From robertwb at math.washington.edu  Wed Apr  1 01:38:21 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 31 Mar 2009 16:38:21 -0700
Subject: [Cython] Package names
In-Reply-To: <e7ba66e40903311348s75e6b7c2he541bef6e6a1e48a@mail.gmail.com>
References: <49D137CC.8070707@student.matnat.uio.no>
	<e7ba66e40903301600y5f3e167w4795df9ec9ea23a0@mail.gmail.com>
	<49D25E4A.8040309@behnel.de>
	<e7ba66e40903311348s75e6b7c2he541bef6e6a1e48a@mail.gmail.com>
Message-ID: <4207A8C5-E0B4-4223-A614-BB0DABF51E9B@math.washington.edu>

On Mar 31, 2009, at 1:48 PM, Lisandro Dalcin wrote:

> On Tue, Mar 31, 2009 at 3:17 PM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>>
>> Lisandro Dalcin wrote:
>>>> 2) Should all the package names be converted to lowercase?
>>>
>>> Perhaps we should follow PEP 8 ?
>>
>> Sure. When we break stuff, make sure we only have to do it once.  
>> That also
>> includes asking on python-dev what a suitable target package would  
>> be for
>> the stdlib.

+1

>>>> 3) Should one perhaps distribute Cython under two seperate package
>>>> names, so that a standalone version can coexist with the one in  
>>>> stdlib?
>>>> This would facilitate easy installation and usage of non-stdlib  
>>>> Cython
>>>> releases. Can't come up with any good names though.
>>>
>>> 1) cython
>>> 2) cython.core
>>> 3) cython.pyximport
>>
>> Yes, one package for everything. So, what you mean is: rename the  
>> current
>> Cython package to "cython.core"? Would that be for the stdlib or  
>> for the
>> external package?
>>
>
> Both stdlib and external :-) ... but doing 'import cython' should not
> have the side-effect of importing 'cython.core' ... Then we could
> decide that stdlib cython.__version__ is old for a project, and
> hot-fix "sys.path" to get a newer Cython, perhaps from a ZIP file.

One thing I just realized is that we use (by necessity) some absolute  
imports in the cython library, which will make having separate names  
rather a pain (though manageable). Are there any other packages that  
have both an internal and external version that we could learn from?

>>>  and also have cython/run.py for cmd line work with -m
>>> switch, and make __init__py use that for newer Python versions that
>>> let you run packages.
>>
>> Note that those aren't that "new" anymore. What do people think about
>> dropping the "cython" frontend alltogether, and making
>>
>>    python -m cython
>>
>> the official way to start the Cython compiler? I doubt that there  
>> are still
>> a lot of developers who do not have access to Python 2.4 or later  
>> in one
>> way or another.
>
> Stefan, the -m switch is know to be broken in Python 2.4. It does not
> work for packages. See yourself:
>
> $ python2.5 -m distutils # the output is nothing
>
> $ python2.4 -m distutils
> python2.4: module distutils has no associated file
>
> that is, 2.4 does not handle __init__.py as the associated file for  
> a package..

This is motivation to keep the frontend around, though we should  
still try and keep as much of the logic outside of that file as  
possible.

- Robert


From robertwb at math.washington.edu  Wed Apr  1 01:46:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 31 Mar 2009 16:46:40 -0700
Subject: [Cython] Naive build system
In-Reply-To: <1f077e770903310820p73c3d014pc7b70ff29f4fb6c7@mail.gmail.com>
References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com>
	<e7ba66e40903310716t436dba00n34ec39f234b38854@mail.gmail.com>
	<1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com>
	<1f077e770903310820p73c3d014pc7b70ff29f4fb6c7@mail.gmail.com>
Message-ID: <F922B7D2-4655-48E1-B205-A43E3D447B30@math.washington.edu>

On Mar 31, 2009, at 8:20 AM, Nicolas Dumazet wrote:

> 2009/4/1 Nicolas Dumazet <nicdumz at gmail.com>:
>> I looked at ways to avoid the extra setup.py work, it seems  
>> there's no
>> "easy way" here.
>> I thought about extending the distutils build command to build the c
>> files before the module build, but it's definitely not simple: if the
>> C files are correctly generated, one would need to register the C
>> extensions on the fly, once the build process has already started;  
>> and
>> I'm not sure that this is possible.
>> And yes, if it is not possible, one would need to write a complex
>> build system handling the whole compilation chain.
>
> Or... the setup.py behavior could be modified:
> When --no-cython-compile is not provided, if the system is not able to
> generate the .c, it bails out, spitting out an error, instead of
> trying to back out "intelligently" to the pure python implementation
> by itself.
>
> This would make the setup.py implementation much more simple:
> * If --no-cython-compile is here, there's almost nothing to do.
> * If it's not here, add the dependences on the C extensions, extend
> the build_ext distutils command to generate the C files before
> building the cython module, (this is actually easy to do) and...
> voil?! Just exit with an error if you can't generate the C files,
> suggest to try --no-cython-compile instead.

I'm -1 on this idea, because it complicates auto-installation of  
packages (for example, if Cython is downloaded via a package manager,  
especially as a dependance to something else). Regarding Scons, it's  
a nice build system, but I'd rather not make it a dependancy for  
Cython (especially as our build needs really aren't super complicated).

Note also that the implict Cython dependancies (i.e. cimports) are  
not handled by this patch either. Perhaps in our case a simple "if  
the .c files are present and all newer than the .pyx files, skip the  
cythonization step."

- Robert


From dalcinl at gmail.com  Wed Apr  1 02:45:33 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 31 Mar 2009 21:45:33 -0300
Subject: [Cython] code documentation tool
In-Reply-To: <49D2904E.2020702@gmail.com>
References: <49CFF3CB.2000408@gmail.com>
	<5b8d13220903291934w69a8cac9y1af6e1722eb4ddca@mail.gmail.com>
	<49D2904E.2020702@gmail.com>
Message-ID: <e7ba66e40903311745v1b0629e1s26694140cd3f5fdd@mail.gmail.com>

On Tue, Mar 31, 2009 at 6:51 PM, Martin Gysel <m.gysel at gmail.com> wrote:
>
> AFAIK sphinx gets the info for autodoc from the docstring but how does
> this work for you because my docstrings are removed in the module
> (according the cython doc, that's the normal behavior).
>

What are you talking about, Martin? Cython does not removes any
docstring (with a few exceptions for some special methods). However,
perhaps you need to asks Cython to auto-generate signatures for you,
just add the commented line at the very beginning of your modules:

#cython: embedsignature=True




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Apr  1 08:54:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 01 Apr 2009 08:54:07 +0200
Subject: [Cython] Refcount question: SetAttrString
In-Reply-To: <49D27184.9020103@student.matnat.uio.no>
References: <49D27184.9020103@student.matnat.uio.no>
Message-ID: <49D30F8F.7090809@behnel.de>


Dag Sverre Seljebotn wrote:
> While ripping out the old temp system (yay!) ...

Thanks for doing that, BTW. This was sooo long overdue.

Stefan



From dagss at student.matnat.uio.no  Wed Apr  1 22:15:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 1 Apr 2009 22:15:04 +0200 (CEST)
Subject: [Cython] Naive build system
In-Reply-To: <F922B7D2-4655-48E1-B205-A43E3D447B30@math.washington.edu>
References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com>
	<e7ba66e40903310716t436dba00n34ec39f234b38854@mail.gmail.com>
	<1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com>
	<1f077e770903310820p73c3d014pc7b70ff29f4fb6c7@mail.gmail.com>
	<F922B7D2-4655-48E1-B205-A43E3D447B30@math.washington.edu>
Message-ID: <070974f7c81c88231be5797308c5ce81.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Mar 31, 2009, at 8:20 AM, Nicolas Dumazet wrote:
>
>> 2009/4/1 Nicolas Dumazet <nicdumz at gmail.com>:
>>> I looked at ways to avoid the extra setup.py work, it seems
>>> there's no
>>> "easy way" here.
>>> I thought about extending the distutils build command to build the c
>>> files before the module build, but it's definitely not simple: if the
>>> C files are correctly generated, one would need to register the C
>>> extensions on the fly, once the build process has already started;
>>> and
>>> I'm not sure that this is possible.
>>> And yes, if it is not possible, one would need to write a complex
>>> build system handling the whole compilation chain.
>>
>> Or... the setup.py behavior could be modified:
>> When --no-cython-compile is not provided, if the system is not able to
>> generate the .c, it bails out, spitting out an error, instead of
>> trying to back out "intelligently" to the pure python implementation
>> by itself.
>>
>> This would make the setup.py implementation much more simple:
>> * If --no-cython-compile is here, there's almost nothing to do.
>> * If it's not here, add the dependences on the C extensions, extend
>> the build_ext distutils command to generate the C files before
>> building the cython module, (this is actually easy to do) and...
>> voil?! Just exit with an error if you can't generate the C files,
>> suggest to try --no-cython-compile instead.
>
> I'm -1 on this idea, because it complicates auto-installation of
> packages (for example, if Cython is downloaded via a package manager,
> especially as a dependance to something else). Regarding Scons, it's
> a nice build system, but I'd rather not make it a dependancy for
> Cython (especially as our build needs really aren't super complicated).

About SCons, iI meant that as an apropos and strictly about support for
end users applications. I.e. it's a matter of finding and bundling an
SCons plugin for Cython in the Tools directory, like we ship an emacs
mode. (And perhaps pointing our users towards SCons rather than investing
too much effort in improving Cython builds using the Cython distutils
plugin; again not for Cython itself but for Cython users.)

Dag Sverre


From m.gysel at gmail.com  Wed Apr  1 23:24:17 2009
From: m.gysel at gmail.com (Martin Gysel)
Date: Wed, 01 Apr 2009 23:24:17 +0200
Subject: [Cython] code documentation tool
In-Reply-To: <e7ba66e40903311745v1b0629e1s26694140cd3f5fdd@mail.gmail.com>
References: <49CFF3CB.2000408@gmail.com>	<5b8d13220903291934w69a8cac9y1af6e1722eb4ddca@mail.gmail.com>	<49D2904E.2020702@gmail.com>
	<e7ba66e40903311745v1b0629e1s26694140cd3f5fdd@mail.gmail.com>
Message-ID: <49D3DB81.2070106@gmail.com>

Lisandro Dalcin schrieb:
> On Tue, Mar 31, 2009 at 6:51 PM, Martin Gysel <m.gysel at gmail.com> wrote:
>   
>> AFAIK sphinx gets the info for autodoc from the docstring but how does
>> this work for you because my docstrings are removed in the module
>> (according the cython doc, that's the normal behavior).
>>     
>
> What are you talking about, Martin? Cython does not removes any
> docstring (with a few exceptions for some special methods). However,
> perhaps you need to asks Cython to auto-generate signatures for you,
> just add the commented line at the very beginning of your modules:
>
> #cython: embedsignature=True
>   
thx for the hint, now everything works as expected

/martin


From robertwb at math.washington.edu  Thu Apr  2 00:23:58 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Apr 2009 15:23:58 -0700
Subject: [Cython] Refcount question: SetAttrString
In-Reply-To: <49D30F8F.7090809@behnel.de>
References: <49D27184.9020103@student.matnat.uio.no>
	<49D30F8F.7090809@behnel.de>
Message-ID: <DDDDB27A-82CC-40E4-AD6B-31F793E909BD@math.washington.edu>

On Mar 31, 2009, at 11:54 PM, Stefan Behnel wrote:

> Dag Sverre Seljebotn wrote:
>> While ripping out the old temp system (yay!) ...
>
> Thanks for doing that, BTW. This was sooo long overdue.

I noticed these changes in the -unstable branch too. Yay!

- Robert


From robertwb at math.washington.edu  Thu Apr  2 00:25:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Apr 2009 15:25:25 -0700
Subject: [Cython] Naive build system
In-Reply-To: <070974f7c81c88231be5797308c5ce81.squirrel@webmail.uio.no>
References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com>
	<e7ba66e40903310716t436dba00n34ec39f234b38854@mail.gmail.com>
	<1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com>
	<1f077e770903310820p73c3d014pc7b70ff29f4fb6c7@mail.gmail.com>
	<F922B7D2-4655-48E1-B205-A43E3D447B30@math.washington.edu>
	<070974f7c81c88231be5797308c5ce81.squirrel@webmail.uio.no>
Message-ID: <8F086E2D-C82F-4723-A073-52A9F235A01F@math.washington.edu>

On Apr 1, 2009, at 1:15 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Mar 31, 2009, at 8:20 AM, Nicolas Dumazet wrote:
>>
>>> 2009/4/1 Nicolas Dumazet <nicdumz at gmail.com>:
>>>> I looked at ways to avoid the extra setup.py work, it seems
>>>> there's no
>>>> "easy way" here.
>>>> I thought about extending the distutils build command to build  
>>>> the c
>>>> files before the module build, but it's definitely not simple:  
>>>> if the
>>>> C files are correctly generated, one would need to register the C
>>>> extensions on the fly, once the build process has already started;
>>>> and
>>>> I'm not sure that this is possible.
>>>> And yes, if it is not possible, one would need to write a complex
>>>> build system handling the whole compilation chain.
>>>
>>> Or... the setup.py behavior could be modified:
>>> When --no-cython-compile is not provided, if the system is not  
>>> able to
>>> generate the .c, it bails out, spitting out an error, instead of
>>> trying to back out "intelligently" to the pure python implementation
>>> by itself.
>>>
>>> This would make the setup.py implementation much more simple:
>>> * If --no-cython-compile is here, there's almost nothing to do.
>>> * If it's not here, add the dependences on the C extensions, extend
>>> the build_ext distutils command to generate the C files before
>>> building the cython module, (this is actually easy to do) and...
>>> voil?! Just exit with an error if you can't generate the C files,
>>> suggest to try --no-cython-compile instead.
>>
>> I'm -1 on this idea, because it complicates auto-installation of
>> packages (for example, if Cython is downloaded via a package manager,
>> especially as a dependance to something else). Regarding Scons, it's
>> a nice build system, but I'd rather not make it a dependancy for
>> Cython (especially as our build needs really aren't super  
>> complicated).
>
> About SCons, iI meant that as an apropos and strictly about support  
> for
> end users applications. I.e. it's a matter of finding and bundling an
> SCons plugin for Cython in the Tools directory, like we ship an emacs
> mode. (And perhaps pointing our users towards SCons rather than  
> investing
> too much effort in improving Cython builds using the Cython distutils
> plugin; again not for Cython itself but for Cython users.)

Ah, thanks for the clarification. Yes, this would be nice to have.

- Robert


From cournape at gmail.com  Thu Apr  2 05:02:17 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 2 Apr 2009 12:02:17 +0900
Subject: [Cython] Naive build system
In-Reply-To: <070974f7c81c88231be5797308c5ce81.squirrel@webmail.uio.no>
References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com>
	<e7ba66e40903310716t436dba00n34ec39f234b38854@mail.gmail.com>
	<1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com>
	<1f077e770903310820p73c3d014pc7b70ff29f4fb6c7@mail.gmail.com>
	<F922B7D2-4655-48E1-B205-A43E3D447B30@math.washington.edu>
	<070974f7c81c88231be5797308c5ce81.squirrel@webmail.uio.no>
Message-ID: <5b8d13220904012002n7eb55a77u977040a4c62818be@mail.gmail.com>

On Thu, Apr 2, 2009 at 5:15 AM, Dag Sverre Seljebotn

> About SCons, iI meant that as an apropos and strictly about support for
> end users applications. I.e. it's a matter of finding and bundling an
> SCons plugin for Cython in the Tools directory, like we ship an emacs
> mode.

I would not mind giving away my cython tool to cython - the tool in
numscons are almost all independent of numscons, and can be used
directly from scons,

cheers,

David

From kwmsmith at gmail.com  Thu Apr  2 22:55:47 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 2 Apr 2009 15:55:47 -0500
Subject: [Cython] What happened to cython-devel repo?
Message-ID: <c266a7930904021355n6ac0201dk3f03055d563107a1@mail.gmail.com>

Hi,

I tried updating my cython-devel, but no luck:

[506]$ pwd
/home/ksmith/Devel/cython/cython-devel

[507]$ hg pull
pulling from http://hg.cython.org/cython-devel
abort: 'http://hg.cython.org/cython-devel' does not appear to be an hg
repository!

Pulling from cython-unstable works, though, and it seems to be the most current.

Kurt

From robertwb at math.washington.edu  Thu Apr  2 23:02:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Apr 2009 14:02:19 -0700
Subject: [Cython] What happened to cython-devel repo?
In-Reply-To: <c266a7930904021355n6ac0201dk3f03055d563107a1@mail.gmail.com>
References: <c266a7930904021355n6ac0201dk3f03055d563107a1@mail.gmail.com>
Message-ID: <BB353352-12B1-4EE4-8863-485D3B2BDF26@math.washington.edu>

In transplanting some patches from -unstable to -devel, I messed  
something up (basically, it was just really messy) and so I took it  
down for a sec to roll it back. Looks like you tried to pull in the  
couple of minutes it was down.

Please try again.

- Robert

On Apr 2, 2009, at 1:55 PM, Kurt Smith wrote:

> Hi,
>
> I tried updating my cython-devel, but no luck:
>
> [506]$ pwd
> /home/ksmith/Devel/cython/cython-devel
>
> [507]$ hg pull
> pulling from http://hg.cython.org/cython-devel
> abort: 'http://hg.cython.org/cython-devel' does not appear to be an hg
> repository!
>
> Pulling from cython-unstable works, though, and it seems to be the  
> most current.
>
> Kurt
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From kwmsmith at gmail.com  Thu Apr  2 23:13:59 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 2 Apr 2009 16:13:59 -0500
Subject: [Cython] What happened to cython-devel repo?
In-Reply-To: <BB353352-12B1-4EE4-8863-485D3B2BDF26@math.washington.edu>
References: <c266a7930904021355n6ac0201dk3f03055d563107a1@mail.gmail.com>
	<BB353352-12B1-4EE4-8863-485D3B2BDF26@math.washington.edu>
Message-ID: <c266a7930904021413y12c9c220t3cb2879b2da4f459@mail.gmail.com>

On Thu, Apr 2, 2009 at 4:02 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> In transplanting some patches from -unstable to -devel, I messed
> something up (basically, it was just really messy) and so I took it
> down for a sec to roll it back. Looks like you tried to pull in the
> couple of minutes it was down.
>
> Please try again.

Thanks -- I'll be more patient next time ;-)

From robertwb at math.washington.edu  Thu Apr  2 23:19:15 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Apr 2009 14:19:15 -0700
Subject: [Cython] What happened to cython-devel repo?
In-Reply-To: <c266a7930904021413y12c9c220t3cb2879b2da4f459@mail.gmail.com>
References: <c266a7930904021355n6ac0201dk3f03055d563107a1@mail.gmail.com>
	<BB353352-12B1-4EE4-8863-485D3B2BDF26@math.washington.edu>
	<c266a7930904021413y12c9c220t3cb2879b2da4f459@mail.gmail.com>
Message-ID: <DEF64E94-75B7-45CD-B27E-95A7927A8FB9@math.washington.edu>

On Apr 2, 2009, at 2:13 PM, Kurt Smith wrote:

> On Thu, Apr 2, 2009 at 4:02 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> In transplanting some patches from -unstable to -devel, I messed
>> something up (basically, it was just really messy) and so I took it
>> down for a sec to roll it back. Looks like you tried to pull in the
>> couple of minutes it was down.
>>
>> Please try again.
>
> Thanks -- I'll be more patient next time ;-)

It's very rare for it to be down--last time was due to moving the  
server to a new machine, so yes, please ping the list if you notice  
problems :).

- Robert


From dalcinl at gmail.com  Fri Apr  3 00:43:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Apr 2009 19:43:15 -0300
Subject: [Cython] new ISO C99 keywords
Message-ID: <e7ba66e40904021543q707cc419x75629091d5bc6513@mail.gmail.com>

Robert, if you add the list below, we get C99..., and you could rename
"ansi_c_keywords" to "iso_c99_keywords".

_Bool
_Complex
_Imaginary
inline
restrict



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Apr  3 00:54:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Apr 2009 15:54:27 -0700
Subject: [Cython] new ISO C99 keywords
In-Reply-To: <e7ba66e40904021543q707cc419x75629091d5bc6513@mail.gmail.com>
References: <e7ba66e40904021543q707cc419x75629091d5bc6513@mail.gmail.com>
Message-ID: <31011F93-8203-42FC-A96E-567B7DCDE892@math.washington.edu>

Sure, I could do that.

On Apr 2, 2009, at 3:43 PM, Lisandro Dalcin wrote:

> Robert, if you add the list below, we get C99..., and you could rename
> "ansi_c_keywords" to "iso_c99_keywords".
>
> _Bool
> _Complex
> _Imaginary
> inline
> restrict
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Fri Apr  3 01:26:20 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Apr 2009 20:26:20 -0300
Subject: [Cython] PATCH: related to ticket #252, special case __weakref__
Message-ID: <e7ba66e40904021626q46b167d7l42b1f39e2d438334@mail.gmail.com>

Robert, likely you forgot about this detail releated weakrefs. Perhaps
the special-case should be done elsewhere?

diff -r 5cabe56c76ab Cython/Compiler/Symtab.py
--- a/Cython/Compiler/Symtab.py	Wed Mar 25 14:23:29 2009 -0700
+++ b/Cython/Compiler/Symtab.py	Thu Apr 02 20:23:49 2009 -0300
@@ -32,7 +32,10 @@

 def c_safe_identifier(cname):
     # There are some C limitations on struct entry names.
-    if (cname[:2] == '__' and not
cname.startswith(Naming.pyrex_prefix)) or cname in ansi_c_keywords:
+    if ((cname[:2] == '__'
+         and not (cname.startswith(Naming.pyrex_prefix)
+                  or cname == '__weakref__'))
+        or  cname in ansi_c_keywords):
         cname = Naming.pyrex_prefix + cname
     return cname


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Apr  3 01:39:51 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Apr 2009 20:39:51 -0300
Subject: [Cython] new patch for #255 (Py->C for integrals) in the tracker
Message-ID: <e7ba66e40904021639r397e1c5dg6063b0c88176ff54@mail.gmail.com>

- The changes to ModuleNode.py and Utils.py are just quick hacks to
get it working.

- I've tried hard to avoid generation of unreachable code like these
"if (0) {...}" blocks.

- Once this is reviewed, commented and eventually accepted, I'll
continue extending this to extern ctypedef integrals.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Apr  3 01:43:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Apr 2009 16:43:42 -0700
Subject: [Cython] PATCH: related to ticket #252, special case __weakref__
In-Reply-To: <e7ba66e40904021626q46b167d7l42b1f39e2d438334@mail.gmail.com>
References: <e7ba66e40904021626q46b167d7l42b1f39e2d438334@mail.gmail.com>
Message-ID: <E70AAD25-536A-49B8-995D-4287724D06D1@math.washington.edu>

On Apr 2, 2009, at 4:26 PM, Lisandro Dalcin wrote:

> Robert, likely you forgot about this detail releated weakrefs. Perhaps
> the special-case should be done elsewhere?
>
> diff -r 5cabe56c76ab Cython/Compiler/Symtab.py
> --- a/Cython/Compiler/Symtab.py	Wed Mar 25 14:23:29 2009 -0700
> +++ b/Cython/Compiler/Symtab.py	Thu Apr 02 20:23:49 2009 -0300
> @@ -32,7 +32,10 @@
>
>  def c_safe_identifier(cname):
>      # There are some C limitations on struct entry names.
> -    if (cname[:2] == '__' and not
> cname.startswith(Naming.pyrex_prefix)) or cname in ansi_c_keywords:
> +    if ((cname[:2] == '__'
> +         and not (cname.startswith(Naming.pyrex_prefix)
> +                  or cname == '__weakref__'))
> +        or  cname in ansi_c_keywords):
>          cname = Naming.pyrex_prefix + cname
>      return cname

Oops, yes. Please push.

- Robert


From robertwb at math.washington.edu  Fri Apr  3 03:52:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Apr 2009 18:52:25 -0700
Subject: [Cython] new patch for #255 (Py->C for integrals) in the tracker
In-Reply-To: <e7ba66e40904021639r397e1c5dg6063b0c88176ff54@mail.gmail.com>
References: <e7ba66e40904021639r397e1c5dg6063b0c88176ff54@mail.gmail.com>
Message-ID: <80E1EA88-7452-461E-8EA6-CF788CA4909A@math.washington.edu>

On Apr 2, 2009, at 4:39 PM, Lisandro Dalcin wrote:

> - The changes to ModuleNode.py and Utils.py are just quick hacks to
> get it working.
>
> - I've tried hard to avoid generation of unreachable code like these
> "if (0) {...}" blocks.
>
> - Once this is reviewed, commented and eventually accepted, I'll
> continue extending this to extern ctypedef integrals.

I've taken a look at the code and it looks good to me. You took care  
of every corner case I thought of checking. I haven't tested it under  
Py3, but it works great under Py2. It might be worth noting that I  
improved the utility code templating to take a type since this is  
such a common case: http://hg.cython.org/cython-devel/file/ 
2289ab00261d/Cython/Utils.py

One comment I had is that in your code

static INLINE TYPE __Pyx_PyInt_AsTYPE(PyObject* x) {
     if (sizeof(TYPE) < sizeof(long)) {
         long val = __Pyx_PyInt_AsLong(x);
         if (unlikely(val == -1 && PyErr_Occurred()))
             return (TYPE)-1;
         if (unlikely(val < 0)) {
             PyErr_SetString(PyExc_OverflowError,
                             "can't convert negative value to TYPE");
             return (TYPE)-1;
         }
         if (unlikely(val != (long)(TYPE)val)) {
             PyErr_SetString(PyExc_OverflowError,
                             "value too large to convert to TYPE");
             return (TYPE)-1;
         }
         return (TYPE)val;
     }
     return (TYPE)__Pyx_PyInt_AsUnsignedLong(x);
}

I believe the successful pathway could be shorted by doing

static INLINE TYPE __Pyx_PyInt_AsTYPE(PyObject* x) {
     if (sizeof(TYPE) < sizeof(long)) {
         long val = __Pyx_PyInt_AsLong(x);
         if (unlikely(val != (long)(TYPE)val)) {
             if (unlikely(val == -1 && PyErr_Occurred()));
             else if (val < 0) {
                 PyErr_SetString(PyExc_OverflowError,
                                 "can't convert negative value to  
TYPE");
             }
             else {
                 PyErr_SetString(PyExc_OverflowError,
                                 "value too large to convert to TYPE");
             }
             return (TYPE)-1;
         }
         return (TYPE)val;
     }
     return (TYPE)__Pyx_PyInt_AsUnsignedLong(x);
}

as sizeof(TYPE) < sizeof(long) guarantees val != (long)(TYPE)val for  
negative val on unsigned TYPE and the error does not need to be  
explicitly tested for otherwise (as the calling function will check  
PyErr_Occurred on a return value of -1).

- Robert


From dsurviver at gmail.com  Fri Apr  3 04:57:07 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Thu, 2 Apr 2009 23:57:07 -0300
Subject: [Cython] [GSoC Proposal] Improve C++ Support
Message-ID: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>

Hi, all.

I just wrote a proposal for GSoC, and I'm posting it here to hav
feedbacks about it.

Here is the link with the wiki with my proposal:
http://wiki.cython.org/gsoc09/daniloaf

Fell free to comment about it.

Thanks


Danilo Freitas

From robertwb at math.washington.edu  Fri Apr  3 05:39:22 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Apr 2009 20:39:22 -0700
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
Message-ID: <8017B8C9-5681-43CE-9024-886D13281783@math.washington.edu>

On Apr 2, 2009, at 7:57 PM, Danilo Freitas wrote:

> Hi, all.
>
> I just wrote a proposal for GSoC, and I'm posting it here to hav
> feedbacks about it.
>
> Here is the link with the wiki with my proposal:
> http://wiki.cython.org/gsoc09/daniloaf
>
> Fell free to comment about it.

First off, I would like to say I think this would be a very good  
project. I would clarify that the changes made to the Cython language  
are not to make natively it know about STL, but rather extending the  
syntax to more easily wrap polymorphic and templated functions/ 
classes so one can use them from Cython. Being able to wrap all of  
the STL library is a very useful and concrete goal.

Note that the application period is drawing to a close tomorrow, but  
feedback can be taken and the application updated until the close of  
the filtering process.

Hopefully we will have a good, pythonic syntax mostly nailed down for  
doing this before the coding period starts.

- Robert


From dagss at student.matnat.uio.no  Fri Apr  3 09:31:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 3 Apr 2009 09:31:35 +0200 (CEST)
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
Message-ID: <fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>

Danilo Freitas wrote:
> Hi, all.
>
> I just wrote a proposal for GSoC, and I'm posting it here to hav
> feedbacks about it.
>
> Here is the link with the wiki with my proposal:
> http://wiki.cython.org/gsoc09/daniloaf

Hi! Here's some feedback (these are my opinion only and not necesarrily of
others here). As Robert says though, please apply *now* and we can talk
about and edit the application later.

I wrote some notes a year ago on http://wiki.cython.org/enhancements/cpp ,
you could have a look (thopugh they are outdated).

Anyway, I see such a project as a) identifying and b) solving a number of
trouble spots for C++ users. Here's my list. I suppose not all of this
would have to be in a gsoc, it's just some feedback for consideration.

1) Using templates -- you already have this. My preferred syntax is using
[] BTW. As Robert said, it should support any template and not only STL.

2) Operator overloading -- you already have this as well. This should be
made generic, so that all C++ operators which have a corresponding Cython
operator can be used on a type.

3) References. Currently there's a problem (I didn't actually try this,
just assuming this from how I know things work):

If you make this C++ function:

void incargs(int& a, int& b) { a++; b++; }

then it has to be declared in Cython like this:

cdef extern void incargs(int a, int b)

and a call like this would result in:

cdef int a = 3
cdef object b = 3
incargs(a, b)
# now a is 4 and b is 3 !!!

My preferred solution for this dilemma would be to declare C++ references
Cython-side and require pointers to be passed (which are then dereferenced
prior to the call), because that's consistent with Cython semantics. I
find it too confusing syntax-wise that "a" above can change value in
Cython.

4) Calling constructors/destructors.

This is an interesting problem. C++ supports both stack and heap
allocation, while Cython/Python (and e.g. Java) only supports heap
allocation (of objects with constructors/destructors).

For instance, one cannot do this in Cython currently: Instead I would have
this 3-step plan (of which perhaps only the first one is gsoc):

a) Introduce syntax for declaring external C++ classes. Such classes can
only be used as pointers; and can be heap-allocated through new
cython.new/cython.delete magic functions.

b) Add support for this, using the non-pointer syntax for the type:

cdef cpp.std.vectot[int] vec = cpp.std.vectot[int](4) # construct
...
vec = something_else # destruct

This requires reference counting the C++ class. This would result in a
Pythonic feel.

c) Use flow control analysis/live variable analysis to optimize b) and
allocate the variables that can on the stack while preserving the same
semantics.

Dag Sverre


From dagss at student.matnat.uio.no  Fri Apr  3 09:56:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Apr 2009 09:56:15 +0200
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
Message-ID: <49D5C11F.3030700@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Danilo Freitas wrote:
>> Hi, all.
>>
>> I just wrote a proposal for GSoC, and I'm posting it here to hav
>> feedbacks about it.
>>
>> Here is the link with the wiki with my proposal:
>> http://wiki.cython.org/gsoc09/daniloaf
> 
> Hi! Here's some feedback (these are my opinion only and not necesarrily of
> others here). As Robert says though, please apply *now* and we can talk
> about and edit the application later.
> 
> I wrote some notes a year ago on http://wiki.cython.org/enhancements/cpp ,
> you could have a look (thopugh they are outdated).
> 
> Anyway, I see such a project as a) identifying and b) solving a number of
> trouble spots for C++ users. Here's my list. I suppose not all of this
> would have to be in a gsoc, it's just some feedback for consideration.
> 
> 1) Using templates -- you already have this. My preferred syntax is using
> [] BTW. As Robert said, it should support any template and not only STL.
> 
> 2) Operator overloading -- you already have this as well. This should be
> made generic, so that all C++ operators which have a corresponding Cython
> operator can be used on a type.
> 
> 3) References. Currently there's a problem (I didn't actually try this,
> just assuming this from how I know things work):
> 
> If you make this C++ function:
> 
> void incargs(int& a, int& b) { a++; b++; }
> 
> then it has to be declared in Cython like this:
> 
> cdef extern void incargs(int a, int b)
> 
> and a call like this would result in:
> 
> cdef int a = 3
> cdef object b = 3
> incargs(a, b)
> # now a is 4 and b is 3 !!!
> 
> My preferred solution for this dilemma would be to declare C++ references
> Cython-side and require pointers to be passed (which are then dereferenced
> prior to the call), because that's consistent with Cython semantics. I
> find it too confusing syntax-wise that "a" above can change value in
> Cython.
> 
> 4) Calling constructors/destructors.
> 
> This is an interesting problem. C++ supports both stack and heap
> allocation, while Cython/Python (and e.g. Java) only supports heap
> allocation (of objects with constructors/destructors).
> 
> For instance, one cannot do this in Cython currently: Instead I would have
> this 3-step plan (of which perhaps only the first one is gsoc):

Something fell out here, sorry about that. Here it is:

For instance, one cannot do this in Cython currently:

cdef my_std_vector_int_iterator it = myvec.begin()

because an STL iterator cannot be constructed without arguments, and 
when that code is Cythonized, it roughly ends up like this:

my_std_vector_int_iterator it; # all declarations at top
...
it = myvec.begin()

(Note that assignment-on-declaration is the same as passing constructor 
arguments in C++).

One solution would be to add a syntax for stack allocation, however that 
interferes with lots of Python rules (variable scoping for instance).

So instead I would have this 3-step plan [...]


-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Apr  3 09:53:47 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 00:53:47 -0700
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
Message-ID: <892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>

On Apr 3, 2009, at 12:31 AM, Dag Sverre Seljebotn wrote:

> Danilo Freitas wrote:
>> Hi, all.
>>
>> I just wrote a proposal for GSoC, and I'm posting it here to hav
>> feedbacks about it.
>>
>> Here is the link with the wiki with my proposal:
>> http://wiki.cython.org/gsoc09/daniloaf
>
> Hi! Here's some feedback (these are my opinion only and not  
> necesarrily of
> others here). As Robert says though, please apply *now* and we can  
> talk
> about and edit the application later.
>
> I wrote some notes a year ago on http://wiki.cython.org/ 
> enhancements/cpp ,
> you could have a look (thopugh they are outdated).
>
> Anyway, I see such a project as a) identifying and b) solving a  
> number of
> trouble spots for C++ users. Here's my list. I suppose not all of this
> would have to be in a gsoc, it's just some feedback for consideration.
>
> 1) Using templates -- you already have this. My preferred syntax is  
> using
> [] BTW. As Robert said, it should support any template and not only  
> STL.
>
> 2) Operator overloading -- you already have this as well. This  
> should be
> made generic, so that all C++ operators which have a corresponding  
> Cython
> operator can be used on a type.
>
> 3) References. Currently there's a problem (I didn't actually try  
> this,
> just assuming this from how I know things work):
>
> If you make this C++ function:
>
> void incargs(int& a, int& b) { a++; b++; }
>
> then it has to be declared in Cython like this:
>
> cdef extern void incargs(int a, int b)
>
> and a call like this would result in:
>
> cdef int a = 3
> cdef object b = 3
> incargs(a, b)
> # now a is 4 and b is 3 !!!
>
> My preferred solution for this dilemma would be to declare C++  
> references
> Cython-side and require pointers to be passed (which are then  
> dereferenced
> prior to the call), because that's consistent with Cython semantics. I
> find it too confusing syntax-wise that "a" above can change value in
> Cython.

+1

> 4) Calling constructors/destructors.
>
> This is an interesting problem. C++ supports both stack and heap
> allocation, while Cython/Python (and e.g. Java) only supports heap
> allocation (of objects with constructors/destructors).
>
> For instance, one cannot do this in Cython currently: Instead I  
> would have
> this 3-step plan (of which perhaps only the first one is gsoc):
>
> a) Introduce syntax for declaring external C++ classes. Such  
> classes can
> only be used as pointers; and can be heap-allocated through new
> cython.new/cython.delete magic functions.
>
> b) Add support for this, using the non-pointer syntax for the type:
>
> cdef cpp.std.vectot[int] vec = cpp.std.vectot[int](4) # construct
> ...
> vec = something_else # destruct
>
> This requires reference counting the C++ class. This would result in a
> Pythonic feel.
>
> c) Use flow control analysis/live variable analysis to optimize b) and
> allocate the variables that can on the stack while preserving the same
> semantics.

Note that the allocate-on-the-stack is not specific to just stacks.  
For example, it is possible to define a C++ class as a member of a  
cdef class, and then its memory is allocated along with the object as  
part of the struct. Currently, one then has to actually initialize  
and destroy manually.

- Robert



From dagss at student.matnat.uio.no  Fri Apr  3 10:59:19 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Apr 2009 10:59:19 +0200
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
Message-ID: <49D5CFE7.6070100@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 12:31 AM, Dag Sverre Seljebotn wrote:
> 
>> Danilo Freitas wrote:
>>> Hi, all.
>>>
>>> I just wrote a proposal for GSoC, and I'm posting it here to hav
>>> feedbacks about it.
>>>
>>> Here is the link with the wiki with my proposal:
>>> http://wiki.cython.org/gsoc09/daniloaf
>> Hi! Here's some feedback (these are my opinion only and not  
>> necesarrily of
>> others here). As Robert says though, please apply *now* and we can  
>> talk
>> about and edit the application later.
>>
>> I wrote some notes a year ago on http://wiki.cython.org/ 
>> enhancements/cpp ,
>> you could have a look (thopugh they are outdated).
>>
>> Anyway, I see such a project as a) identifying and b) solving a  
>> number of
>> trouble spots for C++ users. Here's my list. I suppose not all of this
>> would have to be in a gsoc, it's just some feedback for consideration.
>>
>> 1) Using templates -- you already have this. My preferred syntax is  
>> using
>> [] BTW. As Robert said, it should support any template and not only  
>> STL.
>>
>> 2) Operator overloading -- you already have this as well. This  
>> should be
>> made generic, so that all C++ operators which have a corresponding  
>> Cython
>> operator can be used on a type.
>>
>> 3) References. Currently there's a problem (I didn't actually try  
>> this,
>> just assuming this from how I know things work):
>>
>> If you make this C++ function:
>>
>> void incargs(int& a, int& b) { a++; b++; }
>>
>> then it has to be declared in Cython like this:
>>
>> cdef extern void incargs(int a, int b)
>>
>> and a call like this would result in:
>>
>> cdef int a = 3
>> cdef object b = 3
>> incargs(a, b)
>> # now a is 4 and b is 3 !!!
>>
>> My preferred solution for this dilemma would be to declare C++  
>> references
>> Cython-side and require pointers to be passed (which are then  
>> dereferenced
>> prior to the call), because that's consistent with Cython semantics. I
>> find it too confusing syntax-wise that "a" above can change value in
>> Cython.
> 
> +1
> 
>> 4) Calling constructors/destructors.
>>
>> This is an interesting problem. C++ supports both stack and heap
>> allocation, while Cython/Python (and e.g. Java) only supports heap
>> allocation (of objects with constructors/destructors).
>>
>> For instance, one cannot do this in Cython currently: Instead I  
>> would have
>> this 3-step plan (of which perhaps only the first one is gsoc):
>>
>> a) Introduce syntax for declaring external C++ classes. Such  
>> classes can
>> only be used as pointers; and can be heap-allocated through new
>> cython.new/cython.delete magic functions.
>>
>> b) Add support for this, using the non-pointer syntax for the type:
>>
>> cdef cpp.std.vectot[int] vec = cpp.std.vectot[int](4) # construct
>> ...
>> vec = something_else # destruct
>>
>> This requires reference counting the C++ class. This would result in a
>> Pythonic feel.
>>
>> c) Use flow control analysis/live variable analysis to optimize b) and
>> allocate the variables that can on the stack while preserving the same
>> semantics.
> 
> Note that the allocate-on-the-stack is not specific to just stacks.  
> For example, it is possible to define a C++ class as a member of a  
> cdef class, and then its memory is allocated along with the object as  
> part of the struct. Currently, one then has to actually initialize  
> and destroy manually.

Good point. That's actually fairly tricky to do if the class doesn't 
have a no-arg constructor. If you have this:

class MyClass {
    MyClass(int x) {...} // non-empty constructor
};

Then in order to use it in a struct (without going through a pointer) 
one would actually have to do this:

struct foo {
   MyClass myinstance;
public:
   foo() : myinstance(any expression in globals) {}
// or: foo(x,y,z) : myinstance(any expression in x,y,z,globals) {}
}

Getting Cython output code like this is non-trivial, at least without 
directly copying the member-construction-syntax of C++.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Apr  3 11:00:47 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 02:00:47 -0700
Subject: [Cython] Cython 0.11.1 alpha
Message-ID: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>

In the interest of releasing sooner rather than later, a new alpha is  
up at

http://cython.org/Cython-0.11.1.alpha.tar.gz

There is a good patch by Lisandro that does Python <-> c int  
converstion that I'd like to see make it in, and Sage doesn't yet  
compile (mostly due to the "defined before used" bug catching--should  
this be merely a warning before the next major release?), but at this  
point lets let the -devel branch be primarily focused on resolving  
these issues. I anticipate we could have 0.11.1 out within a week  
with nearly two dozen tickets closed.

- Robert

From psp.reachable at gmail.com  Fri Apr  3 11:10:18 2009
From: psp.reachable at gmail.com (Prajwal Suhas P)
Date: Fri, 3 Apr 2009 14:40:18 +0530
Subject: [Cython] [GSoC Proposal] Control flow analysis for Cython
Message-ID: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>

This is my proposal for gsoc.
http://wiki.cython.org/gsoc/prajwal

Please feel free to give suggestions/comment on it.
-- 
Regards,
Prajwal S.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090403/de72a058/attachment.htm 

From dagss at student.matnat.uio.no  Fri Apr  3 11:28:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Apr 2009 11:28:58 +0200
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
Message-ID: <49D5D6DA.9030201@student.matnat.uio.no>

Robert Bradshaw wrote:
> In the interest of releasing sooner rather than later, a new alpha is  
> up at
> 
> http://cython.org/Cython-0.11.1.alpha.tar.gz
> 
> There is a good patch by Lisandro that does Python <-> c int  
> converstion that I'd like to see make it in, and Sage doesn't yet  
> compile (mostly due to the "defined before used" bug catching--should  
> this be merely a warning before the next major release?), but at this  

+1

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Apr  3 11:31:13 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Apr 2009 11:31:13 +0200
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <49D5D6DA.9030201@student.matnat.uio.no>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
	<49D5D6DA.9030201@student.matnat.uio.no>
Message-ID: <49D5D761.10900@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> In the interest of releasing sooner rather than later, a new alpha is  
>> up at
>>
>> http://cython.org/Cython-0.11.1.alpha.tar.gz
>>
>> There is a good patch by Lisandro that does Python <-> c int  
>> converstion that I'd like to see make it in, and Sage doesn't yet  
>> compile (mostly due to the "defined before used" bug catching--should  
>> this be merely a warning before the next major release?), but at this  
> 
> +1
> 

Though as the for loop semantics also changed, one could also argue to 
keep it as it is but name it 0.12.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Apr  3 11:15:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 02:15:25 -0700
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <49D5CFE7.6070100@student.matnat.uio.no>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no>
Message-ID: <5FE1DE4C-BB47-4546-8CC5-B10235F6E455@math.washington.edu>

On Apr 3, 2009, at 1:59 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>
>> Note that the allocate-on-the-stack is not specific to just stacks.
>> For example, it is possible to define a C++ class as a member of a
>> cdef class, and then its memory is allocated along with the object as
>> part of the struct. Currently, one then has to actually initialize
>> and destroy manually.
>
> Good point. That's actually fairly tricky to do if the class doesn't
> have a no-arg constructor.

I believe it can be done (IIRC constructors can take a chunk of  
memory to initialize rather than allocating it themselves). I'm no  
expert on this though.

> If you have this:
>
> class MyClass {
>     MyClass(int x) {...} // non-empty constructor
> };
>
> Then in order to use it in a struct (without going through a pointer)
> one would actually have to do this:
>
> struct foo {
>    MyClass myinstance;
> public:
>    foo() : myinstance(any expression in globals) {}
> // or: foo(x,y,z) : myinstance(any expression in x,y,z,globals) {}
> }
>
> Getting Cython output code like this is non-trivial, at least without
> directly copying the member-construction-syntax of C++.

Yes, and perhaps outside the scope of this project.

- Robert


From robertwb at math.washington.edu  Fri Apr  3 11:19:22 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 02:19:22 -0700
Subject: [Cython] [GSoC Proposal] Control flow analysis for Cython
In-Reply-To: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>
References: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>
Message-ID: <9CE9AECB-5E68-4FC1-9AFC-B375546E0A93@math.washington.edu>

On Apr 3, 2009, at 2:10 AM, Prajwal Suhas P wrote:

> This is my proposal for gsoc.
> http://wiki.cython.org/gsoc/prajwal

Thanks.

- Robert



From stefan_ml at behnel.de  Fri Apr  3 11:21:20 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 3 Apr 2009 11:21:20 +0200 (CEST)
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <49D5D761.10900@student.matnat.uio.no>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
	<49D5D6DA.9030201@student.matnat.uio.no>
	<49D5D761.10900@student.matnat.uio.no>
Message-ID: <b82fe26b85789bb4d050ff1ae9624745.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> In the interest of releasing sooner rather than later, a new alpha is
>>> up at
>>>
>>> http://cython.org/Cython-0.11.1.alpha.tar.gz
>>>
>>> There is a good patch by Lisandro that does Python <-> c int
>>> converstion that I'd like to see make it in, and Sage doesn't yet
>>> compile (mostly due to the "defined before used" bug catching--should
>>> this be merely a warning before the next major release?), but at this
>>
>> +1
>
> Though as the for loop semantics also changed, one could also argue to
> keep it as it is but name it 0.12.

Although, if everyone who needs the bug fixes for 0.11 has to switch to
0.12 anyway, what good is it to not name it 0.11.1? :)

Stefan


From dagss at student.matnat.uio.no  Fri Apr  3 11:39:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Apr 2009 11:39:25 +0200
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <b82fe26b85789bb4d050ff1ae9624745.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>	<49D5D6DA.9030201@student.matnat.uio.no>	<49D5D761.10900@student.matnat.uio.no>
	<b82fe26b85789bb4d050ff1ae9624745.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49D5D94D.9020108@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert Bradshaw wrote:
>>>> In the interest of releasing sooner rather than later, a new alpha is
>>>> up at
>>>>
>>>> http://cython.org/Cython-0.11.1.alpha.tar.gz
>>>>
>>>> There is a good patch by Lisandro that does Python <-> c int
>>>> converstion that I'd like to see make it in, and Sage doesn't yet
>>>> compile (mostly due to the "defined before used" bug catching--should
>>>> this be merely a warning before the next major release?), but at this
>>> +1
>> Though as the for loop semantics also changed, one could also argue to
>> keep it as it is but name it 0.12.
> 
> Although, if everyone who needs the bug fixes for 0.11 has to switch to
> 0.12 anyway, what good is it to not name it 0.11.1? :)

The question is why we have a major/minor naming scheme at all. It could be

a) Depending on how much time's elapsed since the last major. That's 
roughly our current policy :-)

b) Substantial new features means new major. That's one guildeline, 
which speaks for naming this one 0.11.1.

c) New major when backwards compatability is broken in any way. That's 
another, conflicting guideline, which speaks for naming this one 0.12.

Myself I'm -1 on a) and +0 on b) and c).

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Apr  3 11:24:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 02:24:45 -0700
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <49D5D761.10900@student.matnat.uio.no>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
	<49D5D6DA.9030201@student.matnat.uio.no>
	<49D5D761.10900@student.matnat.uio.no>
Message-ID: <056837A0-5F13-43D1-B0A4-A6342CC6746E@math.washington.edu>

On Apr 3, 2009, at 2:31 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> In the interest of releasing sooner rather than later, a new  
>>> alpha is
>>> up at
>>>
>>> http://cython.org/Cython-0.11.1.alpha.tar.gz
>>>
>>> There is a good patch by Lisandro that does Python <-> c int
>>> converstion that I'd like to see make it in, and Sage doesn't yet
>>> compile (mostly due to the "defined before used" bug catching-- 
>>> should
>>> this be merely a warning before the next major release?), but at  
>>> this
>>
>> +1
>>
>
> Though as the for loop semantics also changed, one could also argue to
> keep it as it is but name it 0.12.

For one thing, I'd like to see the temp allocation changes in 0.12,  
and perhaps some other bigger things that will need lots of testing.  
I'm more OK with the loop semantics changing.

- Robert


From dagss at student.matnat.uio.no  Fri Apr  3 11:41:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Apr 2009 11:41:08 +0200
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <056837A0-5F13-43D1-B0A4-A6342CC6746E@math.washington.edu>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>	<49D5D6DA.9030201@student.matnat.uio.no>	<49D5D761.10900@student.matnat.uio.no>
	<056837A0-5F13-43D1-B0A4-A6342CC6746E@math.washington.edu>
Message-ID: <49D5D9B4.30002@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 2:31 AM, Dag Sverre Seljebotn wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> Robert Bradshaw wrote:
>>>> In the interest of releasing sooner rather than later, a new  
>>>> alpha is
>>>> up at
>>>>
>>>> http://cython.org/Cython-0.11.1.alpha.tar.gz
>>>>
>>>> There is a good patch by Lisandro that does Python <-> c int
>>>> converstion that I'd like to see make it in, and Sage doesn't yet
>>>> compile (mostly due to the "defined before used" bug catching-- 
>>>> should
>>>> this be merely a warning before the next major release?), but at  
>>>> this
>>> +1
>>>
>> Though as the for loop semantics also changed, one could also argue to
>> keep it as it is but name it 0.12.
> 
> For one thing, I'd like to see the temp allocation changes in 0.12,  
> and perhaps some other bigger things that will need lots of testing.  
> I'm more OK with the loop semantics changing.

Ahh, that adds

d) Increase major when more substantial testing is needed (typically a 
merge of -unstable)

which also seems like a good idea.

-- 
Dag Sverre

From stephane.drouard at st.com  Fri Apr  3 11:28:56 2009
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Fri, 3 Apr 2009 11:28:56 +0200
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
Message-ID: <008301c9b43e$9e818470$76ac810a@gnb.st.com>

Hello,

The fix for http://trac.cython.org/cython_trac/ticket/265 does not solve the
issue, we get the same error (stdexcept not included).

Cheers,
Stephane 

-----Original Message-----
From: cython-dev-bounces at codespeak.net
[mailto:cython-dev-bounces at codespeak.net] On Behalf Of Robert Bradshaw
Sent: Friday, April 03, 2009 11:01 AM
To: Cython-dev
Subject: [Cython] Cython 0.11.1 alpha

In the interest of releasing sooner rather than later, a new alpha is up at

http://cython.org/Cython-0.11.1.alpha.tar.gz

There is a good patch by Lisandro that does Python <-> c int converstion
that I'd like to see make it in, and Sage doesn't yet compile (mostly due to
the "defined before used" bug catching--should this be merely a warning
before the next major release?), but at this point lets let the -devel
branch be primarily focused on resolving these issues. I anticipate we could
have 0.11.1 out within a week with nearly two dozen tickets closed.

- Robert
_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri Apr  3 11:33:58 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 02:33:58 -0700
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <008301c9b43e$9e818470$76ac810a@gnb.st.com>
References: <008301c9b43e$9e818470$76ac810a@gnb.st.com>
Message-ID: <65D2DAA2-A974-4DA5-A818-276A539A8320@math.washington.edu>

On Apr 3, 2009, at 2:28 AM, Stephane DROUARD wrote:

> Hello,
>
> The fix for http://trac.cython.org/cython_trac/ticket/265 does not  
> solve the
> issue, we get the same error (stdexcept not included).
>
> Cheers,
> Stephane

Does a

cdef extern from "stdexcept":
     pass

resolve the issue?

- Robert


From robertwb at math.washington.edu  Fri Apr  3 11:40:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 02:40:26 -0700
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <49D5D94D.9020108@student.matnat.uio.no>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>	<49D5D6DA.9030201@student.matnat.uio.no>	<49D5D761.10900@student.matnat.uio.no>
	<b82fe26b85789bb4d050ff1ae9624745.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49D5D94D.9020108@student.matnat.uio.no>
Message-ID: <F169EE31-6CB7-465F-AB5C-3635C5E24E54@math.washington.edu>

On Apr 3, 2009, at 2:39 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert Bradshaw wrote:
>>>>> In the interest of releasing sooner rather than later, a new  
>>>>> alpha is
>>>>> up at
>>>>>
>>>>> http://cython.org/Cython-0.11.1.alpha.tar.gz
>>>>>
>>>>> There is a good patch by Lisandro that does Python <-> c int
>>>>> converstion that I'd like to see make it in, and Sage doesn't yet
>>>>> compile (mostly due to the "defined before used" bug catching-- 
>>>>> should
>>>>> this be merely a warning before the next major release?), but  
>>>>> at this
>>>> +1
>>> Though as the for loop semantics also changed, one could also  
>>> argue to
>>> keep it as it is but name it 0.12.
>>
>> Although, if everyone who needs the bug fixes for 0.11 has to  
>> switch to
>> 0.12 anyway, what good is it to not name it 0.11.1? :)
>
> The question is why we have a major/minor naming scheme at all. It  
> could be
>
> a) Depending on how much time's elapsed since the last major. That's
> roughly our current policy :-)

Well, there's a strong correlation between time elapsed and new  
features.

> b) Substantial new features means new major. That's one guildeline,
> which speaks for naming this one 0.11.1.
>
> c) New major when backwards compatability is broken in any way. That's
> another, conflicting guideline, which speaks for naming this one 0.12.
>
> Myself I'm -1 on a) and +0 on b) and c).

I'm primarily guided by (b), where "substantial new features" may  
include internal re-factoring that requires lots of testing. I think  
(c) is important too, but am not a stickler being pedantically strict  
on this. It should be very safe to do a 0.x.y upgrade, no promises  
that 0.y won't force you to change your code (for the better, though  
it should be avoided--hopefully just making people remove bad/ 
ambiguous/abusive code, or stuff like cdivision).

- Robert


From stefan_ml at behnel.de  Fri Apr  3 11:42:49 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 3 Apr 2009 11:42:49 +0200 (CEST)
Subject: [Cython] [GSoC Proposal] Control flow analysis for Cython
In-Reply-To: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>
References: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>
Message-ID: <77629cb3d9b318f34fdd6e5c48a5f949.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Hi,

Prajwal Suhas P wrote:
> This is my proposal for gsoc.
> http://wiki.cython.org/gsoc/prajwal
>
> Please feel free to give suggestions/comment on it.

My main comment is that your proposal seems to reduce CFA to None check
avoidance. While this is a valid example for CFA application, it does not
quite catch what CFA is by itself. Possible variable values are one thing
that you can *infer* from a control flow graph, possible types are
another.

But the primary goal of this project should be to establish a suitable
representation of the possible execution paths inside a function (and,
less importantly, the global module code). So this really is about finding
(as in reading up) and implementing an efficient way for establishing,
maintaining and querying the control state of a function at each parse
tree node.

Stefan


From stefan_ml at behnel.de  Fri Apr  3 11:48:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 3 Apr 2009 11:48:02 +0200 (CEST)
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <49D5D9B4.30002@student.matnat.uio.no>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
	<49D5D6DA.9030201@student.matnat.uio.no>
	<49D5D761.10900@student.matnat.uio.no>
	<056837A0-5F13-43D1-B0A4-A6342CC6746E@math.washington.edu>
	<49D5D9B4.30002@student.matnat.uio.no>
Message-ID: <b65faf794d2cd1a98685a22183ee4b31.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> d) Increase major when more substantial testing is needed (typically a
> merge of -unstable)
>
> which also seems like a good idea.

Hmm, not that good, IMHO. That's what we have alpha and beta releases for.

Stefan


From stephane.drouard at st.com  Fri Apr  3 11:57:15 2009
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Fri, 3 Apr 2009 11:57:15 +0200
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <65D2DAA2-A974-4DA5-A818-276A539A8320@math.washington.edu>
Message-ID: <008501c9b442$91c38180$76ac810a@gnb.st.com>

Yes it solves it.

I confirm that Cython/Compiler/Nodes.py really has patch
http://hg.cython.org/cython-devel/rev/5d12e7cb8dd9, but stdexcept does not
appear in the generated .C file.

Cheers,
Stephane

-----Original Message-----
From: cython-dev-bounces at codespeak.net
[mailto:cython-dev-bounces at codespeak.net] On Behalf Of Robert Bradshaw
Sent: Friday, April 03, 2009 11:34 AM
To: cython-dev at codespeak.net
Subject: Re: [Cython] Cython 0.11.1 alpha

On Apr 3, 2009, at 2:28 AM, Stephane DROUARD wrote:

> Hello,
>
> The fix for http://trac.cython.org/cython_trac/ticket/265 does not 
> solve the issue, we get the same error (stdexcept not included).
>
> Cheers,
> Stephane

Does a

cdef extern from "stdexcept":
     pass

resolve the issue?

- Robert

_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Fri Apr  3 12:24:19 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 3 Apr 2009 12:24:19 +0200 (CEST)
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <F169EE31-6CB7-465F-AB5C-3635C5E24E54@math.washington.edu>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
	<49D5D6DA.9030201@student.matnat.uio.no>
	<49D5D761.10900@student.matnat.uio.no>
	<b82fe26b85789bb4d050ff1ae9624745.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49D5D94D.9020108@student.matnat.uio.no>
	<F169EE31-6CB7-465F-AB5C-3635C5E24E54@math.washington.edu>
Message-ID: <292c74b8f0dfc4f3218137788f507ae0.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 2:39 AM, Dag Sverre Seljebotn wrote:
>> The question is why we have a major/minor naming scheme at all. It
>> could be
>>
>> a) Depending on how much time's elapsed since the last major. That's
>> roughly our current policy :-)
>
> Well, there's a strong correlation between time elapsed and new
> features.

At least for 0.11, there was also a strong relation between time elapsed
and size of cleanups that needed major testing and consolidation. So the
above doesn't /really/ reflect our current policy.

For the same reason, cython-unstable should become a major 0.xy release.
It changes too many things (not break, just change) to just become a minor
release.


>> b) Substantial new features means new major. That's one guildeline,
>> which speaks for naming this one 0.11.1.
>>
>> c) New major when backwards compatability is broken in any way. That's
>> another, conflicting guideline, which speaks for naming this one 0.12.
>>
>> Myself I'm -1 on a) and +0 on b) and c).
>
> I'm primarily guided by (b), where "substantial new features" may
> include internal re-factoring that requires lots of testing.

+1, even without the "major testing" bit. Substantial code changes do not
belong into a minor release.


> I think
> (c) is important too, but am not a stickler being pedantically strict
> on this. It should be very safe to do a 0.x.y upgrade, no promises
> that 0.y won't force you to change your code (for the better, though
> it should be avoided--hopefully just making people remove bad/
> ambiguous/abusive code, or stuff like cdivision).

That would rather speak for not making major semantic changes before 0.12.
Still, I do consider the loop semantic fixes real bug fixes. The only
reason to go to 0.12 right away would be to say "sorry, 0.11 was a
mistake, it's dead now, please upgrade". I don't think that's true.

Stefan


From psp.reachable at gmail.com  Fri Apr  3 12:25:44 2009
From: psp.reachable at gmail.com (Prajwal Suhas P)
Date: Fri, 3 Apr 2009 15:55:44 +0530
Subject: [Cython] [GSoC Proposal] Control flow analysis for Cython
In-Reply-To: <77629cb3d9b318f34fdd6e5c48a5f949.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>
	<77629cb3d9b318f34fdd6e5c48a5f949.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <fb17e5d0904030325h7ae07e0ep7a5dfecd7c52cff@mail.gmail.com>

On Fri, Apr 3, 2009 at 3:12 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Hi,
>
> Prajwal Suhas P wrote:
> > This is my proposal for gsoc.
> > http://wiki.cython.org/gsoc/prajwal
> >
> > Please feel free to give suggestions/comment on it.
>
> My main comment is that your proposal seems to reduce CFA to None check
> avoidance. While this is a valid example for CFA application, it does not
> quite catch what CFA is by itself. Possible variable values are one thing
> that you can *infer* from a control flow graph, possible types are
> another.
>
Currently i am looking at the None check case for cython CFA. I haven't
covered other cases for CFA application and i thought of adding them as the
project progresses.

>
> But the primary goal of this project should be to establish a suitable
> representation of the possible execution paths inside a function (and,
> less importantly, the global module code). So this really is about finding
> (as in reading up) and implementing an efficient way for establishing,
> maintaining and querying the control state of a function at each parse
> tree node.
>
> Stefan

I have not gone into the depths of generating a CFG for the possible
execution paths inside a function. I will give more thought into
establishing a CFG and update my app.


> _________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Regards,
Prajwal S.

"There is little difference in people,but this little difference makes a big
difference.The little difference is attitude,the big difference is whether
its positive or negative. "
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090403/1ec4a79b/attachment.htm 

From robertwb at math.washington.edu  Fri Apr  3 12:49:31 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 03:49:31 -0700
Subject: [Cython] [GSoC Proposal] Control flow analysis for Cython
In-Reply-To: <fb17e5d0904030325h7ae07e0ep7a5dfecd7c52cff@mail.gmail.com>
References: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>
	<77629cb3d9b318f34fdd6e5c48a5f949.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<fb17e5d0904030325h7ae07e0ep7a5dfecd7c52cff@mail.gmail.com>
Message-ID: <82986048-FCAB-47D9-B945-6C61B6426A97@math.washington.edu>

On Apr 3, 2009, at 3:25 AM, Prajwal Suhas P wrote:

> On Fri, Apr 3, 2009 at 3:12 PM, Stefan Behnel <stefan_ml at behnel.de>  
> wrote:
> Hi,
>
> Prajwal Suhas P wrote:
> > This is my proposal for gsoc.
> > http://wiki.cython.org/gsoc/prajwal
> >
> > Please feel free to give suggestions/comment on it.
>
>> My main comment is that your proposal seems to reduce CFA to None  
>> check
>> avoidance. While this is a valid example for CFA application, it  
>> does not
>> quite catch what CFA is by itself. Possible variable values are  
>> one thing
>> that you can *infer* from a control flow graph, possible types are
>> another.
> Currently i am looking at the None check case for cython CFA. I  
> haven't covered other cases for CFA application and i thought of  
> adding them as the project progresses.

I actually suggested the None check case as a concrete goal for the  
midterm evaluation, as I felt that section needed a real milestone  
and measuring stick. Hopefully by that point the framework would be  
in place enough to be useful in this capacity (even if it will need  
further refining).

>> But the primary goal of this project should be to establish a  
>> suitable
>> representation of the possible execution paths inside a function  
>> (and,
>> less importantly, the global module code). So this really is about  
>> finding
>> (as in reading up) and implementing an efficient way for  
>> establishing,
>> maintaining and querying the control state of a function at each  
>> parse
>> tree node.
>>
>> Stefan
>

+1 Something to this effect should be mentioned in the final evaluation.

- Robert


From robertwb at math.washington.edu  Fri Apr  3 12:51:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 03:51:19 -0700
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <008501c9b442$91c38180$76ac810a@gnb.st.com>
References: <008501c9b442$91c38180$76ac810a@gnb.st.com>
Message-ID: <01C05F17-A4B3-48AE-9303-70E721C24B9E@math.washington.edu>

On Apr 3, 2009, at 2:57 AM, Stephane DROUARD wrote:

> Yes it solves it.
>
> I confirm that Cython/Compiler/Nodes.py really has patch
> http://hg.cython.org/cython-devel/rev/5d12e7cb8dd9, but stdexcept  
> does not
> appear in the generated .C file.

Hmm... I'll look into this some more.

> Cheers,
> Stephane
>
> -----Original Message-----
> From: cython-dev-bounces at codespeak.net
> [mailto:cython-dev-bounces at codespeak.net] On Behalf Of Robert Bradshaw
> Sent: Friday, April 03, 2009 11:34 AM
> To: cython-dev at codespeak.net
> Subject: Re: [Cython] Cython 0.11.1 alpha
>
> On Apr 3, 2009, at 2:28 AM, Stephane DROUARD wrote:
>
>> Hello,
>>
>> The fix for http://trac.cython.org/cython_trac/ticket/265 does not
>> solve the issue, we get the same error (stdexcept not included).
>>
>> Cheers,
>> Stephane
>
> Does a
>
> cdef extern from "stdexcept":
>      pass
>
> resolve the issue?
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri Apr  3 13:03:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 04:03:39 -0700
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <292c74b8f0dfc4f3218137788f507ae0.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <45493547-5D3E-47BD-B526-7AF0B5610871@math.washington.edu>
	<49D5D6DA.9030201@student.matnat.uio.no>
	<49D5D761.10900@student.matnat.uio.no>
	<b82fe26b85789bb4d050ff1ae9624745.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49D5D94D.9020108@student.matnat.uio.no>
	<F169EE31-6CB7-465F-AB5C-3635C5E24E54@math.washington.edu>
	<292c74b8f0dfc4f3218137788f507ae0.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <928EDE1E-BA8F-422E-A2A9-6B91EEBEFC9D@math.washington.edu>

On Apr 3, 2009, at 3:24 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Apr 3, 2009, at 2:39 AM, Dag Sverre Seljebotn wrote:
>>> The question is why we have a major/minor naming scheme at all. It
>>> could be
>>>
>>> a) Depending on how much time's elapsed since the last major. That's
>>> roughly our current policy :-)
>>
>> Well, there's a strong correlation between time elapsed and new
>> features.
>
> At least for 0.11, there was also a strong relation between time  
> elapsed
> and size of cleanups that needed major testing and consolidation.  
> So the
> above doesn't /really/ reflect our current policy.

Hopefully with many smaller releases this will become less of an  
issue :)

> For the same reason, cython-unstable should become a major 0.xy  
> release.
> It changes too many things (not break, just change) to just become  
> a minor
> release.
>
>
>>> b) Substantial new features means new major. That's one guildeline,
>>> which speaks for naming this one 0.11.1.
>>>
>>> c) New major when backwards compatability is broken in any way.  
>>> That's
>>> another, conflicting guideline, which speaks for naming this one  
>>> 0.12.
>>>
>>> Myself I'm -1 on a) and +0 on b) and c).
>>
>> I'm primarily guided by (b), where "substantial new features" may
>> include internal re-factoring that requires lots of testing.
>
> +1, even without the "major testing" bit. Substantial code changes  
> do not
> belong into a minor release.
>
>
>> I think
>> (c) is important too, but am not a stickler being pedantically strict
>> on this. It should be very safe to do a 0.x.y upgrade, no promises
>> that 0.y won't force you to change your code (for the better, though
>> it should be avoided--hopefully just making people remove bad/
>> ambiguous/abusive code, or stuff like cdivision).
>
> That would rather speak for not making major semantic changes  
> before 0.12.
> Still, I do consider the loop semantic fixes real bug fixes. The only
> reason to go to 0.12 right away would be to say "sorry, 0.11 was a
> mistake, it's dead now, please upgrade". I don't think that's true.


There's a lot of good stuff in 0.11.1, but nothing big that I feel  
warrants a "full version" upgrade. Also, it will be good to push out  
stuff like the cdivision directives and warnings.

- Robert


From robertwb at math.washington.edu  Fri Apr  3 13:12:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 04:12:50 -0700
Subject: [Cython] Patches in trac
Message-ID: <296D9E11-7465-42CF-BF92-68A8EE110BFC@math.washington.edu>

I'm planning on setting up a workflow [1] for trac to better keep  
track of tickets that have/need tests/patches. In the meantime, I  
occasionally run into tickets that have patches that I was completely  
unaware of. I hate to let things bitrot, so I'm starting this thread  
to ask if anyone has patches that are sitting in trac (or the email  
archives) that need to be looked at.

- Robert

[1] http://trac.edgewall.org/wiki/WorkFlow

From stefan_ml at behnel.de  Fri Apr  3 13:19:54 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 3 Apr 2009 13:19:54 +0200 (CEST)
Subject: [Cython] Patches in trac
In-Reply-To: <296D9E11-7465-42CF-BF92-68A8EE110BFC@math.washington.edu>
References: <296D9E11-7465-42CF-BF92-68A8EE110BFC@math.washington.edu>
Message-ID: <152284139ca49507df22d43385148d71.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> I'm planning on setting up a workflow [1] for trac to better keep
> track of tickets that have/need tests/patches. In the meantime, I
> occasionally run into tickets that have patches that I was completely
> unaware of. I hate to let things bitrot, so I'm starting this thread
> to ask if anyone has patches that are sitting in trac (or the email
> archives) that need to be looked at.

Isn't there a way to write a query for tickets that have attachments?

Stefan


From robertwb at math.washington.edu  Fri Apr  3 13:30:49 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 04:30:49 -0700
Subject: [Cython] Patches in trac
In-Reply-To: <152284139ca49507df22d43385148d71.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <296D9E11-7465-42CF-BF92-68A8EE110BFC@math.washington.edu>
	<152284139ca49507df22d43385148d71.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <AF206718-4A55-4646-8030-1294ECC58040@math.washington.edu>

On Apr 3, 2009, at 4:19 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> I'm planning on setting up a workflow [1] for trac to better keep
>> track of tickets that have/need tests/patches. In the meantime, I
>> occasionally run into tickets that have patches that I was completely
>> unaware of. I hate to let things bitrot, so I'm starting this thread
>> to ask if anyone has patches that are sitting in trac (or the email
>> archives) that need to be looked at.
>
> Isn't there a way to write a query for tickets that have attachments?

I didn't see one (in my very superficial search) but there likely is.  
Also I'd like to distinguish between patches needing review, patches  
needing work, and patches with positive review (but not yet committed).

- Robert


From stefan_ml at behnel.de  Fri Apr  3 15:40:48 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 3 Apr 2009 15:40:48 +0200 (CEST)
Subject: [Cython] Patches in trac
In-Reply-To: <AF206718-4A55-4646-8030-1294ECC58040@math.washington.edu>
References: <296D9E11-7465-42CF-BF92-68A8EE110BFC@math.washington.edu>
	<152284139ca49507df22d43385148d71.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<AF206718-4A55-4646-8030-1294ECC58040@math.washington.edu>
Message-ID: <7584a7e82fb3a008284fbaa94b5e85d6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 4:19 AM, Stefan Behnel wrote:
>
>> Robert Bradshaw wrote:
>>> I'm planning on setting up a workflow [1] for trac to better keep
>>> track of tickets that have/need tests/patches. In the meantime, I
>>> occasionally run into tickets that have patches that I was completely
>>> unaware of. I hate to let things bitrot, so I'm starting this thread
>>> to ask if anyone has patches that are sitting in trac (or the email
>>> archives) that need to be looked at.
>>
>> Isn't there a way to write a query for tickets that have attachments?
>
> I didn't see one (in my very superficial search) but there likely is.
> Also I'd like to distinguish between patches needing review, patches
> needing work, and patches with positive review (but not yet committed).

Cool. +1

Stefan


From stefan_ml at behnel.de  Fri Apr  3 15:40:48 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 3 Apr 2009 15:40:48 +0200 (CEST)
Subject: [Cython] Patches in trac
In-Reply-To: <AF206718-4A55-4646-8030-1294ECC58040@math.washington.edu>
References: <296D9E11-7465-42CF-BF92-68A8EE110BFC@math.washington.edu>
	<152284139ca49507df22d43385148d71.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<AF206718-4A55-4646-8030-1294ECC58040@math.washington.edu>
Message-ID: <7584a7e82fb3a008284fbaa94b5e85d6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 4:19 AM, Stefan Behnel wrote:
>
>> Robert Bradshaw wrote:
>>> I'm planning on setting up a workflow [1] for trac to better keep
>>> track of tickets that have/need tests/patches. In the meantime, I
>>> occasionally run into tickets that have patches that I was completely
>>> unaware of. I hate to let things bitrot, so I'm starting this thread
>>> to ask if anyone has patches that are sitting in trac (or the email
>>> archives) that need to be looked at.
>>
>> Isn't there a way to write a query for tickets that have attachments?
>
> I didn't see one (in my very superficial search) but there likely is.
> Also I'd like to distinguish between patches needing review, patches
> needing work, and patches with positive review (but not yet committed).

Cool. +1

Stefan


From dalcinl at gmail.com  Fri Apr  3 19:21:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 3 Apr 2009 14:21:17 -0300
Subject: [Cython] new patch for #255 (Py->C for integrals) in the tracker
In-Reply-To: <80E1EA88-7452-461E-8EA6-CF788CA4909A@math.washington.edu>
References: <e7ba66e40904021639r397e1c5dg6063b0c88176ff54@mail.gmail.com>
	<80E1EA88-7452-461E-8EA6-CF788CA4909A@math.washington.edu>
Message-ID: <e7ba66e40904031021n6c6915a9p587356057aa89a27@mail.gmail.com>

Rovert, I've uploaded a new version of the patch following your
suggestions (though slightly different).

Please, see also my last comments on the tracker.

Unfortunately, I'll not be able to work further on this until Monday.
Good weekend!


On Thu, Apr 2, 2009 at 10:52 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 2, 2009, at 4:39 PM, Lisandro Dalcin wrote:
>
>> - The changes to ModuleNode.py and Utils.py are just quick hacks to
>> get it working.
>>
>> - I've tried hard to avoid generation of unreachable code like these
>> "if (0) {...}" blocks.
>>
>> - Once this is reviewed, commented and eventually accepted, I'll
>> continue extending this to extern ctypedef integrals.
>
> I've taken a look at the code and it looks good to me. You took care
> of every corner case I thought of checking. I haven't tested it under
> Py3, but it works great under Py2. It might be worth noting that I
> improved the utility code templating to take a type since this is
> such a common case: http://hg.cython.org/cython-devel/file/
> 2289ab00261d/Cython/Utils.py
>
> One comment I had is that in your code
>
> static INLINE TYPE __Pyx_PyInt_AsTYPE(PyObject* x) {
> ? ? if (sizeof(TYPE) < sizeof(long)) {
> ? ? ? ? long val = __Pyx_PyInt_AsLong(x);
> ? ? ? ? if (unlikely(val == -1 && PyErr_Occurred()))
> ? ? ? ? ? ? return (TYPE)-1;
> ? ? ? ? if (unlikely(val < 0)) {
> ? ? ? ? ? ? PyErr_SetString(PyExc_OverflowError,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? "can't convert negative value to TYPE");
> ? ? ? ? ? ? return (TYPE)-1;
> ? ? ? ? }
> ? ? ? ? if (unlikely(val != (long)(TYPE)val)) {
> ? ? ? ? ? ? PyErr_SetString(PyExc_OverflowError,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? "value too large to convert to TYPE");
> ? ? ? ? ? ? return (TYPE)-1;
> ? ? ? ? }
> ? ? ? ? return (TYPE)val;
> ? ? }
> ? ? return (TYPE)__Pyx_PyInt_AsUnsignedLong(x);
> }
>
> I believe the successful pathway could be shorted by doing
>
> static INLINE TYPE __Pyx_PyInt_AsTYPE(PyObject* x) {
> ? ? if (sizeof(TYPE) < sizeof(long)) {
> ? ? ? ? long val = __Pyx_PyInt_AsLong(x);
> ? ? ? ? if (unlikely(val != (long)(TYPE)val)) {
> ? ? ? ? ? ? if (unlikely(val == -1 && PyErr_Occurred()));
> ? ? ? ? ? ? else if (val < 0) {
> ? ? ? ? ? ? ? ? PyErr_SetString(PyExc_OverflowError,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "can't convert negative value to
> TYPE");
> ? ? ? ? ? ? }
> ? ? ? ? ? ? else {
> ? ? ? ? ? ? ? ? PyErr_SetString(PyExc_OverflowError,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "value too large to convert to TYPE");
> ? ? ? ? ? ? }
> ? ? ? ? ? ? return (TYPE)-1;
> ? ? ? ? }
> ? ? ? ? return (TYPE)val;
> ? ? }
> ? ? return (TYPE)__Pyx_PyInt_AsUnsignedLong(x);
> }
>
> as sizeof(TYPE) < sizeof(long) guarantees val != (long)(TYPE)val for
> negative val on unsigned TYPE and the error does not need to be
> explicitly tested for otherwise (as the calling function will check
> PyErr_Occurred on a return value of -1).
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Apr  3 20:44:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 11:44:23 -0700
Subject: [Cython] new patch for #255 (Py->C for integrals) in the tracker
In-Reply-To: <e7ba66e40904031021n6c6915a9p587356057aa89a27@mail.gmail.com>
References: <e7ba66e40904021639r397e1c5dg6063b0c88176ff54@mail.gmail.com>
	<80E1EA88-7452-461E-8EA6-CF788CA4909A@math.washington.edu>
	<e7ba66e40904031021n6c6915a9p587356057aa89a27@mail.gmail.com>
Message-ID: <8C4DB521-BB67-4524-863A-0D271947C787@math.washington.edu>

On Apr 3, 2009, at 10:21 AM, Lisandro Dalcin wrote:

> Rovert, I've uploaded a new version of the patch following your
> suggestions (though slightly different).
>
> Please, see also my last comments on the tracker.

Cool. I'll take a look.

> Unfortunately, I'll not be able to work further on this until Monday.
> Good weekend!

That's fine. I don't know that I'll have time to resolve the Sage  
issues before then either, so I think we can still get this in 0.11.1.

- Robert

>
>
> On Thu, Apr 2, 2009 at 10:52 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 2, 2009, at 4:39 PM, Lisandro Dalcin wrote:
>>
>>> - The changes to ModuleNode.py and Utils.py are just quick hacks to
>>> get it working.
>>>
>>> - I've tried hard to avoid generation of unreachable code like these
>>> "if (0) {...}" blocks.
>>>
>>> - Once this is reviewed, commented and eventually accepted, I'll
>>> continue extending this to extern ctypedef integrals.
>>
>> I've taken a look at the code and it looks good to me. You took care
>> of every corner case I thought of checking. I haven't tested it under
>> Py3, but it works great under Py2. It might be worth noting that I
>> improved the utility code templating to take a type since this is
>> such a common case: http://hg.cython.org/cython-devel/file/
>> 2289ab00261d/Cython/Utils.py
>>
>> One comment I had is that in your code
>>
>> static INLINE TYPE __Pyx_PyInt_AsTYPE(PyObject* x) {
>>     if (sizeof(TYPE) < sizeof(long)) {
>>         long val = __Pyx_PyInt_AsLong(x);
>>         if (unlikely(val == -1 && PyErr_Occurred()))
>>             return (TYPE)-1;
>>         if (unlikely(val < 0)) {
>>             PyErr_SetString(PyExc_OverflowError,
>>                             "can't convert negative value to TYPE");
>>             return (TYPE)-1;
>>         }
>>         if (unlikely(val != (long)(TYPE)val)) {
>>             PyErr_SetString(PyExc_OverflowError,
>>                             "value too large to convert to TYPE");
>>             return (TYPE)-1;
>>         }
>>         return (TYPE)val;
>>     }
>>     return (TYPE)__Pyx_PyInt_AsUnsignedLong(x);
>> }
>>
>> I believe the successful pathway could be shorted by doing
>>
>> static INLINE TYPE __Pyx_PyInt_AsTYPE(PyObject* x) {
>>     if (sizeof(TYPE) < sizeof(long)) {
>>         long val = __Pyx_PyInt_AsLong(x);
>>         if (unlikely(val != (long)(TYPE)val)) {
>>             if (unlikely(val == -1 && PyErr_Occurred()));
>>             else if (val < 0) {
>>                 PyErr_SetString(PyExc_OverflowError,
>>                                 "can't convert negative value to
>> TYPE");
>>             }
>>             else {
>>                 PyErr_SetString(PyExc_OverflowError,
>>                                 "value too large to convert to  
>> TYPE");
>>             }
>>             return (TYPE)-1;
>>         }
>>         return (TYPE)val;
>>     }
>>     return (TYPE)__Pyx_PyInt_AsUnsignedLong(x);
>> }
>>
>> as sizeof(TYPE) < sizeof(long) guarantees val != (long)(TYPE)val for
>> negative val on unsigned TYPE and the error does not need to be
>> explicitly tested for otherwise (as the calling function will check
>> PyErr_Occurred on a return value of -1).
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Fri Apr  3 20:52:08 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 03 Apr 2009 20:52:08 +0200
Subject: [Cython] [GSoC Proposal] Control flow analysis for Cython
In-Reply-To: <82986048-FCAB-47D9-B945-6C61B6426A97@math.washington.edu>
References: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>	<77629cb3d9b318f34fdd6e5c48a5f949.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<fb17e5d0904030325h7ae07e0ep7a5dfecd7c52cff@mail.gmail.com>
	<82986048-FCAB-47D9-B945-6C61B6426A97@math.washington.edu>
Message-ID: <49D65AD8.1030705@behnel.de>


Robert Bradshaw wrote:
> On Apr 3, 2009, at 3:25 AM, Prajwal Suhas P wrote:
>> Currently i am looking at the None check case for cython CFA. I  
>> haven't covered other cases for CFA application and i thought of  
>> adding them as the project progresses.
> 
> I actually suggested the None check case as a concrete goal for the  
> midterm evaluation, as I felt that section needed a real milestone  
> and measuring stick. Hopefully by that point the framework would be  
> in place enough to be useful in this capacity (even if it will need  
> further refining).

Oh, I'm perfectly fine with a simple application as a first target. I just
wanted to make sure Prajwal (is that your first name?) understands that
this is really just an application and not the goal by itself. The
implementation will have to be a lot more powerful, and thus less trivial
than the application indicates, and the proposal didn't make it obvious to
me that that was clear.

Stefan

From stefan_ml at behnel.de  Fri Apr  3 20:56:42 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 03 Apr 2009 20:56:42 +0200
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <49D5CFE7.6070100@student.matnat.uio.no>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no>
Message-ID: <49D65BEA.9040709@behnel.de>


Dag Sverre Seljebotn wrote:
> [some interesting stuff that doesn't currently work (easily) in Cython]

Are these examples in the Wiki somewhere? Would be great to have a page
that just lists stuff that's non-trivial in current Cython and needs
improvement. That makes it easier to focus on those problems that bring the
best relieve.

Stefan

From robertwb at math.washington.edu  Sat Apr  4 01:04:54 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 16:04:54 -0700
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <01C05F17-A4B3-48AE-9303-70E721C24B9E@math.washington.edu>
References: <008501c9b442$91c38180$76ac810a@gnb.st.com>
	<01C05F17-A4B3-48AE-9303-70E721C24B9E@math.washington.edu>
Message-ID: <9A615BC8-3FDA-40C8-8C70-19E8847B7352@math.washington.edu>

On Apr 3, 2009, at 3:51 AM, Robert Bradshaw wrote:

> On Apr 3, 2009, at 2:57 AM, Stephane DROUARD wrote:
>
>> Yes it solves it.
>>
>> I confirm that Cython/Compiler/Nodes.py really has patch
>> http://hg.cython.org/cython-devel/rev/5d12e7cb8dd9, but stdexcept
>> does not
>> appear in the generated .C file.
>
> Hmm... I'll look into this some more.

Thanks for the feedback--turns out I was only taking care of 2 of the  
3 cases. Should be fixed now.

- Robert


From dsurviver at gmail.com  Sat Apr  4 01:49:41 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Fri, 3 Apr 2009 20:49:41 -0300
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <49D65BEA.9040709@behnel.de>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no> <49D65BEA.9040709@behnel.de>
Message-ID: <45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>

So, I think one thing I need to change, is that instead of creating a
template syntax, I'll work on improving an already existing syntax
right? And so that it should be useful all templates (including STL)?

And should I also work on those ohter problems? I'm afraid of putting
many things on proposal, and don't finish it in time. :/

2009/4/3 Stefan Behnel <stefan_ml at behnel.de>:
>
> Dag Sverre Seljebotn wrote:
>> [some interesting stuff that doesn't currently work (easily) in Cython]
>
> Are these examples in the Wiki somewhere? Would be great to have a page
> that just lists stuff that's non-trivial in current Cython and needs
> improvement. That makes it easier to focus on those problems that bring the
> best relieve.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dsurviver at gmail.com  Sat Apr  4 01:52:06 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Fri, 3 Apr 2009 20:52:06 -0300
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no> <49D65BEA.9040709@behnel.de>
	<45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
Message-ID: <45239150904031652k346c56a7i4215e8242c6d3f97@mail.gmail.com>

Oh... looking now, time is over. :/

I stayed all the evening at university, and got home now. :/

But thanks for the sugestions. If my proposal get accepted, I'll try
to work on the other problems

Thanks for all :D

2009/4/3 Danilo Freitas <dsurviver at gmail.com>:
> So, I think one thing I need to change, is that instead of creating a
> template syntax, I'll work on improving an already existing syntax
> right? And so that it should be useful all templates (including STL)?
>
> And should I also work on those ohter problems? I'm afraid of putting
> many things on proposal, and don't finish it in time. :/
>
> 2009/4/3 Stefan Behnel <stefan_ml at behnel.de>:
>>
>> Dag Sverre Seljebotn wrote:
>>> [some interesting stuff that doesn't currently work (easily) in Cython]
>>
>> Are these examples in the Wiki somewhere? Would be great to have a page
>> that just lists stuff that's non-trivial in current Cython and needs
>> improvement. That makes it easier to focus on those problems that bring the
>> best relieve.
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>

From robertwb at math.washington.edu  Sat Apr  4 03:10:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 18:10:46 -0700
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no>
	<49D65BEA.9040709@behnel.de>
	<45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
Message-ID: <483CA17C-4364-465E-BF62-F87A56815222@math.washington.edu>

On Apr 3, 2009, at 4:49 PM, Danilo Freitas wrote:

> So, I think one thing I need to change, is that instead of creating a
> template syntax, I'll work on improving an already existing syntax
> right?

Probably, yes.

> And so that it should be useful all templates (including STL)?
>
> And should I also work on those ohter problems? I'm afraid of putting
> many things on proposal, and don't finish it in time. :/

As expected, this could turn into a huge wishlist of things that  
would be nice to have, certainly not all feasible for a GSoC  
project :). I would say

(1) Being able to wrap templated code
(2) Polyomorphic functions

would be the priorities, and allow a nearly full wrapping of STL. If  
time permits

(3) operator overloading (we have this for Python classes of course,  
but to allow the + operator, etc. to be used for declared C++ classes)

would be good to.

> 2009/4/3 Stefan Behnel <stefan_ml at behnel.de>:
>>
>> Dag Sverre Seljebotn wrote:
>>> [some interesting stuff that doesn't currently work (easily) in  
>>> Cython]
>>
>> Are these examples in the Wiki somewhere? Would be great to have a  
>> page
>> that just lists stuff that's non-trivial in current Cython and needs
>> improvement. That makes it easier to focus on those problems that  
>> bring the
>> best relieve.
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From vic.kelson at gmail.com  Sat Apr  4 07:13:05 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Sat, 4 Apr 2009 01:13:05 -0400
Subject: [Cython] Status of C complex types
Message-ID: <d5e3120f0904032213nab26d2cl39ff29b196f1bd0c@mail.gmail.com>

Greetings all,
I'm in need of fast computations on complex numbers from Cython. I've been
converting a Python groundwater flow model code to Cython, but I'm reaching
the point where I'd really prefer to have native C complex math. Up to now,
I've used some little C routines that I can call with multiple floats (the C
code converts to/from complex for my purposes). However, I'm nearly to the
point where I'll need to store series coefficients for performance purposes,
and then I'll either need to hide them in float arrays (and that's extremely
nasty), or something like that.

I'd like to say something like



-- 
Vic Kelson
vic.kelson at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090404/80f69b4e/attachment.htm 

From vic.kelson at gmail.com  Sat Apr  4 07:21:24 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Sat, 4 Apr 2009 01:21:24 -0400
Subject: [Cython] Status of C complex types (re-send)
Message-ID: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>

Greetings all,
Sorry for the incomplete message, I'm re-sending it.

I'm in need of fast computations on complex numbers from Cython. I've been
converting a Python groundwater flow model code to Cython, but I'm reaching
the point where I'd really prefer to have native C complex math. Up to now,
I've used some little C routines that I can call with multiple floats (the C
code converts to/from complex for my purposes). However, I'm nearly to the
point where I'll need to store series coefficients for performance purposes,
and then I'll either need to hide them in float arrays (and that's extremely
nasty), or something like that.

I'd like to say something like

cdef class Foo:
    cdef cfloat z

    def __cinit__(self, float x, float y):
        self.z = cfloat(x, y)

    cdef cfloat func(Foo self, float x, float y):
        return-some-gnarly-function of x, y, and self.z


Or something like that, where 'cfloat' maps to the appropriate type in the C
compiler, e.g. "double _Complex" in gcc.

I've seen this discussed in the archives. Has it been implemented? Is there
some pre-release code that essentially works? I only need the four math
functions, conjugate, abs, and log.

THANKS! I think Cython will revolutionize my research work!

--vic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090404/764215b3/attachment.htm 

From robertwb at math.washington.edu  Sat Apr  4 07:37:48 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Apr 2009 22:37:48 -0700
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
Message-ID: <488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>

On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:

> Greetings all,
>
> Sorry for the incomplete message, I'm re-sending it.
>
> I'm in need of fast computations on complex numbers from Cython.  
> I've been converting a Python groundwater flow model code to  
> Cython, but I'm reaching the point where I'd really prefer to have  
> native C complex math. Up to now, I've used some little C routines  
> that I can call with multiple floats (the C code converts to/from  
> complex for my purposes). However, I'm nearly to the point where  
> I'll need to store series coefficients for performance purposes,  
> and then I'll either need to hide them in float arrays (and that's  
> extremely nasty), or something like that.
>
> I'd like to say something like
>
> cdef class Foo:
>     cdef cfloat z
>
>     def __cinit__(self, float x, float y):
>         self.z = cfloat(x, y)
>
>     cdef cfloat func(Foo self, float x, float y):
>         return-some-gnarly-function of x, y, and self.z
>
>
> Or something like that, where 'cfloat' maps to the appropriate type  
> in the C compiler, e.g. "double _Complex" in gcc.
>
> I've seen this discussed in the archives. Has it been implemented?  
> Is there some pre-release code that essentially works? I only need  
> the four math functions, conjugate, abs, and log.

Yes, I've started this, as it comes up in my research too (number  
theory, computing values of L-functions). I hope to have something  
releasable soon. In the meantime, you can do

cdef extern from "complex.h":
     ctypedef double cdouble "double complex"
     cdef double creal(cdouble)
     cdef double cimag(cdouble)
     cdef cdouble _Complex_I

cdef inline cdouble  new_cdouble(double x, double y):
     return x + _Complex_I*y

Of course, the ctypedef is wrong, it will think it can go back to  
doubles (and Python objects) without any problem, but it works for now.

> THANKS! I think Cython will revolutionize my research work!

That's the goal :) Please cite us if it does.

- Robert


From dagss at student.matnat.uio.no  Sat Apr  4 08:44:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 4 Apr 2009 08:44:10 +0200 (CEST)
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <483CA17C-4364-465E-BF62-F87A56815222@math.washington.edu>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no> <49D65BEA.9040709@behnel.de>
	<45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
	<483CA17C-4364-465E-BF62-F87A56815222@math.washington.edu>
Message-ID: <bb7be68cea3d7c54707756b339cf9983.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 4:49 PM, Danilo Freitas wrote:
>
>> So, I think one thing I need to change, is that instead of creating a
>> template syntax, I'll work on improving an already existing syntax
>> right?
>
> Probably, yes.
>
>> And so that it should be useful all templates (including STL)?
>>
>> And should I also work on those ohter problems? I'm afraid of putting
>> many things on proposal, and don't finish it in time. :/
>
> As expected, this could turn into a huge wishlist of things that
> would be nice to have, certainly not all feasible for a GSoC
> project :). I would say
>
> (1) Being able to wrap templated code

What's needed is
a) Parse new syntax for declaring templates
b) Register that information in PyrexTypes/Symtab
c) Output the type arguments directly to C++.

so it's not very much work (on the same order as complex float support, say).

> (2) Polyomorphic functions
>
> would be the priorities, and allow a nearly full wrapping of STL. If
> time permits

Doesn't polymorphic functions work currently? I'd think that were one of
the few C++ things that *do* work in Cython.

Do you mean "overloaded" functions?

Dag Sverre


From robertwb at math.washington.edu  Sat Apr  4 09:25:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 4 Apr 2009 00:25:03 -0700
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <bb7be68cea3d7c54707756b339cf9983.squirrel@webmail.uio.no>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no>
	<49D65BEA.9040709@behnel.de>
	<45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
	<483CA17C-4364-465E-BF62-F87A56815222@math.washington.edu>
	<bb7be68cea3d7c54707756b339cf9983.squirrel@webmail.uio.no>
Message-ID: <A6F88E16-6A49-418C-A6E8-879105B187A4@math.washington.edu>

On Apr 3, 2009, at 11:44 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 3, 2009, at 4:49 PM, Danilo Freitas wrote:
>>
>>> So, I think one thing I need to change, is that instead of  
>>> creating a
>>> template syntax, I'll work on improving an already existing syntax
>>> right?
>>
>> Probably, yes.
>>
>>> And so that it should be useful all templates (including STL)?
>>>
>>> And should I also work on those ohter problems? I'm afraid of  
>>> putting
>>> many things on proposal, and don't finish it in time. :/

BTW, despite being past the deadline, I think the proposal can be  
updated while it's being evaluated.

>> As expected, this could turn into a huge wishlist of things that
>> would be nice to have, certainly not all feasible for a GSoC
>> project :). I would say
>>
>> (1) Being able to wrap templated code
>
> What's needed is
> a) Parse new syntax for declaring templates
> b) Register that information in PyrexTypes/Symtab
> c) Output the type arguments directly to C++.
>
> so it's not very much work (on the same order as complex float  
> support, say).
>
>> (2) Polyomorphic functions
>>
>> would be the priorities, and allow a nearly full wrapping of STL. If
>> time permits
>
> Doesn't polymorphic functions work currently? I'd think that were  
> one of
> the few C++ things that *do* work in Cython.
>
> Do you mean "overloaded" functions?

Yes, that's the more specific term I really meant.

- Robert



From dagss at student.matnat.uio.no  Sat Apr  4 10:34:37 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 04 Apr 2009 10:34:37 +0200
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <A6F88E16-6A49-418C-A6E8-879105B187A4@math.washington.edu>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>	<49D5CFE7.6070100@student.matnat.uio.no>	<49D65BEA.9040709@behnel.de>	<45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>	<483CA17C-4364-465E-BF62-F87A56815222@math.washington.edu>	<bb7be68cea3d7c54707756b339cf9983.squirrel@webmail.uio.no>
	<A6F88E16-6A49-418C-A6E8-879105B187A4@math.washington.edu>
Message-ID: <49D71B9D.9010601@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 11:44 PM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On Apr 3, 2009, at 4:49 PM, Danilo Freitas wrote:
>>>
>>>> So, I think one thing I need to change, is that instead of  
>>>> creating a
>>>> template syntax, I'll work on improving an already existing syntax
>>>> right?
>>> Probably, yes.
>>>
>>>> And so that it should be useful all templates (including STL)?
>>>>
>>>> And should I also work on those ohter problems? I'm afraid of  
>>>> putting
>>>> many things on proposal, and don't finish it in time. :/
> 
> BTW, despite being past the deadline, I think the proposal can be  
> updated while it's being evaluated.

At least it can be updated on the wiki page, which is just as important.

>>> (2) Polyomorphic functions
>>>
>>> would be the priorities, and allow a nearly full wrapping of STL. If
>>> time permits
>> Doesn't polymorphic functions work currently? I'd think that were  
>> one of
>> the few C++ things that *do* work in Cython.
>>
>> Do you mean "overloaded" functions?
> 
> Yes, that's the more specific term I really meant.

:-) I guess polymorphism means a lot of things. But at least in 
C++-lingo it's a common phrase for "virtual methods", i.e. vtables, 
which made it even more confusing.

-- 
Dag Sverre

From psp.reachable at gmail.com  Sat Apr  4 16:42:58 2009
From: psp.reachable at gmail.com (Prajwal Suhas P)
Date: Sat, 4 Apr 2009 20:12:58 +0530
Subject: [Cython] [GSoC Proposal] Control flow analysis for Cython
In-Reply-To: <49D65AD8.1030705@behnel.de>
References: <fb17e5d0904030210q51e8a855ic531f943aca0da7c@mail.gmail.com>
	<77629cb3d9b318f34fdd6e5c48a5f949.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<fb17e5d0904030325h7ae07e0ep7a5dfecd7c52cff@mail.gmail.com>
	<82986048-FCAB-47D9-B945-6C61B6426A97@math.washington.edu>
	<49D65AD8.1030705@behnel.de>
Message-ID: <fb17e5d0904040742q7cf7b65fi79f90fe955fcc483@mail.gmail.com>

On Sat, Apr 4, 2009 at 12:22 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

>
> Robert Bradshaw wrote:
> > On Apr 3, 2009, at 3:25 AM, Prajwal Suhas P wrote:
> >> Currently i am looking at the None check case for cython CFA. I
> >> haven't covered other cases for CFA application and i thought of
> >> adding them as the project progresses.
> >
> > I actually suggested the None check case as a concrete goal for the
> > midterm evaluation, as I felt that section needed a real milestone
> > and measuring stick. Hopefully by that point the framework would be
> > in place enough to be useful in this capacity (even if it will need
> > further refining).
>
> Oh, I'm perfectly fine with a simple application as a first target. I just
> wanted to make sure Prajwal (is that your first name?)
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

 Yes, Prajwal is my first name



-- 
Regards,
Prajwal S.

"There is little difference in people,but this little difference makes a big
difference.The little difference is attitude,the big difference is whether
its positive or negative. "
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090404/25fc033f/attachment-0001.htm 

From vic.kelson at gmail.com  Sat Apr  4 18:09:07 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Sat, 4 Apr 2009 12:09:07 -0400
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
Message-ID: <d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>

Robert,
Thatnks for your helpful note. I can't seem to make it work with gcc
on OS X. I keep getting an error from gcc. I haven't had time to sort
through it yet. I think I'll build a little test problem (I tried
embedding your cod in my existing code and I might have created
another error).
Just to be sure, when I use your code to make a cdouble, the real and
imaginary parts are the same size as a python float, right?
Thanks!
--v


On 4/4/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>
>> Greetings all,
>>
>> Sorry for the incomplete message, I'm re-sending it.
>>
>> I'm in need of fast computations on complex numbers from Cython.
>> I've been converting a Python groundwater flow model code to
>> Cython, but I'm reaching the point where I'd really prefer to have
>> native C complex math. Up to now, I've used some little C routines
>> that I can call with multiple floats (the C code converts to/from
>> complex for my purposes). However, I'm nearly to the point where
>> I'll need to store series coefficients for performance purposes,
>> and then I'll either need to hide them in float arrays (and that's
>> extremely nasty), or something like that.
>>
>> I'd like to say something like
>>
>> cdef class Foo:
>>     cdef cfloat z
>>
>>     def __cinit__(self, float x, float y):
>>         self.z = cfloat(x, y)
>>
>>     cdef cfloat func(Foo self, float x, float y):
>>         return-some-gnarly-function of x, y, and self.z
>>
>>
>> Or something like that, where 'cfloat' maps to the appropriate type
>> in the C compiler, e.g. "double _Complex" in gcc.
>>
>> I've seen this discussed in the archives. Has it been implemented?
>> Is there some pre-release code that essentially works? I only need
>> the four math functions, conjugate, abs, and log.
>
> Yes, I've started this, as it comes up in my research too (number
> theory, computing values of L-functions). I hope to have something
> releasable soon. In the meantime, you can do
>
> cdef extern from "complex.h":
>      ctypedef double cdouble "double complex"
>      cdef double creal(cdouble)
>      cdef double cimag(cdouble)
>      cdef cdouble _Complex_I
>
> cdef inline cdouble  new_cdouble(double x, double y):
>      return x + _Complex_I*y
>
> Of course, the ctypedef is wrong, it will think it can go back to
> doubles (and Python objects) without any problem, but it works for now.
>
>> THANKS! I think Cython will revolutionize my research work!
>
> That's the goal :) Please cite us if it does.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

-- 
Sent from my mobile device

Vic Kelson
vic.kelson at gmail.com

From stefan_ml at behnel.de  Sat Apr  4 20:41:35 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Apr 2009 20:41:35 +0200
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>	<49D5CFE7.6070100@student.matnat.uio.no>
	<49D65BEA.9040709@behnel.de>
	<45239150904031649i78d7f645m3ae4517f5d1bcd53@mail.gmail.com>
Message-ID: <49D7A9DF.4080907@behnel.de>

Hi,

please avoid top-posting. Thanks.

Danilo Freitas wrote:
> 2009/4/3 Stefan Behnel <stefan_ml at behnel.de>:
>> Dag Sverre Seljebotn wrote:
>>> [some interesting stuff that doesn't currently work (easily) in Cython]
>> Are these examples in the Wiki somewhere? Would be great to have a page
>> that just lists stuff that's non-trivial in current Cython and needs
>> improvement. That makes it easier to focus on those problems that bring the
>> best relieve.
>
> So, I think one thing I need to change, is that instead of creating a
> template syntax, I'll work on improving an already existing syntax
> right? And so that it should be useful all templates (including STL)?
> 
> And should I also work on those ohter problems? I'm afraid of putting
> many things on proposal, and don't finish it in time. :/

I didn't mean that everything should appear in your proposal. It's
perfectly right to limit the stated goals to those that appear clearly
reachable.

But it would be helpful to have a separate Wiki page dedicated to
describing missing C++ support features in general (whether they get
implemented (and when) or not) just to keep them in sight.

Stefan

From robertwb at math.washington.edu  Sat Apr  4 20:57:16 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 4 Apr 2009 11:57:16 -0700
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>
Message-ID: <29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>

On Apr 4, 2009, at 9:09 AM, Vic Kelson wrote:

> Robert,
> Thatnks for your helpful note. I can't seem to make it work with gcc
> on OS X.

Hmm... that's the same platform I use.

> I keep getting an error from gcc. I haven't had time to sort
> through it yet. I think I'll build a little test problem (I tried
> embedding your cod in my existing code and I might have created
> another error).

Maybe you could post a snippet of the error you're getting?

> Just to be sure, when I use your code to make a cdouble, the real and
> imaginary parts are the same size as a python float, right?

Yes. Python floats are wrappers of double (and Python complexes wrap  
a pair of doubles).

> Thanks!
> --v
>
>
> On 4/4/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>>
>>> Greetings all,
>>>
>>> Sorry for the incomplete message, I'm re-sending it.
>>>
>>> I'm in need of fast computations on complex numbers from Cython.
>>> I've been converting a Python groundwater flow model code to
>>> Cython, but I'm reaching the point where I'd really prefer to have
>>> native C complex math. Up to now, I've used some little C routines
>>> that I can call with multiple floats (the C code converts to/from
>>> complex for my purposes). However, I'm nearly to the point where
>>> I'll need to store series coefficients for performance purposes,
>>> and then I'll either need to hide them in float arrays (and that's
>>> extremely nasty), or something like that.
>>>
>>> I'd like to say something like
>>>
>>> cdef class Foo:
>>>     cdef cfloat z
>>>
>>>     def __cinit__(self, float x, float y):
>>>         self.z = cfloat(x, y)
>>>
>>>     cdef cfloat func(Foo self, float x, float y):
>>>         return-some-gnarly-function of x, y, and self.z
>>>
>>>
>>> Or something like that, where 'cfloat' maps to the appropriate type
>>> in the C compiler, e.g. "double _Complex" in gcc.
>>>
>>> I've seen this discussed in the archives. Has it been implemented?
>>> Is there some pre-release code that essentially works? I only need
>>> the four math functions, conjugate, abs, and log.
>>
>> Yes, I've started this, as it comes up in my research too (number
>> theory, computing values of L-functions). I hope to have something
>> releasable soon. In the meantime, you can do
>>
>> cdef extern from "complex.h":
>>      ctypedef double cdouble "double complex"
>>      cdef double creal(cdouble)
>>      cdef double cimag(cdouble)
>>      cdef cdouble _Complex_I
>>
>> cdef inline cdouble  new_cdouble(double x, double y):
>>      return x + _Complex_I*y
>>
>> Of course, the ctypedef is wrong, it will think it can go back to
>> doubles (and Python objects) without any problem, but it works for  
>> now.
>>
>>> THANKS! I think Cython will revolutionize my research work!
>>
>> That's the goal :) Please cite us if it does.
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> -- 
> Sent from my mobile device
>
> Vic Kelson
> vic.kelson at gmail.com
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From vic.kelson at gmail.com  Sun Apr  5 02:26:37 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Sat, 4 Apr 2009 20:26:37 -0400
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>
	<29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>
Message-ID: <d5e3120f0904041726ide1e08dxa5638e3cd9c2cd4@mail.gmail.com>

Robert,
Well, as I suspected, it's all my fault! I have  built a tiny module with
your code, a C function and a C class, and it all works! Now to get the
larger code going... Then I'll do some performance testing against some
older Fortran code. If I get 80%, I'm staying with Cython/Python!

THANKS!
--v

On Sat, Apr 4, 2009 at 2:57 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On Apr 4, 2009, at 9:09 AM, Vic Kelson wrote:
>
> > Robert,
> > Thatnks for your helpful note. I can't seem to make it work with gcc
> > on OS X.
>
> Hmm... that's the same platform I use.
>
> > I keep getting an error from gcc. I haven't had time to sort
> > through it yet. I think I'll build a little test problem (I tried
> > embedding your cod in my existing code and I might have created
> > another error).
>
> Maybe you could post a snippet of the error you're getting?
>
> > Just to be sure, when I use your code to make a cdouble, the real and
> > imaginary parts are the same size as a python float, right?
>
> Yes. Python floats are wrappers of double (and Python complexes wrap
> a pair of doubles).
>
> > Thanks!
> > --v
> >
> >
> > On 4/4/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> >> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
> >>
> >>> Greetings all,
> >>>
> >>> Sorry for the incomplete message, I'm re-sending it.
> >>>
> >>> I'm in need of fast computations on complex numbers from Cython.
> >>> I've been converting a Python groundwater flow model code to
> >>> Cython, but I'm reaching the point where I'd really prefer to have
> >>> native C complex math. Up to now, I've used some little C routines
> >>> that I can call with multiple floats (the C code converts to/from
> >>> complex for my purposes). However, I'm nearly to the point where
> >>> I'll need to store series coefficients for performance purposes,
> >>> and then I'll either need to hide them in float arrays (and that's
> >>> extremely nasty), or something like that.
> >>>
> >>> I'd like to say something like
> >>>
> >>> cdef class Foo:
> >>>     cdef cfloat z
> >>>
> >>>     def __cinit__(self, float x, float y):
> >>>         self.z = cfloat(x, y)
> >>>
> >>>     cdef cfloat func(Foo self, float x, float y):
> >>>         return-some-gnarly-function of x, y, and self.z
> >>>
> >>>
> >>> Or something like that, where 'cfloat' maps to the appropriate type
> >>> in the C compiler, e.g. "double _Complex" in gcc.
> >>>
> >>> I've seen this discussed in the archives. Has it been implemented?
> >>> Is there some pre-release code that essentially works? I only need
> >>> the four math functions, conjugate, abs, and log.
> >>
> >> Yes, I've started this, as it comes up in my research too (number
> >> theory, computing values of L-functions). I hope to have something
> >> releasable soon. In the meantime, you can do
> >>
> >> cdef extern from "complex.h":
> >>      ctypedef double cdouble "double complex"
> >>      cdef double creal(cdouble)
> >>      cdef double cimag(cdouble)
> >>      cdef cdouble _Complex_I
> >>
> >> cdef inline cdouble  new_cdouble(double x, double y):
> >>      return x + _Complex_I*y
> >>
> >> Of course, the ctypedef is wrong, it will think it can go back to
> >> doubles (and Python objects) without any problem, but it works for
> >> now.
> >>
> >>> THANKS! I think Cython will revolutionize my research work!
> >>
> >> That's the goal :) Please cite us if it does.
> >>
> >> - Robert
> >>
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >>
> >
> > --
> > Sent from my mobile device
> >
> > Vic Kelson
> > vic.kelson at gmail.com
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Vic Kelson
vic.kelson at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090404/5827b21d/attachment.htm 

From vic.kelson at gmail.com  Sun Apr  5 06:28:09 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Sun, 5 Apr 2009 00:28:09 -0400
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>
	<29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>
Message-ID: <d5e3120f0904042128p84022d5m296981dbdda8c1eb@mail.gmail.com>

Another question: If I need the constant pi, and I use math.pi or numpy.pi
in a C function (or a C method of a C class), does it add an inefficiency
(Python lookup)?
Just wondering...
--v

On Sat, Apr 4, 2009 at 2:57 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On Apr 4, 2009, at 9:09 AM, Vic Kelson wrote:
>
> > Robert,
> > Thatnks for your helpful note. I can't seem to make it work with gcc
> > on OS X.
>
> Hmm... that's the same platform I use.
>
> > I keep getting an error from gcc. I haven't had time to sort
> > through it yet. I think I'll build a little test problem (I tried
> > embedding your cod in my existing code and I might have created
> > another error).
>
> Maybe you could post a snippet of the error you're getting?
>
> > Just to be sure, when I use your code to make a cdouble, the real and
> > imaginary parts are the same size as a python float, right?
>
> Yes. Python floats are wrappers of double (and Python complexes wrap
> a pair of doubles).
>
> > Thanks!
> > --v
> >
> >
> > On 4/4/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> >> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
> >>
> >>> Greetings all,
> >>>
> >>> Sorry for the incomplete message, I'm re-sending it.
> >>>
> >>> I'm in need of fast computations on complex numbers from Cython.
> >>> I've been converting a Python groundwater flow model code to
> >>> Cython, but I'm reaching the point where I'd really prefer to have
> >>> native C complex math. Up to now, I've used some little C routines
> >>> that I can call with multiple floats (the C code converts to/from
> >>> complex for my purposes). However, I'm nearly to the point where
> >>> I'll need to store series coefficients for performance purposes,
> >>> and then I'll either need to hide them in float arrays (and that's
> >>> extremely nasty), or something like that.
> >>>
> >>> I'd like to say something like
> >>>
> >>> cdef class Foo:
> >>>     cdef cfloat z
> >>>
> >>>     def __cinit__(self, float x, float y):
> >>>         self.z = cfloat(x, y)
> >>>
> >>>     cdef cfloat func(Foo self, float x, float y):
> >>>         return-some-gnarly-function of x, y, and self.z
> >>>
> >>>
> >>> Or something like that, where 'cfloat' maps to the appropriate type
> >>> in the C compiler, e.g. "double _Complex" in gcc.
> >>>
> >>> I've seen this discussed in the archives. Has it been implemented?
> >>> Is there some pre-release code that essentially works? I only need
> >>> the four math functions, conjugate, abs, and log.
> >>
> >> Yes, I've started this, as it comes up in my research too (number
> >> theory, computing values of L-functions). I hope to have something
> >> releasable soon. In the meantime, you can do
> >>
> >> cdef extern from "complex.h":
> >>      ctypedef double cdouble "double complex"
> >>      cdef double creal(cdouble)
> >>      cdef double cimag(cdouble)
> >>      cdef cdouble _Complex_I
> >>
> >> cdef inline cdouble  new_cdouble(double x, double y):
> >>      return x + _Complex_I*y
> >>
> >> Of course, the ctypedef is wrong, it will think it can go back to
> >> doubles (and Python objects) without any problem, but it works for
> >> now.
> >>
> >>> THANKS! I think Cython will revolutionize my research work!
> >>
> >> That's the goal :) Please cite us if it does.
> >>
> >> - Robert
> >>
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >>
> >
> > --
> > Sent from my mobile device
> >
> > Vic Kelson
> > vic.kelson at gmail.com
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Vic Kelson
vic.kelson at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090405/8adf7d66/attachment.htm 

From wstein at gmail.com  Sun Apr  5 06:53:51 2009
From: wstein at gmail.com (William Stein)
Date: Sat, 4 Apr 2009 21:53:51 -0700
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <d5e3120f0904042128p84022d5m296981dbdda8c1eb@mail.gmail.com>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com> 
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu> 
	<d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com> 
	<29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu> 
	<d5e3120f0904042128p84022d5m296981dbdda8c1eb@mail.gmail.com>
Message-ID: <85e81ba30904042153u562db1cek6d94ead0f939e901@mail.gmail.com>

On Sat, Apr 4, 2009 at 9:28 PM, Vic Kelson <vic.kelson at gmail.com> wrote:
> Another question: If I need the constant pi, and I use math.pi or numpy.pi
> in a C function (or a C method of a C class), does it add an inefficiency
> (Python lookup)?
> Just wondering...

Yes.  Use Cython's annotate option (or just reading the output) shows
that the "k=math.pi" code in the following:

import math
cdef foo():
    cdef double k
    k = math.pi

generates the following C:

    __pyx_1 = __Pyx_GetName(__pyx_m, __pyx_kp_math); if
(unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 9;
__pyx_clineno = __LINE__; goto __pyx_L1_error;}
  __pyx_2 = PyObject_GetAttr(__pyx_1, __pyx_kp_pi); if
(unlikely(!__pyx_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 9;
__pyx_clineno = __LINE__; goto __pyx_L1_error;}
  Py_DECREF(__pyx_1); __pyx_1 = 0;
  __pyx_3 = __pyx_PyFloat_AsDouble(__pyx_2); if
(unlikely(PyErr_Occurred())) {__pyx_filename = __pyx_f[0];
__pyx_lineno = 9; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
  Py_DECREF(__pyx_2); __pyx_2 = 0;
  __pyx_v_k = __pyx_3;


Timing the following two code snippets shows that using math.pi
instead of a hardcoded double constant for pi can easily slow down a
function by a *factor of 100*.

import math
def foo(int n):
    cdef double k = 0
    cdef int i
    for i in range(n):
        k += math.pi
    return k


import math
def foo2(int n):
    cdef double k=0, pi = 3.14159265358979
    cdef int i
    for i in range(n):
        k += pi
    return k


sage: timeit('foo(10**5)')
25 loops, best of 3: 12.5 ms per loop
sage: timeit('foo2(10**5)')
625 loops, best of 3: 114 ??s per loop
sage: 12.5/0.114
109.649122807018

If you want to write very fast code, you might learn and use the -a
option to Cython.

William

 -- William

From dagss at student.matnat.uio.no  Sun Apr  5 18:13:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 5 Apr 2009 18:13:25 +0200 (CEST)
Subject: [Cython] [GSoC Proposal] Improve C++ Support
In-Reply-To: <49D65BEA.9040709@behnel.de>
References: <45239150904021957l1287e0f5p5e42c7ae7462c991@mail.gmail.com>
	<fea78156d40597b8a7c8ee69ff9013d0.squirrel@webmail.uio.no>
	<892997BA-44F0-4707-96BD-016239A5540A@math.washington.edu>
	<49D5CFE7.6070100@student.matnat.uio.no> <49D65BEA.9040709@behnel.de>
Message-ID: <c1d44784e56a49ac2ae45f67f5d7aa9f.squirrel@webmail.uio.no>

>
> Dag Sverre Seljebotn wrote:
>> [some interesting stuff that doesn't currently work (easily) in Cython]
>
> Are these examples in the Wiki somewhere? Would be great to have a page
> that just lists stuff that's non-trivial in current Cython and needs
> improvement. That makes it easier to focus on those problems that bring
> the
> best relieve.

OK I've fleshed it out and found some more points not supported
(especially upcasts!). I'm sure there's more :-)

http://wiki.cython.org/enhancements/cpp

Dag Sverre


From hoytak at cs.ubc.ca  Sun Apr  5 21:26:15 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Sun, 5 Apr 2009 12:26:15 -0700
Subject: [Cython] Py_New still good way to create classes?
Message-ID: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>

Hello,

I'm using an extension class where I need to create a number of
instances quickly, and I'm looking at the section "Can Cython create
objects or apply operators to locally created objects as pure C code?"
on http://wiki.cython.org/FAQ.  Is this information still correct?

In particular, I'm wondering about it in reference to supporting
cyclical garbage collection and creating object via PyObject_GC_New
etc.  IIRC, cython automatically supports cyclical garbage collection
on classes that have python members, and my extension class falls into
this category.  Should I

a) Use Py_NEW, as specified in the FAQ?
b) Replace Py_NEW with Py_Object_GC_New, and do a?
c) Ignore this and just go with standard creation and deletion,
ignoring the fact that I'll be calling the constructors through
python.

Thanks!
--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From stephane.drouard at st.com  Mon Apr  6 09:36:24 2009
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Mon, 6 Apr 2009 09:36:24 +0200
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <9A615BC8-3FDA-40C8-8C70-19E8847B7352@math.washington.edu>
Message-ID: <003001c9b68a$63e71340$76ac810a@gnb.st.com>

On Apr 3, 2009, at 3:51 AM, Robert Bradshaw wrote:

> Thanks for the feedback--turns out I was only taking care of 2 of the
> 3 cases. Should be fixed now.

Applying patch http://hg.cython.org/cython-devel/rev/49523eb3abb5, it solves
the case I reported. Thanks.

Stephane
 


From dalcinl at gmail.com  Mon Apr  6 18:19:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 6 Apr 2009 13:19:46 -0300
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <d5e3120f0904042128p84022d5m296981dbdda8c1eb@mail.gmail.com>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>
	<29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>
	<d5e3120f0904042128p84022d5m296981dbdda8c1eb@mail.gmail.com>
Message-ID: <e7ba66e40904060919v20fa2709y8baac3d98d646b57@mail.gmail.com>

I would do this (at de module-level):

import math
cdef double PI = math.pi

of perhaps

cdef double PI = 3.14...

and just use "PI" on you cdef functions...

On Sun, Apr 5, 2009 at 1:28 AM, Vic Kelson <vic.kelson at gmail.com> wrote:
> Another question: If I need the constant pi, and I use math.pi or numpy.pi
> in a C function (or a C method of a C class), does it add an inefficiency
> (Python lookup)?
> Just wondering...
> --v
>
> On Sat, Apr 4, 2009 at 2:57 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> On Apr 4, 2009, at 9:09 AM, Vic Kelson wrote:
>>
>> > Robert,
>> > Thatnks for your helpful note. I can't seem to make it work with gcc
>> > on OS X.
>>
>> Hmm... that's the same platform I use.
>>
>> > I keep getting an error from gcc. I haven't had time to sort
>> > through it yet. I think I'll build a little test problem (I tried
>> > embedding your cod in my existing code and I might have created
>> > another error).
>>
>> Maybe you could post a snippet of the error you're getting?
>>
>> > Just to be sure, when I use your code to make a cdouble, the real and
>> > imaginary parts are the same size as a python float, right?
>>
>> Yes. Python floats are wrappers of double (and Python complexes wrap
>> a pair of doubles).
>>
>> > Thanks!
>> > --v
>> >
>> >
>> > On 4/4/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> >> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>> >>
>> >>> Greetings all,
>> >>>
>> >>> Sorry for the incomplete message, I'm re-sending it.
>> >>>
>> >>> I'm in need of fast computations on complex numbers from Cython.
>> >>> I've been converting a Python groundwater flow model code to
>> >>> Cython, but I'm reaching the point where I'd really prefer to have
>> >>> native C complex math. Up to now, I've used some little C routines
>> >>> that I can call with multiple floats (the C code converts to/from
>> >>> complex for my purposes). However, I'm nearly to the point where
>> >>> I'll need to store series coefficients for performance purposes,
>> >>> and then I'll either need to hide them in float arrays (and that's
>> >>> extremely nasty), or something like that.
>> >>>
>> >>> I'd like to say something like
>> >>>
>> >>> cdef class Foo:
>> >>> ? ? cdef cfloat z
>> >>>
>> >>> ? ? def __cinit__(self, float x, float y):
>> >>> ? ? ? ? self.z = cfloat(x, y)
>> >>>
>> >>> ? ? cdef cfloat func(Foo self, float x, float y):
>> >>> ? ? ? ? return-some-gnarly-function of x, y, and self.z
>> >>>
>> >>>
>> >>> Or something like that, where 'cfloat' maps to the appropriate type
>> >>> in the C compiler, e.g. "double _Complex" in gcc.
>> >>>
>> >>> I've seen this discussed in the archives. Has it been implemented?
>> >>> Is there some pre-release code that essentially works? I only need
>> >>> the four math functions, conjugate, abs, and log.
>> >>
>> >> Yes, I've started this, as it comes up in my research too (number
>> >> theory, computing values of L-functions). I hope to have something
>> >> releasable soon. In the meantime, you can do
>> >>
>> >> cdef extern from "complex.h":
>> >> ? ? ?ctypedef double cdouble "double complex"
>> >> ? ? ?cdef double creal(cdouble)
>> >> ? ? ?cdef double cimag(cdouble)
>> >> ? ? ?cdef cdouble _Complex_I
>> >>
>> >> cdef inline cdouble ?new_cdouble(double x, double y):
>> >> ? ? ?return x + _Complex_I*y
>> >>
>> >> Of course, the ctypedef is wrong, it will think it can go back to
>> >> doubles (and Python objects) without any problem, but it works for
>> >> now.
>> >>
>> >>> THANKS! I think Cython will revolutionize my research work!
>> >>
>> >> That's the goal :) Please cite us if it does.
>> >>
>> >> - Robert
>> >>
>> >> _______________________________________________
>> >> Cython-dev mailing list
>> >> Cython-dev at codespeak.net
>> >> http://codespeak.net/mailman/listinfo/cython-dev
>> >>
>> >
>> > --
>> > Sent from my mobile device
>> >
>> > Vic Kelson
>> > vic.kelson at gmail.com
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
>
> --
> Vic Kelson
> vic.kelson at gmail.com
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From vic.kelson at gmail.com  Mon Apr  6 18:53:01 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Mon, 6 Apr 2009 12:53:01 -0400
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <e7ba66e40904060919v20fa2709y8baac3d98d646b57@mail.gmail.com>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>
	<29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>
	<d5e3120f0904042128p84022d5m296981dbdda8c1eb@mail.gmail.com>
	<e7ba66e40904060919v20fa2709y8baac3d98d646b57@mail.gmail.com>
Message-ID: <d5e3120f0904060953n3dae636bxda39c40485bbd27c@mail.gmail.com>

I did the first one. I just wanted to make sure that it was resolved at
compile time.

THANKS!
--v

On Mon, Apr 6, 2009 at 12:19 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> I would do this (at de module-level):
>
> import math
> cdef double PI = math.pi
>
> of perhaps
>
> cdef double PI = 3.14...
>
> and just use "PI" on you cdef functions...
>
> On Sun, Apr 5, 2009 at 1:28 AM, Vic Kelson <vic.kelson at gmail.com> wrote:
> > Another question: If I need the constant pi, and I use math.pi or
> numpy.pi
> > in a C function (or a C method of a C class), does it add an inefficiency
> > (Python lookup)?
> > Just wondering...
> > --v
> >
> > On Sat, Apr 4, 2009 at 2:57 PM, Robert Bradshaw
> > <robertwb at math.washington.edu> wrote:
> >>
> >> On Apr 4, 2009, at 9:09 AM, Vic Kelson wrote:
> >>
> >> > Robert,
> >> > Thatnks for your helpful note. I can't seem to make it work with gcc
> >> > on OS X.
> >>
> >> Hmm... that's the same platform I use.
> >>
> >> > I keep getting an error from gcc. I haven't had time to sort
> >> > through it yet. I think I'll build a little test problem (I tried
> >> > embedding your cod in my existing code and I might have created
> >> > another error).
> >>
> >> Maybe you could post a snippet of the error you're getting?
> >>
> >> > Just to be sure, when I use your code to make a cdouble, the real and
> >> > imaginary parts are the same size as a python float, right?
> >>
> >> Yes. Python floats are wrappers of double (and Python complexes wrap
> >> a pair of doubles).
> >>
> >> > Thanks!
> >> > --v
> >> >
> >> >
> >> > On 4/4/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> >> >> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
> >> >>
> >> >>> Greetings all,
> >> >>>
> >> >>> Sorry for the incomplete message, I'm re-sending it.
> >> >>>
> >> >>> I'm in need of fast computations on complex numbers from Cython.
> >> >>> I've been converting a Python groundwater flow model code to
> >> >>> Cython, but I'm reaching the point where I'd really prefer to have
> >> >>> native C complex math. Up to now, I've used some little C routines
> >> >>> that I can call with multiple floats (the C code converts to/from
> >> >>> complex for my purposes). However, I'm nearly to the point where
> >> >>> I'll need to store series coefficients for performance purposes,
> >> >>> and then I'll either need to hide them in float arrays (and that's
> >> >>> extremely nasty), or something like that.
> >> >>>
> >> >>> I'd like to say something like
> >> >>>
> >> >>> cdef class Foo:
> >> >>>     cdef cfloat z
> >> >>>
> >> >>>     def __cinit__(self, float x, float y):
> >> >>>         self.z = cfloat(x, y)
> >> >>>
> >> >>>     cdef cfloat func(Foo self, float x, float y):
> >> >>>         return-some-gnarly-function of x, y, and self.z
> >> >>>
> >> >>>
> >> >>> Or something like that, where 'cfloat' maps to the appropriate type
> >> >>> in the C compiler, e.g. "double _Complex" in gcc.
> >> >>>
> >> >>> I've seen this discussed in the archives. Has it been implemented?
> >> >>> Is there some pre-release code that essentially works? I only need
> >> >>> the four math functions, conjugate, abs, and log.
> >> >>
> >> >> Yes, I've started this, as it comes up in my research too (number
> >> >> theory, computing values of L-functions). I hope to have something
> >> >> releasable soon. In the meantime, you can do
> >> >>
> >> >> cdef extern from "complex.h":
> >> >>      ctypedef double cdouble "double complex"
> >> >>      cdef double creal(cdouble)
> >> >>      cdef double cimag(cdouble)
> >> >>      cdef cdouble _Complex_I
> >> >>
> >> >> cdef inline cdouble  new_cdouble(double x, double y):
> >> >>      return x + _Complex_I*y
> >> >>
> >> >> Of course, the ctypedef is wrong, it will think it can go back to
> >> >> doubles (and Python objects) without any problem, but it works for
> >> >> now.
> >> >>
> >> >>> THANKS! I think Cython will revolutionize my research work!
> >> >>
> >> >> That's the goal :) Please cite us if it does.
> >> >>
> >> >> - Robert
> >> >>
> >> >> _______________________________________________
> >> >> Cython-dev mailing list
> >> >> Cython-dev at codespeak.net
> >> >> http://codespeak.net/mailman/listinfo/cython-dev
> >> >>
> >> >
> >> > --
> >> > Sent from my mobile device
> >> >
> >> > Vic Kelson
> >> > vic.kelson at gmail.com
> >> > _______________________________________________
> >> > Cython-dev mailing list
> >> > Cython-dev at codespeak.net
> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >>
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> >
> > --
> > Vic Kelson
> > vic.kelson at gmail.com
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Vic Kelson
vic.kelson at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090406/72fb4ae3/attachment-0001.htm 

From dalcinl at gmail.com  Mon Apr  6 18:58:53 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 6 Apr 2009 13:58:53 -0300
Subject: [Cython] updated patch for #255 (Py->C for integrals) in the tracker
Message-ID: <e7ba66e40904060958m1c376150gd7b2af5cb805b55a@mail.gmail.com>

I've uploaded an updated version of the patch for #255.

It is an attempt to reimplement __Pyx_PyNumber_Int() in an smarter
way. Moreover, I've tried to mimic the error messages in core CPython.

Additionally, I've extended the testcase to test for user classes
defining __int__()/__long__(). So far, this test pass on Linux 32 with
Python 2.3 to 2.7 (recent trunk checkout) and 3.0.1

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Mon Apr  6 19:04:10 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 06 Apr 2009 19:04:10 +0200
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <d5e3120f0904060953n3dae636bxda39c40485bbd27c@mail.gmail.com>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>	<d5e3120f0904040909v69416410vf032ffb155adba56@mail.gmail.com>	<29DB5FDB-9FB8-4291-ADC5-5B3234BE4EEF@math.washington.edu>	<d5e3120f0904042128p84022d5m296981dbdda8c1eb@mail.gmail.com>	<e7ba66e40904060919v20fa2709y8baac3d98d646b57@mail.gmail.com>
	<d5e3120f0904060953n3dae636bxda39c40485bbd27c@mail.gmail.com>
Message-ID: <49DA360A.9010601@behnel.de>

Hi,

please try to avoid top-posting.


Vic Kelson wrote:
> On Mon, Apr 6, 2009 at 12:19 PM, Lisandro Dalcin wrote:
> 
>> I would do this (at de module-level):
>>
>> import math
>> cdef double PI = math.pi
>>
>> of perhaps
>>
>> cdef double PI = 3.14...
>>
>> and just use "PI" on you cdef functions...
>>
> I did the first one. I just wanted to make sure that it was resolved at
> compile time.

Well, not really. The value will be resolved and assigned at *runtime*, or
rather module initialisation time. However, *using* the PI variable in your
code will not involve any Python API calls, as it's a plain C variable.

You can also take the value from math.h (cdef extern from ...), which will
avoid any loss in precision due to the type conversion.

Stefan

From dalcinl at gmail.com  Mon Apr  6 19:13:50 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 6 Apr 2009 14:13:50 -0300
Subject: [Cython] pushed fix for __Pyx_SetItemInt_Fast
Message-ID: <e7ba66e40904061013y480938c3q4b994e9c7ee12c1f@mail.gmail.com>

Robert, I've just pushed this fix... please review... (remember "v is
L[i]" can be True)

# HG changeset patch
# User Lisandro Dalcin <dalcinl at gmail.com>
# Date 1239037693 10800
# Node ID ea5b7e6e9db7dcd2f07affbd3f985e8b53b5b335
# Parent  49523eb3abb51b4b783676dc0484b3b1f413e9f8
fix bug in __Pyx_SetItemInt_Fast

diff -r 49523eb3abb5 -r ea5b7e6e9db7 Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py	Fri Apr 03 15:59:21 2009 -0700
+++ b/Cython/Compiler/ExprNodes.py	Mon Apr 06 14:08:13 2009 -0300
@@ -5500,8 +5500,8 @@

 static INLINE int __Pyx_SetItemInt_Fast(PyObject *o, Py_ssize_t i,
PyObject *v, int fits_long) {
     if (PyList_CheckExact(o) && ((0 <= i) & (i < PyList_GET_SIZE(o)))) {
+        Py_INCREF(v);
         Py_DECREF(PyList_GET_ITEM(o, i));
-        Py_INCREF(v);
         PyList_SET_ITEM(o, i, v);
         return 1;
     }


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Mon Apr  6 19:15:38 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 06 Apr 2009 19:15:38 +0200
Subject: [Cython] constant folding for floats?
Message-ID: <49DA38BA.5070704@behnel.de>

Hi,

should we enable constant folding for floats by using the decimal module?

This would not work on Py2.3, but I wouldn't mind just generating less
optimal code when you run Cython under a long out-of-maintenance Python
version.

Stefan

From carl.witty at gmail.com  Mon Apr  6 19:24:14 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Mon, 6 Apr 2009 10:24:14 -0700
Subject: [Cython] constant folding for floats?
In-Reply-To: <49DA38BA.5070704@behnel.de>
References: <49DA38BA.5070704@behnel.de>
Message-ID: <ac1e4c420904061024v73713efai6eea28c87dee9870@mail.gmail.com>

On Mon, Apr 6, 2009 at 10:15 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> should we enable constant folding for floats by using the decimal module?

I don't think so.

1) Is there any advantage to doing constant folding in Cython instead
of letting the C compiler do it?

2) If I write a constant floating-point expression, I expect it to be
evaluated according to IEEE rounding rules.  (So I would prefer doing
the constant folding using Python floats, which will usually match the
computations performed in the target, rather than the decimal module,
which won't match the target.)

Carl

From stefan_ml at behnel.de  Mon Apr  6 19:38:21 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 06 Apr 2009 19:38:21 +0200
Subject: [Cython] constant folding for floats?
In-Reply-To: <ac1e4c420904061024v73713efai6eea28c87dee9870@mail.gmail.com>
References: <49DA38BA.5070704@behnel.de>
	<ac1e4c420904061024v73713efai6eea28c87dee9870@mail.gmail.com>
Message-ID: <49DA3E0D.90504@behnel.de>


Carl Witty wrote:
> On Mon, Apr 6, 2009 at 10:15 AM, Stefan Behnel wrote:
>> should we enable constant folding for floats by using the decimal module?
> 
> I don't think so.
> 
> 1) Is there any advantage to doing constant folding in Cython instead
> of letting the C compiler do it?

Well, yes. In many situations, Cython can use values that are known at
compile time to generate simpler code, e.g. avoid Python type conversions
or move code sections into C completely. Loops are a great example. Cython
0.12 will bring another couple of enhancements here.

Not sure if this really applies to float values, so we might end up
postponing this discussion until there is a concrete reason to enable this.


> 2) If I write a constant floating-point expression, I expect it to be
> evaluated according to IEEE rounding rules.  (So I would prefer doing
> the constant folding using Python floats, which will usually match the
> computations performed in the target, rather than the decimal module,
> which won't match the target.)

Makes sense to me.

Stefan

From robertwb at math.washington.edu  Mon Apr  6 20:24:31 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Apr 2009 11:24:31 -0700
Subject: [Cython] constant folding for floats?
In-Reply-To: <49DA3E0D.90504@behnel.de>
References: <49DA38BA.5070704@behnel.de>
	<ac1e4c420904061024v73713efai6eea28c87dee9870@mail.gmail.com>
	<49DA3E0D.90504@behnel.de>
Message-ID: <4D98C3BE-B202-432B-8827-DCAFC868F865@math.washington.edu>

On Apr 6, 2009, at 10:38 AM, Stefan Behnel wrote:

> Carl Witty wrote:
>> On Mon, Apr 6, 2009 at 10:15 AM, Stefan Behnel wrote:
>>> should we enable constant folding for floats by using the decimal  
>>> module?
>>
>> I don't think so.
>>
>> 1) Is there any advantage to doing constant folding in Cython instead
>> of letting the C compiler do it?
>
> Well, yes. In many situations, Cython can use values that are known at
> compile time to generate simpler code, e.g. avoid Python type  
> conversions
> or move code sections into C completely. Loops are a great example.  
> Cython
> 0.12 will bring another couple of enhancements here.
>
> Not sure if this really applies to float values, so we might end up
> postponing this discussion until there is a concrete reason to  
> enable this.
>
>
>> 2) If I write a constant floating-point expression, I expect it to be
>> evaluated according to IEEE rounding rules.  (So I would prefer doing
>> the constant folding using Python floats, which will usually match  
>> the
>> computations performed in the target, rather than the decimal module,
>> which won't match the target.)
>
> Makes sense to me.

+1 Using the decimals module here seems strange. Note that one needs  
to use repr rather than str to get enough digits to accurately  
represent constants in the source.

- Robert


From dalcinl at gmail.com  Mon Apr  6 21:44:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 6 Apr 2009 16:44:15 -0300
Subject: [Cython] PATCH: the right way to use PyErr_Format with
	Py_ssize_t arguments
In-Reply-To: <2E671135-5B33-4551-82FB-183486183E88@math.washington.edu>
References: <e7ba66e40903110731h65be202ep8b6bed072e9fa9ea@mail.gmail.com>
	<49B7CDF6.3020401@student.matnat.uio.no>
	<e7ba66e40903110901u3c68118fj6dd64441b3f852@mail.gmail.com>
	<2E671135-5B33-4551-82FB-183486183E88@math.washington.edu>
Message-ID: <e7ba66e40904061244s39fd91c0q1595ae0c540da711@mail.gmail.com>

Just opened ticket, http://trac.cython.org/cython_trac/attachment/ticket/274/

The patch is two lines, obviously trivial (for any one with some
knowledge of Python C-API). I've spent more time opening the ticket
and deciding the right Type & Priority tags than working on the fix.
And of course, sum up the time to write this comment ;-). Do we really
need to be so strict about this patch requires ticket policy?


On Wed, Mar 11, 2009 at 8:17 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Mar 11, 2009, at 9:01 AM, Lisandro Dalcin wrote:
>
>> On Wed, Mar 11, 2009 at 12:43 PM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> Lisandro Dalcin wrote:
>>>> Likely Dag should review this... If you agree on the patch, please
>>>> push ... IMHO, this fix should go to 0.11
>
> The patch looks good, can you make a ticket for it? Also, can you use
> mercurial to output diffs so we get your name and a comment with it
> as well?
>
>>> a) Saying that this should go into 0.11 means saying that this
>>> specific
>>> bug is more important than everything else that sits in milestone
>>> 0.11.1
>>> in Trac. We probably put out 0.10 with the same bug and the Cython
>>> project survived! We need to have some disipline and get things
>>> released
>>> -- release early, release often.
>
> Yes, we need to return to the "release early, release often"
> schedule. We can start putting patches up and getting reviews for
> them on trac for 0.11.1 right away. I've added milestone 0.11.2,
> everything that doesn't have a patch or easy fix for 0.11.1 will get
> moved there so we can release 0.11.1 shortly.
>
>> I understand that... I just wanted to point that this fix likely does
>> not have high impact and fix 64 bits issues, more or less like recent
>> Robert's pushes did.
>
>
> The last push was to facilitate getting cython 0.11 into the rpm
> repo, though in retrospect I guess it's lower priority than some
> other stuff. At this point, unless there's a major bug, the only
> thing I can see adding is perhaps testcases.
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue Apr  7 20:40:31 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Apr 2009 11:40:31 -0700
Subject: [Cython] PATCH: the right way to use PyErr_Format with
	Py_ssize_t arguments
In-Reply-To: <e7ba66e40904061244s39fd91c0q1595ae0c540da711@mail.gmail.com>
References: <e7ba66e40903110731h65be202ep8b6bed072e9fa9ea@mail.gmail.com>
	<49B7CDF6.3020401@student.matnat.uio.no>
	<e7ba66e40903110901u3c68118fj6dd64441b3f852@mail.gmail.com>
	<2E671135-5B33-4551-82FB-183486183E88@math.washington.edu>
	<e7ba66e40904061244s39fd91c0q1595ae0c540da711@mail.gmail.com>
Message-ID: <9EF7AF6E-B848-4CC4-8A69-AF9EE8775D4E@math.washington.edu>

On Apr 6, 2009, at 12:44 PM, Lisandro Dalcin wrote:

> Just opened ticket, http://trac.cython.org/cython_trac/attachment/ 
> ticket/274/
>
> The patch is two lines, obviously trivial (for any one with some
> knowledge of Python C-API). I've spent more time opening the ticket
> and deciding the right Type & Priority tags than working on the fix.
> And of course, sum up the time to write this comment ;-). Do we really
> need to be so strict about this patch requires ticket policy?

I feel, at this stage of the project, there are patches simple enough  
to not need a ticket. An obvious example would be fixing typos. This  
is right on the borderline. The trac site serves many purposes, and  
one of them is to be a record of what has been fixed/done (as the  
changelogs are getting lengthy enough that it takes work to extract a  
clear summary from them).

Personally, for a ticket like this, I would probably open a quick  
ticket, not worry to much about the type ("code generation") or  
priority (just leave it at the default) and post a link to the patch  
in the repo. If I found other places where this should be changed, I  
would simply push them (probably adding a link to the already created  
ticket).

- Robert


From bblais at gmail.com  Tue Apr  7 20:48:51 2009
From: bblais at gmail.com (Brian Blais)
Date: Tue, 7 Apr 2009 14:48:51 -0400
Subject: [Cython] preferable way to replace a class
Message-ID: <6F6D1E65-2A4A-4A2F-A8FD-77CAB7D10291@gmail.com>

Hello,

I am writing a tool to do some neural simulation, where I have  
several different neuron groups, which are defined as python  
classes.  Is there a preferred way to write a cython-only replacement  
such that the end-user sees no difference?  What I mean is that they  
should be able to do:

import sim

n1=sim.NeuronGroup1(5)

sim.run_sim([n1])

and if the cython version exists, use that one (for both the neuron  
group and the function to run the simulation), otherwise use the  
python one.


		thanks,

			Brian Blais

--
Brian Blais
bblais at gmail.com
http://web.bryant.edu/~bblais

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090407/e3a31e91/attachment.htm 

From lev at columbia.edu  Tue Apr  7 21:05:18 2009
From: lev at columbia.edu (Lev Givon)
Date: Tue, 7 Apr 2009 15:05:18 -0400
Subject: [Cython] preferable way to replace a class
In-Reply-To: <6F6D1E65-2A4A-4A2F-A8FD-77CAB7D10291@gmail.com>
References: <6F6D1E65-2A4A-4A2F-A8FD-77CAB7D10291@gmail.com>
Message-ID: <20090407190518.GJ957@avicenna.ee.columbia.edu>

Received from Brian Blais on Tue, Apr 07, 2009 at 02:48:51PM EDT:
> Hello,
>
> I am writing a tool to do some neural simulation, where I have several 
> different neuron groups, which are defined as python classes.  Is there a 
> preferred way to write a cython-only replacement such that the end-user 
> sees no difference?  What I mean is that they should be able to do:
>
> import sim
>
> n1=sim.NeuronGroup1(5)
>
> sim.run_sim([n1])
>
> and if the cython version exists, use that one (for both the neuron  
> group and the function to run the simulation), otherwise use the python 
> one.
>
> thanks,
> Brian Blais

You could give the cython and python modules slightly different names
(e.g., sim_cython and sim_python) and create a wrapper module (called
sim) that contains the following code:

try:
    from sim_cython import *
except ImportError:
    from sim_python import *

							L.G.



From bblais at gmail.com  Tue Apr  7 21:29:11 2009
From: bblais at gmail.com (Brian Blais)
Date: Tue, 7 Apr 2009 15:29:11 -0400
Subject: [Cython] preferable way to replace a class
In-Reply-To: <20090407190518.GJ957@avicenna.ee.columbia.edu>
References: <6F6D1E65-2A4A-4A2F-A8FD-77CAB7D10291@gmail.com>
	<20090407190518.GJ957@avicenna.ee.columbia.edu>
Message-ID: <D91B9982-C562-4046-95AC-95EB6AFFE580@gmail.com>

On Apr 7, 2009, at 15:05 , Lev Givon wrote:

> Received from Brian Blais on Tue, Apr 07, 2009 at 02:48:51PM EDT:
>> I am writing a tool to do some neural simulation, where I have  
>> several
>> different neuron groups, which are defined as python classes.  Is  
>> there a
>> preferred way to write a cython-only replacement such that the end- 
>> user
>> sees no difference?  What I mean is that they should be able to do:
>>
>> import sim
>>
>> n1=sim.NeuronGroup1(5)
>>
>> sim.run_sim([n1])
>>
>> and if the cython version exists, use that one (for both the neuron
>> group and the function to run the simulation), otherwise use the  
>> python
>> one.
>>
>>
> You could give the cython and python modules slightly different names
> (e.g., sim_cython and sim_python) and create a wrapper module (called
> sim) that contains the following code:
>
> try:
>     from sim_cython import *
> except ImportError:
>     from sim_python import *
>

Thanks!  Now, if not every class is implemented in sim_cython, should  
I do:

from sim_python import *
try:
     from sim_cython import *
except ImportError:
     pass


		thanks,

			bb

--
Brian Blais
bblais at gmail.com
http://web.bryant.edu/~bblais

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090407/1950955d/attachment.htm 

From robertwb at math.washington.edu  Tue Apr  7 21:33:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Apr 2009 12:33:57 -0700
Subject: [Cython] preferable way to replace a class
In-Reply-To: <D91B9982-C562-4046-95AC-95EB6AFFE580@gmail.com>
References: <6F6D1E65-2A4A-4A2F-A8FD-77CAB7D10291@gmail.com>
	<20090407190518.GJ957@avicenna.ee.columbia.edu>
	<D91B9982-C562-4046-95AC-95EB6AFFE580@gmail.com>
Message-ID: <1EF296C2-624C-4D5A-9FE1-C38EA4696AEA@math.washington.edu>

On Apr 7, 2009, at 12:29 PM, Brian Blais wrote:

> On Apr 7, 2009, at 15:05 , Lev Givon wrote:
>
>> Received from Brian Blais on Tue, Apr 07, 2009 at 02:48:51PM EDT:
>>> I am writing a tool to do some neural simulation, where I have  
>>> several
>>> different neuron groups, which are defined as python classes.  Is  
>>> there a
>>> preferred way to write a cython-only replacement such that the  
>>> end-user
>>> sees no difference?  What I mean is that they should be able to do:
>>>
>>> import sim
>>>
>>> n1=sim.NeuronGroup1(5)
>>>
>>> sim.run_sim([n1])
>>>
>>> and if the cython version exists, use that one (for both the neuron
>>> group and the function to run the simulation), otherwise use the  
>>> python
>>> one.
>>>
>>>
>> You could give the cython and python modules slightly different names
>> (e.g., sim_cython and sim_python) and create a wrapper module (called
>> sim) that contains the following code:
>>
>> try:
>>     from sim_cython import *
>> except ImportError:
>>     from sim_python import *
>>
>
> Thanks!  Now, if not every class is implemented in sim_cython,  
> should I do:
>
> from sim_python import *
> try:
>     from sim_cython import *
> except ImportError:
>     pass

Yes, that would work well.

You might also be interested in http://wiki.cython.org/pure which may  
allow you to keep a single codebase for the compiled and interpreted  
implementations.

- Robert


From lev at columbia.edu  Wed Apr  8 00:59:22 2009
From: lev at columbia.edu (Lev Givon)
Date: Tue, 7 Apr 2009 18:59:22 -0400
Subject: [Cython] preferable way to replace a class
In-Reply-To: <D91B9982-C562-4046-95AC-95EB6AFFE580@gmail.com>
References: <6F6D1E65-2A4A-4A2F-A8FD-77CAB7D10291@gmail.com>
	<20090407190518.GJ957@avicenna.ee.columbia.edu>
	<D91B9982-C562-4046-95AC-95EB6AFFE580@gmail.com>
Message-ID: <20090407225922.GM957@avicenna.ee.columbia.edu>

Received from Brian Blais on Tue, Apr 07, 2009 at 03:29:11PM EDT:

(snip)

>> You could give the cython and python modules slightly different names
>> (e.g., sim_cython and sim_python) and create a wrapper module (called
>> sim) that contains the following code:
>>
>> try:
>>     from sim_cython import *
>> except ImportError:
>>     from sim_python import *
>>
>
> Thanks!  Now, if not every class is implemented in sim_cython, should I 
> do:
>
> from sim_python import *
> try:
>     from sim_cython import *
> except ImportError:
>     pass
>
> thanks,
> bb

Yes, if you intend to replace the symbols imported into the current
namespace from sim_python with their analogues in sim_cython.

						L.G.

From dalcinl at gmail.com  Wed Apr  8 04:15:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Apr 2009 23:15:27 -0300
Subject: [Cython] fix for "exec" statement (ticket #275)
Message-ID: <e7ba66e40904071915j6904a6c3i58efbcea5db02d70@mail.gmail.com>

Just opened ticket and patch attached (tested in Py2.6 and Py3.0 on
Win32 with MSVC compiler, likely ready for push)

http://trac.cython.org/cython_trac/ticket/275

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Apr  8 04:22:29 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Apr 2009 23:22:29 -0300
Subject: [Cython] How to fix Win32 linker errors when running tests?
Message-ID: <e7ba66e40904071922v3d09f0a7v6ff900e9355792fa@mail.gmail.com>

I'm getting a few errors like the following:

coercearraytoptr.obj : error LNK2019: unresolved external symbol _spam
referenced in function ___pyx_f_16coercearraytoptr_eggs

Likely including a fake C header with "cdef extern" will solve some of
the issues. Where should I put such headers?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Apr  8 04:44:55 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Apr 2009 23:44:55 -0300
Subject: [Cython] How to fix Win32 linker errors when running tests?
In-Reply-To: <e7ba66e40904071922v3d09f0a7v6ff900e9355792fa@mail.gmail.com>
References: <e7ba66e40904071922v3d09f0a7v6ff900e9355792fa@mail.gmail.com>
Message-ID: <e7ba66e40904071944v7cf866b1jc2c4716c2ccacfb1@mail.gmail.com>

It seems this issue have an open ticket

http://trac.cython.org/cython_trac/ticket/106

On Tue, Apr 7, 2009 at 11:22 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> I'm getting a few errors like the following:
>
> coercearraytoptr.obj : error LNK2019: unresolved external symbol _spam
> referenced in function ___pyx_f_16coercearraytoptr_eggs
>
> Likely including a fake C header with "cdef extern" will solve some of
> the issues. Where should I put such headers?
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Apr  8 15:58:36 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 8 Apr 2009 15:58:36 +0200 (CEST)
Subject: [Cython] MPI and Cython?
Message-ID: <8fb9f12447b21bacc1fc304859ef7b91.squirrel@webmail.uio.no>

What's the best way of using MPI from Cython? Should one use the C library
directly (has anyone written a pxd?), or is there an MPI wrapper library
with the same Python and Cyhthon interfaces?

Dag Sverre


From dalcinl at gmail.com  Wed Apr  8 16:57:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 8 Apr 2009 11:57:46 -0300
Subject: [Cython] MPI and Cython?
In-Reply-To: <8fb9f12447b21bacc1fc304859ef7b91.squirrel@webmail.uio.no>
References: <8fb9f12447b21bacc1fc304859ef7b91.squirrel@webmail.uio.no>
Message-ID: <e7ba66e40904080757s70955328hb613bda1e1f4dec8@mail.gmail.com>

Well, I would say you should try mpi4py. However, mpi4py hgih-level,
object oriented API is targeted to be used from Python and not Cython
(i.e. no cpdef methods at all). However, you can easily extract the
MPI C handles from the Python objects and call any MPI C API
functions, as you can see here:

http://code.google.com/p/mpi4py/source/browse/trunk/demo/cython/helloworld.pyx

This esample is a bit of mess, because it shows many different ways of
doing things... If you want to make fast MPI calls, you will be
interested in the last part.

Note that "from mpi4py.mpi_c cimport *" should also work (instead of
'include mpi4py/mpi.pxi'), though I've not tested that. That example
was written long ago in a time where likely "cimport *" did not work.



On Wed, Apr 8, 2009 at 10:58 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> What's the best way of using MPI from Cython? Should one use the C library
> directly (has anyone written a pxd?), or is there an MPI wrapper library
> with the same Python and Cyhthon interfaces?
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Apr  8 18:22:12 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 8 Apr 2009 13:22:12 -0300
Subject: [Cython] moving #255 to milestone 0.11.1
Message-ID: <e7ba66e40904080922r909e786p5b7e156ee675d7df@mail.gmail.com>

What do you think about this?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jasone at canonware.com  Wed Apr  8 19:51:09 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 08 Apr 2009 10:51:09 -0700
Subject: [Cython] MPI and Cython?
In-Reply-To: <8fb9f12447b21bacc1fc304859ef7b91.squirrel@webmail.uio.no>
References: <8fb9f12447b21bacc1fc304859ef7b91.squirrel@webmail.uio.no>
Message-ID: <49DCE40D.1010606@canonware.com>

Dag Sverre Seljebotn wrote:
> What's the best way of using MPI from Cython? Should one use the C library
> directly (has anyone written a pxd?), or is there an MPI wrapper library
> with the same Python and Cyhthon interfaces?

I'm using mpi4py in Crux (http://canonware.com/cgi-bin/hg_crux), and am 
very pleased with how it works.  In most places I'm using the cdef 
bindings, but the Python bindings have also been useful in pure-Python 
scripts.

Jason

From robertwb at math.washington.edu  Wed Apr  8 20:15:30 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 8 Apr 2009 11:15:30 -0700
Subject: [Cython] moving #255 to milestone 0.11.1
In-Reply-To: <e7ba66e40904080922r909e786p5b7e156ee675d7df@mail.gmail.com>
References: <e7ba66e40904080922r909e786p5b7e156ee675d7df@mail.gmail.com>
Message-ID: <ECA17B71-E3BD-4CAA-9B32-FA3CC8719943@math.washington.edu>

+1. I've been busy so I haven't had a chance to work on Cython much  
the last couple of days, but was planning on seeing if I can get Sage  
to compile tonight and take a look at this. BTW, just apply the last  
patch?

- Robert

On Apr 8, 2009, at 9:22 AM, Lisandro Dalcin wrote:

> What do you think about this?
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Wed Apr  8 22:27:56 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 8 Apr 2009 17:27:56 -0300
Subject: [Cython] moving #255 to milestone 0.11.1
In-Reply-To: <ECA17B71-E3BD-4CAA-9B32-FA3CC8719943@math.washington.edu>
References: <e7ba66e40904080922r909e786p5b7e156ee675d7df@mail.gmail.com>
	<ECA17B71-E3BD-4CAA-9B32-FA3CC8719943@math.washington.edu>
Message-ID: <e7ba66e40904081327h1025f223xafe1468a8b29fe46@mail.gmail.com>

On Wed, Apr 8, 2009 at 3:15 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> +1. I've been busy so I haven't had a chance to work on Cython much
> the last couple of days, but was planning on seeing if I can get Sage
> to compile tonight and take a look at this. BTW, just apply the last
> patch?

Yes, just the c-int-types.diff (it contains the testcase for the ticket).

>
> - Robert
>
> On Apr 8, 2009, at 9:22 AM, Lisandro Dalcin wrote:
>
>> What do you think about this?
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Apr  9 19:36:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Apr 2009 10:36:23 -0700
Subject: [Cython] Cython 0.11.1 alpha
In-Reply-To: <003001c9b68a$63e71340$76ac810a@gnb.st.com>
References: <003001c9b68a$63e71340$76ac810a@gnb.st.com>
Message-ID: <B1B5470D-8AD1-48A3-811E-F2E41939E980@math.washington.edu>

Just a status update on this--Sage still doesn't compile, but as soon  
as it does I'll push a beta.

- Robert


From robertwb at math.washington.edu  Fri Apr 10 09:19:06 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Apr 2009 00:19:06 -0700
Subject: [Cython] Cython 0.11.1 beta
Message-ID: <943E3D18-3037-4DE7-8684-92D42F420467@math.washington.edu>

Please try it out. http://cython.org/Cython-0.11.1.beta.tar.gz

There are still two blockers:

Exceptions in __Pyx_InitGlobals() lead to a crash
http://trac.cython.org/cython_trac/ticket/216

Sage segfaults
http://trac.cython.org/cython_trac/ticket/278

I'll be working on the second one.

- Robert


From stefan_ml at behnel.de  Fri Apr 10 10:57:25 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Apr 2009 10:57:25 +0200
Subject: [Cython] Cython 0.11.1 beta
In-Reply-To: <943E3D18-3037-4DE7-8684-92D42F420467@math.washington.edu>
References: <943E3D18-3037-4DE7-8684-92D42F420467@math.washington.edu>
Message-ID: <49DF09F5.3060507@behnel.de>


Robert Bradshaw wrote:
> Please try it out. http://cython.org/Cython-0.11.1.beta.tar.gz
> 
> There are still two blockers:
> 
> Exceptions in __Pyx_InitGlobals() lead to a crash
> http://trac.cython.org/cython_trac/ticket/216

I know, I set this to critical (because it's incorrect code that can lead
to a crash), but I don't really think it's that worrisome. The only obvious
reason I can see why there should be an exception in that function is that
we are running out of memory. Rather unlikely for a module import, as it
usually happens at program initialisation time.

Since the fix is not trivial (although not complex either), I'm fine with
moving it to a later milestone (say, 0.12, although 0.11.2 should be ok).

Note that there are still some tests in cython-devel that fail at the C
compiler level:

- cpp_exceptions_T265.cpp seems to lack a file "foo.h"

- bad_c_struct_T252 says:

bad_c_struct_T252.c: In function '__pyx_f_17bad_c_struct_T252_f':
bad_c_struct_T252.c:344: error: expected identifier before 'void'

ERROR

- The test for #232 is also failing, I'll look into that.

Stefan

From stefan_ml at behnel.de  Fri Apr 10 17:29:36 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Apr 2009 17:29:36 +0200
Subject: [Cython] Cython 0.11.1 beta
In-Reply-To: <49DF09F5.3060507@behnel.de>
References: <943E3D18-3037-4DE7-8684-92D42F420467@math.washington.edu>
	<49DF09F5.3060507@behnel.de>
Message-ID: <49DF65E0.10306@behnel.de>


Stefan Behnel wrote:
> - The test for #232 is also failing, I'll look into that.

My bad, that was fixed in cython-unstable, not cython-devel. I just
accidentally copied the test case over into tests/run/ on my side. So this
won't go into 0.11.1 anyway.

Stefan

From riccardo.murri at gmail.com  Mon Apr 13 23:14:32 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Mon, 13 Apr 2009 23:14:32 +0200
Subject: [Cython] very simple iterator crashing Cython-0.11
Message-ID: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>

Hello,

I have started using Cython recently to spped up some Python code; it
works very well and stably, but today I managed to crash Cython 0.11
while trying to compile the following pure Python code:

  # iter.py
  class Iter(object):

      """A docstring."""

      def __iter__(self):
          return self

      def __next__(self):
          raise StopIteration

I'm augmenting the pure Python code with the following .pxd file::

  # iter.pxd
  cpdef class Iter(object)

The Cython-0.11 invocation "cython iter.py" dies with the following backtrace::

  $ cython iter.py

  Error converting Pyrex file to C:
  ------------------------------------------------------------
  ...
  class Iter(object):

      """A docstring."""

      def __iter__(self):
     ^
  ------------------------------------------------------------

  /tmp/iter.py:5:4: Compiler crash in AlignFunctionDefinitions

  ModuleNode.body = StatListNode(iter.py:1:0)
  StatListNode.stats[0] = PyClassDefNode(iter.py:1:0,
      doc = 'A docstring.',
      name = u'Iter')
  CClassDefNode.body = StatListNode(iter.py:5:4)
  StatListNode.stats[0] = DefNode(iter.py:5:4,
      name = u'__iter__',
      num_required_args = 1,
      reqd_kw_flags_cname = '0')

  Compiler crash traceback from this point on:
    File "Visitor.py", line 149, in Cython.Compiler.Visitor.visitchild
(/home/rmurri/phd.cython/src/Cython-0.11/Cython/Compiler/Visitor.c:2814)
    File "Visitor.py", line 45, in
Cython.Compiler.Visitor.BasicVisitor.visit
(/home/rmurri/phd.cython/src/Cython-0.11/Cython/Compiler/Visitor.c:1356)
    File "/home/rmurri/phd.cython/src/Cython-0.11/Cython/Compiler/ParseTreeTransforms.py",
line 788, in visit_DefNode
      pxd_def = self.scope.lookup(node.name)
  AttributeError: 'NoneType' object has no attribute 'lookup'


Compilation is fine if I merge the .pxd and the .py into a .pyx::

  cpdef class Iter(object):

      """A docstring."""

      def __iter__(self):
          return self

      def __next__(self):
          raise StopIteration


Best regards,
Riccardo


-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From dalcinl at gmail.com  Tue Apr 14 01:17:30 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 13 Apr 2009 20:17:30 -0300
Subject: [Cython] very simple iterator crashing Cython-0.11
In-Reply-To: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>
References: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>
Message-ID: <e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>

What happens if you write "cdef class ..." instead of "cpdef class
..." on your pxd?

On Mon, Apr 13, 2009 at 6:14 PM, Riccardo Murri
<riccardo.murri at gmail.com> wrote:
> Hello,
>
> I have started using Cython recently to spped up some Python code; it
> works very well and stably, but today I managed to crash Cython 0.11
> while trying to compile the following pure Python code:
>
> ?# iter.py
> ?class Iter(object):
>
> ? ? ?"""A docstring."""
>
> ? ? ?def __iter__(self):
> ? ? ? ? ?return self
>
> ? ? ?def __next__(self):
> ? ? ? ? ?raise StopIteration
>
> I'm augmenting the pure Python code with the following .pxd file::
>
> ?# iter.pxd
> ?cpdef class Iter(object)
>
> The Cython-0.11 invocation "cython iter.py" dies with the following backtrace::
>
> ?$ cython iter.py
>
> ?Error converting Pyrex file to C:
> ?------------------------------------------------------------
> ?...
> ?class Iter(object):
>
> ? ? ?"""A docstring."""
>
> ? ? ?def __iter__(self):
> ? ? ^
> ?------------------------------------------------------------
>
> ?/tmp/iter.py:5:4: Compiler crash in AlignFunctionDefinitions
>
> ?ModuleNode.body = StatListNode(iter.py:1:0)
> ?StatListNode.stats[0] = PyClassDefNode(iter.py:1:0,
> ? ? ?doc = 'A docstring.',
> ? ? ?name = u'Iter')
> ?CClassDefNode.body = StatListNode(iter.py:5:4)
> ?StatListNode.stats[0] = DefNode(iter.py:5:4,
> ? ? ?name = u'__iter__',
> ? ? ?num_required_args = 1,
> ? ? ?reqd_kw_flags_cname = '0')
>
> ?Compiler crash traceback from this point on:
> ? ?File "Visitor.py", line 149, in Cython.Compiler.Visitor.visitchild
> (/home/rmurri/phd.cython/src/Cython-0.11/Cython/Compiler/Visitor.c:2814)
> ? ?File "Visitor.py", line 45, in
> Cython.Compiler.Visitor.BasicVisitor.visit
> (/home/rmurri/phd.cython/src/Cython-0.11/Cython/Compiler/Visitor.c:1356)
> ? ?File "/home/rmurri/phd.cython/src/Cython-0.11/Cython/Compiler/ParseTreeTransforms.py",
> line 788, in visit_DefNode
> ? ? ?pxd_def = self.scope.lookup(node.name)
> ?AttributeError: 'NoneType' object has no attribute 'lookup'
>
>
> Compilation is fine if I merge the .pxd and the .py into a .pyx::
>
> ?cpdef class Iter(object):
>
> ? ? ?"""A docstring."""
>
> ? ? ?def __iter__(self):
> ? ? ? ? ?return self
>
> ? ? ?def __next__(self):
> ? ? ? ? ?raise StopIteration
>
>
> Best regards,
> Riccardo
>
>
> --
> Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From riccardo.murri at gmail.com  Tue Apr 14 10:01:14 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Tue, 14 Apr 2009 10:01:14 +0200
Subject: [Cython] very simple iterator crashing Cython-0.11
In-Reply-To: <e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>
References: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>
	<e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>
Message-ID: <1c34ba170904140101l37100db6k7805277f5c50db9d@mail.gmail.com>

On Tue, Apr 14, 2009 at 1:17 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> What happens if you write "cdef class ..." instead of "cpdef class
> ..." on your pxd?
>

Same error.

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From robertwb at math.washington.edu  Tue Apr 14 10:09:14 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 01:09:14 -0700
Subject: [Cython] very simple iterator crashing Cython-0.11
In-Reply-To: <1c34ba170904140101l37100db6k7805277f5c50db9d@mail.gmail.com>
References: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>
	<e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>
	<1c34ba170904140101l37100db6k7805277f5c50db9d@mail.gmail.com>
Message-ID: <530BC1C3-5A3C-404B-AD51-DD7B1DD5A61B@math.washington.edu>

On Apr 14, 2009, at 1:01 AM, Riccardo Murri wrote:

> On Tue, Apr 14, 2009 at 1:17 AM, Lisandro Dalcin  
> <dalcinl at gmail.com> wrote:
>> What happens if you write "cdef class ..." instead of "cpdef class
>> ..." on your pxd?
>>
>
> Same error.

I'm surprised that "cpdef class" works at all (it shouldn't). In any  
case, this is most certainly a bug, we are tracking it at http:// 
trac.cython.org/cython_trac/ticket/279 .

- Robert



From dagss at student.matnat.uio.no  Tue Apr 14 14:41:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 14 Apr 2009 14:41:25 +0200
Subject: [Cython] Poll: Buffer access exception under nogil
Message-ID: <49E48475.9090406@student.matnat.uio.no>

Cython currently segfaults when using a buffer in nogil and accessing 
out of bounds (which triggers an IndexError). This is a bug, however I 
would like to do more than disallow it.

I.e. what should happen here:

     cdef object[int] buf = ... # buffer of length < 6
     with nogil:
         buf[5] = 0

Option 1) Reacquire the GIL temporarily to construct the IndexError 
instance. This seems to work wonderfully in my current patch; but is it 
too unintuitive to raise an exception from an operation in a nogil 
section? Will any other operation trigger an exception within a nogil 
section?

Option 2) Only allow buffer use in nogil if @cython.boundscheck(False). 
This has some usability problems; it would be a pain to turn on/off 
nogil for debugging.

Option 3) Print to standard output using printf instead of raising 
exception.



-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Apr 14 15:30:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Apr 2009 15:30:44 +0200 (CEST)
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E48475.9090406@student.matnat.uio.no>
References: <49E48475.9090406@student.matnat.uio.no>
Message-ID: <c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> Cython currently segfaults when using a buffer in nogil and accessing
> out of bounds (which triggers an IndexError). This is a bug, however I
> would like to do more than disallow it.
>
> I.e. what should happen here:
>
>      cdef object[int] buf = ... # buffer of length < 6
>      with nogil:
>          buf[5] = 0
>
> Option 1) Reacquire the GIL temporarily to construct the IndexError
> instance. This seems to work wonderfully in my current patch; but is it
> too unintuitive to raise an exception from an operation in a nogil
> section?

Well, exceptions happen to be the error reporting mechanism in Python. In
any case, the above must exit the nogil block immediately and should not
crash (as long as we can prevent it in a reasonable way). To me, this
sounds like an exception is the right thing.

However, it's quite possible that this kind of control flow makes error
handling and C-level cleanup measures a nightmare. How would such a nogil
block interact with try-except or try-finally, for example? We can't allow
try-except /inside/ the block, so you'd have to anticipate all errors and
wrap separate sections of the nogil block in their own try-except or
try-finally statements to make sure the cleanup works as required.


> Will any other operation trigger an exception within a nogil
> section?

You can have exceptions that get raised in callbacks that re-acquire the
GIL, so you can't be sure that nogil blocks do not raise exceptions
anyway.

Think of code like this (which does not currently work):

    cdef extern from *:
        # some external C library function
        void call_into_c(...) nogil except *

    cdef void callback(void* data) with gil except *:
        ((...*)data).failed = 1   # cause abort in calling C library
        raise IndexError

    try:
        with nogil:
            call_into_c(callback, &some_data)
            printf("we never get here!")
    except IndexError:
        print "holy moly!"

To support this, Cython would have to generate code to re-acquire the GIL
after the call to "call_into_c()", check for an error and then either jump
out of the failed nogil block or release the GIL and continue.

If we start raising exceptions inside nogil blocks, this would be the next
obvious step. I have not made up my mind yet if this is a good thing. At
least, I know a couple of cases in lxml where I could use it, but this
will certainly need some prior exploration before deciding to make this an
official language feature.


> Option 2) Only allow buffer use in nogil if @cython.boundscheck(False).
> This has some usability problems; it would be a pain to turn on/off
> nogil for debugging.

Sounds like a viable alternative, though. Should be followed if exception
raising is not considered accurate from inside nogil blocks.


> Option 3) Print to standard output using printf instead of raising
> exception.

-1 This is a clear error case that should not pass silently.

Stefan


From dagss at student.matnat.uio.no  Tue Apr 14 18:17:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 14 Apr 2009 18:17:28 +0200
Subject: [Cython] C int division by zero
Message-ID: <49E4B718.5020502@student.matnat.uio.no>

I was just thinking about this:

     cdef int a = 10, b = 0
     print a // b

which currently gives a C exception terminating the Python interpreter.

On one hand: We have already changed division to Python behaviour (with 
a speed penalty); so using the same arguments (which I won't reiterate) 
it would make sense to raise a Python exception instead here.

It seems to make sense to couple this as well to the cdivision directive 
and cython.cdiv function.

On the other hand: Well, all the arguments that was against changing // 
to Python behaviour.

But in addition:

This is coupled with the "exception under nogil" issue that I raised in 
another thread today, as one should definitely be allowed to do C 
division within a nogil section.

One solution would be to work on accepting exceptions under nogil, 
another one would be to require use of cdivision directive/cdiv to do 
division in nogil sections.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Apr 14 20:19:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 14 Apr 2009 20:19:40 +0200
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49E4D3BC.8040809@student.matnat.uio.no>

Thanks for the feedback. I'm quite keen to get this resolved myself so a 
patch is sure to follow once I can get the semantics part sorted.

Stefan Behnel wrote:
> Well, exceptions happen to be the error reporting mechanism in Python. In
> any case, the above must exit the nogil block immediately and should not
> crash (as long as we can prevent it in a reasonable way). To me, this
> sounds like an exception is the right thing.
> 
> However, it's quite possible that this kind of control flow makes error
> handling and C-level cleanup measures a nightmare. How would such a nogil
> block interact with try-except or try-finally, for example? We can't allow
> try-except /inside/ the block, so you'd have to anticipate all errors and
> wrap separate sections of the nogil block in their own try-except or
> try-finally statements to make sure the cleanup works as required.

I'm not sure I get what you mean here; but anyway:

Looking at generated C source, it seems that "with nogil" sets up a 
plain "try/finally" block, with an acquisition of the GIL in the finally 
part. So even as it is it is perfectly possible to raise an exception 
the normal way; labels are set correctly and the exception-raising-goto 
path is resumed after acquisition of the GIL.

However I don't think any code makes use of this (?), and if it is, it 
raises an issue with the (lack of) thread-safety of the 
__pyx_filename/lineno/clineno variables (which in other places are 
protected by the GIL).

How about this:

Change things so that instead of a GIL block setting up a "try/finally" 
to reacquire the GIL, any exception-raising code within a GIL block is 
responsible for reacquiring the GIL. (Basically, code.funcstate.in_nogil 
is maintained and if nodes want to raise an exception they reacquire the 
GIL as part of doing that.)

>> Option 2) Only allow buffer use in nogil if @cython.boundscheck(False).
>> This has some usability problems; it would be a pain to turn on/off
>> nogil for debugging.
> 
> Sounds like a viable alternative, though. Should be followed if exception
> raising is not considered accurate from inside nogil blocks.

I might end up doing this for 0.11 and await input for the scheme above 
for 0.12 (if I decide to bother with 0.11).

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Apr 14 20:44:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Apr 2009 20:44:40 +0200
Subject: [Cython] very simple iterator crashing Cython-0.11
In-Reply-To: <530BC1C3-5A3C-404B-AD51-DD7B1DD5A61B@math.washington.edu>
References: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>	<e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>	<1c34ba170904140101l37100db6k7805277f5c50db9d@mail.gmail.com>
	<530BC1C3-5A3C-404B-AD51-DD7B1DD5A61B@math.washington.edu>
Message-ID: <49E4D998.1030205@behnel.de>

Hi,

Robert Bradshaw wrote:
> On Apr 14, 2009, at 1:01 AM, Riccardo Murri wrote:
> 
>> On Tue, Apr 14, 2009 at 1:17 AM, Lisandro Dalcin  
>> <dalcinl at gmail.com> wrote:
>>> What happens if you write "cdef class ..." instead of "cpdef class
>>> ..." on your pxd?
>>>
>> Same error.
> 
> I'm surprised that "cpdef class" works at all (it shouldn't). In any  
> case, this is most certainly a bug, we are tracking it at http:// 
> trac.cython.org/cython_trac/ticket/279 .

The syntax of the cdef statement is waaay too forgiving. It even allows you
to write

	cdef pass

for a "pass" statement. I'll try to fix it up.

Stefan

From robertwb at math.washington.edu  Tue Apr 14 20:53:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 11:53:17 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <49E4B718.5020502@student.matnat.uio.no>
References: <49E4B718.5020502@student.matnat.uio.no>
Message-ID: <67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>

On Apr 14, 2009, at 9:17 AM, Dag Sverre Seljebotn wrote:

> I was just thinking about this:
>
>      cdef int a = 10, b = 0
>      print a // b
>
> which currently gives a C exception terminating the Python  
> interpreter.
>
> On one hand: We have already changed division to Python behaviour  
> (with
> a speed penalty); so using the same arguments (which I won't  
> reiterate)
> it would make sense to raise a Python exception instead here.
>
> It seems to make sense to couple this as well to the cdivision  
> directive
> and cython.cdiv function.
>
> On the other hand: Well, all the arguments that was against  
> changing //
> to Python behaviour.

Since we changed it, we should chang it all the way. It will be  
somewhat uglier as this means when writing a / (c-c) one needs to  
store the denominator in a temp and check it rather than simply  
letting the expression through to C (I already did this for warnings).

> But in addition:
>
> This is coupled with the "exception under nogil" issue that I  
> raised in
> another thread today, as one should definitely be allowed to do C
> division within a nogil section.
>
> One solution would be to work on accepting exceptions under nogil,
> another one would be to require use of cdivision directive/cdiv to do
> division in nogil sections.

I don't think exceptions are safe without the gil. With cdivision on  
it won't raise anything.

- Robert


From collette at physics.ucla.edu  Tue Apr 14 21:04:46 2009
From: collette at physics.ucla.edu (Andrew Collette)
Date: Tue, 14 Apr 2009 12:04:46 -0700
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E4D3BC.8040809@student.matnat.uio.no>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49E4D3BC.8040809@student.matnat.uio.no>
Message-ID: <f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>

Just for another data point w.r.t. C callbacks and the GIL, I actually
use this technique in my project HDF5 for Python
(h5py.googlecode.com).  The HDF5 library allows you to set an
arbitrary callback function, so I set it to (abridged):

void err_callback(...) with gil:
    raise SomeException("Whoops!" )

The library functions I care about are all imported with an exception
clause as e.g:

void HDF5_read_data(...) except -1

so that when I call into HDF5, if an error occurs, a Python exception
is constructed automatically.  In particular, the code for reading
data is wrapped in a nogil block:

try:
    with nogil:
        HDF5_read_data(...)
finally:
    ... some cleanup ...

In practice this seems to work just fine, although I haven't
extensively tested it with threads.  However, all Cython functions in
this module are protected by (the same) reentrant lock as HDF5 isn't
thread safe, so maybe this has been helping me avoid trouble.

Importantly, if the function HDF5_read_data is declared "except *",
Cython immediately crashes if an exception is raised; apparently it's
not valid to check for an exception without holding the GIL.
Declaring with "except <n>" seems to work just fine.

The code in question is at:
http://code.google.com/p/h5py/source/browse/trunk/h5py/h5.pyx (line 651)
and
http://code.google.com/p/h5py/source/browse/trunk/h5py/h5d.pyx (line 161)

Andrew Collette

On Tue, Apr 14, 2009 at 11:19 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Thanks for the feedback. I'm quite keen to get this resolved myself so a
> patch is sure to follow once I can get the semantics part sorted.
>
> Stefan Behnel wrote:
>> Well, exceptions happen to be the error reporting mechanism in Python. In
>> any case, the above must exit the nogil block immediately and should not
>> crash (as long as we can prevent it in a reasonable way). To me, this
>> sounds like an exception is the right thing.
>>
>> However, it's quite possible that this kind of control flow makes error
>> handling and C-level cleanup measures a nightmare. How would such a nogil
>> block interact with try-except or try-finally, for example? We can't allow
>> try-except /inside/ the block, so you'd have to anticipate all errors and
>> wrap separate sections of the nogil block in their own try-except or
>> try-finally statements to make sure the cleanup works as required.
>
> I'm not sure I get what you mean here; but anyway:
>
> Looking at generated C source, it seems that "with nogil" sets up a
> plain "try/finally" block, with an acquisition of the GIL in the finally
> part. So even as it is it is perfectly possible to raise an exception
> the normal way; labels are set correctly and the exception-raising-goto
> path is resumed after acquisition of the GIL.
>
> However I don't think any code makes use of this (?), and if it is, it
> raises an issue with the (lack of) thread-safety of the
> __pyx_filename/lineno/clineno variables (which in other places are
> protected by the GIL).
>
> How about this:
>
> Change things so that instead of a GIL block setting up a "try/finally"
> to reacquire the GIL, any exception-raising code within a GIL block is
> responsible for reacquiring the GIL. (Basically, code.funcstate.in_nogil
> is maintained and if nodes want to raise an exception they reacquire the
> GIL as part of doing that.)
>
>>> Option 2) Only allow buffer use in nogil if @cython.boundscheck(False).
>>> This has some usability problems; it would be a pain to turn on/off
>>> nogil for debugging.
>>
>> Sounds like a viable alternative, though. Should be followed if exception
>> raising is not considered accurate from inside nogil blocks.
>
> I might end up doing this for 0.11 and await input for the scheme above
> for 0.12 (if I decide to bother with 0.11).
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From stefan_ml at behnel.de  Tue Apr 14 21:13:19 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Apr 2009 21:13:19 +0200
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>
References: <49E48475.9090406@student.matnat.uio.no>	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49E4D3BC.8040809@student.matnat.uio.no>
	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>
Message-ID: <49E4E04F.603@behnel.de>

Hi,

just a quick note on this:

Andrew Collette wrote:
> void HDF5_read_data(...) except -1
> [...]
> try:
>     with nogil:
>         HDF5_read_data(...)
> finally:
>     ... some cleanup ...
>
> Importantly, if the function HDF5_read_data is declared "except *",
> Cython immediately crashes if an exception is raised

It shouldn't allow you to do that at all. I assume you declared the
function "nogil" to call it in a "with nogil" block. Combining "nogil" with
exception handling should raise an error at compile time.

There may be cases where this rule could potentially be relaxed, but the
compiler should catch this early in any case.

Stefan


From dagss at student.matnat.uio.no  Tue Apr 14 21:18:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 14 Apr 2009 21:18:55 +0200 (CEST)
Subject: [Cython] C int division by zero
In-Reply-To: <67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
Message-ID: <8ffacc54543ae4591accfdb1a58bce93.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Apr 14, 2009, at 9:17 AM, Dag Sverre Seljebotn wrote:
>
>> I was just thinking about this:
>>
>>      cdef int a = 10, b = 0
>>      print a // b
>>
>> which currently gives a C exception terminating the Python
>> interpreter.
>>
>> On one hand: We have already changed division to Python behaviour
>> (with
>> a speed penalty); so using the same arguments (which I won't
>> reiterate)
>> it would make sense to raise a Python exception instead here.
>
> Since we changed it, we should chang it all the way. It will be
> somewhat uglier as this means when writing a / (c-c) one needs to
> store the denominator in a temp and check it rather than simply
> letting the expression through to C (I already did this for warnings).

We're already in beta, but still: Can we get a warning cycle for this as
well into 0.11.1 so that people only have to deal with C int semantic
changes once? I can probably make it tomorrow but I'll have a higher
learning curve as you did the previous change.

>> One solution would be to work on accepting exceptions under nogil,
>> another one would be to require use of cdivision directive/cdiv to do
>> division in nogil sections.
>
> I don't think exceptions are safe without the gil. With cdivision on
> it won't raise anything.

This is an exceptional case; I think it is OK to acquire the GIL
temporarily to construct the exception if the exception hits, as long as
non-exception paths doesn't require waiting for a GIL lock?

See also the other thread on this.

Dag Sverre


From stefan_ml at behnel.de  Tue Apr 14 21:24:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Apr 2009 21:24:40 +0200
Subject: [Cython] very simple iterator crashing Cython-0.11
In-Reply-To: <49E4D998.1030205@behnel.de>
References: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>	<e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>	<1c34ba170904140101l37100db6k7805277f5c50db9d@mail.gmail.com>	<530BC1C3-5A3C-404B-AD51-DD7B1DD5A61B@math.washington.edu>
	<49E4D998.1030205@behnel.de>
Message-ID: <49E4E2F8.5040907@behnel.de>


Stefan Behnel wrote:
> The syntax of the cdef statement is waaay too forgiving. It even allows you
> to write
> 
> 	cdef pass
> 
> for a "pass" statement.

I now wonder why "pass" appears inside the "cdef" parsing function at all.
I don't mean using "pass" inside of a cdef block, just right in the
declaration itself. Is there any combination of "cdef" with "pass" that
makes any sense? Any kind of cdef-extern, struct/enum/union or class
declaration line?

Stefan

From robertwb at math.washington.edu  Tue Apr 14 21:33:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 12:33:51 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <8ffacc54543ae4591accfdb1a58bce93.squirrel@webmail.uio.no>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
	<8ffacc54543ae4591accfdb1a58bce93.squirrel@webmail.uio.no>
Message-ID: <AF86289D-B18E-478D-89E5-0BEB5DFC2C05@math.washington.edu>

On Apr 14, 2009, at 12:18 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 14, 2009, at 9:17 AM, Dag Sverre Seljebotn wrote:
>>
>>> I was just thinking about this:
>>>
>>>      cdef int a = 10, b = 0
>>>      print a // b
>>>
>>> which currently gives a C exception terminating the Python
>>> interpreter.
>>>
>>> On one hand: We have already changed division to Python behaviour
>>> (with
>>> a speed penalty); so using the same arguments (which I won't
>>> reiterate)
>>> it would make sense to raise a Python exception instead here.
>>
>> Since we changed it, we should chang it all the way. It will be
>> somewhat uglier as this means when writing a / (c-c) one needs to
>> store the denominator in a temp and check it rather than simply
>> letting the expression through to C (I already did this for  
>> warnings).
>
> We're already in beta, but still: Can we get a warning cycle for  
> this as
> well into 0.11.1 so that people only have to deal with C int semantic
> changes once? I can probably make it tomorrow but I'll have a higher
> learning curve as you did the previous change.

Yes, I'll do this. I think segfault -> ZeroDivisionError exception  
will be an easy migration path for most people :).

>>> One solution would be to work on accepting exceptions under nogil,
>>> another one would be to require use of cdivision directive/cdiv  
>>> to do
>>> division in nogil sections.
>>
>> I don't think exceptions are safe without the gil. With cdivision on
>> it won't raise anything.
>
> This is an exceptional case; I think it is OK to acquire the GIL
> temporarily to construct the exception if the exception hits, as  
> long as
> non-exception paths doesn't require waiting for a GIL lock?
>
> See also the other thread on this.

Hmm... I don't have a definite opinion on this yet, but that might be  
reasonable. Propagating an exception requires the gil too...it  
dilutes the pureness of "no gil."

- Robert



From dagss at student.matnat.uio.no  Tue Apr 14 22:19:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 14 Apr 2009 22:19:24 +0200
Subject: [Cython] Exceptions under nogil
In-Reply-To: <c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49E4EFCC.30402@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Cython currently segfaults when using a buffer in nogil and accessing
>> out of bounds (which triggers an IndexError). This is a bug, however I
>> would like to do more than disallow it.
>>
>> I.e. what should happen here:
>>
>>      cdef object[int] buf = ... # buffer of length < 6
>>      with nogil:
>>          buf[5] = 0
>>
>> Option 1) Reacquire the GIL temporarily to construct the IndexError
>> instance. This seems to work wonderfully in my current patch; but is it
>> too unintuitive to raise an exception from an operation in a nogil
>> section?
> 
> Well, exceptions happen to be the error reporting mechanism in Python. In
> any case, the above must exit the nogil block immediately and should not
> crash (as long as we can prevent it in a reasonable way). To me, this
> sounds like an exception is the right thing.

However if the disadvantages outweighs the benefits... also, as Robert 
says, exception control flow would "water out" the nogil.

I'm OK with waiting for a GIL if an exception is raised, but on a mere 
function call seems much worse.

How about

  a) only flag for exceptions while in nogil, using a thread-local variable
  b) upon exiting the nogil section a 
ExceptionRaisedWhileInNoGilSectionError can be raised?
  c) Possibly add a function cython.nogil_exception_occurred so that 
exceptions can be checked for in nogil sections.

Try/except cannot ever be used in nogil sections after all.

After all, requiring compiler directives explictly silencing exceptions 
doesn't seem better than getting the exception a little late :-)

BTW, does anybody know if there's perhaps a way of querying 
PyErr_Occurred without the GIL, by Python-interpreter-dependant hacks if 
necesarry? It's just a read of a flag which is likely pointed to by a 
thread-local variable after all.

If this could be done I'm all for exception propagation in nogil 
sections (and a GIL is automatically acquired if exception must be 
constructed).

-- 
Dag Sverre

From carl.witty at gmail.com  Tue Apr 14 22:04:05 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Tue, 14 Apr 2009 13:04:05 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
Message-ID: <ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>

On Tue, Apr 14, 2009 at 11:53 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 14, 2009, at 9:17 AM, Dag Sverre Seljebotn wrote:
>
>> I was just thinking about this:
>>
>> ? ? ?cdef int a = 10, b = 0
>> ? ? ?print a // b
>>
>> which currently gives a C exception terminating the Python
>> interpreter.
>>
>> On one hand: We have already changed division to Python behaviour
>> (with
>> a speed penalty); so using the same arguments (which I won't
>> reiterate)
>> it would make sense to raise a Python exception instead here.
>>
>> It seems to make sense to couple this as well to the cdivision
>> directive
>> and cython.cdiv function.
>>
>> On the other hand: Well, all the arguments that was against
>> changing //
>> to Python behaviour.
>
> Since we changed it, we should chang it all the way. It will be
> somewhat uglier as this means when writing a / (c-c) one needs to
> store the denominator in a temp and check it rather than simply
> letting the expression through to C (I already did this for warnings).

If we're going through and listing special cases for division, there's
one more to consider (I think this is the last one):

On x86 processors, dividing -2**31 by -1 gives a processor signal;
under Linux, this results in exiting the process with "Floating point
exception".  (Even though it's not floating point.)  (Presumably, the
same thing happens when dividing -2**63 by -1 on a 64-bit processor; I
haven't tested.)

Carl

From collette at physics.ucla.edu  Tue Apr 14 22:35:17 2009
From: collette at physics.ucla.edu (Andrew Collette)
Date: Tue, 14 Apr 2009 13:35:17 -0700
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E4E04F.603@behnel.de>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49E4D3BC.8040809@student.matnat.uio.no>
	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>
	<49E4E04F.603@behnel.de>
Message-ID: <f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>

Hi,

Well, the function signature in question is (in the referred file h5d.pyx):

"cdef int H5PY_H5Dread(<stuff>) nogil except -1"

which compiles just fine... is this going to stop working?  Is there a
better way to do it?  It seems like "except -1" simply  translates to
"after this function is called, if the return code is -1, a Python
exception happened in this thread".  It seems like the GIL should be
reacquired, the nogil block exited immediately, and the appropriate
branches of any try...except...finally statement should be executed.
If -1 isn't returned, the GIL isn't touched.

Whatever happens, I need to be able to run nogil code (reading in
data) which can possibly acquire the GIL and set an exception (via a
callback), and then be able to detect and propagate the exception back
to Python.   I am certainly open to refactoring the code if there's a
better way to do this in Cython.

Andrew Collette


On Tue, Apr 14, 2009 at 12:13 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> just a quick note on this:
>
> Andrew Collette wrote:
>> void HDF5_read_data(...) except -1
>> [...]
>> try:
>> ? ? with nogil:
>> ? ? ? ? HDF5_read_data(...)
>> finally:
>> ? ? ... some cleanup ...
>>
>> Importantly, if the function HDF5_read_data is declared "except *",
>> Cython immediately crashes if an exception is raised
>
> It shouldn't allow you to do that at all. I assume you declared the
> function "nogil" to call it in a "with nogil" block. Combining "nogil" with
> exception handling should raise an error at compile time.
>
> There may be cases where this rule could potentially be relaxed, but the
> compiler should catch this early in any case.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From stefan_ml at behnel.de  Tue Apr 14 22:47:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Apr 2009 22:47:17 +0200
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
References: <49E48475.9090406@student.matnat.uio.no>	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49E4D3BC.8040809@student.matnat.uio.no>	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>	<49E4E04F.603@behnel.de>
	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
Message-ID: <49E4F655.8080001@behnel.de>

Hi,

please don't top-post.

Andrew Collette wrote:
> Well, the function signature in question is (in the referred file h5d.pyx):
> 
> "cdef int H5PY_H5Dread(<stuff>) nogil except -1"
> 
> which compiles just fine... is this going to stop working?

I think this is the one case where exception propagation will easily work
in a nogil block. If we get the exception propagation semantics right here,
we can leave this in. But it's not entirely clear (at least to me) if this
currently behaves 'correctly'.

It *might* be possible to enable the same for "except *" somehow, but until
then, Cython should raise an error at compile time rather than letting the
code crash at runtime.

Stefan


From stefan_ml at behnel.de  Tue Apr 14 23:14:49 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Apr 2009 23:14:49 +0200
Subject: [Cython] very simple iterator crashing Cython-0.11
In-Reply-To: <49E4D998.1030205@behnel.de>
References: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>	<e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>	<1c34ba170904140101l37100db6k7805277f5c50db9d@mail.gmail.com>	<530BC1C3-5A3C-404B-AD51-DD7B1DD5A61B@math.washington.edu>
	<49E4D998.1030205@behnel.de>
Message-ID: <49E4FCC9.80704@behnel.de>


Stefan Behnel wrote:
> The syntax of the cdef statement is waaay too forgiving. It even allows you
> to write
> 
> 	cdef pass
> 
> for a "pass" statement. I'll try to fix it up.

Patch is up at

http://trac.cython.org/cython_trac/ticket/280

Not sure if this is safe to go into 0.11.1, so I'd be happy to see it
tested on some real code.

Stefan

From collette at physics.ucla.edu  Tue Apr 14 23:56:23 2009
From: collette at physics.ucla.edu (Andrew Collette)
Date: Tue, 14 Apr 2009 14:56:23 -0700
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E4F655.8080001@behnel.de>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49E4D3BC.8040809@student.matnat.uio.no>
	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>
	<49E4E04F.603@behnel.de>
	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
	<49E4F655.8080001@behnel.de>
Message-ID: <f125f5e0904141456k1c81d0bbl6e88e30b96c8ddf1@mail.gmail.com>

> please don't top-post.

Well excuse me. :) I didn't notice Gmail was stuffing the whole thing
under my reply.

> I think this is the one case where exception propagation will easily work
> in a nogil block. If we get the exception propagation semantics right here,
> we can leave this in. But it's not entirely clear (at least to me) if this
> currently behaves 'correctly'.

OK, it sounds like I should rework this so that it fits better with
the current defined behavior of Cython, to avoid unexpected breakage.

> It *might* be possible to enable the same for "except *" somehow, but until
> then, Cython should raise an error at compile time rather than letting the
> code crash at runtime.

I agree, this shouldn't even compile with "except *"; re-acquiring the
GIL inside a nogil block every time an "except-*" declared function
returns seems like it's too surprising, especially in the (common)
case of infrequently raised exceptions.  Nogil code shouldn't touch
the GIL unless it really, really has to.

Thanks,
Andrew Collette

From robertwb at math.washington.edu  Wed Apr 15 00:27:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 15:27:08 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
Message-ID: <9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>

On Apr 14, 2009, at 1:04 PM, Carl Witty wrote:

> On Tue, Apr 14, 2009 at 11:53 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 14, 2009, at 9:17 AM, Dag Sverre Seljebotn wrote:
>>
>>> I was just thinking about this:
>>>
>>>      cdef int a = 10, b = 0
>>>      print a // b
>>>
>>> which currently gives a C exception terminating the Python
>>> interpreter.
>>>
>>> On one hand: We have already changed division to Python behaviour
>>> (with
>>> a speed penalty); so using the same arguments (which I won't
>>> reiterate)
>>> it would make sense to raise a Python exception instead here.
>>>
>>> It seems to make sense to couple this as well to the cdivision
>>> directive
>>> and cython.cdiv function.
>>>
>>> On the other hand: Well, all the arguments that was against
>>> changing //
>>> to Python behaviour.
>>
>> Since we changed it, we should chang it all the way. It will be
>> somewhat uglier as this means when writing a / (c-c) one needs to
>> store the denominator in a temp and check it rather than simply
>> letting the expression through to C (I already did this for  
>> warnings).

OK, pushed. http://hg.cython.org/cython-devel/rev/a09a309febdc

> If we're going through and listing special cases for division, there's
> one more to consider (I think this is the last one):
>
> On x86 processors, dividing -2**31 by -1 gives a processor signal;
> under Linux, this results in exiting the process with "Floating point
> exception".  (Even though it's not floating point.)  (Presumably, the
> same thing happens when dividing -2**63 by -1 on a 64-bit processor; I
> haven't tested.)

Interesting corner case. This seems more like an overflow issue, and  
we're currently not trying to protect the user from those.

- Robert


From robertwb at math.washington.edu  Wed Apr 15 01:14:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 16:14:39 -0700
Subject: [Cython] very simple iterator crashing Cython-0.11
In-Reply-To: <49E4FCC9.80704@behnel.de>
References: <1c34ba170904131414s42ea8446o7c06b5bbbd876705@mail.gmail.com>	<e7ba66e40904131617n3d464c8ct2dd1dae693f651f1@mail.gmail.com>	<1c34ba170904140101l37100db6k7805277f5c50db9d@mail.gmail.com>	<530BC1C3-5A3C-404B-AD51-DD7B1DD5A61B@math.washington.edu>
	<49E4D998.1030205@behnel.de> <49E4FCC9.80704@behnel.de>
Message-ID: <08592E04-036D-464C-94B2-32AF1473663F@math.washington.edu>

On Apr 14, 2009, at 2:14 PM, Stefan Behnel wrote:

>
> Stefan Behnel wrote:
>> The syntax of the cdef statement is waaay too forgiving. It even  
>> allows you
>> to write
>>
>> 	cdef pass
>>
>> for a "pass" statement. I'll try to fix it up.
>
> Patch is up at
>
> http://trac.cython.org/cython_trac/ticket/280
>
> Not sure if this is safe to go into 0.11.1, so I'd be happy to see it
> tested on some real code.

I don't see anything wrong with it, but better safe than sorry.

- Robert


From robertwb at math.washington.edu  Wed Apr 15 01:21:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 16:21:23 -0700
Subject: [Cython] Cython 0.11.1 release candidate
Message-ID: <27D772B7-E8B6-4A8A-9701-49D7C10A9DA0@math.washington.edu>

Unless something is horribly broken, I will release in the next day  
or two.

http://cython.org/Cython-0.11.1.rc1.tar.gz

- Robert

From dagss at student.matnat.uio.no  Wed Apr 15 08:44:19 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Apr 2009 08:44:19 +0200
Subject: [Cython] C int division by zero
In-Reply-To: <9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
References: <49E4B718.5020502@student.matnat.uio.no>	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
Message-ID: <49E58243.6090704@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 14, 2009, at 1:04 PM, Carl Witty wrote:
>> On x86 processors, dividing -2**31 by -1 gives a processor signal;
>> under Linux, this results in exiting the process with "Floating point
>> exception".  (Even though it's not floating point.)  (Presumably, the
>> same thing happens when dividing -2**63 by -1 on a 64-bit processor; I
>> haven't tested.)
> 
> Interesting corner case. This seems more like an overflow issue, and  
> we're currently not trying to protect the user from those.

Well, yes, but usually with overflows one just gets the overflowed value 
silently. With division overflow one crashes the application, which is 
IMO much worse even if it is a very rare corner case.

I think we should raise OverflowError here under the same circumstances 
as ZeroDivisionError.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Apr 15 08:49:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Apr 2009 08:49:27 +0200
Subject: [Cython] C int division by zero
In-Reply-To: <9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
References: <49E4B718.5020502@student.matnat.uio.no>	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
Message-ID: <49E58377.8020103@student.matnat.uio.no>

Robert Bradshaw wrote:
> 
> OK, pushed. http://hg.cython.org/cython-devel/rev/a09a309febdc

Note that this now segfaults:

@cython.cdivision(False)
def f():
     cdef int a=10, b=0
     with nogil:
         a = a // b
     return a

-- 
Dag Sverre

From stefan_ml at behnel.de  Wed Apr 15 08:35:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Apr 2009 08:35:12 +0200 (CEST)
Subject: [Cython] C int division by zero
In-Reply-To: <9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
Message-ID: <4d3dda8bda5f2d8cd9b626b526cac84f.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
>>> On Apr 14, 2009, at 9:17 AM, Dag Sverre Seljebotn wrote:
>>>
>>>> I was just thinking about this:
>>>>
>>>>      cdef int a = 10, b = 0
>>>>      print a // b
>>>>
>>>> which currently gives a C exception terminating the Python
>>>> interpreter.
>>>
>>> Since we changed it, we should chang it all the way. It will be
>>> somewhat uglier as this means when writing a / (c-c) one needs to
>>> store the denominator in a temp and check it rather than simply
>>> letting the expression through to C (I already did this for
>>> warnings).
>
> OK, pushed. http://hg.cython.org/cython-devel/rev/a09a309febdc

Note that this requires a GIL check now (just like the warning case),
haven't checked if this case is currently handled.

Stefan


From robertwb at math.washington.edu  Wed Apr 15 08:41:34 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 23:41:34 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <49E58377.8020103@student.matnat.uio.no>
References: <49E4B718.5020502@student.matnat.uio.no>	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
	<49E58377.8020103@student.matnat.uio.no>
Message-ID: <B21DBA68-07FD-4C89-B28F-91985AE1DF6C@math.washington.edu>

On Apr 14, 2009, at 11:49 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>
>> OK, pushed. http://hg.cython.org/cython-devel/rev/a09a309febdc
>
> Note that this now segfaults:
>
> @cython.cdivision(False)
> def f():
>      cdef int a=10, b=0
>      with nogil:
>          a = a // b
>      return a

Ah, because raising the exception requires the GIL. I'll have to add  
a check in this case.

- Robert


From robertwb at math.washington.edu  Wed Apr 15 08:42:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 23:42:40 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <49E58243.6090704@student.matnat.uio.no>
References: <49E4B718.5020502@student.matnat.uio.no>	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
	<49E58243.6090704@student.matnat.uio.no>
Message-ID: <0EB5D934-2764-464E-A7E8-38D614FAC5A9@math.washington.edu>

On Apr 14, 2009, at 11:44 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 14, 2009, at 1:04 PM, Carl Witty wrote:
>>> On x86 processors, dividing -2**31 by -1 gives a processor signal;
>>> under Linux, this results in exiting the process with "Floating  
>>> point
>>> exception".  (Even though it's not floating point.)  (Presumably,  
>>> the
>>> same thing happens when dividing -2**63 by -1 on a 64-bit  
>>> processor; I
>>> haven't tested.)
>>
>> Interesting corner case. This seems more like an overflow issue, and
>> we're currently not trying to protect the user from those.
>
> Well, yes, but usually with overflows one just gets the overflowed  
> value
> silently. With division overflow one crashes the application, which is
> IMO much worse even if it is a very rare corner case.
>
> I think we should raise OverflowError here under the same  
> circumstances
> as ZeroDivisionError.

Ah, I hadn't realized it segfaults (well, it does on my machine). OK,  
another check coming right up.

- Robert



From stefan_ml at behnel.de  Wed Apr 15 08:43:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Apr 2009 08:43:00 +0200 (CEST)
Subject: [Cython] C int division by zero
In-Reply-To: <49E58243.6090704@student.matnat.uio.no>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
	<49E58243.6090704@student.matnat.uio.no>
Message-ID: <998260cf7038809d556b704c199bb10e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> usually with overflows one just gets the overflowed value
> silently. With division overflow one crashes the application, which is
> IMO much worse even if it is a very rare corner case.
>
> I think we should raise OverflowError here under the same circumstances
> as ZeroDivisionError.

+1, this matches the patch Lisandro provided.

BTW, isn't there a way to catch these signals in a way that would allow us
to raise an exception instead? Catching signals isn't the hard part, I
guess, but we'd have to extract the current state and code position from
the signal handler, so that we can jump back to the right label and
continue.

As long as they don't occur, signals are a lot cheaper than doing runtime
checks all over the place.

Stefan


From robertwb at math.washington.edu  Wed Apr 15 08:53:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Apr 2009 23:53:38 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <998260cf7038809d556b704c199bb10e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
	<49E58243.6090704@student.matnat.uio.no>
	<998260cf7038809d556b704c199bb10e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <71603C95-E8C0-4933-BA96-CBA9005BFE97@math.washington.edu>

On Apr 14, 2009, at 11:43 PM, Stefan Behnel wrote:

> Dag Sverre Seljebotn wrote:
>> usually with overflows one just gets the overflowed value
>> silently. With division overflow one crashes the application,  
>> which is
>> IMO much worse even if it is a very rare corner case.
>>
>> I think we should raise OverflowError here under the same  
>> circumstances
>> as ZeroDivisionError.
>
> +1, this matches the patch Lisandro provided.
>
> BTW, isn't there a way to catch these signals in a way that would  
> allow us
> to raise an exception instead? Catching signals isn't the hard part, I
> guess, but we'd have to extract the current state and code position  
> from
> the signal handler, so that we can jump back to the right label and
> continue.
>
> As long as they don't occur, signals are a lot cheaper than doing  
> runtime
> checks all over the place.

Yes, this would be much better. I just am not familiar enough with  
signals to know how to trap them. Also, are there platform  
compatibility issues? (We may be able to use macros which use signals  
when available, and runtime tests if not.)

In any case, I'm going to push a patch to get the behavior right the  
"easy way" for now.

- Robert


From dagss at student.matnat.uio.no  Wed Apr 15 09:14:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Apr 2009 09:14:40 +0200
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E4F655.8080001@behnel.de>
References: <49E48475.9090406@student.matnat.uio.no>	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49E4D3BC.8040809@student.matnat.uio.no>	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>	<49E4E04F.603@behnel.de>	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
	<49E4F655.8080001@behnel.de>
Message-ID: <49E58960.7090802@student.matnat.uio.no>

Stefan Behnel wrote:
> I think this is the one case where exception propagation will easily work
> in a nogil block. If we get the exception propagation semantics right here,
> we can leave this in. But it's not entirely clear (at least to me) if this
> currently behaves 'correctly'.
> 
> It *might* be possible to enable the same for "except *" somehow, but until
> then, Cython should raise an error at compile time rather than letting the
> code crash at runtime.

I think we can pull of except * without the GIL:

When releasing the GIL, we get a PyThreadState* pointing to our current 
thread state, in _save.

When nested functions reacquire the GIL in order to construct and raise 
an exception, they should get the same PyThreadState* object.

So back in our original function, we can simply directly inspect 
_save->exc_type rather than use PyErr_Occurred.

All in all I propose this:
  - It is possible to raise exceptions from a nogil context, by fetching 
the GIL temporarily to raise the exception, release the GIL, and return 
the normal exceptional way.
  - In order to propagate the exception, nogil code can look at 
_save->exc_type, which is very fast.
  - If returning from a nogil function with an exception set, the 
function will acquire the GIL, push exception stack information, and 
release the GIL.

This will mean that
a) Normally nogil code does not acquire the GIL.
b) If an exception is raised, the GIL is still acquired/reacquired, once 
per stack level.

-- 
Dag Sverre

From robertwb at math.washington.edu  Wed Apr 15 09:31:01 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 15 Apr 2009 00:31:01 -0700
Subject: [Cython] C int division by zero
In-Reply-To: <71603C95-E8C0-4933-BA96-CBA9005BFE97@math.washington.edu>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
	<49E58243.6090704@student.matnat.uio.no>
	<998260cf7038809d556b704c199bb10e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<71603C95-E8C0-4933-BA96-CBA9005BFE97@math.washington.edu>
Message-ID: <CB38D686-3918-45A1-91E5-CB3872FF7CCA@math.washington.edu>

On Apr 14, 2009, at 11:53 PM, Robert Bradshaw wrote:

> In any case, I'm going to push a patch to get the behavior right the
> "easy way" for now.

OK, http://hg.cython.org/cython-devel/rev/bbf007cb50f5 . Thanks Carl  
and Dag for pointing these bugs out.

- Robert


From stefan_ml at behnel.de  Wed Apr 15 10:49:45 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Apr 2009 10:49:45 +0200 (CEST)
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E58960.7090802@student.matnat.uio.no>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49E4D3BC.8040809@student.matnat.uio.no>
	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>
	<49E4E04F.603@behnel.de>
	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
	<49E4F655.8080001@behnel.de> <49E58960.7090802@student.matnat.uio.no>
Message-ID: <30350dc4037cf40ac415a1b6685c91fe.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> So back in our original function, we can simply directly inspect
> _save->exc_type rather than use PyErr_Occurred.

Just for correctness, that would be _save->curexc_type.

Stefan


From stefan_ml at behnel.de  Wed Apr 15 10:53:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Apr 2009 10:53:52 +0200 (CEST)
Subject: [Cython] C int division by zero
In-Reply-To: <CB38D686-3918-45A1-91E5-CB3872FF7CCA@math.washington.edu>
References: <49E4B718.5020502@student.matnat.uio.no>
	<67E81388-686E-4B1E-A878-EBBD3B8A3662@math.washington.edu>
	<ac1e4c420904141304o20e87d71tfd0fc5d3ced9a681@mail.gmail.com>
	<9411A55A-C607-4303-A7A7-64F259CC8E21@math.washington.edu>
	<49E58243.6090704@student.matnat.uio.no>
	<998260cf7038809d556b704c199bb10e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<71603C95-E8C0-4933-BA96-CBA9005BFE97@math.washington.edu>
	<CB38D686-3918-45A1-91E5-CB3872FF7CCA@math.washington.edu>
Message-ID: <2a94782f6aed546bd32424fcb11fd68e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Apr 14, 2009, at 11:53 PM, Robert Bradshaw wrote:
>
>> In any case, I'm going to push a patch to get the behavior right the
>> "easy way" for now.
>
> OK, http://hg.cython.org/cython-devel/rev/bbf007cb50f5

Just a quick reminder that the GIL checks work different in
cython-unstable, where we have a dedicated tree visitor to do this. So
this is ok for cython-devel, but should be moved to the appropriate method
in cython-unstable.

Stefan


From stefan_ml at behnel.de  Wed Apr 15 11:04:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Apr 2009 11:04:46 +0200 (CEST)
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E58960.7090802@student.matnat.uio.no>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49E4D3BC.8040809@student.matnat.uio.no>
	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>
	<49E4E04F.603@behnel.de>
	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
	<49E4F655.8080001@behnel.de> <49E58960.7090802@student.matnat.uio.no>
Message-ID: <88afe4ab5f4012a00253e6add6830ac4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> When releasing the GIL, we get a PyThreadState* pointing to our current
> thread state, in _save.

That is true for the nogil-block case, but not for nogil functions.

I don't see a way to retrieve the original thread state from a nogil
function without acquiring the GIL first (maybe I'm missing something). So
how would calling a function that raises an exception work from within a
nogil function?

Stefan


From robertwb at math.washington.edu  Wed Apr 15 11:16:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 15 Apr 2009 02:16:55 -0700
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <49E58960.7090802@student.matnat.uio.no>
References: <49E48475.9090406@student.matnat.uio.no>	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49E4D3BC.8040809@student.matnat.uio.no>	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>	<49E4E04F.603@behnel.de>	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
	<49E4F655.8080001@behnel.de>
	<49E58960.7090802@student.matnat.uio.no>
Message-ID: <566293ED-750B-4737-B57E-AAC3123128BF@math.washington.edu>

On Apr 15, 2009, at 12:14 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> I think this is the one case where exception propagation will  
>> easily work
>> in a nogil block. If we get the exception propagation semantics  
>> right here,
>> we can leave this in. But it's not entirely clear (at least to me)  
>> if this
>> currently behaves 'correctly'.
>>
>> It *might* be possible to enable the same for "except *" somehow,  
>> but until
>> then, Cython should raise an error at compile time rather than  
>> letting the
>> code crash at runtime.
>
> I think we can pull of except * without the GIL:
>
> When releasing the GIL, we get a PyThreadState* pointing to our  
> current
> thread state, in _save.
>
> When nested functions reacquire the GIL in order to construct and  
> raise
> an exception, they should get the same PyThreadState* object.
>
> So back in our original function, we can simply directly inspect
> _save->exc_type rather than use PyErr_Occurred.
>
> All in all I propose this:
>   - It is possible to raise exceptions from a nogil context, by  
> fetching
> the GIL temporarily to raise the exception, release the GIL, and  
> return
> the normal exceptional way.
>   - In order to propagate the exception, nogil code can look at
> _save->exc_type, which is very fast.
>   - If returning from a nogil function with an exception set, the
> function will acquire the GIL, push exception stack information, and
> release the GIL.
>
> This will mean that
> a) Normally nogil code does not acquire the GIL.
> b) If an exception is raised, the GIL is still acquired/reacquired,  
> once
> per stack level.

In other words, you're changing the definition of nogil to be "gets  
the GIL if it needs" rather than "doesn't use (or need) the GIL."  
Granted, exceptions are exceptional, but I'm still a bit wary.  
Specifically, can implicitly acquiring the GIL for this cause block  
and/or lead to deadlock/race conditions?

- Robert



From robertwb at math.washington.edu  Wed Apr 15 11:31:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 15 Apr 2009 02:31:42 -0700
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>
Message-ID: <1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>

On Apr 5, 2009, at 12:26 PM, Hoyt Koepke wrote:

> Hello,
>
> I'm using an extension class where I need to create a number of
> instances quickly, and I'm looking at the section "Can Cython create
> objects or apply operators to locally created objects as pure C code?"
> on http://wiki.cython.org/FAQ.  Is this information still correct?
>
> In particular, I'm wondering about it in reference to supporting
> cyclical garbage collection and creating object via PyObject_GC_New
> etc.  IIRC, cython automatically supports cyclical garbage collection
> on classes that have python members, and my extension class falls into
> this category.  Should I
>
> a) Use Py_NEW, as specified in the FAQ?

Yes, if one has an absolute need for speed. Cyclic garbage collection  
is supported via a flag on the type.

> b) Replace Py_NEW with Py_Object_GC_New, and do a?

Py_Object_GC_New will allocate the space, but will not call the  
__cinit__ functions, and in particular will not initialize the base  
fields, so don't use this unless you really know what you're doing.

> c) Ignore this and just go with standard creation and deletion,
> ignoring the fact that I'll be calling the constructors through
> python.

This is what I would do, at least until I determined object creation  
was a large bottleneck. PY_NEW as stated on the wiki calls the  
__cinit__ functions, but avoids calling the __init__ functions, so  
depending on your usecase it can be a savings. Also, it would be  
desirable and feasible to potentially call __init__ via a C call  
rather than a Python call (maybe, if the signature is simple enough,  
using the cpdef trick to avoid the pythonic argument packing/ 
unpacking overhead.)

- Robert


From dagss at student.matnat.uio.no  Wed Apr 15 12:14:32 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Apr 2009 12:14:32 +0200
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <566293ED-750B-4737-B57E-AAC3123128BF@math.washington.edu>
References: <49E48475.9090406@student.matnat.uio.no>	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49E4D3BC.8040809@student.matnat.uio.no>	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>	<49E4E04F.603@behnel.de>	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>	<49E4F655.8080001@behnel.de>	<49E58960.7090802@student.matnat.uio.no>
	<566293ED-750B-4737-B57E-AAC3123128BF@math.washington.edu>
Message-ID: <49E5B388.8000408@student.matnat.uio.no>

Robert Bradshaw wrote:
>> This will mean that
>> a) Normally nogil code does not acquire the GIL.
>> b) If an exception is raised, the GIL is still acquired/reacquired,  
>> once
>> per stack level.
> 
> In other words, you're changing the definition of nogil to be "gets  
> the GIL if it needs" rather than "doesn't use (or need) the GIL."  
> Granted, exceptions are exceptional, but I'm still a bit wary.  
> Specifically, can implicitly acquiring the GIL for this cause block  
> and/or lead to deadlock/race conditions?

Thanks, that was very helpful to my thoughts. I'll be clearer.

This is actually two orthogonal issues:

I) Exception propagation in nogil.

The following currently works and is fully supported:

cdef int raise_some_error() except -1 with gil:
     raise ValueError("test")

def f():
     cdef int a
     try:
         with nogil:
             a = 0
             if 1 + 1 == 2: # very exceptional case
                 raise_some_error()
             a = 1
     except ValueError:
         a = 3
     print a # prints 3

However "except *" is not supported, which is a limitation to this.

II) Whether division and buffer access is allowed to imply a hidden 
"with gil" section in their exceptional cases.

To answer your question: The only thing I can think of is if you start 
spawning new threads using C functions; then reacquiring the GIL will 
fail using those threads, and calling functions declared "with gil" is 
perhaps more explicit and can more easily be avoided. (But that's a 
rather expert case and there's always compiler directives.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Apr 15 13:37:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Apr 2009 13:37:53 +0200
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <88afe4ab5f4012a00253e6add6830ac4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49E48475.9090406@student.matnat.uio.no>	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49E4D3BC.8040809@student.matnat.uio.no>	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>	<49E4E04F.603@behnel.de>	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>	<49E4F655.8080001@behnel.de>
	<49E58960.7090802@student.matnat.uio.no>
	<88afe4ab5f4012a00253e6add6830ac4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49E5C711.8040800@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> When releasing the GIL, we get a PyThreadState* pointing to our current
>> thread state, in _save.
> 
> That is true for the nogil-block case, but not for nogil functions.
> 
> I don't see a way to retrieve the original thread state from a nogil
> function without acquiring the GIL first (maybe I'm missing something). So
> how would calling a function that raises an exception work from within a
> nogil function?

No, you appear to be right, I didn't consider that :-(

However as there is already a way of propagating exceptions [1], using 
return values, it appears to be orthogonal to the question of whether 
buffer access or int division should imply an automatic "with gil" 
section raising an exception.

(We could try to get a patch into Python for PyErr_OccurredWithoutGIL, 
don't know if such a thing would be accepted -- it would be easy to 
write. And then only acquire a GIL in the current Python versions. After 
all, it would only happen for something which is currently not allowed.)

[1] From what I can see it works ok, except for a highly unlikely race 
condition on the __pyx_filename/lineno/... which can corrupt the 
exception stack description.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Apr 15 16:24:33 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Apr 2009 16:24:33 +0200
Subject: [Cython] Cython 0.11.1 release candidate
In-Reply-To: <27D772B7-E8B6-4A8A-9701-49D7C10A9DA0@math.washington.edu>
References: <27D772B7-E8B6-4A8A-9701-49D7C10A9DA0@math.washington.edu>
Message-ID: <49E5EE21.80505@student.matnat.uio.no>

Robert Bradshaw wrote:
> Unless something is horribly broken, I will release in the next day  
> or two.
> 
> http://cython.org/Cython-0.11.1.rc1.tar.gz

I know it is late (sorry!), but I just remembered some changes to 
directives behaviour which I'd like to get in; emitting a warning for 
change of behaviour in 0.12. I think they're rather safe, but I'll 
definitely take no for an answer.

Attaching a patch as I don't want to disturb the freeze (tell me and 
I'll push).

-- 
Dag Sverre


# HG changeset patch
# User Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
# Date 1239804937 -7200
# Node ID c8d2f8143120ecc3efd1e20d675b26a55e5c8684
# Parent  d22204d89cfdbdfa05e2f191b7a1b33d2aaa6dc2
Add warning in situations where command line directive behaviour will 
change in 0.12

diff -r d22204d89cfd -r c8d2f8143120 Cython/Compiler/ParseTreeTransforms.py
--- a/Cython/Compiler/ParseTreeTransforms.py    Wed Apr 15 15:53:26 2009 
+0200
+++ b/Cython/Compiler/ParseTreeTransforms.py    Wed Apr 15 16:15:37 2009 
+0200
@@ -332,13 +332,20 @@ class InterpretCompilerDirectives(Cython

      def __init__(self, context, compilation_option_overrides):
          super(InterpretCompilerDirectives, self).__init__(context)
-        self.compilation_option_overrides = compilation_option_overrides
+        self.compilation_option_overrides = {}
+        for key, value in compilation_option_overrides.iteritems():
+            self.compilation_option_overrides[unicode(key)] = value
          self.cython_module_names = set()
          self.option_names = {}

      # Set up processing and handle the cython: comments.
      def visit_ModuleNode(self, node):
          options = copy.copy(Options.option_defaults)
+        for key, value in self.compilation_option_overrides.iteritems():
+            if key in node.option_comments and 
node.option_comments[key] != value:
+                warning(node.pos, "Compiler directive differs between 
environment and file header; this will change "
+                        "in Cython 0.12. See 
http://article.gmane.org/gmane.comp.python.cython.devel/5233", 2)
+                break
          options.update(node.option_comments)
          options.update(self.compilation_option_overrides)
          self.options = options
# HG changeset patch
# User Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
# Date 1239803606 -7200
# Node ID d22204d89cfdbdfa05e2f191b7a1b33d2aaa6dc2
# Parent  bbf007cb50f57bcf0c010661e66b883191958aa3
Make -X cmdline switch apply to directives, as help text says

diff -r bbf007cb50f5 -r d22204d89cfd Cython/Compiler/CmdLine.py
--- a/Cython/Compiler/CmdLine.py        Wed Apr 15 00:27:57 2009 -0700
+++ b/Cython/Compiler/CmdLine.py        Wed Apr 15 15:53:26 2009 +0200
@@ -79,7 +79,7 @@ def parse_command_line(args):
                  options.use_listing_file = 1
              elif option in ("-C", "--compile"):
                  options.c_only = 0
-            elif option in ("-X", "--link"):
+            elif option in ("--link"):
                  options.c_only = 0
                  options.obj_only = 0
              elif option in ("-+", "--cplus"):




-------------- next part --------------
A non-text attachment was scrubbed...
Name: directives.diff
Type: text/x-diff
Size: 2598 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090415/4918b6fe/attachment-0001.bin 

From bpederse at gmail.com  Wed Apr 15 19:39:01 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 15 Apr 2009 10:39:01 -0700
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>
	<1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>
Message-ID: <e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com>

On Wed, Apr 15, 2009 at 2:31 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 5, 2009, at 12:26 PM, Hoyt Koepke wrote:
>
>> Hello,
>>
>> I'm using an extension class where I need to create a number of
>> instances quickly, and I'm looking at the section "Can Cython create
>> objects or apply operators to locally created objects as pure C code?"
>> on http://wiki.cython.org/FAQ. ?Is this information still correct?
>>
>> In particular, I'm wondering about it in reference to supporting
>> cyclical garbage collection and creating object via PyObject_GC_New
>> etc. ?IIRC, cython automatically supports cyclical garbage collection
>> on classes that have python members, and my extension class falls into
>> this category. ?Should I
>>
>> a) Use Py_NEW, as specified in the FAQ?
>
> Yes, if one has an absolute need for speed. Cyclic garbage collection
> is supported via a flag on the type.
>
>> b) Replace Py_NEW with Py_Object_GC_New, and do a?
>
> Py_Object_GC_New will allocate the space, but will not call the
> __cinit__ functions, and in particular will not initialize the base
> fields, so don't use this unless you really know what you're doing.
>
>> c) Ignore this and just go with standard creation and deletion,
>> ignoring the fact that I'll be calling the constructors through
>> python.
>
> This is what I would do, at least until I determined object creation
> was a large bottleneck. PY_NEW as stated on the wiki calls the
> __cinit__ functions, but avoids calling the __init__ functions, so
> depending on your usecase it can be a savings. Also, it would be
> desirable and feasible to potentially call __init__ via a C call
> rather than a Python call (maybe, if the signature is simple enough,
> using the cpdef trick to avoid the pythonic argument packing/
> unpacking overhead.)
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

hi, i have a use-case where i'm creating lots of objects as well. so i
wrote up a quick test to see the speed of various methods:
http://gist.github.com/95916

assuming i haven't done anything stupid, results in seconds for that case are:

PY_NEW on Cython class 0.652909994125
__init__ on Python class 1.51110291481
batch PY_NEW 0.434900999069
__init__ on Cython class 0.673034906387
batch __init__ on Cython class 0.532235145569

-b

From stefan_ml at behnel.de  Wed Apr 15 20:25:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Apr 2009 20:25:44 +0200
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>	<1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>
	<e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com>
Message-ID: <49E626A8.4000608@behnel.de>

Hi,

thanks for sharing that.

Brent Pedersen wrote:
> assuming i haven't done anything stupid

No, nothing stupid, but something that can reduce the comparability of the
timings. You are creating a 1000000 item list on each benchmark, using a
call to range() in some cases and a list comprehension in others.

It's usually better to move initialisations out of the timings, e.g. by
creating a large range() object once and re-using it. That reduces the
impact of unrelated operations on the absolute numbers.

Stefan


From robertwb at math.washington.edu  Wed Apr 15 20:25:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 15 Apr 2009 11:25:23 -0700
Subject: [Cython] Cython 0.11.1 release candidate
In-Reply-To: <49E5EE21.80505@student.matnat.uio.no>
References: <27D772B7-E8B6-4A8A-9701-49D7C10A9DA0@math.washington.edu>
	<49E5EE21.80505@student.matnat.uio.no>
Message-ID: <27887959-53F9-491B-8991-7EC5535CD251@math.washington.edu>

On Apr 15, 2009, at 7:24 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> Unless something is horribly broken, I will release in the next  
>> day  or two.
>> http://cython.org/Cython-0.11.1.rc1.tar.gz
>
> I know it is late (sorry!), but I just remembered some changes to  
> directives behaviour which I'd like to get in; emitting a warning  
> for change of behaviour in 0.12. I think they're rather safe, but  
> I'll definitely take no for an answer.

The warning looks like it'd be fine to slip in at the last moment,  
but I'm not as sure about removing the -X option--perhaps a warning  
about that would be in order though.

> Attaching a patch as I don't want to disturb the freeze (tell me  
> and I'll push).
>
> -- 
> Dag Sverre
>
>
> # HG changeset patch
> # User Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
> # Date 1239804937 -7200
> # Node ID c8d2f8143120ecc3efd1e20d675b26a55e5c8684
> # Parent  d22204d89cfdbdfa05e2f191b7a1b33d2aaa6dc2
> Add warning in situations where command line directive behaviour  
> will change in 0.12
>
> diff -r d22204d89cfd -r c8d2f8143120 Cython/Compiler/ 
> ParseTreeTransforms.py
> --- a/Cython/Compiler/ParseTreeTransforms.py    Wed Apr 15 15:53:26  
> 2009 +0200
> +++ b/Cython/Compiler/ParseTreeTransforms.py    Wed Apr 15 16:15:37  
> 2009 +0200
> @@ -332,13 +332,20 @@ class InterpretCompilerDirectives(Cython
>
>      def __init__(self, context, compilation_option_overrides):
>          super(InterpretCompilerDirectives, self).__init__(context)
> -        self.compilation_option_overrides =  
> compilation_option_overrides
> +        self.compilation_option_overrides = {}
> +        for key, value in compilation_option_overrides.iteritems():
> +            self.compilation_option_overrides[unicode(key)] = value
>          self.cython_module_names = set()
>          self.option_names = {}
>
>      # Set up processing and handle the cython: comments.
>      def visit_ModuleNode(self, node):
>          options = copy.copy(Options.option_defaults)
> +        for key, value in  
> self.compilation_option_overrides.iteritems():
> +            if key in node.option_comments and node.option_comments 
> [key] != value:
> +                warning(node.pos, "Compiler directive differs  
> between environment and file header; this will change "
> +                        "in Cython 0.12. See http:// 
> article.gmane.org/gmane.comp.python.cython.devel/5233", 2)
> +                break
>          options.update(node.option_comments)
>          options.update(self.compilation_option_overrides)
>          self.options = options
> # HG changeset patch
> # User Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
> # Date 1239803606 -7200
> # Node ID d22204d89cfdbdfa05e2f191b7a1b33d2aaa6dc2
> # Parent  bbf007cb50f57bcf0c010661e66b883191958aa3
> Make -X cmdline switch apply to directives, as help text says
>
> diff -r bbf007cb50f5 -r d22204d89cfd Cython/Compiler/CmdLine.py
> --- a/Cython/Compiler/CmdLine.py        Wed Apr 15 00:27:57 2009 -0700
> +++ b/Cython/Compiler/CmdLine.py        Wed Apr 15 15:53:26 2009 +0200
> @@ -79,7 +79,7 @@ def parse_command_line(args):
>                  options.use_listing_file = 1
>              elif option in ("-C", "--compile"):
>                  options.c_only = 0
> -            elif option in ("-X", "--link"):
> +            elif option in ("--link"):
>                  options.c_only = 0
>                  options.obj_only = 0
>              elif option in ("-+", "--cplus"):
>
>
>
>
> # HG changeset patch
> # User Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
> # Date 1239804937 -7200
> # Node ID c8d2f8143120ecc3efd1e20d675b26a55e5c8684
> # Parent  d22204d89cfdbdfa05e2f191b7a1b33d2aaa6dc2
> Add warning in situations where command line directive behaviour  
> will change in 0.12
>
> diff -r d22204d89cfd -r c8d2f8143120 Cython/Compiler/ 
> ParseTreeTransforms.py
> --- a/Cython/Compiler/ParseTreeTransforms.py	Wed Apr 15 15:53:26  
> 2009 +0200
> +++ b/Cython/Compiler/ParseTreeTransforms.py	Wed Apr 15 16:15:37  
> 2009 +0200
> @@ -332,13 +332,20 @@ class InterpretCompilerDirectives(Cython
>
>      def __init__(self, context, compilation_option_overrides):
>          super(InterpretCompilerDirectives, self).__init__(context)
> -        self.compilation_option_overrides =  
> compilation_option_overrides
> +        self.compilation_option_overrides = {}
> +        for key, value in compilation_option_overrides.iteritems():
> +            self.compilation_option_overrides[unicode(key)] = value
>          self.cython_module_names = set()
>          self.option_names = {}
>
>      # Set up processing and handle the cython: comments.
>      def visit_ModuleNode(self, node):
>          options = copy.copy(Options.option_defaults)
> +        for key, value in  
> self.compilation_option_overrides.iteritems():
> +            if key in node.option_comments and node.option_comments 
> [key] != value:
> +                warning(node.pos, "Compiler directive differs  
> between environment and file header; this will change "
> +                        "in Cython 0.12. See http:// 
> article.gmane.org/gmane.comp.python.cython.devel/5233", 2)
> +                break
>          options.update(node.option_comments)
>          options.update(self.compilation_option_overrides)
>          self.options = options
> # HG changeset patch
> # User Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
> # Date 1239803606 -7200
> # Node ID d22204d89cfdbdfa05e2f191b7a1b33d2aaa6dc2
> # Parent  bbf007cb50f57bcf0c010661e66b883191958aa3
> Make -X cmdline switch apply to directives, as help text says
>
> diff -r bbf007cb50f5 -r d22204d89cfd Cython/Compiler/CmdLine.py
> --- a/Cython/Compiler/CmdLine.py	Wed Apr 15 00:27:57 2009 -0700
> +++ b/Cython/Compiler/CmdLine.py	Wed Apr 15 15:53:26 2009 +0200
> @@ -79,7 +79,7 @@ def parse_command_line(args):
>                  options.use_listing_file = 1
>              elif option in ("-C", "--compile"):
>                  options.c_only = 0
> -            elif option in ("-X", "--link"):
> +            elif option in ("--link"):
>                  options.c_only = 0
>                  options.obj_only = 0
>              elif option in ("-+", "--cplus"):
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Wed Apr 15 22:14:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Apr 2009 22:14:00 +0200
Subject: [Cython] Cython 0.11.1 release candidate
In-Reply-To: <27887959-53F9-491B-8991-7EC5535CD251@math.washington.edu>
References: <27D772B7-E8B6-4A8A-9701-49D7C10A9DA0@math.washington.edu>	<49E5EE21.80505@student.matnat.uio.no>
	<27887959-53F9-491B-8991-7EC5535CD251@math.washington.edu>
Message-ID: <49E64008.4000203@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 15, 2009, at 7:24 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> Unless something is horribly broken, I will release in the next  
>>> day  or two.
>>> http://cython.org/Cython-0.11.1.rc1.tar.gz
>> I know it is late (sorry!), but I just remembered some changes to  
>> directives behaviour which I'd like to get in; emitting a warning  
>> for change of behaviour in 0.12. I think they're rather safe, but  
>> I'll definitely take no for an answer.
> 
> The warning looks like it'd be fine to slip in at the last moment,  
> but I'm not as sure about removing the -X option--perhaps a warning  
> about that would be in order though.

OK, I've pushed changes to this effect. This also fixes the help text 
which is currently WRONG in this area.

In the list of patches I should have done earlier, there's also

http://trac.cython.org/cython_trac/attachment/ticket/210/buffernogil.patch

Again I'd be really surprised to have this break anything, and it is 
something there's been expressed a need for in the NumPy community.

I know how irritating this must be though :-) If you say no I won't 
complain.

-- 
Dag Sverre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: deprecateX.patch
Type: text/x-diff
Size: 1665 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090415/8586ce36/attachment.bin 

From robertwb at math.washington.edu  Wed Apr 15 22:14:15 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 15 Apr 2009 13:14:15 -0700
Subject: [Cython] Cython 0.11.1 release candidate
In-Reply-To: <49E64008.4000203@student.matnat.uio.no>
References: <27D772B7-E8B6-4A8A-9701-49D7C10A9DA0@math.washington.edu>	<49E5EE21.80505@student.matnat.uio.no>
	<27887959-53F9-491B-8991-7EC5535CD251@math.washington.edu>
	<49E64008.4000203@student.matnat.uio.no>
Message-ID: <E7EF79BB-55A9-4671-A298-ED280E969F04@math.washington.edu>

On Apr 15, 2009, at 1:14 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 15, 2009, at 7:24 AM, Dag Sverre Seljebotn wrote:
>>> Robert Bradshaw wrote:
>>>> Unless something is horribly broken, I will release in the next   
>>>> day  or two.
>>>> http://cython.org/Cython-0.11.1.rc1.tar.gz
>>> I know it is late (sorry!), but I just remembered some changes  
>>> to  directives behaviour which I'd like to get in; emitting a  
>>> warning  for change of behaviour in 0.12. I think they're rather  
>>> safe, but  I'll definitely take no for an answer.
>> The warning looks like it'd be fine to slip in at the last  
>> moment,  but I'm not as sure about removing the -X option--perhaps  
>> a warning  about that would be in order though.
>
> OK, I've pushed changes to this effect. This also fixes the help  
> text which is currently WRONG in this area.

Thanks.

>
> In the list of patches I should have done earlier, there's also
>
> http://trac.cython.org/cython_trac/attachment/ticket/210/ 
> buffernogil.patch
>
> Again I'd be really surprised to have this break anything, and it  
> is something there's been expressed a need for in the NumPy community.
>
> I know how irritating this must be though :-) If you say no I won't  
> complain.

No. First thing for 0.11.2 (or 0.12) though. Unless a major bug is  
discovered, 0.11.1 will be the current tip of cython-devel http:// 
hg.cython.org/cython-devel/shortlog/53e4979c9b84 , released tonight.

- Robert


From bpederse at gmail.com  Thu Apr 16 04:31:29 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 15 Apr 2009 19:31:29 -0700
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <49E626A8.4000608@behnel.de>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>
	<1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>
	<e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com>
	<49E626A8.4000608@behnel.de>
Message-ID: <e183a99d0904151931l7b3e5997id566dbb511b39cda@mail.gmail.com>

On Wed, Apr 15, 2009 at 11:25 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> thanks for sharing that.
>
> Brent Pedersen wrote:
>> assuming i haven't done anything stupid
>
> No, nothing stupid, but something that can reduce the comparability of the
> timings. You are creating a 1000000 item list on each benchmark, using a
> call to range() in some cases and a list comprehension in others.
>
> It's usually better to move initialisations out of the timings, e.g. by
> creating a large range() object once and re-using it. That reduces the
> impact of unrelated operations on the absolute numbers.
>

ah, i see, updated that. fixing that makes the python constructor look
even slower.
now it assumes that creating a list comprehension without assgning to
a variable is
the same as calling a function that returns an array--also without assigning.

here are the new timings:

PY_NEW on Cython class: 1.137
__init__ on Python class: 28.468
__init__ on Python class with slots: 9.936
batch PY_NEW total: 0.821 , interval only: 0.363
batch __init__ on Cython class total 0.975 , interval_only: 0.524
__init__ on Cython class 1.154

so for this case using PY_NEW macro actual doesnt improve speed that
much over a cdef'ed class.
especially if using a "batch" method is applicable (as it is for my use-case).

i didnt realize slots would affect on object creation time that much
-- in this case it's 3x faster with slots!
(and then another 6-10x with a cdef'ed class)

here's my updated gist:
http://gist.github.com/95916


> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From greg.ewing at canterbury.ac.nz  Thu Apr 16 02:56:55 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 16 Apr 2009 12:56:55 +1200
Subject: [Cython] Poll: Buffer access exception under nogil
In-Reply-To: <566293ED-750B-4737-B57E-AAC3123128BF@math.washington.edu>
References: <49E48475.9090406@student.matnat.uio.no>
	<c37ebdbdf4fe28e78fbe0036ccb602c3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49E4D3BC.8040809@student.matnat.uio.no>
	<f125f5e0904141204s1626d871sf88a949388d82717@mail.gmail.com>
	<49E4E04F.603@behnel.de>
	<f125f5e0904141335u2711af22qdb11781ab3d63a28@mail.gmail.com>
	<49E4F655.8080001@behnel.de> <49E58960.7090802@student.matnat.uio.no>
	<566293ED-750B-4737-B57E-AAC3123128BF@math.washington.edu>
Message-ID: <49E68257.7020205@canterbury.ac.nz>

Robert Bradshaw wrote:

> In other words, you're changing the definition of nogil to be "gets  
> the GIL if it needs" rather than "doesn't use (or need) the GIL."  

But it has never meant that, at least as far as
Pyrex is concerned. It's always meant "this function
can be called without the gil" and no more. So
there's hing wrong with this in principle if you
can make it work.

-- 
Greg

From stefan_ml at behnel.de  Thu Apr 16 07:05:13 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Apr 2009 07:05:13 +0200
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <e183a99d0904151931l7b3e5997id566dbb511b39cda@mail.gmail.com>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>	<1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>	<e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com>	<49E626A8.4000608@behnel.de>
	<e183a99d0904151931l7b3e5997id566dbb511b39cda@mail.gmail.com>
Message-ID: <49E6BC89.7020400@behnel.de>

Hi,

now, those are interesting timings. Which versions of Cython and Python did
you use?

Brent Pedersen wrote:
> here are the new timings:
> 
> PY_NEW on Cython class: 1.137
> __init__ on Cython class 1.154

You're right, that's almost negligible. However, the __init__ benchmark
involves the overhead of a Python call to create_interval, which uses the
same Python calling convention as the constructor. So it's more or less
doing the same thing in both cases, just additionally calling __init__
through a C struct member dereference in the second case (but without
repacking the arguments).

Also note that it can actually be a feature that PY_NEW doesn't call
__init__(). I happily misuse that in lxml.etree to assign different
behaviour to a user class instantiation and an internal proxy creation.


> batch PY_NEW total: 0.821 , interval only: 0.363
> batch __init__ on Cython class total 0.975 , interval_only: 0.524

As expected, the difference is a lot larger here. Ignoring the setup
overhead, that's some 30% faster, as PY_NEW does the field initialisation
inside a plain C call, whereas the call to __init__ requires the Python
calling convention (i.e. tuple packing and unpacking).


> __init__ on Python class: 28.468
> __init__ on Python class with slots: 9.936

Ok, that's a completely different performance league. The first benchmark
requires setting up a dictionary for the instance and putting the
attributes there, whereas the second happily assigns the attributes to
predefined slots, without the overhead of a dict.

To sum it up, using PY_NEW from the beginning looks like a premature
optimisation to me. If you notice that class creation becomes a bottleneck
later on, *and* your classes are instantiated internally in the relevant
cases, *and* the setup involves working with plain C field values (such as
the 'start' and 'end' ints in your case), switching to PY_NEW can give you
about 30% in plain class instantiation time (YMWV, don't forget to
benchmark your own code).

Thanks for sharing,

Stefan

From robertwb at math.washington.edu  Thu Apr 16 10:36:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Apr 2009 01:36:42 -0700
Subject: [Cython] Cython 0.11.1 released
Message-ID: <36C7FFE0-FBE8-4FC5-ADDA-AFDF8ACDA034@math.washington.edu>

Cython 0.11.1 is released and can be downloaded from http:// 
cython.org/. The major changes in this release are better conversion  
between c integer and Python types, better error reporting, and more  
correct (and consistent) for-from and for-in loops. More python  
constructions are supported (from package import module and integer  
powering) and several bugs were fixed. CEP 516 (http:// 
wiki.cython.org/enhancements/division) was also implemented,  
providing Python semantics for division, but is off by default until  
0.12. See

http://trac.cython.org/cython_trac/query? 
group=component&milestone=0.11.1

for a list of all tickets closed.

- Robert


From ndbecker2 at gmail.com  Thu Apr 16 13:49:48 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 16 Apr 2009 07:49:48 -0400
Subject: [Cython] Cython 0.11.1 released
References: <36C7FFE0-FBE8-4FC5-ADDA-AFDF8ACDA034@math.washington.edu>
Message-ID: <gs760s$43o$2@ger.gmane.org>

Where can I find release notes?  (Need to point to it for the fedora update)



From stefan_ml at behnel.de  Thu Apr 16 14:46:09 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Apr 2009 14:46:09 +0200 (CEST)
Subject: [Cython] Cython 0.11.1 released
In-Reply-To: <gs760s$43o$2@ger.gmane.org>
References: <36C7FFE0-FBE8-4FC5-ADDA-AFDF8ACDA034@math.washington.edu>
	<gs760s$43o$2@ger.gmane.org>
Message-ID: <58cffd5f7d955299f62535b329d9ad77.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Neal Becker wrote:
> Where can I find release notes?  (Need to point to it for the fedora
> update)

You can just point to the trac query page that Robert sent a link for.

Here is another query that works for me:

http://trac.cython.org/cython_trac/query?status=closed&order=type&col=id&col=summary&col=type&col=owner&milestone=0.11.1

You can also link to the corresponding RSS feed if you prefer.

Stefan


From bpederse at gmail.com  Thu Apr 16 17:04:19 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Thu, 16 Apr 2009 08:04:19 -0700
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <49E6BC89.7020400@behnel.de>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>
	<1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>
	<e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com>
	<49E626A8.4000608@behnel.de>
	<e183a99d0904151931l7b3e5997id566dbb511b39cda@mail.gmail.com>
	<49E6BC89.7020400@behnel.de>
Message-ID: <e183a99d0904160804i75517678p92ea2ceb1750e47f@mail.gmail.com>

On Wed, Apr 15, 2009 at 10:05 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> now, those are interesting timings. Which versions of Cython and Python did
> you use?

hi,
i am using python 2.5 with cython .10 and .11 -- which both give same ballpark.
thanks for the explanation.
-brent

>
> Brent Pedersen wrote:
>> here are the new timings:
>>
>> PY_NEW on Cython class: 1.137
>> __init__ on Cython class 1.154
>
> You're right, that's almost negligible. However, the __init__ benchmark
> involves the overhead of a Python call to create_interval, which uses the
> same Python calling convention as the constructor. So it's more or less
> doing the same thing in both cases, just additionally calling __init__
> through a C struct member dereference in the second case (but without
> repacking the arguments).
>
> Also note that it can actually be a feature that PY_NEW doesn't call
> __init__(). I happily misuse that in lxml.etree to assign different
> behaviour to a user class instantiation and an internal proxy creation.
>
>
>> batch PY_NEW total: 0.821 , interval only: 0.363
>> batch __init__ on Cython class total 0.975 , interval_only: 0.524
>
> As expected, the difference is a lot larger here. Ignoring the setup
> overhead, that's some 30% faster, as PY_NEW does the field initialisation
> inside a plain C call, whereas the call to __init__ requires the Python
> calling convention (i.e. tuple packing and unpacking).
>
>
>> __init__ on Python class: 28.468
>> __init__ on Python class with slots: 9.936
>
> Ok, that's a completely different performance league. The first benchmark
> requires setting up a dictionary for the instance and putting the
> attributes there, whereas the second happily assigns the attributes to
> predefined slots, without the overhead of a dict.
>
> To sum it up, using PY_NEW from the beginning looks like a premature
> optimisation to me. If you notice that class creation becomes a bottleneck
> later on, *and* your classes are instantiated internally in the relevant
> cases, *and* the setup involves working with plain C field values (such as
> the 'start' and 'end' ints in your case), switching to PY_NEW can give you
> about 30% in plain class instantiation time (YMWV, don't forget to
> benchmark your own code).
>
> Thanks for sharing,
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dalcinl at gmail.com  Thu Apr 16 17:27:13 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Apr 2009 12:27:13 -0300
Subject: [Cython] Why the tarball release includes '.hg' directory?
Message-ID: <e7ba66e40904160827t5ebcac9ated6ab21916294fb5@mail.gmail.com>

Do we really need this?

$ tar -zxf Cython-0.11.1.tar.gz
$ du -sh Cython-0.11.1
11M	Cython-0.11.1

$ du -sh Cython-0.11.1/.hg
6.2M	Cython-0.11.1/.hg

$ rm -rf Cython-0.11.1/.hg
$ du -sh Cython-0.11.1
4.5M	Cython-0.11.1

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Apr 16 17:38:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Apr 2009 12:38:21 -0300
Subject: [Cython] moving some tests to the same place
Message-ID: <e7ba66e40904160838r30fb85a6j653bd7c63071bf9@mail.gmail.com>

See this...

$ find Cython -name 'Test*'
Cython/TestUtils.py
Cython/Compiler/Tests
Cython/Compiler/Tests/TestParseTreeTransforms.py
Cython/Compiler/Tests/TestTreeFragment.py
Cython/Compiler/Tests/TestBuffer.py
Cython/Compiler/Tests/TestDecorators.py
Cython/Tests
Cython/Tests/TestCodeWriter.py

What do you think about moving ALL that inside "Cython/Tests"...

BTW, Why setup.py lists  'Cython.Tests' and 'Cython.Compiler.Tests' as
packages? Do a final user-targeted installation really need these?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From hoytak at cs.ubc.ca  Thu Apr 16 19:24:34 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Thu, 16 Apr 2009 10:24:34 -0700
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <4db580fd0904161022w5d1d4fd1g72377f97ef806df4@mail.gmail.com>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com> 
	<1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu> 
	<e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com> 
	<49E626A8.4000608@behnel.de>
	<e183a99d0904151931l7b3e5997id566dbb511b39cda@mail.gmail.com> 
	<49E6BC89.7020400@behnel.de>
	<e183a99d0904160804i75517678p92ea2ceb1750e47f@mail.gmail.com> 
	<4db580fd0904161022w5d1d4fd1g72377f97ef806df4@mail.gmail.com>
Message-ID: <4db580fd0904161024l30136b0cxba4457a3919212e2@mail.gmail.com>

Thanks everyone, this discussion is quite informative.  My classes all
fall in the potential-30% class, and I've already written it in, so I
think I'll keep things in.

-- Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From dagss at student.matnat.uio.no  Thu Apr 16 20:32:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Apr 2009 20:32:09 +0200
Subject: [Cython] moving some tests to the same place
In-Reply-To: <e7ba66e40904160838r30fb85a6j653bd7c63071bf9@mail.gmail.com>
References: <e7ba66e40904160838r30fb85a6j653bd7c63071bf9@mail.gmail.com>
Message-ID: <49E779A9.5070105@student.matnat.uio.no>

Lisandro Dalcin wrote:
> See this...
> 
> $ find Cython -name 'Test*'
> Cython/TestUtils.py

This is a collection of utilities for writing tests, and is so not in 
the same category as the rest...

> Cython/Compiler/Tests
> Cython/Compiler/Tests/TestParseTreeTransforms.py
> Cython/Compiler/Tests/TestTreeFragment.py
> Cython/Compiler/Tests/TestBuffer.py
> Cython/Compiler/Tests/TestDecorators.py
> Cython/Tests
> Cython/Tests/TestCodeWriter.py
> 
> What do you think about moving ALL that inside "Cython/Tests"...

Why?

These are unit tests (and NOT integration tests, like tests/*). I think 
it is very uncommon to remove tests that far from the package they are 
testing. The idea is that the Cython.Compiler package has its unit tests 
in Cython.Compiler.Tests, and the Cython package has its unit tests in 
Cython.Tests.

(The reason it is very uncommon to do what you suggest is that they test 
1:1 with the file they refer to. What do you do with your scheme if both 
Cython/Foo.py and Cython/Compiler/Foo.py exists?)

If anything, I think all the test files should be moved alongside the 
other files; but since unit tests are so little in use that would be a 
bit intrusive..

Another common convention is to put the unit tests in the same script 
files as they are testing (perhaps as doctests), but that is


> BTW, Why setup.py lists  'Cython.Tests' and 'Cython.Compiler.Tests' as
> packages? Do a final user-targeted installation really need these?

I don't mind either way, but it's rather common for a Python package to 
package its test (e.g. with NumPy there's a method you can call to run 
the test suite). But in that case, tests/* would be much more important 
anyway, so removing these seems more consistent.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Apr 16 20:38:20 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Apr 2009 20:38:20 +0200
Subject: [Cython] Why the tarball release includes '.hg' directory?
In-Reply-To: <e7ba66e40904160827t5ebcac9ated6ab21916294fb5@mail.gmail.com>
References: <e7ba66e40904160827t5ebcac9ated6ab21916294fb5@mail.gmail.com>
Message-ID: <49E77B1C.2090009@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Do we really need this?
> 
> $ tar -zxf Cython-0.11.1.tar.gz
> $ du -sh Cython-0.11.1
> 11M	Cython-0.11.1
> 
> $ du -sh Cython-0.11.1/.hg
> 6.2M	Cython-0.11.1/.hg
> 
> $ rm -rf Cython-0.11.1/.hg
> $ du -sh Cython-0.11.1
> 4.5M	Cython-0.11.1
> 

Well, 11M \approx 4.5M \approx 0M :-)

An argument in favor is that end-users can very easily contribute a 
bugfix. If users suddenly encounter a bug they can fix then the fix they 
do is very likely to happen in the downloaded release directory, and 
having done that then it's a lot less effort to submit a patch if it's a 
working repository.

Especially with Cython, whose end-users are all developers, this is a 
nice idea.

I don't really mind though. But I have a feeling that the time of making 
a reliable fix for doing releases without .hg, or even writing this 
email, far surpass the download time of those few extra MBs...

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Apr 16 20:51:49 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Apr 2009 11:51:49 -0700
Subject: [Cython] Why the tarball release includes '.hg' directory?
In-Reply-To: <49E77B1C.2090009@student.matnat.uio.no>
References: <e7ba66e40904160827t5ebcac9ated6ab21916294fb5@mail.gmail.com>
	<49E77B1C.2090009@student.matnat.uio.no>
Message-ID: <013DBDD2-DFB9-4961-B21C-93625D2A42EA@math.washington.edu>

On Apr 16, 2009, at 11:38 AM, Dag Sverre Seljebotn wrote:

> Lisandro Dalcin wrote:
>> Do we really need this?
>>
>> $ tar -zxf Cython-0.11.1.tar.gz
>> $ du -sh Cython-0.11.1
>> 11M	Cython-0.11.1
>>
>> $ du -sh Cython-0.11.1/.hg
>> 6.2M	Cython-0.11.1/.hg
>>
>> $ rm -rf Cython-0.11.1/.hg
>> $ du -sh Cython-0.11.1
>> 4.5M	Cython-0.11.1
>>
>
> Well, 11M \approx 4.5M \approx 0M :-)
>
> An argument in favor is that end-users can very easily contribute a
> bugfix. If users suddenly encounter a bug they can fix then the fix  
> they
> do is very likely to happen in the downloaded release directory, and
> having done that then it's a lot less effort to submit a patch if  
> it's a
> working repository.
>
> Especially with Cython, whose end-users are all developers, this is a
> nice idea.

This is exactly why we do it. Sage ships this way and I think it's  
been a significant factor in attracting developers.

> I don't really mind though. But I have a feeling that the time of  
> making
> a reliable fix for doing releases without .hg, or even writing this
> email, far surpass the download time of those few extra MBs...

This is a good point too.

- Robert


From robertwb at math.washington.edu  Thu Apr 16 21:30:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Apr 2009 12:30:50 -0700
Subject: [Cython] test/bugs dirctory
Message-ID: <6307A74B-D2C1-46F2-9339-E3697BCDE602@math.washington.edu>

I like being able to create testfiles for specific tickets and easily  
test them, but having to move tickets back and forth between bugs and  
error/compile/run is cumbersome. Is anyone opposed to putting the  
tests directly in the appropriate test directories, and then having a  
"outstanding bugs" list in runtests.py that it uses to skip over the  
Txxx ones that are still unresolved?

- Robert


From dagss at student.matnat.uio.no  Thu Apr 16 22:02:36 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Apr 2009 22:02:36 +0200
Subject: [Cython] test/bugs dirctory
In-Reply-To: <6307A74B-D2C1-46F2-9339-E3697BCDE602@math.washington.edu>
References: <6307A74B-D2C1-46F2-9339-E3697BCDE602@math.washington.edu>
Message-ID: <49E78EDC.8030108@student.matnat.uio.no>

Robert Bradshaw wrote:
> I like being able to create testfiles for specific tickets and easily  
> test them, but having to move tickets back and forth between bugs and  
> error/compile/run is cumbersome. Is anyone opposed to putting the  
> tests directly in the appropriate test directories, and then having a  
> "outstanding bugs" list in runtests.py that it uses to skip over the  
> Txxx ones that are still unresolved?

+1, though I'd like that to be a seperate unsolved.txt rather than part 
of runtests.py (prettier commit history).

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Apr 16 22:07:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Apr 2009 13:07:03 -0700
Subject: [Cython] test/bugs dirctory
In-Reply-To: <49E78EDC.8030108@student.matnat.uio.no>
References: <6307A74B-D2C1-46F2-9339-E3697BCDE602@math.washington.edu>
	<49E78EDC.8030108@student.matnat.uio.no>
Message-ID: <A5E2F57E-6514-4796-9FF1-ADBC2C5D9A8A@math.washington.edu>

On Apr 16, 2009, at 1:02 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> I like being able to create testfiles for specific tickets and easily
>> test them, but having to move tickets back and forth between bugs and
>> error/compile/run is cumbersome. Is anyone opposed to putting the
>> tests directly in the appropriate test directories, and then having a
>> "outstanding bugs" list in runtests.py that it uses to skip over the
>> Txxx ones that are still unresolved?
>
> +1, though I'd like that to be a seperate unsolved.txt rather than  
> part
> of runtests.py (prettier commit history).

Sure.

- Robert


From dagss at student.matnat.uio.no  Thu Apr 16 22:28:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Apr 2009 22:28:55 +0200
Subject: [Cython] test/bugs dirctory
In-Reply-To: <49E78EDC.8030108@student.matnat.uio.no>
References: <6307A74B-D2C1-46F2-9339-E3697BCDE602@math.washington.edu>
	<49E78EDC.8030108@student.matnat.uio.no>
Message-ID: <49E79507.3090602@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> I like being able to create testfiles for specific tickets and easily  
>> test them, but having to move tickets back and forth between bugs and  
>> error/compile/run is cumbersome. Is anyone opposed to putting the  
>> tests directly in the appropriate test directories, and then having a  
>> "outstanding bugs" list in runtests.py that it uses to skip over the  
>> Txxx ones that are still unresolved?
> 
> +1, though I'd like that to be a seperate unsolved.txt rather than part 
> of runtests.py (prettier commit history).
> 

I'd now prefer something slighly more ambituous though: Mark each 
testcase as known to fail in the testcase itself (e.g. as a comment on 
the first line).




Going on an utopic rant:

If I really had my wish without anyone spending time to implement it 
:-), I would actually prefer to structure all testcases in a free-form 
hierarchy, and instead have a header which

  a) indicate whether it is run/compile/error
  b) which ticket(s) it is related to
  c) module requirements
  d) some kind of tag system for the feature which is tested -- tags 
could be e.g. "IndexNode", "temps"


-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Apr 16 22:22:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Apr 2009 13:22:38 -0700
Subject: [Cython] test/bugs dirctory
In-Reply-To: <49E79507.3090602@student.matnat.uio.no>
References: <6307A74B-D2C1-46F2-9339-E3697BCDE602@math.washington.edu>
	<49E78EDC.8030108@student.matnat.uio.no>
	<49E79507.3090602@student.matnat.uio.no>
Message-ID: <E1004CEC-4D9B-4557-B0DA-087CAFE0958A@math.washington.edu>

On Apr 16, 2009, at 1:28 PM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> I like being able to create testfiles for specific tickets and  
>>> easily
>>> test them, but having to move tickets back and forth between bugs  
>>> and
>>> error/compile/run is cumbersome. Is anyone opposed to putting the
>>> tests directly in the appropriate test directories, and then  
>>> having a
>>> "outstanding bugs" list in runtests.py that it uses to skip over the
>>> Txxx ones that are still unresolved?
>>
>> +1, though I'd like that to be a seperate unsolved.txt rather than  
>> part
>> of runtests.py (prettier commit history).
>>
>
> I'd now prefer something slighly more ambituous though: Mark each
> testcase as known to fail in the testcase itself (e.g. as a comment on
> the first line).

I think it's nice to have this kind of data external, as it makes it  
much easier to quickly at a glance see what's "still bad" and not  
have untested files fall through the cracks.

> Going on an utopic rant:
>
> If I really had my wish without anyone spending time to implement it
> :-), I would actually prefer to structure all testcases in a free-form
> hierarchy, and instead have a header which
>
>   a) indicate whether it is run/compile/error
>   b) which ticket(s) it is related to
>   c) module requirements
>   d) some kind of tag system for the feature which is tested -- tags
> could be e.g. "IndexNode", "temps"

Maybe someday, but one think I like about the current system is that  
it's insanely simplicity.

- Robert


From m.gysel at gmail.com  Thu Apr 16 23:15:46 2009
From: m.gysel at gmail.com (Martin Gysel)
Date: Thu, 16 Apr 2009 23:15:46 +0200
Subject: [Cython] subclassing builtin
Message-ID: <49E7A002.50002@gmail.com>

Hi

I wanna subclass the builtin dict. I read through the doc but
unfortunately I haven't got it...
I think I need to do something like:

cdef extern from "dictobject.h":
    ctypedef class __builtin__.dict [object PyDictObject]

but then what else, how do I define it after declaration?

thx
martin

From dalcinl at gmail.com  Thu Apr 16 23:43:58 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Apr 2009 18:43:58 -0300
Subject: [Cython] subclassing builtin
In-Reply-To: <49E7A002.50002@gmail.com>
References: <49E7A002.50002@gmail.com>
Message-ID: <e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>

On Thu, Apr 16, 2009 at 6:15 PM, Martin Gysel <m.gysel at gmail.com> wrote:
> Hi
>
> I wanna subclass the builtin dict. I read through the doc but
> unfortunately I haven't got it...
> I think I need to do something like:
>
> cdef extern from "dictobject.h":
> ? ?ctypedef class __builtin__.dict [object PyDictObject]
>

I woud just use "Python.h" ... and add ": pass" at the end (perhaps
add a newline after ":")


> but then what else, how do I define it after declaration?

cdef class MyDict(dict):
   # implement me !!


>
> thx
> martin
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Apr 16 23:49:29 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Apr 2009 18:49:29 -0300
Subject: [Cython] make __pyx_modulename have static storage
Message-ID: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>

Just noticed this line (reported by Intel compiler)

const char *__pyx_modulename = "mpi4py.MPI";

Any good reason to not make it have "static" storage?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From kwmsmith at gmail.com  Fri Apr 17 05:28:34 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 16 Apr 2009 22:28:34 -0500
Subject: [Cython] cdef list variable raises exception on SetItem
Message-ID: <c266a7930904162028q23e5156do975e1bf8ff17f596@mail.gmail.com>

Hi,

The first function below "in_range" works fine (the last line
generates a call to PyObject_SetItem) but the "out_of_range" function
does not.  The last line in it generates a call to PyList_SetItem as
it should, but the 'ob' PyObject is not converted to a Py_ssize_t.

[590]$ cat _out_of_range.pyx
def in_range():
    lst = range(11)
    ob = 10
    lst[ob] = -10

def out_of_range():
    cdef list lst = range(11)
    ob = 10
    lst[ob] = -10


Compiling raises a warning, as it should:

[591]$ python setup.py build_ext --inplace
running build_ext
cythoning _out_of_range.pyx to _out_of_range.c
building '_out_of_range' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c _out_of_range.c
-o build/temp.linux-x86_64-2.5/_out_of_range.o
_out_of_range.c: In function ?__pyx_pf_13_out_of_range_out_of_range?:
_out_of_range.c:429: warning: passing argument 2 of ?PyList_SetItem?
makes integer from pointer without a cast
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
build/temp.linux-x86_64-2.5/_out_of_range.o -o _out_of_range.so

And calling it raises the exception:

[592]$ python out_of_range.py
after in_range
Traceback (most recent call last):
  File "out_of_range.py", line 5, in <module>
    _out_of_range.out_of_range()
  File "_out_of_range.pyx", line 9, in _out_of_range.out_of_range
(_out_of_range.c:429)
    lst[ob] = -10
IndexError: list assignment index out of range

[593]$ cat out_of_range.py
import _out_of_range

_out_of_range.in_range()
print "after in_range"
_out_of_range.out_of_range()
print "after out_of_range"

From kwmsmith at gmail.com  Fri Apr 17 06:54:29 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 16 Apr 2009 23:54:29 -0500
Subject: [Cython] cdef list variable raises exception on SetItem
In-Reply-To: <c266a7930904162028q23e5156do975e1bf8ff17f596@mail.gmail.com>
References: <c266a7930904162028q23e5156do975e1bf8ff17f596@mail.gmail.com>
Message-ID: <c266a7930904162154q7da19f8bi1e0bec1a853d86ec@mail.gmail.com>

On Thu, Apr 16, 2009 at 10:28 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
> Hi,
>
> The first function below "in_range" works fine (the last line
> generates a call to PyObject_SetItem) but the "out_of_range" function
> does not. ?The last line in it generates a call to PyList_SetItem as
> it should, but the 'ob' PyObject is not converted to a Py_ssize_t.
>

See

http://trac.cython.org/cython_trac/ticket/284

From kwmsmith at gmail.com  Fri Apr 17 07:21:06 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 17 Apr 2009 00:21:06 -0500
Subject: [Cython] cdef list variable raises exception on SetItem
In-Reply-To: <c266a7930904162154q7da19f8bi1e0bec1a853d86ec@mail.gmail.com>
References: <c266a7930904162028q23e5156do975e1bf8ff17f596@mail.gmail.com>
	<c266a7930904162154q7da19f8bi1e0bec1a853d86ec@mail.gmail.com>
Message-ID: <c266a7930904162221g4bfbabd3g499965591476278a@mail.gmail.com>

On Thu, Apr 16, 2009 at 11:54 PM, Kurt Smith <kwmsmith at gmail.com> wrote:


>
> See
>
> http://trac.cython.org/cython_trac/ticket/284
>

Here's a testcase.

diff -r 6fc636aebe32 -r 1097ad309db6 tests/run/cdef_setitem_T284.pyx
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/tests/run/cdef_setitem_T284.pyx   Fri Apr 17 00:15:48 2009 -0500
@@ -0,0 +1,17 @@
+__doc__ = u'''
+>>> no_cdef()
+>>> with_cdef()
+'''
+def no_cdef():
+    lst = range(11)
+    ob = 10L
+    lst[ob] = -10
+    dd = {}
+    dd[ob] = -10
+
+def with_cdef():
+    cdef list lst = range(11)
+    ob = 10L
+    lst[ob] = -10
+    cdef dict dd = {}
+    dd[ob] = -10

And here's a trivial fix -- it seems hacky and if there's a better way
to do it please say so...

diff -r 6fc636aebe32 -r 1097ad309db6 Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py      Thu Apr 16 19:10:51 2009 -0700
+++ b/Cython/Compiler/ExprNodes.py      Fri Apr 17 00:15:48 2009 -0500
@@ -1858,6 +1858,7 @@ class IndexNode(ExprNode):
                 function = "PyDict_SetItem"
             elif self.base.type is list_type:
                 function = "PyList_SetItem"
+                index_code = "PyInt_AsSsize_t(%s)" % index_code
             # don't use PyTuple_SetItem(), as we'd normally get a
             # TypeError when changing a tuple, while PyTuple_SetItem()
             # would allow updates

From stefan_ml at behnel.de  Fri Apr 17 07:39:06 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Apr 2009 07:39:06 +0200
Subject: [Cython] make __pyx_modulename have static storage
In-Reply-To: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>
References: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>
Message-ID: <49E815FA.1030504@behnel.de>


Lisandro Dalcin wrote:
> Just noticed this line (reported by Intel compiler)
> 
> const char *__pyx_modulename = "mpi4py.MPI";
> 
> Any good reason to not make it have "static" storage?

No, it shouldn't be used anywhere else anyway.

Stefan

From robertwb at math.washington.edu  Fri Apr 17 07:58:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Apr 2009 22:58:59 -0700
Subject: [Cython] cdef list variable raises exception on SetItem
In-Reply-To: <c266a7930904162221g4bfbabd3g499965591476278a@mail.gmail.com>
References: <c266a7930904162028q23e5156do975e1bf8ff17f596@mail.gmail.com>
	<c266a7930904162154q7da19f8bi1e0bec1a853d86ec@mail.gmail.com>
	<c266a7930904162221g4bfbabd3g499965591476278a@mail.gmail.com>
Message-ID: <72AA5609-31A3-454C-8916-E33AA5CB3093@math.washington.edu>

On Apr 16, 2009, at 10:21 PM, Kurt Smith wrote:

> On Thu, Apr 16, 2009 at 11:54 PM, Kurt Smith <kwmsmith at gmail.com>  
> wrote:
>
>
>>
>> See
>>
>> http://trac.cython.org/cython_trac/ticket/284
>>
>
> Here's a testcase.
>
> diff -r 6fc636aebe32 -r 1097ad309db6 tests/run/cdef_setitem_T284.pyx
> --- /dev/null   Thu Jan 01 00:00:00 1970 +0000
> +++ b/tests/run/cdef_setitem_T284.pyx   Fri Apr 17 00:15:48 2009 -0500
> @@ -0,0 +1,17 @@
> +__doc__ = u'''
> +>>> no_cdef()
> +>>> with_cdef()
> +'''
> +def no_cdef():
> +    lst = range(11)
> +    ob = 10L
> +    lst[ob] = -10
> +    dd = {}
> +    dd[ob] = -10
> +
> +def with_cdef():
> +    cdef list lst = range(11)
> +    ob = 10L
> +    lst[ob] = -10
> +    cdef dict dd = {}
> +    dd[ob] = -10
>

Thanks.

> And here's a trivial fix -- it seems hacky and if there's a better way
> to do it please say so...
>
> diff -r 6fc636aebe32 -r 1097ad309db6 Cython/Compiler/ExprNodes.py
> --- a/Cython/Compiler/ExprNodes.py      Thu Apr 16 19:10:51 2009 -0700
> +++ b/Cython/Compiler/ExprNodes.py      Fri Apr 17 00:15:48 2009 -0500
> @@ -1858,6 +1858,7 @@ class IndexNode(ExprNode):
>                  function = "PyDict_SetItem"
>              elif self.base.type is list_type:
>                  function = "PyList_SetItem"
> +                index_code = "PyInt_AsSsize_t(%s)" % index_code
>              # don't use PyTuple_SetItem(), as we'd normally get a
>              # TypeError when changing a tuple, while  
> PyTuple_SetItem()
>              # would allow updates

This will actually raise the wrong kind of error for non-int objects-- 
perhaps we should just remove the entire line? Optimized setting/ 
getting still happens with int indices.

- Robert


From riccardo.murri at gmail.com  Fri Apr 17 12:54:30 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Fri, 17 Apr 2009 12:54:30 +0200
Subject: [Cython] cannot compile derived class? (cython 0.11.1)
Message-ID: <1c34ba170904170354j576ba8a8i58e1ed246d3ecc9b@mail.gmail.com>

Hello,

I cannot get Cython 0.11.1 to compile the following simple example::

  $ cat a.pxd
  cdef class A(object)
  cdef class B(A)

  $ cat a.pyx
  cdef class A(object):
       pass

  cdef class B(A):
       pass

  $ cat c.pyx
  from a cimport B

  cdef class C(B):
       pass

File "a.pyx" compiles correctly, but Cython chockes on "c.pxd"::

  $ cython -v a.pyx c.pyx
  Compiling /tmp/a.pyx
  Compiling /tmp/c.pyx
  Traceback (most recent call last):
    File "/home/rmurri/sw/Cython-0.11.1/bin/cython", line 8, in <module>
      main(command_line = 1)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/Main.py", line
734, in main
      result = compile(sources, options)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/Main.py", line
711, in compile
      return compile_multiple(source, options)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/Main.py", line
681, in compile_multiple
      result = run_pipeline(source, options)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/Main.py", line
543, in run_pipeline
      err, enddata = context.run_pipeline(pipeline, source)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/Main.py", line
198, in run_pipeline
      data = phase(data)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/Main.py", line
139, in generate_pyx_code
      module_node.process_implementation(options, result)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/ModuleNode.py",
line 71, in process_implementation
      self.generate_c_code(env, options, result)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/ModuleNode.py",
line 269, in generate_c_code
      self.generate_typeobj_definitions(env, code)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/ModuleNode.py",
line 825, in generate_typeobj_definitions
      self.generate_new_function(scope, code)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/ModuleNode.py",
line 885, in generate_new_function
      tp_new = TypeSlots.get_base_slot_function(scope, tp_slot)
    File "/home/rmurri/sw/Cython-0.11.1/Cython/Compiler/TypeSlots.py",
line 429, in get_base_slot_function
      if scope.parent_scope is base_type.scope.parent_scope:
  AttributeError: 'NoneType' object has no attribute 'parent_scope'

  $ cython --version
  Cython version 0.11.1

Same error if I change "class A(object)" to "class A".

Is this a bug or am I doing something wrong?

Best regards,
Riccardo

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From riccardo.murri at gmail.com  Fri Apr 17 14:50:47 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Fri, 17 Apr 2009 14:50:47 +0200
Subject: [Cython] incorrect C code generated on method assignment (cython
	0.11.1)
Message-ID: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>

Hello,

while converting a class from pure Python to Cython, I've come across
the following:

  $ cat r.pyx
  cdef class r(object):
       cdef long long n
       cdef long long d

       def __init__(self, long long n, long long d):
           self.n = n
           self.d = d

       def __int__(self):
           return (self.n // self.d)

       __long__ = __int__

  $ cython -v r.pyx
  Compiling /tmp/r.pyx

  $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c r.c -o r.o
  r.c:487: error: ?None? undeclared here (not in a function)

  $ sed -e '484,490!d;488i^^^^^^^^^^^^^^^^^^^^^^^^ up here' r.c
    0, /*nb_coerce*/
    #endif
    __pyx_pf_1r_1r___int__, /*nb_int*/
    None, /*nb_long*/
  ^^^^^^^^^^^^^^^^^^^^^^^^ up here
    0, /*nb_float*/
    #if PY_MAJOR_VERSION < 3
    0, /*nb_oct*/

I understand that "__long__ = __int__" is not the correct way to do
method aliasing in Cython, but the compiler should probably emit a
warning -- generating incorrect C code looks like a bug.

Best regards,
Riccardo

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From stefan_ml at behnel.de  Fri Apr 17 15:09:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Apr 2009 15:09:02 +0200 (CEST)
Subject: [Cython] incorrect C code generated on method assignment
 (cython 0.11.1)
In-Reply-To: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
Message-ID: <09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Hi,

thanks for the report.

Riccardo Murri wrote:
> while converting a class from pure Python to Cython, I've come across
> the following:
>
>   $ cat r.pyx
>   cdef class r(object):
>        cdef long long n
>        cdef long long d
>
>        def __init__(self, long long n, long long d):
>            self.n = n
>            self.d = d
>
>        def __int__(self):
>            return (self.n // self.d)
>
>        __long__ = __int__
>
>   $ cython -v r.pyx
>   Compiling /tmp/r.pyx
>
>   $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c r.c -o r.o
>   r.c:487: error: ???None??? undeclared here (not in a function)
>
>   $ sed -e '484,490!d;488i^^^^^^^^^^^^^^^^^^^^^^^^ up here' r.c
>     0, /*nb_coerce*/
>     #endif
>     __pyx_pf_1r_1r___int__, /*nb_int*/
>     None, /*nb_long*/
>   ^^^^^^^^^^^^^^^^^^^^^^^^ up here
>     0, /*nb_float*/
>     #if PY_MAJOR_VERSION < 3
>     0, /*nb_oct*/

Yes, that's a bug. I assume that the above code does not make Cython set
up the C level function name.

Raising an error here would work for now.


> I understand that "__long__ = __int__" is not the correct way to do
> method aliasing in Cython

... and that's a feature request, it would be nice to make that work.
Could you create two tickets for those?

http://trac.cython.org/cython_trac/

If you don't have a trac account yet, please send a htpasswd password line
to Robert Bradshaw.

Thanks!

Stefan


From dalcinl at gmail.com  Fri Apr 17 15:33:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 10:33:15 -0300
Subject: [Cython] make __pyx_modulename have static storage
In-Reply-To: <49E815FA.1030504@behnel.de>
References: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>
	<49E815FA.1030504@behnel.de>
Message-ID: <e7ba66e40904170633g2fa55eb3rd6392ac2725282ba@mail.gmail.com>

Sorry, I did not understad you comment, perhaps I was not clear
enough... This is a global var, not one loca to function, then I'm
asking to emit the C code like this:

static const char *__pyx_modulename = "mpi4py.MPI";

See this:

$ grep __pyx_modulename src/mpi4py.MPI.c
const char *__pyx_modulename = "mpi4py.MPI";

$ nm ~/lib/python/mpi4py/MPI.so | grep __pyx_modulename
000c0fa0 D __pyx_modulename

What's the use case for that symbol to be dynamic??

All this was introduced by Robert when implementing the new C division stuff

changeset:   1916:413a75b92fc0
user:        Robert Bradshaw <robertwb at math.washington.edu>
date:        Sat Mar 28 02:22:07 2009 -0700
summary:     Use warnings framework for c division.


On Fri, Apr 17, 2009 at 2:39 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>> Just noticed this line (reported by Intel compiler)
>>
>> const char *__pyx_modulename = "mpi4py.MPI";
>>
>> Any good reason to not make it have "static" storage?
>
> No, it shouldn't be used anywhere else anyway.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From riccardo.murri at gmail.com  Fri Apr 17 15:51:42 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Fri, 17 Apr 2009 15:51:42 +0200
Subject: [Cython] subclassing builtin
In-Reply-To: <e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>
References: <49E7A002.50002@gmail.com>
	<e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>
Message-ID: <1c34ba170904170651i3a2cf1cbkb9f1b4ea6906de14@mail.gmail.com>

Hi,

I cannot get the documented method to work::

  $ cat t.pyx
  cdef extern from "tupleobject.h":
      ctypedef class __builtin__.tuple [object PyTupleObject]:
          pass

  cdef class MyTuple(tuple):
      pass

  $ cython t.pyx
  warning: /tmp/t.pyx:2:4: tuple already a builtin Cython type
  $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c t.c -o t.o
  $ gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions t.o -o t.so

  $ python
  Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49)
  [GCC 4.3.2] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import t
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "t.pyx", line 2, in t (t.c:610)
      ctypedef class __builtin__.tuple [object PyTupleObject]:
  ValueError: __builtin__.tuple does not appear to be the correct type object

Relevant lines in t.c are::

  $ sed -e '547,551!d' t.c
    /*--- Function export code ---*/
    /*--- Type init code ---*/
    __pyx_ptype_1t_tuple = __Pyx_ImportType(__Pyx_BUILTIN_MODULE_NAME,
"tuple", sizeof(PyTupleObject)); if (unlikely(!__pyx_ptype_1t_tuple))
{__pyx_filename = __pyx_f[0]; __pyx_lineno = 2; __pyx_clineno =
__LINE__; goto __pyx_L1_error;}
    __pyx_type_1t_MyTuple.tp_base = __pyx_ptype_1t_tuple;
    if (PyType_Ready(&__pyx_type_1t_MyTuple) < 0) {__pyx_filename =
__pyx_f[0]; __pyx_lineno = 5; __pyx_clineno = __LINE__; goto
__pyx_L1_error;}

I'm using Cython 0.11.1; thanks for any help!

Best regards,
Riccardo

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From dalcinl at gmail.com  Fri Apr 17 16:01:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 11:01:09 -0300
Subject: [Cython] incorrect C code generated on method assignment
	(cython 0.11.1)
In-Reply-To: <09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>

On Fri, Apr 17, 2009 at 10:09 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> ... and that's a feature request, it would be nice to make that work.
> Could you create two tickets for those?
>

Mmm... we will have to have care here... In Py3k, nb_long was renamed
to nb_reserved (moreover, now it is a bare void* pointer). So, In
Py3k, if a class only defines one of __int__() or __long__(), Cython
should generate code filling the 'nb_int' slot, and put a NULL in
'nb_reserved'. What I do not know is what to do if a class defines
both special methods...

Also note that the in implementation of __Pyx_PyNumber_Int() (from my
patch for C-int conversion), "nb_long" takes precedence over "nb_int"
in Python 2. IMHO, that was the best thing to do...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Apr 17 16:13:04 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 11:13:04 -0300
Subject: [Cython] subclassing builtin
In-Reply-To: <1c34ba170904170651i3a2cf1cbkb9f1b4ea6906de14@mail.gmail.com>
References: <49E7A002.50002@gmail.com>
	<e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>
	<1c34ba170904170651i3a2cf1cbkb9f1b4ea6906de14@mail.gmail.com>
Message-ID: <e7ba66e40904170713i3572d04dv8dfc47691b210bda@mail.gmail.com>

On Fri, Apr 17, 2009 at 10:51 AM, Riccardo Murri
<riccardo.murri at gmail.com> wrote:
> Hi,
>
> I cannot get the documented method to work::
>
> ?$ cat t.pyx
> ?cdef extern from "tupleobject.h":
> ? ? ?ctypedef class __builtin__.tuple [object PyTupleObject]:
> ? ? ? ? ?pass
>
> ?cdef class MyTuple(tuple):
> ? ? ?pass
>

Tuples are a different beast... They are PyObject_VAR_HEAD structures
(like strings in Py2, and perhaps in Py3 in the future, no right
now)...

typedef struct {
    PyObject_VAR_HEAD
    PyObject *ob_item[1];
} PyTupleObject;

Cython cannot currently handle this...

Perhaps the error message could be made more explicit in this case (by
checking if tp_itemsize != 0 ) ... ?


> ?$ cython t.pyx
> ?warning: /tmp/t.pyx:2:4: tuple already a builtin Cython type
> ?$ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c t.c -o t.o
> ?$ gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions t.o -o t.so
>
> ?$ python
> ?Python 2.5.2 (r252:60911, Oct ?5 2008, 19:24:49)
> ?[GCC 4.3.2] on linux2
> ?Type "help", "copyright", "credits" or "license" for more information.
> ?>>> import t
> ?Traceback (most recent call last):
> ? ?File "<stdin>", line 1, in <module>
> ? ?File "t.pyx", line 2, in t (t.c:610)
> ? ? ?ctypedef class __builtin__.tuple [object PyTupleObject]:
> ?ValueError: __builtin__.tuple does not appear to be the correct type object
>
> Relevant lines in t.c are::
>
> ?$ sed -e '547,551!d' t.c
> ? ?/*--- Function export code ---*/
> ? ?/*--- Type init code ---*/
> ? ?__pyx_ptype_1t_tuple = __Pyx_ImportType(__Pyx_BUILTIN_MODULE_NAME,
> "tuple", sizeof(PyTupleObject)); if (unlikely(!__pyx_ptype_1t_tuple))
> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 2; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}
> ? ?__pyx_type_1t_MyTuple.tp_base = __pyx_ptype_1t_tuple;
> ? ?if (PyType_Ready(&__pyx_type_1t_MyTuple) < 0) {__pyx_filename =
> __pyx_f[0]; __pyx_lineno = 5; __pyx_clineno = __LINE__; goto
> __pyx_L1_error;}
>
> I'm using Cython 0.11.1; thanks for any help!
>
> Best regards,
> Riccardo
>
> --
> Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From riccardo.murri at gmail.com  Fri Apr 17 16:32:13 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Fri, 17 Apr 2009 16:32:13 +0200
Subject: [Cython] subclassing builtin
In-Reply-To: <e7ba66e40904170713i3572d04dv8dfc47691b210bda@mail.gmail.com>
References: <49E7A002.50002@gmail.com>
	<e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>
	<1c34ba170904170651i3a2cf1cbkb9f1b4ea6906de14@mail.gmail.com>
	<e7ba66e40904170713i3572d04dv8dfc47691b210bda@mail.gmail.com>
Message-ID: <1c34ba170904170732k245bd6c3q3d619204aa914607@mail.gmail.com>

On Fri, Apr 17, 2009 at 4:13 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On Fri, Apr 17, 2009 at 10:51 AM, Riccardo Murri <riccardo.murri at gmail.com> wrote:
>> I cannot get the documented method to work::
>>
>> ?$ cat t.pyx
>> ?cdef extern from "tupleobject.h":
>> ? ? ?ctypedef class __builtin__.tuple [object PyTupleObject]:
>> ? ? ? ? ?pass
>>
>> ?cdef class MyTuple(tuple):
>> ? ? ?pass
>>
>
> Tuples are a different beast... They are PyObject_VAR_HEAD structures
> (like strings in Py2, and perhaps in Py3 in the future, no right
> now)...
>
> typedef struct {
> ? ?PyObject_VAR_HEAD
> ? ?PyObject *ob_item[1];
> } PyTupleObject;
>
> Cython cannot currently handle this...
>

Thank you very much!  I have updated the Wiki page, stating that not
only strings but also tuples cannot be subclassed:

http://wiki.cython.org/FAQ/cdef_derive#preview

Riccardo

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From stefan_ml at behnel.de  Fri Apr 17 16:39:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Apr 2009 16:39:04 +0200 (CEST)
Subject: [Cython] subclassing builtin
In-Reply-To: <e7ba66e40904170713i3572d04dv8dfc47691b210bda@mail.gmail.com>
References: <49E7A002.50002@gmail.com>
	<e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>
	<1c34ba170904170651i3a2cf1cbkb9f1b4ea6906de14@mail.gmail.com>
	<e7ba66e40904170713i3572d04dv8dfc47691b210bda@mail.gmail.com>
Message-ID: <867910dd5a852ea84f8c11706783366b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> On Fri, Apr 17, 2009 at 10:51 AM, Riccardo Murri wrote:
>> I cannot get the documented method to work::
>>
>> ?$ cat t.pyx
>> ?cdef extern from "tupleobject.h":
>> ? ? ?ctypedef class __builtin__.tuple [object PyTupleObject]:
>> ? ? ? ? ?pass
>>
>> ?cdef class MyTuple(tuple):
>> ? ? ?pass
>>
>
> Tuples are a different beast... They are PyObject_VAR_HEAD structures

Yep, that's a missing feature. It's not trivial to implement as this
requires a special case in the way Cython deals with subtypes, likely with
a major code impact.

http://trac.cython.org/cython_trac/ticket/152

It's usually best to use a normal Python class to subtype str (Py2.x) and
tuple, instead of a cdef class.

Stefan


From stefan_ml at behnel.de  Fri Apr 17 16:49:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Apr 2009 16:49:47 +0200 (CEST)
Subject: [Cython] make __pyx_modulename have static storage
In-Reply-To: <e7ba66e40904170633g2fa55eb3rd6392ac2725282ba@mail.gmail.com>
References: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>
	<49E815FA.1030504@behnel.de>
	<e7ba66e40904170633g2fa55eb3rd6392ac2725282ba@mail.gmail.com>
Message-ID: <df4cd3fdb1630b6f453aa61b0dac1992.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> Sorry, I did not understad you comment, perhaps I was not clear
> enough... This is a global var, not one loca to function, then I'm
> asking to emit the C code like this:
>
> static const char *__pyx_modulename = "mpi4py.MPI";

Yes, that's what I meant. It's not supposed to be used anywhere outside of
the module, so "static" is correct here. I assume that Robert added the
constant because utility functions are not currently module specific.

Stefan


From dalcinl at gmail.com  Fri Apr 17 16:53:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 11:53:27 -0300
Subject: [Cython] make __pyx_modulename have static storage
In-Reply-To: <df4cd3fdb1630b6f453aa61b0dac1992.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>
	<49E815FA.1030504@behnel.de>
	<e7ba66e40904170633g2fa55eb3rd6392ac2725282ba@mail.gmail.com>
	<df4cd3fdb1630b6f453aa61b0dac1992.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40904170753r5d54eb44pd1ad1d57ad7d9abf@mail.gmail.com>

On Fri, Apr 17, 2009 at 11:49 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin wrote:
>>
>> static const char *__pyx_modulename = "mpi4py.MPI";
>
> Yes, that's what I meant.

OK, then I'll push a fix in cython-devel.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Apr 17 17:06:39 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Apr 2009 17:06:39 +0200 (CEST)
Subject: [Cython] incorrect C code generated on method assignment
 (cython 0.11.1)
In-Reply-To: <e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>
Message-ID: <63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> On Fri, Apr 17, 2009 at 10:09 AM, Stefan Behnel wrote:
>> ... and that's a feature request, it would be nice to make that work.
>> Could you create two tickets for those?

What I meant was: add support for setting special methods by assignment,
at least in some cases. I didn't specifically mean "__int__" and
"__long__".


> Mmm... we will have to have care here... In Py3k, nb_long was renamed
> to nb_reserved (moreover, now it is a bare void* pointer).

Yes, they removed all sorts of stuff (and I lobbied for removing all sorts
of dead meat), but they forgot to remove that one in time. It's still in
3.1a1.


> So, In
> Py3k, if a class only defines one of __int__() or __long__(), Cython
> should generate code filling the 'nb_int' slot,

Sure.


> and put a NULL in 'nb_reserved'.

Always in Py3.


> What I do not know is what to do
> if a class defines both special methods...

Good question. I think if __long__ is defined, that's what best resembles
nb_int in Py3. People are more likely to expect C-int compatibility in
__int__, so the only reason why one would define __long__ is to pass
larger numbers.

If both are defined, I'd vote for using __long__ in Py3.


> Also note that the in implementation of __Pyx_PyNumber_Int() (from my
> patch for C-int conversion), "nb_long" takes precedence over "nb_int"
> in Python 2. IMHO, that was the best thing to do...

What's CPython doing here?

Stefan


From dalcinl at gmail.com  Fri Apr 17 18:04:58 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 13:04:58 -0300
Subject: [Cython] incorrect C code generated on method assignment
	(cython 0.11.1)
In-Reply-To: <63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>
	<63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40904170904w54c3fe86j29dc2f08cac7195a@mail.gmail.com>

On Fri, Apr 17, 2009 at 12:06 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin wrote:
>> On Fri, Apr 17, 2009 at 10:09 AM, Stefan Behnel wrote:
>>> ... and that's a feature request, it would be nice to make that work.
>>> Could you create two tickets for those?
>
> What I meant was: add support for setting special methods by assignment,
> at least in some cases. I didn't specifically mean "__int__" and
> "__long__".
>
>
>> Mmm... we will have to have care here... In Py3k, nb_long was renamed
>> to nb_reserved (moreover, now it is a bare void* pointer).
>
> Yes, they removed all sorts of stuff (and I lobbied for removing all sorts
> of dead meat), but they forgot to remove that one in time. It's still in
> 3.1a1.
>
>
>> So, In
>> Py3k, if a class only defines one of __int__() or __long__(), Cython
>> should generate code filling the 'nb_int' slot,
>
> Sure.
>
>
>> and put a NULL in 'nb_reserved'.
>
> Always in Py3.
>
>
>> What I do not know is what to do
>> if a class defines both special methods...
>
> Good question. I think if __long__ is defined, that's what best resembles
> nb_int in Py3. People are more likely to expect C-int compatibility in
> __int__, so the only reason why one would define __long__ is to pass
> larger numbers.
>
> If both are defined, I'd vote for using __long__ in Py3.
>

I agree in all above points with you.

>
>> Also note that the in implementation of __Pyx_PyNumber_Int() (from my
>> patch for C-int conversion), "nb_long" takes precedence over "nb_int"
>> in Python 2. IMHO, that was the best thing to do...
>
> What's CPython doing here?
>

CPython is a real mess here... At the price of manually inspecting the
existence of nb_int/nb_long slots, __Pyx_PyNumber_Int(ob) is my best
attempt for making all work as expected for Cython. I've worked hard
to make to avoid the PyInt/PyLong distinction. So if you do "cdef
short j = obj" it will work for PyInt/PyLong in all supported Python
runtimes.

The only possible issue is that if a integral-like type defines both
nb_int/nb_long and they return Python integrals with different values
(in the == sense), then the return of "nb_long" will be used and
"nb_int" will not be ever called. However, what would be the use case
for such implementations of __int__() and __long__() ???. However,
this work on Python.

In [1]: class A(object):
   ...:     def __int__(self): return 2
   ...:     def __long__(self): return 3
   ...:
   ...:

In [2]:

In [3]: int(A())
Out[3]: 2

In [4]: long(A())
Out[4]: 3





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From kwmsmith at gmail.com  Fri Apr 17 18:40:13 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 17 Apr 2009 11:40:13 -0500
Subject: [Cython] cdef list variable raises exception on SetItem
In-Reply-To: <72AA5609-31A3-454C-8916-E33AA5CB3093@math.washington.edu>
References: <c266a7930904162028q23e5156do975e1bf8ff17f596@mail.gmail.com>
	<c266a7930904162154q7da19f8bi1e0bec1a853d86ec@mail.gmail.com>
	<c266a7930904162221g4bfbabd3g499965591476278a@mail.gmail.com>
	<72AA5609-31A3-454C-8916-E33AA5CB3093@math.washington.edu>
Message-ID: <c266a7930904170940h2c1e549ye23d8b44d74e50b9@mail.gmail.com>

On Fri, Apr 17, 2009 at 12:58 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:

>
> This will actually raise the wrong kind of error for non-int objects--
> perhaps we should just remove the entire line? Optimized setting/
> getting still happens with int indices.

Here's a patch that comments out the lines and adds a comment.

diff -r fc73225aaea1 Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py	Fri Apr 17 09:11:16 2009 +0200
+++ b/Cython/Compiler/ExprNodes.py	Fri Apr 17 11:37:44 2009 -0500
@@ -1856,8 +1856,12 @@
             index_code = self.index.py_result()
             if self.base.type is dict_type:
                 function = "PyDict_SetItem"
-            elif self.base.type is list_type:
-                function = "PyList_SetItem"
+            # PyList_SetItem() requires the index to be converted to a
+            # Py_ssize_t, so we fall through to PyObject_SetItem instead.
+            #
+            #elif self.base.type is list_type:
+            #    function = "PyList_SetItem"
+
             # don't use PyTuple_SetItem(), as we'd normally get a
             # TypeError when changing a tuple, while PyTuple_SetItem()
             # would allow updates

Kurt

From stefan_ml at behnel.de  Fri Apr 17 18:52:48 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Apr 2009 18:52:48 +0200
Subject: [Cython] incorrect C code generated on method
 assignment	(cython 0.11.1)
In-Reply-To: <e7ba66e40904170904w54c3fe86j29dc2f08cac7195a@mail.gmail.com>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>	<63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40904170904w54c3fe86j29dc2f08cac7195a@mail.gmail.com>
Message-ID: <49E8B3E0.2050403@behnel.de>


Lisandro Dalcin wrote:
> On Fri, Apr 17, 2009 at 12:06 PM, Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>> What I do not know is what to do
>>> if a class defines both special methods...
>> Good question. I think if __long__ is defined, that's what best resembles
>> nb_int in Py3. People are more likely to expect C-int compatibility in
>> __int__, so the only reason why one would define __long__ is to pass
>> larger numbers.

BTW, in Py3, PyNumber_Int() was replaced in favour of PyNumber_Long(). That
makes it a bit funny, the C layer uses "long", whereas the Python layer
uses "int".


>>> Also note that the in implementation of __Pyx_PyNumber_Int() (from my
>>> patch for C-int conversion), "nb_long" takes precedence over "nb_int"
>>> in Python 2. IMHO, that was the best thing to do...
>> What's CPython doing here?
> 
> CPython is a real mess here... At the price of manually inspecting the
> existence of nb_int/nb_long slots, __Pyx_PyNumber_Int(ob) is my best
> attempt for making all work as expected for Cython. I've worked hard
> to make to avoid the PyInt/PyLong distinction. So if you do "cdef
> short j = obj" it will work for PyInt/PyLong in all supported Python
> runtimes.
> 
> The only possible issue is that if a integral-like type defines both
> nb_int/nb_long and they return Python integrals with different values
> (in the == sense), then the return of "nb_long" will be used and
> "nb_int" will not be ever called. However, what would be the use case
> for such implementations of __int__() and __long__() ???. However,
> this work on Python.

I don't see a major use case either, but calling nb_int when we convert to
a C-int or a smaller type might still make sense in Py2. The function might
actually do some work (as opposed to returning a constant value). I'd
suspect an __int__() function to be potentially more optimised by users
than something that already knows that it has to return a PyLong.

Note that Py2 allows __int__() to return a PyLong if the value doesn't fit
into a PyInt, while __long__() must return a PyLong:

http://docs.python.org/c-api/number.html#PyNumber_Int

"""
PyObject* PyNumber_Int(PyObject *o)
    Return value: New reference.

    Returns the o converted to an integer object on success, or NULL on
failure. If the argument is outside the integer range a long object will be
returned instead. This is the equivalent of the Python expression int(o).

PyObject* PyNumber_Long(PyObject *o)
    Return value: New reference.

    Returns the o converted to a long integer object on success, or NULL on
failure. This is the equivalent of the Python expression long(o).
"""

Stefan

From dalcinl at gmail.com  Fri Apr 17 19:19:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 14:19:28 -0300
Subject: [Cython] incorrect C code generated on method assignment
	(cython 0.11.1)
In-Reply-To: <49E8B3E0.2050403@behnel.de>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>
	<63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40904170904w54c3fe86j29dc2f08cac7195a@mail.gmail.com>
	<49E8B3E0.2050403@behnel.de>
Message-ID: <e7ba66e40904171019u6e3b48f7tc8e8d3535a9d9d87@mail.gmail.com>

On Fri, Apr 17, 2009 at 1:52 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> BTW, in Py3, PyNumber_Int() was replaced in favour of PyNumber_Long(). That
> makes it a bit funny, the C layer uses "long", whereas the Python layer
> uses "int".
>

Of course... They should have to remove nb_int, and mantain nb_long...
I remember there were some recent discussions about this
inconsistencies on Python-Dev...

>
> I don't see a major use case either, but calling nb_int when we convert to
> a C-int or a smaller type might still make sense in Py2.

Well, my implementation first check if nb_long exists, if not, it
checks nb_int. So all this work just fine in Py2.

> Note that Py2 allows __int__() to return a PyLong if the value doesn't fit
> into a PyInt, while __long__() must return a PyLong:
>

Then I would vote for getting rid of "nb_long" at all, and always fill
"nb_int" with these rule:

1) if "__int__()" is implemented, always use it and ignore
"__long__()" if present, then fill the "nb_int" slot in Py2/Py3, and
set "nb_long"/"nb_reserved" to NULL.

2) if only "__long__()" is implemented, then fill the "nb_int" slot in
Py2/Py3, and set "nb_long"/"nb_reserved" to NULL.

BTW, filling only the "nb_long" slot has issues, see for example the
implementation "PyNumber_Check()" in Python 2. A  cdef class only
implementing "__long__()" will likely fail that check, right?


What do you think?


> http://docs.python.org/c-api/number.html#PyNumber_Int
>
> """
> PyObject* PyNumber_Int(PyObject *o)
> ? ?Return value: New reference.
>
> ? ?Returns the o converted to an integer object on success, or NULL on
> failure. If the argument is outside the integer range a long object will be
> returned instead. This is the equivalent of the Python expression int(o).
>
> PyObject* PyNumber_Long(PyObject *o)
> ? ?Return value: New reference.
>
> ? ?Returns the o converted to a long integer object on success, or NULL on
> failure. This is the equivalent of the Python expression long(o).
> """
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Apr 17 19:56:14 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 17 Apr 2009 10:56:14 -0700
Subject: [Cython] make __pyx_modulename have static storage
In-Reply-To: <df4cd3fdb1630b6f453aa61b0dac1992.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>
	<49E815FA.1030504@behnel.de>
	<e7ba66e40904170633g2fa55eb3rd6392ac2725282ba@mail.gmail.com>
	<df4cd3fdb1630b6f453aa61b0dac1992.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <665B9775-8C01-4EBA-9A08-31021C3E8259@math.washington.edu>

On Apr 17, 2009, at 7:49 AM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> Sorry, I did not understad you comment, perhaps I was not clear
>> enough... This is a global var, not one loca to function, then I'm
>> asking to emit the C code like this:
>>
>> static const char *__pyx_modulename = "mpi4py.MPI";
>
> Yes, that's what I meant. It's not supposed to be used anywhere  
> outside of
> the module, so "static" is correct here. I assume that Robert added  
> the
> constant because utility functions are not currently module specific.

Yep, IIRC, I needed to provide the module name in the warning. The  
above should be totally fine.

- Robert



From robertwb at math.washington.edu  Fri Apr 17 20:01:00 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 17 Apr 2009 11:01:00 -0700
Subject: [Cython] cannot compile derived class? (cython 0.11.1)
In-Reply-To: <1c34ba170904170354j576ba8a8i58e1ed246d3ecc9b@mail.gmail.com>
References: <1c34ba170904170354j576ba8a8i58e1ed246d3ecc9b@mail.gmail.com>
Message-ID: <991E4AA8-4F7C-42DB-B070-F9ECC1B5A867@math.washington.edu>

On Apr 17, 2009, at 3:54 AM, Riccardo Murri wrote:

> Hello,
>
> I cannot get Cython 0.11.1 to compile the following simple example::
>
>   $ cat a.pxd
>   cdef class A(object)
>   cdef class B(A)
>
>   $ cat a.pyx
>   cdef class A(object):
>        pass
>
>   cdef class B(A):
>        pass
>
>   $ cat c.pyx
>   from a cimport B
>
>   cdef class C(B):
>        pass

[...]

> Is this a bug or am I doing something wrong?

No, I think this is a bug. However, does it work if you write

   cdef class A(object):
        pass

   cdef class B(A):
        pass

in your .pxd?

- Robert


From dalcinl at gmail.com  Fri Apr 17 21:04:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 16:04:28 -0300
Subject: [Cython] make __pyx_modulename have static storage
In-Reply-To: <665B9775-8C01-4EBA-9A08-31021C3E8259@math.washington.edu>
References: <e7ba66e40904161449x1dcc7d41t20e3dedb823a10b4@mail.gmail.com>
	<49E815FA.1030504@behnel.de>
	<e7ba66e40904170633g2fa55eb3rd6392ac2725282ba@mail.gmail.com>
	<df4cd3fdb1630b6f453aa61b0dac1992.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<665B9775-8C01-4EBA-9A08-31021C3E8259@math.washington.edu>
Message-ID: <e7ba66e40904171204x797720fdtd8aa712ca740be31@mail.gmail.com>

Well, if we add "static" we start getting unused GCC warns .. So I've
decided to change this to a #define'd macro... Just in case, I've not
removed the definition in Naming.py.

On Fri, Apr 17, 2009 at 2:56 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 17, 2009, at 7:49 AM, Stefan Behnel wrote:
>
>> Lisandro Dalcin wrote:
>>> Sorry, I did not understad you comment, perhaps I was not clear
>>> enough... This is a global var, not one loca to function, then I'm
>>> asking to emit the C code like this:
>>>
>>> static const char *__pyx_modulename = "mpi4py.MPI";
>>
>> Yes, that's what I meant. It's not supposed to be used anywhere
>> outside of
>> the module, so "static" is correct here. I assume that Robert added
>> the
>> constant because utility functions are not currently module specific.
>
> Yep, IIRC, I needed to provide the module name in the warning. The
> above should be totally fine.
>
> - Robert
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Apr 17 21:14:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Apr 2009 21:14:07 +0200
Subject: [Cython] incorrect C code generated on method assignment
 (cython 0.11.1)
In-Reply-To: <e7ba66e40904171019u6e3b48f7tc8e8d3535a9d9d87@mail.gmail.com>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>	<63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<e7ba66e40904170904w54c3fe86j29dc2f08cac7195a@mail.gmail.com>	<49E8B3E0.2050403@behnel.de>
	<e7ba66e40904171019u6e3b48f7tc8e8d3535a9d9d87@mail.gmail.com>
Message-ID: <49E8D4FF.1050003@behnel.de>


Lisandro Dalcin wrote:
> On Fri, Apr 17, 2009 at 1:52 PM, Stefan Behnel wrote:
>> I don't see a major use case either, but calling nb_int when we convert to
>> a C-int or a smaller type might still make sense in Py2.
> 
> Well, my implementation first check if nb_long exists, if not, it
> checks nb_int. So all this work just fine in Py2.

What I meant was: if we convert to a type that fits into a C int, we should
prefer nb_int over nb_long, not the other way round.


>> Note that Py2 allows __int__() to return a PyLong if the value doesn't fit
>> into a PyInt, while __long__() must return a PyLong:
> 
> Then I would vote for getting rid of "nb_long" at all, and always fill
> "nb_int" with these rule:
> 
> 1) if "__int__()" is implemented, always use it and ignore
> "__long__()" if present, then fill the "nb_int" slot in Py2/Py3, and
> set "nb_long"/"nb_reserved" to NULL.
> 
> 2) if only "__long__()" is implemented, then fill the "nb_int" slot in
> Py2/Py3, and set "nb_long"/"nb_reserved" to NULL.

I think you say this because it follows the Py3 rules. I'm fine with that,
but I do think that we should use both methods in Py2 if defined. So 1)
should only apply to Py3.

Given that there is no obvious use case for __long__, I also think that
defining it should emit a deprecation warning in general, not only in case 2)

Stefan


From riccardo.murri at gmail.com  Fri Apr 17 21:17:18 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Fri, 17 Apr 2009 21:17:18 +0200
Subject: [Cython] cannot compile derived class? (cython 0.11.1)
In-Reply-To: <991E4AA8-4F7C-42DB-B070-F9ECC1B5A867@math.washington.edu>
References: <1c34ba170904170354j576ba8a8i58e1ed246d3ecc9b@mail.gmail.com>
	<991E4AA8-4F7C-42DB-B070-F9ECC1B5A867@math.washington.edu>
Message-ID: <1c34ba170904171217m1f6b193ckc5065667c70d409b@mail.gmail.com>

On Fri, Apr 17, 2009 at 8:01 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> No, I think this is a bug. However, does it work if you write
>
> ? cdef class A(object):
> ? ? ? ?pass
>
> ? cdef class B(A):
> ? ? ? ?pass
>
> in your .pxd?
>

Yes, it does.

Riccardo

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From dalcinl at gmail.com  Fri Apr 17 22:36:03 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 17:36:03 -0300
Subject: [Cython] incorrect C code generated on method assignment
	(cython 0.11.1)
In-Reply-To: <49E8D4FF.1050003@behnel.de>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>
	<63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40904170904w54c3fe86j29dc2f08cac7195a@mail.gmail.com>
	<49E8B3E0.2050403@behnel.de>
	<e7ba66e40904171019u6e3b48f7tc8e8d3535a9d9d87@mail.gmail.com>
	<49E8D4FF.1050003@behnel.de>
Message-ID: <e7ba66e40904171336q5cb44517jf3ba8c57f06b5378@mail.gmail.com>

On Fri, Apr 17, 2009 at 4:14 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>> On Fri, Apr 17, 2009 at 1:52 PM, Stefan Behnel wrote:
>>> I don't see a major use case either, but calling nb_int when we convert to
>>> a C-int or a smaller type might still make sense in Py2.
>>
>> Well, my implementation first check if nb_long exists, if not, it
>> checks nb_int. So all this work just fine in Py2.
>
> What I meant was: if we convert to a type that fits into a C int, we should
> prefer nb_int over nb_long, not the other way round.
>

Well, to do it the right way, nb_int it should take precedence if the
type fits in a C int, or if it fits in a "signed long" but not
"unsigned long". It seems that I was lazy, though I've actually tried
to maintain the implementation of _Pyx_PyNumber_Int() simple and make
it take a single argument (like the one in core CPython). I'll try to
review my work and provide a patch, though I'm not sure if further
work will pay that much... For core CPython types, the only case where
all this can have some impact in for float -> int conversion.

>
>>> Note that Py2 allows __int__() to return a PyLong if the value doesn't fit
>>> into a PyInt, while __long__() must return a PyLong:
>>
>> Then I would vote for getting rid of "nb_long" at all, and always fill
>> "nb_int" with these rule:
>>
>> 1) if "__int__()" is implemented, always use it and ignore
>> "__long__()" if present, then fill the "nb_int" slot in Py2/Py3, and
>> set "nb_long"/"nb_reserved" to NULL.
>>
>> 2) if only "__long__()" is implemented, then fill the "nb_int" slot in
>> Py2/Py3, and set "nb_long"/"nb_reserved" to NULL.
>
> I think you say this because it follows the Py3 rules. I'm fine with that,
> but I do think that we should use both methods in Py2 if defined. So 1)
> should only apply to Py3.
>

As I've started to work on a patch, let put it clear case by case...

Python 2
----------
a) if only __int__(), fill as "nb_int = __int__" and "nb_long = NULL"
b) if only __long__(), fill as "nb_int = __long__" and "nb_long = NULL"
c) if __int__() AND __long__(): fill as "nb_int = __int__" and
"nb_long = __long__"

Python 3
----------
a) if only __int__(), fill as "nb_int = __int__" and "nb_reserved = NULL"
b) if only __long__(), fill as "nb_int = __long__" and "nb_reserved = NULL"
c) if __int__() AND __long__(): fill as "nb_int = __int__" and
"nb_reserved = NULL"

So only cases (c) makes a difference for Py2/Py3...

>
> Given that there is no obvious use case for __long__, I also think that
> defining it should emit a deprecation warning in general, not only in case 2)
>

I'm fine with that, but I'll likely ask you to do the work :-)



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From kwmsmith at gmail.com  Fri Apr 17 22:45:00 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 17 Apr 2009 15:45:00 -0500
Subject: [Cython] endianness bug fix in numpy_test.pyx
Message-ID: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>

The current numpy_test.pyx file fails for PowerPC macs due to
endianness issues in the dtype.  This is a small fix to make it work
(and make all tests pass on my machine).  It also adds an explicit
big-endian test to the doctest.

diff -r fc73225aaea1 tests/run/numpy_test.pyx
--- a/tests/run/numpy_test.pyx	Fri Apr 17 09:11:16 2009 +0200
+++ b/tests/run/numpy_test.pyx	Fri Apr 17 15:43:11 2009 -0500
@@ -132,13 +132,20 @@
     >>> test_recordarray()

     >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
-            ('a', np.dtype('i,i')),\
-            ('b', np.dtype('i,i'))\
+            ('a', np.dtype('<i,<i')),\
+            ('b', np.dtype('<i,<i'))\
         ])))
     array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
           dtype=[('a', [('f0', '<i4'), ('f1', '<i4')]), ('b', [('f0',
'<i4'), ('f1', '<i4')])])

     >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
+            ('a', np.dtype('>i,>i')),\
+            ('b', np.dtype('>i,>i'))\
+        ])))
+    array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
+          dtype=[('a', [('f0', '>i4'), ('f1', '>i4')]), ('b', [('f0',
'>i4'), ('f1', '>i4')])])
+
+    >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
             ('a', np.dtype('i,f')),\
             ('b', np.dtype('i,i'))\
         ])))
@@ -297,6 +304,10 @@
     arr[1].b.x = arr[0].a.y + 1
     arr[1].b.y = 4
     arr[2] = arr[1]
+    if not arr.dtype.isnative:
+        # required to get the endianness
+        # indicator (< or >) correct in output.
+        arr.byteswap(True)
     return arr

 def test_bad_nested_dtypes():

From robertwb at math.washington.edu  Fri Apr 17 23:48:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 17 Apr 2009 14:48:28 -0700
Subject: [Cython] endianness bug fix in numpy_test.pyx
In-Reply-To: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
References: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
Message-ID: <1D04E878-D6F2-4279-98E3-B727916FF58D@math.washington.edu>

On Apr 17, 2009, at 1:45 PM, Kurt Smith wrote:

> The current numpy_test.pyx file fails for PowerPC macs due to
> endianness issues in the dtype.  This is a small fix to make it work
> (and make all tests pass on my machine).  It also adds an explicit
> big-endian test to the doctest.

Hmm... This looks like a bug in the code itself, not just the test.

>
> diff -r fc73225aaea1 tests/run/numpy_test.pyx
> --- a/tests/run/numpy_test.pyx	Fri Apr 17 09:11:16 2009 +0200
> +++ b/tests/run/numpy_test.pyx	Fri Apr 17 15:43:11 2009 -0500
> @@ -132,13 +132,20 @@
>>>> test_recordarray()
>
>>>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
> -            ('a', np.dtype('i,i')),\
> -            ('b', np.dtype('i,i'))\
> +            ('a', np.dtype('<i,<i')),\
> +            ('b', np.dtype('<i,<i'))\
>          ])))
>      array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
>            dtype=[('a', [('f0', '<i4'), ('f1', '<i4')]), ('b', [('f0',
> '<i4'), ('f1', '<i4')])])
>
>>>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
> +            ('a', np.dtype('>i,>i')),\
> +            ('b', np.dtype('>i,>i'))\
> +        ])))
> +    array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
> +          dtype=[('a', [('f0', '>i4'), ('f1', '>i4')]), ('b', [('f0',
> '>i4'), ('f1', '>i4')])])
> +
> +    >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
>              ('a', np.dtype('i,f')),\
>              ('b', np.dtype('i,i'))\
>          ])))
> @@ -297,6 +304,10 @@
>      arr[1].b.x = arr[0].a.y + 1
>      arr[1].b.y = 4
>      arr[2] = arr[1]
> +    if not arr.dtype.isnative:
> +        # required to get the endianness
> +        # indicator (< or >) correct in output.
> +        arr.byteswap(True)
>      return arr
>
>  def test_bad_nested_dtypes():
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Sat Apr 18 00:12:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Apr 2009 19:12:19 -0300
Subject: [Cython] setting cdef members of cdef classes in nogil functions
Message-ID: <e7ba66e40904171512w5769dd28ha3d3aed52c862ed2@mail.gmail.com>

I'm working on providing support in mpi4py for complex types, and then
I did more or les this:

cdef class Datatype: # cdef class
    MPI_Datatype ob_mpi

cdef Datatype COMPLEX = Datatype() # cdef global

cdef void setup_complex_types() nogil: # nogil function
   cdef MPI_Datatype newtype
   create_complex_type(&newtype) # implemented elsewhere
   COMPLEX.ob_mpi = newtype # Look HERE  !

As you see, I'm touching a C field of the object structure inside a
"nogil" function.

 I'm very hapy it worked!! . AFAIK, there are no GIL issues in this
code. Anyway, I would like to know your opinion on this.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sat Apr 18 02:59:01 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 17 Apr 2009 17:59:01 -0700
Subject: [Cython] setting cdef members of cdef classes in nogil functions
In-Reply-To: <e7ba66e40904171512w5769dd28ha3d3aed52c862ed2@mail.gmail.com>
References: <e7ba66e40904171512w5769dd28ha3d3aed52c862ed2@mail.gmail.com>
Message-ID: <A1498618-5FE3-41F5-B7E2-B1E8EAE9AC13@math.washington.edu>

On Apr 17, 2009, at 3:12 PM, Lisandro Dalcin wrote:

> I'm working on providing support in mpi4py for complex types, and then
> I did more or les this:
>
> cdef class Datatype: # cdef class
>     MPI_Datatype ob_mpi
>
> cdef Datatype COMPLEX = Datatype() # cdef global
>
> cdef void setup_complex_types() nogil: # nogil function
>    cdef MPI_Datatype newtype
>    create_complex_type(&newtype) # implemented elsewhere
>    COMPLEX.ob_mpi = newtype # Look HERE  !
>
> As you see, I'm touching a C field of the object structure inside a
> "nogil" function.
>
>  I'm very hapy it worked!! . AFAIK, there are no GIL issues in this
> code. Anyway, I would like to know your opinion on this.

Yes, there's no reason at all that this should need the GIL.

- Robert



From cournape at gmail.com  Sat Apr 18 07:30:42 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 18 Apr 2009 14:30:42 +0900
Subject: [Cython] endianness bug fix in numpy_test.pyx
In-Reply-To: <1D04E878-D6F2-4279-98E3-B727916FF58D@math.washington.edu>
References: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
	<1D04E878-D6F2-4279-98E3-B727916FF58D@math.washington.edu>
Message-ID: <5b8d13220904172230h44c1a86fg41dd4493366a3047@mail.gmail.com>

On Sat, Apr 18, 2009 at 6:48 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 17, 2009, at 1:45 PM, Kurt Smith wrote:
>
>> The current numpy_test.pyx file fails for PowerPC macs due to
>> endianness issues in the dtype. ?This is a small fix to make it work
>> (and make all tests pass on my machine). ?It also adds an explicit
>> big-endian test to the doctest.
>
> Hmm... This looks like a bug in the code itself, not just the test.

This may be related to a python 2.6 bug on ppc:

http://bugs.python.org/issue4728

There is a workaround in numpy 1.3 to bypass the problem altogether -
but python 2.6 + ppc + numpy < 1.3 will not work,

David

From stefan_ml at behnel.de  Sat Apr 18 08:26:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 18 Apr 2009 08:26:11 +0200
Subject: [Cython] incorrect C code generated on method assignment
 (cython 0.11.1)
In-Reply-To: <e7ba66e40904171336q5cb44517jf3ba8c57f06b5378@mail.gmail.com>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<e7ba66e40904170701h4c6af104m7f6d0ab9328c83cf@mail.gmail.com>	<63f6aed263dda2ed3b6d431bad1e93c6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<e7ba66e40904170904w54c3fe86j29dc2f08cac7195a@mail.gmail.com>	<49E8B3E0.2050403@behnel.de>	<e7ba66e40904171019u6e3b48f7tc8e8d3535a9d9d87@mail.gmail.com>	<49E8D4FF.1050003@behnel.de>
	<e7ba66e40904171336q5cb44517jf3ba8c57f06b5378@mail.gmail.com>
Message-ID: <49E97283.9000505@behnel.de>


Lisandro Dalcin wrote:
> Well, to do it the right way, nb_int it should take precedence if the
> type fits in a C int, or if it fits in a "signed long" but not
> "unsigned long". It seems that I was lazy, though I've actually tried
> to maintain the implementation of _Pyx_PyNumber_Int() simple and make
> it take a single argument (like the one in core CPython). I'll try to
> review my work and provide a patch

Thanks!


> though I'm not sure if further
> work will pay that much... For core CPython types, the only case where
> all this can have some impact in for float -> int conversion.

... plus, all of this will go away in Py3 anyway. So, don't invest too much
time, but if you can get it right, I think it would make things more robust
and consistent.


> As I've started to work on a patch, let put it clear case by case...
> 
> Python 2
> ----------
> a) if only __int__(), fill as "nb_int = __int__" and "nb_long = NULL"
> b) if only __long__(), fill as "nb_int = __long__" and "nb_long = NULL"
> c) if __int__() AND __long__(): fill as "nb_int = __int__" and
> "nb_long = __long__"
> 
> Python 3
> ----------
> a) if only __int__(), fill as "nb_int = __int__" and "nb_reserved = NULL"
> b) if only __long__(), fill as "nb_int = __long__" and "nb_reserved = NULL"

Fine so far.


> c) if __int__() AND __long__(): fill as "nb_int = __int__" and
> "nb_reserved = NULL"

I'm still unsure whether Py3 should prefer __long__ or __int__. I guess it
would be more consistent to use __int__ for nb_int in both Py2 and Py3, but
since nb_int *must* return a PyLong in Py3, it might be cleaner (and
easier?) to use __long__ instead.

If we use __int__ in Py3, I think it would make sense to raise a warning
only for case b). That would help users to get their code right. That might
be the best solution after all: use __int__ when available, support
__long__ only in Py2, and raise a warning when someone forgets to define
__int__ next to it.

Stefan


From robertwb at math.washington.edu  Sat Apr 18 10:03:48 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 18 Apr 2009 01:03:48 -0700
Subject: [Cython] cdef list variable raises exception on SetItem
In-Reply-To: <c266a7930904162221g4bfbabd3g499965591476278a@mail.gmail.com>
References: <c266a7930904162028q23e5156do975e1bf8ff17f596@mail.gmail.com>
	<c266a7930904162154q7da19f8bi1e0bec1a853d86ec@mail.gmail.com>
	<c266a7930904162221g4bfbabd3g499965591476278a@mail.gmail.com>
Message-ID: <32A99FF0-83AB-44E6-BC64-BF7F082CDE7E@math.washington.edu>

On Apr 16, 2009, at 10:21 PM, Kurt Smith wrote:

> On Thu, Apr 16, 2009 at 11:54 PM, Kurt Smith <kwmsmith at gmail.com>  
> wrote:
>
>> See
>>
>> http://trac.cython.org/cython_trac/ticket/284

Fixed. Thanks for tracking this down.

> Here's a testcase.
>
> diff -r 6fc636aebe32 -r 1097ad309db6 tests/run/cdef_setitem_T284.pyx
> --- /dev/null   Thu Jan 01 00:00:00 1970 +0000
> +++ b/tests/run/cdef_setitem_T284.pyx   Fri Apr 17 00:15:48 2009 -0500

BTW, do you know how to output mercurial diffs?

- Robert

From dagss at student.matnat.uio.no  Sat Apr 18 10:22:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 18 Apr 2009 10:22:30 +0200
Subject: [Cython] endianness bug fix in numpy_test.pyx
In-Reply-To: <1D04E878-D6F2-4279-98E3-B727916FF58D@math.washington.edu>
References: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
	<1D04E878-D6F2-4279-98E3-B727916FF58D@math.washington.edu>
Message-ID: <49E98DC6.2040803@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 17, 2009, at 1:45 PM, Kurt Smith wrote:
> 
>> The current numpy_test.pyx file fails for PowerPC macs due to
>> endianness issues in the dtype.  This is a small fix to make it work
>> (and make all tests pass on my machine).  It also adds an explicit
>> big-endian test to the doctest.
> 
> Hmm... This looks like a bug in the code itself, not just the test.

Thanks for bringing this up Kurt.

Using typed buffers in Cython only support the native endianness, so 
what one should get is an exception on one of the platforms.

It seems that the problem is that numpy.pxd does not include endianness 
in the formatting string; in fact it looks like this wasn't considered 
at all.

http://trac.cython.org/cython_trac/ticket/285

When it comes to the testcase, one should have two tests:

a) With native endianness, which should succeed. I think the best way is 
to modify the result:

x = ...
print(repr(x).replace(">","<")) # then follows little endian result

b) With non-native endianness, which should result in an exception from 
buffer acquisition.


> 
>> diff -r fc73225aaea1 tests/run/numpy_test.pyx
>> --- a/tests/run/numpy_test.pyx	Fri Apr 17 09:11:16 2009 +0200
>> +++ b/tests/run/numpy_test.pyx	Fri Apr 17 15:43:11 2009 -0500
>> @@ -132,13 +132,20 @@
>>>>> test_recordarray()
>>>>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
>> -            ('a', np.dtype('i,i')),\
>> -            ('b', np.dtype('i,i'))\
>> +            ('a', np.dtype('<i,<i')),\
>> +            ('b', np.dtype('<i,<i'))\
>>          ])))
>>      array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
>>            dtype=[('a', [('f0', '<i4'), ('f1', '<i4')]), ('b', [('f0',
>> '<i4'), ('f1', '<i4')])])
>>
>>>>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
>> +            ('a', np.dtype('>i,>i')),\
>> +            ('b', np.dtype('>i,>i'))\
>> +        ])))
>> +    array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
>> +          dtype=[('a', [('f0', '>i4'), ('f1', '>i4')]), ('b', [('f0',
>> '>i4'), ('f1', '>i4')])])
>> +
>> +    >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
>>              ('a', np.dtype('i,f')),\
>>              ('b', np.dtype('i,i'))\
>>          ])))
>> @@ -297,6 +304,10 @@
>>      arr[1].b.x = arr[0].a.y + 1
>>      arr[1].b.y = 4
>>      arr[2] = arr[1]
>> +    if not arr.dtype.isnative:
>> +        # required to get the endianness
>> +        # indicator (< or >) correct in output.
>> +        arr.byteswap(True)
>>      return arr
>>
>>  def test_bad_nested_dtypes():
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Apr 18 10:15:29 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 18 Apr 2009 01:15:29 -0700
Subject: [Cython] Py_New still good way to create classes?
In-Reply-To: <e183a99d0904151931l7b3e5997id566dbb511b39cda@mail.gmail.com>
References: <4db580fd0904051226u64d9dfe5n73674e3d258ba677@mail.gmail.com>
	<1228653C-157E-48C6-A305-E9D2CFE27AB0@math.washington.edu>
	<e183a99d0904151039r7e33bf7aj97044f4fe4285844@mail.gmail.com>
	<49E626A8.4000608@behnel.de>
	<e183a99d0904151931l7b3e5997id566dbb511b39cda@mail.gmail.com>
Message-ID: <C270C8B3-7EE2-4FA1-A59A-6E5D1956D2F7@math.washington.edu>

On Apr 15, 2009, at 7:31 PM, Brent Pedersen wrote:

> On Wed, Apr 15, 2009 at 11:25 AM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> thanks for sharing that.
>>
>> Brent Pedersen wrote:
>>> assuming i haven't done anything stupid
>>
>> No, nothing stupid, but something that can reduce the  
>> comparability of the
>> timings. You are creating a 1000000 item list on each benchmark,  
>> using a
>> call to range() in some cases and a list comprehension in others.
>>
>> It's usually better to move initialisations out of the timings,  
>> e.g. by
>> creating a large range() object once and re-using it. That reduces  
>> the
>> impact of unrelated operations on the absolute numbers.
>>
>
> ah, i see, updated that. fixing that makes the python constructor look
> even slower.
> now it assumes that creating a list comprehension without assgning to
> a variable is
> the same as calling a function that returns an array--also without  
> assigning.
>
> here are the new timings:
>
> PY_NEW on Cython class: 1.137
> __init__ on Python class: 28.468
> __init__ on Python class with slots: 9.936
> batch PY_NEW total: 0.821 , interval only: 0.363
> batch __init__ on Cython class total 0.975 , interval_only: 0.524
> __init__ on Cython class 1.154
>
> so for this case using PY_NEW macro actual doesnt improve speed that
> much over a cdef'ed class.
> especially if using a "batch" method is applicable (as it is for my  
> use-case).

Here's some more data points, specific to Sage (Python 2.5, Cython 0.11)

-------------------

%cython

from sage.rings.integer cimport Integer

cdef class A:
     pass

def time_py_new(long N):
     cdef long i
     for i from 0 <= i < N:
         z = PY_NEW(Integer)

def time_py_init(long N):
     cdef long i
     for i from 0 <= i < N:
         z = Integer()

def time_py_new_A(long N):
     cdef long i
     for i from 0 <= i < N:
         z = PY_NEW(A)

def time_py_init_A(long N):
     cdef long i
     for i from 0 <= i < N:
         z = A()

--------------------

sage: time time_py_init(10**7)
Time: CPU 0.67 s, Wall: 0.68 s
sage: time time_py_new(10**7)
Time: CPU 0.16 s, Wall: 0.17 s
sage: time time_py_init_A(10**7)
Time: CPU 0.66 s, Wall: 0.67 s
sage: time time_py_new_A(10**7)
Time: CPU 0.47 s, Wall: 0.48 s

Note that I have an empty constructor in both cases. In summary,  
Integer has a custom tp_new slot with a pool to avoid allocation/ 
deallocation overhead, and PY_NEW here saves quite a bit. The 30%  
speed difference for a standard cdef class seems to hold about right-- 
probably two thirds of the time is probably in allocating memory from  
the heap (and also releasing it, in my test).

- Robert


From dagss at student.matnat.uio.no  Sat Apr 18 10:39:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 18 Apr 2009 10:39:51 +0200
Subject: [Cython] endianness bug fix in numpy_test.pyx
In-Reply-To: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
References: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
Message-ID: <49E991D7.5000809@student.matnat.uio.no>

Kurt Smith wrote:
> The current numpy_test.pyx file fails for PowerPC macs due to
> endianness issues in the dtype.  This is a small fix to make it work
> (and make all tests pass on my machine).  It also adds an explicit
> big-endian test to the doctest.
> 
> diff -r fc73225aaea1 tests/run/numpy_test.pyx
> --- a/tests/run/numpy_test.pyx	Fri Apr 17 09:11:16 2009 +0200
> +++ b/tests/run/numpy_test.pyx	Fri Apr 17 15:43:11 2009 -0500
> @@ -132,13 +132,20 @@
>      >>> test_recordarray()
> 
>      >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
> -            ('a', np.dtype('i,i')),\
> -            ('b', np.dtype('i,i'))\
> +            ('a', np.dtype('<i,<i')),\
> +            ('b', np.dtype('<i,<i'))\
>          ])))
>      array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
>            dtype=[('a', [('f0', '<i4'), ('f1', '<i4')]), ('b', [('f0',
> '<i4'), ('f1', '<i4')])])

Any ideas on why this particular case actually passes on your machine? 
The Cython code alwayws works with big-endian on your machine, so I 
wouldn't expect to see ((1, 2), (1, 4)) there, but the byteswapped 
versions of those numbers...

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Apr 18 14:57:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 18 Apr 2009 14:57:05 +0200
Subject: [Cython] endianness bug fix in numpy_test.pyx
In-Reply-To: <49E991D7.5000809@student.matnat.uio.no>
References: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
	<49E991D7.5000809@student.matnat.uio.no>
Message-ID: <49E9CE21.8040105@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Kurt Smith wrote:
>> The current numpy_test.pyx file fails for PowerPC macs due to
>> endianness issues in the dtype.  This is a small fix to make it work
>> (and make all tests pass on my machine).  It also adds an explicit
>> big-endian test to the doctest.
>>
>> diff -r fc73225aaea1 tests/run/numpy_test.pyx
>> --- a/tests/run/numpy_test.pyx	Fri Apr 17 09:11:16 2009 +0200
>> +++ b/tests/run/numpy_test.pyx	Fri Apr 17 15:43:11 2009 -0500
>> @@ -132,13 +132,20 @@
>>      >>> test_recordarray()
>>
>>      >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
>> -            ('a', np.dtype('i,i')),\
>> -            ('b', np.dtype('i,i'))\
>> +            ('a', np.dtype('<i,<i')),\
>> +            ('b', np.dtype('<i,<i'))\
>>          ])))
>>      array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
>>            dtype=[('a', [('f0', '<i4'), ('f1', '<i4')]), ('b', [('f0',
>> '<i4'), ('f1', '<i4')])])
> 
> Any ideas on why this particular case actually passes on your machine? 
> The Cython code alwayws works with big-endian on your machine, so I 
> wouldn't expect to see ((1, 2), (1, 4)) there, but the byteswapped 
> versions of those numbers...
> 

Ahh right; the only value read from the input array (in the test 
function) is zero, which is the same in both endians, and you added 
explicit byteswap on the output.

-- 
Dag Sverre

From m.gysel at gmail.com  Sat Apr 18 15:15:48 2009
From: m.gysel at gmail.com (Martin Gysel)
Date: Sat, 18 Apr 2009 15:15:48 +0200
Subject: [Cython] subclassing builtin
In-Reply-To: <e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>
References: <49E7A002.50002@gmail.com>
	<e7ba66e40904161443n34129729t2ef0ffce7c9c62f6@mail.gmail.com>
Message-ID: <49E9D284.10905@gmail.com>

Lisandro Dalcin schrieb:
> On Thu, Apr 16, 2009 at 6:15 PM, Martin Gysel <m.gysel at gmail.com> wrote:
>   
>> cdef extern from "dictobject.h":
>>    ctypedef class __builtin__.dict [object PyDictObject]
>>     
> I woud just use "Python.h" ... and add ": pass" at the end (perhaps
> add a newline after ":")
>   
thx for the hint with the ': pass'

cheers
martin


From kwmsmith at gmail.com  Sat Apr 18 19:11:35 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sat, 18 Apr 2009 12:11:35 -0500
Subject: [Cython] endianness bug fix in numpy_test.pyx
In-Reply-To: <49E9CE21.8040105@student.matnat.uio.no>
References: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>
	<49E991D7.5000809@student.matnat.uio.no>
	<49E9CE21.8040105@student.matnat.uio.no>
Message-ID: <c266a7930904181011t32fce0b9u7495655efbf48059@mail.gmail.com>

On Sat, Apr 18, 2009 at 7:57 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> Kurt Smith wrote:
>>> The current numpy_test.pyx file fails for PowerPC macs due to
>>> endianness issues in the dtype. ?This is a small fix to make it work
>>> (and make all tests pass on my machine). ?It also adds an explicit
>>> big-endian test to the doctest.
>>>
>>> diff -r fc73225aaea1 tests/run/numpy_test.pyx
>>> --- a/tests/run/numpy_test.pyx ? ? ? Fri Apr 17 09:11:16 2009 +0200
>>> +++ b/tests/run/numpy_test.pyx ? ? ? Fri Apr 17 15:43:11 2009 -0500
>>> @@ -132,13 +132,20 @@
>>> ? ? ?>>> test_recordarray()
>>>
>>> ? ? ?>>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
>>> - ? ? ? ? ? ?('a', np.dtype('i,i')),\
>>> - ? ? ? ? ? ?('b', np.dtype('i,i'))\
>>> + ? ? ? ? ? ?('a', np.dtype('<i,<i')),\
>>> + ? ? ? ? ? ?('b', np.dtype('<i,<i'))\
>>> ? ? ? ? ?])))
>>> ? ? ?array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
>>> ? ? ? ? ? ?dtype=[('a', [('f0', '<i4'), ('f1', '<i4')]), ('b', [('f0',
>>> '<i4'), ('f1', '<i4')])])
>>
>> Any ideas on why this particular case actually passes on your machine?
>> The Cython code alwayws works with big-endian on your machine, so I
>> wouldn't expect to see ((1, 2), (1, 4)) there, but the byteswapped
>> versions of those numbers...
>>
>
> Ahh right; the only value read from the input array (in the test
> function) is zero, which is the same in both endians, and you added
> explicit byteswap on the output.

Sorry -- can't devote the time to address these questions right now
but I'll get to them as soon as I can (Monday?).

Thanks,

Kurt

From dagss at student.matnat.uio.no  Sat Apr 18 20:36:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 18 Apr 2009 20:36:10 +0200
Subject: [Cython] endianness bug fix in numpy_test.pyx
In-Reply-To: <c266a7930904181011t32fce0b9u7495655efbf48059@mail.gmail.com>
References: <c266a7930904171345s7fe30968g1fa469dba070ee7d@mail.gmail.com>	<49E991D7.5000809@student.matnat.uio.no>	<49E9CE21.8040105@student.matnat.uio.no>
	<c266a7930904181011t32fce0b9u7495655efbf48059@mail.gmail.com>
Message-ID: <49EA1D9A.1040103@student.matnat.uio.no>

Kurt Smith wrote:
> On Sat, Apr 18, 2009 at 7:57 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Dag Sverre Seljebotn wrote:
>>> Kurt Smith wrote:
>>>> The current numpy_test.pyx file fails for PowerPC macs due to
>>>> endianness issues in the dtype.  This is a small fix to make it work
>>>> (and make all tests pass on my machine).  It also adds an explicit
>>>> big-endian test to the doctest.
>>>>
>>>> diff -r fc73225aaea1 tests/run/numpy_test.pyx
>>>> --- a/tests/run/numpy_test.pyx       Fri Apr 17 09:11:16 2009 +0200
>>>> +++ b/tests/run/numpy_test.pyx       Fri Apr 17 15:43:11 2009 -0500
>>>> @@ -132,13 +132,20 @@
>>>>      >>> test_recordarray()
>>>>
>>>>      >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\
>>>> -            ('a', np.dtype('i,i')),\
>>>> -            ('b', np.dtype('i,i'))\
>>>> +            ('a', np.dtype('<i,<i')),\
>>>> +            ('b', np.dtype('<i,<i'))\
>>>>          ])))
>>>>      array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))],
>>>>            dtype=[('a', [('f0', '<i4'), ('f1', '<i4')]), ('b', [('f0',
>>>> '<i4'), ('f1', '<i4')])])
>>> Any ideas on why this particular case actually passes on your machine?
>>> The Cython code alwayws works with big-endian on your machine, so I
>>> wouldn't expect to see ((1, 2), (1, 4)) there, but the byteswapped
>>> versions of those numbers...
>>>
>> Ahh right; the only value read from the input array (in the test
>> function) is zero, which is the same in both endians, and you added
>> explicit byteswap on the output.
> 
> Sorry -- can't devote the time to address these questions right now
> but I'll get to them as soon as I can (Monday?).

I wasn't implying that you should fix this :-) just that the problem is 
larger than the patch.

Actually struct buffer access isn't fully worked out yet (the 
rudimentary support that is there now was added some months ago and 
contains some weaknesses). I now actually think some rather involved 
fixes on "struct packing" (native vs. packed) must be done as well to 
support most common numpy dtypes; and those are very connected, so may 
do those at the same time.

If you like we can do it together. We'll see; my work on it must happen 
sporadically over the course of some weeks anyway.

-- 
Dag Sverre

From riccardo.murri at gmail.com  Sun Apr 19 09:39:01 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Sun, 19 Apr 2009 09:39:01 +0200
Subject: [Cython] cannot compile derived class? (cython 0.11.1)
In-Reply-To: <991E4AA8-4F7C-42DB-B070-F9ECC1B5A867@math.washington.edu>
References: <1c34ba170904170354j576ba8a8i58e1ed246d3ecc9b@mail.gmail.com>
	<991E4AA8-4F7C-42DB-B070-F9ECC1B5A867@math.washington.edu>
Message-ID: <1c34ba170904190039v7722b723o58e0028741c6aec4@mail.gmail.com>

On Fri, Apr 17, 2009 at 8:01 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 17, 2009, at 3:54 AM, Riccardo Murri wrote:
>
>> I cannot get Cython 0.11.1 to compile the following simple example::
>>
>> ? $ cat a.pxd
>> ? cdef class A(object)
>> ? cdef class B(A)
>>
>> ? $ cat a.pyx
>> ? cdef class A(object):
>> ? ? ? ?pass
>>
>> ? cdef class B(A):
>> ? ? ? ?pass
>>
>> ? $ cat c.pyx
>> ? from a cimport B
>>
>> ? cdef class C(B):
>> ? ? ? ?pass
>
> [...]
>
>> Is this a bug or am I doing something wrong?
>
> No, I think this is a bug.

Reported in ticket #286: http://trac.cython.org/cython_trac/ticket/286

Riccardo

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From riccardo.murri at gmail.com  Sun Apr 19 10:02:12 2009
From: riccardo.murri at gmail.com (Riccardo Murri)
Date: Sun, 19 Apr 2009 10:02:12 +0200
Subject: [Cython] incorrect C code generated on method assignment
	(cython 0.11.1)
In-Reply-To: <09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <1c34ba170904190102g5d60ca61ve9045fcb204a93b6@mail.gmail.com>

On Fri, Apr 17, 2009 at 3:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Riccardo Murri wrote:
>> while converting a class from pure Python to Cython, I've come across
>> the following:
>>
>> ? $ cat r.pyx
>> ? cdef class r(object):
>> ? ? ? ?cdef long long n
>> ? ? ? ?cdef long long d
>>
>> ? ? ? ?def __init__(self, long long n, long long d):
>> ? ? ? ? ? ?self.n = n
>> ? ? ? ? ? ?self.d = d
>>
>> ? ? ? ?def __int__(self):
>> ? ? ? ? ? ?return (self.n // self.d)
>>
>> ? ? ? ?__long__ = __int__
>>
>> ? $ cython -v r.pyx
>> ? Compiling /tmp/r.pyx
>>
>> ? $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
>> -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c r.c -o r.o
>> ? r.c:487: error: ???None??? undeclared here (not in a function)
>>
>> ? $ sed -e '484,490!d;488i^^^^^^^^^^^^^^^^^^^^^^^^ up here' r.c
>> ? ? 0, /*nb_coerce*/
>> ? ? #endif
>> ? ? __pyx_pf_1r_1r___int__, /*nb_int*/
>> ? ? None, /*nb_long*/
>> ? ^^^^^^^^^^^^^^^^^^^^^^^^ up here
>> ? ? 0, /*nb_float*/
>> ? ? #if PY_MAJOR_VERSION < 3
>> ? ? 0, /*nb_oct*/
>
> Yes, that's a bug. I assume that the above code does not make Cython set
> up the C level function name.
>
> Raising an error here would work for now.
>
>
>> I understand that "__long__ = __int__" is not the correct way to do
>> method aliasing in Cython
>
> ... and that's a feature request, it would be nice to make that work.
> Could you create two tickets for those?
>

Done: tickets #287 and #288

http://trac.cython.org/cython_trac/ticket/287
http://trac.cython.org/cython_trac/ticket/288

Thank you!

Riccardo

-- 
Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)

From lrhazi at gmail.com  Sun Apr 19 12:56:22 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Sun, 19 Apr 2009 06:56:22 -0400
Subject: [Cython] Cython just to avoid coding C?
Message-ID: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>

Hello,

Please excuse these naive questions of mine...

Would it make sense to use Cython merely to avoid writing C? That is,
am I missing something in thinking that I could do that?

Assume I wrote a foo.pyx, which uses an external C lib. foo calls
functions on the C lib and defines callbacks the latter lib will be
calling. I then compile the foo.pyx into foo.so, then write a shell
script that runs: python -c 'import foo'

Could someone explain, in OS level terms, what would be happening when
this shell script is running? what would the Python VM be doing? would
it be idle all the time, while the Cython generated code is all that's
running? is there a way to avoid that by compiling the foo.pyx
directly into native executable?

It's late and am having trouble following my own thoughts. I hope
someone out there can detect the source of my confusion and help me
clear it out.

I am looking at using Cython because I need to use Python libs as
well, but started wondering, if I did not need any Python libs at
all.. Could I just use Cython for its pythonic syntax and semantics?

Thanks a lot,
Mohamed.

From dagss at student.matnat.uio.no  Sun Apr 19 13:48:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 19 Apr 2009 13:48:42 +0200 (CEST)
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
Message-ID: <f512b545782c02ba410bd78bcfcddf8d.squirrel@webmail.uio.no>

Mohamed Lrhazi wrote:
> Would it make sense to use Cython merely to avoid writing C? That is,
> am I missing something in thinking that I could do that?
>
> Assume I wrote a foo.pyx, which uses an external C lib. foo calls
> functions on the C lib and defines callbacks the latter lib will be
> calling. I then compile the foo.pyx into foo.so, then write a shell
> script that runs: python -c 'import foo'
>
> Could someone explain, in OS level terms, what would be happening when
> this shell script is running? what would the Python VM be doing? would
> it be idle all the time, while the Cython generated code is all that's
> running? is there a way to avoid that by compiling the foo.pyx
> directly into native executable?

Well, look at the Python executable as two parts:
 a) A library with lots of objects, interfaces etc. for strings, numbers,
classes etc
 b) An interpreter for Python code

The interpreter part would hardly be run at all -- it would interpret
"import foo" and then wait on that. But that is just some code sitting in
memory and having it around doesn't hurt.

You need part a) for Cython programs, and that answers your last question
("no").

All in all, what happens is that the interpreter part of Python does some
things, then does a call into the C code Cython generates which doesn't
return until your application closes.

Bottom line: Unless the memory usage or startup time is a problem, don't
worry about this, just do it :-)

(Hmm.. perhaps someone could comment on any other threads that are running
or reference cycle collection -- but I wouldn't expect it to be a
problem.)

Dag Sverre


From stefan_ml at behnel.de  Sun Apr 19 13:58:43 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 19 Apr 2009 13:58:43 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
Message-ID: <49EB11F3.1080908@behnel.de>


Mohamed Lrhazi wrote:
> Would it make sense to use Cython merely to avoid writing C? That is,
> am I missing something in thinking that I could do that?
> 
> Assume I wrote a foo.pyx, which uses an external C lib. foo calls
> functions on the C lib and defines callbacks the latter lib will be
> calling. I then compile the foo.pyx into foo.so, then write a shell
> script that runs: python -c 'import foo'

That should work. It's basically the same as starting a Python module
manually. You have to keep the interpreter alive while the C lib does its
work, though. That usually means calling into some kind of event loop of
the C library that does not return but only calls predefined callbacks (GUI
or networking toolkits come to mind).

Note that you can use pyximport to build the module for you, i.e.

    $ ls
    foo.pyx
    $ export PYTHONPATH=$PWD
    $ python -c 'import pyximport; pyximport.install(); import foo'

At least during development, this proves very helpful, as you safe the
extra "run Cython and build the .so" steps.

BTW, it does not seem to work to just run

    $ python -m foo

for a "foo.so", the error being "No code object available for foo". Not
sure if there is anything we can do about that...


> Could someone explain, in OS level terms, what would be happening when
> this shell script is running? what would the Python VM be doing? would
> it be idle all the time, while the Cython generated code is all that's
> running?

The Python VM (with which I mean the normal eval loop) would only be
involved when you import or call normal Python code.


> I am looking at using Cython because I need to use Python libs as
> well, but started wondering, if I did not need any Python libs at
> all.. Could I just use Cython for its pythonic syntax and semantics?

Well, Cython is not designed for writing stand-alone C programs. You always
have the overhead of requiring CPython as a dependency. That's great if you
can make use of Python libraries or at least Python data types or
exceptions, but if you use none of those, you just end up with a large,
unused dependency, so that's not the optimal use case.

Stefan

From lodatom at gmail.com  Sun Apr 19 15:16:46 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Sun, 19 Apr 2009 09:16:46 -0400
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
Message-ID: <ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>

On Sun, Apr 19, 2009 at 6:56 AM, Mohamed Lrhazi <lrhazi at gmail.com> wrote:
> is there a way to avoid that by compiling the foo.pyx
> directly into native executable?

There's no way to get rid of the Python interpreter, but you could
embed the interpreter and statically link your module.  This has two
main advantages: (1) you only have a single file to deal with, rather
than the script and the .so, and (2) you can profile using `gprof',
which does not work with shared objects.

Here's an example:

--------------------------------- mylib.pyx --------------------------
import sys

cdef extern from "math.h":
    double lgamma(double)
    double exp(double)


cdef inline double gamma(double n):
    return exp( lgamma(n) )


def main(argv = None):
    if argv is None:
        argv = sys.argv
    if len(argv) != 2:
        name = argv[0]  if argv  else '<prog>'
        sys.stderr.write("USAGE: %s n\nPrints gamma(n).\n" % name)
        sys.exit(1)

    print gamma(float(argv[1]))
----------------------------------------------------------------------

----------------------------------- main.c ---------------------------
#include <Python.h>

// For each Cython module you want to embed, you must declare an
// init<module> function, like so:
PyMODINIT_FUNC initmylib(void);

int
main(int argc, char *argv[])
{
    // The first step is to set up the Python interpreter:
    Py_Initialize();
    PySys_SetArgv(argc, argv);

    // Next, we need to tell Python that our module exists.  Call each
    // of the functions you declared above.
    initmylib();

    // Now do some Python stuff.  The easiest thing to do is to give
    // the interpreter a string of Python code that imports your
    // module and calls it.
    PyRun_SimpleString("from mylib import main\n"
                       "main()\n");

    // When we're done, tell Python to clean up.
    Py_Finalize();
    return 0;
}
----------------------------------------------------------------------

On my machine, I compile with:

cython  mylib.pyx
gcc -fPIC -g -O2 -I /usr/include/python2.5  -c -o main.o main.c
gcc -fPIC -g -O2 -I /usr/include/python2.5  -c -o mylib.o mylib.c
gcc -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions \
    main.o mylib.o /usr/lib/python2.5/config/libpython2.5.a \
    -lm -ldl -pthread -lutil \
    -o main

(See http://docs.python.org/extending/embedding.html for details on
how to properly compile.)

Now I can run my program as a normal executable:

$ ./main
USAGE: ./main n
Prints gamma(n).
$ ./main 6
120.0
$ ./main 100
9.33262154439e+155


--
Mark

From lrhazi at gmail.com  Sun Apr 19 16:06:09 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Sun, 19 Apr 2009 10:06:09 -0400
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
Message-ID: <c4ab18df0904190706p50d00031vaf9be5e40cbb5906@mail.gmail.com>

Thank you so much eveybody. you answered all my questions and gave me
more. Now, when you're nice to people, they want more... so let me ask
for a bit more advice:

What I want to write is indeed, like Stefan guessed, a plugin for a
network OS. So my Cython code would be calling functions to register
for events of interest, and then define function callbacks, then call
a poll() function and wait for events to fire and callbacks to be
called.

Because the Cython code would never return, the "import foo" would be
the last thing to run, essentially. So, How can I define the callbacks
in my normal Python land, if I wanted to?

The poll function for the system actually comes in flavors, one of
which is nonblocking, but am still trying to understand all options.

I am also wondering if there was an obvious way to "connect" this
event loop, in this NOS, with Twisted Python reactors? but that's even
more vague a question, aint it?

Thank you very much all, and thanks for Cython. BTW, how do you
pronounce it? with an "s" sound or "k" sound?

Mohamed.


On Sun, Apr 19, 2009 at 9:16 AM, Mark Lodato <lodatom at gmail.com> wrote:
> On Sun, Apr 19, 2009 at 6:56 AM, Mohamed Lrhazi <lrhazi at gmail.com> wrote:
>> is there a way to avoid that by compiling the foo.pyx
>> directly into native executable?
>
> There's no way to get rid of the Python interpreter, but you could
> embed the interpreter and statically link your module. ?This has two
> main advantages: (1) you only have a single file to deal with, rather
> than the script and the .so, and (2) you can profile using `gprof',
> which does not work with shared objects.
>
> Here's an example:
>
> --------------------------------- mylib.pyx --------------------------
> import sys
>
> cdef extern from "math.h":
> ? ?double lgamma(double)
> ? ?double exp(double)
>
>
> cdef inline double gamma(double n):
> ? ?return exp( lgamma(n) )
>
>
> def main(argv = None):
> ? ?if argv is None:
> ? ? ? ?argv = sys.argv
> ? ?if len(argv) != 2:
> ? ? ? ?name = argv[0] ?if argv ?else '<prog>'
> ? ? ? ?sys.stderr.write("USAGE: %s n\nPrints gamma(n).\n" % name)
> ? ? ? ?sys.exit(1)
>
> ? ?print gamma(float(argv[1]))
> ----------------------------------------------------------------------
>
> ----------------------------------- main.c ---------------------------
> #include <Python.h>
>
> // For each Cython module you want to embed, you must declare an
> // init<module> function, like so:
> PyMODINIT_FUNC initmylib(void);
>
> int
> main(int argc, char *argv[])
> {
> ? ?// The first step is to set up the Python interpreter:
> ? ?Py_Initialize();
> ? ?PySys_SetArgv(argc, argv);
>
> ? ?// Next, we need to tell Python that our module exists. ?Call each
> ? ?// of the functions you declared above.
> ? ?initmylib();
>
> ? ?// Now do some Python stuff. ?The easiest thing to do is to give
> ? ?// the interpreter a string of Python code that imports your
> ? ?// module and calls it.
> ? ?PyRun_SimpleString("from mylib import main\n"
> ? ? ? ? ? ? ? ? ? ? ? "main()\n");
>
> ? ?// When we're done, tell Python to clean up.
> ? ?Py_Finalize();
> ? ?return 0;
> }
> ----------------------------------------------------------------------
>
> On my machine, I compile with:
>
> cython ?mylib.pyx
> gcc -fPIC -g -O2 -I /usr/include/python2.5 ?-c -o main.o main.c
> gcc -fPIC -g -O2 -I /usr/include/python2.5 ?-c -o mylib.o mylib.c
> gcc -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions \
> ? ?main.o mylib.o /usr/lib/python2.5/config/libpython2.5.a \
> ? ?-lm -ldl -pthread -lutil \
> ? ?-o main
>
> (See http://docs.python.org/extending/embedding.html for details on
> how to properly compile.)
>
> Now I can run my program as a normal executable:
>
> $ ./main
> USAGE: ./main n
> Prints gamma(n).
> $ ./main 6
> 120.0
> $ ./main 100
> 9.33262154439e+155
>
>
> --
> Mark
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
" Logic merely sanctions the conquests of the intuition."
Jacques Hadamard

From stefan_ml at behnel.de  Sun Apr 19 16:46:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 19 Apr 2009 16:46:27 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
Message-ID: <49EB3943.9090409@behnel.de>


Mark Lodato wrote:
> you could embed the interpreter and statically link your module.
> [...]
> ----------------------------------- main.c ---------------------------
> #include <Python.h>
> 
> // For each Cython module you want to embed, you must declare an
> // init<module> function, like so:
> PyMODINIT_FUNC initmylib(void);
> 
> int
> main(int argc, char *argv[])
> {
>     // The first step is to set up the Python interpreter:
>     Py_Initialize();
>     PySys_SetArgv(argc, argv);
> 
>     // Next, we need to tell Python that our module exists.  Call each
>     // of the functions you declared above.
>     initmylib();
> 
>     // Now do some Python stuff.  The easiest thing to do is to give
>     // the interpreter a string of Python code that imports your
>     // module and calls it.
>     PyRun_SimpleString("from mylib import main\n"
>                        "main()\n");
> 
>     // When we're done, tell Python to clean up.
>     Py_Finalize();
>     return 0;
> }
> ----------------------------------------------------------------------

I wonder if it makes sense to add support for this to Cython. You could
have a command line option that adds a suitable main() function that runs
the module code in an embedded CPython interpreter (and that does the setup
correctly for Py2 and Py3).

Or maybe an additional tool like Python's "freeze" (what about naming it
"cool" or "maincyfy"? Or maybe "(em)bedcy"?) could take a list of Cython
modules and generate a main .c file like the one above, which would simply
run the module initialisation in order. That would make it trivial to write
a 'main module' (passed as last argument) that simply contains the main
program code in its body.

We might even be able to check module interdependencies here, so that the
tool can bail out if it knows that the sequential module initialisations
cannot work in an embedded setup.

Tons of cool stuff to do here ...

Stefan

From dagss at student.matnat.uio.no  Sun Apr 19 22:50:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 19 Apr 2009 22:50:02 +0200
Subject: [Cython] Proposal: Packed struct support?
Message-ID: <49EB8E7A.5060106@student.matnat.uio.no>

Could we have some kind of keyword to define packed structs in Cython?
This is not part of the C standard, but is supported in most C compilers
through extensions. (Obviously one can do it now through writing a
header file, but that is inconvenient.)

I.e. something like:

cdef packed struct MyStruct:
     char a
     double b

would output conditional code to do either __attribute__((__packed__))
or #pragma pack(push,1), #pragma pack(pop) (gcc/icc and Visual C, 
respectively).

PEP-3118 buffers (like NumPy arrays) can contain arbitrarily packed 
data, and this would help me write readable testing code for that (as 
well as anyone actually using such types, however of course one should 
always align ones data properly...)

BTW, any volunteers for Visual C testing on this? Also, does anyone know 
wheter it is possible to have a #pragma in a #define, like this:

#define SOMEMACRO #pragma pack(push, 1)

?

-- 
Dag Sverre


From greg.ewing at canterbury.ac.nz  Mon Apr 20 02:52:30 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 20 Apr 2009 12:52:30 +1200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <f512b545782c02ba410bd78bcfcddf8d.squirrel@webmail.uio.no>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<f512b545782c02ba410bd78bcfcddf8d.squirrel@webmail.uio.no>
Message-ID: <49EBC74E.8090608@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> Bottom line: Unless the memory usage or startup time is a problem, don't
> worry about this, just do it :-)

If your code won't be making any use of Python objects
or Python libraries, there probably won't be much
advantage to using Cython -- you'll still be writing
C code, just with a slightly different syntax.

On the other hand, if you want to take advantage of
Python features in your code, this is a perfectly
reasonable thing to do.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Mon Apr 20 02:56:13 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 20 Apr 2009 12:56:13 +1200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
Message-ID: <49EBC82D.9030200@canterbury.ac.nz>

Mohamed Lrhazi wrote:

> is there a way to avoid that by compiling the foo.pyx
> directly into native executable?

It's probably not feasible to create a Cython module
that will work with no Python interpreter at all.
Even a minimal Cython source file probably generates
references to various Python runtime routines.

-- 
Greg

From sturla at molden.no  Mon Apr 20 13:56:48 2009
From: sturla at molden.no (Sturla Molden)
Date: Mon, 20 Apr 2009 13:56:48 +0200
Subject: [Cython] Proposal: Packed struct support?
In-Reply-To: <49EB8E7A.5060106@student.matnat.uio.no>
References: <49EB8E7A.5060106@student.matnat.uio.no>
Message-ID: <49EC6300.1050907@molden.no>

On 4/19/2009 10:50 PM, Dag Sverre Seljebotn wrote:

> Could we have some kind of keyword to define packed structs in Cython?
> This is not part of the C standard, but is supported in most C compilers
> through extensions. 

Since it's not in the standard, there is no standard syntax for this.

But you can fake it using a char array and some constants storing offsets.

    /*
    cdef packed struct MyStruct:
       char a
       double b
    */
    const int mystruct_size = sizeof(char)+sizeof(double);
    const int mystruct_offset_a = 0;
    const int mystruct_offset_b = sizeof(char);

    /*
    cdef MyStruct mystruct
    */
    char mystruct[mystruct_size]; /* sizeof(char) is 1 by defintion */


    /*
    cdef char a
    cdef double b
    a = mystruct.a
    b = mystruct.b
    */
    char a;
    double b;
    a = *(char *)((char *)mystruct + mystruct_offset_a);
    b = *(double *)((char *)mystruct + mystruct_offset_b);



Sturla Molden


From dagss at student.matnat.uio.no  Mon Apr 20 16:17:32 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 20 Apr 2009 16:17:32 +0200 (CEST)
Subject: [Cython] Proposal: Packed struct support?
In-Reply-To: <49EC6300.1050907@molden.no>
References: <49EB8E7A.5060106@student.matnat.uio.no>
	<49EC6300.1050907@molden.no>
Message-ID: <999877f81f0134f2d1acd578effc2769.squirrel@webmail.uio.no>

Sturla Molden wrote:
> On 4/19/2009 10:50 PM, Dag Sverre Seljebotn wrote:
>
>> Could we have some kind of keyword to define packed structs in Cython?
>> This is not part of the C standard, but is supported in most C compilers
>> through extensions.
>
> Since it's not in the standard, there is no standard syntax for this.
>
> But you can fake it using a char array and some constants storing offsets.

Thanks, good idea. Indeed, since Cython would know the struct definitions,
we could just directly output the necesarry pointer arithmetic :-)

However using the various compiler extensions would take less time to
implement in Cython and make for more readable code. I'm not that worried
about compatability -- we're only considering the subset of C compilers on
which CPython compiles anyway, and it's a fairly universal compiler
extension (though with differing syntax).

So (assuming a positive vote on this) I suppose I'll go that route first,
and then take your approach if there's a need.

Dag Sverre


From jf.moulin at gmail.com  Mon Apr 20 17:17:29 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Mon, 20 Apr 2009 17:17:29 +0200
Subject: [Cython] help with function pointer
Message-ID: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>

Hi

I am struggling with a function pointer problem:

cdef class LstFileReader_FAST:
    cdef tuple (*do_read_ev)(LstFileReader_FAST)

     def __init__(self,fname):

        self.fname = fname
        self.goto_start()
        self.do_read_ev = self._next_ev
.....

results in a compilation error:

Cannot assign type 'tuple (LstFileReader_FAST, int
__pyx_skip_dispatch)' to 'tuple (*)(LstFileReader_FAST)'

I found no clear explanation for this __pyx_skip_dispatch... and
somewhere above in my code I have an almostt similar statement which
works:

cdef int (*locate)(DelayLine, np.ndarray)
...
    def __init__(self):
               ....
                self.locate = self.locate_single_terminal


Any clue?
Thanks in advance for helping a beginner!
JF

From dalcinl at gmail.com  Mon Apr 20 17:29:18 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 20 Apr 2009 12:29:18 -0300
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
Message-ID: <e7ba66e40904200829q19871895kf3c6f48214caf5ff@mail.gmail.com>

In the past, Cython accepted such code. Unfortunately, at some point
this functionality was lost. However, you can add a forward
declaration and a ctypedef like this:

cdef class LstFileReader_FAST # forward declaration

ctypedef tuple read_fn_t(LstFileReader_FAST) # ctypedef for function type

cdef class LstFileReader_FAST:
   cdef read_fn_t *do_read_ev # note do_read_ev is a pointer to read_fn_t !!!



On Mon, Apr 20, 2009 at 12:17 PM, Jean-Francois Moulin
<jf.moulin at gmail.com> wrote:
> Hi
>
> I am struggling with a function pointer problem:
>
> cdef class LstFileReader_FAST:
> ? ?cdef tuple (*do_read_ev)(LstFileReader_FAST)
>
> ? ? def __init__(self,fname):
>
> ? ? ? ?self.fname = fname
> ? ? ? ?self.goto_start()
> ? ? ? ?self.do_read_ev = self._next_ev
> .....
>
> results in a compilation error:
>
> Cannot assign type 'tuple (LstFileReader_FAST, int
> __pyx_skip_dispatch)' to 'tuple (*)(LstFileReader_FAST)'
>
> I found no clear explanation for this __pyx_skip_dispatch... and
> somewhere above in my code I have an almostt similar statement which
> works:
>
> cdef int (*locate)(DelayLine, np.ndarray)
> ...
> ? ?def __init__(self):
> ? ? ? ? ? ? ? ....
> ? ? ? ? ? ? ? ?self.locate = self.locate_single_terminal
>
>
> Any clue?
> Thanks in advance for helping a beginner!
> JF
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Apr 20 17:34:52 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 20 Apr 2009 12:34:52 -0300
Subject: [Cython] Proposal: Packed struct support?
In-Reply-To: <49EB8E7A.5060106@student.matnat.uio.no>
References: <49EB8E7A.5060106@student.matnat.uio.no>
Message-ID: <e7ba66e40904200834w3fbc3114gee600d5832110bd9@mail.gmail.com>

On Sun, Apr 19, 2009 at 5:50 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>
> #define SOMEMACRO #pragma pack(push, 1)
>

No way, the preprocessor will complain. You'll need the C99
"_Pragma()" operator.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Mon Apr 20 20:38:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 20 Apr 2009 11:38:26 -0700
Subject: [Cython] Proposal: Packed struct support?
In-Reply-To: <999877f81f0134f2d1acd578effc2769.squirrel@webmail.uio.no>
References: <49EB8E7A.5060106@student.matnat.uio.no>
	<49EC6300.1050907@molden.no>
	<999877f81f0134f2d1acd578effc2769.squirrel@webmail.uio.no>
Message-ID: <1ADDF964-8FE1-4BA9-8B10-B31B06B1F8B5@math.washington.edu>

On Apr 20, 2009, at 7:17 AM, Dag Sverre Seljebotn wrote:

> Sturla Molden wrote:
>> On 4/19/2009 10:50 PM, Dag Sverre Seljebotn wrote:
>>
>>> Could we have some kind of keyword to define packed structs in  
>>> Cython?
>>> This is not part of the C standard, but is supported in most C  
>>> compilers
>>> through extensions.
>>
>> Since it's not in the standard, there is no standard syntax for this.
>>
>> But you can fake it using a char array and some constants storing  
>> offsets.
>
> Thanks, good idea. Indeed, since Cython would know the struct  
> definitions,
> we could just directly output the necesarry pointer arithmetic :-)

I'd rather avoid such messiness in the C code if we can. +1 to being  
able to specify a packed keyword though.

- Robert


From robertwb at math.washington.edu  Mon Apr 20 20:42:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 20 Apr 2009 11:42:03 -0700
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
Message-ID: <6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>

On Apr 20, 2009, at 8:17 AM, Jean-Francois Moulin wrote:

> Hi
>
> I am struggling with a function pointer problem:
>
> cdef class LstFileReader_FAST:
>     cdef tuple (*do_read_ev)(LstFileReader_FAST)
>
>      def __init__(self,fname):
>
>         self.fname = fname
>         self.goto_start()
>         self.do_read_ev = self._next_ev
> .....
>
> results in a compilation error:
>
> Cannot assign type 'tuple (LstFileReader_FAST, int
> __pyx_skip_dispatch)' to 'tuple (*)(LstFileReader_FAST)'
>
> I found no clear explanation for this __pyx_skip_dispatch...

I bet you've declared self._next_ev to be a cpdef (not cdef)  
function. cpdef functions have an extra implicit argument  
"__pyx_skip_dispatch" which lets them know whether or not they should  
skip a possible lookup in the objects dictionary.

This should probably be a clearer error though...

- Robert

From robertwb at math.washington.edu  Mon Apr 20 20:48:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 20 Apr 2009 11:48:28 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <49EB3943.9090409@behnel.de>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
Message-ID: <4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>

On Apr 19, 2009, at 7:46 AM, Stefan Behnel wrote:

> Mark Lodato wrote:
>> you could embed the interpreter and statically link your module.
>> [...]
>> ----------------------------------- main.c  
>> ---------------------------
>> #include <Python.h>
>>
>> // For each Cython module you want to embed, you must declare an
>> // init<module> function, like so:
>> PyMODINIT_FUNC initmylib(void);
>>
>> int
>> main(int argc, char *argv[])
>> {
>>     // The first step is to set up the Python interpreter:
>>     Py_Initialize();
>>     PySys_SetArgv(argc, argv);
>>
>>     // Next, we need to tell Python that our module exists.  Call  
>> each
>>     // of the functions you declared above.
>>     initmylib();
>>
>>     // Now do some Python stuff.  The easiest thing to do is to give
>>     // the interpreter a string of Python code that imports your
>>     // module and calls it.
>>     PyRun_SimpleString("from mylib import main\n"
>>                        "main()\n");
>>
>>     // When we're done, tell Python to clean up.
>>     Py_Finalize();
>>     return 0;
>> }
>> --------------------------------------------------------------------- 
>> -
>
> I wonder if it makes sense to add support for this to Cython. You  
> could
> have a command line option that adds a suitable main() function  
> that runs
> the module code in an embedded CPython interpreter (and that does  
> the setup
> correctly for Py2 and Py3).

+1, I think that's an excellent idea. http://trac.cython.org/ 
cython_trac/ticket/289

> Or maybe an additional tool like Python's "freeze" (what about  
> naming it
> "cool" or "maincyfy"? Or maybe "(em)bedcy"?) could take a list of  
> Cython
> modules and generate a main .c file like the one above, which would  
> simply
> run the module initialisation in order. That would make it trivial  
> to write
> a 'main module' (passed as last argument) that simply contains the  
> main
> program code in its body.
>
> We might even be able to check module interdependencies here, so  
> that the
> tool can bail out if it knows that the sequential module  
> initialisations
> cannot work in an embedded setup.
>
> Tons of cool stuff to do here ...

Yep, that'd be cool. With enough work, we could totally do a  
"freeze." (Actually, if we use freeze it might not be that much work  
at all, but I haven't looked into it yet.)

- Robert



From stefan_ml at behnel.de  Tue Apr 21 07:43:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 07:43:04 +0200
Subject: [Cython] Path towards Python completeness
Message-ID: <49ED5CE8.6090708@behnel.de>

Hi,

I just read the release announcement of PyPy's 1.1 beta:

http://morepypy.blogspot.com/2009/04/beta-for-110-released.html

According to their benchmarks, they are now less than 2x slower than
CPython in most cases and in some cases even 20% faster. Cython is
currently about 30% faster than CPython for plain Python code, but that
only concerns the "Python code that Cython supports", which is not at all
comparable to what PyPy supports.

My feeling is that the goal to make CPython support "most of Python" is
getting more important simply because PyPy exists. There is not that much
that is missing, but the sad thing is that the main part that *is* missing
is a very important part of the language: closures.

I feel that closure support has become a real blocker on the path towards
Python completeness. Closure support will actually open up a can of worms,
but a friendly one, as it will lay out the path towards lambda support and
nested functions/classes. Especially support for generators seems to be an
extremely worthy goal, but in the current state of Cython, it just cannot
be done.

Robert has started work on this at the first Cython workshop, but it turned
out to be more work than what was doable at the time. So there are parts of
this implementation in Cython's code base already, but it still needs some
fair investment to finish it up.

I would like to focus some (wo)manpower on this topic. For that, I need to
understand better what is there and what remains to be done, so that we can
break down the task into doable steps and get going.

Anyone else interested?

Stefan

From stefan_ml at behnel.de  Tue Apr 21 08:30:06 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 08:30:06 +0200
Subject: [Cython] Path towards Python completeness
In-Reply-To: <49ED5CE8.6090708@behnel.de>
References: <49ED5CE8.6090708@behnel.de>
Message-ID: <49ED67EE.5090801@behnel.de>


Stefan Behnel wrote:
> My feeling is that the goal to make CPython support "most of Python"

... that may not sound like an interesting goal, unless you replace
"CPython" with "Cython". ;)

Stefan


From jf.moulin at gmail.com  Tue Apr 21 09:45:20 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 21 Apr 2009 09:45:20 +0200
Subject: [Cython] help with function pointer
In-Reply-To: <6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
Message-ID: <af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>

Hi,

first of all, thanks for the replies.

As for the first answer by Lisandro, I do not understand then why my
first code fragment at all works...
The only difference is that the function called has two params, one
instance of one of my classes and a numpy array, while the faulty
piece of code should be called on an instance
of one of my classes only...

As for the second, from Robert, now, very well... how do I specify a
value for this implicit parameter then!? It is a bint I guess... I
tried to set it to 1 and got new error messages:
delay_line.pyx:961:30: Cannot assign type 'tuple (LstFileReader_FAST,
int __pyx_skip_dispatch, struct
__pyx_opt_args_12delay_line_C_18LstFileReader_FAST_next_ev
*__pyx_optional_args)' to 'tuple (*)(LstFileReader_FAST)'

I also tried to use cdef in all cases, I get no compile time error but
at runtime my object LstFileReader_FAST appears no longer to be
defined!? I do not understand because all the calls are made from
within Cython code... (so that I expect the cpdef is not really
needed)


Thanks a lot again for your help!
JF

On Mon, Apr 20, 2009 at 8:42 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 20, 2009, at 8:17 AM, Jean-Francois Moulin wrote:
>
>> Hi
>>
>> I am struggling with a function pointer problem:
>>
>> cdef class LstFileReader_FAST:
>>     cdef tuple (*do_read_ev)(LstFileReader_FAST)
>>
>>      def __init__(self,fname):
>>
>>         self.fname = fname
>>         self.goto_start()
>>         self.do_read_ev = self._next_ev
>> .....
>>
>> results in a compilation error:
>>
>> Cannot assign type 'tuple (LstFileReader_FAST, int
>> __pyx_skip_dispatch)' to 'tuple (*)(LstFileReader_FAST)'
>>
>> I found no clear explanation for this __pyx_skip_dispatch...
>
> I bet you've declared self._next_ev to be a cpdef (not cdef)
> function. cpdef functions have an extra implicit argument
> "__pyx_skip_dispatch" which lets them know whether or not they should
> skip a possible lookup in the objects dictionary.
>
> This should probably be a clearer error though...
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Tue Apr 21 10:02:05 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 10:02:05 +0200 (CEST)
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
Message-ID: <fe44d821ed0a3a47a662ce0f9dff0957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Jean-Francois Moulin wrote:
> As for the second, from Robert, now, very well... how do I specify a
> value for this implicit parameter then!?

You don't. "cpdef" functions have a special signature, so you can't use
them as callbacks.


> I also tried to use cdef in all cases, I get no compile time error

:)


> but at runtime my object LstFileReader_FAST appears no longer to be
> defined!?

Then you changed other things in your code that made this break. Unless
you show us the code you try to compile, we cannot help you.

Stefan


From robertwb at math.washington.edu  Tue Apr 21 10:23:06 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 01:23:06 -0700
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
Message-ID: <CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>

On Apr 21, 2009, at 12:45 AM, Jean-Francois Moulin wrote:

> Hi,
>
> first of all, thanks for the replies.
>
> As for the first answer by Lisandro, I do not understand then why my
> first code fragment at all works...
> The only difference is that the function called has two params, one
> instance of one of my classes and a numpy array, while the faulty
> piece of code should be called on an instance
> of one of my classes only...
>
> As for the second, from Robert, now, very well... how do I specify a
> value for this implicit parameter then!? It is a bint I guess... I
> tried to set it to 1 and got new error messages:
> delay_line.pyx:961:30: Cannot assign type 'tuple (LstFileReader_FAST,
> int __pyx_skip_dispatch, struct
> __pyx_opt_args_12delay_line_C_18LstFileReader_FAST_next_ev
> *__pyx_optional_args)' to 'tuple (*)(LstFileReader_FAST)'

You can't. I agree the errors are more cryptic than they should be,  
but C functions have no concept of runtime dispatching or default  
arguments, so it doesn't work to cast such a function to a C function  
(well, maybe one could hack it, but it'd be ugly and fragile).

> I also tried to use cdef in all cases, I get no compile time error but
> at runtime my object LstFileReader_FAST appears no longer to be
> defined!? I do not understand because all the calls are made from
> within Cython code... (so that I expect the cpdef is not really
> needed)

I am guessing you're doing something like

a. LstFileReader_FAST(...)

and it's complaining that "a has not attribute 'LstFileReader_FAST'".  
Usually this indicates that a is not typed correctly (e.g. it thinks  
a is an ordinary object). You can run cython with the -a option to  
easily see this kind of thing.

- Robert


From robertwb at math.washington.edu  Tue Apr 21 10:34:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 01:34:51 -0700
Subject: [Cython] Path towards Python completeness
In-Reply-To: <49ED5CE8.6090708@behnel.de>
References: <49ED5CE8.6090708@behnel.de>
Message-ID: <80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>

On Apr 20, 2009, at 10:43 PM, Stefan Behnel wrote:

> Hi,
>
> I just read the release announcement of PyPy's 1.1 beta:
>
> http://morepypy.blogspot.com/2009/04/beta-for-110-released.html
>
> According to their benchmarks, they are now less than 2x slower than
> CPython in most cases and in some cases even 20% faster. Cython is
> currently about 30% faster than CPython for plain Python code, but  
> that
> only concerns the "Python code that Cython supports", which is not  
> at all
> comparable to what PyPy supports.
>
> My feeling is that the goal to make CPython support "most of  
> Python" is
> getting more important simply because PyPy exists. There is not  
> that much
> that is missing, but the sad thing is that the main part that *is*  
> missing
> is a very important part of the language: closures.
>
> I feel that closure support has become a real blocker on the path  
> towards
> Python completeness. Closure support will actually open up a can of  
> worms,
> but a friendly one, as it will lay out the path towards lambda  
> support and
> nested functions/classes. Especially support for generators seems  
> to be an
> extremely worthy goal, but in the current state of Cython, it just  
> cannot
> be done.

All of the above are syntactic sugar for special closures.

> Robert has started work on this at the first Cython workshop, but  
> it turned
> out to be more work than what was doable at the time. So there are  
> parts of
> this implementation in Cython's code base already, but it still  
> needs some
> fair investment to finish it up.
>
> I would like to focus some (wo)manpower on this topic. For that, I  
> need to
> understand better what is there and what remains to be done, so  
> that we can
> break down the task into doable steps and get going.
>
> Anyone else interested?

This has been bugging me for a *long* time. I'll pull out and go  
though the random scattered stuff I've done over the past year  
(something else more urgent has always come up everytime I've set out  
to do this) and put it up within the week. Should I make a closure  
branch on the site?

- Robert



From robertwb at math.washington.edu  Tue Apr 21 11:16:05 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 02:16:05 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
Message-ID: <4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>

On Apr 20, 2009, at 11:48 AM, Robert Bradshaw wrote:

> On Apr 19, 2009, at 7:46 AM, Stefan Behnel wrote:
>
>> Mark Lodato wrote:
>>> you could embed the interpreter and statically link your module.
>>> [...]
>>> ----------------------------------- main.c
>>> ---------------------------
>>> #include <Python.h>
>>>
>>> // For each Cython module you want to embed, you must declare an
>>> // init<module> function, like so:
>>> PyMODINIT_FUNC initmylib(void);
>>>
>>> int
>>> main(int argc, char *argv[])
>>> {
>>>     // The first step is to set up the Python interpreter:
>>>     Py_Initialize();
>>>     PySys_SetArgv(argc, argv);
>>>
>>>     // Next, we need to tell Python that our module exists.  Call
>>> each
>>>     // of the functions you declared above.
>>>     initmylib();
>>>
>>>     // Now do some Python stuff.  The easiest thing to do is to give
>>>     // the interpreter a string of Python code that imports your
>>>     // module and calls it.
>>>     PyRun_SimpleString("from mylib import main\n"
>>>                        "main()\n");
>>>
>>>     // When we're done, tell Python to clean up.
>>>     Py_Finalize();
>>>     return 0;
>>> }
>>> -------------------------------------------------------------------- 
>>> -
>>> -
>>
>> I wonder if it makes sense to add support for this to Cython. You
>> could
>> have a command line option that adds a suitable main() function
>> that runs
>> the module code in an embedded CPython interpreter (and that does
>> the setup
>> correctly for Py2 and Py3).
>
> +1, I think that's an excellent idea. http://trac.cython.org/
> cython_trac/ticket/289

There's an --embed command line option now, which generates an  
appropriate main() method. One then compiles the resulting C file  
into an executable (which still needs to be linked against the Python  
library).

- Robert

From jf.moulin at gmail.com  Tue Apr 21 11:19:06 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 21 Apr 2009 11:19:06 +0200
Subject: [Cython] help with function pointer
In-Reply-To: <CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
Message-ID: <af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>

Hi!
thanks for the answers!

Here is the runtime error message:

python read_sweep_C.py
LstFileReader_FAST('/home/jfmoulin/Lab/Experiments/refsans/07_11_PMB/BPSi-JFM-B1-087.lst')
Traceback (most recent call last):
  File "read_sweep_C.py", line 278, in <module>
    ui = exple2()
  File "read_sweep_C.py", line 225, in exple2
    tof_resol=.05
  File "delay_line.pyx", line 103, in delay_line_C.UI.__init__
(delay_line.c:1212)
    self.det_reader = DetectorReader(fname=self.fname,reader=reader,
  File "delay_line.pyx", line 328, in
delay_line_C.DetectorReader.__init__ (delay_line.c:3445)
    self.list_mode_file = [ListModeFile(fname=fname,reader=reader)
  File "delay_line.pyx", line 719, in
delay_line_C.ListModeFile.__init__ (delay_line.c:7367)
    self.reader = eval('%s(\'%s\')'%(reader,self.fname))
  File "<string>", line 1, in <module>
NameError: name 'LstFileReader_FAST' is not defined

So it seems not to be a type error for an object which would have
a LstFileReader_FAST property or method, right ?

Okay, you asked for it..., here is the full code (yes it is still very
messy and full of FIXMEs but the python version does run...albeit
sooooo slow, hence Cython)

Cheers, JF

from __future__ import division
import numpy as np
cimport numpy as np
import scipy # FIXME??? cimport?
import exceptions
import types
import struct
import string
import sys
import subprocess
import cPickle
import time
import exceptions
from os.path import getsize
import struct



DTYPE_int = np.int
DTYPE_long = np.long
DTYPE_double = np.double
ctypedef np.int_t DTYPE_int_t
ctypedef np.long_t DTYPE_long_t
ctypedef np.double_t DTYPE_double_t

# Forward class declaration
cdef class UI
cdef class DetectorReader
cdef class TofSlabber
cdef class LstFileReader_FAST
cdef class DelayLine

cdef class UI:

    """
    A class dealing with the setup of a list mode file reader :

       fname: a filename or a list of filenames (lst files)
       reader: the name of the low-level reader class to use for this lst file
       delay_line_pars: is a dictonary or list of dicts describing the
                        delay line, keys are:
            'name': 'X' or 'Y' is the name of the delay line
            'anode': [devicenum,channelnum] is a list of 2 values telling where
                    to look for the anode signal (the files are numbered in
                    the same order as in fname)
            'cathodes': [[devnum1,channum1],[devnum2,channum2]] is a list of
                       one or two signal adresses in the format as above.
                       Each of these one/two addresses corresponds to one
                       end of the cathode
            'length': 342.7 is a value in ns for the length of the delay line
            'tol': 20 a tolerance in ns for the discrimination of events
            'npix': 256 the number of pixels for the full delay line length

      tof_resol : .1 the tof resolution used for histogramming
      tof_lims : the tof limits used for the histogram
                 (None goes to a 0-Tmax default)
      timing_accuracy : maximum possible mesuered missmatch in ns between two
                        events which are to be considered simultaneous
                        (mismatch could be due to cabling or timing of the
                        different cards, 3 ns should be safe)

    """

    cdef list fname # FIXME
    cdef list delay_line_pars
    cdef int timing_accuracy
    cdef DetectorReader det_reader
    cdef TofSlabber slabber
    cdef dict strobe_seq
    cdef str tof_histo

    cdef public np.ndarray slab, histo
    cdef public nsweep

    def __init__(self,\
                 fname,\
                 reader='LstFileReader_FAST',\
                 delay_line_pars=[{'name':'X',
                                   'anode':[0,0],
                                   'cathodes':[[0,1],[0,2]],
                                   'length':342.7,
                                   'tol':20,
                                   'npix':256
                                   },
                                  {'name':'Y',
                                   'anode':[0,0],
                                   'cathodes':[0,3],
                                   'length':342.7,
                                   'tol':20,
                                   'npix':256
                                  }
                                ],\
                 tof_resol = .1,\
                 tof_lims = None,\
                 timing_accuracy = 3


                ):

        self.fname = fname
        self.delay_line_pars = delay_line_pars
        self.timing_accuracy = timing_accuracy

        self.det_reader = DetectorReader(fname=self.fname,reader=reader,
                                         delay_line_pars=self.delay_line_pars,
                                         timing_accuracy = self.timing_accuracy

                                        )
        self.slabber = TofSlabber(resol=tof_resol,tof_lims=tof_lims,
                                  npix=(delay_line_pars[0]['npix'],
                                        delay_line_pars[1]['npix']
                                       )
                                 )
        #self.reporter = Lst_reporter(self.slabber)
        self.strobe_seq = {}
        tof_histo = 'tof_histo'


    def old_read(self,nsweep=1):
        """reads nsweep of data and does the tof analysis."""
        for i in range(nsweep):
            sweep = self.det_reader.next_sweep()

            cleaned = self.det_reader.clean_sweep(sweep)
            self.slabber._incr(cleaned)

    cdef bint _is_ok_sweep(self, tuple sweep):
        """
        Tests if a sweep is valid (full time between two starts) and increments
        the slabber.
        Returns False if sweep not valid, True if valid
        """
        cdef bint sweep_empty
        cdef int count, swc
        cdef str fi
        cdef tuple sw
        cdef tuple cleaned
        sweep_empty = False

        # For each subfile we test if we got a complete sweep (i.e not None)
        # and if it contains events or not
        for count,fi in enumerate(self.fname):
#           print 'uuuu',sweep[count]
            for swc,sw in enumerate(sweep):
                if sw is None:
                    print 'reached EOF for %s'%count
                    return False
                # A valid sweep maybe empty...
                if sw == ():
                    print 'No counts in last read() for line %s'\
                            %swc
                    sweep_empty = True
            # A valid non empty sweep needs cleaning and in either case
            # we should increment the slabber with the adequate values
            if not sweep_empty:
                cleaned = self.det_reader.clean_sweep(sweep)
                self.slabber._incr(cleaned)
                # let us make the slabs and histo directly
                # accessible from the UI (...in cython seems to be too deep)
                self.slab = self.slabber.slab.view()
                self.histo = self.slabber.tof_histo.view()
                self.nsweep = self.slabber.nsweep
            else:
                self.slabber._incr(())
                # let us make the slabs and histo directly
                # accessible from the UI (...in cython seems to be too
deep) FIXME property
                self.slab = self.slabber.slab.view()
                self.histo = self.slabber.tof_histo.view()
                self.nsweep = self.slabber.nsweep
            return True

    cpdef int read(self,int nsweep=0):
        """
        read nsweep sweeps.
        nsweep > 0 reads from present position in file.
        nsweep == 0 or None reads all remaining sweeps till EOF.
        nsweep <0 reads last nsweep sweeps in file.
        Returns the number of successfuly read sweeps
        """
        cdef int nsweeps_count
        cdef int answ
        cdef tuple sweep
        nsweeps_count = 0
        if nsweep == 0:
            answ = 1
            while  answ :#is not None:
                answ = self.read(1)# FAILS HERE!
            return nsweeps_count
        else:
            if nsweep > 0:
                for i in range(nsweep):
                    sweep = self.det_reader.next_sweep()
                    nsweeps_count += 1
                    if not self._is_ok_sweep(sweep):
                        return nsweeps_count - 1
            else:
                # To read the last sweeps we first need to go to EOF
                self.det_reader.goto_end()
                for i in range(-nsweep):
                    sweep = self.det_reader.prev_sweep()
                    nsweeps_count += 1
                    if not self._is_ok_sweep(sweep):
                        return nsweeps_count - 1
        return nsweeps_count

    def strobe(self,nrepeat=1,nsweep=100):
        """
        Stroboscopic measurement over nrepeat blocks of nsweep sweeps
        Updates self.strobe_seq into {'slab','tof_histo','intensity'}
        where each key leads to a list of corresponding values
        """
        li_intensity = []
        li_slab = []
        li_histo = []
        n_slabs = len(self.slabber.li_tof)+1
        npix = self.slabber.npix
        for repeat in range(nrepeat):
            self.slabber.slab = scipy.zeros((n_slabs,npix[0],npix[1]))
            self.slabber.tof_histo = scipy.zeros((n_slabs))
            for i in range(nsweep):
                sweep = self.det_reader.next_sweep()
                if sweep is not None:
                    cleaned = self.det_reader.clean_sweep(sweep)
                    self.slabber._incr(cleaned)
            try:
                li_slab.append(self.slabber.slab)
                li_histo.append(self.slabber.tof_histo)
                li_intensity.append(self.slabber.slab[0].sum())
            except:
                pass
        self.strobe_seq = {'slab':li_slab,
                           'tof_histo':li_histo,
                           'intensity':li_intensity
                          }

    def follow(self, intervall=5, begin_at_EOF=False):
        """
        reads up to EOF every intervall seconds.
        if begin_at_EOF is True waits for one intervall and then reads
        the data acquired since follow() was called (not since startMeas())
        """
        if begin_at_EOF:
            self.det_reader.goto_end()
            time.sleep(intervall)

        while True:
            print 'I start reading again!'
            self.read()
            print 'Done! I\'ll have a nap for %s seconds.'%intervall
            time.sleep(intervall)

    def analyze(self, nsweep=0, out='full'):
        """
        A convenience function which reads nsweep and returns data only
        (unlike read() which just updates the UI properties)
        out == 'full' returns {'intensity','histo','nsweep'}
        out == 'intensity' returns a Numpy array of intensities
        out == 'histo' returns a list of histogramms
        """
        # input sanity check
        #FIXME

        self.read(nsweep=nsweep)
        if out == 'full':
            out = {'intensity':self.slabber.slab,
                   'histo':self.slabber.tof_histo,
                   'nsweep':self.slabber.nsweep
                  }

        elif out == 'intensity':
            out = self.slabber.slab
        elif out == 'histo':
            out = self.slabber.tof_histo,
        else:
            raise exceptions.InputError

        return out

    def save_data(self,filename,gzip=True):
        """
        save the slab array to a file in [gziped] cPickle format
        """
        fiout = open(filename,'w')
        cPickle.dump(self.slabber.slab,fiout)
        fiout.close()
        if gzip:
            subprocess.call(['gzip',filename])
#       p.save(filename+'.gz',self.slabber.slab[0])

    #def display(self,slicenum=0):
    #    """
    #    Minimalistic display function
    #    """
    #    p.imshow(p.log10(self.slabber.slab[slicenum]+1),interpolation='nearest')
    #    p.colorbar()
    #    p.figure()
    #    p.semilogy(self.slabber.tof_histo + 1)


cdef class DetectorReader:
    """
    Reads a list mode file and returns the 2D slices using
    delay line parameters
    """

    cdef list fname # FIXME enforce it
    cdef list list_mode_file
    cdef int n_devices
    cdef list delay_line_pars # FIXME
    cdef list delay_line
    cdef int timing_accuracy

    def __init__(self,fname=None,reader='LstFileReader_FAST',\
                 delay_line_pars={'name':'X',\
                                  'anode':[0,0],\
                                  'cathodes':[[0,1],[0,2]] ,\
                                  'length':300,\
                                  'tol':4.00,\
                                  'npix':256\
                                 },\
                 timing_accuracy =3\
                ):


        self.fname = list(fname)


        # create a list of ListModeFile readers (one per input file)
        self.list_mode_file = [ListModeFile(fname=fname,reader=reader)
                               for fname in self.fname
                              ]
        self.n_devices = len(self.list_mode_file)

        if type(delay_line_pars) != types.ListType:
            delay_line_pars = [delay_line_pars]
        self.delay_line_pars = delay_line_pars

        # create a list of DelayLine instances
        # (one per dictionary of parameters given as input to __init__)
        self.delay_line = [DelayLine(dlp) for dlp in self.delay_line_pars]
        self.timing_accuracy = timing_accuracy

    def goto_end(self):
        """
        Goes to the last full sweep in lst
        """
        for fi in self.list_mode_file:
            fi.reader.goto_end()

    cdef tuple next_sweep(self):
        """
        Returns a raw sweep in the form [[signals dev0][signal dev1]...].
        Useful for sweep analysis (counter behaviour)
        """
        cdef tuple sweep
        cdef ListModeFile fi

        sweep = tuple([fi.next_sweep() for fi in self.list_mode_file])
        return sweep

    def prev_sweep(self):
        """
        Returns a raw sweep in the form [[signals dev0][signal dev1]...].
        Useful for sweep analysis (counter behaviour)
        """
        sweep = tuple([fi.prev_sweep() for fi in self.list_mode_file])
        return sweep


    cdef dict format_sweep(self,tuple raw_sweep,tuple
li_signals=('anode','cathodes')):
        """
        Returns a dictionary of signals according to the definitions
        given in self.delay_line_pars.
        This output is more human-readable
        """

        cdef dict out_sweep
        cdef dict dlp
        cdef str dl_name
        cdef str signal

        out_sweep = {}
        for dlp in self.delay_line_pars:
            dl_name = dlp['name']
            out_sweep[dl_name] = {}
            for signal in li_signals:

                #print 'tttttt',type(dlp[signal][0])
                #print 'dlp[sig]',type(dlp[signal][0])
                if type(dlp[signal][0]) != types.IntType:
                    try:
                        out_sweep[dl_name][signal] =
tuple([raw_sweep[dlp[signal]
                                                               [terminal]
                                                               [0]
                                                           ]
                                                           [dlp[signal]
                                                               [terminal]
                                                               [1]
                                                           ]
                                                  for terminal in [0,1]
                                                ])
                    except:
                        print 'Signal not available'
                        return None # FIXME dangerous for a tuple???
                #except:
                else:
                    try:
                        out_sweep[dl_name][signal] = raw_sweep[dlp[signal][0]]\
                                                          [dlp[signal][1]]
                    except:
                        print 'Signal not available'
                        return None

                #except: #exceptions.TypeError:

                    ## we got an empty raw_sweep!
                    #print 'we got an empty raw sweep',raw_sweep
                    #return None


        return out_sweep

    cpdef tuple clean_sweep(self, tuple sweep):
        """
        Performs the discri on the sweep and tries to reconstruct missing
        events as much as possible
        """
        cdef dict formated_sweep
        cdef np.ndarray discriminatedX, discriminatedY,evx # FIXME USE
buffer nottation !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!11
        cdef list ev#FIXME!!!!!!!!!!!!evx to tuple??
        cdef np.ndarray index_evy
        formated_sweep = self.format_sweep(sweep)

        if formated_sweep is None:
            return ()

        discriminatedX = self.delay_line[0].do_discri(formated_sweep)
        discriminatedY = self.delay_line[1].do_discri(formated_sweep)
        #print 'discriX',discriminatedX
        #print 'discriY',discriminatedY
        # !!!! FIXME should avoid the search by being cleverer
        ev = []
        for evx in discriminatedX:
            try:
                index_evy = scipy.nonzero(abs(discriminatedY[:,0]-evx[0]) <
                                          self.timing_accuracy
                                         )[0]
                if len(index_evy)!= 0:
                    ev.append(tuple([evx[0],self.delay_line[0]._locate(evx),
                                self.delay_line[1]._locate(
                                                discriminatedY[index_evy][0]
                                                    )
                            ])
                            )# FIXME !!!be sure that there coulld be
only one evy
            except:
                pass
            #    print 'fixme with user defined error npixrange 11'
            #except:
                #print 'ahaaaaa!'
                #pass
        return tuple(ev)



cdef class DelayLine:
    """
    A class describing a delay line (addresses and length) as well
    as providing the discrimination functions
    """
    cdef public dict delay_line_pars # FIXME is this optimal
    cdef bint has_anode #bool fails withdelay_line.c:443: warning:
implicit declaration of function ?PyBool_CheckExact? at compile and
<type 'exceptions.ImportError'>:
/home/jfmoulin/My_Progs/refsans/refsans_cython/read_sweep/C/delay_line_C.so:
undefined symbol: PyBool_CheckExact at runtime
    cdef int has_cathodes
    cdef double length
    cdef double tol
    cdef public tuple window
    cdef list li_sig #FIXME
    cdef int npix
    cdef double npix_2, npix_length, npix_length_2
    cdef int (*locate)(DelayLine, np.ndarray) #FIXME
http://www.mail-archive.com/cython-dev at codespeak.net/msg02700.html)
    cdef bint (*discri)(DelayLine, tuple)
        #------------------------

    def __init__(self,dict delay_line_pars):
        self.delay_line_pars = delay_line_pars
        self.has_anode = self.delay_line_pars.has_key('anode')
        self.has_cathodes = 0
        self.length = self.delay_line_pars['length']
        self.tol = self.delay_line_pars['tol']
        self.window = (self.length - self.tol , self.length + self.tol)
        if self.delay_line_pars.has_key('cathodes'):
            try:
                self.delay_line_pars['cathodes'][1][0]
                self.has_cathodes = 2
            except:
                self.has_cathodes = 1
                self.locate = self.locate_single_terminal
        if self.has_anode and self.has_cathodes == 2:
            self.li_sig = ['anode','cathodes']
            self.locate = self.locate_two_terminals
        self.npix = self.delay_line_pars['npix']
        self.npix_2 = self.npix * .5
        self.npix_length = self.npix  / self.length
        self.npix_length_2 = self.npix_2  / self.length

    cpdef int _locate(self, np.ndarray signals):
        return self.locate(self, signals) # FIXME cython strangeness cdef fails
#even if call is from cython code!

    cdef bint discri_complete(self, tuple signals):
        """
        The easy case where we have exactly one anode and two cathodes signals
        """
        cdef int calc_length
        cdef bint discri_sum

        calc_length = signals[1] + signals[2] - 2 * signals[0]
        discri_sum = calc_length >= self.window[0] and \
                     calc_length <= self.window[1]
        return discri_sum

    cdef bint discri_single_cath(self, tuple signals):
        #FIXME is tuplethe best here???
        cdef int calc_pos
        cdef bint discri_pos

        calc_pos = signals[1] - signals[0]
        discri_pos = calc_pos <= self.window[1]
        return discri_pos

    cpdef np.ndarray do_discri(self,dict sweep):
        cdef list li_discriminated
        cdef int count_anode
        cdef int anode
        cdef tuple li_anode,sig # FIXME NAME??
        cdef bint only_one_ev
        cdef tuple li_cathodes

        self.discri = self.discri_complete
        # extract what is relevant to this delay line only
        sweep = sweep[self.delay_line_pars['name']]
        li_discriminated = []
        for count_anode , anode in enumerate(sweep['anode']):

            li_anode = self.within_delay(sweep['anode'],anode)
            only_one_ev = len(li_anode) == 1
            if self.has_cathodes == 2:
                li_cathodes =
tuple([self.within_delay(sweep['cathodes'][cathcount],
                                                 anode
                                                )
                               for cathcount in [0,1]
                              ])
                only_one_ev = only_one_ev and \
                              len(li_cathodes[0]) == 1 and \
                              len(li_cathodes[1]) == 1
            else:
                li_cathodes = self.within_delay(sweep['cathodes'],anode)
                only_one_ev = only_one_ev and len(li_cathodes) == 1
                self.discri = self.discri_single_cath

            if only_one_ev:
                if self.has_cathodes==2:
                    sig = (li_anode[0],li_cathodes[0][0],li_cathodes[1][0])
                else:
                    sig = (li_anode[0],li_cathodes[0])
                if self.discri(self,sig):# FIXME why must I put self
in the args!?
                    li_discriminated.append(sig)
                #else:
                #    print 'rej',sig

        discriminated = scipy.array(li_discriminated)
        return discriminated

    cdef tuple within_delay(self,tuple li_signals,int anode):
        cdef int tmax
        cdef int ev
        tmax = self.window[1] + anode
        return tuple([ev for ev in li_signals if ev>=anode and ev<=tmax])

    cdef int locate_single_terminal(self,np.ndarray signals):
        """
        Position the event on a self.pix scale (if self.npix is None,
        the return is continuous over [0,1]
        Input should be an array [anode, c1]
        """
        cdef int pos
        pos = int(round((signals[1] - signals[0]) * self.npix_length))
        return pos

    cdef int locate_two_terminals(self,np.ndarray signals):
        """
        Position the event on a self.pix scale (if self.npix  is None,
        the return is continuous over [0,1]
        Input should be an array [anode, c1, c2]
        """
        cdef int pos

        try:
            pos =  int(round(self.npix_2 + \
                       (signals[:,2] - signals[:,1]) * self.npix_length_2)
                      )
        except:
            pos =  int(round(self.npix_2 + \
                       (signals[2] - signals[1]) * self.npix_length_2)
                      )
        return pos

cdef class TofSlabber:
    """A class implementing the TOF histogramming """

    # NOTE:  for performance reasons defined directly only for 2D images
    # do it for 1D counters...

    # FIXME add a check to avoid incrementing severall times for
    # the same sweep?????


    cdef tuple tof_lims
    cdef double resol
    cdef tuple npix
    cdef public int nsweep
    cdef np.ndarray li_tof, li_tof_low, li_tof_high
    cdef public np.ndarray slab, tof_histo



    def __init__(self,tof_lims=None,resol=.1,npix=(256,256)):
        self.tof_lims = tof_lims
        self.resol = resol
        self.npix = npix
        self.nsweep = 0

        if self.tof_lims  is not None:
            self.init_tof_array()

    def init_tof_array(self):
        """
        Initializes the tof histogram
        """
        n_slabs =
round(scipy.log(self.tof_lims[1]/self.tof_lims[0])/scipy.log(1 +
self.resol)) + 1
        #print 'tof_lims/resol (init_tof_array)',self.tof_lims,self.resol
        #print 'n_slabs (init_tof_array)',n_slabs
        self.li_tof = scipy.append(scipy.array([0]),
                                   self.tof_lims[0] *
                                   scipy.logspace(0,
                                                  n_slabs,
                                                  num=n_slabs,
                                                  base=1 + self.resol
                                                 )
                                  )
        #print 'li_tof (init_tof_array)',self.li_tof

        self.li_tof_low = (1 - .5 * self.resol) * self.li_tof
        self.li_tof_high = (1 + .5 * self.resol) * self.li_tof

        self.slab = scipy.zeros((n_slabs,self.npix[0],self.npix[1]))

        self.tof_histo = scipy.zeros((n_slabs))

    def _incr(self,sweep):
        self.incr(sweep)

    cdef void incr(self,tuple sweep) except *:
        """
        Increments the tof histogramm and the different channels with the
        events found in a sweep.
        sweep should be a tuple.
        """


        cdef tuple ev
        cdef int slab_num

        #print 'sweep (input to incr)',sweep
        self.nsweep += 1
        if sweep == () :

            return
        if self.tof_lims  is None:
#            print (s.array(sweep[0]).min() *
.9,s.array(sweep[:0]).max() * 1.15)
            self.tof_lims = (int(scipy.array(sweep)[:,0].min() * .9),\
                             1 + int(scipy.array(sweep)[:,0].max() * 1.5)
                            )
            self.init_tof_array()

        for ev in sweep:
            slab_num = scipy.nonzero(ev[0]>=self.li_tof_low)[0][-1]
            # try to increment the correct tof slice
        #    try:
            #print 'slab_num (incr)', slab_num
            if slab_num > len(self.li_tof)-2:
                print 'Event rejected, tof>tmax'
            else:
                try:
                    self.slab[slab_num][ev[1],ev[2]] += 1
                    # increment the integrated signal
                    self.slab[0][ev[1],ev[2]] += 1
                    # increment the tof histogram
                    self.tof_histo[slab_num] += 1
                except exceptions.IndexError:
                    print 'fixme with user error about range'#,sweep
              #  pass


cdef class ListModeFile:
    """
    Implementation of all operations on list mode files.
    This class provides a level of abstraction to separate from the
    details of the file structure.
    fname: name of the file to be read
    reader: the name of a class defining a reader for this file.
            See LstFileReader_FAST for details of necessary methods.
    """

    cdef str fname
    cdef  LstFileReader_FAST reader# FIXME!!!!

    def __init__(self,fname,reader='LstFileReader_FAST'):
        self.fname = fname
        print '%s(\'%s\')'%(reader,self.fname)
        self.reader = eval('%s(\'%s\')'%(reader,self.fname))

#        self.reader = eval('low_level_reader.%s(\'%s\')'\
#                           %(reader,self.fname)
#                          )#FIXME into something less harmful

    cpdef tuple next_sweep(self):
        """
        Returns the next available full sweep starting from the present
        position in the file
        """
        return self.reader.next_sweep()

    def prev_sweep(self):
        """
        Reads the previous full sweep starting from present pos in file
        """
        return self.reader.prev_sweep()

    def goto_start(self):
        """
        Goes to first sweep in file.
        """
        self.reader.goto_start()

    def goto_end(self):
        """
        Goes to last full sweep in file.
        """
        self.reader.goto_end()

    def nevents(self):
        """
        Returns the number of events in the file (incl start signals)
        """
        return self.reader.nevents()


cdef class LstFileReader_FAST:

    """
    Implement operations on a FAST  list mode file (.lst).
    This is the low level reader
    """
    cdef str fname
    cdef tuple (*do_read_ev)(LstFileReader_FAST)
    cdef file lstfile # FIXME
    cdef list header


    def __init__(self,fname):

        self.fname = fname
        self.goto_start()
        # we define a function pointer to the reader.next in order to be
        # able to change the reading direction easily
        self.do_read_ev = self._next_ev

    def __iter__(self):

        return self

    def goto_start(self):

        """
        Read the header. Goto first event in file.
        """

        try:
            self.lstfile.close()
        except exceptions.AttributeError:
            pass

        self.lstfile = open(self.fname,'rb')

        self.header = []
        while True:
            line = self.lstfile.readline()
            #print line
            if '[DATA]' in line:
                break
            else:
                self.header.append(line)


    def goto_end(self):

        """
        Goto the position of the last fully written event in file.
        """

        # Count the events in file, there are 4 bytes per event...
        nev = self.nevents()
        end_pos = nev * 4
        last_ev_pos = (nev - 1) * 4
        self.lstfile.seek(last_ev_pos,1)

    def nevents(self):
        """
        return the number of events in file (incl starts)
        """
        self.goto_start()
        start_offset = self.lstfile.tell()
        nev = round((getsize(self.fname) - start_offset)/4)
        return nev

    cdef tuple prev(self):
        return self._prev()

    cdef tuple _prev(self):

        """
        read previous event in file, jump back before its position and
        return [channel,value].
        Use to read a file backwards.
        """

        cdef tuple ev

        try:
            self.lstfile.seek(-4,1)
        except exceptions.IOError:
            return None
        ev = self.next_ev()
        self.lstfile.seek(-4,1)
        return ev


    cdef tuple next_ev(self):
        return self._next_ev()

    cdef tuple _next_ev(self):

        """
        Read the next event in file and decode it. Return [chan_number, value]
        """
        cdef int input_chan
        cdef int value
        cdef str st_data
        cdef int data
        cdef int bit30
        cdef int bit31

        input_chan = -1#None
        value =  -1#None
        # cython trick for strings from python (see manual p15))!
#        s = self.lstfile.read(4)
        #print self.lstfile.tell()
        #print 's',type(s),s
        st_data = self.lstfile.read(4)
        #print 'st_data',st_data
        #print data
        try:
            data = struct.unpack('i',st_data)[0]
         #   print 'data', data

        except struct.error:
          #  print 'struct_error!return None'
            return (None,)
        #Reading bits 30 and 31 for input channel:
        #struct.unpack('>i','\x40\x00\x00\x00')[0] != 0)
        bit30 = int(data & 1073741824!=0)
        #struct.unpack('>i','\x80\x00\x00\x00')[0] !=0)
        bit31 = int(data & -2147483648!=0)
        #print 'bits', bit30, bit31
        input_chan = 1 * bit30 + 2 * bit31
#        key = str(bit30) + str(bit31)
        #print 'key',key
        #print bit30, bit31, 'event @',dict_input[key]

        #value of int(bits0_29): turn the bits 30 and 31
        # off and convert the value
        #struct.unpack('>i','\x3f\xff\xff\xff')[0]
        value = 32 + data & 1073741823
        #print value
        #vals[dict_input[key]-1].append(value)
        #input_chan = LstFileReader_FAST.dict_input[key]
        #print 'chan,val',input_chan, value
        return (input_chan, value)

    cdef tuple next_sweep(self):
        return self._next_sweep()

    cdef tuple _next_sweep(self):



        """
        Returns the next available full sweep starting from the present
        position in the file (implement premature eof!!!) FIXME
        """
        cdef list channels#, last_ev
        cdef tuple last_ev
        cdef list chan


        channels = []
        last_ev = (None, None)
      #  last_ev = [[],[]]
        # if necessary look for start event
        try:
            while last_ev[1] != 32:
                last_ev = self.do_read_ev(self)# FIXME
         #       print 'last_ev',last_ev
        except exceptions.TypeError:
            return None
        last_ev = (None,None)
        # then accumulate data up to next start
        while last_ev[1] != 32:
            last_ev = self.do_read_ev(self) # FIXME
            # in case of EOF before START signal,
            # we return None (uncomplete sweep)
            if last_ev[0] is None : return None
            # if we got a start signal we are done
            if last_ev[1] == 32:
                return tuple([tuple(chan) for chan in channels])
            else:
                # auto adjusting the number of channels
                try:
                    channels[last_ev[0]].append(last_ev[1])
                except:
                    channels.extend([[] for count in
                                     range(last_ev[0]-len(channels)+1)
                                    ]
                                   )
                    channels[last_ev[0]].append(last_ev[1])

        return tuple([tuple(chan) for chan in channels])

    cdef tuple prev_sweep(self):
        """
        Reads previous full sweep and goes back to its starting pos.
        Use to read backwards.
        """
        cdef tuple channels
        self.do_read_ev = self.prev
        channels = self.next_sweep()
        if channels is not None:
            # we read backwards... so we have to reorder the events
            # in increasing time
            channels = [chan[::-1] for chan in channels]
            # restore the normal reading direction of the file
        self.do_read_ev = self.next_ev

        return channels


    def progress(self):
        """
        progress() returns (Mbytes_read,file_progress(%))
        """
        return(self.lstfile.tell()/(1024*1024))#,fprogress(self.lstfile))






On Tue, Apr 21, 2009 at 10:23 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 21, 2009, at 12:45 AM, Jean-Francois Moulin wrote:
>
>> Hi,
>>
>> first of all, thanks for the replies.
>>
>> As for the first answer by Lisandro, I do not understand then why my
>> first code fragment at all works...
>> The only difference is that the function called has two params, one
>> instance of one of my classes and a numpy array, while the faulty
>> piece of code should be called on an instance
>> of one of my classes only...
>>
>> As for the second, from Robert, now, very well... how do I specify a
>> value for this implicit parameter then!? It is a bint I guess... I
>> tried to set it to 1 and got new error messages:
>> delay_line.pyx:961:30: Cannot assign type 'tuple (LstFileReader_FAST,
>> int __pyx_skip_dispatch, struct
>> __pyx_opt_args_12delay_line_C_18LstFileReader_FAST_next_ev
>> *__pyx_optional_args)' to 'tuple (*)(LstFileReader_FAST)'
>
> You can't. I agree the errors are more cryptic than they should be,
> but C functions have no concept of runtime dispatching or default
> arguments, so it doesn't work to cast such a function to a C function
> (well, maybe one could hack it, but it'd be ugly and fragile).
>
>> I also tried to use cdef in all cases, I get no compile time error but
>> at runtime my object LstFileReader_FAST appears no longer to be
>> defined!? I do not understand because all the calls are made from
>> within Cython code... (so that I expect the cpdef is not really
>> needed)
>
> I am guessing you're doing something like
>
> a. LstFileReader_FAST(...)
>
> and it's complaining that "a has not attribute 'LstFileReader_FAST'".
> Usually this indicates that a is not typed correctly (e.g. it thinks
> a is an ordinary object). You can run cython with the -a option to
> easily see this kind of thing.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Tue Apr 21 11:23:45 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 11:23:45 +0200 (CEST)
Subject: [Cython] Path towards Python completeness
In-Reply-To: <80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>
References: <49ED5CE8.6090708@behnel.de>
	<80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>
Message-ID: <380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Apr 20, 2009, at 10:43 PM, Stefan Behnel wrote:
>> Closure support will actually open up a can of worms,
>> but a friendly one, as it will lay out the path towards lambda
>> support and nested functions/classes. Especially support for
>> generators seems to be an extremely worthy goal, but in the current
>> state of Cython, it just cannot be done.
>
> All of the above are syntactic sugar for special closures.

Sure, but I would say that closure support is more or less finished when
nested functions work. The rest are additional steps to add more syntax
features on top of that.

For example, generator support needs code that supports continuing
function execution. Lambda functions need a transform into nested
functions. That's additional work, but nothing that's worth caring about
right now.

The only way to get this working is to set achievable partial goals that
we (or others) can build on. Even nested classes are out of scope for the
first milestone, IMHO.


> This has been bugging me for a *long* time. I'll pull out and go
> though the random scattered stuff I've done over the past year
> (something else more urgent has always come up everytime I've set out
> to do this) and put it up within the week.

Great!


> Should I make a closure branch on the site?

You'd create it from cython-unstable, I assume. Given that the current
development works reasonably well with two branches (pull-direction
unstable<-devel), I doubt that a third "closures" branch (pull-direction
closures<-unstable) would add any major additional overhead, while still
allowing us to finish up and release 0.12 from cython-unstable when it's
time.

I think that's a good idea. Should we make (basic?) closure support an
official 0.13 goal?

Stefan


From stefan_ml at behnel.de  Tue Apr 21 11:39:54 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 11:39:54 +0200 (CEST)
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
Message-ID: <7e35120d0d391d2840926cf8e034e41b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Jean-Francois Moulin wrote:
> Okay, you asked for it..., here is the full code

I didn't really ask for the "full" code. It's rather uncommon to post code
of that size to a mailing list, especially as (fragile) part of the
message body. Next time, please consider pasting it into a web site
(pastebin or whatever), or at least *attach* it to an e-mail, so that
people can safely safe it to a file.

In this specific case, the traceback would have been enough, or at least a
hint that you are using eval():

   delay_line_C.ListModeFile.__init__ (delay_line.c:7367)
       self.reader = eval('%s(\'%s\')'%(reader,self.fname))
     File "<string>", line 1, in <module>
   NameError: name 'LstFileReader_FAST' is not defined

Cython does not currently support globals(), which is used by eval()
internally. Any reason why you pass the class name as parameter instead of
the class itself?

Stefan


From robertwb at math.washington.edu  Tue Apr 21 11:39:54 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 02:39:54 -0700
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
Message-ID: <90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>

On Apr 21, 2009, at 2:19 AM, Jean-Francois Moulin wrote:

> Hi!
> thanks for the answers!
>
> Here is the runtime error message:
>
> python read_sweep_C.py
> LstFileReader_FAST('/home/jfmoulin/Lab/Experiments/refsans/ 
> 07_11_PMB/BPSi-JFM-B1-087.lst')
> Traceback (most recent call last):
>   File "read_sweep_C.py", line 278, in <module>
>     ui = exple2()
>   File "read_sweep_C.py", line 225, in exple2
>     tof_resol=.05
>   File "delay_line.pyx", line 103, in delay_line_C.UI.__init__
> (delay_line.c:1212)
>     self.det_reader = DetectorReader(fname=self.fname,reader=reader,
>   File "delay_line.pyx", line 328, in
> delay_line_C.DetectorReader.__init__ (delay_line.c:3445)
>     self.list_mode_file = [ListModeFile(fname=fname,reader=reader)
>   File "delay_line.pyx", line 719, in
> delay_line_C.ListModeFile.__init__ (delay_line.c:7367)
>     self.reader = eval('%s(\'%s\')'%(reader,self.fname))
>   File "<string>", line 1, in <module>
> NameError: name 'LstFileReader_FAST' is not defined
>
> So it seems not to be a type error for an object which would have
> a LstFileReader_FAST property or method, right ?

Well, it was a blind guess not having the code :).

> Okay, you asked for it..., here is the full code (yes it is still very
> messy and full of FIXMEs but the python version does run...albeit
> sooooo slow, hence Cython)

Not sure exactly what the bug is here, but it does look more like a  
bug. Maybe you could boil this down to a isolated couple of lines  
that produces the error?

- Robert


From robertwb at math.washington.edu  Tue Apr 21 11:40:58 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 02:40:58 -0700
Subject: [Cython] Path towards Python completeness
In-Reply-To: <380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49ED5CE8.6090708@behnel.de>
	<80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>
	<380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <85093F3A-89F0-4324-9156-9DADE52C8B95@math.washington.edu>

On Apr 21, 2009, at 2:23 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Apr 20, 2009, at 10:43 PM, Stefan Behnel wrote:
>>> Closure support will actually open up a can of worms,
>>> but a friendly one, as it will lay out the path towards lambda
>>> support and nested functions/classes. Especially support for
>>> generators seems to be an extremely worthy goal, but in the current
>>> state of Cython, it just cannot be done.
>>
>> All of the above are syntactic sugar for special closures.
>
> Sure, but I would say that closure support is more or less finished  
> when
> nested functions work. The rest are additional steps to add more  
> syntax
> features on top of that.
>
> For example, generator support needs code that supports continuing
> function execution. Lambda functions need a transform into nested
> functions. That's additional work, but nothing that's worth caring  
> about
> right now.
>
> The only way to get this working is to set achievable partial goals  
> that
> we (or others) can build on. Even nested classes are out of scope  
> for the
> first milestone, IMHO.

+1

>
>
>> This has been bugging me for a *long* time. I'll pull out and go
>> though the random scattered stuff I've done over the past year
>> (something else more urgent has always come up everytime I've set out
>> to do this) and put it up within the week.
>
> Great!
>
>
>> Should I make a closure branch on the site?
>
> You'd create it from cython-unstable, I assume. Given that the current
> development works reasonably well with two branches (pull-direction
> unstable<-devel), I doubt that a third "closures" branch (pull- 
> direction
> closures<-unstable) would add any major additional overhead, while  
> still
> allowing us to finish up and release 0.12 from cython-unstable when  
> it's
> time.
>
> I think that's a good idea. Should we make (basic?) closure support an
> official 0.13 goal?

Yep, sounds like a good plan.

- Robert


From jf.moulin at gmail.com  Tue Apr 21 12:06:18 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 21 Apr 2009 12:06:18 +0200
Subject: [Cython] help with function pointer
In-Reply-To: <90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
Message-ID: <af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>

Very sorry for this netiquette mistake... was too impatient I guess :0(

I will try to produce something shorter as requested!

As for the use of eval and the name of the class, it comes straight of
my python version were I used to import a module which contained the
class definition. So the code was self adapting in each
new case...
The idea was that this part of the code, which is a low-level reader
for some binary file could be changed to accomodate different vendor
versions or types of input files.
Maybe that is a stupid way of doing...
So stupid question now...: can I use a pointer to a class?

From stefan_ml at behnel.de  Tue Apr 21 13:16:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 13:16:31 +0200 (CEST)
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
Message-ID: <b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Jean-Francois Moulin wrote:
> As for the use of eval and the name of the class, it comes straight of
> my python version were I used to import a module which contained the
> class definition. So the code was self adapting in each
> new case...

Well, eval() is not a good idea even in Python, as it requires you to
convert the parameters to a string first and then parse them again, which
may break things if they cannot be property escaped (see "SQL injection" &
friends).


> The idea was that this part of the code, which is a low-level reader
> for some binary file could be changed to accomodate different vendor
> versions or types of input files.
> Maybe that is a stupid way of doing...
> So stupid question now...: can I use a pointer to a class?

Sure, a class is a Python object. Just do what you'd do in Python: pass
the object and call it.

Stefan


From stefan_ml at behnel.de  Tue Apr 21 13:16:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 13:16:31 +0200 (CEST)
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
Message-ID: <b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Jean-Francois Moulin wrote:
> As for the use of eval and the name of the class, it comes straight of
> my python version were I used to import a module which contained the
> class definition. So the code was self adapting in each
> new case...

Well, eval() is not a good idea even in Python, as it requires you to
convert the parameters to a string first and then parse them again, which
may break things if they cannot be property escaped (see "SQL injection" &
friends).


> The idea was that this part of the code, which is a low-level reader
> for some binary file could be changed to accomodate different vendor
> versions or types of input files.
> Maybe that is a stupid way of doing...
> So stupid question now...: can I use a pointer to a class?

Sure, a class is a Python object. Just do what you'd do in Python: pass
the object and call it.

Stefan


From dagss at student.matnat.uio.no  Tue Apr 21 14:04:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 21 Apr 2009 14:04:56 +0200
Subject: [Cython] Fortran GSoC accepted
Message-ID: <49EDB668.7030502@student.matnat.uio.no>

I'm very happy to report that Kurt Smith has been accepted for a Google 
Summer of Code project for working on Cython/Fortran integration!

http://wiki.cython.org/kwsmith/soc09

I'm his mentor for the project, and I start this thread to discuss how 
the GSoC should be arranged. (I suppose part of this could apply to the 
other accepted project as well:

http://socghop.appspot.com/student_project/show/google/gsoc2009/python/t124024629578

).

The project is roughly going to be divided in two parts:
  (A) Use the f2py parser and output code for a Cython binding
  (B) Improve Cython in some areas (especially buffers) so that the 
bindings generated in a) can be much less messy and more convenient to use.

Some things:

1) Where should the mentoring take place? Opinions on how much should 
happen on this list and how much should be done in private mail? 
Obviously language decisions etc. must end up here in the end, but what 
about day-to-day mentoring etc.?

One possibility is setting up a seperate Google group just for this 
project, if it turns out that it would be too high traffic for the 
mailing list but other people would like to follow the project as well.

2) Code and patch workflow.

On the (A) side, the status with f2py is that the author started a full 
rewrite (f2py 3G) which hasn't seen work for half a year 
(http://f2py.org/). So anything can happen here and we have to get back 
to how it's going to be done and where it should live. A relevant 
question though is whether we would like to ship the tool in (A) as part 
of Cython or have it as an independent download. (I don't expect it to 
add anything significant to download size, but Python core inclusion 
would be something to consider for or against here.)

When it comes to (B), the obvious thing to do is for Kurt to pass the 
code to me for review and then merge it as often as we can with either 
-devel or -unstable (not sure which yet, possibly both).

I'm wondering though whether perhaps Kurt could just get commit rights 
for the repos, under the strict understanding that he doesn't push 
anything which is not reviewed (for now at least). That could take some 
day-to-day merges off my back and make Kurt responsible for doing stable 
merges and think about the code integration schedule right from the start.

We could then use e.g. codereview.appspot.com for doing the code 
reviews; I would then simply give thumbs up and Kurt would do the push.

3) Feedback

There should be some kind of feedback mechanism; I expect to hear from 
Kurt at least weekly over the summer unless otherwise is arranged. I 
expect the ticket system will serve as the status update, and hope Kurt 
will make his progress visible through the ticket system (we can use the 
"fortran" tag on all GSoC related tickets for instance).

I guess (A) would perhaps use a different tracker than (B), we need to 
get back to that. We could set something up at e.g. Google Code if the 
rest of the Cython team wouldn't like (A)-related tickets cluttering up 
the trac. We could use a seperate milestone though...

Some mentors require students to keep a blog about what they are doing. 
Unless the Python Software Foundation makes this a requirement I'm going 
to leave this one up to Kurt; if the status is dutifully reported via 
tickets I'll have what I need personally, though a blog would be 
interesting as well.

Dag Sverre

From dagss at student.matnat.uio.no  Tue Apr 21 14:18:07 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 21 Apr 2009 14:18:07 +0200
Subject: [Cython] Path towards Python completeness
In-Reply-To: <49ED5CE8.6090708@behnel.de>
References: <49ED5CE8.6090708@behnel.de>
Message-ID: <49EDB97F.4030004@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> I just read the release announcement of PyPy's 1.1 beta:
> 
> http://morepypy.blogspot.com/2009/04/beta-for-110-released.html
> 
> According to their benchmarks, they are now less than 2x slower than
> CPython in most cases and in some cases even 20% faster. Cython is
> currently about 30% faster than CPython for plain Python code, but that
> only concerns the "Python code that Cython supports", which is not at all
> comparable to what PyPy supports.
> 
> My feeling is that the goal to make CPython support "most of Python" is
> getting more important simply because PyPy exists. There is not that much
> that is missing, but the sad thing is that the main part that *is* missing
> is a very important part of the language: closures. 

I'm very happy that work on closures will be resumed and this is simply 
a digression:

I don't think we have to think of ourselves as in direct competition 
with PyPy (and/or Unladen Swallow). Personally I'm more than happy to 
use pure Python rather than Cython for the kind of things that can be 
sped up by those projects anyway (makes for a shorter compile/run cycle).

What I think sets Cython apart is that we add a superset to the Python 
language for adding types; which are simply needed in some situations. I 
think that is where we "compete".

When PyPy runs as fast as CPython, perhaps PyPy can start supporting the 
Cython type annotations (at least the pure Python mode ones); Jython 
could support the same annotations in time, and so on.

I.e., I see Cython as the language and philosophy at least as important 
as the speed we can run pure Python code at vs. PyPy :-)

-- 
Dag Sverre

From robin at reportlab.com  Tue Apr 21 14:54:04 2009
From: robin at reportlab.com (Robin Becker)
Date: Tue, 21 Apr 2009 13:54:04 +0100
Subject: [Cython] download for 0.11.1
Message-ID: <49EDC1EC.7030703@chamonix.reportlab.co.uk>

Is it intentional that the zip file contains the .hg folder. I see 5.365Mb in 
total of which 3.193Mb appeasr to be in the .hg folder?
-- 
Robin Becker

From ilmarw at simula.no  Tue Apr 21 14:57:20 2009
From: ilmarw at simula.no (Ilmar Wilbers)
Date: Tue, 21 Apr 2009 14:57:20 +0200
Subject: [Cython] download for 0.11.1
In-Reply-To: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
Message-ID: <49EDC2B0.3060700@simula.no>

Discussed recently:
http://codespeak.net/pipermail/cython-dev/2009-April/004883.html

ilmar

Robin Becker wrote:
> Is it intentional that the zip file contains the .hg folder. I see 5.365Mb in 
> total of which 3.193Mb appeasr to be in the .hg folder?
>   

From vic.kelson at gmail.com  Tue Apr 21 15:07:10 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Tue, 21 Apr 2009 09:07:10 -0400
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <49EDB668.7030502@student.matnat.uio.no>
References: <49EDB668.7030502@student.matnat.uio.no>
Message-ID: <d5e3120f0904210607tb699866q5202a115f691926b@mail.gmail.com>

Congratulations!

This will be very useful in my research work. I'll be happy to assist with
testing, if that would be helpful.

Best of luck,
Vic

On Tue, Apr 21, 2009 at 8:04 AM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> I'm very happy to report that Kurt Smith has been accepted for a Google
> Summer of Code project for working on Cython/Fortran integration!
>
> http://wiki.cython.org/kwsmith/soc09
>
> I'm his mentor for the project, and I start this thread to discuss how
> the GSoC should be arranged. (I suppose part of this could apply to the
> other accepted project as well:
>
>
> http://socghop.appspot.com/student_project/show/google/gsoc2009/python/t124024629578
>
> ).
>
> The project is roughly going to be divided in two parts:
>  (A) Use the f2py parser and output code for a Cython binding
>  (B) Improve Cython in some areas (especially buffers) so that the
> bindings generated in a) can be much less messy and more convenient to use.
>
> Some things:
>
> 1) Where should the mentoring take place? Opinions on how much should
> happen on this list and how much should be done in private mail?
> Obviously language decisions etc. must end up here in the end, but what
> about day-to-day mentoring etc.?
>
> One possibility is setting up a seperate Google group just for this
> project, if it turns out that it would be too high traffic for the
> mailing list but other people would like to follow the project as well.
>
> 2) Code and patch workflow.
>
> On the (A) side, the status with f2py is that the author started a full
> rewrite (f2py 3G) which hasn't seen work for half a year
> (http://f2py.org/). So anything can happen here and we have to get back
> to how it's going to be done and where it should live. A relevant
> question though is whether we would like to ship the tool in (A) as part
> of Cython or have it as an independent download. (I don't expect it to
> add anything significant to download size, but Python core inclusion
> would be something to consider for or against here.)
>
> When it comes to (B), the obvious thing to do is for Kurt to pass the
> code to me for review and then merge it as often as we can with either
> -devel or -unstable (not sure which yet, possibly both).
>
> I'm wondering though whether perhaps Kurt could just get commit rights
> for the repos, under the strict understanding that he doesn't push
> anything which is not reviewed (for now at least). That could take some
> day-to-day merges off my back and make Kurt responsible for doing stable
> merges and think about the code integration schedule right from the start.
>
> We could then use e.g. codereview.appspot.com for doing the code
> reviews; I would then simply give thumbs up and Kurt would do the push.
>
> 3) Feedback
>
> There should be some kind of feedback mechanism; I expect to hear from
> Kurt at least weekly over the summer unless otherwise is arranged. I
> expect the ticket system will serve as the status update, and hope Kurt
> will make his progress visible through the ticket system (we can use the
> "fortran" tag on all GSoC related tickets for instance).
>
> I guess (A) would perhaps use a different tracker than (B), we need to
> get back to that. We could set something up at e.g. Google Code if the
> rest of the Cython team wouldn't like (A)-related tickets cluttering up
> the trac. We could use a seperate milestone though...
>
> Some mentors require students to keep a blog about what they are doing.
> Unless the Python Software Foundation makes this a requirement I'm going
> to leave this one up to Kurt; if the status is dutifully reported via
> tickets I'll have what I need personally, though a blog would be
> interesting as well.
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Vic Kelson
vic.kelson at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090421/f7783b82/attachment.htm 

From robin at reportlab.com  Tue Apr 21 15:11:53 2009
From: robin at reportlab.com (Robin Becker)
Date: Tue, 21 Apr 2009 14:11:53 +0100
Subject: [Cython] download for 0.11.1
In-Reply-To: <49EDC2B0.3060700@simula.no>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
	<49EDC2B0.3060700@simula.no>
Message-ID: <49EDC619.5000602@chamonix.reportlab.co.uk>

Ilmar Wilbers wrote:
> Discussed recently:
> http://codespeak.net/pipermail/cython-dev/2009-April/004883.html
> 
> ilmar
> 
> Robin Becker wrote:
>> Is it intentional that the zip file contains the .hg folder. I see 5.365Mb in 
>> total of which 3.193Mb appeasr to be in the .hg folder?
>>   
>......

so just laziness then :)
-- 
Robin Becker

From dalcinl at gmail.com  Tue Apr 21 15:17:36 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 21 Apr 2009 10:17:36 -0300
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
Message-ID: <e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>

Robert, your main() implementation needs more work IMHO.

1) For Py2, you need to if(PyErr_Occurred())
PyErr_WriteUnraisable(PyErr_Occurred()).

2) In Py3, you have to save the return of PyInit_<modname>, if NULL,
report error (WriteUnraisable again?), if not, decref it.

3) In case of errors, process return value should match what core
CPython returns en case of unhandled execptions.

3) However, a SystemExit exceptions should need some special
consideration, as you should extract the process return value from it,
as it could be 0 (zero).







On Tue, Apr 21, 2009 at 6:16 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 20, 2009, at 11:48 AM, Robert Bradshaw wrote:
>
>> On Apr 19, 2009, at 7:46 AM, Stefan Behnel wrote:
>>
>>> Mark Lodato wrote:
>>>> you could embed the interpreter and statically link your module.
>>>> [...]
>>>> ----------------------------------- main.c
>>>> ---------------------------
>>>> #include <Python.h>
>>>>
>>>> // For each Cython module you want to embed, you must declare an
>>>> // init<module> function, like so:
>>>> PyMODINIT_FUNC initmylib(void);
>>>>
>>>> int
>>>> main(int argc, char *argv[])
>>>> {
>>>> ? ? // The first step is to set up the Python interpreter:
>>>> ? ? Py_Initialize();
>>>> ? ? PySys_SetArgv(argc, argv);
>>>>
>>>> ? ? // Next, we need to tell Python that our module exists. ?Call
>>>> each
>>>> ? ? // of the functions you declared above.
>>>> ? ? initmylib();
>>>>
>>>> ? ? // Now do some Python stuff. ?The easiest thing to do is to give
>>>> ? ? // the interpreter a string of Python code that imports your
>>>> ? ? // module and calls it.
>>>> ? ? PyRun_SimpleString("from mylib import main\n"
>>>> ? ? ? ? ? ? ? ? ? ? ? ?"main()\n");
>>>>
>>>> ? ? // When we're done, tell Python to clean up.
>>>> ? ? Py_Finalize();
>>>> ? ? return 0;
>>>> }
>>>> --------------------------------------------------------------------
>>>> -
>>>> -
>>>
>>> I wonder if it makes sense to add support for this to Cython. You
>>> could
>>> have a command line option that adds a suitable main() function
>>> that runs
>>> the module code in an embedded CPython interpreter (and that does
>>> the setup
>>> correctly for Py2 and Py3).
>>
>> +1, I think that's an excellent idea. http://trac.cython.org/
>> cython_trac/ticket/289
>
> There's an --embed command line option now, which generates an
> appropriate main() method. One then compiles the resulting C file
> into an executable (which still needs to be linked against the Python
> library).
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Apr 21 15:35:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 21 Apr 2009 15:35:06 +0200
Subject: [Cython] download for 0.11.1
In-Reply-To: <49EDC619.5000602@chamonix.reportlab.co.uk>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>	<49EDC2B0.3060700@simula.no>
	<49EDC619.5000602@chamonix.reportlab.co.uk>
Message-ID: <49EDCB8A.6040705@student.matnat.uio.no>

Robin Becker wrote:
> Ilmar Wilbers wrote:
>> Discussed recently:
>> http://codespeak.net/pipermail/cython-dev/2009-April/004883.html
>>
>> ilmar
>>
>> Robin Becker wrote:
>>> Is it intentional that the zip file contains the .hg folder. I see 5.365Mb in 
>>> total of which 3.193Mb appeasr to be in the .hg folder?
>>>   
>> ......
> 
> so just laziness then :)

No, quote:

 > An argument in favor is that end-users can very easily contribute a
 > bugfix. If users suddenly encounter a bug they can fix then the fix
 > they
 > do is very likely to happen in the downloaded release directory, and
 > having done that then it's a lot less effort to submit a patch if
 > it's a
 > working repository.
 >
 > Especially with Cython, whose end-users are all developers, this is a
 > nice idea.

This is exactly why we do it. Sage ships this way and I think it's
been a significant factor in attracting developers.


-- 
Dag Sverre

From jf.moulin at gmail.com  Tue Apr 21 15:33:41 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 21 Apr 2009 15:33:41 +0200
Subject: [Cython] help with function pointer
In-Reply-To: <b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
	<b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>

Great! Just tried it and it works fine!
I gain something like a tenfold factor in speed with respect to my
initial python code.
>From what I read I expect I should still gain quite a lot by
optimizing the calls to the Numpy functions.
Namely using the buffer notation defining the type of array I am using.
A question arises, should I also apply this notation in the definition
of my pointers? and if yes, how?

so, I presently have statements like:
cdef int (*locate)(DelayLine, np.ndarray)

and would write something like:
 cdef int (*locate)(DelayLine, np.ndarray[DTYPE_t,ndim=2])

but compilation fails saying that a closing square bracket is missing
(])... in the description of the notation in the docs, one sees that
the name of the numpy array directly follows the
description... How to proceed?

From robin at reportlab.com  Tue Apr 21 15:48:00 2009
From: robin at reportlab.com (Robin Becker)
Date: Tue, 21 Apr 2009 14:48:00 +0100
Subject: [Cython] inherit from list
Message-ID: <49EDCE90.2050604@chamonix.reportlab.co.uk>

I have some older code that used to run with pyrex without complaint, but now 
the following

cdef extern from "listobject.h":
	ctypedef class __builtin__.list [object PyListObject]:
		pass


causes a warning
> warning: C:\code\users\robin\_tex_wrap.pyx:144:1: list already a builtin Cython type


however, I cannot remove the above code as that causes an error at my class 
declaration



> cdef class ObjectList(list):


what's the correct cython way to inherit from a list or other builtin? I have 
looked in the extension_types section, but am no wiser.
--
Robin Becker

From dagss at student.matnat.uio.no  Tue Apr 21 16:23:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 21 Apr 2009 16:23:18 +0200 (CEST)
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
	<b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>
Message-ID: <d416dd90cdcf283d0aae347b41e40d28.squirrel@webmail.uio.no>

Jean-Francois Moulin wrote:
> Great! Just tried it and it works fine!
> I gain something like a tenfold factor in speed with respect to my
> initial python code.
>>From what I read I expect I should still gain quite a lot by
> optimizing the calls to the Numpy functions.
> Namely using the buffer notation defining the type of array I am using.
> A question arises, should I also apply this notation in the definition
> of my pointers? and if yes, how?
>
> so, I presently have statements like:
> cdef int (*locate)(DelayLine, np.ndarray)
>
> and would write something like:
>  cdef int (*locate)(DelayLine, np.ndarray[DTYPE_t,ndim=2])

This is simply not supported with cdef functions. You need to do

cdef int (*locate)(DelayLine, np.ndarray)

and then inside the implementation of the function you assign to it, you do

cdef int (*locate)(DelayLine a, np.ndarray b)
    cdef np.ndarray[DTYPE_t, ndim=2] b_buf = b
    .... access b through b_buf ...

Dag Sverre


From stefan_ml at behnel.de  Tue Apr 21 18:26:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 18:26:17 +0200 (CEST)
Subject: [Cython] Path towards Python completeness
In-Reply-To: <49EDB97F.4030004@student.matnat.uio.no>
References: <49ED5CE8.6090708@behnel.de>
	<49EDB97F.4030004@student.matnat.uio.no>
Message-ID: <ab1e0fdf3d34a12989a138890c192bfd.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> I don't think we have to think of ourselves as in direct competition
> with PyPy (and/or Unladen Swallow). Personally I'm more than happy to
> use pure Python rather than Cython for the kind of things that can be
> sped up by those projects anyway (makes for a shorter compile/run cycle).

I do actually see PyPy as a direct competitor of Cython (much more than
Jython and IronPython, BTW), as both projects aim to compile Python code
to fast native code. Currently, we are faster and they are more feature
complete. But if we can have the cake and eat it, too...


> What I think sets Cython apart is that we add a superset to the Python
> language for adding types; which are simply needed in some situations. I
> think that is where we "compete".

That's rather where we are beyond competition.


> When PyPy runs as fast as CPython, perhaps PyPy can start supporting the
> Cython type annotations (at least the pure Python mode ones); Jython
> could support the same annotations in time, and so on.

Let's wait and see, but I doubt that there will be a major interest in
doing that. I actually find it more important for Cython to gain type
inference mechanisms that lift the requirement for defining types in the
first place, at least for standard use cases.

Stefan


From stefan_ml at behnel.de  Tue Apr 21 18:32:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 18:32:30 +0200 (CEST)
Subject: [Cython] inherit from list
In-Reply-To: <49EDCE90.2050604@chamonix.reportlab.co.uk>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
Message-ID: <36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robin Becker wrote:
> I have some older code that used to run with pyrex without complaint, but
> now the following
>
> cdef extern from "listobject.h":
> 	ctypedef class __builtin__.list [object PyListObject]:
> 		pass
>
> causes a warning
>> warning: C:\code\users\robin\_tex_wrap.pyx:144:1: list already a builtin
>> Cython type

Yes, that warning is currently a bit misleading in this specific case, as
there is not yet a way to achieve inheritance of builtin types without the
above declaration. It's meant more for code that uses the declared type in
other places than supertypes, where Cython now handles these types better
on its own.

This is what ticket #258 is supposed to mean.

http://trac.cython.org/cython_trac/ticket/258

See also ticket #170:

http://trac.cython.org/cython_trac/ticket/170

If you use the "list" type only for subtypes, just ignore the warning. It
will become an error once Cython can deal with this case directly, so you
will notice when it's time to remove the declaration.

Stefan


From robin at reportlab.com  Tue Apr 21 18:40:16 2009
From: robin at reportlab.com (Robin Becker)
Date: Tue, 21 Apr 2009 17:40:16 +0100
Subject: [Cython] inherit from list
In-Reply-To: <36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49EDF6F0.504@chamonix.reportlab.co.uk>

.........
> 
> Yes, that warning is currently a bit misleading in this specific case, as
> there is not yet a way to achieve inheritance of builtin types without the
> above declaration. It's meant more for code that uses the declared type in
> other places than supertypes, where Cython now handles these types better
> on its own.
> 
> This is what ticket #258 is supposed to mean.
> 
> http://trac.cython.org/cython_trac/ticket/258
> 
> See also ticket #170:
> 
> http://trac.cython.org/cython_trac/ticket/170
> 
> If you use the "list" type only for subtypes, just ignore the warning. It
> will become an error once Cython can deal with this case directly, so you
> will notice when it's time to remove the declaration.
> 
> Stefan
......

OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
from builtin types to be handled automatically as it is in python where list is 
both a constructor and a type?
-- 
Robin Becker

From stefan_ml at behnel.de  Tue Apr 21 19:01:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 19:01:27 +0200 (CEST)
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <49EDB668.7030502@student.matnat.uio.no>
References: <49EDB668.7030502@student.matnat.uio.no>
Message-ID: <d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> I'm very happy to report that Kurt Smith has been accepted for a Google
> Summer of Code project for working on Cython/Fortran integration!

Yep, congrats!


> I'm his mentor for the project, and I start this thread to discuss how
> the GSoC should be arranged. (I suppose part of this could apply to the
> other accepted project as well:
>
> http://socghop.appspot.com/student_project/show/google/gsoc2009/python/t124024629578

Congrats on that one, too!

Good luck to our GSoCers.


> 1) Where should the mentoring take place? Opinions on how much should
> happen on this list and how much should be done in private mail?
> Obviously language decisions etc. must end up here in the end, but what
> about day-to-day mentoring etc.?

Depends. Trivial/basic things may go into private mail, but I would prefer
having most of the discussions on the list, where others can
help/find/learn, too. This is an open source project, so things should
happen in the open.


> One possibility is setting up a seperate Google group just for this
> project, if it turns out that it would be too high traffic for the
> mailing list but other people would like to follow the project as well.

Let's see about that when we notice that it gets too much traffic.

I would lean more towards a separate cython-users list than a separate
sub-dev list. (I have ignored threads before, so I know I'll cope ;)


> 2) Code and patch workflow.
>
> On the (A) side, the status with f2py is that the author started a full
> rewrite (f2py 3G) which hasn't seen work for half a year
> (http://f2py.org/). So anything can happen here and we have to get back
> to how it's going to be done and where it should live.

Sounds like the right thing to ask the author first.


> A relevant
> question though is whether we would like to ship the tool in (A) as part
> of Cython or have it as an independent download. (I don't expect it to
> add anything significant to download size, but Python core inclusion
> would be something to consider for or against here.)

Let's decide that when we can take a look at it.


> When it comes to (B), the obvious thing to do is for Kurt to pass the
> code to me for review and then merge it as often as we can with either
> -devel or -unstable (not sure which yet, possibly both).

Definitely not the current cython-devel, but without further detail (that
may become available once the project has left the initial planning
phase), I can't tell if it makes more sense to add another branch. Given
the time-frame of the GSoC, maybe even cython-unstable is too close to
release to add major new, unfinished features. Parts may go in earlier,
obviously, so a branch from which we can import stuff may make sense.

IMHO, too early to decide.


> I'm wondering though whether perhaps Kurt could just get commit rights
> for the repos, under the strict understanding that he doesn't push
> anything which is not reviewed (for now at least). That could take some
> day-to-day merges off my back and make Kurt responsible for doing stable
> merges and think about the code integration schedule right from the start.

I wouldn't mind giving him write access, though I don't see an immediate
need for that. Patches will require reviews anyway, even major design
reviews.


> We could then use e.g. codereview.appspot.com for doing the code
> reviews; I would then simply give thumbs up and Kurt would do the push.

Looks like they didn't fix the login issue yet. It still requires a Google
account to comment.


> 3) Feedback
>
> There should be some kind of feedback mechanism; I expect to hear from
> Kurt at least weekly over the summer unless otherwise is arranged. I
> expect the ticket system will serve as the status update, and hope Kurt
> will make his progress visible through the ticket system (we can use the
> "fortran" tag on all GSoC related tickets for instance).

Sure. Please take care to cut down the subject of tickets to make sure it
has a provable result. Otherwise, it's not clear when the ticket is
solved.

It's fine to have a couple of tickets that depend on each other.


> I guess (A) would perhaps use a different tracker than (B), we need to
> get back to that.

Why? As long as we consider integrating it with Cython, I'm fine with
tracking the required work here.


> Some mentors require students to keep a blog about what they are doing.
> Unless the Python Software Foundation makes this a requirement I'm going
> to leave this one up to Kurt; if the status is dutifully reported via
> tickets I'll have what I need personally, though a blog would be
> interesting as well.

It would be nice, yes.

Stefan


From stefan_ml at behnel.de  Tue Apr 21 19:04:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 19:04:12 +0200 (CEST)
Subject: [Cython] inherit from list
In-Reply-To: <49EDF6F0.504@chamonix.reportlab.co.uk>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
Message-ID: <5868d3e61cc56e30541512544f9c00e8.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robin Becker wrote:
> Is the intent to allow inheriting
> from builtin types to be handled automatically as it is in python where
> list is both a constructor and a type?

Yes, that's what ticket #258 is supposed to track (although the
description is a bit short to really get the idea).

Stefan


From dagss at student.matnat.uio.no  Tue Apr 21 19:39:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 21 Apr 2009 19:39:28 +0200 (CEST)
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu
	-darmstadt.de>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <4becd63dd8da72233794f517ffa03fea.squirrel@webmail.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> We could then use e.g. codereview.appspot.com for doing the code
>> reviews; I would then simply give thumbs up and Kurt would do the push.
>
> Looks like they didn't fix the login issue yet. It still requires a Google
> account to comment.

Have I missed a post somewhere on the dangers of getting a Google account?

Me and Kurt are obviously required to have those anyway, in order to
participate in the program.

Dag Sverre


From kwmsmith at gmail.com  Tue Apr 21 19:42:27 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 21 Apr 2009 12:42:27 -0500
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>

On Tue, Apr 21, 2009 at 12:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn wrote:
>> I'm very happy to report that Kurt Smith has been accepted for a Google
>> Summer of Code project for working on Cython/Fortran integration!
>
> Yep, congrats!

Thanks!  And thanks to you, Robert, Dag and everyone else for
accepting the project.  I'm pleased to see that Cython has 2 GSoC
projects this year, too (one more than last year?).


>> 1) Where should the mentoring take place? Opinions on how much should
>> happen on this list and how much should be done in private mail?
>> Obviously language decisions etc. must end up here in the end, but what
>> about day-to-day mentoring etc.?
>
> Depends. Trivial/basic things may go into private mail, but I would prefer
> having most of the discussions on the list, where others can
> help/find/learn, too. This is an open source project, so things should
> happen in the open.

I'm with you there -- we'll get a feel as time goes on, but by default
the substantial threads would be on the list.

>
>
>> One possibility is setting up a seperate Google group just for this
>> project, if it turns out that it would be too high traffic for the
>> mailing list but other people would like to follow the project as well.
>
> Let's see about that when we notice that it gets too much traffic.
>
> I would lean more towards a separate cython-users list than a separate
> sub-dev list. (I have ignored threads before, so I know I'll cope ;)
>
>
>> 2) Code and patch workflow.
>>
>> On the (A) side, the status with f2py is that the author started a full
>> rewrite (f2py 3G) which hasn't seen work for half a year
>> (http://f2py.org/). So anything can happen here and we have to get back
>> to how it's going to be done and where it should live.
>
> Sounds like the right thing to ask the author first.

Dag just sent off an email to Pearu this morning outlining our
approach and possible options.  We'll let you know the results & cc
the list if need be.

>
>
>> A relevant
>> question though is whether we would like to ship the tool in (A) as part
>> of Cython or have it as an independent download. (I don't expect it to
>> add anything significant to download size, but Python core inclusion
>> would be something to consider for or against here.)
>
> Let's decide that when we can take a look at it.
>
>
>> When it comes to (B), the obvious thing to do is for Kurt to pass the
>> code to me for review and then merge it as often as we can with either
>> -devel or -unstable (not sure which yet, possibly both).
>
> Definitely not the current cython-devel, but without further detail (that
> may become available once the project has left the initial planning
> phase), I can't tell if it makes more sense to add another branch. Given
> the time-frame of the GSoC, maybe even cython-unstable is too close to
> release to add major new, unfinished features. Parts may go in earlier,
> obviously, so a branch from which we can import stuff may make sense.
>
> IMHO, too early to decide.

I/we should have a better grasp on this in a couple of weeks at most.

>
>
>> I'm wondering though whether perhaps Kurt could just get commit rights
>> for the repos, under the strict understanding that he doesn't push
>> anything which is not reviewed (for now at least). That could take some
>> day-to-day merges off my back and make Kurt responsible for doing stable
>> merges and think about the code integration schedule right from the start.
>
> I wouldn't mind giving him write access, though I don't see an immediate
> need for that. Patches will require reviews anyway, even major design
> reviews.

I have no great eagerness for write access, and I'll need to prove
trustworthiness to get there.  I hope that at a future date after
patches are reviewed and committed by yourselves it'll make more sense
to grant me write access, with the understanding that I'll put up the
patches for review first.

Honestly, my mercurial-fu isn't marvelous at the moment (I can do the
basic things and am establishing my workflow, but if I were to mess
something up I'd need help fixing it) and I'd like some more
experience in the day-to-day work before being given access to
something that matters.

>
>
>> We could then use e.g. codereview.appspot.com for doing the code
>> reviews; I would then simply give thumbs up and Kurt would do the push.
>
> Looks like they didn't fix the login issue yet. It still requires a Google
> account to comment.
>
>
>> 3) Feedback
>>
>> There should be some kind of feedback mechanism; I expect to hear from
>> Kurt at least weekly over the summer unless otherwise is arranged. I
>> expect the ticket system will serve as the status update, and hope Kurt
>> will make his progress visible through the ticket system (we can use the
>> "fortran" tag on all GSoC related tickets for instance).
>
> Sure. Please take care to cut down the subject of tickets to make sure it
> has a provable result. Otherwise, it's not clear when the ticket is
> solved.
>
> It's fine to have a couple of tickets that depend on each other.

Sounds fine to me.

>> I guess (A) would perhaps use a different tracker than (B), we need to
>> get back to that.
>
> Why? As long as we consider integrating it with Cython, I'm fine with
> tracking the required work here.
>
>
>> Some mentors require students to keep a blog about what they are doing.
>> Unless the Python Software Foundation makes this a requirement I'm going
>> to leave this one up to Kurt; if the status is dutifully reported via
>> tickets I'll have what I need personally, though a blog would be
>> interesting as well.
>
> It would be nice, yes.

I'll get one set up.  Any canonical place for GSoC blogs that anyone knows of?

Kurt

From dagss at student.matnat.uio.no  Tue Apr 21 19:48:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 21 Apr 2009 19:48:25 +0200 (CEST)
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
Message-ID: <f6fe0d10df80e021bd5c74a315121eb2.squirrel@webmail.uio.no>

Kurt Smith wrote:
> On Tue, Apr 21, 2009 at 12:01 PM, Stefan Behnel <stefan_ml at behnel.de>
> wrote:
>> Dag Sverre Seljebotn wrote:
>>> Some mentors require students to keep a blog about what they are doing.
>>> Unless the Python Software Foundation makes this a requirement I'm
>>> going
>>> to leave this one up to Kurt; if the status is dutifully reported via
>>> tickets I'll have what I need personally, though a blog would be
>>> interesting as well.
>>
>> It would be nice, yes.
>
> I'll get one set up.  Any canonical place for GSoC blogs that anyone knows
> of?

I don't think so, what happened last year though is that PSF aggregated
the GSoC blogs here: http://soc.python.org/ . So as long as you have RSS
support you're good.

Dag Sverre


From fperez.net at gmail.com  Tue Apr 21 20:29:35 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 21 Apr 2009 11:29:35 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <49EDB668.7030502@student.matnat.uio.no>
References: <49EDB668.7030502@student.matnat.uio.no>
Message-ID: <db6b5ecc0904211129g4d48d632y95a8dbba5bafff8f@mail.gmail.com>

On Tue, Apr 21, 2009 at 5:04 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> I'm very happy to report that Kurt Smith has been accepted for a Google
> Summer of Code project for working on Cython/Fortran integration!
>
> http://wiki.cython.org/kwsmith/soc09
>
> I'm his mentor for the project, and I start this thread to discuss how

I just wanted to congratulate you and thank you for getting this
going!  Fortran support is an oft-overlooked yet *critical* piece of
making Python a serious scientific computing tool, and f2py has been
getting a bit long in the tooth.  For a long time many of us have
thought that the cython-f2py marriage would be an ideal evolution,
seeing how conceptually they can share so much infrastructure, and
cython is a much more lively project than f2py today.

So a huge thanks from me, and best wishes for the success of this
project over the summer!

Cheers,

f

From robertwb at math.washington.edu  Tue Apr 21 20:33:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 11:33:43 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
Message-ID: <E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>

On Apr 21, 2009, at 6:17 AM, Lisandro Dalcin wrote:

> Robert, your main() implementation needs more work IMHO.

Thanks for the feedback. I just did the basic embedding, as on David  
McNab's site. It would be good to have it behave more like python -m  
http://trac.cython.org/cython_trac/ticket/291


> 1) For Py2, you need to if(PyErr_Occurred())
> PyErr_WriteUnraisable(PyErr_Occurred()).

Actually, it might be nice to dump the whole traceback here.

> 2) In Py3, you have to save the return of PyInit_<modname>, if NULL,
> report error (WriteUnraisable again?), if not, decref it.

Is PyErr_Occurred not a good enough check here? Yes, I should  
probably get and decref the module (though it won't matter much as I  
call Py_Finalize next.

> 3) In case of errors, process return value should match what core
> CPython returns en case of unhandled execptions.

Do you know of a good list? (Otherwise, I'll just try some.)

> 3) However, a SystemExit exceptions should need some special
> consideration, as you should extract the process return value from it,
> as it could be 0 (zero).

Good point.

- Robert

From robertwb at math.washington.edu  Tue Apr 21 20:36:49 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 11:36:49 -0700
Subject: [Cython] help with function pointer
In-Reply-To: <d416dd90cdcf283d0aae347b41e40d28.squirrel@webmail.uio.no>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<6F08D037-7371-4D43-B7FE-8185FD8B25DB@math.washington.edu>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
	<b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>
	<d416dd90cdcf283d0aae347b41e40d28.squirrel@webmail.uio.no>
Message-ID: <AC1069B7-A950-492A-A075-9D20FE4EB6A0@math.washington.edu>

On Apr 21, 2009, at 7:23 AM, Dag Sverre Seljebotn wrote:

> Jean-Francois Moulin wrote:
>> Great! Just tried it and it works fine!
>> I gain something like a tenfold factor in speed with respect to my
>> initial python code.
>>> From what I read I expect I should still gain quite a lot by
>> optimizing the calls to the Numpy functions.
>> Namely using the buffer notation defining the type of array I am  
>> using.
>> A question arises, should I also apply this notation in the  
>> definition
>> of my pointers? and if yes, how?
>>
>> so, I presently have statements like:
>> cdef int (*locate)(DelayLine, np.ndarray)
>>
>> and would write something like:
>>  cdef int (*locate)(DelayLine, np.ndarray[DTYPE_t,ndim=2])
>
> This is simply not supported with cdef functions. You need to do
>
> cdef int (*locate)(DelayLine, np.ndarray)
>
> and then inside the implementation of the function you assign to  
> it, you do
>
> cdef int (*locate)(DelayLine a, np.ndarray b)
>     cdef np.ndarray[DTYPE_t, ndim=2] b_buf = b
>     .... access b through b_buf ...

I may be missing something about your use cases, but is there any  
reason you're passing/storing function pointers all over the place  
instead of just calling cdef methods on a cdef class? That would  
probably make your code a lot cleaner and easier to understand (and  
write).

- Robert


From robertwb at math.washington.edu  Tue Apr 21 20:50:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 11:50:17 -0700
Subject: [Cython] Path towards Python completeness
In-Reply-To: <ab1e0fdf3d34a12989a138890c192bfd.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49ED5CE8.6090708@behnel.de>
	<49EDB97F.4030004@student.matnat.uio.no>
	<ab1e0fdf3d34a12989a138890c192bfd.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <70B6BB83-90F4-4148-A677-1C4574C224D5@math.washington.edu>

On Apr 21, 2009, at 9:26 AM, Stefan Behnel wrote:

> Dag Sverre Seljebotn wrote:
>> I don't think we have to think of ourselves as in direct competition
>> with PyPy (and/or Unladen Swallow). Personally I'm more than happy to
>> use pure Python rather than Cython for the kind of things that can be
>> sped up by those projects anyway (makes for a shorter compile/run  
>> cycle).
>
> I do actually see PyPy as a direct competitor of Cython (much more  
> than
> Jython and IronPython, BTW), as both projects aim to compile Python  
> code
> to fast native code. Currently, we are faster and they are more  
> feature
> complete. But if we can have the cake and eat it, too...

It's also important to note that each has non-intersecting goals on  
top of making Python faster, and I could see them all being around  
for a long time. One thing that sets us apart is the ease at which we  
can integrate existing C (and soon C++/Fortran) code, which is  
extremely valuable. Our greatest weakness is that we are tied to the  
non-treadsafe CPython interpreter--my hope is that eventually CPython  
will be improved to use a concurrent garbage collector and its  
shipped modules fixed accordingly. (With the current trend of the  
number of cores increasing faster than the clockspeed, one could  
dedicate a core to GC alone if speed is a concern as it has been in  
the past...)

>> What I think sets Cython apart is that we add a superset to the  
>> Python
>> language for adding types; which are simply needed in some  
>> situations. I
>> think that is where we "compete".
>
> That's rather where we are beyond competition.
>
>
>> When PyPy runs as fast as CPython, perhaps PyPy can start  
>> supporting the
>> Cython type annotations (at least the pure Python mode ones); Jython
>> could support the same annotations in time, and so on.
>
> Let's wait and see, but I doubt that there will be a major interest in
> doing that. I actually find it more important for Cython to gain type
> inference mechanisms that lift the requirement for defining types  
> in the
> first place, at least for standard use cases.

+1, inference is something else that would be really nice to finally  
have. There was enough interest, however, to add function signature  
type annotations.

- Robert


From dagss at student.matnat.uio.no  Tue Apr 21 20:55:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 21 Apr 2009 20:55:55 +0200 (CEST)
Subject: [Cython] Path towards Python completeness
In-Reply-To: <70B6BB83-90F4-4148-A677-1C4574C224D5@math.washington.edu>
References: <49ED5CE8.6090708@behnel.de>
	<49EDB97F.4030004@student.matnat.uio.no>
	<ab1e0fdf3d34a12989a138890c192bfd.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<70B6BB83-90F4-4148-A677-1C4574C224D5@math.washington.edu>
Message-ID: <ae9a9d59754e2c145d679b856262369a.squirrel@webmail.uio.no>

>> Something I snipped prematurely

Yes, I'm very happy that Unladen Swallow say they might make another
charge on the GIL, I hope they succeed.

Robert Bradshaw wrote:
> On Apr 21, 2009, at 9:26 AM, Stefan Behnel wrote:
>
>> Dag Sverre Seljebotn wrote:
>>> When PyPy runs as fast as CPython, perhaps PyPy can start
>>> supporting the
>>> Cython type annotations (at least the pure Python mode ones); Jython
>>> could support the same annotations in time, and so on.
>>
>> Let's wait and see, but I doubt that there will be a major interest in
>> doing that. I actually find it more important for Cython to gain type
>> inference mechanisms that lift the requirement for defining types
>> in the
>> first place, at least for standard use cases.
>
> +1, inference is something else that would be really nice to finally
> have. There was enough interest, however, to add function signature
> type annotations.

I was thinking of this at a slightly broader level. For instance, I don't
see automatic inference for doing vtable dispatches rather than dict
lookups any time soon -- yet they are critical for writing fast data
structures.

Cython provides vtable dispatches, etc., which is outside of the goal of
the other projects. (And that's a major part of "typing").

Dag Sverre


From stefan_ml at behnel.de  Tue Apr 21 21:00:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Apr 2009 21:00:31 +0200
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <4becd63dd8da72233794f517ffa03fea.squirrel@webmail.uio.no>
References: <49EDB668.7030502@student.matnat.uio.no>	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4becd63dd8da72233794f517ffa03fea.squirrel@webmail.uio.no>
Message-ID: <49EE17CF.409@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> We could then use e.g. codereview.appspot.com for doing the code
>>> reviews; I would then simply give thumbs up and Kurt would do the push.
>> Looks like they didn't fix the login issue yet. It still requires a Google
>> account to comment.
> 
> Have I missed a post somewhere on the dangers of getting a Google account?

Last time I checked, they had all sorts of evil paragraphs in their "terms
of service". I rechecked now and it seems they have learned something and
stripped it down from 21 to 11 paragraphs. They no longer require me to
acknowledge their patents, for example.

There is one paragraph that requires me to accept updates to software
installed on my computer (which I don't have to install, so that's fine).

There is another paragraph that states that they are allowed to change
their TOS at any time, and move it onto my shoulders to poll their TOS for
updates regularly. That's not quite what I call fair contract conditions.
It's more like: "well, we can change the contract at any time, so if we
feel like suing you, we will make sure that we adapt it accordingly before
sending out lawyers".

But, all in all, the new TOS are actually quite acceptable.

Stefan

From kwmsmith at gmail.com  Tue Apr 21 21:29:01 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 21 Apr 2009 14:29:01 -0500
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <f6fe0d10df80e021bd5c74a315121eb2.squirrel@webmail.uio.no>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<f6fe0d10df80e021bd5c74a315121eb2.squirrel@webmail.uio.no>
Message-ID: <c266a7930904211229w5e596b29i6b38a055ff6f54f7@mail.gmail.com>

On Tue, Apr 21, 2009 at 12:48 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Kurt Smith wrote:
>>
>> I'll get one set up. ?Any canonical place for GSoC blogs that anyone knows
>> of?
>
> I don't think so, what happened last year though is that PSF aggregated
> the GSoC blogs here: http://soc.python.org/ . So as long as you have RSS
> support you're good.
>

Here it is:

http://fortrancython.wordpress.com/

From jf.moulin at gmail.com  Tue Apr 21 23:05:51 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 21 Apr 2009 23:05:51 +0200
Subject: [Cython] help with function pointer
In-Reply-To: <AC1069B7-A950-492A-A075-9D20FE4EB6A0@math.washington.edu>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
	<b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>
	<d416dd90cdcf283d0aae347b41e40d28.squirrel@webmail.uio.no>
	<AC1069B7-A950-492A-A075-9D20FE4EB6A0@math.washington.edu>
Message-ID: <af6d35810904211405p4a4cca29n44d3e2f9312f09fe@mail.gmail.com>

> I may be missing something about your use cases, but is there any
> reason you're passing/storing function pointers all over the place
> instead of just calling cdef methods on a cdef class? That would
> probably make your code a lot cleaner and easier to understand (and
> write).

Well, that might be due to me being a newbie ;0)
Let me briefly explain my problem and idea:
I need to a apply a discrimination function on some data, depending on
the structure of these data, the discri fn can take two forms (possibly
more in the future).
Rather than to use an if / else test at each iteration to branch to the
relevant function, I do the test once at the beginning (first time I see the
data) and then define via a pointer which fn to use. Same thing for the locate
function, which can vary depending on the available data.
I hope this make sense, and when not, I am more then willing to learn of a
better approach... I am learning the hard way!


JF

From robertwb at math.washington.edu  Wed Apr 22 02:17:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 17:17:32 -0700
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904211405p4a4cca29n44d3e2f9312f09fe@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<af6d35810904210045k105c4178t29f68b621d1354c5@mail.gmail.com>
	<CAD789E2-DAD2-4605-9878-318E7A2ECD33@math.washington.edu>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
	<b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>
	<d416dd90cdcf283d0aae347b41e40d28.squirrel@webmail.uio.no>
	<AC1069B7-A950-492A-A075-9D20FE4EB6A0@math.washington.edu>
	<af6d35810904211405p4a4cca29n44d3e2f9312f09fe@mail.gmail.com>
Message-ID: <6D796CC5-B26D-4033-BBF7-AF5AFE62F74D@math.washington.edu>

On Apr 21, 2009, at 2:05 PM, Jean-Francois Moulin wrote:

>> I may be missing something about your use cases, but is there any
>> reason you're passing/storing function pointers all over the place
>> instead of just calling cdef methods on a cdef class? That would
>> probably make your code a lot cleaner and easier to understand (and
>> write).
>
> Well, that might be due to me being a newbie ;0)
> Let me briefly explain my problem and idea:
> I need to a apply a discrimination function on some data, depending on
> the structure of these data, the discri fn can take two forms  
> (possibly
> more in the future).
> Rather than to use an if / else test at each iteration to branch to  
> the
> relevant function, I do the test once at the beginning (first time  
> I see the
> data) and then define via a pointer which fn to use. Same thing for  
> the locate
> function, which can vary depending on the available data.

BTW, calling a function pointer can be more expensive than a  
predictable branch, depends on what you're doing. In Python, of  
course, it can make a huge difference.

> I hope this make sense, and when not, I am more then willing to  
> learn of a
> better approach... I am learning the hard way!


Here's what I would do--I'd make a class that has cdef discrimination  
and cdef locate functions, then subclass it for all the different  
data structures you want to handle. You can then do

cdef DataHandlerClass f = [decide which one here]
for x in data:
     y = f.locate(x)
     ...

locate can be a nice cdef (or even cpdef) function, with optional  
arguments and all the other niceness that you've come to expect. I  
also think this better encapsulates what you're trying to do, but  
that's subjective of course.

- Robert


From greg.ewing at canterbury.ac.nz  Wed Apr 22 02:34:02 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 22 Apr 2009 12:34:02 +1200
Subject: [Cython] inherit from list
In-Reply-To: <49EDF6F0.504@chamonix.reportlab.co.uk>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
Message-ID: <49EE65FA.6080501@canterbury.ac.nz>

Robin Becker wrote:

> OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
> from builtin types to be handled automatically as it is in python where list is 
> both a constructor and a type?

FWIW, the current version of Pyrex handles inheriting
from the builtin list type okay, without the need for
any declarations.

-- 
Greg

From robertwb at math.washington.edu  Wed Apr 22 02:44:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 17:44:59 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
Message-ID: <B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>

On Apr 21, 2009, at 10:42 AM, Kurt Smith wrote:

> On Tue, Apr 21, 2009 at 12:01 PM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>> Dag Sverre Seljebotn wrote:
>>> I'm very happy to report that Kurt Smith has been accepted for a  
>>> Google
>>> Summer of Code project for working on Cython/Fortran integration!
>>
>> Yep, congrats!
>
> Thanks!  And thanks to you, Robert, Dag and everyone else for
> accepting the project.  I'm pleased to see that Cython has 2 GSoC
> projects this year, too (one more than last year?).

Yep, and it looks like both are going to be big steps forward in  
terms of usability.

>>> 1) Where should the mentoring take place? Opinions on how much  
>>> should
>>> happen on this list and how much should be done in private mail?
>>> Obviously language decisions etc. must end up here in the end,  
>>> but what
>>> about day-to-day mentoring etc.?
>>
>> Depends. Trivial/basic things may go into private mail, but I  
>> would prefer
>> having most of the discussions on the list, where others can
>> help/find/learn, too. This is an open source project, so things  
>> should
>> happen in the open.
>
> I'm with you there -- we'll get a feel as time goes on, but by default
> the substantial threads would be on the list.

Day-to-day mentoring should probably be off-list, but I imagine the  
bulk of what goes on will be on-list. Maybe ask yourself if someone  
using the fortran-cython interface a year from now could possibly  
have any interest in reading the thread.

>>> One possibility is setting up a seperate Google group just for this
>>> project, if it turns out that it would be too high traffic for the
>>> mailing list but other people would like to follow the project as  
>>> well.
>>
>> Let's see about that when we notice that it gets too much traffic.
>>
>> I would lean more towards a separate cython-users list than a  
>> separate
>> sub-dev list. (I have ignored threads before, so I know I'll cope ;)

I brought the idea of a cython-users/cython-support list before, and  
there wasn't much interested. (Of course, those who couldn't handle  
the traffic of cython-devel never got the message...) If it's too  
specific to be on cython-devel, it probably make sense to have it  
just between Dag and Kurt.

>>> 2) Code and patch workflow.
>>>
>>> A relevant
>>> question though is whether we would like to ship the tool in (A)  
>>> as part
>>> of Cython or have it as an independent download. (I don't expect  
>>> it to
>>> add anything significant to download size, but Python core inclusion
>>> would be something to consider for or against here.)
>>
>> Let's decide that when we can take a look at it.

Lets not count on doing so for now at least. There are license  
considerations to consider too--f2py is GPL and as such wouldn't make  
it into Python without relicensing.

>>> When it comes to (B), the obvious thing to do is for Kurt to pass  
>>> the
>>> code to me for review and then merge it as often as we can with  
>>> either
>>> -devel or -unstable (not sure which yet, possibly both).
>>
>> Definitely not the current cython-devel, but without further  
>> detail (that
>> may become available once the project has left the initial planning
>> phase), I can't tell if it makes more sense to add another branch.  
>> Given
>> the time-frame of the GSoC, maybe even cython-unstable is too  
>> close to
>> release to add major new, unfinished features. Parts may go in  
>> earlier,
>> obviously, so a branch from which we can import stuff may make sense.
>>
>> IMHO, too early to decide.
>
> I/we should have a better grasp on this in a couple of weeks at most.

I think the best option is to have a new branch for this project.  
This at the very least will be a public, open place for people to  
look at and review the code. I anticipate it will get pulled into - 
devel or -unstable on a regular basis by Dag (and, depending on how  
things develop, hopefully Kurt will get push access too).

>>> I'm wondering though whether perhaps Kurt could just get commit  
>>> rights
>>> for the repos, under the strict understanding that he doesn't push
>>> anything which is not reviewed (for now at least). That could  
>>> take some
>>> day-to-day merges off my back and make Kurt responsible for doing  
>>> stable
>>> merges and think about the code integration schedule right from  
>>> the start.
>>
>> I wouldn't mind giving him write access, though I don't see an  
>> immediate
>> need for that. Patches will require reviews anyway, even major design
>> reviews.
>
> I have no great eagerness for write access, and I'll need to prove
> trustworthiness to get there.  I hope that at a future date after
> patches are reviewed and committed by yourselves it'll make more sense
> to grant me write access, with the understanding that I'll put up the
> patches for review first.

If you're pushing to a public repo (see above) it will be relatively  
easy for us to pull your changes over (e.g. when you're ready, you  
pull from -unstable, merge, make sure all is well, and then when  
we've reviewed your stuff we pull your changes over to -unstable).

> Honestly, my mercurial-fu isn't marvelous at the moment (I can do the
> basic things and am establishing my workflow, but if I were to mess
> something up I'd need help fixing it) and I'd like some more
> experience in the day-to-day work before being given access to
> something that matters.
>
>>> We could then use e.g. codereview.appspot.com for doing the code
>>> reviews; I would then simply give thumbs up and Kurt would do the  
>>> push.
>>
>> Looks like they didn't fix the login issue yet. It still requires  
>> a Google
>> account to comment.

Whatever works best for you two, that's what we should use.

>>> 3) Feedback
>>>
>>> There should be some kind of feedback mechanism; I expect to hear  
>>> from
>>> Kurt at least weekly over the summer unless otherwise is arranged. I
>>> expect the ticket system will serve as the status update, and  
>>> hope Kurt
>>> will make his progress visible through the ticket system (we can  
>>> use the
>>> "fortran" tag on all GSoC related tickets for instance).
>>
>> Sure. Please take care to cut down the subject of tickets to make  
>> sure it
>> has a provable result. Otherwise, it's not clear when the ticket is
>> solved.
>>
>> It's fine to have a couple of tickets that depend on each other.
>
> Sounds fine to me.
>
>>> I guess (A) would perhaps use a different tracker than (B), we  
>>> need to
>>> get back to that.
>>
>> Why? As long as we consider integrating it with Cython, I'm fine with
>> tracking the required work here.
>>
>>
>>> Some mentors require students to keep a blog about what they are  
>>> doing.
>>> Unless the Python Software Foundation makes this a requirement  
>>> I'm going
>>> to leave this one up to Kurt; if the status is dutifully reported  
>>> via
>>> tickets I'll have what I need personally, though a blog would be
>>> interesting as well.
>>
>> It would be nice, yes.
>
> I'll get one set up.  Any canonical place for GSoC blogs that  
> anyone knows of?

Thanks. Danilo, do you want to set one up for your project as well.

- Robert



From dalcinl at gmail.com  Wed Apr 22 04:31:13 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 21 Apr 2009 23:31:13 -0300
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
Message-ID: <e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>

On Tue, Apr 21, 2009 at 3:33 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 21, 2009, at 6:17 AM, Lisandro Dalcin wrote:
>
>> Robert, your main() implementation needs more work IMHO.
>
> Thanks for the feedback. I just did the basic embedding, as on David
> McNab's site.

>>>import this
...
Now is better than never.
Although never is often better than *right* now.
...

Or perhaps Cython should have its own Zen ;-) ???

> It would be good to have it behave more like python -m
> http://trac.cython.org/cython_trac/ticket/291
>

Mmm... I?m not sure python -m will be equivalent to your embeding...
Py_Main() does a lot of things other thanInitialize() && SetArgv(), I
do not remember right now, but IIRC some environ vars are inspected in
Py_Main()

>
>> 1) For Py2, you need to if(PyErr_Occurred())
>> PyErr_WriteUnraisable(PyErr_Occurred()).
>
> Actually, it might be nice to dump the whole traceback here.
>

OK, of course... you want the equivalent to "import traceback;
traceback.print_exc()". I do not remember right now if there is a
direct C-API call for this.

>> 2) In Py3, you have to save the return of PyInit_<modname>, if NULL,
>> report error (WriteUnraisable again?), if not, decref it.
>
> Is PyErr_Occurred not a good enough check here? Yes, I should
> probably get and decref the module (though it won't matter much as I
> call Py_Finalize next.
>

Well, I would ask to maintain Cython-generated code valgrind-friendly
if possible :-)

>> 3) In case of errors, process return value should match what core
>> CPython returns en case of unhandled execptions.
>
> Do you know of a good list? (Otherwise, I'll just try some.)
>

Do not remember right now, but I rememberd "-1" as a hard-wired return
value in core CPython (likely in pythonrun.c).

>> 3) However, a SystemExit exceptions should need some special
>> consideration, as you should extract the process return value from it,
>> as it could be 0 (zero).
>
> Good point.
>

In short, I would check for SystemExit and honor its return code, or
just return "-1"


> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Apr 22 04:50:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 21 Apr 2009 23:50:19 -0300
Subject: [Cython] download for 0.11.1
In-Reply-To: <49EDCB8A.6040705@student.matnat.uio.no>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
	<49EDC2B0.3060700@simula.no>
	<49EDC619.5000602@chamonix.reportlab.co.uk>
	<49EDCB8A.6040705@student.matnat.uio.no>
Message-ID: <e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>

On Tue, Apr 21, 2009 at 10:35 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>
> This is exactly why we do it. Sage ships this way and I think it's
> been a significant factor in attracting developers.
>

Sorry, can't resist... Could you tell me why a developer would be
attracted by this? To make use of ".hg", they need to have mercurial,
then... Do you think any DEVELOPER will find it cumbersome to do "hg
clone http://hg.cython.org/cython" ??



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Apr 22 04:50:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 19:50:46 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>
Message-ID: <C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>

On Apr 21, 2009, at 7:31 PM, Lisandro Dalcin wrote:

> On Tue, Apr 21, 2009 at 3:33 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 21, 2009, at 6:17 AM, Lisandro Dalcin wrote:
>>
>>> Robert, your main() implementation needs more work IMHO.
>>
>> Thanks for the feedback. I just did the basic embedding, as on David
>> McNab's site.
>
>>>> import this
> ...
> Now is better than never.
> Although never is often better than *right* now.
> ...
>
> Or perhaps Cython should have its own Zen ;-) ???

:-) I was actually just working on this *right* now.

>
>> It would be good to have it behave more like python -m
>> http://trac.cython.org/cython_trac/ticket/291
>
> Mmm... I?m not sure python -m will be equivalent to your embeding...
> Py_Main() does a lot of things other thanInitialize() && SetArgv(), I
> do not remember right now, but IIRC some environ vars are inspected in
> Py_Main()

Hmm... I'd like to see a list of what happens before we decide to opt  
for python -m semantics or simplicity. I'm going to leave the ticket  
open for now.

>
>>
>>> 1) For Py2, you need to if(PyErr_Occurred())
>>> PyErr_WriteUnraisable(PyErr_Occurred()).
>>
>> Actually, it might be nice to dump the whole traceback here.
>
> OK, of course... you want the equivalent to "import traceback;
> traceback.print_exc()". I do not remember right now if there is a
> direct C-API call for this.

There's a C-API call for this.

>>> 2) In Py3, you have to save the return of PyInit_<modname>, if NULL,
>>> report error (WriteUnraisable again?), if not, decref it.
>>
>> Is PyErr_Occurred not a good enough check here? Yes, I should
>> probably get and decref the module (though it won't matter much as I
>> call Py_Finalize next.
>
> Well, I would ask to maintain Cython-generated code valgrind-friendly
> if possible :-)

Me too. Done.

>
>>> 3) In case of errors, process return value should match what core
>>> CPython returns en case of unhandled execptions.
>>
>> Do you know of a good list? (Otherwise, I'll just try some.)
>>
>
> Do not remember right now, but I rememberd "-1" as a hard-wired return
> value in core CPython (likely in pythonrun.c).

I'm getting 1 unless otherwise specified. I've seen -1 hard coded,  
but never seen it come out (both on mac and linux).

>
>>> 3) However, a SystemExit exceptions should need some special
>>> consideration, as you should extract the process return value  
>>> from it,
>>> as it could be 0 (zero).
>>
>> Good point.
>>
>
> In short, I would check for SystemExit and honor its return code, or
> just return "-1"

Interestingly enough, the C-API call PyErr_PrintEx does this for you.  
(Yeah, I might call it a bit hackish, but that's what Python seems to  
use).

- Robert



From dalcinl at gmail.com  Wed Apr 22 05:06:24 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Apr 2009 00:06:24 -0300
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>
	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>
Message-ID: <e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>

On Tue, Apr 21, 2009 at 11:50 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>>
>>> It would be good to have it behave more like python -m
>>> http://trac.cython.org/cython_trac/ticket/291
>>
>> Mmm... I?m not sure python -m will be equivalent to your embeding...
>> Py_Main() does a lot of things other thanInitialize() && SetArgv(), I
>> do not remember right now, but IIRC some environ vars are inspected in
>> Py_Main()
>
> Hmm... I'd like to see a list of what happens before we decide to opt
> for python -m semantics or simplicity. I'm going to leave the ticket
> open for now.
>

Perhaps I was not clear enough... I mean, 'python -m' is one thing, a
binary executable embedding Python is a different beast... You are
adding support for generating a bin exe, so IMHO you should keep it
SIMPLE, at least for now...

>>
>> In short, I would check for SystemExit and honor its return code, or
>> just return "-1"
>
> Interestingly enough, the C-API call PyErr_PrintEx does this for you.
> (Yeah, I might call it a bit hackish, but that's what Python seems to
> use).
>

Wow... I see... So you call PyErr_PrintEx(0) (likely passing 1 will be
pointless, right?) and you are done.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Wed Apr 22 05:06:42 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 21 Apr 2009 20:06:42 -0700
Subject: [Cython] download for 0.11.1
In-Reply-To: <e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>	<49EDC2B0.3060700@simula.no>	<49EDC619.5000602@chamonix.reportlab.co.uk>	<49EDCB8A.6040705@student.matnat.uio.no>
	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
Message-ID: <49EE89C2.4000805@gmail.com>

Lisandro Dalcin wrote:
> On Tue, Apr 21, 2009 at 10:35 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> This is exactly why we do it. Sage ships this way and I think it's
>> been a significant factor in attracting developers.
>>
> 
> Sorry, can't resist... Could you tell me why a developer would be
> attracted by this? To make use of ".hg", they need to have mercurial,
> then... Do you think any DEVELOPER will find it cumbersome to do "hg
> clone http://hg.cython.org/cython" ??

The developer still has to do it :)

With Sage telling people to check in their changes and to send us a 
patch is absolutely trivial. This is also due to the integration of hg 
and the different repos in Sage to the command line in Sage. Once you 
get someone to contribute that way the first time it is often possible 
to convert such a person into a Sage developer since the main problem 
often enough is to get non-technical mathematicians to play with version 
control, etc.

Debugging problems is also much easier if you ship with a repo since you 
can verify the state, figure out what patches are applied, etc. But I 
see this to be less of an argument for Cython, but I agree with Robert 
that it cannot hurt to have the extra 4MB in the archive.

Cheers,

Michael


From robertwb at math.washington.edu  Wed Apr 22 05:07:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 20:07:25 -0700
Subject: [Cython] download for 0.11.1
In-Reply-To: <e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
	<49EDC2B0.3060700@simula.no>
	<49EDC619.5000602@chamonix.reportlab.co.uk>
	<49EDCB8A.6040705@student.matnat.uio.no>
	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
Message-ID: <DAE326CC-76C3-42AB-BB8B-E10922EDA167@math.washington.edu>

On Apr 21, 2009, at 7:50 PM, Lisandro Dalcin wrote:

> On Tue, Apr 21, 2009 at 10:35 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>>
>> This is exactly why we do it. Sage ships this way and I think it's
>> been a significant factor in attracting developers.
>
> Sorry, can't resist... Could you tell me why a developer would be
> attracted by this? To make use of ".hg", they need to have mercurial,
> then... Do you think any DEVELOPER will find it cumbersome to do "hg
> clone http://hg.cython.org/cython" ??

With the Sage project, most (in my anecdotal evidence) developers  
started out as users, then notice a typo in the documentation, or  
missing functionality, or just look at the code (it's easy to get the  
code for any function at the prompt) and say "I could do this  
better." I have the feeling that a lot of them would not go so far as  
to download the repo separately to start hacking on it, or at least  
would not have done so so soon. In fact, many people edit first, then  
want to submit changes later.

Of course there are several differences between Sage and Cython,  
noteably the tech-savvyness of the intended audience, the fact that  
Sage ships with many components as well as the core source (e.g. it  
ships with mercurial--which is the right choice for Sage), and its  
sheer size. But I think a lot remains the same. Other than saving a  
few MB on a users hard drive, I have yet to see any compelling  
reasons against it.

- Robert


From robertwb at math.washington.edu  Wed Apr 22 05:13:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 20:13:07 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<ca433830904190616v9cf56f9r26ca2ed803a4423e@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>
	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>
	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>
Message-ID: <CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>

On Apr 21, 2009, at 8:06 PM, Lisandro Dalcin wrote:

> On Tue, Apr 21, 2009 at 11:50 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>>
>>>> It would be good to have it behave more like python -m
>>>> http://trac.cython.org/cython_trac/ticket/291
>>>
>>> Mmm... I?m not sure python -m will be equivalent to your embeding...
>>> Py_Main() does a lot of things other thanInitialize() && SetArgv 
>>> (), I
>>> do not remember right now, but IIRC some environ vars are  
>>> inspected in
>>> Py_Main()
>>
>> Hmm... I'd like to see a list of what happens before we decide to opt
>> for python -m semantics or simplicity. I'm going to leave the ticket
>> open for now.
>>
>
> Perhaps I was not clear enough... I mean, 'python -m' is one thing, a
> binary executable embedding Python is a different beast... You are
> adding support for generating a bin exe, so IMHO you should keep it
> SIMPLE, at least for now...

I agree. I'll change the subject of the ticket title.

>>> In short, I would check for SystemExit and honor its return code, or
>>> just return "-1"
>>
>> Interestingly enough, the C-API call PyErr_PrintEx does this for you.
>> (Yeah, I might call it a bit hackish, but that's what Python seems to
>> use).
>>
>
> Wow... I see... So you call PyErr_PrintEx(0) (likely passing 1 will be
> pointless, right?)

There might be memory cleanup advantages one way or the other. I'm  
calling PyErr_Print() which defaults to 1.

> and you are done.

Well, it only exits if it's a SystemExit, but basically, yes.

- Robert


From ellisonbg.net at gmail.com  Wed Apr 22 05:21:28 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 21 Apr 2009 20:21:28 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
Message-ID: <6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>

> Lets not count on doing so for now at least. There are license
> considerations to consider too--f2py is GPL and as such wouldn't make
> it into Python without relicensing.

Mmmm, f2py ships with numpy and thus can't be GPL.  The home page of
f2py lists is as LGPL.  Now whether or not that is sufficient for
Python itself is another question.

Brian

From robertwb at math.washington.edu  Wed Apr 22 05:59:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 20:59:43 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
Message-ID: <EAEECCB6-FFEF-488C-A90F-C82361A5A413@math.washington.edu>

On Apr 21, 2009, at 8:21 PM, Brian Granger wrote:

>> Lets not count on doing so for now at least. There are license
>> considerations to consider too--f2py is GPL and as such wouldn't make
>> it into Python without relicensing.
>
> Mmmm, f2py ships with numpy and thus can't be GPL.  The home page of
> f2py lists is as LGPL.  Now whether or not that is sufficient for
> Python itself is another question.

Yes, sorry, LGPL. In any case, lets treat it as an external  
dependancy for now.

- Robert


From wstein at gmail.com  Wed Apr 22 06:01:39 2009
From: wstein at gmail.com (William Stein)
Date: Tue, 21 Apr 2009 21:01:39 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <EAEECCB6-FFEF-488C-A90F-C82361A5A413@math.washington.edu>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
	<EAEECCB6-FFEF-488C-A90F-C82361A5A413@math.washington.edu>
Message-ID: <85e81ba30904212101n2a3af5edr52d0f86fc636cc84@mail.gmail.com>

2009/4/21 Robert Bradshaw <robertwb at math.washington.edu>:
> On Apr 21, 2009, at 8:21 PM, Brian Granger wrote:
>
>>> Lets not count on doing so for now at least. There are license
>>> considerations to consider too--f2py is GPL and as such wouldn't make
>>> it into Python without relicensing.
>>
>> Mmmm, f2py ships with numpy and thus can't be GPL. ?The home page of
>> f2py lists is as LGPL. ?Now whether or not that is sufficient for
>> Python itself is another question.
>
> Yes, sorry, LGPL. In any case, lets treat it as an external
> dependancy for now.

1. I think there is no way Python would ship LGPL'd code as part of
standard Python.

2. That f2py ships with numpy doesn't imply that f2py isn't GPL
(though it of course isn't GPL).
Numpy's license is modified BSD which can be combined with GPL'd code.

William

From kwmsmith at gmail.com  Wed Apr 22 06:18:19 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 21 Apr 2009 23:18:19 -0500
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
Message-ID: <c266a7930904212118j969a0c9i94105708b9e4be6a@mail.gmail.com>

On Tue, Apr 21, 2009 at 10:21 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Lets not count on doing so for now at least. There are license
>> considerations to consider too--f2py is GPL and as such wouldn't make
>> it into Python without relicensing.
>
> Mmmm, f2py ships with numpy and thus can't be GPL. ?The home page of
> f2py lists is as LGPL. ?Now whether or not that is sufficient for
> Python itself is another question.

Thanks for bringing this up.

We'll have to discuss licensing/packaging issues in more detail later,
and anything I say is open to correction from those more knowledgeable
(Pearu, Dag, numpy/scipy/cython devs...).

Our project won't be using the current f2py as exists in the
numpy/scipy distribution.  Instead we're using the fparser submodule
from the G3 F2PY project for the fortran parsing.  We'll be writing a
cython backend to this parser to allow cython code to call fortran
code.  From the source code for the fparser module:
(https://launchpad.net/f2py/)

"Permission to use, modify, and distribute this software is given under the
terms of the NumPy License. See http://scipy.org.

NO WARRANTY IS EXPRESSED OR IMPLIED.  USE AT YOUR OWN RISK."

The fparser module states it has a "BSD license (revised)."

I see that the f2py page (version 2) has LGPL listed at the top, but
the f2py (again, version 2) packaged with numpy has the same numpy
licensing as listed in the source code.  One of the licenses is out of
date since numpy/scipy explicitly states that they won't package GPL
or LGPL code.

http://scipy.org/License_Compatibility

The numpy/scipy license is apparently BSD-like (see
http://scipy.org/FAQ and http://scipy.org/License_Compatibility).
I've not delved into licensing issues in any great detail, and we'll
have to look at this carefully when considering packaging, etc.

At any rate, we'll work under the assumption that it is a separate dependency.

Kurt

From ellisonbg.net at gmail.com  Wed Apr 22 07:27:52 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 21 Apr 2009 22:27:52 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <85e81ba30904212101n2a3af5edr52d0f86fc636cc84@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
	<EAEECCB6-FFEF-488C-A90F-C82361A5A413@math.washington.edu>
	<85e81ba30904212101n2a3af5edr52d0f86fc636cc84@mail.gmail.com>
Message-ID: <6ce0ac130904212227j1ef89fc7k703f3942c071322a@mail.gmail.com>

> 1. I think there is no way Python would ship LGPL'd code as part of
> standard Python.

I don't know about that, but I would not be surprised if they want the
Python license for all things that end up in the std lib.

> 2. That f2py ships with numpy doesn't imply that f2py isn't GPL
> (though it of course isn't GPL).

True in the formal sense.  The expanded version of my argument goes
something like: the numpy and scipy devs are very careful about
license related things and thus I know they wouldn't violate the GPL
by shipping a combined GPL+BSD work under the BSD license.

> Numpy's license is modified BSD which can be combined with GPL'd code.

...as long as the combined work is released as GPL...

NumPy ships with f2py.  In my mind, that makes NumPy a derived work.
Thus if f2py were GPL, NumPy would be violating the GPL (if NumPy were
still BSD).

Cheers,

Brian




> William
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ellisonbg.net at gmail.com  Wed Apr 22 07:28:43 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 21 Apr 2009 22:28:43 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <c266a7930904212118j969a0c9i94105708b9e4be6a@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
	<c266a7930904212118j969a0c9i94105708b9e4be6a@mail.gmail.com>
Message-ID: <6ce0ac130904212228xcb3ee29x22e375ec0daf96c9@mail.gmail.com>

If there are questions about the exact license of certain parts of
f2py, please ask on the numpy list about this.

Brian

2009/4/21 Kurt Smith <kwmsmith at gmail.com>:
> On Tue, Apr 21, 2009 at 10:21 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>>> Lets not count on doing so for now at least. There are license
>>> considerations to consider too--f2py is GPL and as such wouldn't make
>>> it into Python without relicensing.
>>
>> Mmmm, f2py ships with numpy and thus can't be GPL. ?The home page of
>> f2py lists is as LGPL. ?Now whether or not that is sufficient for
>> Python itself is another question.
>
> Thanks for bringing this up.
>
> We'll have to discuss licensing/packaging issues in more detail later,
> and anything I say is open to correction from those more knowledgeable
> (Pearu, Dag, numpy/scipy/cython devs...).
>
> Our project won't be using the current f2py as exists in the
> numpy/scipy distribution. ?Instead we're using the fparser submodule
> from the G3 F2PY project for the fortran parsing. ?We'll be writing a
> cython backend to this parser to allow cython code to call fortran
> code. ?From the source code for the fparser module:
> (https://launchpad.net/f2py/)
>
> "Permission to use, modify, and distribute this software is given under the
> terms of the NumPy License. See http://scipy.org.
>
> NO WARRANTY IS EXPRESSED OR IMPLIED. ?USE AT YOUR OWN RISK."
>
> The fparser module states it has a "BSD license (revised)."
>
> I see that the f2py page (version 2) has LGPL listed at the top, but
> the f2py (again, version 2) packaged with numpy has the same numpy
> licensing as listed in the source code. ?One of the licenses is out of
> date since numpy/scipy explicitly states that they won't package GPL
> or LGPL code.
>
> http://scipy.org/License_Compatibility
>
> The numpy/scipy license is apparently BSD-like (see
> http://scipy.org/FAQ and http://scipy.org/License_Compatibility).
> I've not delved into licensing issues in any great detail, and we'll
> have to look at this carefully when considering packaging, etc.
>
> At any rate, we'll work under the assumption that it is a separate dependency.
>
> Kurt
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Wed Apr 22 07:43:25 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 22 Apr 2009 07:43:25 +0200
Subject: [Cython] inherit from list
In-Reply-To: <49EE65FA.6080501@canterbury.ac.nz>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz>
Message-ID: <49EEAE7D.2040507@behnel.de>


Greg Ewing wrote:
> Robin Becker wrote:
> 
>> OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
>> from builtin types to be handled automatically as it is in python where list is 
>> both a constructor and a type?
> 
> FWIW, the current version of Pyrex handles inheriting
> from the builtin list type okay, without the need for
> any declarations.

So, when you say

    cdef class mylist(list):
        pass

this will work? I assume this also works for all other (non-PyVarObject)
builtin types?

Will Pyrex handle the above type in any special way because it knows that
mylist is of type 'list', or will it ignore the builtin type specific
optimisations for it?

Stefan

From dagss at student.matnat.uio.no  Wed Apr 22 08:01:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 22 Apr 2009 08:01:02 +0200
Subject: [Cython] download for 0.11.1
In-Reply-To: <e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>	<49EDC2B0.3060700@simula.no>	<49EDC619.5000602@chamonix.reportlab.co.uk>	<49EDCB8A.6040705@student.matnat.uio.no>
	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
Message-ID: <49EEB29E.8080602@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Apr 21, 2009 at 10:35 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> This is exactly why we do it. Sage ships this way and I think it's
>> been a significant factor in attracting developers.
>>
> 
> Sorry, can't resist... Could you tell me why a developer would be
> attracted by this? To make use of ".hg", they need to have mercurial,
> then... Do you think any DEVELOPER will find it cumbersome to do "hg
> clone http://hg.cython.org/cython" ??

I usually appriciate myself when projects do this, because otherwise 
when I encounter something I can fix I must
  a) Go to the website, find the right repo to clone..
  b) Close all my emacs windows and reopen the files in the new location 
and find back to where I was.. (I'm sure there's a macro for that 
somewhere but again, that's overhead)
  c) Transplant any changes I did manually..

The point is that even a) and b) are way too much of an overhead if I'm 
just hacking away to see if I bother to make a patch (of which there is 
perhaps a 10% chance) -- then if I suddenly did it, I'd like to avoid c).


-- 
Dag Sverre

From jf.moulin at gmail.com  Wed Apr 22 07:50:46 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Wed, 22 Apr 2009 07:50:46 +0200
Subject: [Cython] help with function pointer
In-Reply-To: <6D796CC5-B26D-4033-BBF7-AF5AFE62F74D@math.washington.edu>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
	<b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>
	<d416dd90cdcf283d0aae347b41e40d28.squirrel@webmail.uio.no>
	<AC1069B7-A950-492A-A075-9D20FE4EB6A0@math.washington.edu>
	<af6d35810904211405p4a4cca29n44d3e2f9312f09fe@mail.gmail.com>
	<6D796CC5-B26D-4033-BBF7-AF5AFE62F74D@math.washington.edu>
Message-ID: <af6d35810904212250y6b0759c7peb8998848e675d14@mail.gmail.com>

Sounds good and simple indeed...
Could you shortly elaborate on the cost of the pointers (or point me to some
reference ;0)?

Anyway, I'll try that asap (you are right that should be easy to
implement.. I just have to see if
subclassing in Cython is as easy as in Python)

and for sure I'll come with plenty of new questions!
Thanks for your input!
Cheers.
 JF

From robertwb at math.washington.edu  Wed Apr 22 08:30:20 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 23:30:20 -0700
Subject: [Cython] inherit from list
In-Reply-To: <49EEAE7D.2040507@behnel.de>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
Message-ID: <15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>

On Apr 21, 2009, at 10:43 PM, Stefan Behnel wrote:

> Greg Ewing wrote:
>> Robin Becker wrote:
>>
>>> OK thanks. I noticed that all seems to work. Is the intent to  
>>> allow inheriting
>>> from builtin types to be handled automatically as it is in python  
>>> where list is
>>> both a constructor and a type?
>>
>> FWIW, the current version of Pyrex handles inheriting
>> from the builtin list type okay, without the need for
>> any declarations.
>
> So, when you say
>
>     cdef class mylist(list):
>         pass
>
> this will work? I assume this also works for all other (non- 
> PyVarObject)
> builtin types?
>
> Will Pyrex handle the above type in any special way because it  
> knows that
> mylist is of type 'list', or will it ignore the builtin type specific
> optimisations for it?

You have to ignore the type-specific optimizations, because mylist  
(or a subclass thereof) might override the default behavior.

- Robert


From robertwb at math.washington.edu  Wed Apr 22 08:33:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Apr 2009 23:33:17 -0700
Subject: [Cython] help with function pointer
In-Reply-To: <af6d35810904212250y6b0759c7peb8998848e675d14@mail.gmail.com>
References: <af6d35810904200817p62e92366ld616a491bf9b63da@mail.gmail.com>
	<af6d35810904210219y45a12adbr644cbb219a3d6e1d@mail.gmail.com>
	<90EEE860-DAED-474D-A2E3-F1857ED62699@math.washington.edu>
	<af6d35810904210306p7ab1c80bk6f5433074f8eec2a@mail.gmail.com>
	<b2476612f3966aba2541532431243305.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<af6d35810904210633i39bb1b4fs113379a5094c17b7@mail.gmail.com>
	<d416dd90cdcf283d0aae347b41e40d28.squirrel@webmail.uio.no>
	<AC1069B7-A950-492A-A075-9D20FE4EB6A0@math.washington.edu>
	<af6d35810904211405p4a4cca29n44d3e2f9312f09fe@mail.gmail.com>
	<6D796CC5-B26D-4033-BBF7-AF5AFE62F74D@math.washington.edu>
	<af6d35810904212250y6b0759c7peb8998848e675d14@mail.gmail.com>
Message-ID: <D1A9B9D2-731A-4EE2-B5C3-E9F92203D48C@math.washington.edu>

On Apr 21, 2009, at 10:50 PM, Jean-Francois Moulin wrote:

> Sounds good and simple indeed...
> Could you shortly elaborate on the cost of the pointers (or point  
> me to some
> reference ;0)?

I don't have a good reference, but it will probably vary a lot from  
processor to processor (depending on what kind of branch prediction  
algorithms they use--calling a function pointer is a branch of sorts  
too).

> Anyway, I'll try that asap (you are right that should be easy to
> implement.. I just have to see if
> subclassing in Cython is as easy as in Python)

Yep, it is.

> and for sure I'll come with plenty of new questions!
> Thanks for your input!

No problem.

- Robert


From stefan_ml at behnel.de  Wed Apr 22 08:57:26 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 22 Apr 2009 08:57:26 +0200 (CEST)
Subject: [Cython] inherit from list
In-Reply-To: <15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
	<15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
Message-ID: <4779bb0f4c07ca36f4a27b3e7bfbbb68.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Apr 21, 2009, at 10:43 PM, Stefan Behnel wrote:
>>     cdef class mylist(list):
>>         pass
>>
>> Will Pyrex handle the above type in any special way because it
>> knows that mylist is of type 'list', or will it ignore the
>> builtin type specific optimisations for it?
>
> You have to ignore the type-specific optimizations, because mylist
> (or a subclass thereof) might override the default behavior.

As would Cython, currently (so that's the answer I'd expect). That's a
different topic now, but if the subtype does not override __getitem__(),
most list optimisations would work just fine, for example. Same for the
append() optimisation if the type does not override append() or
__getattr(ibute)__().

Now that we have a transform for builtin type method optimisations anyway,
we could check the type definition itself, instead of restricting it to
the plain builtin types.

One reason more to fix Cython here so that we can make builtin type
redefinition a real error.

Stefan


From dagss at student.matnat.uio.no  Wed Apr 22 10:25:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 22 Apr 2009 10:25:05 +0200
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49EED461.9090705@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> 2) Code and patch workflow.
>>
>> On the (A) side, the status with f2py is that the author started a full
>> rewrite (f2py 3G) which hasn't seen work for half a year
>> (http://f2py.org/). So anything can happen here and we have to get back
>> to how it's going to be done and where it should live.
> 
> Sounds like the right thing to ask the author first.

This is the current consensus between us and Pearu:

He considers f2py G3 still in development, he just doesn't have time for 
it at the time. And he's noticed interest in the parser component, 
"fparser", from other parties as well, and has seperated it out in a 
seperate module.

Myself and Kurt will get access to f2py G3's Launchpad project and use 
f2py's repo, tracker and mailing list for any work that is strictly 
related to fparser, in cooperation with Pearu.

It seems then that the fparser -> Cython part of it will live in the 
Cython project and use Cython's tracker etc.

(The reason I don't think it is premature to deal with this now is that 
I consider it important that discussions happen in the right places, and 
the coming month is the one that's going to be most discussion-intensive.)


-- 
Dag Sverre

From stefan_ml at behnel.de  Wed Apr 22 11:21:35 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 22 Apr 2009 11:21:35 +0200 (CEST)
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <49EED461.9090705@student.matnat.uio.no>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EED461.9090705@student.matnat.uio.no>
Message-ID: <60c4fbcf7c2deb5c4edbf8e0310a30f9.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> This is the current consensus between us and Pearu:
>
> He considers f2py G3 still in development, he just doesn't have time for
> it at the time. And he's noticed interest in the parser component,
> "fparser", from other parties as well, and has seperated it out in a
> seperate module.
>
> Myself and Kurt will get access to f2py G3's Launchpad project and use
> f2py's repo, tracker and mailing list for any work that is strictly
> related to fparser, in cooperation with Pearu.
>
> It seems then that the fparser -> Cython part of it will live in the
> Cython project and use Cython's tracker etc.

That sounds like a suitable separation. It also avoids any licensing issues.

So Cython would depend on the separate fparser package only if a user
wants to interface with Fortran code.

Stefan


From robin at reportlab.com  Wed Apr 22 13:28:57 2009
From: robin at reportlab.com (Robin Becker)
Date: Wed, 22 Apr 2009 12:28:57 +0100
Subject: [Cython] inherit from list
In-Reply-To: <49EE65FA.6080501@canterbury.ac.nz>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz>
Message-ID: <49EEFF79.7090508@chamonix.reportlab.co.uk>

Greg Ewing wrote:
> Robin Becker wrote:
> 
>> OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
>> from builtin types to be handled automatically as it is in python where list is 
>> both a constructor and a type?
> 
> FWIW, the current version of Pyrex handles inheriting
> from the builtin list type okay, without the need for
> any declarations.
> 
yes this code is a bit out of date.

-- 
Robin Becker

From stefan_ml at behnel.de  Wed Apr 22 14:59:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 22 Apr 2009 14:59:27 +0200 (CEST)
Subject: [Cython] inherit from list
In-Reply-To: <49EEFF79.7090508@chamonix.reportlab.co.uk>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz>
	<49EEFF79.7090508@chamonix.reportlab.co.uk>
Message-ID: <0078405dad363fd8c2fe36b6533917f2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robin Becker wrote:
> Greg Ewing wrote:
>> FWIW, the current version of Pyrex handles inheriting
>> from the builtin list type okay, without the need for
>> any declarations.
>>
> yes this code is a bit out of date.

Not out of date (in the sense that it hasn't been worked on), but lacking
a port. This Pyrex feature dates back to March 2008 (Pyrex 0.9.7, hg rev.
26-37). It won't be trivial to port, and from a quick glance, it looks a
bit like a clever hack. It will likely interfere with Cython's builtin
caching feature, although that might become redundant in many cases.
True/False handling is another thing that works different in Cython now,
so that must be left out.

I also wonder how exception subtyping is handled for Python versions
before 2.5. I assume this is left to the user, so that

    cdef class MyExc(Exception): pass

will require Py2.5+, whereas

    class MyExc(Exception): pass

should still work in Py2.3.

It's too bad that Pyrex's test suite doesn't really test anything. It will
be a bit of work to add suitable test cases to Cython.

Stefan


From robin at reportlab.com  Wed Apr 22 15:39:38 2009
From: robin at reportlab.com (Robin Becker)
Date: Wed, 22 Apr 2009 14:39:38 +0100
Subject: [Cython] inherit from list
In-Reply-To: <0078405dad363fd8c2fe36b6533917f2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49EDF6F0.504@chamonix.reportlab.co.uk>	<49EE65FA.6080501@canterbury.ac.nz>	<49EEFF79.7090508@chamonix.reportlab.co.uk>
	<0078405dad363fd8c2fe36b6533917f2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49EF1E1A.5070706@chamonix.reportlab.co.uk>

Stefan Behnel wrote:
> Robin Becker wrote:
>> Greg Ewing wrote:
>>> FWIW, the current version of Pyrex handles inheriting
>>> from the builtin list type okay, without the need for
>>> any declarations.
>>>
>> yes this code is a bit out of date.
> 



> Not out of date (in the sense that it hasn't been worked on), but lacking
> a port. This Pyrex feature dates back to March 2008 (Pyrex 0.9.7, hg rev.
> 26-37). It won't be trivial to port, and from a quick glance, it looks a
> bit like a clever hack. It will likely interfere with Cython's builtin
> caching feature, although that might become redundant in many cases.
> True/False handling is another thing that works different in Cython now,
> so that must be left out.
> 


you mistake me :) I was referring to my old pyrex compatible code; I wrote it 
about two years ago and pyrex has advanced since then so apparently I don't need 
the declarations with modern pyrex.

So far as I'm aware cython is advancing wave upon wave and will eventually win 
great victories.

.........

-- 
Robin Becker

From cournape at gmail.com  Wed Apr 22 16:39:05 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 22 Apr 2009 23:39:05 +0900
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <85e81ba30904212101n2a3af5edr52d0f86fc636cc84@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
	<EAEECCB6-FFEF-488C-A90F-C82361A5A413@math.washington.edu>
	<85e81ba30904212101n2a3af5edr52d0f86fc636cc84@mail.gmail.com>
Message-ID: <5b8d13220904220739q1965e66cua99ba9b0e700d768@mail.gmail.com>

On Wed, Apr 22, 2009 at 1:01 PM, William Stein <wstein at gmail.com> wrote:
> 2009/4/21 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Apr 21, 2009, at 8:21 PM, Brian Granger wrote:
>>
>>>> Lets not count on doing so for now at least. There are license
>>>> considerations to consider too--f2py is GPL and as such wouldn't make
>>>> it into Python without relicensing.
>>>
>>> Mmmm, f2py ships with numpy and thus can't be GPL. ?The home page of
>>> f2py lists is as LGPL. ?Now whether or not that is sufficient for
>>> Python itself is another question.
>>
>> Yes, sorry, LGPL. In any case, lets treat it as an external
>> dependancy for now.
>
> 1. I think there is no way Python would ship LGPL'd code as part of
> standard Python.
>
> 2. That f2py ships with numpy doesn't imply that f2py isn't GPL
> (though it of course isn't GPL).

It indeed does not imply anything legally, but we would rather not
include any shipped code which is not BSD. I was actually very
surprised to learn that f2py was not BSD-like, and I checked the
sources of f2py as shipped in numpy: those are explicitly covered by
the Numpy license, as can be seen by numpy/f2py/docs/README.txt:

.. -*- rest -*-

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 F2PY: Fortran to Python interface generator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:Author: Pearu Peterson <pearu at cens.ioc.ee>
:License: NumPy License
:Web-site: http://cens.ioc.ee/projects/f2py2e/
:Discussions to: `f2py-users mailing list`_
:Documentation: `User's Guide`__, FAQ__
:Platforms: All
:Date: $Date: 2005/01/30 18:54:53 $

...

Davd

From dagss at student.matnat.uio.no  Wed Apr 22 18:19:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 22 Apr 2009 18:19:08 +0200
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
Message-ID: <49EF437C.90605@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
> 
>> I'm in need of fast computations on complex numbers from Cython.  
> 
> Yes, I've started this, as it comes up in my research too (number  
> theory, computing values of L-functions). I hope to have something  
> releasable soon. In the meantime, you can do

Is there anything I can do to help you with this? (Or perhaps take over, 
if you now prioritize closures higher.)

I'm holding a full-day tutorial on scientific use of Cython, 18th of 
May, and I'd feel less embarassed if this was in a release by then, so 
that's a nice goal, although not all-overriding of course :-) But my 
recent focus on correct struct support with NumPy could certainly be 
shifted below complex numbers.

-- 
Dag Sverre

From robert.kern at gmail.com  Wed Apr 22 22:31:18 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 22 Apr 2009 15:31:18 -0500
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <6ce0ac130904212228xcb3ee29x22e375ec0daf96c9@mail.gmail.com>
References: <49EDB668.7030502@student.matnat.uio.no>	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>	<c266a7930904212118j969a0c9i94105708b9e4be6a@mail.gmail.com>
	<6ce0ac130904212228xcb3ee29x22e375ec0daf96c9@mail.gmail.com>
Message-ID: <gsnuqn$j9v$1@ger.gmane.org>

On 2009-04-22 00:28, Brian Granger wrote:
> If there are questions about the exact license of certain parts of
> f2py, please ask on the numpy list about this.

This page is very old:

   http://cens.ioc.ee/projects/f2py2e/

Pay no attention to it. It refers to a previous incarnation of f2py.

When Pearu integrated f2py into numpy, he did so under numpy's BSD license. 
Additionally, his G3F2PY work is also under a BSD license:

   https://launchpad.net/f2py

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ellisonbg.net at gmail.com  Wed Apr 22 22:36:15 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 22 Apr 2009 13:36:15 -0700
Subject: [Cython] Fortran GSoC accepted
In-Reply-To: <gsnuqn$j9v$1@ger.gmane.org>
References: <49EDB668.7030502@student.matnat.uio.no>
	<d86a3133f5a7b216f98fb14045e6f6f6.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<c266a7930904211042o373f4356r546ccabbfe392d3d@mail.gmail.com>
	<B39BC9BA-0F4B-46ED-A2A8-EEA5055A3E8E@math.washington.edu>
	<6ce0ac130904212021q26da8adag2b1bb9955c963838@mail.gmail.com>
	<c266a7930904212118j969a0c9i94105708b9e4be6a@mail.gmail.com>
	<6ce0ac130904212228xcb3ee29x22e375ec0daf96c9@mail.gmail.com>
	<gsnuqn$j9v$1@ger.gmane.org>
Message-ID: <6ce0ac130904221336x289f722ei6c61186e492251d3@mail.gmail.com>

> Pay no attention to it. It refers to a previous incarnation of f2py.
>
> When Pearu integrated f2py into numpy, he did so under numpy's BSD license.
> Additionally, his G3F2PY work is also under a BSD license:
>
>   https://launchpad.net/f2py


Thanks Robert for the clarification!

Brian

From dalcinl at gmail.com  Thu Apr 23 01:03:49 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Apr 2009 20:03:49 -0300
Subject: [Cython] weird difference in Py 2.3&2.4 versus 2.5 and above
	regarding __int__()/__long__() for old-style (aka classic)
	classes...
Message-ID: <e7ba66e40904221603h477bb6d5gf3a5f07612921c68@mail.gmail.com>

I do not have the time do dive deep enough to figure out the root of
this difference, but here you have a patch that modifies a testcase to
exercise the problem, and a quick fix for __Pyx_PyNumber_Int fixing
the issue.

If there are no objections, I'll push the fix to cython-devel... As
this is related to rather old Python versions, I'm being a bit lazy as
to re-openi #255, I would just push the fix :-)


diff -r b550ee890c72 Cython/Compiler/PyrexTypes.py
--- a/Cython/Compiler/PyrexTypes.py	Tue Apr 21 19:43:22 2009 -0700
+++ b/Cython/Compiler/PyrexTypes.py	Wed Apr 22 19:52:13 2009 -0300
@@ -1524,7 +1524,13 @@
     return Py_INCREF(x), x;
   m = Py_TYPE(x)->tp_as_number;
 #if PY_VERSION_HEX < 0x03000000
+#if PY_VERSION_HEX < 0x02050000
+  if (m && m->nb_long &&
+      (!PyInstance_Check(x) ||
+       PyObject_HasAttrString(x, (char*)"__long__"))) {
+#else
   if (m && m->nb_long) {
+#endif
     name = "long";
     res = PyNumber_Long(x);
   }
diff -r b550ee890c72 tests/run/c_int_types_T255.pyx
--- a/tests/run/c_int_types_T255.pyx	Tue Apr 21 19:43:22 2009 -0700
+++ b/tests/run/c_int_types_T255.pyx	Wed Apr 22 19:52:13 2009 -0300
@@ -655,7 +655,7 @@
 # -------------------------------------------------------------------
 import sys

-class MyInt(object):
+class MyInt:
    def __init__(self, value):
       self.value = value
    def __int__(self):


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Apr 23 01:49:29 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Apr 2009 11:49:29 +1200
Subject: [Cython] download for 0.11.1
In-Reply-To: <e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
	<49EDC2B0.3060700@simula.no>
	<49EDC619.5000602@chamonix.reportlab.co.uk>
	<49EDCB8A.6040705@student.matnat.uio.no>
	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
Message-ID: <49EFAD09.2030104@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Sorry, can't resist... Could you tell me why a developer would be
> attracted by this? To make use of ".hg", they need to have mercurial,
> then... Do you think any DEVELOPER will find it cumbersome to do "hg
> clone http://hg.cython.org/cython" ??

Also I think it's a bit impolite to force all the people
who aren't developers to download twice as much data as
they're going to use.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Apr 23 02:07:52 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Apr 2009 12:07:52 +1200
Subject: [Cython] inherit from list
In-Reply-To: <49EEAE7D.2040507@behnel.de>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
Message-ID: <49EFB158.2000905@canterbury.ac.nz>

Stefan Behnel wrote:

>     cdef class mylist(list):
>         pass
> 
> this will work?

I haven't tested it extensively, but the above certainly
compiles, and I don't know of any reason why it shouldn't
work. Pyrex just preloads the symbol table as if you had
written an external declaration for type 'list', so the
end result should be the same.

> I assume this also works for all other (non-PyVarObject)
> builtin types?

It should work for the ones Pyrex knows about. You will
need to provide your own declarations for any others.

> Will Pyrex handle the above type in any special way because it knows that
> mylist is of type 'list'

It *should*, but trying it just now, there seems to be
a bug... I'll see about fixing it.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Thu Apr 23 02:20:12 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Apr 2009 12:20:12 +1200
Subject: [Cython] inherit from list
In-Reply-To: <15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
	<15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
Message-ID: <49EFB43C.9010709@canterbury.ac.nz>

Robert Bradshaw wrote:

> You have to ignore the type-specific optimizations, because mylist  
> (or a subclass thereof) might override the default behavior.

That's somewhat tricky in Pyrex, because it makes its
optimisation decisions based on the static type of the
object. So if you pass your list subclass to something
declared as taking a list, the list optimisastions will
get done on it anyway.

I'm not sure what's the best thing to do here -- only
do the optimisations for something declared as exactly
a list, or for any subclass of list.

-- 
Greg

From dalcinl at gmail.com  Thu Apr 23 02:29:13 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Apr 2009 21:29:13 -0300
Subject: [Cython] download for 0.11.1
In-Reply-To: <49EFAD09.2030104@canterbury.ac.nz>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
	<49EDC2B0.3060700@simula.no>
	<49EDC619.5000602@chamonix.reportlab.co.uk>
	<49EDCB8A.6040705@student.matnat.uio.no>
	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
	<49EFAD09.2030104@canterbury.ac.nz>
Message-ID: <e7ba66e40904221729x7882e079v2a92bd04af8e9318@mail.gmail.com>

On Wed, Apr 22, 2009 at 8:49 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> Sorry, can't resist... Could you tell me why a developer would be
>> attracted by this? To make use of ".hg", they need to have mercurial,
>> then... Do you think any DEVELOPER will find it cumbersome to do "hg
>> clone http://hg.cython.org/cython" ??
>
> Also I think it's a bit impolite to force all the people
> who aren't developers to download twice as much data as
> they're going to use.
>

Moreover, I frequently use easy_install when I need a package in a
hurry. In such cases, tarball releases are downloaded, the package
installed, and next the tarball is just thrown away...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Apr 23 02:31:53 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Apr 2009 21:31:53 -0300
Subject: [Cython] inherit from list
In-Reply-To: <49EFB43C.9010709@canterbury.ac.nz>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
	<15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
	<49EFB43C.9010709@canterbury.ac.nz>
Message-ID: <e7ba66e40904221731y3e9a7defr4933f026262031e4@mail.gmail.com>

On Wed, Apr 22, 2009 at 9:20 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Robert Bradshaw wrote:
>
>> You have to ignore the type-specific optimizations, because mylist
>> (or a subclass thereof) might override the default behavior.
>
> I'm not sure what's the best thing to do here -- only
> do the optimisations for something declared as exactly
> a list, or for any subclass of list.
>

If you dive into core CPython sources, I would not be surprised to
find places where PyList_Check() is done and PyList_{Set|Get}Item() is
next called.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Apr 23 02:35:54 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Apr 2009 17:35:54 -0700
Subject: [Cython] inherit from list
In-Reply-To: <49EFB43C.9010709@canterbury.ac.nz>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
	<15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
	<49EFB43C.9010709@canterbury.ac.nz>
Message-ID: <F3F61AB7-72D9-4BFA-A6DF-9C379D3BD646@math.washington.edu>

On Apr 22, 2009, at 5:20 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> You have to ignore the type-specific optimizations, because mylist
>> (or a subclass thereof) might override the default behavior.
>
> That's somewhat tricky in Pyrex, because it makes its
> optimisation decisions based on the static type of the
> object. So if you pass your list subclass to something
> declared as taking a list, the list optimisastions will
> get done on it anyway.
>
> I'm not sure what's the best thing to do here -- only
> do the optimisations for something declared as exactly
> a list, or for any subclass of list.

In Cython we only allow None or exactly a list to be assigned to a  
cdef list variable, not subclass thereof. We discussed this when we  
introduced it, and I am still convinced it is the right thing to do,  
but perhaps this makes subclassing messier.

- Robert


From greg.ewing at canterbury.ac.nz  Thu Apr 23 02:59:31 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Apr 2009 12:59:31 +1200
Subject: [Cython] inherit from list
In-Reply-To: <0078405dad363fd8c2fe36b6533917f2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz>
	<49EEFF79.7090508@chamonix.reportlab.co.uk>
	<0078405dad363fd8c2fe36b6533917f2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49EFBD73.7010205@canterbury.ac.nz>

Stefan Behnel wrote:

> I also wonder how exception subtyping is handled for Python versions
> before 2.5. I assume this is left to the user, so that
> 
>     cdef class MyExc(Exception): pass
> 
> will require Py2.5+, whereas
> 
>     class MyExc(Exception): pass
> 
> should still work in Py2.3.

Yes. Do you think it can/should be handled differently?

-- 
Greg

From ndbecker2 at gmail.com  Thu Apr 23 03:14:54 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 22 Apr 2009 21:14:54 -0400
Subject: [Cython] Status of C complex types (re-send)
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<49EF437C.90605@student.matnat.uio.no>
Message-ID: <gsofee$vg4$3@ger.gmane.org>

Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>> 
>>> I'm in need of fast computations on complex numbers from Cython.
>> 
>> Yes, I've started this, as it comes up in my research too (number
>> theory, computing values of L-functions). I hope to have something
>> releasable soon. In the meantime, you can do
> 
> Is there anything I can do to help you with this? (Or perhaps take over,
> if you now prioritize closures higher.)
> 
> I'm holding a full-day tutorial on scientific use of Cython, 18th of
> May, and I'd feel less embarassed if this was in a release by then, so
> that's a nice goal, although not all-overriding of course :-) But my
> recent focus on correct struct support with NumPy could certainly be
> shifted below complex numbers.
> 

Any chance of making this tutorial publically available?



From greg.ewing at canterbury.ac.nz  Thu Apr 23 03:37:56 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Apr 2009 13:37:56 +1200
Subject: [Cython] inherit from list
In-Reply-To: <e7ba66e40904221731y3e9a7defr4933f026262031e4@mail.gmail.com>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
	<15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
	<49EFB43C.9010709@canterbury.ac.nz>
	<e7ba66e40904221731y3e9a7defr4933f026262031e4@mail.gmail.com>
Message-ID: <49EFC674.2080008@canterbury.ac.nz>

Lisandro Dalcin wrote:

> If you dive into core CPython sources, I would not be surprised to
> find places where PyList_Check() is done and PyList_{Set|Get}Item() is
> next called.

But that's a dynamic check, which might be all right
for Cython, but it's not the way Pyrex does things.

-- 
Greg

From dalcinl at gmail.com  Thu Apr 23 03:37:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Apr 2009 22:37:22 -0300
Subject: [Cython] inherit from list
In-Reply-To: <49EFBD73.7010205@canterbury.ac.nz>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz>
	<49EEFF79.7090508@chamonix.reportlab.co.uk>
	<0078405dad363fd8c2fe36b6533917f2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EFBD73.7010205@canterbury.ac.nz>
Message-ID: <e7ba66e40904221837r42ff8d93o91f7104ff1515a64@mail.gmail.com>

On Wed, Apr 22, 2009 at 9:59 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Stefan Behnel wrote:
>
>> I also wonder how exception subtyping is handled for Python versions
>> before 2.5. I assume this is left to the user, so that
>>
>> ? ? cdef class MyExc(Exception): pass
>>
>> will require Py2.5+, whereas
>>
>> ? ? class MyExc(Exception): pass
>>
>> should still work in Py2.3.
>
> Yes. Do you think it can/should be handled differently?
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

AFAIK, you cannot handle it differently... old-style, aka classic
classes are actually instances (of types.ClassType), so a cdef class
class (implicitly new-style) cannot inherit from an instance, very
much like "class A([]): pass" ends-up failing in core Py2.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Apr 23 03:43:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Apr 2009 22:43:17 -0300
Subject: [Cython] inherit from list
In-Reply-To: <49EFC674.2080008@canterbury.ac.nz>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
	<15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
	<49EFB43C.9010709@canterbury.ac.nz>
	<e7ba66e40904221731y3e9a7defr4933f026262031e4@mail.gmail.com>
	<49EFC674.2080008@canterbury.ac.nz>
Message-ID: <e7ba66e40904221843k16308cb5w8af769b9bbaca8d8@mail.gmail.com>

On Wed, Apr 22, 2009 at 10:37 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> If you dive into core CPython sources, I would not be surprised to
>> find places where PyList_Check() is done and PyList_{Set|Get}Item() is
>> next called.
>
> But that's a dynamic check, which might be all right
> for Cython, but it's not the way Pyrex does things.
>

Did you noticed I'm talking about core CPython, not Cython?

Anyway, IMHO, Cython/Pyrex should optimize ONLY if PyList_CheckExact()
returns 1... Just to avoid surprises...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lodatom at gmail.com  Thu Apr 23 04:18:20 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Wed, 22 Apr 2009 22:18:20 -0400
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>
	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>
	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>
	<CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>
Message-ID: <ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>

The embedded mode is a really good idea, but I think it would make
more sense as a separate script for several reasons:
1. If it is separate, Cython itself doesn't have to change.
2. The same Cython C output file be made into a module and also embedded.
3. Multiple Cython modules can be embedded into a single executable.

I have included a first draft of such a script.  It is a modification
of the embedded mode that was checked into cython-devel.  Notably, I
use PyImport_ExtendInittab to register all the modules as built-ins,
and I only import the first one listed (which would ideally be called
"__main__"; see below).  This seems to be a cleaner way to do things.
This also means that any extra module listed on the command-line that
is not imported by the first module will not be run.

Other (minor) changes:
* I call Py_SetProgramName(), which seems like a good idea.
* I create a MODINIT() macro to create the module initialization name,
but I don't know how portable the ## concatenation is.

Please realize this is a first draft, and thus I am simply printing to
standard output.  I hope this can be of use to someone.



An additional idea (not implemented) - it would be awesome to have the
first module listed (that is, the one directly imported) to have
__name__ == "__main__", so the normal Python idiom works.  I
researched this for the past few hours, but I could not find a way to
do this without modifying the module's code.  My best solution is the
following:
1. For each module, mymod.pyx:
1a. Add a new externally-visible global variable:
    int __pyx_mymod_is___main__ = 0;
1b. When converting the code for __name__, if __pyx_mymod_is___main__,
return "__main__", else the module name.
2. In the script below, set
    __pyx_mainmod_is___main__ = 1;
right before the call to PyImport_ImportModule(), where "mainmod" is
the main module.

I tried the above by modifying the Cython C output by hand and it
seemed to work OK.



Quick question, is the following a typo?
        if (Py_FlushLine()); PyErr_Clear();
Should that middle semi-colon be there?  If so, why the if statement?



--Mark




-------------------------- cython_freeze.py --------------------
#!/usr/bin/env python

import sys

if len(sys.argv) < 2:
    print >>sys.stderr, "USAGE: %s module [module ...]" % sys.argv[0]
    sys.exit(1)

print """
#include <Python.h>
#include <stdio.h>
#include <stdlib.h>

#if PY_MAJOR_VERSION < 3
# define MODINIT(name)  init ## name
#else
# define MODINIT(name)  PyInit_ ## name
#endif
"""

for name in sys.argv[1:]:
    print "PyMODINIT_FUNC MODINIT(%s) (void);" % name

print """
static struct _inittab inittab[] = {"""

for name in sys.argv[1:]:
    print '    {"%(name)s", MODINIT(%(name)s)},' % {'name' : name}

print """    {NULL, NULL}
};

int main(int argc, char** argv) {
    int r = 0;
    PyObject *m = NULL;
    if (PyImport_ExtendInittab(inittab)) {
        fprintf(stderr, "No memory\\n");
        exit(1);
    }
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    PySys_SetArgv(argc, argv);
    m = PyImport_ImportModule(inittab[0].name);
    if (!m) {
        r = 1;
        PyErr_Print(); /* This exits with the right code if SystemExit. */
        if (Py_FlushLine()); PyErr_Clear();
    }
    Py_XDECREF(m);
    Py_Finalize();
    return r;
}
"""

From robertwb at math.washington.edu  Thu Apr 23 05:55:37 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Apr 2009 20:55:37 -0700
Subject: [Cython] inherit from list
In-Reply-To: <e7ba66e40904221843k16308cb5w8af769b9bbaca8d8@mail.gmail.com>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>
	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EDF6F0.504@chamonix.reportlab.co.uk>
	<49EE65FA.6080501@canterbury.ac.nz> <49EEAE7D.2040507@behnel.de>
	<15258211-C631-4432-9325-C2597ECBC477@math.washington.edu>
	<49EFB43C.9010709@canterbury.ac.nz>
	<e7ba66e40904221731y3e9a7defr4933f026262031e4@mail.gmail.com>
	<49EFC674.2080008@canterbury.ac.nz>
	<e7ba66e40904221843k16308cb5w8af769b9bbaca8d8@mail.gmail.com>
Message-ID: <264E3E72-55AB-403F-BF97-265E790013CF@math.washington.edu>

On Apr 22, 2009, at 6:43 PM, Lisandro Dalcin wrote:

> On Wed, Apr 22, 2009 at 10:37 PM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
>> Lisandro Dalcin wrote:
>>
>>> If you dive into core CPython sources, I would not be surprised to
>>> find places where PyList_Check() is done and PyList_{Set|Get}Item 
>>> () is
>>> next called.
>>
>> But that's a dynamic check, which might be all right
>> for Cython, but it's not the way Pyrex does things.

Cython checks the type on assignment, not every time it gets accessed  
(just like extension types).

> Did you noticed I'm talking about core CPython, not Cython?
>
> Anyway, IMHO, Cython/Pyrex should optimize ONLY if PyList_CheckExact()
> returns 1... Just to avoid surprises...
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Thu Apr 23 06:26:49 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Apr 2009 06:26:49 +0200 (CEST)
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <gsofee$vg4$3@ger.gmane.org>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<49EF437C.90605@student.matnat.uio.no> <gsofee$vg4$3@ger.gmane.org>
Message-ID: <506795580178627c462474966d2e59bc.squirrel@webmail.uio.no>

Neal Becker wrote:
> Dag Sverre Seljebotn wrote:
>
>> Robert Bradshaw wrote:
>>> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>>>
>>>> I'm in need of fast computations on complex numbers from Cython.
>>>
>>> Yes, I've started this, as it comes up in my research too (number
>>> theory, computing values of L-functions). I hope to have something
>>> releasable soon. In the meantime, you can do
>>
>> Is there anything I can do to help you with this? (Or perhaps take over,
>> if you now prioritize closures higher.)
>>
>> I'm holding a full-day tutorial on scientific use of Cython, 18th of
>> May, and I'd feel less embarassed if this was in a release by then, so
>> that's a nice goal, although not all-overriding of course :-) But my
>> recent focus on correct struct support with NumPy could certainly be
>> shifted below complex numbers.
>>
>
> Any chance of making this tutorial publically available?

The written material/slides: Sure, I just need to write it first :-)

I don't know whether they will video tape it, but it's going to be over
4-5 hours so nobody would bother to watch it anyway (i.e. probably no
video).

http://www.notur.no/notur2009/programme.html

Dag Sverre


From stefan_ml at behnel.de  Thu Apr 23 07:13:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Apr 2009 07:13:12 +0200
Subject: [Cython] inherit from list
In-Reply-To: <49EFBD73.7010205@canterbury.ac.nz>
References: <49EDCE90.2050604@chamonix.reportlab.co.uk>	<36cc114ef2dc49b452300d16e4a7660d.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49EDF6F0.504@chamonix.reportlab.co.uk>	<49EE65FA.6080501@canterbury.ac.nz>	<49EEFF79.7090508@chamonix.reportlab.co.uk>	<0078405dad363fd8c2fe36b6533917f2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49EFBD73.7010205@canterbury.ac.nz>
Message-ID: <49EFF8E8.6020408@behnel.de>


Greg Ewing wrote:
> Stefan Behnel wrote:
> 
>> I also wonder how exception subtyping is handled for Python versions
>> before 2.5. I assume this is left to the user, so that
>>
>>     cdef class MyExc(Exception): pass
>>
>> will require Py2.5+, whereas
>>
>>     class MyExc(Exception): pass
>>
>> should still work in Py2.3.
> 
> Yes. Do you think it can/should be handled differently?

No, that's what I expected. Cython doesn't currently handle exception types
any special, so it's not a problem here (unless users declare the exception
types themselves).

I think the behaviour of Pyrex makes sense here. After all, it's a user's
decision if speed is more important than backwards portability for exceptions.

I would expect a compile error in that case, though, like Cython does in a
couple of other cases. Something like a conditional

"""
#if PY_VERSION_HEX < 0x0205000000
#error This module requires new style exceptions, which are only \
       available  in Python 2.5 or later.
#endif
"""

Stefan

From stefan_ml at behnel.de  Thu Apr 23 07:16:09 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Apr 2009 07:16:09 +0200
Subject: [Cython] download for 0.11.1
In-Reply-To: <e7ba66e40904221729x7882e079v2a92bd04af8e9318@mail.gmail.com>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>	<49EDC2B0.3060700@simula.no>	<49EDC619.5000602@chamonix.reportlab.co.uk>	<49EDCB8A.6040705@student.matnat.uio.no>	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>	<49EFAD09.2030104@canterbury.ac.nz>
	<e7ba66e40904221729x7882e079v2a92bd04af8e9318@mail.gmail.com>
Message-ID: <49EFF999.90307@behnel.de>


Lisandro Dalcin wrote:
> On Wed, Apr 22, 2009 at 8:49 PM, Greg Ewing wrote:
>> Lisandro Dalcin wrote:
>>
>>> Sorry, can't resist... Could you tell me why a developer would be
>>> attracted by this? To make use of ".hg", they need to have mercurial,
>>> then... Do you think any DEVELOPER will find it cumbersome to do "hg
>>> clone http://hg.cython.org/cython" ??
>> Also I think it's a bit impolite to force all the people
>> who aren't developers to download twice as much data as
>> they're going to use.
> 
> Moreover, I frequently use easy_install when I need a package in a
> hurry. In such cases, tarball releases are downloaded, the package
> installed, and next the tarball is just thrown away...

What about shipping two distributions from PyPI then? One called "Cython"
that contains only the installable sources, and one called
"Cython-complete" that easy_install will not use due to its different name,
and that additionally contains the .hg directory.

Stefan

From stefan_ml at behnel.de  Thu Apr 23 07:33:15 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Apr 2009 07:33:15 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>	<CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>
	<ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>
Message-ID: <49EFFD9B.3070200@behnel.de>

Hi,

Mark Lodato wrote:
> The embedded mode is a really good idea, but I think it would make
> more sense as a separate script for several reasons:
> 1. If it is separate, Cython itself doesn't have to change.
> 2. The same Cython C output file be made into a module and also embedded.
> 3. Multiple Cython modules can be embedded into a single executable.

Yep, that's what I thought, too.


> I have included a first draft of such a script.  It is a modification
> of the embedded mode that was checked into cython-devel.  Notably, I
> use PyImport_ExtendInittab to register all the modules as built-ins,

1) does that work for modules in packages?

2) will this decref the modules in Py3?


> and I only import the first one listed (which would ideally be called
> "__main__"; see below). This seems to be a cleaner way to do things.
> This also means that any extra module listed on the command-line that
> is not imported by the first module will not be run.

When is the init code of the modules called, and in which order? The main
module init code should be able to import and use all other modules, even
before executing its own init code.


> Other (minor) changes:
> * I call Py_SetProgramName(), which seems like a good idea.

Yes.


> * I create a MODINIT() macro to create the module initialization name,
> but I don't know how portable the ## concatenation is.

Also, the functions return different things in Py2 and Py3, so they are not
easily interchangeable.


> Please realize this is a first draft, and thus I am simply printing to
> standard output.  I hope this can be of use to someone.

Sure, thanks!


> An additional idea (not implemented) - it would be awesome to have the
> first module listed (that is, the one directly imported) to have
> __name__ == "__main__", so the normal Python idiom works.  I
> researched this for the past few hours, but I could not find a way to
> do this without modifying the module's code.  My best solution is the
> following:
> 1. For each module, mymod.pyx:
> 1a. Add a new externally-visible global variable:
>     int __pyx_mymod_is___main__ = 0;
> 1b. When converting the code for __name__, if __pyx_mymod_is___main__,
> return "__main__", else the module name.
> 2. In the script below, set
>     __pyx_mainmod_is___main__ = 1;
> right before the call to PyImport_ImportModule(), where "mainmod" is
> the main module.
> 
> I tried the above by modifying the Cython C output by hand and it
> seemed to work OK.

I thought about special casing the

    if __name__ == '__main__':

paradigm so that it generates a function with only global variables instead
that would be callable from the embedding main(). In that case, we'd have a
problem with code like this:

    a = 1
    if __name__ == '__main__':
        a = 2
    print a

So I think that we'd need something like you describe.

BTW, my impression is that this isn't ready for cython-devel yet, given the
amount of open questions and design choices.

Stefan


From stefan_ml at behnel.de  Thu Apr 23 07:37:29 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Apr 2009 07:37:29 +0200
Subject: [Cython] weird difference in Py 2.3&2.4 versus 2.5 and above
 regarding __int__()/__long__() for old-style (aka classic)	classes...
In-Reply-To: <e7ba66e40904221603h477bb6d5gf3a5f07612921c68@mail.gmail.com>
References: <e7ba66e40904221603h477bb6d5gf3a5f07612921c68@mail.gmail.com>
Message-ID: <49EFFE99.80301@behnel.de>

Hi Lisandro,

I can't currently comment on the change itself, but I would prefer
duplicating the relevant part of the test case instead of replacing it, so
that we test both types of classes.

Stefan


Lisandro Dalcin wrote:
> I do not have the time do dive deep enough to figure out the root of
> this difference, but here you have a patch that modifies a testcase to
> exercise the problem, and a quick fix for __Pyx_PyNumber_Int fixing
> the issue.
> 
> If there are no objections, I'll push the fix to cython-devel... As
> this is related to rather old Python versions, I'm being a bit lazy as
> to re-openi #255, I would just push the fix :-)
> 
> 
> diff -r b550ee890c72 Cython/Compiler/PyrexTypes.py
> --- a/Cython/Compiler/PyrexTypes.py	Tue Apr 21 19:43:22 2009 -0700
> +++ b/Cython/Compiler/PyrexTypes.py	Wed Apr 22 19:52:13 2009 -0300
> @@ -1524,7 +1524,13 @@
>      return Py_INCREF(x), x;
>    m = Py_TYPE(x)->tp_as_number;
>  #if PY_VERSION_HEX < 0x03000000
> +#if PY_VERSION_HEX < 0x02050000
> +  if (m && m->nb_long &&
> +      (!PyInstance_Check(x) ||
> +       PyObject_HasAttrString(x, (char*)"__long__"))) {
> +#else
>    if (m && m->nb_long) {
> +#endif
>      name = "long";
>      res = PyNumber_Long(x);
>    }
> diff -r b550ee890c72 tests/run/c_int_types_T255.pyx
> --- a/tests/run/c_int_types_T255.pyx	Tue Apr 21 19:43:22 2009 -0700
> +++ b/tests/run/c_int_types_T255.pyx	Wed Apr 22 19:52:13 2009 -0300
> @@ -655,7 +655,7 @@
>  # -------------------------------------------------------------------
>  import sys
> 
> -class MyInt(object):
> +class MyInt:
>     def __init__(self, value):
>        self.value = value
>     def __int__(self):
> 
> 

From robertwb at math.washington.edu  Thu Apr 23 08:53:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Apr 2009 23:53:51 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>
	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>
	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>
	<CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>
	<ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>
Message-ID: <199324C4-ECDE-4511-BEE4-F0FDDE703B0F@math.washington.edu>

On Apr 22, 2009, at 7:18 PM, Mark Lodato wrote:

> The embedded mode is a really good idea, but I think it would make
> more sense as a separate script for several reasons:
> 1. If it is separate, Cython itself doesn't have to change.
> 2. The same Cython C output file be made into a module and also  
> embedded.
> 3. Multiple Cython modules can be embedded into a single executable.

I was going for something really simple--running Cython files should  
be as easy as running Python files (with a compile step in there of  
course). What you're describing is valuable too, but I don't think  
the two are mutually exclusive. Also, I'm not scared of changing  
Cython :)

> I have included a first draft of such a script.

Thanks.

> Other (minor) changes:
> * I call Py_SetProgramName(), which seems like a good idea.

Good.

> * I create a MODINIT() macro to create the module initialization name,
> but I don't know how portable the ## concatenation is.

Also, the two functions have different signatures.

> An additional idea (not implemented) - it would be awesome to have the
> first module listed (that is, the one directly imported) to have
> __name__ == "__main__", so the normal Python idiom works.  I
> researched this for the past few hours, but I could not find a way to
> do this without modifying the module's code.  My best solution is the
> following:
> 1. For each module, mymod.pyx:
> 1a. Add a new externally-visible global variable:
>     int __pyx_mymod_is___main__ = 0;
> 1b. When converting the code for __name__, if __pyx_mymod_is___main__,
> return "__main__", else the module name.
> 2. In the script below, set
>     __pyx_mainmod_is___main__ = 1;
> right before the call to PyImport_ImportModule(), where "mainmod" is
> the main module.

I actually thought about doing this, and this is the only way I could  
come up with as well. If there's no objection I could push this so  
it'd be available for all generated .c files by default.

> Quick question, is the following a typo?
>         if (Py_FlushLine()); PyErr_Clear();
> Should that middle semi-colon be there?  If so, why the if statement?

Yes, oops. Fixed.

- Robert



From robertwb at math.washington.edu  Thu Apr 23 08:56:21 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Apr 2009 23:56:21 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <49EFFD9B.3070200@behnel.de>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>	<CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>
	<ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>
	<49EFFD9B.3070200@behnel.de>
Message-ID: <79E0D0D9-112A-41A4-B760-36A8DA6CABE3@math.washington.edu>

On Apr 22, 2009, at 10:33 PM, Stefan Behnel wrote:

> BTW, my impression is that this isn't ready for cython-devel yet,  
> given the
> amount of open questions and design choices.


I think there's a lot one could do here (e.g. automatically finding/ 
including dependancies, etc.) but that something really simple (you  
can make a .c file into an executable, why not a .pyx file) is  
valuable too.

In terms of -devel vs. -unstable, I see -unstable as a place to put  
things that may break Cython (e.g. internal reorganizations, messing  
with the type system, or other deeper stuff). My view is that all  
(non-bug) tests should always pass in the tip of -devel, and of  
course not break things we don't have tests for yet.

- Robert


From robertwb at math.washington.edu  Thu Apr 23 08:58:34 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Apr 2009 23:58:34 -0700
Subject: [Cython] download for 0.11.1
In-Reply-To: <49EFF999.90307@behnel.de>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>	<49EDC2B0.3060700@simula.no>	<49EDC619.5000602@chamonix.reportlab.co.uk>	<49EDCB8A.6040705@student.matnat.uio.no>	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>	<49EFAD09.2030104@canterbury.ac.nz>
	<e7ba66e40904221729x7882e079v2a92bd04af8e9318@mail.gmail.com>
	<49EFF999.90307@behnel.de>
Message-ID: <C32F141B-EF70-484A-B289-EEADBB229538@math.washington.edu>

On Apr 22, 2009, at 10:16 PM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> On Wed, Apr 22, 2009 at 8:49 PM, Greg Ewing wrote:
>>> Lisandro Dalcin wrote:
>>>
>>>> Sorry, can't resist... Could you tell me why a developer would be
>>>> attracted by this? To make use of ".hg", they need to have  
>>>> mercurial,
>>>> then... Do you think any DEVELOPER will find it cumbersome to do  
>>>> "hg
>>>> clone http://hg.cython.org/cython" ??
>>> Also I think it's a bit impolite to force all the people
>>> who aren't developers to download twice as much data as
>>> they're going to use.
>>
>> Moreover, I frequently use easy_install when I need a package in a
>> hurry. In such cases, tarball releases are downloaded, the package
>> installed, and next the tarball is just thrown away...
>
> What about shipping two distributions from PyPI then? One called  
> "Cython"
> that contains only the installable sources, and one called
> "Cython-complete" that easy_install will not use due to its  
> different name,
> and that additionally contains the .hg directory.

I like the default to be one with the repo. Is it really painful for  
users to download the extra couple MB, or is it the principle of the  
thing?

- Robert



From dagss at student.matnat.uio.no  Thu Apr 23 09:40:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Apr 2009 09:40:42 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <79E0D0D9-112A-41A4-B760-36A8DA6CABE3@math.washington.edu>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>	<CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>	<ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>	<49EFFD9B.3070200@behnel.de>
	<79E0D0D9-112A-41A4-B760-36A8DA6CABE3@math.washington.edu>
Message-ID: <49F01B7A.3080905@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 22, 2009, at 10:33 PM, Stefan Behnel wrote:
> 
>> BTW, my impression is that this isn't ready for cython-devel yet,  
>> given the
>> amount of open questions and design choices.
> 
> 
> I think there's a lot one could do here (e.g. automatically finding/ 
> including dependancies, etc.) but that something really simple (you  
> can make a .c file into an executable, why not a .pyx file) is  
> valuable too.
> 
> In terms of -devel vs. -unstable, I see -unstable as a place to put  
> things that may break Cython (e.g. internal reorganizations, messing  
> with the type system, or other deeper stuff). My view is that all  
> (non-bug) tests should always pass in the tip of -devel, and of  
> course not break things we don't have tests for yet.

Well, it makes the tree "unstable" in the sense that it adds 
functionality which one must then continue to support in order to be 
backwards compatible, which is a decision which could stall a release of 
the branch.

Remove the command-line switch and it is fixed.

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Apr 23 10:48:01 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 01:48:01 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <49F01B7A.3080905@student.matnat.uio.no>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>	<CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>	<ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>	<49EFFD9B.3070200@behnel.de>
	<79E0D0D9-112A-41A4-B760-36A8DA6CABE3@math.washington.edu>
	<49F01B7A.3080905@student.matnat.uio.no>
Message-ID: <0D189C52-BD3F-4D29-8CBB-F910ADAFC4FA@math.washington.edu>

On Apr 23, 2009, at 12:40 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 22, 2009, at 10:33 PM, Stefan Behnel wrote:
>>
>>> BTW, my impression is that this isn't ready for cython-devel yet,
>>> given the
>>> amount of open questions and design choices.
>>
>>
>> I think there's a lot one could do here (e.g. automatically finding/
>> including dependancies, etc.) but that something really simple (you
>> can make a .c file into an executable, why not a .pyx file) is
>> valuable too.
>>
>> In terms of -devel vs. -unstable, I see -unstable as a place to put
>> things that may break Cython (e.g. internal reorganizations, messing
>> with the type system, or other deeper stuff). My view is that all
>> (non-bug) tests should always pass in the tip of -devel, and of
>> course not break things we don't have tests for yet.
>
> Well, it makes the tree "unstable" in the sense that it adds
> functionality which one must then continue to support in order to be
> backwards compatible, which is a decision which could stall a  
> release of
> the branch.
>
> Remove the command-line switch and it is fixed.

Or even remove the option from the help--undocumented flags are  
always subject to change. But it's a whole different class than the  
much-needed, but scary, reworking of temps or adding closures.

- Robert



From lodatom at gmail.com  Thu Apr 23 12:45:56 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Thu, 23 Apr 2009 06:45:56 -0400
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <49EFFD9B.3070200@behnel.de>
References: <c4ab18df0904190356w6298837boec858b2e18034567@mail.gmail.com>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	<e7ba66e40904210617t230c5142v2bae1671504ebc0@mail.gmail.com>
	<E7DA7325-D6F2-416C-B104-072D2CF34281@math.washington.edu>
	<e7ba66e40904211931n517d5c94rddde2ec6000308cc@mail.gmail.com>
	<C12EFD39-0428-45CC-98D0-06B62EAAF5AF@math.washington.edu>
	<e7ba66e40904212006m4e907fc7m5e3191a22b5e341e@mail.gmail.com>
	<CD5A11A1-D2A6-41D4-87CE-55A1ABE02B72@math.washington.edu>
	<ca433830904221918v83574d4u7355a3819c1d1b87@mail.gmail.com>
	<49EFFD9B.3070200@behnel.de>
Message-ID: <ca433830904230345i7a825a55vb0de871017d955c6@mail.gmail.com>

On Thu, Apr 23, 2009 at 1:33 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> I have included a first draft of such a script. ?It is a modification
>> of the embedded mode that was checked into cython-devel. ?Notably, I
>> use PyImport_ExtendInittab to register all the modules as built-ins,
>
> 1) does that work for modules in packages?

I don't know.  I have never really used the Python C API before, so
what I wrote is just a guess.  I have no idea how to deal with
packages.  More testing is probably warranted.

> 2) will this decref the modules in Py3?

Sorry, I don't really know exactly what that means, nor do I know how
to test it.  I've never written and Py3 C code before.


>> and I only import the first one listed (which would ideally be called
>> "__main__"; see below). This seems to be a cleaner way to do things.
>> This also means that any extra module listed on the command-line that
>> is not imported by the first module will not be run.
>
> When is the init code of the modules called, and in which order? The main
> module init code should be able to import and use all other modules, even
> before executing its own init code.

The init code of the first (main) module will be called when
PyImport_ImportModule is called.  The init code of others will be
called if and when the main module (or some other module) imports
them.  By adding the modules to the inittab, they act like built-ins,
so I think (but have not verified) that when they are imported, no
search of sys.path is performed.

>> * I create a MODINIT() macro to create the module initialization name,
>> but I don't know how portable the ## concatenation is.
>
> Also, the functions return different things in Py2 and Py3, so they are not
> easily interchangeable.

This is true, but it doesn't matter.  The value PyMODINIT_FUNC is set
to "void" in Py2 and "PyObject*" in Py3, so only the name changes.
(Cython uses this.)  I don't actually call the function - I just load
a pointer to the function into the struct _inittab.


Thanks for the positive responses everyone!

-- Mark

From robertwb at math.washington.edu  Thu Apr 23 12:58:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 03:58:51 -0700
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <49EF437C.90605@student.matnat.uio.no>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>
	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>
	<49EF437C.90605@student.matnat.uio.no>
Message-ID: <E90F6960-A3F7-474A-9527-370234F013B7@math.washington.edu>

On Apr 22, 2009, at 9:19 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>>
>>> I'm in need of fast computations on complex numbers from Cython.
>>
>> Yes, I've started this, as it comes up in my research too (number
>> theory, computing values of L-functions). I hope to have something
>> releasable soon. In the meantime, you can do
>
> Is there anything I can do to help you with this? (Or perhaps take  
> over,
> if you now prioritize closures higher.)

It'd be really helpful if you handle the buffer support side of things.

> I'm holding a full-day tutorial on scientific use of Cython, 18th of
> May, and I'd feel less embarassed if this was in a release by then, so
> that's a nice goal, although not all-overriding of course :-) But my
> recent focus on correct struct support with NumPy could certainly be
> shifted below complex numbers.

Easily done by then.

- Robert


From dagss at student.matnat.uio.no  Thu Apr 23 13:02:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Apr 2009 13:02:12 +0200
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <E90F6960-A3F7-474A-9527-370234F013B7@math.washington.edu>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>	<49EF437C.90605@student.matnat.uio.no>
	<E90F6960-A3F7-474A-9527-370234F013B7@math.washington.edu>
Message-ID: <49F04AB4.9090706@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 22, 2009, at 9:19 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>>>
>>>> I'm in need of fast computations on complex numbers from Cython.
>>> Yes, I've started this, as it comes up in my research too (number
>>> theory, computing values of L-functions). I hope to have something
>>> releasable soon. In the meantime, you can do
>> Is there anything I can do to help you with this? (Or perhaps take  
>> over,
>> if you now prioritize closures higher.)
> 
> It'd be really helpful if you handle the buffer support side of things.

Oh definitely, I was expecting you to drop that on me anyway :-)

I need the basics in before it will be useful to working on that though. 
So unless I hear something else I'll assume you're working on it and 
will drop me a note when I can add buffer support.

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Apr 23 13:03:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 04:03:07 -0700
Subject: [Cython] GSoC branches
Message-ID: <9A5529CE-F1FE-4B24-A17A-F68C1C7E7814@math.washington.edu>

I've made two branches http://hg.cython.org/gsoc-danilo/ and http:// 
hg.cython.org/gsoc-kurt/ for GSoC projects. They are currently at  
cython-devel--pull in cython-unstable if you'd rather start from  
there. Feel free to use them as you like, but it'd be nice to push  
there often (I'll purge them at your request if necessary) and that  
way people can see what you're up to and make it easy to pull code  
into the main development branches.

- Robert


From robertwb at math.washington.edu  Thu Apr 23 13:07:12 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 04:07:12 -0700
Subject: [Cython] Path towards Python completeness
In-Reply-To: <380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49ED5CE8.6090708@behnel.de>
	<80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>
	<380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <BFD20AC6-2309-4634-98FB-D30BB66A6B2F@math.washington.edu>

On Apr 21, 2009, at 2:23 AM, Stefan Behnel wrote:

>> Should I make a closure branch on the site?
>
> You'd create it from cython-unstable, I assume. Given that the current
> development works reasonably well with two branches (pull-direction
> unstable<-devel), I doubt that a third "closures" branch (pull- 
> direction
> closures<-unstable) would add any major additional overhead, while  
> still
> allowing us to finish up and release 0.12 from cython-unstable when  
> it's
> time.

See http://hg.cython.org/cython-closures/ . I found some old stuff I  
did, remembered why it didn't work (but there was some useful stuff  
in there) and hacked on it a bit more. The tests at

http://hg.cython.org/cython-closures/file/c4ee199f08a2/tests/run/ 
closures_T82.pyx

pass. Before you get your hopes up too much, I want to add a big  
disclaimer that though I tried to be careful, this is very first- 
draft code and there are bound to be issues (e.g. I think I've run  
into some with respect to Python arguments).

- Robert


From robertwb at math.washington.edu  Thu Apr 23 13:09:35 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 04:09:35 -0700
Subject: [Cython] Status of C complex types (re-send)
In-Reply-To: <49F04AB4.9090706@student.matnat.uio.no>
References: <d5e3120f0904032221v43ee94cdseec8a1fd859f5b63@mail.gmail.com>	<488BAB80-5474-4BE8-9ED9-9C33133CFFDE@math.washington.edu>	<49EF437C.90605@student.matnat.uio.no>
	<E90F6960-A3F7-474A-9527-370234F013B7@math.washington.edu>
	<49F04AB4.9090706@student.matnat.uio.no>
Message-ID: <D5B30872-FDE2-4B7A-A768-FF44CE09B52A@math.washington.edu>

On Apr 23, 2009, at 4:02 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 22, 2009, at 9:19 AM, Dag Sverre Seljebotn wrote:
>>
>>> Robert Bradshaw wrote:
>>>> On Apr 3, 2009, at 10:21 PM, Vic Kelson wrote:
>>>>
>>>>> I'm in need of fast computations on complex numbers from Cython.
>>>> Yes, I've started this, as it comes up in my research too (number
>>>> theory, computing values of L-functions). I hope to have something
>>>> releasable soon. In the meantime, you can do
>>> Is there anything I can do to help you with this? (Or perhaps take
>>> over,
>>> if you now prioritize closures higher.)
>>
>> It'd be really helpful if you handle the buffer support side of  
>> things.
>
> Oh definitely, I was expecting you to drop that on me anyway :-)
>
> I need the basics in before it will be useful to working on that  
> though.
> So unless I hear something else I'll assume you're working on it and
> will drop me a note when I can add buffer support.

Yes, feel free to keep pinging me. I've been working on an algorithm  
that uses arbitrary-precision complex numbers, and it's taken a while  
to get that up and running, which is why I haven't worked on the  
double-precision case lately.

- Robert


From dalcinl at gmail.com  Thu Apr 23 13:50:38 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Apr 2009 08:50:38 -0300
Subject: [Cython] weird difference in Py 2.3&2.4 versus 2.5 and above
	regarding __int__()/__long__() for old-style (aka classic)
	classes...
In-Reply-To: <49EFFE99.80301@behnel.de>
References: <e7ba66e40904221603h477bb6d5gf3a5f07612921c68@mail.gmail.com>
	<49EFFE99.80301@behnel.de>
Message-ID: <e7ba66e40904230450m150b33cdwa477724aa7285da3@mail.gmail.com>

On Thu, Apr 23, 2009 at 2:37 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi Lisandro,
>
> I can't currently comment on the change itself, but I would prefer
> duplicating the relevant part of the test case instead of replacing it, so
> that we test both types of classes.
>

Well, there is other test case that covers new-style classes, but as
it also has to do with __int__&__long__(), I'll duplicate the former.


> Stefan
>
>
> Lisandro Dalcin wrote:
>> I do not have the time do dive deep enough to figure out the root of
>> this difference, but here you have a patch that modifies a testcase to
>> exercise the problem, and a quick fix for __Pyx_PyNumber_Int fixing
>> the issue.
>>
>> If there are no objections, I'll push the fix to cython-devel... As
>> this is related to rather old Python versions, I'm being a bit lazy as
>> to re-openi #255, I would just push the fix :-)
>>
>>
>> diff -r b550ee890c72 Cython/Compiler/PyrexTypes.py
>> --- a/Cython/Compiler/PyrexTypes.py ? Tue Apr 21 19:43:22 2009 -0700
>> +++ b/Cython/Compiler/PyrexTypes.py ? Wed Apr 22 19:52:13 2009 -0300
>> @@ -1524,7 +1524,13 @@
>> ? ? ?return Py_INCREF(x), x;
>> ? ?m = Py_TYPE(x)->tp_as_number;
>> ?#if PY_VERSION_HEX < 0x03000000
>> +#if PY_VERSION_HEX < 0x02050000
>> + ?if (m && m->nb_long &&
>> + ? ? ?(!PyInstance_Check(x) ||
>> + ? ? ? PyObject_HasAttrString(x, (char*)"__long__"))) {
>> +#else
>> ? ?if (m && m->nb_long) {
>> +#endif
>> ? ? ?name = "long";
>> ? ? ?res = PyNumber_Long(x);
>> ? ?}
>> diff -r b550ee890c72 tests/run/c_int_types_T255.pyx
>> --- a/tests/run/c_int_types_T255.pyx ?Tue Apr 21 19:43:22 2009 -0700
>> +++ b/tests/run/c_int_types_T255.pyx ?Wed Apr 22 19:52:13 2009 -0300
>> @@ -655,7 +655,7 @@
>> ?# -------------------------------------------------------------------
>> ?import sys
>>
>> -class MyInt(object):
>> +class MyInt:
>> ? ? def __init__(self, value):
>> ? ? ? ?self.value = value
>> ? ? def __int__(self):
>>
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Apr 23 13:57:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Apr 2009 13:57:31 +0200 (CEST)
Subject: [Cython] Path towards Python completeness
In-Reply-To: <BFD20AC6-2309-4634-98FB-D30BB66A6B2F@math.washington.edu>
References: <49ED5CE8.6090708@behnel.de>
	<80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>
	<380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<BFD20AC6-2309-4634-98FB-D30BB66A6B2F@math.washington.edu>
Message-ID: <2038f7c55d17516c79c8b3b2a9385fb1.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> http://hg.cython.org/cython-closures/file/c4ee199f08a2/tests/run/closures_T82.pyx

Way cool. It's always impressive how much work such a tiny test case can
imply.


> Before you get your hopes up too much, I want to add a big
> disclaimer that though I tried to be careful, this is very first-
> draft code and there are bound to be issues (e.g. I think I've run
> into some with respect to Python arguments).

I'll give it a try during the next days.

Stefan


From dalcinl at gmail.com  Thu Apr 23 14:21:13 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Apr 2009 09:21:13 -0300
Subject: [Cython] download for 0.11.1
In-Reply-To: <C32F141B-EF70-484A-B289-EEADBB229538@math.washington.edu>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
	<49EDC2B0.3060700@simula.no>
	<49EDC619.5000602@chamonix.reportlab.co.uk>
	<49EDCB8A.6040705@student.matnat.uio.no>
	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
	<49EFAD09.2030104@canterbury.ac.nz>
	<e7ba66e40904221729x7882e079v2a92bd04af8e9318@mail.gmail.com>
	<49EFF999.90307@behnel.de>
	<C32F141B-EF70-484A-B289-EEADBB229538@math.washington.edu>
Message-ID: <e7ba66e40904230521n1fc87745vecb21e34e84d26f0@mail.gmail.com>

On Thu, Apr 23, 2009 at 3:58 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 22, 2009, at 10:16 PM, Stefan Behnel wrote:
>
>
> I like the default to be one with the repo. Is it really painful for
> users to download the extra couple MB, or is it the principle of the
> thing?
>

Cython is small, so the extra couple MB is not really a big issue,
though my internet connection at work is really slow ;-(. But as Greg
commented, this is unpolite. If all projects out there did the same,
even you will complain (think of core CPython doing this). For
example, I'm involved in developing PETSc. The latest release tarball
weights 14MB; if we add the .hg directectory, that would add 85MB; and
many PETSc users are also developers...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Apr 23 14:23:42 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Apr 2009 09:23:42 -0300
Subject: [Cython] weird difference in Py 2.3&2.4 versus 2.5 and above
	regarding __int__()/__long__() for old-style (aka classic)
	classes...
In-Reply-To: <e7ba66e40904230450m150b33cdwa477724aa7285da3@mail.gmail.com>
References: <e7ba66e40904221603h477bb6d5gf3a5f07612921c68@mail.gmail.com>
	<49EFFE99.80301@behnel.de>
	<e7ba66e40904230450m150b33cdwa477724aa7285da3@mail.gmail.com>
Message-ID: <e7ba66e40904230523l602a9be5k74c726f9671ff7bc@mail.gmail.com>

On Thu, Apr 23, 2009 at 8:50 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On Thu, Apr 23, 2009 at 2:37 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi Lisandro,
>>
>> I can't currently comment on the change itself, but I would prefer
>> duplicating the relevant part of the test case instead of replacing it, so
>> that we test both types of classes.
>>
>
> Well, there is other test case that covers new-style classes, but as
> it also has to do with __int__&__long__(), I'll duplicate the former.
>

Well, I've now duplicated the tests. All pass from 2.3 to 3.0,
including recent 2.7 trunk.

I'll wait for Robert's comments or more from Stefan's side.


>
>> Stefan
>>
>>
>> Lisandro Dalcin wrote:
>>> I do not have the time do dive deep enough to figure out the root of
>>> this difference, but here you have a patch that modifies a testcase to
>>> exercise the problem, and a quick fix for __Pyx_PyNumber_Int fixing
>>> the issue.
>>>
>>> If there are no objections, I'll push the fix to cython-devel... As
>>> this is related to rather old Python versions, I'm being a bit lazy as
>>> to re-openi #255, I would just push the fix :-)
>>>
>>>
>>> diff -r b550ee890c72 Cython/Compiler/PyrexTypes.py
>>> --- a/Cython/Compiler/PyrexTypes.py ? Tue Apr 21 19:43:22 2009 -0700
>>> +++ b/Cython/Compiler/PyrexTypes.py ? Wed Apr 22 19:52:13 2009 -0300
>>> @@ -1524,7 +1524,13 @@
>>> ? ? ?return Py_INCREF(x), x;
>>> ? ?m = Py_TYPE(x)->tp_as_number;
>>> ?#if PY_VERSION_HEX < 0x03000000
>>> +#if PY_VERSION_HEX < 0x02050000
>>> + ?if (m && m->nb_long &&
>>> + ? ? ?(!PyInstance_Check(x) ||
>>> + ? ? ? PyObject_HasAttrString(x, (char*)"__long__"))) {
>>> +#else
>>> ? ?if (m && m->nb_long) {
>>> +#endif
>>> ? ? ?name = "long";
>>> ? ? ?res = PyNumber_Long(x);
>>> ? ?}
>>> diff -r b550ee890c72 tests/run/c_int_types_T255.pyx
>>> --- a/tests/run/c_int_types_T255.pyx ?Tue Apr 21 19:43:22 2009 -0700
>>> +++ b/tests/run/c_int_types_T255.pyx ?Wed Apr 22 19:52:13 2009 -0300
>>> @@ -655,7 +655,7 @@
>>> ?# -------------------------------------------------------------------
>>> ?import sys
>>>
>>> -class MyInt(object):
>>> +class MyInt:
>>> ? ? def __init__(self, value):
>>> ? ? ? ?self.value = value
>>> ? ? def __int__(self):
>>>
>>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Apr 23 18:36:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Apr 2009 13:36:06 -0300
Subject: [Cython] about Python completeness, other round about bytes/unicode
Message-ID: <e7ba66e40904230936j313d256di2d812358b25ab3af@mail.gmail.com>

Supose you construct a dict with string keys on Cython code, and that
dict is intended to be used as a **kargs for calling some function.
Moreover, supose you construct such dict as a literal, and the keys
are also string literals... How can I write that in such a way that
works both in Py2 and Py3 without duplicating code? If you use b"abc"
for the keys, it does not work on Py3. If you use u"abc", it does not
work on Py2. If you use a bare "abc", it will work on Py2 or Py3 (but
not both) depending on "Future.unicode_literals".

In short, string literals are a big PITA, at least for me. And I
believe that other Cython users do not complain loudly just because
they do not have Py3 as a target runtime yet.

* All my Cython sources are ASCII (I even misspell my last name just
to maintain them pure ASCII).
* In all cases, my bare string literals (I mean, "abc") are pure ASCII.
* In the few cases I need a byte-string (to match PyBytes types in
Py3), I do use b"abc".

Why then cannot I have an option or compiler directive (or perhaps
just a global option) that will make bare "abc" string literals become
PyString in Py2 and PyUnicode in Py3?

Even if these literals were restricted to be pure-ASCII (Cython
generating a error if I write a bare "abcX" were X cannot be
ASCII-encoded), I would be more than happy.

Stefan, Can you imagine any potential problem with this? Please take
into account I'm not talking about the conversion Py <-> char*, just
(pure-ASCII) string literals with unspecified kind. Am I (once more)
missing something?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Apr 23 19:10:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 10:10:38 -0700
Subject: [Cython] download for 0.11.1
In-Reply-To: <e7ba66e40904230521n1fc87745vecb21e34e84d26f0@mail.gmail.com>
References: <49EDC1EC.7030703@chamonix.reportlab.co.uk>
	<49EDC2B0.3060700@simula.no>
	<49EDC619.5000602@chamonix.reportlab.co.uk>
	<49EDCB8A.6040705@student.matnat.uio.no>
	<e7ba66e40904211950w342a618avfddba19ce31c7031@mail.gmail.com>
	<49EFAD09.2030104@canterbury.ac.nz>
	<e7ba66e40904221729x7882e079v2a92bd04af8e9318@mail.gmail.com>
	<49EFF999.90307@behnel.de>
	<C32F141B-EF70-484A-B289-EEADBB229538@math.washington.edu>
	<e7ba66e40904230521n1fc87745vecb21e34e84d26f0@mail.gmail.com>
Message-ID: <ABE5B761-8E44-479D-BF4B-0198366A1325@math.washington.edu>


On Apr 23, 2009, at 5:21 AM, Lisandro Dalcin wrote:

> On Thu, Apr 23, 2009 at 3:58 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 22, 2009, at 10:16 PM, Stefan Behnel wrote:
>>
>>
>> I like the default to be one with the repo. Is it really painful for
>> users to download the extra couple MB, or is it the principle of the
>> thing?
>>
>
> Cython is small, so the extra couple MB is not really a big issue,
> though my internet connection at work is really slow ;-(. But as Greg
> commented, this is unpolite.

That is a matter of opinion--in my book anything that a community  
does to remove barriers to contribution is a mark on the "more  
polite" side. It's provided as a convenience, not because we think  
users couldn't cope otherwise. Most people probably won't even notice  
it's there until they go to edit the source.

> If all projects out there did the same,
> even you will complain (think of core CPython doing this). For
> example, I'm involved in developing PETSc. The latest release tarball
> weights 14MB; if we add the .hg directectory, that would add 85MB; and
> many PETSc users are also developers...

Only if it got in the way, e.g. 85MB would. Let's revisit when when  
the potential savings hit 10MB.

- Robert


From dagss at student.matnat.uio.no  Thu Apr 23 20:17:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Apr 2009 20:17:12 +0200
Subject: [Cython] Documentation tickets
Message-ID: <49F0B0A8.40701@student.matnat.uio.no>

I added a "Documentation" milestone and component. Rushing ahead to do 
other useful stuff instead of staying behind perfecting documentation 
can't be helped IMO, but filing a ticket is much easier to do and such a 
list is probably very useful.

I could even much easier add the actual documentation text as I go to a 
ticket, which has much lower startup overhead than editing the 
documentation itself.

Let me know if I'm wrong and should remove it again :-)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Apr 23 20:21:16 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Apr 2009 20:21:16 +0200
Subject: [Cython] Documentation tickets
In-Reply-To: <49F0B0A8.40701@student.matnat.uio.no>
References: <49F0B0A8.40701@student.matnat.uio.no>
Message-ID: <49F0B19C.8070301@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> I added a "Documentation" milestone and component. Rushing ahead to do 
> other useful stuff instead of staying behind perfecting documentation 
> can't be helped IMO, but filing a ticket is much easier to do and such a 
> list is probably very useful.
> 
> I could even much easier add the actual documentation text as I go to a 
> ticket, which has much lower startup overhead than editing the 
> documentation itself.
> 
> Let me know if I'm wrong and should remove it again :-)

Also I propose that every such ticket has a "Since" field in the 
description field. E.g. for packed structs I stated our next release.

http://trac.cython.org/cython_trac/ticket/294

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Apr 23 20:36:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 11:36:42 -0700
Subject: [Cython] Documentation tickets
In-Reply-To: <49F0B19C.8070301@student.matnat.uio.no>
References: <49F0B0A8.40701@student.matnat.uio.no>
	<49F0B19C.8070301@student.matnat.uio.no>
Message-ID: <80E1EE0A-340A-4220-BFAA-E48A09D477EA@math.washington.edu>


On Apr 23, 2009, at 11:21 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> I added a "Documentation" milestone and component. Rushing ahead  
>> to do
>> other useful stuff instead of staying behind perfecting documentation
>> can't be helped IMO, but filing a ticket is much easier to do and  
>> such a
>> list is probably very useful.
>>
>> I could even much easier add the actual documentation text as I go  
>> to a
>> ticket, which has much lower startup overhead than editing the
>> documentation itself.
>>
>> Let me know if I'm wrong and should remove it again :-)

No, that's a very good idea.

> Also I propose that every such ticket has a "Since" field in the
> description field. E.g. for packed structs I stated our next release.
>
> http://trac.cython.org/cython_trac/ticket/294

Isn't this redundant with the milestone field?

- Robert


From dagss at student.matnat.uio.no  Thu Apr 23 21:17:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Apr 2009 21:17:04 +0200
Subject: [Cython] Documentation tickets
In-Reply-To: <80E1EE0A-340A-4220-BFAA-E48A09D477EA@math.washington.edu>
References: <49F0B0A8.40701@student.matnat.uio.no>	<49F0B19C.8070301@student.matnat.uio.no>
	<80E1EE0A-340A-4220-BFAA-E48A09D477EA@math.washington.edu>
Message-ID: <49F0BEB0.50909@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 23, 2009, at 11:21 AM, Dag Sverre Seljebotn wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> I added a "Documentation" milestone and component. Rushing ahead  
>>> to do
>>> other useful stuff instead of staying behind perfecting documentation
>>> can't be helped IMO, but filing a ticket is much easier to do and  
>>> such a
>>> list is probably very useful.
>>>
>>> I could even much easier add the actual documentation text as I go  
>>> to a
>>> ticket, which has much lower startup overhead than editing the
>>> documentation itself.
>>>
>>> Let me know if I'm wrong and should remove it again :-)
> 
> No, that's a very good idea.
> 
>> Also I propose that every such ticket has a "Since" field in the
>> description field. E.g. for packed structs I stated our next release.
>>
>> http://trac.cython.org/cython_trac/ticket/294
> 
> Isn't this redundant with the milestone field?

I'm sure it can be discussed, but that's an emphatic "no" from me.

Milestones are mainly for scheduling -- scheduling work and scheduling 
branches. If I assigned a doc ticket to 0.11.2, I think that's the same 
as saying "we want to see if we can get this documented before we 
release 0.11.2". Which is silly at this stage. (But e.g. in Sage you'd 
easily do something like that, i.e. "we want documentation to have this 
quality before releasing". Like the doctests goals.)

What I want with the "Since" field is to have that more or less 
literally end up in the documentation.

Example: I could create a new doc ticket now for something in buffers 
which I remembered, assign it to milestone 0.12 because I'd really like 
to get it done by the 0.12 release, but have a "Since" field of 0.10 
since the feature happened to be available by then.

You *do* have the "Version" trac field, which we don't use. Using a 
comment just seemed simpler.

BTW I prefer my trac tickets served by report 3

http://trac.cython.org/cython_trac/report/3

This may be less important in other reports.

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Apr 23 21:36:34 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 12:36:34 -0700
Subject: [Cython] Documentation tickets
In-Reply-To: <49F0BEB0.50909@student.matnat.uio.no>
References: <49F0B0A8.40701@student.matnat.uio.no>	<49F0B19C.8070301@student.matnat.uio.no>
	<80E1EE0A-340A-4220-BFAA-E48A09D477EA@math.washington.edu>
	<49F0BEB0.50909@student.matnat.uio.no>
Message-ID: <4116E00C-C872-4BDC-8DD2-E6FB46751651@math.washington.edu>

On Apr 23, 2009, at 12:17 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 23, 2009, at 11:21 AM, Dag Sverre Seljebotn wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> I added a "Documentation" milestone and component. Rushing ahead
>>>> to do
>>>> other useful stuff instead of staying behind perfecting  
>>>> documentation
>>>> can't be helped IMO, but filing a ticket is much easier to do and
>>>> such a
>>>> list is probably very useful.
>>>>
>>>> I could even much easier add the actual documentation text as I go
>>>> to a
>>>> ticket, which has much lower startup overhead than editing the
>>>> documentation itself.
>>>>
>>>> Let me know if I'm wrong and should remove it again :-)
>>
>> No, that's a very good idea.
>>
>>> Also I propose that every such ticket has a "Since" field in the
>>> description field. E.g. for packed structs I stated our next  
>>> release.
>>>
>>> http://trac.cython.org/cython_trac/ticket/294
>>
>> Isn't this redundant with the milestone field?
>
> I'm sure it can be discussed, but that's an emphatic "no" from me.
>
> Milestones are mainly for scheduling -- scheduling work and scheduling
> branches. If I assigned a doc ticket to 0.11.2, I think that's the  
> same
> as saying "we want to see if we can get this documented before we
> release 0.11.2". Which is silly at this stage. (But e.g. in Sage you'd
> easily do something like that, i.e. "we want documentation to have  
> this
> quality before releasing". Like the doctests goals.)
>
> What I want with the "Since" field is to have that more or less
> literally end up in the documentation.
>
> Example: I could create a new doc ticket now for something in buffers
> which I remembered, assign it to milestone 0.12 because I'd really  
> like
> to get it done by the 0.12 release, but have a "Since" field of 0.10
> since the feature happened to be available by then.

Ok, you're just proposing a since field for documentation tickets.  
Yes, that makes more sense now.

- Robert


From stefan_ml at behnel.de  Thu Apr 23 22:39:49 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Apr 2009 22:39:49 +0200
Subject: [Cython] Cython + tuple unpacking
In-Reply-To: <mailman.4434.1240496587.11746.python-list@python.org>
References: <mailman.4434.1240496587.11746.python-list@python.org>
Message-ID: <49F0D215.7020504@behnel.de>

Hugues Salamin wrote:
> The following code will crash with a segfault when compiled using cython (v0.11)
> 
> def func():
>     for (a, b) ,c ,d in zip(zip(range(3), range(3)), range(3), range(3)):
>         print a, b
>         print c
>         print d # This line segfault
> 
> Compilation is done using distutils.
> 
> If the module is imported in python and func is called, I got a segmentation
> fault at the line "print d".

Yes, this works for me in the latest (unstable) developer branch, but fails
in the release branch:

  $ cd cython-unstable
  $ cat rangeloop.pyx
  def func():
      for (a, b) ,c ,d in zip(zip(range(3), range(3)), range(3), range(3)):
          print a, b
          print c
          print d # This line segfault

  $ python2.6 -c 'import pyximport; pyximport.install(); \
                                 import rangeloop; rangeloop.func()'
  0 0
  0
  0
  1 1
  1
  1
  2 2
  2
  2

  $ cd ../cython-release
  $ python2.6 -c 'import pyximport; pyximport.install(); \
                                 import rangeloop; rangeloop.func()'
  0 0
  0
  Segmentation fault

Not sure why. I'm moving this to the Cython mailing list, we should be able
to help you there.

Stefan

From dalcinl at gmail.com  Thu Apr 23 23:50:07 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Apr 2009 18:50:07 -0300
Subject: [Cython] incorrect C code generated on method assignment
	(cython 0.11.1)
In-Reply-To: <1c34ba170904190102g5d60ca61ve9045fcb204a93b6@mail.gmail.com>
References: <1c34ba170904170550jc9a800cue5c1dad48d30e256@mail.gmail.com>
	<09c458c88760dc5ea641d522c6ec939e.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<1c34ba170904190102g5d60ca61ve9045fcb204a93b6@mail.gmail.com>
Message-ID: <e7ba66e40904231450h771bd5b3nd51234fe4bb88e37@mail.gmail.com>

I've uploaded a patch to the tracker fixing #287.

Deciding on what to do here was a real pain. I really think that this
is the best thing to do, for many reasons I'm now so tired to write
down. If any of you think any part of this is wrong, comments very
welcome, but I would appreciate you provide also a valid use case or
Python/Cython code that prove me wrong; I was not able to find any
single one.


On Sun, Apr 19, 2009 at 5:02 AM, Riccardo Murri
<riccardo.murri at gmail.com> wrote:
> On Fri, Apr 17, 2009 at 3:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Riccardo Murri wrote:
>>> while converting a class from pure Python to Cython, I've come across
>>> the following:
>>>
>>> ? $ cat r.pyx
>>> ? cdef class r(object):
>>> ? ? ? ?cdef long long n
>>> ? ? ? ?cdef long long d
>>>
>>> ? ? ? ?def __init__(self, long long n, long long d):
>>> ? ? ? ? ? ?self.n = n
>>> ? ? ? ? ? ?self.d = d
>>>
>>> ? ? ? ?def __int__(self):
>>> ? ? ? ? ? ?return (self.n // self.d)
>>>
>>> ? ? ? ?__long__ = __int__
>>>
>>> ? $ cython -v r.pyx
>>> ? Compiling /tmp/r.pyx
>>>
>>> ? $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
>>> -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c r.c -o r.o
>>> ? r.c:487: error: ???None??? undeclared here (not in a function)
>>>
>>> ? $ sed -e '484,490!d;488i^^^^^^^^^^^^^^^^^^^^^^^^ up here' r.c
>>> ? ? 0, /*nb_coerce*/
>>> ? ? #endif
>>> ? ? __pyx_pf_1r_1r___int__, /*nb_int*/
>>> ? ? None, /*nb_long*/
>>> ? ^^^^^^^^^^^^^^^^^^^^^^^^ up here
>>> ? ? 0, /*nb_float*/
>>> ? ? #if PY_MAJOR_VERSION < 3
>>> ? ? 0, /*nb_oct*/
>>
>> Yes, that's a bug. I assume that the above code does not make Cython set
>> up the C level function name.
>>
>> Raising an error here would work for now.
>>
>>
>>> I understand that "__long__ = __int__" is not the correct way to do
>>> method aliasing in Cython
>>
>> ... and that's a feature request, it would be nice to make that work.
>> Could you create two tickets for those?
>>
>
> Done: tickets #287 and #288
>
> http://trac.cython.org/cython_trac/ticket/287
> http://trac.cython.org/cython_trac/ticket/288
>
> Thank you!
>
> Riccardo
>
> --
> Riccardo Murri, via Ross?e 17, 6987 Caslano (CH)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Apr 24 04:00:10 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Apr 2009 23:00:10 -0300
Subject: [Cython] PATCH: honoring Options.optimize_simple_methods for
	methods and not just for functions
Message-ID: <e7ba66e40904231900m65c057e0s85c689185e5e5a94@mail.gmail.com>

If there are no objections, I'm going to push the one-line patch below.

BTW, Why this optimization is on by default? Despite the comment on
Options.py, IMHO we should emulate Python semantics here. A METH_O
method cannot be called like this "func(arg, **{})", but that works
for methods defined in Python code.


$ hg diff
diff -r c3bf82e34e54 Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py  Thu Apr 23 20:08:25 2009 +0200
+++ b/Cython/Compiler/Nodes.py  Thu Apr 23 22:49:15 2009 -0300
@@ -1699,7 +1699,7 @@
                 elif len(self.args) == 1:
                     if self.args[0].default is None and not
self.args[0].kw_only:
                         self.entry.signature = TypeSlots.pyfunction_onearg
-            elif self.entry.signature is TypeSlots.pymethod_signature:
+            elif self.entry.signature is TypeSlots.pymethod_signature
and Options.optimize_simple_methods:
                 if len(self.args) == 1:
                     self.entry.signature = TypeSlots.unaryfunc
                 elif len(self.args) == 2:



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Apr 24 04:31:49 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Apr 2009 19:31:49 -0700
Subject: [Cython] PATCH: honoring Options.optimize_simple_methods for
	methods and not just for functions
In-Reply-To: <e7ba66e40904231900m65c057e0s85c689185e5e5a94@mail.gmail.com>
References: <e7ba66e40904231900m65c057e0s85c689185e5e5a94@mail.gmail.com>
Message-ID: <27A96D63-8B70-4FE8-B472-DEDB74C0B807@math.washington.edu>

On Apr 23, 2009, at 7:00 PM, Lisandro Dalcin wrote:

> If there are no objections, I'm going to push the one-line patch  
> below.

Yes, we should do that. Actually, it probably makes more sense  
logically to do

if self.entry.is_special:
     ...
elif Options.optimize_simple_methods and not self.star_arg and not  
self.starstar_arg:
     ...

> BTW, Why this optimization is on by default? Despite the comment on
> Options.py, IMHO we should emulate Python semantics here. A METH_O
> method cannot be called like this "func(arg, **{})", but that works
> for methods defined in Python code.

This optimization is on by default because it is extremely common and  
does have a noticeable impact on timings (don't recall offhand any  
actual numbers here). Now that we have compiler directives, we should  
be using them here, but unlike the C division case I think it's a bad  
idea to change the default because it gives a (rather comprehensible)  
error rather than silently continuing with different behavior. You're  
the second person since this optimization was introduced well over a  
year ago to even notice.

I am still in favor of a emulate-pure-python-no-matter-what-the-cost  
compiler directive that would disable this (and other discrepancies  
we may find). It should also be better documented. http:// 
trac.cython.org/cython_trac/ticket/295

- Robert

> $ hg diff
> diff -r c3bf82e34e54 Cython/Compiler/Nodes.py
> --- a/Cython/Compiler/Nodes.py  Thu Apr 23 20:08:25 2009 +0200
> +++ b/Cython/Compiler/Nodes.py  Thu Apr 23 22:49:15 2009 -0300
> @@ -1699,7 +1699,7 @@
>                  elif len(self.args) == 1:
>                      if self.args[0].default is None and not
> self.args[0].kw_only:
>                          self.entry.signature =  
> TypeSlots.pyfunction_onearg
> -            elif self.entry.signature is  
> TypeSlots.pymethod_signature:
> +            elif self.entry.signature is TypeSlots.pymethod_signature
> and Options.optimize_simple_methods:
>                  if len(self.args) == 1:
>                      self.entry.signature = TypeSlots.unaryfunc
>                  elif len(self.args) == 2:
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Fri Apr 24 06:54:51 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Apr 2009 06:54:51 +0200
Subject: [Cython] PATCH: honoring Options.optimize_simple_methods for
 methods and not just for functions
In-Reply-To: <27A96D63-8B70-4FE8-B472-DEDB74C0B807@math.washington.edu>
References: <e7ba66e40904231900m65c057e0s85c689185e5e5a94@mail.gmail.com>
	<27A96D63-8B70-4FE8-B472-DEDB74C0B807@math.washington.edu>
Message-ID: <49F1461B.7090106@behnel.de>


Robert Bradshaw wrote:
> On Apr 23, 2009, at 7:00 PM, Lisandro Dalcin wrote:
>> BTW, Why this optimization is on by default? Despite the comment on
>> Options.py, IMHO we should emulate Python semantics here. A METH_O
>> method cannot be called like this "func(arg, **{})", but that works
>> for methods defined in Python code.
> 
> This optimization is on by default because it is extremely common and  
> does have a noticeable impact on timings (don't recall offhand any  
> actual numbers here). Now that we have compiler directives, we should  
> be using them here, but unlike the C division case I think it's a bad  
> idea to change the default because it gives a (rather comprehensible)  
> error rather than silently continuing with different behavior. You're  
> the second person since this optimization was introduced well over a  
> year ago to even notice.

I agree. It's extremely rare to call one-arg functions with keywords (and
there are tons of C extensions out there that disallow it already). The
only sensible case I can think of is the generic function wrapper, i.e.

    def wrap(f, *args, **kwargs):
        ...
        f(*args, **kwargs)

BTW, given that the above raises an error (as you said, didn't try), I'm
more inclined to consider that a bug in CPython. If the kwdict is empty, it
can just be ignored in the call mechanism instead of letting the call fail.
There'd be nothing wrong with that.

Stefan

From stefan_ml at behnel.de  Fri Apr 24 07:17:59 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Apr 2009 07:17:59 +0200
Subject: [Cython] Cython + tuple unpacking
In-Reply-To: <49F0D215.7020504@behnel.de>
References: <mailman.4434.1240496587.11746.python-list@python.org>
	<49F0D215.7020504@behnel.de>
Message-ID: <49F14B87.8010406@behnel.de>


Stefan Behnel wrote:
> Hugues Salamin wrote:
>> The following code will crash with a segfault when compiled using cython (v0.11)
>>
>> def func():
>>     for (a, b) ,c ,d in zip(zip(range(3), range(3)), range(3), range(3)):
>>         print a, b
>>         print c
>>         print d # This line segfault
>>
>> Compilation is done using distutils.
>>
>> If the module is imported in python and func is called, I got a segmentation
>> fault at the line "print d".
> 
> Yes, this works for me in the latest (unstable) developer branch, but fails
> in the release branch:
> 
>   $ cd cython-unstable
>   $ cat rangeloop.pyx
>   def func():
>       for (a, b) ,c ,d in zip(zip(range(3), range(3)), range(3), range(3)):
>           print a, b
>           print c
>           print d # This line segfault
> 
>   $ python2.6 -c 'import pyximport; pyximport.install(); \
>                                  import rangeloop; rangeloop.func()'
>   0 0
>   0
>   0
>   1 1
>   1
>   1
>   2 2
>   2
>   2
> 
>   $ cd ../cython-release
>   $ python2.6 -c 'import pyximport; pyximport.install(); \
>                                  import rangeloop; rangeloop.func()'
>   0 0
>   0
>   Segmentation fault


The generated code is incorrect in this case. Note how the (old-style) temp
var __pyx_4 is used for both the 'a' and 'd' variables.

    if (PyTuple_CheckExact(__pyx_t_5) && \
                        likely(PyTuple_GET_SIZE(__pyx_t_5) == 3)) {
      PyObject* tuple = __pyx_t_5;
      __pyx_2 = PyTuple_GET_ITEM(tuple, 0); __Pyx_INCREF(__pyx_2);
      __pyx_3 = PyTuple_GET_ITEM(tuple, 1); __Pyx_INCREF(__pyx_3);
      __pyx_4 = PyTuple_GET_ITEM(tuple, 2); __Pyx_INCREF(__pyx_4);
      __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;
      if (PyTuple_CheckExact(__pyx_2) && \
                          likely(PyTuple_GET_SIZE(__pyx_2) == 2)) {
        PyObject* tuple = __pyx_2;
        __pyx_4 = PyTuple_GET_ITEM(tuple, 0); __Pyx_INCREF(__pyx_4);
        __pyx_5 = PyTuple_GET_ITEM(tuple, 1); __Pyx_INCREF(__pyx_5);
        __Pyx_DECREF(__pyx_2); __pyx_2 = 0;
        __Pyx_DECREF(__pyx_v_a);
        __pyx_v_a = __pyx_4;
        __pyx_4 = 0;
        __Pyx_DECREF(__pyx_v_b);
        __pyx_v_b = __pyx_5;
        __pyx_5 = 0;

      ...

      }
      __Pyx_DECREF(__pyx_v_c);
      __pyx_v_c = __pyx_3;
      __pyx_3 = 0;
      __Pyx_DECREF(__pyx_v_d);
      __pyx_v_d = __pyx_4;
      __pyx_4 = 0;

Given that this works in cython-unstable (and I know that there were many,
many changes regarding temp allocation there again, it now uses new-style
temps everywhere), the backport may not be as simple as "copy one line".

Dag, any idea?

Stefan

From roland at utk.edu  Fri Apr 24 09:10:51 2009
From: roland at utk.edu (Roland Schulz)
Date: Fri, 24 Apr 2009 03:10:51 -0400
Subject: [Cython] Operators for ctypedef struct
Message-ID: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>

Hi,

is it possible (or planned) to add operators to a ctypdef struct?

Using the WrappingCPlusPlus example I would like to be able to do:
cdef c_Rectangle a = c_Rectangle(1,2,3,4)
a=a+a

but adding
c_Rectangle __add__ "operator+"(c_Rectangle right)
to c_Rectange does not work.

I don't want to use the Python wrapper class because I don't want the
operator to produce a PyNumber_Add call.

Will it be in the long run possible to wrap a C++ linear algebra
template library like Eigen (1) or NT2 (2)? This would be very nice
because one could very easy speed up numpy code.

E.g. in numpy
a,b,c,d=[N.random.random(100000) for i in range(4)] #just creating sample data
a*b*c*d #element wise multiplication

is pretty slow because the multiplications are done after each other
and thus cache is used poorly. By doing the same expression with
either template library, the expression templates make it just one
for-loop and even SIMD it.

Roland

Links:
1) eigen.tuxfamily.org/
2) nt2.sourceforge.net/
-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309

From dagss at student.matnat.uio.no  Fri Apr 24 09:35:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 24 Apr 2009 09:35:29 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
Message-ID: <49F16BC1.5050708@student.matnat.uio.no>

Roland Schulz wrote:
> Hi,
> 
> is it possible (or planned) to add operators to a ctypdef struct?

Better C++ support is definitely planned; in fact there is a GSoC 
project on it where this could likely be a part (though Danilo or Robert 
would have to comment on that).

I started a wiki page listing the things that could be improved for C++ 
support and

http://wiki.cython.org/enhancements/cpp

but I became exhausted right before fixing up the operator overloading 
section; perhaps you could have a look and come with suggestions?

Especially, I'm concerned with operators typically returning references 
like operator[]:

cdef struct Polygon:
     Point& operator[](int edge)

where one would like to be able to do

mypoly[3] = mypoint

However this also contradicts the scetched solution for dealing with 
references which means having Cython treat references as pointers; 
making it necesarry to do

mypoly[3][0] = mypoint

Perhaps some map between Python's __setitem__ and reference return 
values in C++ operator[], but it gets messy syntax-wise quickly...

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Apr 24 10:24:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 24 Apr 2009 10:24:30 +0200 (CEST)
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F16BC1.5050708@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<49F16BC1.5050708@student.matnat.uio.no>
Message-ID: <9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>

BTW, sorry to go off in a digression like this -- it is just something a
problem I remembered with this and I am writing as much to Danilo and
Robert as you. At least I also answered your question :-)

Some more thoughts:

> Roland Schulz wrote:
>> Hi,
>>
>> is it possible (or planned) to add operators to a ctypdef struct?
>
> Better C++ support is definitely planned; in fact there is a GSoC
> project on it where this could likely be a part (though Danilo or Robert
> would have to comment on that).
>
> I started a wiki page listing the things that could be improved for C++
> support and
>
> http://wiki.cython.org/enhancements/cpp
>
> but I became exhausted right before fixing up the operator overloading
> section; perhaps you could have a look and come with suggestions?
>
> Especially, I'm concerned with operators typically returning references
> like operator[]:
>
> cdef struct Polygon:
>      Point& operator[](int edge)
>
> where one would like to be able to do
>
> mypoly[3] = mypoint
>
> However this also contradicts the scetched solution for dealing with
> references which means having Cython treat references as pointers;
> making it necesarry to do
>
> mypoly[3][0] = mypoint
>
> Perhaps some map between Python's __setitem__ and reference return
> values in C++ operator[], but it gets messy syntax-wise quickly...

I should probably provide some more context here. In Python there's
getitem and setitem as seperate operators. In C++ you would instead return
an object with overridden assignment operator (for which there is no
Python equivalent, of course, as assignments are not allowed to have
side-effects in Python).

So in the example

Point& operator[](int edge);

if you wanted to print something when changing a point, you could do

PointInPolygon operator[](int edge);

instead, and PointInPolygon would contain both a reference to the Polygon
and the index of the edge, and have overloaded "operator=" and "operator
Point" which would work as setitem and getitem, respecticely.

So what I'm saying is: Mapping C++ operators to Python operators and vice
versa is difficult, with possibly many special cases where e.g. a Python
setitem operator would map to an entire C++ expression involving many
operators.

Starting out with the binary operators only is simpler and probably a good
idea :-)

Dag Sverre


From dagss at student.matnat.uio.no  Fri Apr 24 10:38:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 24 Apr 2009 10:38:48 +0200 (CEST)
Subject: [Cython] Fortran support design issues
Message-ID: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>

(This is a move of my and Kurt's discussion into the public; I'll give 
some context but not that much.)

Kurt: This will go into a broader discussion of buffers in Cython as such
and not be directly Fortran-relevant. I want to have Fortran support
integrated in a nice way into Cython, so I think this is necesarry to some
degree, but I'll try not to overdo it.

So, you wanted to target contiguous buffer passing first. And I was 
saying that fixing

http://trac.cython.org/cython_trac/ticket/177

would for the most part amount to what we need. Adding support for

   cdef void myfunc(object[int, mode="c"] buf)

which would have the C signature

   void myfunc(Py_buffer* buf) // + a contract on int/contiguous

and have the caller be responsible for getting the buffer, gets us far. 
It would mean that you could do

   myfunc(myobject)

and it would turn into

   Py_buffer buf = acquire buffer from myobject
   raise exception if buf is not contiguous
   myfunc(&buf)
   release buf

I first thought that adding one component more would get us all the way: 
Automatic copying into contiguous buffers. So assuming #177 is
implemented, one would then move on to having it instead turn into:

   Py_buffer buf = acquire buffer from myobject
   if (buf is contiguous) {
     myfunc(&buf);
   } else {
     Py_buffer buf2 = make contiguous copy of buf
     myfunc(&buf2)
   }
   release buf

However, there's a (big) problem here: What Python object does buf2 refer to?

Using the one of "buf" would be too confusing as they point to different
memory areas. One solution is just setting it to None, perhaps. However
once #177 is solved one will expect to be able to do

   cdef void myfunc(np.ndarray[int, mode="c"] buf)

and work simultaneously with NumPy functions/operators and buffer access,
and that works nice in itself with only #177, but with the addition of
copy-in/copy-out one then has a problem.

All of this makes me think about pushing the "new buffer syntax" a bit
harder and get it started on in your GSoC. With that,

   cdef void myfunc(int[:] buf)

could easily give non-surprising effects for #177 and copy-in/copy-out, as
the Python object is not "part of the deal".

Your GSoC would then Cython-side consist more or less of

a) #177, with new syntax
b) A generic mechanism for automatic coercion between buffers of different
modes. That is:

cdef int[::contiguous] buf # new syntax for mode="c" w/o Python obj?
buf = some_object

Here, if the buffer of some_object is not contiguous, a contiguous copy
will be made! And when releasing buf, it would be copied back.

This would make the parts necesarry for Fortran support tremendously
useful elsewhere for what I believe will not be much extra effort.

(Though I could help out with the parts of those not needed for Fortran
support in order to not derail your project.)

I should probably describe some of this better, let me know which part!

-- 
Dag Sverre







From dagss at student.matnat.uio.no  Fri Apr 24 11:18:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 24 Apr 2009 11:18:02 +0200
Subject: [Cython] Fortran support design issues
In-Reply-To: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>
References: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>
Message-ID: <49F183CA.4050401@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Your GSoC would then Cython-side consist more or less of
> 
> a) #177, with new syntax
> b) A generic mechanism for automatic coercion between buffers of different
> modes. That is:
> 
> cdef int[::contiguous] buf # new syntax for mode="c" w/o Python obj?
> buf = some_object
> 
> Here, if the buffer of some_object is not contiguous, a contiguous copy
> will be made! And when releasing buf, it would be copied back.

BTW, I want to make it clear that I /am/ moving around a bit here myself 
and am still considering the best way of getting Fortran support and PEP 
3118 buffers working together in the best way possible. By thinking 
aloud you'll hopefully pick up on the design constrains around 
buffers-in-Cython as we go; I don't expect it all to be clear right now.

Anyway, b) above is not good, it raises a lot of questions on semantics 
(what is int[:] really -- a reference to memory or the memory itself -- 
what happens if one acquires two buffers simultaneously, do they point 
to the same memory -- etc). So replace b) above with:

b) A mechanism for automatically making contiguous copy-in/copy-out on 
#177-style method calls if necesarry.

-- 
Dag Sverre

From dalcinl at gmail.com  Fri Apr 24 16:17:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Apr 2009 11:17:02 -0300
Subject: [Cython] PATCH: honoring Options.optimize_simple_methods for
	methods and not just for functions
In-Reply-To: <49F1461B.7090106@behnel.de>
References: <e7ba66e40904231900m65c057e0s85c689185e5e5a94@mail.gmail.com>
	<27A96D63-8B70-4FE8-B472-DEDB74C0B807@math.washington.edu>
	<49F1461B.7090106@behnel.de>
Message-ID: <e7ba66e40904240717r62c6b552x8ca6ac6eab42a4e1@mail.gmail.com>

On Fri, Apr 24, 2009 at 1:54 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Robert Bradshaw wrote:
>> On Apr 23, 2009, at 7:00 PM, Lisandro Dalcin wrote:
>>> BTW, Why this optimization is on by default? Despite the comment on
>>> Options.py, IMHO we should emulate Python semantics here. A METH_O
>>> method cannot be called like this "func(arg, **{})", but that works
>>> for methods defined in Python code.
>>
>> This optimization is on by default because it is extremely common and
>> does have a noticeable impact on timings (don't recall offhand any
>> actual numbers here). Now that we have compiler directives, we should
>> be using them here, but unlike the C division case I think it's a bad
>> idea to change the default because it gives a (rather comprehensible)
>> error rather than silently continuing with different behavior.

OK, I agree...

>> You're
>> the second person since this optimization was introduced well over a
>> year ago to even notice.
>

Just because now I need that all the args of my methods can be passed
as keywords...


> I agree. It's extremely rare to call one-arg functions with keywords (and
> there are tons of C extensions out there that disallow it already). The
> only sensible case I can think of is the generic function wrapper, i.e.
>
> ? ?def wrap(f, *args, **kwargs):
> ? ? ? ?...
> ? ? ? ?f(*args, **kwargs)
>
> BTW, given that the above raises an error (as you said, didn't try), I'm
> more inclined to consider that a bug in CPython. If the kwdict is empty, it
> can just be ignored in the call mechanism instead of letting the call fail.
> There'd be nothing wrong with that.
>

Sorry, I got confused and provided a bad example... Your "wrap" do
work as long as you your single method appears in *args. In short,
what I need is that methods behave as if they were defined in pure
Python code.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Apr 24 17:17:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Apr 2009 12:17:43 -0300
Subject: [Cython] a question about Shadow.py
Message-ID: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>

I'm working on #295, but there is something I do not understand... We
can use directives as decorators like this:

@cython.cdivision(True)
def foo(): pass

But then... Why Shadow.py does not expose a "cdivision" for using it
as a decorator??



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Apr 24 17:33:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Apr 2009 12:33:09 -0300
Subject: [Cython] asking for advice on patch ticket #295 (compiler directive
	to control METH_O and METH_NOARGS flags)
Message-ID: <e7ba66e40904240833n41ace96bs8773fc6901facc33@mail.gmail.com>

I have a patch almost ready... but

1) Now that we have a proper compiler directive, should I remove the
global option in Options.py ??

2) Up to now, I'm named the directive "optimize_simple_methods" ...
but it is a bit longish, any better name? If not, should I use
"..._methods" in plural, or just "..._method" ?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Apr 24 19:58:10 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Apr 2009 19:58:10 +0200
Subject: [Cython] Cython now runs on Py3
Message-ID: <49F1FDB2.5080909@behnel.de>

Hi,

I just managed to integrate the 2to3 tool into Cython's build, so that the
sources get translated to Py3 on install.

This means that Cython now installs and works on Python 3. I tested this
with Py3.1a1.

The relevant changes are in cython-unstable, so you can pull them from there:

http://hg.cython.org/cython-unstable/

Note that you have to *install* Cython under Py3 in order to use it. It
will not work to run the test suite directly from the source directory, and
Cython will not compile itself in Py3.

There are still a few failing tests that need fixing, but it's usually just
the doctests that are broken, not the generated code

If you use Py3, please give it a try.

Stefan

From stefan_ml at behnel.de  Fri Apr 24 20:48:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Apr 2009 20:48:07 +0200
Subject: [Cython] about Python completeness,
	other round about bytes/unicode
In-Reply-To: <e7ba66e40904230936j313d256di2d812358b25ab3af@mail.gmail.com>
References: <e7ba66e40904230936j313d256di2d812358b25ab3af@mail.gmail.com>
Message-ID: <49F20967.4000406@behnel.de>

Hi Lisandro,

Lisandro Dalcin wrote:
> Why then cannot I have an option or compiler directive (or perhaps
> just a global option) that will make bare "abc" string literals become
> PyString in Py2 and PyUnicode in Py3?
> 
> Stefan, Can you imagine any potential problem with this?

I'm still convinced that this will do more harm than good, as it will break
user code in the same way that Python 3 breaks user code. Your argument is
that a compiler directive would solve this, but that's because you know how
to handle these things and want a way to do that. Unicode is something you
have to care about to get it right, and many people do not care until they
notice they have a problem.


> Supose you construct a dict with string keys on Cython code, and that
> dict is intended to be used as a **kargs for calling some function.
> Moreover, supose you construct such dict as a literal, and the keys
> are also string literals... How can I write that in such a way that
> works both in Py2 and Py3 without duplicating code?

So far, the use case you present is the only real use case that I can see.
Risking user code breakage just for this one case sounds like a rather
large drawback.

One way to achieve what you want in current Cython (or at least a common
incarnation of this problem) is this:

    kwargs = dict(argument1 = some_value, arg2 = u"test")

This will create a dict with identifiers as keys, i.e. strings that are
byte strings in Py2 and unicode strings in Py3. It will also do this
efficiently as it is transformed into a literal dict construction.

The more general case, where you build a dict step by step or add keywords
to it, is not currently efficient, but you can always do the above and then
use the resulting dict to update another dict. Given that keyword arguments
are a lot slower than positional arguments, and that dynamically
constructed sets of keyword arguments are rather rare, I doubt that this
use case is performance critical enough to change the string semantics just
to achieve a performance improvement here.

Stefan

From stefan_ml at behnel.de  Fri Apr 24 20:50:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Apr 2009 20:50:44 +0200
Subject: [Cython] a question about Shadow.py
In-Reply-To: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
Message-ID: <49F20A04.4040102@behnel.de>


Lisandro Dalcin wrote:
> I'm working on #295, but there is something I do not understand... We
> can use directives as decorators like this:
> 
> @cython.cdivision(True)
> def foo(): pass
> 
> But then... Why Shadow.py does not expose a "cdivision" for using it
> as a decorator??

Sounds like an oversight to me.

Stefan


From robertwb at math.washington.edu  Fri Apr 24 21:15:52 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Apr 2009 12:15:52 -0700
Subject: [Cython] PATCH: honoring Options.optimize_simple_methods for
	methods and not just for functions
In-Reply-To: <e7ba66e40904240717r62c6b552x8ca6ac6eab42a4e1@mail.gmail.com>
References: <e7ba66e40904231900m65c057e0s85c689185e5e5a94@mail.gmail.com>
	<27A96D63-8B70-4FE8-B472-DEDB74C0B807@math.washington.edu>
	<49F1461B.7090106@behnel.de>
	<e7ba66e40904240717r62c6b552x8ca6ac6eab42a4e1@mail.gmail.com>
Message-ID: <0DF9AF70-FFCD-4334-B708-99DBD869E542@math.washington.edu>

On Apr 24, 2009, at 7:17 AM, Lisandro Dalcin wrote:

> On Fri, Apr 24, 2009 at 1:54 AM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>>
>> Robert Bradshaw wrote:
>>> On Apr 23, 2009, at 7:00 PM, Lisandro Dalcin wrote:
>>>> BTW, Why this optimization is on by default? Despite the comment on
>>>> Options.py, IMHO we should emulate Python semantics here. A METH_O
>>>> method cannot be called like this "func(arg, **{})", but that works
>>>> for methods defined in Python code.
>>>
>>> This optimization is on by default because it is extremely common  
>>> and
>>> does have a noticeable impact on timings (don't recall offhand any
>>> actual numbers here). Now that we have compiler directives, we  
>>> should
>>> be using them here, but unlike the C division case I think it's a  
>>> bad
>>> idea to change the default because it gives a (rather  
>>> comprehensible)
>>> error rather than silently continuing with different behavior.
>
> OK, I agree...
>
>>> You're
>>> the second person since this optimization was introduced well over a
>>> year ago to even notice.
>>
>
> Just because now I need that all the args of my methods can be passed
> as keywords...

We'll make this into a compiler directive the next release then.

- Robert



From robertwb at math.washington.edu  Fri Apr 24 21:20:33 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Apr 2009 12:20:33 -0700
Subject: [Cython] asking for advice on patch ticket #295 (compiler
	directive to control METH_O and METH_NOARGS flags)
In-Reply-To: <e7ba66e40904240833n41ace96bs8773fc6901facc33@mail.gmail.com>
References: <e7ba66e40904240833n41ace96bs8773fc6901facc33@mail.gmail.com>
Message-ID: <7355E4E1-987F-4323-AB88-20DAF1C5D5FF@math.washington.edu>

On Apr 24, 2009, at 8:33 AM, Lisandro Dalcin wrote:

> I have a patch almost ready... but
>
> 1) Now that we have a proper compiler directive, should I remove the
> global option in Options.py ??

Yes, once the directive is implemented.

> 2) Up to now, I'm named the directive "optimize_simple_methods" ...
> but it is a bit longish, any better name? If not, should I use
> "..._methods" in plural, or just "..._method" ?

Maybe

@cython.simple_calling(False)
def foo():
     ...

or optimized_call or simple_dispatch or ??

I'd like to hear some more suggestions though, the original name is  
pretty verbose to actually expose to users...

- Robert



From robertwb at math.washington.edu  Fri Apr 24 21:22:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Apr 2009 12:22:39 -0700
Subject: [Cython] Cython now runs on Py3
In-Reply-To: <49F1FDB2.5080909@behnel.de>
References: <49F1FDB2.5080909@behnel.de>
Message-ID: <B8ADF778-1D38-45CB-990C-D616ACB9BB40@math.washington.edu>

On Apr 24, 2009, at 10:58 AM, Stefan Behnel wrote:

> Hi,
>
> I just managed to integrate the 2to3 tool into Cython's build, so  
> that the
> sources get translated to Py3 on install.
>
> This means that Cython now installs and works on Python 3. I tested  
> this
> with Py3.1a1.

Very cool! Personally, my target will be 2.x for a while (until Sage  
switches, which in turn depends on a large number of upstream  
packages switching), but I still think this is a very good thing to  
have.

- Robert


From robertwb at math.washington.edu  Fri Apr 24 21:27:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Apr 2009 12:27:11 -0700
Subject: [Cython] a question about Shadow.py
In-Reply-To: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
Message-ID: <39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu>

On Apr 24, 2009, at 8:17 AM, Lisandro Dalcin wrote:

> I'm working on #295, but there is something I do not understand... We
> can use directives as decorators like this:
>
> @cython.cdivision(True)
> def foo(): pass
>
> But then... Why Shadow.py does not expose a "cdivision" for using it
> as a decorator??

Because I forgot to add it when I added cdiv/cmod. Note that  
cdivision does not guarantee certain semantics for division, just  
says that Python semantics are not enforced.

The other compiler directives should be added as well.

- Robert


From dalcinl at gmail.com  Fri Apr 24 22:46:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Apr 2009 17:46:27 -0300
Subject: [Cython] Cython now runs on Py3
In-Reply-To: <49F1FDB2.5080909@behnel.de>
References: <49F1FDB2.5080909@behnel.de>
Message-ID: <e7ba66e40904241346x525476f5o1f6be9d05e598251@mail.gmail.com>

On Fri, Apr 24, 2009 at 2:58 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I just managed to integrate the 2to3 tool into Cython's build, so that the
> sources get translated to Py3 on install.
>

Does 2to3 run under Py3? I'm thinking about an import hook to make
runtests use it in Py3 ...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Apr 24 22:48:45 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Apr 2009 17:48:45 -0300
Subject: [Cython] a question about Shadow.py
In-Reply-To: <39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
	<39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu>
Message-ID: <e7ba66e40904241348wba1528bl51fd3ca359700ec6@mail.gmail.com>

On Fri, Apr 24, 2009 at 4:27 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 24, 2009, at 8:17 AM, Lisandro Dalcin wrote:
>
>> I'm working on #295, but there is something I do not understand... We
>> can use directives as decorators like this:
>>
>> @cython.cdivision(True)
>> def foo(): pass
>>
>> But then... Why Shadow.py does not expose a "cdivision" for using it
>> as a decorator??
>
> Because I forgot to add it when I added cdiv/cmod. Note that
> cdivision does not guarantee certain semantics for division, just
> says that Python semantics are not enforced.
>

Of course, then if  @cython.cdivision(True) is used, the code in
Shadow should emit a warning, right?

>
> The other compiler directives should be added as well.
>

OK, though I likely will not take this...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Apr 24 23:08:32 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Apr 2009 18:08:32 -0300
Subject: [Cython] asking for advice on patch ticket #295 (compiler
	directive to control METH_O and METH_NOARGS flags)
In-Reply-To: <7355E4E1-987F-4323-AB88-20DAF1C5D5FF@math.washington.edu>
References: <e7ba66e40904240833n41ace96bs8773fc6901facc33@mail.gmail.com>
	<7355E4E1-987F-4323-AB88-20DAF1C5D5FF@math.washington.edu>
Message-ID: <e7ba66e40904241408w6df7c20dy31c7a85bb5dad408@mail.gmail.com>

On Fri, Apr 24, 2009 at 4:20 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 24, 2009, at 8:33 AM, Lisandro Dalcin wrote:
>
>> I have a patch almost ready... but
>>
>> 1) Now that we have a proper compiler directive, should I remove the
>> global option in Options.py ??
>
> Yes, once the directive is implemented.
>
>> 2) Up to now, I'm named the directive "optimize_simple_methods" ...
>> but it is a bit longish, any better name? If not, should I use
>> "..._methods" in plural, or just "..._method" ?
>
> Maybe
>
> @cython.simple_calling(False)
> def foo():
> ? ? ...

I like this, but not strongly... I would be fine with anything...

I would not bother too much, perhaps I would be the only one using this ;-)



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Apr 24 23:41:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Apr 2009 14:41:55 -0700
Subject: [Cython] a question about Shadow.py
In-Reply-To: <e7ba66e40904241348wba1528bl51fd3ca359700ec6@mail.gmail.com>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
	<39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu>
	<e7ba66e40904241348wba1528bl51fd3ca359700ec6@mail.gmail.com>
Message-ID: <F7AC106B-4809-40DF-9E4A-F8774B5BAEA4@math.washington.edu>

On Apr 24, 2009, at 1:48 PM, Lisandro Dalcin wrote:

> On Fri, Apr 24, 2009 at 4:27 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 24, 2009, at 8:17 AM, Lisandro Dalcin wrote:
>>
>>> I'm working on #295, but there is something I do not  
>>> understand... We
>>> can use directives as decorators like this:
>>>
>>> @cython.cdivision(True)
>>> def foo(): pass
>>>
>>> But then... Why Shadow.py does not expose a "cdivision" for using it
>>> as a decorator??
>>
>> Because I forgot to add it when I added cdiv/cmod. Note that
>> cdivision does not guarantee certain semantics for division, just
>> says that Python semantics are not enforced.
>>
>
> Of course, then if  @cython.cdivision(True) is used, the code in
> Shadow should emit a warning, right?

No idea how one would do this...

>
>>
>> The other compiler directives should be added as well.
>>
>
> OK, though I likely will not take this...
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri Apr 24 23:42:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Apr 2009 14:42:28 -0700
Subject: [Cython] asking for advice on patch ticket #295 (compiler
	directive to control METH_O and METH_NOARGS flags)
In-Reply-To: <e7ba66e40904241408w6df7c20dy31c7a85bb5dad408@mail.gmail.com>
References: <e7ba66e40904240833n41ace96bs8773fc6901facc33@mail.gmail.com>
	<7355E4E1-987F-4323-AB88-20DAF1C5D5FF@math.washington.edu>
	<e7ba66e40904241408w6df7c20dy31c7a85bb5dad408@mail.gmail.com>
Message-ID: <AE7A5887-FD8F-4143-9452-58E86AE74A25@math.washington.edu>

On Apr 24, 2009, at 2:08 PM, Lisandro Dalcin wrote:

> On Fri, Apr 24, 2009 at 4:20 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 24, 2009, at 8:33 AM, Lisandro Dalcin wrote:
>>
>>> I have a patch almost ready... but
>>>
>>> 1) Now that we have a proper compiler directive, should I remove the
>>> global option in Options.py ??
>>
>> Yes, once the directive is implemented.
>>
>>> 2) Up to now, I'm named the directive "optimize_simple_methods" ...
>>> but it is a bit longish, any better name? If not, should I use
>>> "..._methods" in plural, or just "..._method" ?
>>
>> Maybe
>>
>> @cython.simple_calling(False)
>> def foo():
>>     ...
>
> I like this, but not strongly... I would be fine with anything...
>
> I would not bother too much, perhaps I would be the only one using  
> this ;-)

It's worth it, it's ugly sticking things in Options.py anyways.

- Robert

From dalcinl at gmail.com  Sat Apr 25 00:07:47 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Apr 2009 19:07:47 -0300
Subject: [Cython] a question about Shadow.py
In-Reply-To: <F7AC106B-4809-40DF-9E4A-F8774B5BAEA4@math.washington.edu>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
	<39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu>
	<e7ba66e40904241348wba1528bl51fd3ca359700ec6@mail.gmail.com>
	<F7AC106B-4809-40DF-9E4A-F8774B5BAEA4@math.washington.edu>
Message-ID: <e7ba66e40904241507h44cc2d34x11ef714330844a2e@mail.gmail.com>

On Fri, Apr 24, 2009 at 6:41 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 24, 2009, at 1:48 PM, Lisandro Dalcin wrote:
>
>> On Fri, Apr 24, 2009 at 4:27 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>> On Apr 24, 2009, at 8:17 AM, Lisandro Dalcin wrote:
>>
>> Of course, then if ?@cython.cdivision(True) is used, the code in
>> Shadow should emit a warning, right?
>
> No idea how one would do this...
>

Perhaps something like this (sorry if identation is wrong, still
missing native support for fixed-width fonts when composing mails in
Gmail)

# In Shadow.py
class cdivision:
? def __init__(self, value):
? ? ? ?assert type(value) is bool
? ? ? ?self.value = value
? def __call__(self, function):
? ? ? ?if self.value: warn("C division semantics cannot work on Python code")
? ? ? ?return function



>>
>>>
>>> The other compiler directives should be added as well.
>>>
>>
>> OK, though I likely will not take this...
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



--
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Apr 25 04:52:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Apr 2009 04:52:47 +0200
Subject: [Cython] Cython now runs on Py3
In-Reply-To: <e7ba66e40904241346x525476f5o1f6be9d05e598251@mail.gmail.com>
References: <49F1FDB2.5080909@behnel.de>
	<e7ba66e40904241346x525476f5o1f6be9d05e598251@mail.gmail.com>
Message-ID: <49F27AFF.4080700@behnel.de>


Lisandro Dalcin wrote:
> On Fri, Apr 24, 2009 at 2:58 PM, Stefan Behnel wrote:
>> I just managed to integrate the 2to3 tool into Cython's build, so that the
>> sources get translated to Py3 on install.
> 
> Does 2to3 run under Py3?

It runs /only/ on Py3.


> I'm thinking about an import hook to make runtests use it in Py3 ...

You could do what setup.py currently does: use distutils' build_py_2to3 to
transform the sources so that they end up in the build/... directory (maybe
you can retarget that), and then simply prepend that to sys.path before you
import Cython. That also works incrementally, so it will only work on
sources with modifications on later calls.

Stefan

From stefan_ml at behnel.de  Sat Apr 25 05:01:13 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Apr 2009 05:01:13 +0200
Subject: [Cython] Cython now runs on Py3
In-Reply-To: <49F27AFF.4080700@behnel.de>
References: <49F1FDB2.5080909@behnel.de>	<e7ba66e40904241346x525476f5o1f6be9d05e598251@mail.gmail.com>
	<49F27AFF.4080700@behnel.de>
Message-ID: <49F27CF9.5090500@behnel.de>


Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>> I'm thinking about an import hook to make runtests use it in Py3 ...
> 
> You could do what setup.py currently does: use distutils' build_py_2to3 to
> transform the sources so that they end up in the build/... directory (maybe
> you can retarget that), and then simply prepend that to sys.path before you
> import Cython. That also works incrementally, so it will only work on
> sources with modifications on later calls.

On second thought, maybe we should always install Cython to a local
directory as part of the test suite run. That would add compiling Cython to
the normal test run as well (which would prevent a couple of build
breakages that we had in the past).

Part of that would be to make Cython's build truly incremental (mainly the
compiler run), so that the test suite run doesn't unnecessarily loose time.

Stefan


From robertwb at math.washington.edu  Sat Apr 25 09:43:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Apr 2009 00:43:51 -0700
Subject: [Cython] a question about Shadow.py
In-Reply-To: <e7ba66e40904241507h44cc2d34x11ef714330844a2e@mail.gmail.com>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
	<39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu>
	<e7ba66e40904241348wba1528bl51fd3ca359700ec6@mail.gmail.com>
	<F7AC106B-4809-40DF-9E4A-F8774B5BAEA4@math.washington.edu>
	<e7ba66e40904241507h44cc2d34x11ef714330844a2e@mail.gmail.com>
Message-ID: <3A0D02E0-A9A1-4A21-B455-20E03B13A85F@math.washington.edu>

On Apr 24, 2009, at 3:07 PM, Lisandro Dalcin wrote:

> On Fri, Apr 24, 2009 at 6:41 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 24, 2009, at 1:48 PM, Lisandro Dalcin wrote:
>>
>>> On Fri, Apr 24, 2009 at 4:27 PM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>>> On Apr 24, 2009, at 8:17 AM, Lisandro Dalcin wrote:
>>>
>>> Of course, then if  @cython.cdivision(True) is used, the code in
>>> Shadow should emit a warning, right?
>>
>> No idea how one would do this...
>>
>
> Perhaps something like this

Sorry, for some reason I was thinking about  
@cython.cdivision_warnings which would be really hard to emulate in  
pure Python...

> (sorry if identation is wrong, still
> missing native support for fixed-width fonts when composing mails in
> Gmail)
>
> # In Shadow.py
> class cdivision:
>   def __init__(self, value):
>        assert type(value) is bool
>        self.value = value
>   def __call__(self, function):
>        if self.value: warn("C division semantics cannot work on  
> Python code")
>        return function

C has undefined division semantics, so Python semantics are fine with  
cython.cdivision(True) (though I have yet to run into that in  
practice). However, almost every compiler directive can't be emulated  
in pure Python mode (e.g. we don't truncate integers), so I am of the  
opinion that they should just be vacuous and ignored rather than all  
raise warnings.

- Robert


From robertwb at math.washington.edu  Sat Apr 25 10:07:18 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Apr 2009 01:07:18 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<49F16BC1.5050708@student.matnat.uio.no>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
Message-ID: <44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>

On Apr 24, 2009, at 1:24 AM, Dag Sverre Seljebotn wrote:

> BTW, sorry to go off in a digression like this -- it is just  
> something a
> problem I remembered with this and I am writing as much to Danilo and
> Robert as you. At least I also answered your question :-)

Yep, this is a good discussion to be having right now...

> Some more thoughts:
>
>> Roland Schulz wrote:
>>> Hi,
>>>
>>> is it possible (or planned) to add operators to a ctypdef struct?
>>
>> Better C++ support is definitely planned; in fact there is a GSoC
>> project on it where this could likely be a part (though Danilo or  
>> Robert
>> would have to comment on that).
>>
>> I started a wiki page listing the things that could be improved  
>> for C++
>> support and
>>
>> http://wiki.cython.org/enhancements/cpp
>>
>> but I became exhausted right before fixing up the operator  
>> overloading
>> section; perhaps you could have a look and come with suggestions?

It would be good, Danilo, if you could flesh this out some more as a  
starting point (including tagging priorities and approximate  
timelines for the stuff you plan on doing as part of your project).

>> Especially, I'm concerned with operators typically returning  
>> references
>> like operator[]:
>>
>> cdef struct Polygon:
>>      Point& operator[](int edge)
>>
>> where one would like to be able to do
>>
>> mypoly[3] = mypoint
>>
>> However this also contradicts the scetched solution for dealing with
>> references which means having Cython treat references as pointers;
>> making it necesarry to do
>>
>> mypoly[3][0] = mypoint
>>
>> Perhaps some map between Python's __setitem__ and reference return
>> values in C++ operator[], but it gets messy syntax-wise quickly...
>
> I should probably provide some more context here. In Python there's
> getitem and setitem as seperate operators. In C++ you would instead  
> return
> an object with overridden assignment operator (for which there is no
> Python equivalent, of course, as assignments are not allowed to have
> side-effects in Python).
>
> So in the example
>
> Point& operator[](int edge);
>
> if you wanted to print something when changing a point, you could do
>
> PointInPolygon operator[](int edge);
>
> instead, and PointInPolygon would contain both a reference to the  
> Polygon
> and the index of the edge, and have overloaded "operator=" and  
> "operator
> Point" which would work as setitem and getitem, respecticely.
>
> So what I'm saying is: Mapping C++ operators to Python operators  
> and vice
> versa is difficult, with possibly many special cases where e.g. a  
> Python
> setitem operator would map to an entire C++ expression involving many
> operators.
>
> Starting out with the binary operators only is simpler and probably  
> a good
> idea :-)

Perhaps I'm taking a too simplistic view of things, but it seems that  
all we have to do is figure out whether or not we can pass a given  
operator onto the C++ code, which is much easier to reason about.  
Also, because in Python assignment is not an expression, we don't  
have to worry about its return value.

The point about handling references (esp. with regard to set- vs.  
getitem still holds, I'm not sure the best way to handle this.

- Robert



From dagss at student.matnat.uio.no  Sat Apr 25 11:59:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 25 Apr 2009 11:59:57 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
Message-ID: <49F2DF1D.7090403@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 24, 2009, at 1:24 AM, Dag Sverre Seljebotn wrote:
>> So in the example
>>
>> Point& operator[](int edge);
>>
>> if you wanted to print something when changing a point, you could do
>>
>> PointInPolygon operator[](int edge);
>>
>> instead, and PointInPolygon would contain both a reference to the  
>> Polygon
>> and the index of the edge, and have overloaded "operator=" and  
>> "operator
>> Point" which would work as setitem and getitem, respecticely.
>>
>> So what I'm saying is: Mapping C++ operators to Python operators  
>> and vice
>> versa is difficult, with possibly many special cases where e.g. a  
>> Python
>> setitem operator would map to an entire C++ expression involving many
>> operators.
>>
>> Starting out with the binary operators only is simpler and probably  
>> a good
>> idea :-)
> 
> Perhaps I'm taking a too simplistic view of things, but it seems that  
> all we have to do is figure out whether or not we can pass a given  
> operator onto the C++ code, which is much easier to reason about.  
> Also, because in Python assignment is not an expression, we don't  
> have to worry about its return value.

Hmm. That's a nice perspective to have; but I don't think it is 
"obvious" and leads to some fairly concrete consequences IMO: Operators 
must be declared using the Python operator syntax. So you are never 
declaring anything about "operator[]" in Cython, instead you are 
declaring "what types are allowed for the full statement 'x[i] = y'".

(If we start using C++ syntax but don't support the whole thing in the 
same way it will bring lots of confusion I think.)

This pushes the hard work over to writing of the C++ pxd wrapper 
(whether that's a person or in time an automatic script). I think that's 
a good idea. It also makes Cython's grammar simpler than the alternative.

However, a consequence is that if we support overloading, we must also 
support overloading solely on return values :-) See below.

Some examples:
   int operator[](int idx)
would give
   int __getitem__(int idx)

while

   int& operator[](int idx)
would be declared as
   int __getitem__(int idx)
   void __setitem__(int idx, int value)


A more complicated example, with a collection which can retrieve items 
as either type A or B, but store them as types C or D, using either key 
type K or T:

class ItemAccessor {
     operator A() { ... } // coerce to A
     operator B() { ...}
     ItemAccessor operator=(C value) {
         ...
     }
     ItemAccessor operator=(D value) {
         ...
     }
     ItemAccessor(MyCollection col, K idx) { ... }
     ItemAccessor(MyCollection col, T idx) { ... }
}

class MyCollection {
     ItemAccessor operator[](K idx) { return ItemAccessor(this, idx);}
     ItemAccessor operator[](T idx) { return ItemAccessor(this, idx);}
}

In Cython this would be represtented rather different:

class MyCollection:
     A __getitem__(K idx)
     A __getitem__(T idx)
     B __getitem__(K idx) # note: overloads on return value...
     B __getitem__(T idx)
     void __setitem__(K idx, A value)
     void __setitem__(T idx, A value)
     void __setitem__(K idx, B value)
     void __setitem__(T idx, B value)

So it's not too bad :-)  A combinatorial explosion happens but I don't 
worry too much, in practice it's not likely to become large.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Apr 25 12:04:17 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 25 Apr 2009 12:04:17 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F2DF1D.7090403@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
Message-ID: <49F2E021.5000209@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> On Apr 24, 2009, at 1:24 AM, Dag Sverre Seljebotn wrote:
>>> So in the example
>>>
>>> Point& operator[](int edge);
>>>
>>> if you wanted to print something when changing a point, you could do
>>>
>>> PointInPolygon operator[](int edge);
>>>
>>> instead, and PointInPolygon would contain both a reference to the  
>>> Polygon
>>> and the index of the edge, and have overloaded "operator=" and  
>>> "operator
>>> Point" which would work as setitem and getitem, respecticely.
>>>
>>> So what I'm saying is: Mapping C++ operators to Python operators  
>>> and vice
>>> versa is difficult, with possibly many special cases where e.g. a  
>>> Python
>>> setitem operator would map to an entire C++ expression involving many
>>> operators.
>>>
>>> Starting out with the binary operators only is simpler and probably  
>>> a good
>>> idea :-)
>> Perhaps I'm taking a too simplistic view of things, but it seems that  
>> all we have to do is figure out whether or not we can pass a given  
>> operator onto the C++ code, which is much easier to reason about.  
>> Also, because in Python assignment is not an expression, we don't  
>> have to worry about its return value.
> 
> Hmm. That's a nice perspective to have; but I don't think it is 
> "obvious" and leads to some fairly concrete consequences IMO: Operators 
> must be declared using the Python operator syntax. So you are never 
> declaring anything about "operator[]" in Cython, instead you are 
> declaring "what types are allowed for the full statement 'x[i] = y'".
> 
> (If we start using C++ syntax but don't support the whole thing in the 
> same way it will bring lots of confusion I think.)
> 
> This pushes the hard work over to writing of the C++ pxd wrapper 
> (whether that's a person or in time an automatic script). I think that's 
> a good idea. It also makes Cython's grammar simpler than the alternative.
> 
> However, a consequence is that if we support overloading, we must also 
> support overloading solely on return values :-) See below.
> 
> Some examples:
>    int operator[](int idx)
> would give
>    int __getitem__(int idx)
> 
> while
> 
>    int& operator[](int idx)
> would be declared as
>    int __getitem__(int idx)
>    void __setitem__(int idx, int value)
> 
> 
> A more complicated example, with a collection which can retrieve items 
> as either type A or B, but store them as types C or D, using either key 
> type K or T:
> 
> class ItemAccessor {
>      operator A() { ... } // coerce to A
>      operator B() { ...}
>      ItemAccessor operator=(C value) {
>          ...
>      }
>      ItemAccessor operator=(D value) {
>          ...
>      }
>      ItemAccessor(MyCollection col, K idx) { ... }
>      ItemAccessor(MyCollection col, T idx) { ... }
> }
> 
> class MyCollection {
>      ItemAccessor operator[](K idx) { return ItemAccessor(this, idx);}
>      ItemAccessor operator[](T idx) { return ItemAccessor(this, idx);}
> }
> 
> In Cython this would be represtented rather different:
> 
> class MyCollection:
>      A __getitem__(K idx)
>      A __getitem__(T idx)
>      B __getitem__(K idx) # note: overloads on return value...
>      B __getitem__(T idx)
>      void __setitem__(K idx, A value)
>      void __setitem__(T idx, A value)
>      void __setitem__(K idx, B value)
>      void __setitem__(T idx, B value)

Note that it should be C and D for the last four lines.

(Seems my C++ is a bit rusty as well but I won't bother correcting it, 
consider it psuedo-code :-) )

-- 
Dag Sverre

From carl.witty at gmail.com  Sat Apr 25 17:36:18 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Sat, 25 Apr 2009 08:36:18 -0700
Subject: [Cython] a question about Shadow.py
In-Reply-To: <3A0D02E0-A9A1-4A21-B455-20E03B13A85F@math.washington.edu>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com> 
	<39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu> 
	<e7ba66e40904241348wba1528bl51fd3ca359700ec6@mail.gmail.com> 
	<F7AC106B-4809-40DF-9E4A-F8774B5BAEA4@math.washington.edu> 
	<e7ba66e40904241507h44cc2d34x11ef714330844a2e@mail.gmail.com> 
	<3A0D02E0-A9A1-4A21-B455-20E03B13A85F@math.washington.edu>
Message-ID: <ac1e4c420904250836i85ac01bg83531de71b75b895@mail.gmail.com>

On Sat, Apr 25, 2009 at 12:43 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> C has undefined division semantics, so Python semantics are fine with
> cython.cdivision(True) (though I have yet to run into that in
> practice). However, almost every compiler directive can't be emulated
> in pure Python mode (e.g. we don't truncate integers), so I am of the
> opinion that they should just be vacuous and ignored rather than all
> raise warnings.

The current version of ANSI C has defined division semantics (the
first version of ANSI C did not).

Carl

From robertwb at math.washington.edu  Sat Apr 25 18:34:29 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Apr 2009 09:34:29 -0700
Subject: [Cython] a question about Shadow.py
In-Reply-To: <ac1e4c420904250836i85ac01bg83531de71b75b895@mail.gmail.com>
References: <e7ba66e40904240817t24735256o590401806ff5f4ea@mail.gmail.com>
	<39B9279C-BC89-4D77-815A-3BBF26C2F7DF@math.washington.edu>
	<e7ba66e40904241348wba1528bl51fd3ca359700ec6@mail.gmail.com>
	<F7AC106B-4809-40DF-9E4A-F8774B5BAEA4@math.washington.edu>
	<e7ba66e40904241507h44cc2d34x11ef714330844a2e@mail.gmail.com>
	<3A0D02E0-A9A1-4A21-B455-20E03B13A85F@math.washington.edu>
	<ac1e4c420904250836i85ac01bg83531de71b75b895@mail.gmail.com>
Message-ID: <90BA23EB-20F9-48B2-8EA9-FB08C6A59783@math.washington.edu>

On Apr 25, 2009, at 8:36 AM, Carl Witty wrote:

> On Sat, Apr 25, 2009 at 12:43 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> C has undefined division semantics, so Python semantics are fine with
>> cython.cdivision(True) (though I have yet to run into that in
>> practice). However, almost every compiler directive can't be emulated
>> in pure Python mode (e.g. we don't truncate integers), so I am of the
>> opinion that they should just be vacuous and ignored rather than all
>> raise warnings.
>
> The current version of ANSI C has defined division semantics (the
> first version of ANSI C did not).

Yes, it does, but we don't rely on things being c99+ compliant.

- Robert


From robertwb at math.washington.edu  Sat Apr 25 18:41:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Apr 2009 09:41:25 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F2DF1D.7090403@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
Message-ID: <FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>

On Apr 25, 2009, at 2:59 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>
>> Perhaps I'm taking a too simplistic view of things, but it seems that
>> all we have to do is figure out whether or not we can pass a given
>> operator onto the C++ code, which is much easier to reason about.
>> Also, because in Python assignment is not an expression, we don't
>> have to worry about its return value.
>
> Hmm. That's a nice perspective to have; but I don't think it is
> "obvious" and leads to some fairly concrete consequences IMO:  
> Operators
> must be declared using the Python operator syntax.

I'm not so sure about this, I think using declaring the C++ operatorX  
notation may be more transparent to what is actually going on,  
despite a (minor) complication in the parser.

> So you are never
> declaring anything about "operator[]" in Cython, instead you are
> declaring "what types are allowed for the full statement 'x[i] = y'".

Yes, I think that's as far as we need to go.

- Robert


From dagss at student.matnat.uio.no  Sat Apr 25 18:54:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 25 Apr 2009 18:54:29 +0200 (CEST)
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<49F16BC1.5050708@student.matnat.uio.no>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
Message-ID: <d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Apr 25, 2009, at 2:59 AM, Dag Sverre Seljebotn wrote:
>
>> Robert Bradshaw wrote:
>>>
>>> Perhaps I'm taking a too simplistic view of things, but it seems that
>>> all we have to do is figure out whether or not we can pass a given
>>> operator onto the C++ code, which is much easier to reason about.
>>> Also, because in Python assignment is not an expression, we don't
>>> have to worry about its return value.
>>
>> Hmm. That's a nice perspective to have; but I don't think it is
>> "obvious" and leads to some fairly concrete consequences IMO:
>> Operators
>> must be declared using the Python operator syntax.
>
> I'm not so sure about this, I think using declaring the C++ operatorX
> notation may be more transparent to what is actually going on,
> despite a (minor) complication in the parser.

I think maintaining a hybrid solution using C++ syntax but different
semantics (i.e. Cython semantic) is bound to give lots of confusion. I'd
have no idea what a natural syntax would be personally; we would invent
something new which just looked like C++.

How would you represent my latter example in my previous mail at using C++
syntax in Cython?

>> So you are never
>> declaring anything about "operator[]" in Cython, instead you are
>> declaring "what types are allowed for the full statement 'x[i] = y'".
>
> Yes, I think that's as far as we need to go.

This is what I think conflicts with letting the operator[] notation into
Cython. operator[] means nothing of the sort of what we are really
supporting.

C++ is a very complicated language and it's semantics differ a lot from
Cython. Which is why I'm afraid that using C++ syntax in Cython will be
confusing.

Dag Sverre


From dagss at student.matnat.uio.no  Sat Apr 25 21:05:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 25 Apr 2009 21:05:00 +0200
Subject: [Cython] PATCH: Serious temp unpacking bug
Message-ID: <49F35EDC.7000503@student.matnat.uio.no>

As the web servers appears to be down at the moment, I'm posting it here 
for the moment.

This is the bug reported recently. It's related to a problem in 
allocate_temps of SequenceNode (basically two nodes could get the same 
temp), and has likely been there for a long time.

It hits some nested tuple unpacking cases depending on the number of 
variables in the nested vs. outer tuple.

The refnanny catches a segfault on trivial code like this:

def g():
     return ((3, 2), 1, 0)

def func2():
     (a, b), c, d = g()


Of course, the whole function which was buggy has disappeared in 
cython-unstable. So this is kind of depressing :-) But at least now 
0.11.2 isn't blocked by having to merge -unstable.

-- 
Dag Sverre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tupleunpackbug.diff
Type: text/x-diff
Size: 1560 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090425/4c52dad7/attachment.bin 

From robertwb at math.washington.edu  Sat Apr 25 22:50:35 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Apr 2009 13:50:35 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<49F16BC1.5050708@student.matnat.uio.no>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
Message-ID: <4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>

I start with a disclaimer I should have made several posts back: I  
have done very little programming with C++, so please excuse any  
naivet? I have in this regard...

On Apr 25, 2009, at 9:54 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 25, 2009, at 2:59 AM, Dag Sverre Seljebotn wrote:
>>
>>> Robert Bradshaw wrote:
>>>>
>>>> Perhaps I'm taking a too simplistic view of things, but it seems  
>>>> that
>>>> all we have to do is figure out whether or not we can pass a given
>>>> operator onto the C++ code, which is much easier to reason about.
>>>> Also, because in Python assignment is not an expression, we don't
>>>> have to worry about its return value.
>>>
>>> Hmm. That's a nice perspective to have; but I don't think it is
>>> "obvious" and leads to some fairly concrete consequences IMO:
>>> Operators
>>> must be declared using the Python operator syntax.
>>
>> I'm not so sure about this, I think using declaring the C++ operatorX
>> notation may be more transparent to what is actually going on,
>> despite a (minor) complication in the parser.
>
> I think maintaining a hybrid solution using C++ syntax but different
> semantics (i.e. Cython semantic) is bound to give lots of  
> confusion. I'd
> have no idea what a natural syntax would be personally; we would  
> invent
> something new which just looked like C++.
>
> How would you represent my latter example in my previous mail at  
> using C++
> syntax in Cython?

I would expose your ItemAccessor class as well as your MyCollection  
class in the Cython declaration. This may necessitate exposing the  
notion of a C++ reference to Cython as well. At the very least, I  
think trying to overload on the return type is bad.

>>> So you are never
>>> declaring anything about "operator[]" in Cython, instead you are
>>> declaring "what types are allowed for the full statement 'x[i] =  
>>> y'".
>>
>> Yes, I think that's as far as we need to go.
>
> This is what I think conflicts with letting the operator[] notation  
> into
> Cython. operator[] means nothing of the sort of what we are really
> supporting.

Hmm... OK, maybe I'll take back that statement and say Cython does  
need to go deeper than this. Right now one has to learn the "magic"  
of giving Cython declarations to generate the right code, which is  
undesirable. I would rather not make users have to learn a new  
"magic" but really state what's going on on the C++ side.

The only two non-simple lvalues we need to worry about are indexing  
and attribute access, right?

> C++ is a very complicated language and it's semantics differ a lot  
> from
> Cython. Which is why I'm afraid that using C++ syntax in Cython  
> will be
> confusing.

Unless we give them actual C++ semantics.

- Robert


From dsurviver at gmail.com  Sat Apr 25 23:37:06 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Sat, 25 Apr 2009 18:37:06 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<49F16BC1.5050708@student.matnat.uio.no>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
Message-ID: <45239150904251437n6efba4bbme6746a346579593f@mail.gmail.com>

I'll consider all this for my project on GSoC.

For overloading operators, template maybe solve this, maybe not.
But I don't think using template for this is a good idea.


The following code:

cdef extern from "Rectangle.h":
    ctypedef struct Rectangle:
        Rectangle(int) # constructor takes an int
        Rectangle operator+(Rectangle)
        Coordinate& operator[](Corner corner) # & means result is assignable

from Dag wiki for enchacements, as said there, a change in the parser
will be needed.

I think that if we can make the syntax closer to Python, it would be better.
We could use something like __cadd__, __csetitem__, __cgetitem__, etc.
I think it would be more interesting, and still more for Cython users
(I think Cython users know more Python than C/C++).


But for all this, changes in the parser are needed.

From lrhazi at gmail.com  Sun Apr 26 03:25:46 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Sat, 25 Apr 2009 21:25:46 -0400
Subject: [Cython] Callback functions for C lib in Python
Message-ID: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>

Hello,

I am trying to write an extension module, to make a C library
available for use by the Python user.

The lib needs to be passed pointers to callback functions. how do I
code this in Cython, so the actual functions are provided by the
Python code?

Any code examples I could look at for inspiration?

Thanks a lot.
Mohamed.

From common at gmx.ch  Sun Apr 26 03:35:32 2009
From: common at gmx.ch (common at gmx.ch)
Date: Sun, 26 Apr 2009 03:35:32 +0200
Subject: [Cython] Cython & python3: str/bytes
Message-ID: <49F3BA64.6050407@gmx.ch>

Hi,

having issues with cython/python3 I found the thread
http://www.mail-archive.com/cython-dev at codespeak.net/msg02346.html

Actually, I have the same problem:
As *all* strings in cython code get converted to bytes, nothing works out.

I my case I have a c function like:

void foo(char *)

cdef c_foo "foo"(char *)


cdef class A:
   def __init__(self,arg0):
     c_foo(arg0)


calling from python with
x = A('test')

in python the argument 'test' is of type str, cython expects the
argument for c_foo to be of type bytes as char * is mapped to bytes.

Now, as I do not want to change all calls to all functions which take
str arguments, I tried to convert 'test' from str to bytes.

arg0 = arg0.encode("ascii")

Result:
TypeError: encode() argument 1 must be string, not bytes.

Even worse:
if isinstance(arg0, str):

actually checks if arg0 is of type bytes ...

Thats really bad, you can neither create nor recognize or convert
python3 str's in cython.


Maybe mapping "char *" to PyString_Type as a default for all internal
strings, and "unsigned char *" to PyBytes_Type?

Any other suggestions?


MfG
Markus


From robertwb at math.washington.edu  Sun Apr 26 04:02:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Apr 2009 19:02:40 -0700
Subject: [Cython] Callback functions for C lib in Python
In-Reply-To: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>
References: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>
Message-ID: <F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>

On Apr 25, 2009, at 6:25 PM, Mohamed Lrhazi wrote:

> Hello,
>
> I am trying to write an extension module, to make a C library
> available for use by the Python user.
>
> The lib needs to be passed pointers to callback functions. how do I
> code this in Cython, so the actual functions are provided by the
> Python code?
>
> Any code examples I could look at for inspiration?

Yes, there's an example of callbacks in the Demos folder of Cython.

- Robert



From kwmsmith at gmail.com  Sun Apr 26 04:08:27 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sat, 25 Apr 2009 21:08:27 -0500
Subject: [Cython] Fortran support design issues
In-Reply-To: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>
References: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>
Message-ID: <c266a7930904251908w76559b51lf4bd9ae4e9e34f57@mail.gmail.com>

Clarifying question:  In #177, it says,

"Currently there's no way of creating efficient functions taking a
buffer argument -- for starters cdef functions doesn't support
declaring an argument as a buffer, and even if it did, it would mean a
costly (in this context) buffer acquisition on every call."

more precisely, should it say "There's no way of creating efficient
functions taking a buffer argument /using the buffer syntax/..." since
one could always just have the function take a Py_buffer(*) argument,
right?


On Fri, Apr 24, 2009 at 3:38 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> (This is a move of my and Kurt's discussion into the public; I'll give
> some context but not that much.)
>
> Kurt: This will go into a broader discussion of buffers in Cython as such
> and not be directly Fortran-relevant. I want to have Fortran support
> integrated in a nice way into Cython, so I think this is necesarry to some
> degree, but I'll try not to overdo it.
>
> So, you wanted to target contiguous buffer passing first. And I was
> saying that fixing
>
> http://trac.cython.org/cython_trac/ticket/177
>
> would for the most part amount to what we need. Adding support for
>
> ? cdef void myfunc(object[int, mode="c"] buf)
>
> which would have the C signature
>
> ? void myfunc(Py_buffer* buf) // + a contract on int/contiguous
>
> and have the caller be responsible for getting the buffer, gets us far.
> It would mean that you could do
>
> ? myfunc(myobject)
>
> and it would turn into
>
> ? Py_buffer buf = acquire buffer from myobject
> ? raise exception if buf is not contiguous
> ? myfunc(&buf)
> ? release buf

Just stating my knowledge as it is right now -- for

def func2(object[int, mode='c'] buf):
    # body here

The 'buf' is passed in as a python object and a stack Py_buffer is
generated inside func2's body that acquires the buffer from buf.  So
for 'def' functions the buffer is acquired/released within the
function scope, while for 'cdef' functions it is acquired/released
outside.  I'd imagine that cpdef functions would be outside, too.

>
> I first thought that adding one component more would get us all the way:
> Automatic copying into contiguous buffers. So assuming #177 is
> implemented, one would then move on to having it instead turn into:
>
> ? Py_buffer buf = acquire buffer from myobject
> ? if (buf is contiguous) {
> ? ? myfunc(&buf);
> ? } else {
> ? ? Py_buffer buf2 = make contiguous copy of buf
> ? ? myfunc(&buf2)
> ? }
> ? release buf
>
> However, there's a (big) problem here: What Python object does buf2 refer to?
>
> Using the one of "buf" would be too confusing as they point to different
> memory areas. One solution is just setting it to None, perhaps. However
> once #177 is solved one will expect to be able to do

How hard would it be to have a contiguous -> strided (or indirect)
buffer copy utility function in Cython, and this copy be triggered
after the myfunc(&buf2) call?  The downside would be a 2 copy penalty
for passing a strided (or indirect) object to the function, but it
would then handle the problem above, right?  The programmer should use
contiguous arrays to avoid the inefficiency.

Please correct any misunderstanding.

>
> ? cdef void myfunc(np.ndarray[int, mode="c"] buf)
>
> and work simultaneously with NumPy functions/operators and buffer access,
> and that works nice in itself with only #177, but with the addition of
> copy-in/copy-out one then has a problem.
>
> All of this makes me think about pushing the "new buffer syntax" a bit
> harder and get it started on in your GSoC. With that,
>
> ? cdef void myfunc(int[:] buf)
>
> could easily give non-surprising effects for #177 and copy-in/copy-out, as
> the Python object is not "part of the deal".

Let me get this right:  the "int[:] buf" is syntactic sugar for a
Py_buffer, whereas "object[int] buf" represents a Python object that
conforms to the buffer protocol.  Access to the "int[:] buf" would be
always done at C level (perhaps a conversion on the indices), whereas
access to the "object[int] buf" is by default done through the Python
API layer, unless the indices are c ints.  So dealing with just the
Py_buffers Cython side avoids much of the mucking around with
PyObjects, etc, and would be much more efficient, since a Py_buffer is
effectively a souped-up C array (N-dimensions, strides, suboffsets,
etc.)  The "int[:] buf" can always be converted into a memoryview
object for use in the Python layer.

So, in the example above with the new syntax:

cdef void myfunc(int[:] buf)

would have the C prototype

void myfunc(Py_buffer *buf) // + contract on contiguous/int

A call within Cython code with a PyObject argument would require an
explicit cast, wouldn't it?

So one would have to do

myfunc(<int[:]>myobject)

Which would generate:

Py_buffer temp_buffer = acquire buffer from myobject
/* everything else the same, without object reference worries */

If the programmer wants to pass in a buffer and get something back, do
in Cython:

cdef int[:] pass_buf = <int[:]>myobject
myfunc(pass_buf)
# convert pass_buf to other object, use its contents, etc.

>
> Your GSoC would then Cython-side consist more or less of
>
> a) #177, with new syntax
> b) A generic mechanism for automatic coercion between buffers of different
> modes. That is:
>
> cdef int[::contiguous] buf # new syntax for mode="c" w/o Python obj?
> buf = some_object
>
> Here, if the buffer of some_object is not contiguous, a contiguous copy
> will be made! And when releasing buf, it would be copied back.
>
> This would make the parts necesarry for Fortran support tremendously
> useful elsewhere for what I believe will not be much extra effort.
>
> (Though I could help out with the parts of those not needed for Fortran
> support in order to not derail your project.)

I'll need to digest this a bit, but I like it.

Kurt

From dalcinl at gmail.com  Sun Apr 26 05:47:34 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sun, 26 Apr 2009 00:47:34 -0300
Subject: [Cython] Cython & python3: str/bytes
In-Reply-To: <49F3BA64.6050407@gmx.ch>
References: <49F3BA64.6050407@gmx.ch>
Message-ID: <e7ba66e40904252047i319c251cxd3c17eebf94f6d1a@mail.gmail.com>

use this on your Cython code:

arg0 = arg0.encode(u"ascii")

Hopefully, this will work on both under a Py2 and Py3 runtime.


On Sat, Apr 25, 2009 at 10:35 PM, common at gmx.ch <common at gmx.ch> wrote:
> Hi,
>
> having issues with cython/python3 I found the thread
> http://www.mail-archive.com/cython-dev at codespeak.net/msg02346.html
>
> Actually, I have the same problem:
> As *all* strings in cython code get converted to bytes, nothing works out.
>
> I my case I have a c function like:
>
> void foo(char *)
>
> cdef c_foo "foo"(char *)
>
>
> cdef class A:
> ? def __init__(self,arg0):
> ? ? c_foo(arg0)
>
>
> calling from python with
> x = A('test')
>
> in python the argument 'test' is of type str, cython expects the
> argument for c_foo to be of type bytes as char * is mapped to bytes.
>
> Now, as I do not want to change all calls to all functions which take
> str arguments, I tried to convert 'test' from str to bytes.
>
> arg0 = arg0.encode("ascii")
>
> Result:
> TypeError: encode() argument 1 must be string, not bytes.
>
> Even worse:
> if isinstance(arg0, str):
>
> actually checks if arg0 is of type bytes ...
>
> Thats really bad, you can neither create nor recognize or convert
> python3 str's in cython.
>
>
> Maybe mapping "char *" to PyString_Type as a default for all internal
> strings, and "unsigned char *" to PyBytes_Type?
>
> Any other suggestions?
>
>
> MfG
> Markus
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sun Apr 26 07:30:22 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 07:30:22 +0200 (CEST)
Subject: [Cython] Fortran support design issues
In-Reply-To: <c266a7930904251908w76559b51lf4bd9ae4e9e34f57@mail.gmail.com>
References: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>
	<c266a7930904251908w76559b51lf4bd9ae4e9e34f57@mail.gmail.com>
Message-ID: <1eda0ddb60d232e8f855f154db7be1e8.squirrel@webmail.uio.no>

Kurt Smit wrote:
> Clarifying question:  In #177, it says,
>
> "Currently there's no way of creating efficient functions taking a
> buffer argument -- for starters cdef functions doesn't support
> declaring an argument as a buffer, and even if it did, it would mean a
> costly (in this context) buffer acquisition on every call."
>
> more precisely, should it say "There's no way of creating efficient
> functions taking a buffer argument /using the buffer syntax/..." since
> one could always just have the function take a Py_buffer(*) argument,
> right?

Sure :-) But if you go that route, you need to access it by manually
multiplying strides and calculating offsets etc.; and when I say "buffer
argument" I refer to "special Cython buffers" :-) Py_buffer is just a
normal struct seen from Cython's perspective.

> On Fri, Apr 24, 2009 at 3:38 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> So, you wanted to target contiguous buffer passing first. And I was
>> saying that fixing
>>
>> http://trac.cython.org/cython_trac/ticket/177
>>
>> would for the most part amount to what we need. Adding support for
>>
>> ? cdef void myfunc(object[int, mode="c"] buf)
>>
>> which would have the C signature
>>
>> ? void myfunc(Py_buffer* buf) // + a contract on int/contiguous
>>
>> and have the caller be responsible for getting the buffer, gets us far.
>> It would mean that you could do
>>
>> ? myfunc(myobject)
>>
>> and it would turn into
>>
>> ? Py_buffer buf = acquire buffer from myobject
>> ? raise exception if buf is not contiguous
>> ? myfunc(&buf)
>> ? release buf
>
> Just stating my knowledge as it is right now -- for
>
> def func2(object[int, mode='c'] buf):
>     # body here
>
> The 'buf' is passed in as a python object and a stack Py_buffer is
> generated inside func2's body that acquires the buffer from buf.  So
> for 'def' functions the buffer is acquired/released within the
> function scope, while for 'cdef' functions it is acquired/released
> outside.  I'd imagine that cpdef functions would be outside, too.

You are right about def. There is not support for cdef or cpdef currently,
but what you say for cdef would be #177. cpdef would need to be
in-between: It generates a cdef function would would have the #177
behaviour, and a def wrapper which would acquire the buffer.

This is exactly the same as for e.g. "(c/cp)def func(int a)" currently BTW:
 - With def, conversion from Python object happen within function
 - With cdef, the caller has to convert prior to calling (or, it could
already have an int)
 - With cpdef, both of the above, as a wrapper def is created

>> I first thought that adding one component more would get us all the way:
>> Automatic copying into contiguous buffers. So assuming #177 is
>> implemented, one would then move on to having it instead turn into:
>>
>> ? Py_buffer buf = acquire buffer from myobject
>> ? if (buf is contiguous) {
>> ? ? myfunc(&buf);
>> ? } else {
>> ? ? Py_buffer buf2 = make contiguous copy of buf
>> ? ? myfunc(&buf2)
>> ? }
>> ? release buf
>>
>> However, there's a (big) problem here: What Python object does buf2
>> refer to?
>>
>> Using the one of "buf" would be too confusing as they point to different
>> memory areas. One solution is just setting it to None, perhaps. However
>> once #177 is solved one will expect to be able to do
>
> How hard would it be to have a contiguous -> strided (or indirect)
> buffer copy utility function in Cython, and this copy be triggered
> after the myfunc(&buf2) call?  The downside would be a 2 copy penalty
> for passing a strided (or indirect) object to the function, but it
> would then handle the problem above, right?  The programmer should use
> contiguous arrays to avoid the inefficiency.

This was just a mistake on my part in my psuedo-code, I should have copied
out as well. (And we likely need "in/inout/out" specifications for speed
to optionally drop one copy, but let us by all means deal with that
later.)

But the problem with there not being a Python object to fill in for buf2's
Py_buffer remains. The buffer protocol doesn't define any way of creating
a new buffer of the same Python object type -- i.e. we couldn't create a a
new NumPy ndarray in the right way without adding an additional protocol.
Passing buf's Python object on buf2's behalf would just be wrong as then
accessing the buffer through e.g. slices (which goes through Python layer,
at least currently) would access a different memory area than through item
indexing...

>> All of this makes me think about pushing the "new buffer syntax" a bit
>> harder and get it started on in your GSoC. With that,
>>
>> ? cdef void myfunc(int[:] buf)
>>
>> could easily give non-surprising effects for #177 and copy-in/copy-out,
>> as
>> the Python object is not "part of the deal".
>
> Let me get this right:  the "int[:] buf" is syntactic sugar for a
> Py_buffer, whereas "object[int] buf" represents a Python object that
> conforms to the buffer protocol.  Access to the "int[:] buf" would be
> always done at C level (perhaps a conversion on the indices), whereas
> access to the "object[int] buf" is by default done through the Python
> API layer, unless the indices are c ints.  So dealing with just the

Yes! Glad you got it, the description was a bit short.

> Py_buffers Cython side avoids much of the mucking around with
> PyObjects, etc, and would be much more efficient, since a Py_buffer is
> effectively a souped-up C array (N-dimensions, strides, suboffsets,
> etc.)  The "int[:] buf" can always be converted into a memoryview
> object for use in the Python layer.
>
> So, in the example above with the new syntax:
>
> cdef void myfunc(int[:] buf)
>
> would have the C prototype
>
> void myfunc(Py_buffer *buf) // + contract on contiguous/int

This is a digression:

Actually I was planning on having int[:] mean strided, and perhaps
int[::1] or something like that mean contiguous. One could use the third
field for any kind of stride configuration:

 int[:,::1] - C-contiguous
 int[::1,:] - Fortran-contiguous
 int[:,:] - Strided

These would be magic short-hands for more explicit specifications. Some more:
 int[::strided, ::indirect] - Matrix stored as pointers to strided columns
 int[::full, ::1] - First index could use any scheme (if-test required to
see if suboffset is -1 or not), while second index is always contiguous.

It just means the syntax is extensible, we would start with only
supporting contiguous.

> A call within Cython code with a PyObject argument would require an
> explicit cast, wouldn't it?
>
> So one would have to do
>
> myfunc(<int[:]>myobject)

I think the conversion could be automatic -- on the grounds that it
currently is if "def myfunc(int foo)". But it is what happens under the
hood anyway.

> Which would generate:
>
> Py_buffer temp_buffer = acquire buffer from myobject
> /* everything else the same, without object reference worries */
>
> If the programmer wants to pass in a buffer and get something back, do
> in Cython:
>
> cdef int[:] pass_buf = <int[:]>myobject
> myfunc(pass_buf)
> # convert pass_buf to other object, use its contents, etc.

Ah OK now I see where you are heading with the explicit cast. No, I didn't
think of it like this. Py_buffer is just a reference to existing memory --
we can "fake" that memory being contiguous by copying for a moment, but
that shouldn't change semantics -- an "int[:]" variable is a reference,
not allocated memory.

This is very different from Fortran, but matches the way Python works
where everything is a reference.

I.e. you could also do

myfunc(myobject)
cdef int[:] buf = myobject

and buf would get the output of myfunc. And:

cdef int[:] a = ...
cdef int[:] b = a
b[3] = 2 # also shows up in a[3]

and so on.

Note that in general, we should support all cases for cdef functions:

cdef func(int[::full] buf) # or something like that

would never require copying as it would support all 1D buffers. But if you do

cdef func(int[::1] buf)

then the buffer is forced to be contiguous, if necesarry by copying in and
out.


>> Your GSoC would then Cython-side consist more or less of
>>
>> a) #177, with new syntax
>> b) A generic mechanism for automatic coercion between buffers of
>> different
>> modes. That is:
>>
>> cdef int[::contiguous] buf # new syntax for mode="c" w/o Python obj?
>> buf = some_object
>>
>> Here, if the buffer of some_object is not contiguous, a contiguous copy
>> will be made! And when releasing buf, it would be copied back.
>>
>> This would make the parts necesarry for Fortran support tremendously
>> useful elsewhere for what I believe will not be much extra effort.
>>
>> (Though I could help out with the parts of those not needed for Fortran
>> support in order to not derail your project.)
>
> I'll need to digest this a bit, but I like it.

Sure. Note that I corrected my b) point in a seperate mail.

Dag Sverre


From stefan_ml at behnel.de  Sun Apr 26 07:52:32 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 07:52:32 +0200
Subject: [Cython] Cython & python3: str/bytes
In-Reply-To: <49F3BA64.6050407@gmx.ch>
References: <49F3BA64.6050407@gmx.ch>
Message-ID: <49F3F6A0.6030007@behnel.de>

Hi,

common at gmx.ch wrote:
> As *all* strings in cython code get converted to bytes, nothing works out.

Actuall, Python byte strings and unicode strings are equally well supported
in Cython, for both Py2 and Py3. Cython just uses (mainly) Py2.x syntax.


> I my case I have a c function like:
> 
> void foo(char *)
> 
> cdef c_foo "foo"(char *)
> 
> 
> cdef class A:
>    def __init__(self,arg0):
>      c_foo(arg0)

Unless you are dealing with binary data, you should not do this.


> calling from python with
> x = A('test')
> 
> in python the argument 'test' is of type str, cython expects the
> argument for c_foo to be of type bytes as char * is mapped to bytes.
> 
> Now, as I do not want to change all calls to all functions which take
> str arguments, I tried to convert 'test' from str to bytes.
> 
> arg0 = arg0.encode("ascii")
> 
> Result:
> TypeError: encode() argument 1 must be string, not bytes.

This is because you passed a bytes object into the encode() method, which
requires a text value as parameter. Try

	arg0 = arg0.encode(u"ascii")

This will work in both Py2 and Py3.


> Even worse:
> if isinstance(arg0, str):
> 
> actually checks if arg0 is of type bytes ...

Yes, because Cython uses the builtin names "str" and "unicode" of Python
2.x. They will be mapped to the Py3 'equivalents' as expected, though, so
if you test for the "unicode" type, this will continue to work in Py3.

Stefan


From stefan_ml at behnel.de  Sun Apr 26 08:12:59 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 08:12:59 +0200
Subject: [Cython] PATCH: Serious temp unpacking bug
In-Reply-To: <49F35EDC.7000503@student.matnat.uio.no>
References: <49F35EDC.7000503@student.matnat.uio.no>
Message-ID: <49F3FB6B.4000903@behnel.de>


Dag Sverre Seljebotn wrote:
> As the web servers appears to be down at the moment, I'm posting it here
> for the moment.
> 
> This is the bug reported recently. It's related to a problem in
> allocate_temps of SequenceNode (basically two nodes could get the same
> temp), and has likely been there for a long time.
> 
> It hits some nested tuple unpacking cases depending on the number of
> variables in the nested vs. outer tuple.
> 
> The refnanny catches a segfault on trivial code like this:
> 
> def g():
>     return ((3, 2), 1, 0)
> 
> def func2():
>     (a, b), c, d = g()
> 
> 
> Of course, the whole function which was buggy has disappeared in
> cython-unstable. So this is kind of depressing :-) But at least now
> 0.11.2 isn't blocked by having to merge -unstable.

http://trac.cython.org/cython_trac/ticket/298

Closed as "fixed" in 0.11.2.

Stefan

From lrhazi at gmail.com  Sun Apr 26 08:21:01 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Sun, 26 Apr 2009 02:21:01 -0400
Subject: [Cython] Callback functions for C lib in Python
In-Reply-To: <F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>
References: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>
	<F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>
Message-ID: <c4ab18df0904252321j55cfcb64x5cf163d034fbc813@mail.gmail.com>

Thanks Robert.

In the example, the external C function is declared like this:

void find_cheeses(cheesefunc user_func, void *user_data)

and user_data is used to pass the pointer, to the Python function, to
the lib, and the lib sends it back to the callback.

What if the external lib function expects just one value, and that is
the pointer to a callback?

Thanks a lot,
Mohamed.


On Sat, Apr 25, 2009 at 10:02 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 25, 2009, at 6:25 PM, Mohamed Lrhazi wrote:
>
>> Hello,
>>
>> I am trying to write an extension module, to make a C library
>> available for use by the Python user.
>>
>> The lib needs to be passed pointers to callback functions. how do I
>> code this in Cython, so the actual functions are provided by the
>> Python code?
>>
>> Any code examples I could look at for inspiration?
>
> Yes, there's an example of callbacks in the Demos folder of Cython.
>
> - Robert
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
" Logic merely sanctions the conquests of the intuition."
Jacques Hadamard

From stefan_ml at behnel.de  Sun Apr 26 08:29:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 08:29:30 +0200
Subject: [Cython] Callback functions for C lib in Python
In-Reply-To: <c4ab18df0904252321j55cfcb64x5cf163d034fbc813@mail.gmail.com>
References: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>	<F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>
	<c4ab18df0904252321j55cfcb64x5cf163d034fbc813@mail.gmail.com>
Message-ID: <49F3FF4A.4020306@behnel.de>

Hi,

please don't top-post.

Mohamed Lrhazi wrote:
> In the example, the external C function is declared like this:
> 
> void find_cheeses(cheesefunc user_func, void *user_data)
> 
> and user_data is used to pass the pointer, to the Python function, to
> the lib, and the lib sends it back to the callback.
> 
> What if the external lib function expects just one value, and that is
> the pointer to a callback?

In that case, pass the pointer to the callback.

I imagine that your question implies: "how do I pass some state into the
callback function?". That depends on the library you are using, maybe it
passes something into the callback function that you can control from your
outside code.

Stefan

From lrhazi at gmail.com  Sun Apr 26 08:34:15 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Sun, 26 Apr 2009 02:34:15 -0400
Subject: [Cython] Callback functions for C lib in Python
In-Reply-To: <49F3FF4A.4020306@behnel.de>
References: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>
	<F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>
	<c4ab18df0904252321j55cfcb64x5cf163d034fbc813@mail.gmail.com>
	<49F3FF4A.4020306@behnel.de>
Message-ID: <c4ab18df0904252334l791638d2wd1709d16625c67bc@mail.gmail.com>

On Sun, Apr 26, 2009 at 2:29 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> please don't top-post.
>

Sorry, and thanks for the hints.

From dagss at student.matnat.uio.no  Sun Apr 26 09:19:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 09:19:40 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
Message-ID: <49F40B0C.2050707@student.matnat.uio.no>

Robert Bradshaw wrote:
> I start with a disclaimer I should have made several posts back: I  
> have done very little programming with C++, so please excuse any  
> naivet? I have in this regard...

Sure, but I must also expose it :-)

So, to sum up the two solutions so far:

A) Introduce the C++ operators to Cython. This will require adding C++ 
semantics to the Cython language in other areas as well.

B) Define C++ snippets of code which correspond to Cython 
operators/semantics. This will only support a (natural) subset of what 
is possible to do in C++; say, as if the C++ library had been wrapped 
for the Python runtime.

We seem to agree that A should use C++ syntax (that's really a 
tangential issue but is a nice peg to remember things by).

I argue below that A must have effects far beyond simple operator 
overloading.

> On Apr 25, 2009, at 9:54 AM, Dag Sverre Seljebotn wrote:
>> How would you represent my latter example in my previous mail at  
>> using C++
>> syntax in Cython?
> 
> I would expose your ItemAccessor class as well as your MyCollection  
> class in the Cython declaration. This may necessitate exposing the  
> notion of a C++ reference to Cython as well. At the very least, I  
> think trying to overload on the return type is bad.

(Note that C++ overloading on return type in the specific case of the 
conversion operator, "operator SomeType". I.e. you can have "int a = x; 
float b = x" invoke two different methods on an x object to do the 
conversion.)

Exposing ItemAccessor as is, and support its use, will have a dramatic 
effect on the Cython language, as operator= is what is really used here.

I've attached some C++ code to demonstrate the full example. Note that 
it is possible to do:

if (1 == 1) {
   ItemAccessor acc = some_collection[2];
   acc = 3.2;
   cout << some_collection[2] << endl; // outputs 3.2
}

Now, translate this to Cython to see how bad this looks:

if 1 == 1:
     cdef ItemAccessor acc = some_collection[2] # C++ var scoping?

     acc = 3.2
     # assignment triggers operator= but leaves
     # existing object in place?

     print some_collection[2]
# upon exiting if-tests scope, the
# destructor of ItemAccessor is run which could do anything..


Summary:
  - C++ does *only* have "getitem", not "setitem".
  - However, we return an object with overriden operator= and "operator 
double" in order to simulate setitem.
  - operator[] is not what is at stake here, one could also have a 
method return an ItemAccessor:

some_collection.get(2) = 3.2;

This is just one example -- that doesn't mean that it's a corner case, 
it means that there are a hundred other cases which are equally 
different to represent sanely with Cython semantics. C++ code is just 
pieced together in a different way, where many C++ objects and operators 
can make up one statement in Python.

>> This is what I think conflicts with letting the operator[] notation  
>> into
>> Cython. operator[] means nothing of the sort of what we are really
>> supporting.
> 
> Hmm... OK, maybe I'll take back that statement and say Cython does  
> need to go deeper than this. Right now one has to learn the "magic"  
> of giving Cython declarations to generate the right code, which is  
> undesirable. I would rather not make users have to learn a new  
> "magic" but really state what's going on on the C++ side.
> 
> The only two non-simple lvalues we need to worry about are indexing  
> and attribute access, right?

You mean Cython-side? Yes. It's all we can support with the current 
Pythonic syntax.

C++ can do "anything", like

some_collection.get(2) = 3.2;

or whatever, just overload "operator=" on the lhs return value.

The real issue here is with C++ allowing objects as stack variables with 
copy constructors and operator= etc. etc.; things I personally don't 
want to have in the Cython language. And then some kind of "wrapping" 
where the semantics are not 1:1 is needed.

There are lots of other cases where Cython cannot easily express C++ 
code without modification:
  - ++ is not the same as += 1, e.g. linked list types would support 
only the first on iterators
  - a->b and (*a).b could have different values and types, just overload 
operator-> differently from operator*.

All of these can be relatively easily handled if we have a philosophy 
where we make it easy to wrap "sane" code but don't support the whole 
C++ set, which seem to imply (B).

-- 
Dag Sverre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.cpp
Type: text/x-c++src
Size: 1275 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090426/d0367279/attachment.cpp 

From dagss at student.matnat.uio.no  Sun Apr 26 09:53:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 09:53:00 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F40B0C.2050707@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
Message-ID: <49F412DC.60200@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> All of these can be relatively easily handled if we have a philosophy 
> where we make it easy to wrap "sane" code but don't support the whole 
> C++ set, which seem to imply (B).

To be concrete, I think it is better to allow

cdef extern "C++":
   class MyClass:
     double get(int key)
     void set(int key, double value) "(*this)(key) = value;"

where one gets help for conveniently defining a new "set" method in 
Cython (which would in this case not exist in C++), and where one has to 
write some C++ template (macro) code in a string, than to extend Cython 
to allow

obj.get(3) = 2

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun Apr 26 09:56:13 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 09:56:13 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F412DC.60200@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>
	<49F412DC.60200@student.matnat.uio.no>
Message-ID: <49F4139D.1080205@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> All of these can be relatively easily handled if we have a philosophy 
>> where we make it easy to wrap "sane" code but don't support the whole 
>> C++ set, which seem to imply (B).
> 
> To be concrete, I think it is better to allow
> 
> cdef extern "C++":
>    class MyClass:
>      double get(int key)
>      void set(int key, double value) "(*this)(key) = value;"

Should be "(*this).get(key) = value;".

And we would then predefine e.g. __setitem__ as "(*this)[key] = value;".

-- 
Dag Sverre

From robertwb at math.washington.edu  Sun Apr 26 12:38:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 26 Apr 2009 03:38:57 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F40B0C.2050707@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
Message-ID: <A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>

On Apr 26, 2009, at 12:19 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> I start with a disclaimer I should have made several posts back:  
>> I  have done very little programming with C++, so please excuse  
>> any  naivet? I have in this regard...
>
> Sure, but I must also expose it :-)
>
> So, to sum up the two solutions so far:
>
> A) Introduce the C++ operators to Cython. This will require adding C 
> ++ semantics to the Cython language in other areas as well.
>
> B) Define C++ snippets of code which correspond to Cython operators/ 
> semantics. This will only support a (natural) subset of what is  
> possible to do in C++; say, as if the C++ library had been wrapped  
> for the Python runtime.
>
> We seem to agree that A should use C++ syntax (that's really a  
> tangential issue but is a nice peg to remember things by).

That's a fair summary. I am also for supporting only a subset of what  
is possible--if one really needs crazy stuff like differently  
implemented prefix/postfix decrements and -> vs (*). one can code in C 
++ or use clever macros. (Would we need to expose ++, a.k.a  
"__next__" for STL iterators?)

> I argue below that A must have effects far beyond simple operator  
> overloading.
>
>> On Apr 25, 2009, at 9:54 AM, Dag Sverre Seljebotn wrote:
>>> How would you represent my latter example in my previous mail at   
>>> using C++
>>> syntax in Cython?
>> I would expose your ItemAccessor class as well as your  
>> MyCollection  class in the Cython declaration. This may  
>> necessitate exposing the  notion of a C++ reference to Cython as  
>> well. At the very least, I  think trying to overload on the return  
>> type is bad.
>
> (Note that C++ overloading on return type in the specific case of  
> the conversion operator, "operator SomeType". I.e. you can have  
> "int a = x; float b = x" invoke two different methods on an x  
> object to do the conversion.)
>
> Exposing ItemAccessor as is, and support its use, will have a  
> dramatic effect on the Cython language, as operator= is what is  
> really used here.

I think we're going to want to support the operator= in a limited  
sense that one is allowed to do conversions from one type to another  
(the declaration could be totally different, and potentially the same/ 
similar as whatever we settle on for "operator T()"). So rather than  
the hypothetical

class MyCollection:
      A __getitem__(K idx)
      B __getitem__(K idx) # note: overloads on return value...

I would prefer

class MyCollection:
      X __getitem__(K idx)

with X being typed to convert either to A or B.

- Robert


From greg.ewing at canterbury.ac.nz  Sun Apr 26 13:24:01 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 26 Apr 2009 23:24:01 +1200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F40B0C.2050707@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<49F16BC1.5050708@student.matnat.uio.no>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
Message-ID: <49F44451.4020901@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> (Note that C++ overloading on return type in the specific case of the 
> conversion operator, "operator SomeType". I.e. you can have "int a = x; 
> float b = x" invoke two different methods on an x object to do the 
> conversion.)

But that's not overloading on return type -- it's not
even overloading at all, since they're different
methods.

Overloading on return type would be if

   int a = x.foo()

and

   float b = x.foo()

were to select different versions of the foo() method
based on their return types. I don't think C++ does
that (although I might be wrong).

> Summary:
>  - C++ does *only* have "getitem", not "setitem".

It's more accurate to say that it has neither "getitem"
nor "setitem", but rather a single "get reference to item",
the result of which can be used as either an lvalue or
an rvalue (or at least it's possible to define an operator[]
that works that way).

All of these complications are part of the reason I've
been very cautious about adding C++ features to Pyrex
so far. Trying to sip from that firehose could have
enormous consequences.

-- 
Greg

From common at gmx.ch  Sun Apr 26 13:46:49 2009
From: common at gmx.ch (common at gmx.ch)
Date: Sun, 26 Apr 2009 13:46:49 +0200
Subject: [Cython] Cython & python3: str/bytes
In-Reply-To: <49F3F6A0.6030007@behnel.de>
References: <49F3BA64.6050407@gmx.ch> <49F3F6A0.6030007@behnel.de>
Message-ID: <49F449A9.8010509@gmx.ch>

Hi,

thanks for assistance.

Stefan Behnel wrote:
> common at gmx.ch wrote:
>> As *all* strings in cython code get converted to bytes, nothing works out.
> 
> Actuall, Python byte strings and unicode strings are equally well supported
> in Cython, for both Py2 and Py3. Cython just uses (mainly) Py2.x syntax.
> 
> 
>> I my case I have a c function like:
>>
>> void foo(char *)
>>
>> cdef c_foo "foo"(char *)
>>
>>
>> cdef class A:
>>    def __init__(self,arg0):
>>      c_foo(arg0)
> 
> Unless you are dealing with binary data, you should not do this.

foo(char *) is expected to get an zero terminated string.

What exactly, and whats is the correct way of doing it?


MfG
Markus

From dagss at student.matnat.uio.no  Sun Apr 26 15:09:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 15:09:29 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
Message-ID: <49F45D09.6040202@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 26, 2009, at 12:19 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> I start with a disclaimer I should have made several posts back:  
>>> I  have done very little programming with C++, so please excuse  
>>> any  naivet? I have in this regard...
>> Sure, but I must also expose it :-)
>>
>> So, to sum up the two solutions so far:
>>
>> A) Introduce the C++ operators to Cython. This will require adding C 
>> ++ semantics to the Cython language in other areas as well.
>>
>> B) Define C++ snippets of code which correspond to Cython operators/ 
>> semantics. This will only support a (natural) subset of what is  
>> possible to do in C++; say, as if the C++ library had been wrapped  
>> for the Python runtime.
>>
>> We seem to agree that A should use C++ syntax (that's really a  
>> tangential issue but is a nice peg to remember things by).
> 
> That's a fair summary. I am also for supporting only a subset of what  
> is possible--if one really needs crazy stuff like differently  
> implemented prefix/postfix decrements and -> vs (*). one can code in C 
> ++ or use clever macros. (Would we need to expose ++, a.k.a  
> "__next__" for STL iterators?)

Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any 
form of +=, but we can just compile += 1 and -= 1 to those operators in 
general (also for plain ints for that matter), so it's not really a 
problem. (We are then requiring that += 1 would have the same semantics, 
which I'm 100% fine with.) This holds for both my and your stance BTW 
and is orthogonal.

As for crazy .. well, I'd argue that "polygon.vertices(2) = Vertex(3,4)" 
is not outside the realm of what one could sanely do in C++ -- I'd go 
for [] myself, but that's not really up to me. Yet this is out of reach 
for current Cython semantics.

>> Exposing ItemAccessor as is, and support its use, will have a  
>> dramatic effect on the Cython language, as operator= is what is  
>> really used here.
> 
> I think we're going to want to support the operator= in a limited  
> sense that one is allowed to do conversions from one type to another  
> (the declaration could be totally different, and potentially the same/ 
> similar as whatever we settle on for "operator T()"). So rather than  
> the hypothetical

a) I don't see where operator= enters the picture here.
b) Even if you could do this, this is only hot-fixing one very specific 
consequence of a much larger body of problems. I have a feeling you 
could go around forever fixing specific cases with native Cython support 
and get something massively complicated (and impossible to guess) 
compared to my proposal for embedded C++ macros.

I'll believe this will work when I see a more concrete proposal.

(But at the moment, let "func(2) = 3" serve as my main worry rather than 
the overloading by return type -- I'll agree that that is a problem.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun Apr 26 15:14:43 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 15:14:43 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F44451.4020901@canterbury.ac.nz>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>
	<49F44451.4020901@canterbury.ac.nz>
Message-ID: <49F45E43.8080703@student.matnat.uio.no>

Greg Ewing wrote:
> Dag Sverre Seljebotn wrote:
> 
>> (Note that C++ overloading on return type in the specific case of the 
>> conversion operator, "operator SomeType". I.e. you can have "int a = x; 
>> float b = x" invoke two different methods on an x object to do the 
>> conversion.)
> 
> But that's not overloading on return type -- it's not
> even overloading at all, since they're different
> methods.
> 
> Overloading on return type would be if
> 
>    int a = x.foo()
> 
> and
> 
>    float b = x.foo()
> 
> were to select different versions of the foo() method
> based on their return types. I don't think C++ does
> that (although I might be wrong).

I suppose you are right. Although the *effect* can be simulated by 
having foo() simply return an object with conversion operators which 
call two different methods:

class Result {
public:
   operator float() { return 2; }
   operator int() { return 3; }
}
Result foo() { return Result(); }

int a = foo() // a == 3
float b = foo() // b == 4
foo() // nothing can happen, no side-effects possible

So the reason I end up with overloading on result type in Cython is 
because then you can declare the interface while ignoring the Result 
class and have things work the right way.

But it was just a throw-away idea, like I said to Robert, I'm much more 
worried about

foo() = 3;

at the moment. And, if you decide to declare Result Cython-side, I'm 
worried it might force support of stuff like

if True:
   cdef Result result = foo() # inside block!
   # destructor of result called here

and so on.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun Apr 26 15:46:34 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 15:46:34 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F45D09.6040202@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
Message-ID: <49F465BA.8070503@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> I think we're going to want to support the operator= in a limited  
>> sense that one is allowed to do conversions from one type to another  
>> (the declaration could be totally different, and potentially the same/ 
>> similar as whatever we settle on for "operator T()"). So rather than  
>> the hypothetical
> 
> a) I don't see where operator= enters the picture here.
> b) Even if you could do this, this is only hot-fixing one very specific 
> consequence of a much larger body of problems. I have a feeling you 
> could go around forever fixing specific cases with native Cython support 
> and get something massively complicated (and impossible to guess) 
> compared to my proposal for embedded C++ macros.

This was rather unfair of me. You said:

"""
I would prefer

class MyCollection:
       X __getitem__(K idx)
"""

Which is not by any means a hotfix, as it is just mapping over the C++ 
declaration!

So I'll rephrase: Until I see it, I remain curious about how one is 
going to select a subset of C++ semantics to support which
  a) maps to Cython semantics without too many wierd, exceptional rules 
which one has no chance of guessing (which then defeats the purpose of 
pretending to support operatorX-notation in the first place)
  b) allows wrapping of most C++ libraries without writing a wrapper 
C++-side which is more Cython-subset-friendly.

(Problem with wrapper C++ side: I think one would have to subclass 
classes C++-side, which would leave descendants in other libraries in 
the cold. Though multiple inheritance *could* save us here.)

-- 
Dag Sverre

From stefan_ml at behnel.de  Sun Apr 26 16:33:06 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 16:33:06 +0200
Subject: [Cython] Cython & python3: str/bytes
In-Reply-To: <49F449A9.8010509@gmx.ch>
References: <49F3BA64.6050407@gmx.ch> <49F3F6A0.6030007@behnel.de>
	<49F449A9.8010509@gmx.ch>
Message-ID: <49F470A2.10003@behnel.de>


common at gmx.ch wrote:
> Stefan Behnel wrote:
>> common at gmx.ch wrote:
>>> I my case I have a c function like:
>>>
>>> void foo(char *)
>>>
>>> cdef c_foo "foo"(char *)
>>>
>>>
>>> cdef class A:
>>>    def __init__(self,arg0):
>>>      c_foo(arg0)
>> Unless you are dealing with binary data, you should not do this.
> 
> foo(char *) is expected to get an zero terminated string.
> 
> What exactly

This:

    def __init__(self,arg0):
        c_foo(arg0)

because it leaks implementation details into your Python-level API. If your
users are required to pass byte strings (e.g. binary data), this is
perfectly fine. But if the idea is to pass some kind of textual data (i.e.
unicode strings), then you have to do some kind of conversion, usually by
encoding the string to a byte sequence. The specific encoding depends on
what your C function can handle and should be hidden from user code.

It's ok to strictly require unicode input here in Py3, where unicode
strings and byte strings are distinct types. You may or may not want to
find a suitable way to deal with byte string input in Py2, but that is
application specific. A helper function that does the conversion for any
input value you get from Python space (and raises an appropriate exception
on errors) will likely prove useful here.

I think I'll actually write this up as a FAQ entry.

Stefan

From stefan_ml at behnel.de  Sun Apr 26 17:19:01 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 17:19:01 +0200
Subject: [Cython] Cython & python3: str/bytes
In-Reply-To: <49F470A2.10003@behnel.de>
References: <49F3BA64.6050407@gmx.ch>
	<49F3F6A0.6030007@behnel.de>	<49F449A9.8010509@gmx.ch>
	<49F470A2.10003@behnel.de>
Message-ID: <49F47B65.2080602@behnel.de>


Stefan Behnel wrote:
> I think I'll actually write this up as a FAQ entry.

Does this help?

http://wiki.cython.org/FAQ#HowdoIpassaPythonstringparameterontoaClibrary.3F

Stefan


From lrhazi at gmail.com  Sun Apr 26 17:44:21 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Sun, 26 Apr 2009 11:44:21 -0400
Subject: [Cython] Callback functions for C lib in Python
In-Reply-To: <F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>
References: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>
	<F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>
Message-ID: <c4ab18df0904260844n9a3e45cw8869b5b67ebd7856@mail.gmail.com>

On Sat, Apr 25, 2009 at 10:02 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Yes, there's an example of callbacks in the Demos folder of Cython.
>

I found a lot better than that example, I found a treasure, should be
on Cython's documentation page :)

http://code.google.com/p/pyevent/source/browse/#svn/trunk/pyevent

Thanks for the author,
Mohamed.

From dsurviver at gmail.com  Sun Apr 26 17:55:59 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Sun, 26 Apr 2009 12:55:59 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F45D09.6040202@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
Message-ID: <45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>

>> That's a fair summary. I am also for supporting only a subset of what
>> is possible--if one really needs crazy stuff like differently
>> implemented prefix/postfix decrements and -> vs (*). one can code in C
>> ++ or use clever macros. (Would we need to expose ++, a.k.a
>> "__next__" for STL iterators?)
>
> Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any
> form of +=, but we can just compile += 1 and -= 1 to those operators in
> general (also for plain ints for that matter), so it's not really a
> problem. (We are then requiring that += 1 would have the same semantics,
> which I'm 100% fine with.) This holds for both my and your stance BTW
> and is orthogonal.
>

Could we define a function int() and dec() for that?

like, inc(iterator), dec(iterator).

Would this be possible?
And, of course, better?

From dsurviver at gmail.com  Sun Apr 26 17:59:18 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Sun, 26 Apr 2009 12:59:18 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F45E43.8080703@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<49F44451.4020901@canterbury.ac.nz>
	<49F45E43.8080703@student.matnat.uio.no>
Message-ID: <45239150904260859y7d3c02d0q8dba33dfcf94ac69@mail.gmail.com>

2009/4/26 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
> Greg Ewing wrote:
>> Dag Sverre Seljebotn wrote:
>>
>>> (Note that C++ overloading on return type in the specific case of the
>>> conversion operator, "operator SomeType". I.e. you can have "int a = x;
>>> float b = x" invoke two different methods on an x object to do the
>>> conversion.)
>>
>> But that's not overloading on return type -- it's not
>> even overloading at all, since they're different
>> methods.
>>
>> Overloading on return type would be if
>>
>> ? ?int a = x.foo()
>>
>> and
>>
>> ? ?float b = x.foo()
>>
>> were to select different versions of the foo() method
>> based on their return types. I don't think C++ does
>> that (although I might be wrong).
>
> I suppose you are right. Although the *effect* can be simulated by
> having foo() simply return an object with conversion operators which
> call two different methods:
>
> class Result {
> public:
> ? operator float() { return 2; }
> ? operator int() { return 3; }
> }
> Result foo() { return Result(); }
>
> int a = foo() // a == 3
> float b = foo() // b == 4
> foo() // nothing can happen, no side-effects possible
>
> So the reason I end up with overloading on result type in Cython is
> because then you can declare the interface while ignoring the Result
> class and have things work the right way.
>
> But it was just a throw-away idea, like I said to Robert, I'm much more
> worried about
>
> foo() = 3;

If no type defined, maybe an overloaded function with PyObject work.
Or using type() function.
So, the parser would know the overloaded method we want him to use.

From dagss at student.matnat.uio.no  Sun Apr 26 18:04:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 18:04:26 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
Message-ID: <49F4860A.9090106@student.matnat.uio.no>

Danilo Freitas wrote:
>>> That's a fair summary. I am also for supporting only a subset of what
>>> is possible--if one really needs crazy stuff like differently
>>> implemented prefix/postfix decrements and -> vs (*). one can code in C
>>> ++ or use clever macros. (Would we need to expose ++, a.k.a
>>> "__next__" for STL iterators?)
>> Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any
>> form of +=, but we can just compile += 1 and -= 1 to those operators in
>> general (also for plain ints for that matter), so it's not really a
>> problem. (We are then requiring that += 1 would have the same semantics,
>> which I'm 100% fine with.) This holds for both my and your stance BTW
>> and is orthogonal.
>>
> 
> Could we define a function int() and dec() for that?
> 
> like, inc(iterator), dec(iterator).

Good idea! +1, if it is done in the "cython" namespace:

from cython import inc, dec

Other "magic" functions are already like in the cython namespace (it's 
not a real module).

Note that for STL iterators we also need *it rather than [0], so add 
"deref" to that list. Or possibly we just add * to Cython?

We could then also move on to define the next builtin for C++ classes, 
which would translate like this:

obj = next(it)

would become

cython.inc(it)
obj = cython.deref(it)

-- 
Dag Sverre

From dsurviver at gmail.com  Sun Apr 26 18:18:03 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Sun, 26 Apr 2009 13:18:03 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4860A.9090106@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
Message-ID: <45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>

>
> Good idea! +1, if it is done in the "cython" namespace:
>
> from cython import inc, dec
>
> Other "magic" functions are already like in the cython namespace (it's
> not a real module).
>
> Note that for STL iterators we also need *it rather than [0], so add
> "deref" to that list. Or possibly we just add * to Cython?
>
> We could then also move on to define the next builtin for C++ classes,
> which would translate like this:
>
> obj = next(it)
>
> would become
>
> cython.inc(it)
> obj = cython.deref(it)
>

As I said before, I prefer a Pythonic syntax for Cython.
So, deref() would be better in this case.
But, for all this and more, we need to define a default syntax.
Mixing syntax is a little weird and harder to use.

If we translate C++ syntax to Pythonic  syntax, I think it will be
easier to use it.

So, if we gonna do it, many other functions will be added to the list.
I have some Ideas. Don't know if they are good. But it sounds like:

my_list = clist(int) #just an example
it = cython.get_begin_iterator(my_list)
while it != cython.get_end_iterator(my_list):
    if something == cython.deref(it): #just a normal if
        break
    cython.inc(it)

So, we could get iterator fom many ways. Just like in STL.

It's just a simple idea I have in mind

From dagss at student.matnat.uio.no  Sun Apr 26 18:53:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 18:53:44 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>	<49F4860A.9090106@student.matnat.uio.no>
	<45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>
Message-ID: <49F49198.4050105@student.matnat.uio.no>

Danilo Freitas wrote:
> As I said before, I prefer a Pythonic syntax for Cython.
> So, deref() would be better in this case.
> But, for all this and more, we need to define a default syntax.
> Mixing syntax is a little weird and harder to use.
> 
> If we translate C++ syntax to Pythonic  syntax, I think it will be
> easier to use it.
> 
> So, if we gonna do it, many other functions will be added to the list.
> I have some Ideas. Don't know if they are good. But it sounds like:
> 
> my_list = clist(int) #just an example
> it = cython.get_begin_iterator(my_list)
> while it != cython.get_end_iterator(my_list):
>     if something == cython.deref(it): #just a normal if
>         break
>     cython.inc(it)
> 
> So, we could get iterator fom many ways. Just like in STL.

Yes. (Though my vote is for my_list.begin() and my_list.end() instead.)

The main issue with using STL iterators in Cython though is what types 
to add.


For comparison, this is how it is /currently/ possible in Cython to do this:

# the proper extern declarations

# use vector in example, then we don't need inc/dec/deref

cdef cpp_vector_int* vec = new_cpp_vector_int()
cdef cpp_vector_int_iterator* it = \
   new_cpp_vector_int_iterator(vec.begin())
cdef cpp_vector_int_iterator* end = \
   new_cpp_vector_int_iterator(vec.end())

while it[0] != end[0]:
   if something = it[0][0]:
     break
   it[0] += 1

cpp_delete(it)
cpp_delete(end)
cpp_delete(vec)


Not pretty. The types are the problem -- if one could simply do

cdef cpp_vector_int_iterator it = vec.begin()

things would look much nicer.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sun Apr 26 19:33:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 19:33:44 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F45D09.6040202@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
Message-ID: <49F49AF8.7070409@behnel.de>


Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> I am also for supporting only a subset of what  
>> is possible--if one really needs crazy stuff like differently  
>> implemented prefix/postfix decrements and -> vs (*). one can code in C 
>> ++ or use clever macros. (Would we need to expose ++, a.k.a  
>> "__next__" for STL iterators?)
> 
> Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any 
> form of +=

Am I missing something here? Why isn't the for-loop syntax enough for C++
iterators?

Stefan

From dsurviver at gmail.com  Sun Apr 26 19:34:29 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Sun, 26 Apr 2009 14:34:29 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F49198.4050105@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>
	<49F49198.4050105@student.matnat.uio.no>
Message-ID: <45239150904261034y1772e9edmdebfc7d94f4fc9ee@mail.gmail.com>

>
> Yes. (Though my vote is for my_list.begin() and my_list.end() instead.)
>
Yes. It may be better so, through we have it similar in C++
(list<T>::iterator it = myList.begin();)

> The main issue with using STL iterators in Cython though is what types
> to add.
>
>
> For comparison, this is how it is /currently/ possible in Cython to do this:
>
> # the proper extern declarations
>
> # use vector in example, then we don't need inc/dec/deref
>
> cdef cpp_vector_int* vec = new_cpp_vector_int()
> cdef cpp_vector_int_iterator* it = \
> ? new_cpp_vector_int_iterator(vec.begin())
> cdef cpp_vector_int_iterator* end = \
> ? new_cpp_vector_int_iterator(vec.end())
>
> while it[0] != end[0]:
> ? if something = it[0][0]:
> ? ? break
> ? it[0] += 1
>
> cpp_delete(it)
> cpp_delete(end)
> cpp_delete(vec)
>
>
> Not pretty. The types are the problem -- if one could simply do
>
> cdef cpp_vector_int_iterator it = vec.begin()
>
> things would look much nicer.
>
Is it all implemented dinamically?
If we find a way to implement it static, or just doing it dinamic, but
automated, we could avoid cpp_delete() (I dont't know ow this works in
C++)

From common at gmx.ch  Sun Apr 26 19:44:58 2009
From: common at gmx.ch (common at gmx.ch)
Date: Sun, 26 Apr 2009 19:44:58 +0200
Subject: [Cython] Cython & python3: str/bytes
In-Reply-To: <49F47B65.2080602@behnel.de>
References: <49F3BA64.6050407@gmx.ch>	<49F3F6A0.6030007@behnel.de>	<49F449A9.8010509@gmx.ch>	<49F470A2.10003@behnel.de>
	<49F47B65.2080602@behnel.de>
Message-ID: <49F49D9A.5050500@gmx.ch>

Stefan Behnel wrote:
> Stefan Behnel wrote:
>> I think I'll actually write this up as a FAQ entry.
> 
> Does this help?
> 
> http://wiki.cython.org/FAQ#HowdoIpassaPythonstringparameterontoaClibrary.3F

Yes, thanks.


Markus

From dagss at student.matnat.uio.no  Sun Apr 26 19:55:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 19:55:42 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F49AF8.7070409@behnel.de>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>
	<49F49AF8.7070409@behnel.de>
Message-ID: <49F4A01E.4040108@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> I am also for supporting only a subset of what  
>>> is possible--if one really needs crazy stuff like differently  
>>> implemented prefix/postfix decrements and -> vs (*). one can code in C 
>>> ++ or use clever macros. (Would we need to expose ++, a.k.a  
>>> "__next__" for STL iterators?)
>> Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any 
>> form of +=
> 
> Am I missing something here? Why isn't the for-loop syntax enough for C++
> iterators?

Who's saying that you always want to iterate through all elements from 
beginning to end in one go?

But yes, one could use the for-loop syntax for that specific case.

(However myself I'm in favor of even getting the most basic things 
working before adding support in higher-level constructs.)

-- 
Dag Sverre

From stefan_ml at behnel.de  Sun Apr 26 20:19:28 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 20:19:28 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4A01E.4040108@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<49F49AF8.7070409@behnel.de>
	<49F4A01E.4040108@student.matnat.uio.no>
Message-ID: <49F4A5B0.7070301@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert Bradshaw wrote:
>>>> I am also for supporting only a subset of what  
>>>> is possible--if one really needs crazy stuff like differently  
>>>> implemented prefix/postfix decrements and -> vs (*). one can code in C 
>>>> ++ or use clever macros. (Would we need to expose ++, a.k.a  
>>>> "__next__" for STL iterators?)
>>> Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any 
>>> form of +=
>> Am I missing something here? Why isn't the for-loop syntax enough for C++
>> iterators?
> 
> Who's saying that you always want to iterate through all elements from 
> beginning to end in one go?

Well, it's the most common case, at least. For the rest, there's reversed()
and islice(), which we could define (and optimise) on C++ iterators. Maybe
just our own version (as we might have to fix up the semantics), such as
"cython.cppitertools.islice()". Although overriding the standard function
might make life a lot easier for users.

I (basically) never used C++, so I may not be aware of all implications,
but I would prefer not reinventing stuff that Python already has, just
because we are dealing with a different 'backend'. The fewer things we need
to add to the language, the more seamlessly the C++ integration will become.


> But yes, one could use the for-loop syntax for that specific case.
> 
> (However myself I'm in favor of even getting the most basic things 
> working before adding support in higher-level constructs.)

But isn't the for-in-loop the most basic thing that you could get to work
for iterators? (at least from a Python POV...)

Stefan

From dagss at student.matnat.uio.no  Sun Apr 26 20:37:20 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 20:37:20 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4A5B0.7070301@behnel.de>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<49F49AF8.7070409@behnel.de>	<49F4A01E.4040108@student.matnat.uio.no>
	<49F4A5B0.7070301@behnel.de>
Message-ID: <49F4A9E0.6090608@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert Bradshaw wrote:
>>>>> I am also for supporting only a subset of what  
>>>>> is possible--if one really needs crazy stuff like differently  
>>>>> implemented prefix/postfix decrements and -> vs (*). one can code in C 
>>>>> ++ or use clever macros. (Would we need to expose ++, a.k.a  
>>>>> "__next__" for STL iterators?)
>>>> Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any 
>>>> form of +=
>>> Am I missing something here? Why isn't the for-loop syntax enough for C++
>>> iterators?
>> Who's saying that you always want to iterate through all elements from 
>> beginning to end in one go?
> 
> Well, it's the most common case, at least. For the rest, there's reversed()
> and islice(), which we could define (and optimise) on C++ iterators. Maybe
> just our own version (as we might have to fix up the semantics), such as
> "cython.cppitertools.islice()". Although overriding the standard function
> might make life a lot easier for users.
> 
> I (basically) never used C++, so I may not be aware of all implications,
> but I would prefer not reinventing stuff that Python already has, just
> because we are dealing with a different 'backend'. The fewer things we need
> to add to the language, the more seamlessly the C++ integration will become.
> 
> 
>> But yes, one could use the for-loop syntax for that specific case.
>>
>> (However myself I'm in favor of even getting the most basic things 
>> working before adding support in higher-level constructs.)
> 
> But isn't the for-in-loop the most basic thing that you could get to work
> for iterators? (at least from a Python POV...)

The solution you scetch sounds to me like adding support for the Python 
"os" module to Cython :-) And we don't do that, we use getattr/etc. 
etc., because then we support all the other modules and usecases as well.

Ahh; the missing piece: Iterators are nothing special in C++. There's 
absolutely *no* concept of an iterator in the language. It's just a set 
of conventions.

So that's why I propose focusing on just getting the very basic 
interfacing of the languages done, and get it all supported instead of 
just STL.

When it comes to ++/--, basic fact is that it can be used for whatever 
purpose in C++. We are, after all, talking about the language which 
overloads "<<" for being "output to stream" -- again, with *no* explicit 
support for streams in the language; just by the usual operator overloading.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun Apr 26 20:42:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 20:42:24 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4A9E0.6090608@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<49F49AF8.7070409@behnel.de>	<49F4A01E.4040108@student.matnat.uio.no>	<49F4A5B0.7070301@behnel.de>
	<49F4A9E0.6090608@student.matnat.uio.no>
Message-ID: <49F4AB10.5060406@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Stefan Behnel wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>> Robert Bradshaw wrote:
>>>>>> I am also for supporting only a subset of what  
>>>>>> is possible--if one really needs crazy stuff like differently  
>>>>>> implemented prefix/postfix decrements and -> vs (*). one can code in C 
>>>>>> ++ or use clever macros. (Would we need to expose ++, a.k.a  
>>>>>> "__next__" for STL iterators?)
>>>>> Yes, you need ++/-- for std::list<T>::iterator, which doesn't accept any 
>>>>> form of +=
>>>> Am I missing something here? Why isn't the for-loop syntax enough for C++
>>>> iterators?
>>> Who's saying that you always want to iterate through all elements from 
>>> beginning to end in one go?
>> Well, it's the most common case, at least. For the rest, there's reversed()
>> and islice(), which we could define (and optimise) on C++ iterators. Maybe
>> just our own version (as we might have to fix up the semantics), such as
>> "cython.cppitertools.islice()". Although overriding the standard function
>> might make life a lot easier for users.
>>
>> I (basically) never used C++, so I may not be aware of all implications,
>> but I would prefer not reinventing stuff that Python already has, just
>> because we are dealing with a different 'backend'. The fewer things we need
>> to add to the language, the more seamlessly the C++ integration will become.
>>
>>
>>> But yes, one could use the for-loop syntax for that specific case.
>>>
>>> (However myself I'm in favor of even getting the most basic things 
>>> working before adding support in higher-level constructs.)
>> But isn't the for-in-loop the most basic thing that you could get to work
>> for iterators? (at least from a Python POV...)
> 
> The solution you scetch sounds to me like adding support for the Python 
> "os" module to Cython :-) And we don't do that, we use getattr/etc. 
> etc., because then we support all the other modules and usecases as well.
> 
> Ahh; the missing piece: Iterators are nothing special in C++. There's 
> absolutely *no* concept of an iterator in the language. It's just a set 
> of conventions.
> 
> So that's why I propose focusing on just getting the very basic 
> interfacing of the languages done, and get it all supported instead of 
> just STL.

Critical piece: I believe myself that this basic support will be about 
the same amount of work as explicit support for STL. It could of course 
make good sense otherwise.


-- 
Dag Sverre

From stefan_ml at behnel.de  Sun Apr 26 21:03:13 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Apr 2009 21:03:13 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4A9E0.6090608@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<49F49AF8.7070409@behnel.de>	<49F4A01E.4040108@student.matnat.uio.no>	<49F4A5B0.7070301@behnel.de>
	<49F4A9E0.6090608@student.matnat.uio.no>
Message-ID: <49F4AFF1.40503@behnel.de>

Hi Dag,

given that I'm really far from a C++ expert, I'll try to keep my comments
at a meta-level.

Dag Sverre Seljebotn wrote:
> Ahh; the missing piece: Iterators are nothing special in C++. There's 
> absolutely *no* concept of an iterator in the language. It's just a set 
> of conventions.

Ah, thanks, didn't know that. But that doesn't mean they don't exist. It's
perfectly ok to support only the common conventions, and if users need
more, help them getting there, but without impacting the language itself.

Supporting /use cases/ in a pythonic way is IMHO much better than giving
users only the basic, c-ish tools and then stepping aside to leave them
alone with their use. You can potentially do a lot more with low-level
tools, but you will also have to go low-level to use them right. Cython is
not a low-level language.


> So that's why I propose focusing on just getting the very basic 
> interfacing of the languages done, and get it all supported instead of 
> just STL.

I think it's worth treating each operator separately (as you already seem
to have started). Some may fit the Cython language well, others will have
to be integrated in a way that doesn't break the language but keeps most of
the C++ power intact. Side effects are a clear sign for not simply mapping
them to a Python operator, for example.


> When it comes to ++/--, basic fact is that it can be used for whatever 
> purpose in C++. We are, after all, talking about the language which 
> overloads "<<" for being "output to stream" -- again, with *no* explicit 
> support for streams in the language; just by the usual operator overloading.

I don't feel a need to hold the gun when users shoot their own foot. If
they want to, the language should try not to get in their way (in the worst
case, you can always go and define a header file with wild macros), but it
should not lead them there, either.

There is no use in /integrating/ C++ into Cython. I think the idea is more
to make the sane parts of C++ easy to use and interface with. Side effects
on arbitrary operators that are visible from Cython code do not fall into
that category, IMHO. After all, you will read the Cython code and expect it
to make sense, even if you do not know that there is some wild C++ operator
overloading trickery going on in the background.

Stefan

From dagss at student.matnat.uio.no  Sun Apr 26 21:27:31 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Apr 2009 21:27:31 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4AFF1.40503@behnel.de>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<49F49AF8.7070409@behnel.de>	<49F4A01E.4040108@student.matnat.uio.no>	<49F4A5B0.7070301@behnel.de>	<49F4A9E0.6090608@student.matnat.uio.no>
	<49F4AFF1.40503@behnel.de>
Message-ID: <49F4B5A3.8010504@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi Dag,
> 
> given that I'm really far from a C++ expert, I'll try to keep my comments
> at a meta-level.
> 
> Dag Sverre Seljebotn wrote:
>> Ahh; the missing piece: Iterators are nothing special in C++. There's 
>> absolutely *no* concept of an iterator in the language. It's just a set 
>> of conventions.
> 
> Ah, thanks, didn't know that. But that doesn't mean they don't exist. It's
> perfectly ok to support only the common conventions, and if users need
> more, help them getting there, but without impacting the language itself.

I'd love to write "for item in some_cpp_col" eventually, I just think 
the basics should be considered first.

> I don't feel a need to hold the gun when users shoot their own foot. If
> they want to, the language should try not to get in their way (in the worst
> case, you can always go and define a header file with wild macros), but it
> should not lead them there, either.
> 
> There is no use in /integrating/ C++ into Cython. I think the idea is more
> to make the sane parts of C++ easy to use and interface with. Side effects
> on arbitrary operators that are visible from Cython code do not fall into
> that category, IMHO. After all, you will read the Cython code and expect it
> to make sense, even if you do not know that there is some wild C++ operator
> overloading trickery going on in the background.

Thanks for summing up my sentiments so nicely :-) We agree here.

My current stance is therefore that  I think we should *not* try to make 
sense of what C++ declarations would mean in Cython, but rather make it 
easy to write Cythonic wrappers for C++ code.

But then you need to be able to call the operators in C++ somehow, to 
create the wrapper, as there often is no other way of calling them. The 
problem is that C++ is so powerful and people do use that power.

So my/current thoughts (though Robert and Danilo are calling the shots) 
is along the lines of allowing inline C++ code. The declared interface 
must be Cythonic, but you can use C++ in the bridge in the C++. Ugly but 
effective.

Thus the ++/-- problem can be solved like this (a bit verbose, but 
syntax could perhaps be improved):

cdef extern "C++" class SomeIterator:

      # expose C++ only operation as Cython-only method call
     cdef inline SomeIterator movenext(self):
         return inline_cpp("++self")

     cdef inline SomeIterator moveprev(self):
         return inline_cpp("--self")

     cdef inline double __next__(self):
         return inline_cpp("*self++")

and so on. Obviously one could improve syntax to make it less verbose, 
provide defaults for operators etc.

(Some are left, like the conversion operators, copy constructors, 
operator=, etc., which must still be handled case-by-case)

-- 
Dag Sverre

From greg.ewing at canterbury.ac.nz  Mon Apr 27 01:28:35 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 27 Apr 2009 11:28:35 +1200
Subject: [Cython] Callback functions for C lib in Python
In-Reply-To: <49F3FF4A.4020306@behnel.de>
References: <c4ab18df0904251825n3e2a9454jc3d796707762a34a@mail.gmail.com>
	<F7FDD752-DD5C-4E54-B20B-90E5BABC905D@math.washington.edu>
	<c4ab18df0904252321j55cfcb64x5cf163d034fbc813@mail.gmail.com>
	<49F3FF4A.4020306@behnel.de>
Message-ID: <49F4EE23.2070900@canterbury.ac.nz>

Stefan Behnel wrote:

> I imagine that your question implies: "how do I pass some state into the
> callback function?". That depends on the library you are using, maybe it
> passes something into the callback function that you can control from your
> outside code.

And if there really isn't anything passed to the callback
that you can attach your own data to, you'll just have
to store the Python function globally and put up with
only being able to have one callback installed at a
time.

-- 
Greg


From lrhazi at gmail.com  Mon Apr 27 07:54:48 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 01:54:48 -0400
Subject: [Cython] typedef struct FOO *FOO
Message-ID: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>

exposing further my ignorance...

In the header file I am working with, they have several declarations
like: typedef struct FOO *FOO

FOO itself in not defined anywhere else!

What is FOO, then? and how do I translate and use that into Cython?

Thanks a lot.
Mohamed.

From m.gysel at gmail.com  Mon Apr 27 09:08:40 2009
From: m.gysel at gmail.com (Martin Gysel)
Date: Mon, 27 Apr 2009 09:08:40 +0200
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
Message-ID: <49F559F8.7010608@gmail.com>

Mohamed Lrhazi schrieb:
> exposing further my ignorance...
>
> In the header file I am working with, they have several declarations
> like: typedef struct FOO *FOO
>
> FOO itself in not defined anywhere else!
>
> What is FOO, then? and how do I translate and use that into Cython?
>   
it's an opaque pointer, see:
http://en.wikipedia.org/wiki/Opaque_pointer#C

from extern "header.h":
    struct FOO:
        pass
    ctypedef FOO *FOO  #here, I'm not sure if this works

/martin


From dsurviver at gmail.com  Mon Apr 27 12:43:38 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Mon, 27 Apr 2009 07:43:38 -0300
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <49F559F8.7010608@gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
Message-ID: <45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>

> from extern "header.h":
> ? ?struct FOO:
> ? ? ? ?pass
> ? ?ctypedef FOO *FOO ?#here, I'm not sure if this works
>
Nor me. I think the names in typedef (or ctypedef, whatever) should be
different.

From dagss at student.matnat.uio.no  Mon Apr 27 13:11:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 27 Apr 2009 13:11:18 +0200
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
Message-ID: <49F592D6.2020403@student.matnat.uio.no>

Danilo Freitas wrote:
>> from extern "header.h":
>>    struct FOO:
>>        pass
>>    ctypedef FOO *FOO  #here, I'm not sure if this works
>>
>>     
> Nor me. I think the names in typedef (or ctypedef, whatever) should be
> different.
>   

How about
cdef extern ...:
    ctypedef void* FOO

Dag Sverre

From lrhazi at gmail.com  Mon Apr 27 15:41:01 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 09:41:01 -0400
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <49F592D6.2020403@student.matnat.uio.no>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
Message-ID: <c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>

On Mon, Apr 27, 2009 at 7:11 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> How about
> cdef extern ...:
> ? ?ctypedef void* FOO

That makes more sense to me, since I dont know what the struct FOO is, anyways.

Now, this lib requires me to create one such FOO, and initialize it to
NULL, then pass a pointer to it around.
The C example does something like this:

static FOO s_foo=NULL;

then:

lib_func(&foo)

How would I do this in Cython?

cdef FOO s_foo=NULL

Does not work.
Thanks a lot,
Mohamed.

From dalcinl at gmail.com  Mon Apr 27 15:58:04 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Apr 2009 10:58:04 -0300
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
Message-ID: <e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>

What does not work? Cython or the C compiler?

I really do not like these generic "ctypedef void* FOO". I would do this:

cdef extern from "header.h":
   cdef struct struct_FOO "FOO":
      pass
   ctypedef struct_FOO* FOO


Now, you said the C example does "something like":

static FOO s_foo=NULL;

Are you sure it does that? In such case, it should work. If not,
please provide us more info about the Cython or C compiler failure...






On Mon, Apr 27, 2009 at 10:41 AM, Mohamed Lrhazi <lrhazi at gmail.com> wrote:
> On Mon, Apr 27, 2009 at 7:11 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> How about
>> cdef extern ...:
>> ? ?ctypedef void* FOO
>
> That makes more sense to me, since I dont know what the struct FOO is, anyways.
>
> Now, this lib requires me to create one such FOO, and initialize it to
> NULL, then pass a pointer to it around.
> The C example does something like this:
>
> static FOO s_foo=NULL;
>
> then:
>
> lib_func(&foo)
>
> How would I do this in Cython?
>
> cdef FOO s_foo=NULL
>
> Does not work.
> Thanks a lot,
> Mohamed.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Apr 27 16:09:36 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Apr 2009 11:09:36 -0300
Subject: [Cython] patch for #295 uploaded
Message-ID: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>

I've just called the new compiler directive "optimize_simple_methods",
like the former global option. This should be changed to a better name
once a decision is taken. A testcase is also provided.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lrhazi at gmail.com  Mon Apr 27 16:17:54 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 10:17:54 -0400
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
Message-ID: <c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>

On Mon, Apr 27, 2009 at 9:58 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> What does not work? Cython or the C compiler?
>
> I really do not like these generic "ctypedef void* FOO". I would do this:
>
> cdef extern from "header.h":
> ? cdef struct struct_FOO "FOO":
> ? ? ?pass
> ? ctypedef struct_FOO* FOO
>
>
> Now, you said the C example does "something like":
>
> static FOO s_foo=NULL;
>
> Are you sure it does that? In such case, it should work. If not,
> please provide us more info about the Cython or C compiler failure...

I tried the following:

cdef extern from "mylib.h":
  cdef struct struct_FOO "FOO":
    pass
  ctypedef struct_FOO* FOO
  int mylib_func(FOO* c)

cdef FOO s_foo
s_foo=NULL
mylib_func(&s_foo)

Cython does not complain. gcc gives a warnings: assignment from
incompatible pointer type
gcc completes fine, but when I run my code, I get a seg fault at that
last function call..

Thanks a lot and please do excuse my ignorance, it's been very long
time since I did C and pointers, got spoiled with Python.

Mohamed.

From lrhazi at gmail.com  Mon Apr 27 16:28:07 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 10:28:07 -0400
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
Message-ID: <c4ab18df0904270728k17d89e20q3b539c8e688a6f0@mail.gmail.com>

On Mon, Apr 27, 2009 at 10:17 AM, Mohamed Lrhazi <lrhazi at gmail.com> wrote:
> cdef extern from "mylib.h":
> ?cdef struct struct_FOO "FOO":
> ? ?pass
> ?ctypedef struct_FOO* FOO
> ?int mylib_func(FOO* c)
>
> cdef FOO s_foo
> s_foo=NULL
> mylib_func(&s_foo)
>

Also, cdef static FOO s_foo
gives me: Syntax error in C variable declaration

From stefan_ml at behnel.de  Mon Apr 27 16:31:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 27 Apr 2009 16:31:12 +0200 (CEST)
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
Message-ID: <fa8f8f90c259408e6e1f5eec93bb2e7c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Mohamed Lrhazi wrote:
> On Mon, Apr 27, 2009 at 9:58 AM, Lisandro Dalcin wrote:
>> What does not work? Cython or the C compiler?
>>
>> I really do not like these generic "ctypedef void* FOO". I would do
>> this:
>>
>> cdef extern from "header.h":
>> ? cdef struct struct_FOO "FOO":
>> ? ? ?pass
>> ? ctypedef struct_FOO* FOO
>>
>>
>> Now, you said the C example does "something like":
>>
>> static FOO s_foo=NULL;
>>
>> Are you sure it does that? In such case, it should work. If not,
>> please provide us more info about the Cython or C compiler failure...
>
> I tried the following:
>
> cdef extern from "mylib.h":
>   cdef struct struct_FOO "FOO":
>     pass
>   ctypedef struct_FOO* FOO
>   int mylib_func(FOO* c)
>
> cdef FOO s_foo
> s_foo=NULL
> mylib_func(&s_foo)
>
> Cython does not complain. gcc gives a warnings: assignment from
> incompatible pointer type
> gcc completes fine, but when I run my code, I get a seg fault at that
> last function call..

Could you please post the relevant parts of

a) the C code generated by Cython for the above and
b) the original definitions used in the C header files and
c) the output of the C compiler

It's cumbersome to collect all of these from separate e-mails, and to ask
back for more details on each new try.

Stefan


From lrhazi at gmail.com  Mon Apr 27 16:34:16 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 10:34:16 -0400
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <fa8f8f90c259408e6e1f5eec93bb2e7c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
	<fa8f8f90c259408e6e1f5eec93bb2e7c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <c4ab18df0904270734s234b31e4je60029b0cc7e1eda@mail.gmail.com>

On Mon, Apr 27, 2009 at 10:31 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Could you please post the relevant parts of
>
> a) the C code generated by Cython for the above and
> b) the original definitions used in the C header files and
> c) the output of the C compiler
>
> It's cumbersome to collect all of these from separate e-mails, and to ask
> back for more details on each new try.
>

I randomly kept trying variations, for lack of true understanding....
and found one that seems to work :)

I will post proper detailed info next time, when I hit my next obstacle..  :)

Thank you all.
Mohamed.

From stefan_ml at behnel.de  Mon Apr 27 16:34:33 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 27 Apr 2009 16:34:33 +0200 (CEST)
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <c4ab18df0904270728k17d89e20q3b539c8e688a6f0@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
	<c4ab18df0904270728k17d89e20q3b539c8e688a6f0@mail.gmail.com>
Message-ID: <da863e2fd17d2efc1b2076bf79ede2f9.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Mohamed Lrhazi wrote:
> On Mon, Apr 27, 2009 at 10:17 AM, Mohamed Lrhazi <lrhazi at gmail.com> wrote:
>> cdef extern from "mylib.h":
>> ?cdef struct struct_FOO "FOO":
>> ? ?pass
>> ?ctypedef struct_FOO* FOO
>> ?int mylib_func(FOO* c)
>>
>> cdef FOO s_foo
>> s_foo=NULL
>> mylib_func(&s_foo)
>>
>
> Also, cdef static FOO s_foo
> gives me: Syntax error in C variable declaration

That's because everything that you do not declare "public" is static by
default, so there is no special "static" modifier.

Stefan


From m.gysel at gmail.com  Mon Apr 27 16:34:36 2009
From: m.gysel at gmail.com (Martin Gysel)
Date: Mon, 27 Apr 2009 16:34:36 +0200
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>	<49F559F8.7010608@gmail.com>	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>	<49F592D6.2020403@student.matnat.uio.no>	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
Message-ID: <49F5C27C.3050104@gmail.com>

Mohamed Lrhazi schrieb:
> On Mon, Apr 27, 2009 at 9:58 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>   
> I tried the following:
>
> cdef extern from "mylib.h":
>   cdef struct struct_FOO "FOO":
>     pass
>   ctypedef struct_FOO* FOO
>   int mylib_func(FOO* c)
>
> cdef FOO s_foo
> s_foo=NULL
> mylib_func(&s_foo)
>   
in this case, s_foo is a pointer, right? does the lib really what the
address of a pointer rather than the pointer?

/martin


From lrhazi at gmail.com  Mon Apr 27 16:39:15 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 10:39:15 -0400
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <49F5C27C.3050104@gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
	<49F5C27C.3050104@gmail.com>
Message-ID: <c4ab18df0904270739u1f22737k517b0c8e62cfc3df@mail.gmail.com>

You guys are very generous... is there a cython-users? I feel bad
annoying people on a dev list.

Mohamed.

From m.gysel at gmail.com  Mon Apr 27 16:49:02 2009
From: m.gysel at gmail.com (Martin Gysel)
Date: Mon, 27 Apr 2009 16:49:02 +0200
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <c4ab18df0904270739u1f22737k517b0c8e62cfc3df@mail.gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>	<49F559F8.7010608@gmail.com>	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>	<49F592D6.2020403@student.matnat.uio.no>	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>	<49F5C27C.3050104@gmail.com>
	<c4ab18df0904270739u1f22737k517b0c8e62cfc3df@mail.gmail.com>
Message-ID: <49F5C5DE.2050502@gmail.com>

Mohamed Lrhazi schrieb:
> You guys are very generous... is there a cython-users? I feel bad
> annoying people on a dev list.
>   
I think there was some discussion recently about a separate mailing-list
but if I remember correctly most preferred to have both users and devs
on the same list...
there was also a discussion about an irc channel #cython on freenode,
but sadly it is most time empty... suppose most cython guys don't like
irc very much :-(

/martin


From lrhazi at gmail.com  Mon Apr 27 17:01:22 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 11:01:22 -0400
Subject: [Cython] typedef struct FOO *FOO
In-Reply-To: <49F5C5DE.2050502@gmail.com>
References: <c4ab18df0904262254v109966acn793865ee31365849@mail.gmail.com>
	<49F559F8.7010608@gmail.com>
	<45239150904270343n78e21265meaf4e06f4202da7e@mail.gmail.com>
	<49F592D6.2020403@student.matnat.uio.no>
	<c4ab18df0904270641g71ef51dena466956342ab2efd@mail.gmail.com>
	<e7ba66e40904270658n1086bf13q826a503129c517fb@mail.gmail.com>
	<c4ab18df0904270717w4d9026efxa512d96fb67724c7@mail.gmail.com>
	<49F5C27C.3050104@gmail.com>
	<c4ab18df0904270739u1f22737k517b0c8e62cfc3df@mail.gmail.com>
	<49F5C5DE.2050502@gmail.com>
Message-ID: <c4ab18df0904270801i6d713b71o3685c0164ade4e5e@mail.gmail.com>

On Mon, Apr 27, 2009 at 10:49 AM, Martin Gysel <m.gysel at gmail.com> wrote:
> I think there was some discussion recently about a separate mailing-list
> but if I remember correctly most preferred to have both users and devs
> on the same list...
> there was also a discussion about an irc channel #cython on freenode,
> but sadly it is most time empty... suppose most cython guys don't like
> irc very much :-(

Yeah, I tried #cython on freenode yesterday and found no one.
Why dont you guys just gimme your phone numbers? :)

Mohamed.

From lrhazi at gmail.com  Mon Apr 27 17:12:36 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 11:12:36 -0400
Subject: [Cython] About extension types...
Message-ID: <c4ab18df0904270812m5968a7bw8f3e9d12a1dfa9df@mail.gmail.com>

Extension types are defined as "class", which the python user would
"instantiate".

Can I assume that every Python instance, using my module,  will have
its own instance of that "class"?

Thanks,
Mohamed.

From robertwb at math.washington.edu  Mon Apr 27 21:20:49 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 12:20:49 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F465BA.8070503@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<49F465BA.8070503@student.matnat.uio.no>
Message-ID: <FB1CC970-248C-44D8-A152-751FF5B8E264@math.washington.edu>

On Apr 26, 2009, at 6:46 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> I think we're going to want to support the operator= in a limited
>>> sense that one is allowed to do conversions from one type to another
>>> (the declaration could be totally different, and potentially the  
>>> same/
>>> similar as whatever we settle on for "operator T()"). So rather than
>>> the hypothetical
>>
>> a) I don't see where operator= enters the picture here.
>> b) Even if you could do this, this is only hot-fixing one very  
>> specific
>> consequence of a much larger body of problems. I have a feeling you
>> could go around forever fixing specific cases with native Cython  
>> support
>> and get something massively complicated (and impossible to guess)
>> compared to my proposal for embedded C++ macros.
>
> This was rather unfair of me. You said:
>
> """
> I would prefer
>
> class MyCollection:
>        X __getitem__(K idx)
> """
>
> Which is not by any means a hotfix, as it is just mapping over the C++
> declaration!

My point was that the concept of "overloading based on return type"  
is foreign to Python, Cython, C++, and C, so I'd rather not introduce  
it. (It also very easily leads to ambiguities.) What exactly X is  
need not be exposed, just that it can be converted to both an A and a B.

> So I'll rephrase: Until I see it, I remain curious about how one is
> going to select a subset of C++ semantics to support which
>   a) maps to Cython semantics without too many wierd, exceptional  
> rules
> which one has no chance of guessing (which then defeats the purpose of
> pretending to support operatorX-notation in the first place)
>   b) allows wrapping of most C++ libraries without writing a wrapper
> C++-side which is more Cython-subset-friendly.

This is a thorny problem. In terms of operators, I would like to see  
at least:

1) Binary operators
2) A sensible getitem/setitem syntax
3) The ability to declare conversions (e.g. you can assign an int to  
this type, or assign this type to a double).
4) (maybe) an easy way to use the dereferencing * and ++/-- because  
of iterators.

- Robert


From robertwb at math.washington.edu  Mon Apr 27 21:32:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 12:32:17 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4B5A3.8010504@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<49F49AF8.7070409@behnel.de>	<49F4A01E.4040108@student.matnat.uio.no>	<49F4A5B0.7070301@behnel.de>	<49F4A9E0.6090608@student.matnat.uio.no>
	<49F4AFF1.40503@behnel.de> <49F4B5A3.8010504@student.matnat.uio.no>
Message-ID: <EA295FD8-921E-405E-8C07-891FAFB533DB@math.washington.edu>

On Apr 26, 2009, at 12:27 PM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Hi Dag,
>>
>> given that I'm really far from a C++ expert, I'll try to keep my  
>> comments
>> at a meta-level.
>>
>> Dag Sverre Seljebotn wrote:
>>> Ahh; the missing piece: Iterators are nothing special in C++.  
>>> There's
>>> absolutely *no* concept of an iterator in the language. It's just  
>>> a set
>>> of conventions.
>>
>> Ah, thanks, didn't know that. But that doesn't mean they don't  
>> exist. It's
>> perfectly ok to support only the common conventions, and if users  
>> need
>> more, help them getting there, but without impacting the language  
>> itself.
>
> I'd love to write "for item in some_cpp_col" eventually, I just think
> the basics should be considered first.
>
>> I don't feel a need to hold the gun when users shoot their own  
>> foot. If
>> they want to, the language should try not to get in their way (in  
>> the worst
>> case, you can always go and define a header file with wild  
>> macros), but it
>> should not lead them there, either.
>>
>> There is no use in /integrating/ C++ into Cython. I think the idea  
>> is more
>> to make the sane parts of C++ easy to use and interface with. Side  
>> effects
>> on arbitrary operators that are visible from Cython code do not  
>> fall into
>> that category, IMHO. After all, you will read the Cython code and  
>> expect it
>> to make sense, even if you do not know that there is some wild C++  
>> operator
>> overloading trickery going on in the background.
>
> Thanks for summing up my sentiments so nicely :-) We agree here.

+1 from me too.

> My current stance is therefore that  I think we should *not* try to  
> make
> sense of what C++ declarations would mean in Cython, but rather  
> make it
> easy to write Cythonic wrappers for C++ code.
>
> But then you need to be able to call the operators in C++ somehow, to
> create the wrapper, as there often is no other way of calling them.  
> The
> problem is that C++ is so powerful and people do use that power.
>
> So my/current thoughts (though Robert and Danilo are calling the  
> shots)
> is along the lines of allowing inline C++ code. The declared interface
> must be Cythonic, but you can use C++ in the bridge in the C++.  
> Ugly but
> effective.

My first impression is that this is basically a more structured  
version of the string-substitution way of wrapping C++ that we are  
trying to get away from.

- Robert


From kwmsmith at gmail.com  Mon Apr 27 21:36:23 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 27 Apr 2009 14:36:23 -0500
Subject: [Cython] Fortran support design issues
In-Reply-To: <1eda0ddb60d232e8f855f154db7be1e8.squirrel@webmail.uio.no>
References: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>
	<c266a7930904251908w76559b51lf4bd9ae4e9e34f57@mail.gmail.com>
	<1eda0ddb60d232e8f855f154db7be1e8.squirrel@webmail.uio.no>
Message-ID: <c266a7930904271236t7c9a59e7j5fffbc37614d1bf3@mail.gmail.com>

On Sun, Apr 26, 2009 at 12:30 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Kurt Smit wrote:
>> Clarifying question: ?In #177, it says,
>>
>> "Currently there's no way of creating efficient functions taking a
>> buffer argument -- for starters cdef functions doesn't support
>> declaring an argument as a buffer, and even if it did, it would mean a
>> costly (in this context) buffer acquisition on every call."
>>
>> more precisely, should it say "There's no way of creating efficient
>> functions taking a buffer argument /using the buffer syntax/..." since
>> one could always just have the function take a Py_buffer(*) argument,
>> right?
>
> Sure :-) But if you go that route, you need to access it by manually
> multiplying strides and calculating offsets etc.; and when I say "buffer
> argument" I refer to "special Cython buffers" :-) Py_buffer is just a
> normal struct seen from Cython's perspective.

Gotcha -- it's straightened out in my mind.  Not suggesting actually
going that route, just wanted to clarify.

>
>> On Fri, Apr 24, 2009 at 3:38 AM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
[snip]
>>
>> The 'buf' is passed in as a python object and a stack Py_buffer is
>> generated inside func2's body that acquires the buffer from buf. ?So
>> for 'def' functions the buffer is acquired/released within the
>> function scope, while for 'cdef' functions it is acquired/released
>> outside. ?I'd imagine that cpdef functions would be outside, too.
>
> You are right about def. There is not support for cdef or cpdef currently,
> but what you say for cdef would be #177. cpdef would need to be
> in-between: It generates a cdef function would would have the #177
> behaviour, and a def wrapper which would acquire the buffer.

Again, thanks.  Should've made it clear I was talking about the end
state after things are done, not the current state of Cython.  The
cpdef case makes sense.


>>> I first thought that adding one component more would get us all the way:
>>> Automatic copying into contiguous buffers. So assuming #177 is
>>> implemented, one would then move on to having it instead turn into:
>>>
>>> ? Py_buffer buf = acquire buffer from myobject
>>> ? if (buf is contiguous) {
>>> ? ? myfunc(&buf);
>>> ? } else {
>>> ? ? Py_buffer buf2 = make contiguous copy of buf
>>> ? ? myfunc(&buf2)
>>> ? }
>>> ? release buf
>>>
>>> However, there's a (big) problem here: What Python object does buf2
>>> refer to?
>>>
>>> Using the one of "buf" would be too confusing as they point to different
>>> memory areas. One solution is just setting it to None, perhaps. However
>>> once #177 is solved one will expect to be able to do
[snip]
> But the problem with there not being a Python object to fill in for buf2's
> Py_buffer remains. The buffer protocol doesn't define any way of creating
> a new buffer of the same Python object type -- i.e. we couldn't create a a
> new NumPy ndarray in the right way without adding an additional protocol.
> Passing buf's Python object on buf2's behalf would just be wrong as then
> accessing the buffer through e.g. slices (which goes through Python layer,
> at least currently) would access a different memory area than through item
> indexing...

I see.  So having 2 ways of getting to & modifying the data (fast
through the buffer with typed indices, or slowly through the Python
API and the PyObject pointer) makes it imperative that whatever buffer
is passed to the function references the same data as the PyObject.

>
>>> All of this makes me think about pushing the "new buffer syntax" a bit
>>> harder and get it started on in your GSoC. With that,
>>>
>>> ? cdef void myfunc(int[:] buf)
>>>
>>> could easily give non-surprising effects for #177 and copy-in/copy-out,
>>> as
>>> the Python object is not "part of the deal".

Revisiting this point:  myfunc would get a Py_buffer passed to it,
that Py_buffer would have a reference to some python object.  If
myfunc gets the buf->obj PyObject and accesses it in a way that would
use the Python API, (e.g. a slice on the referenced object) we'd run
into the same problems as above, right?  Unless we say that form of
access is not allowed or is undefined.  I.e. one could only access &
modify the buffer's data through the buffer's void *buf field, not
going through the buf->obj PyObject reference.

Would it be simply a matter of not allowing the programmer access to
the buffer's PyObject reference in the new buffer syntax?  This would
work within the Cython file itself, but an external C function which
got the Py_buffer would have to be aware of the pitfalls.

A better question:  if a contiguous Py_buffer is created from a
discontiguous PyObject with a contiguous copy, what to do with the
buffer's obj reference, since that PyObject will have a different
memory area?  Would this be 'out of bounds' since it is understood
that by using a Py_buffer one must go through the buffer's void *buf
field?

If slicing or buffer operations were supported, we'd have to
explicitly support it ourselves in Cython, right?  This would be a
future enhancement.

> This is a digression:
>
> Actually I was planning on having int[:] mean strided, and perhaps
> int[::1] or something like that mean contiguous. One could use the third
> field for any kind of stride configuration:
>
> ?int[:,::1] - C-contiguous
> ?int[::1,:] - Fortran-contiguous
> ?int[:,:] - Strided
>
> These would be magic short-hands for more explicit specifications. Some more:
> ?int[::strided, ::indirect] - Matrix stored as pointers to strided columns
> ?int[::full, ::1] - First index could use any scheme (if-test required to
> see if suboffset is -1 or not), while second index is always contiguous.
>
> It just means the syntax is extensible, we would start with only
> supporting contiguous.

[snip]

> Ah OK now I see where you are heading with the explicit cast. No, I didn't
> think of it like this. Py_buffer is just a reference to existing memory --
> we can "fake" that memory being contiguous by copying for a moment, but
> that shouldn't change semantics -- an "int[:]" variable is a reference,
> not allocated memory.
>
> This is very different from Fortran, but matches the way Python works
> where everything is a reference.
>
> I.e. you could also do
>
> myfunc(myobject)
> cdef int[:] buf = myobject
>
> and buf would get the output of myfunc. And:
>
> cdef int[:] a = ...
> cdef int[:] b = a
> b[3] = 2 # also shows up in a[3]
>
> and so on.
>
> Note that in general, we should support all cases for cdef functions:
>
> cdef func(int[::full] buf) # or something like that
>
> would never require copying as it would support all 1D buffers. But if you do
>
> cdef func(int[::1] buf)
>
> then the buffer is forced to be contiguous, if necesarry by copying in and
> out.

I'm with you on the above.  Would we need to draw up a CEP detailing
the new syntax, etc and get it approved by the community?  (basically
expanding http://wiki.cython.org/enhancements/buffersyntax)

>>> Your GSoC would then Cython-side consist more or less of
>>>
>>> a) #177, with new syntax
>>> b) A generic mechanism for automatic coercion between buffers of
>>> different
>>> modes. That is:
>>>
>>> cdef int[::contiguous] buf # new syntax for mode="c" w/o Python obj?
>>> buf = some_object
>>>
>>> Here, if the buffer of some_object is not contiguous, a contiguous copy
>>> will be made! And when releasing buf, it would be copied back.
>>>
>>> This would make the parts necesarry for Fortran support tremendously
>>> useful elsewhere for what I believe will not be much extra effort.
>>>
>>> (Though I could help out with the parts of those not needed for Fortran
>>> support in order to not derail your project.)
>>
>> I'll need to digest this a bit, but I like it.
>
> Sure. Note that I corrected my b) point in a seperate mail.

Right.  To put it in one place:

> Anyway, b) above is not good, it raises a lot of questions on semantics
> (what is int[:] really -- a reference to memory or the memory itself --
> what happens if one acquires two buffers simultaneously, do they point
> to the same memory -- etc). So replace b) above with:
>
> b) A mechanism for automatically making contiguous copy-in/copy-out on
> #177-style method calls if necesarry.

What needs to be done to write up the CEP and to start implementing
the new syntax, or a subset of it for the GSoC puprposes?  What
blockers are there that need to be addressed?  Would we need to
resolve the problem raised in the above section before starting?

Also, presuming the new buffer syntax is fully in place (not as a
result of my project, which just starts on its implementation) would
you expect it to be a replacement for the old syntax?  We'd need some
way to convert a Py_buffer to a PyObject; as you mention we could do
this in the numpy.pxd with a __frombuffer__ method, or by backporting
the memoryview to older Python versions.

Kurt

From robertwb at math.washington.edu  Mon Apr 27 21:39:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 12:39:19 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F49198.4050105@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>	<49F4860A.9090106@student.matnat.uio.no>
	<45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>
	<49F49198.4050105@student.matnat.uio.no>
Message-ID: <2CFCB592-1109-49C0-8E6E-82B74985E38A@math.washington.edu>

On Apr 26, 2009, at 9:53 AM, Dag Sverre Seljebotn wrote:

> Danilo Freitas wrote:
>> As I said before, I prefer a Pythonic syntax for Cython.
>> So, deref() would be better in this case.
>> But, for all this and more, we need to define a default syntax.
>> Mixing syntax is a little weird and harder to use.
>>
>> If we translate C++ syntax to Pythonic  syntax, I think it will be
>> easier to use it.
>>
>> So, if we gonna do it, many other functions will be added to the  
>> list.
>> I have some Ideas. Don't know if they are good. But it sounds like:
>>
>> my_list = clist(int) #just an example
>> it = cython.get_begin_iterator(my_list)
>> while it != cython.get_end_iterator(my_list):
>>     if something == cython.deref(it): #just a normal if
>>         break
>>     cython.inc(it)
>>
>> So, we could get iterator fom many ways. Just like in STL.
>
> Yes. (Though my vote is for my_list.begin() and my_list.end()  
> instead.)

Yes, this doesn't take anything special.

> The main issue with using STL iterators in Cython though is what types
> to add.
>
>
> For comparison, this is how it is /currently/ possible in Cython to  
> do this:
>
> # the proper extern declarations
>
> # use vector in example, then we don't need inc/dec/deref
>
> cdef cpp_vector_int* vec = new_cpp_vector_int()
> cdef cpp_vector_int_iterator* it = \
>    new_cpp_vector_int_iterator(vec.begin())
> cdef cpp_vector_int_iterator* end = \
>    new_cpp_vector_int_iterator(vec.end())
>
> while it[0] != end[0]:
>    if something = it[0][0]:
>      break
>    it[0] += 1
>
> cpp_delete(it)
> cpp_delete(end)
> cpp_delete(vec)
>
>
> Not pretty. The types are the problem -- if one could simply do
>
> cdef cpp_vector_int_iterator it = vec.begin()
>
> things would look much nicer.

What is the obstruction to this? One should be able to declare the  
vector<T> class as having a begin method returning an iterator<T>,  
it's all about templates, not operators (well the initializing of it  
at least).

- Robert



From robertwb at math.washington.edu  Mon Apr 27 21:46:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 12:46:23 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F4860A.9090106@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
Message-ID: <5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>

On Apr 26, 2009, at 9:04 AM, Dag Sverre Seljebotn wrote:

> Danilo Freitas wrote:
>>>> That's a fair summary. I am also for supporting only a subset of  
>>>> what
>>>> is possible--if one really needs crazy stuff like differently
>>>> implemented prefix/postfix decrements and -> vs (*). one can  
>>>> code in C
>>>> ++ or use clever macros. (Would we need to expose ++, a.k.a
>>>> "__next__" for STL iterators?)
>>> Yes, you need ++/-- for std::list<T>::iterator, which doesn't  
>>> accept any
>>> form of +=, but we can just compile += 1 and -= 1 to those  
>>> operators in
>>> general (also for plain ints for that matter), so it's not really a
>>> problem. (We are then requiring that += 1 would have the same  
>>> semantics,
>>> which I'm 100% fine with.) This holds for both my and your stance  
>>> BTW
>>> and is orthogonal.
>>>
>>
>> Could we define a function int() and dec() for that?
>>
>> like, inc(iterator), dec(iterator).
>
> Good idea! +1, if it is done in the "cython" namespace:
>
> from cython import inc, dec
>
> Other "magic" functions are already like in the cython namespace (it's
> not a real module).
>
> Note that for STL iterators we also need *it rather than [0], so add
> "deref" to that list. Or possibly we just add * to Cython?
>
> We could then also move on to define the next builtin for C++ classes,
> which would translate like this:
>
> obj = next(it)
>
> would become
>
> cython.inc(it)
> obj = cython.deref(it)

In general, I like this proposal, but I do see some drawbacks when  
one extends beyond inc/dec. One problem is what would the return type  
of cython.decref be? I would rather see them as methods attached to  
objects (which is what they really are) than top-level functions.  
(cython.inc/dec could always return "void" (even thought they don't)  
and no function overloading needs to go on here).

I could see supporting unary * as a native Cython operator (though  
one would still want to be able to specify its type when operated on  
a class overloading it).

- Robert


From dagss at student.matnat.uio.no  Mon Apr 27 22:05:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 27 Apr 2009 22:05:23 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <2CFCB592-1109-49C0-8E6E-82B74985E38A@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>	<49F4860A.9090106@student.matnat.uio.no>	<45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>	<49F49198.4050105@student.matnat.uio.no>
	<2CFCB592-1109-49C0-8E6E-82B74985E38A@math.washington.edu>
Message-ID: <49F61003.3050208@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 26, 2009, at 9:53 AM, Dag Sverre Seljebotn wrote:
>> cdef cpp_vector_int* vec = new_cpp_vector_int()
>> cdef cpp_vector_int_iterator* it = \
>>    new_cpp_vector_int_iterator(vec.begin())
>> cdef cpp_vector_int_iterator* end = \
>>    new_cpp_vector_int_iterator(vec.end())
>>
>> while it[0] != end[0]:
>>    if something = it[0][0]:
>>      break
>>    it[0] += 1
>>
>> cpp_delete(it)
>> cpp_delete(end)
>> cpp_delete(vec)
>>
>>
>> Not pretty. The types are the problem -- if one could simply do
>>
>> cdef cpp_vector_int_iterator it = vec.begin()
>>
>> things would look much nicer.
> 
> What is the obstruction to this? One should be able to declare the  
> vector<T> class as having a begin method returning an iterator<T>,  
> it's all about templates, not operators (well the initializing of it  
> at least).

The BIGGEST problem I see with C++ support in Cython right now lies 
right here. I beg you to make sure you fully understand this.

The issue is that vector<T>::iterator does not have a no-arg constructor.

I.e. it is legal to do

vector<int>::iterator it = vec.begin()

but this gives a compilation error:

vector<int>::iterator it;

So, to get it to work, one must basically do

vector<int>::iterator* it = NULL;
...even get hold of vec somehow...
*it = vec.begin()
... now, first element in vec is **it

There seems to be two solutions, of which I strongly dislike one and 
like the other.

The one I strongly dislike:

I,ntroduce C++ variable scoping in Cython (otherwise you cannot supply 
the constructor arguments when you have them, and run the destructor at 
the right time)

The one I like is here, under "Stack allocation":
http://wiki.cython.org/enhancements/cpp


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Apr 27 22:16:07 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 27 Apr 2009 22:16:07 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <EA295FD8-921E-405E-8C07-891FAFB533DB@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F16BC1.5050708@student.matnat.uio.no>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<49F49AF8.7070409@behnel.de>	<49F4A01E.4040108@student.matnat.uio.no>	<49F4A5B0.7070301@behnel.de>	<49F4A9E0.6090608@student.matnat.uio.no>	<49F4AFF1.40503@behnel.de>
	<49F4B5A3.8010504@student.matnat.uio.no>
	<EA295FD8-921E-405E-8C07-891FAFB533DB@math.washington.edu>
Message-ID: <49F61287.6030101@student.matnat.uio.no>

Robert Bradshaw wrote:
>> My current stance is therefore that  I think we should *not* try to  
>> make
>> sense of what C++ declarations would mean in Cython, but rather  
>> make it
>> easy to write Cythonic wrappers for C++ code.
>>
>> But then you need to be able to call the operators in C++ somehow, to
>> create the wrapper, as there often is no other way of calling them.  
>> The
>> problem is that C++ is so powerful and people do use that power.
>>
>> So my/current thoughts (though Robert and Danilo are calling the  
>> shots)
>> is along the lines of allowing inline C++ code. The declared interface
>> must be Cythonic, but you can use C++ in the bridge in the C++.  
>> Ugly but
>> effective.
> 
> My first impression is that this is basically a more structured  
> version of the string-substitution way of wrapping C++ that we are  
> trying to get away from.

Good observation. I now believe strongly that it has merits though: It 
allows one to skip the whole issue by keeping Cython semantics in Cython 
and C++ semantics in C++.

I'm sure noone asked for it, but even so here's my draft on how I'd like 
to see C++ support.

http://wiki.cython.org/enhancements/inlinecpp

Note:
  a) It has many uses outside of C++ wrapping as well
  b) It is probably easier to implement than anything else we could come 
up with (not that that is a deciding factor)
  c) By its hacky nature, it gives full support at once rather than 
covering case-by-case

Think about being the author of a C++ library wanting to wrap it in 
Cython -- would this be so bad?

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Apr 27 22:23:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 27 Apr 2009 22:23:58 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F61003.3050208@student.matnat.uio.no>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>	<49F4860A.9090106@student.matnat.uio.no>	<45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>	<49F49198.4050105@student.matnat.uio.no>	<2CFCB592-1109-49C0-8E6E-82B74985E38A@math.washington.edu>
	<49F61003.3050208@student.matnat.uio.no>
Message-ID: <49F6145E.4010808@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> On Apr 26, 2009, at 9:53 AM, Dag Sverre Seljebotn wrote:
>>> cdef cpp_vector_int* vec = new_cpp_vector_int()
>>> cdef cpp_vector_int_iterator* it = \
>>>    new_cpp_vector_int_iterator(vec.begin())
>>> cdef cpp_vector_int_iterator* end = \
>>>    new_cpp_vector_int_iterator(vec.end())
>>>
>>> while it[0] != end[0]:
>>>    if something = it[0][0]:
>>>      break
>>>    it[0] += 1
>>>
>>> cpp_delete(it)
>>> cpp_delete(end)
>>> cpp_delete(vec)
>>>
>>>
>>> Not pretty. The types are the problem -- if one could simply do
>>>
>>> cdef cpp_vector_int_iterator it = vec.begin()
>>>
>>> things would look much nicer.
>> What is the obstruction to this? One should be able to declare the  
>> vector<T> class as having a begin method returning an iterator<T>,  
>> it's all about templates, not operators (well the initializing of it  
>> at least).
> 
> The BIGGEST problem I see with C++ support in Cython right now lies 
> right here. I beg you to make sure you fully understand this.
> 
> The issue is that vector<T>::iterator does not have a no-arg constructor.
> 
> I.e. it is legal to do
> 
> vector<int>::iterator it = vec.begin()
> 
> but this gives a compilation error:
> 
> vector<int>::iterator it;
> 
> So, to get it to work, one must basically do
> 
> vector<int>::iterator* it = NULL;
> ...even get hold of vec somehow...
> *it = vec.begin()
> ... now, first element in vec is **it

Ouch. That obviously does not work. It should be this:

vector<int>::iterator* it = NULL;
... get vec
it = new vector<int>::iterator(vec.begin());
... use it, value is in **it
delete it;

-- 
Dag Sverre

From stefan_ml at behnel.de  Mon Apr 27 22:50:37 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 27 Apr 2009 22:50:37 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
Message-ID: <49F61A9D.3000505@behnel.de>


Robert Bradshaw wrote:
> On Apr 26, 2009, at 9:04 AM, Dag Sverre Seljebotn wrote:
>> Note that for STL iterators we also need *it rather than [0], so add
>> "deref" to that list. Or possibly we just add * to Cython?
>>
>> We could then also move on to define the next builtin for C++ classes,
>> which would translate like this:
>>
>> obj = next(it)
>>
>> would become
>>
>> cython.inc(it)
>> obj = cython.deref(it)
> 
> I could see supporting unary * as a native Cython operator (though  
> one would still want to be able to specify its type when operated on  
> a class overloading it).

Note that there would be an ambiguity with PEP 3132, though. We (and Pyrex)
did pretty well without unary* for quite a while. I'd be very careful with
adding it now. Especially for something like iterators, i.e. a concept that
Python already has (even if it looks and works different in the two
languages), and for which dereferencing is a totally unrelated concept from
a Python POV.

Stefan

From kwmsmith at gmail.com  Mon Apr 27 22:51:55 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 27 Apr 2009 15:51:55 -0500
Subject: [Cython] First round of benchmarks for GSoC fortran project
Message-ID: <c266a7930904271351w1cbb5623rfe398f358da10aca@mail.gmail.com>

Hi All,

It was suggested off list that I gather some simple benchmarks
comparing the different ways of passing a struct with a contiguous
array pointer inside -- through a copy, through a pointer, and through
explicit unpacking of the struct.

So far, I find almost no difference whatsoever between the 3 modes,
which is encouraging.  If these benchmarks hold up, it means we can
choose the 'best' way to pass Py_buffer structs around without worry
of a speed hit.

Comments welcome.

reps are the number of repetitions per measurement, length is the
length of the array.  Each loop was run 3 times and the times are
reported below.

Using -O2 optimization, gcc 4.2.4.

copy passes a struct copy (i.e. func(struct_arg)), ptr passes a
reference (func(&struct_arg)) and nostruct is func(float *dta,
unsigned int length).

# On a Core 2 Duo, 3.16 GHz, 6 MB L2 Cache.
# reps: 10000 length: 10000
# copy: 0.280, 0.280, 0.290
# nostruct: 0.280, 0.290, 0.290
# ptr: 0.280, 0.280, 0.290

# reps: 100000 length: 1000
# copy: 0.280, 0.280, 0.290
# nostruct: 0.280, 0.290, 0.290
# ptr: 0.280, 0.290, 0.290

# reps: 1000000 length: 100
# copy: 0.280, 0.280, 0.300
# nostruct: 0.290, 0.290, 0.290
# ptr: 0.290, 0.290, 0.290

# reps: 10000000 length: 10
# copy: 0.140, 0.150, 0.150
# nostruct: 0.140, 0.140, 0.140
# ptr: 0.140, 0.140, 0.150

Kurt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: benchmarks.tgz
Type: application/x-gzip
Size: 908 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090427/71638b12/attachment.bin 

From dagss at student.matnat.uio.no  Mon Apr 27 22:55:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 27 Apr 2009 22:55:42 +0200
Subject: [Cython] Fortran support design issues
In-Reply-To: <c266a7930904271236t7c9a59e7j5fffbc37614d1bf3@mail.gmail.com>
References: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>	<c266a7930904251908w76559b51lf4bd9ae4e9e34f57@mail.gmail.com>	<1eda0ddb60d232e8f855f154db7be1e8.squirrel@webmail.uio.no>
	<c266a7930904271236t7c9a59e7j5fffbc37614d1bf3@mail.gmail.com>
Message-ID: <49F61BCE.8000809@student.matnat.uio.no>

You seem to be fully up to speed now, happy to see that :-)

Kurt Smith wrote:
> On Sun, Apr 26, 2009 at 12:30 AM, Dag Sverre Seljebotn wrote:
>>>> All of this makes me think about pushing the "new buffer syntax" a bit
>>>> harder and get it started on in your GSoC. With that,
>>>>
>>>>   cdef void myfunc(int[:] buf)
>>>>
>>>> could easily give non-surprising effects for #177 and copy-in/copy-out,
>>>> as
>>>> the Python object is not "part of the deal".
> 
> Revisiting this point:  myfunc would get a Py_buffer passed to it,
> that Py_buffer would have a reference to some python object.  If
> myfunc gets the buf->obj PyObject and accesses it in a way that would
> use the Python API, (e.g. a slice on the referenced object) we'd run
> into the same problems as above, right?  Unless we say that form of
> access is not allowed or is undefined.  I.e. one could only access &
> modify the buffer's data through the buffer's void *buf field, not
> going through the buf->obj PyObject reference.
> 
> Would it be simply a matter of not allowing the programmer access to
> the buffer's PyObject reference in the new buffer syntax?  This would
> work within the Cython file itself, but an external C function which
> got the Py_buffer would have to be aware of the pitfalls.

True.

A)

An interesting question raised here is whether Py_buffer should be 
passed at all. Perhaps
   object[int] would mean passing a Py_buffer
   int[::1] would mean passing struct {void* buf; size_t shape0;}
   int[:] would mean passing struct {void* buf; size_t shape0, stride0;}

Hmm. But refcounts, possible storage and releasing the buffer etc. are 
needed, aren't they...

So the above doesn't work; I just wanted to make a point about the 
actual shape of the buffer passing not being set in stone. I think 
benchmarks could help here -- the Py_buffer struct is big and you need 
pointer lookups to get what you need onto the stack in the reciever. If 
necesarry, Py_buffer could be allocated on the heap and passed as a 
pointer and only used for refcounting. Like this:

struct {
   Py_buffer buf;
   size_t refcount; // users of buffer, release buffer on 0
} __Pyx_buffer;

And then int[:] would mean passing
struct { __Pyx_buffer* buf; void* bif; size_t shape0, stride0; }

(that doesn't add information, but could be faster)

B)

I was thinking of perhaps adding some attributes to this new buffer 
type. One could allow

cdef int[:] buf = ...
print buf.object

which would get the original object from which the buffer was once 
acquired -- but without.

But, this object could also be set to None. And I think for contiguous 
copies, as well as a possible future feature; casting from C pointers:

cdef int[:] buf = <int[:]>someptr

it should be set to None.

> 
> A better question:  if a contiguous Py_buffer is created from a
> discontiguous PyObject with a contiguous copy, what to do with the
> buffer's obj reference, since that PyObject will have a different
> memory area?  Would this be 'out of bounds' since it is understood
> that by using a Py_buffer one must go through the buffer's void *buf
> field?

Hmm. Actually, when doing a contiguous copy, we need to manage that 
memory somehow (and the called function could assign the int[:] to a 
global int[:] var or whatever; I want to start supporting those).

So we can create our own class for that (subclassing memoryview under 
Py2.6+ only).

> If slicing or buffer operations were supported, we'd have to
> explicitly support it ourselves in Cython, right?  This would be a
> future enhancement.

Yep. And then the underlying object and the int[:] would definitely get 
out of sync; I think that should just be documented and "how it works".

> I'm with you on the above.  Would we need to draw up a CEP detailing
> the new syntax, etc and get it approved by the community?  (basically
> expanding http://wiki.cython.org/enhancements/buffersyntax)

I did raise the issue when I wrote that CEP, the response amounted to 
Robert saying "interesting!" :-) So at least noone were outright 
rejecting it at the time.

There's also a thread on the NumPy list:

http://thread.gmane.org/gmane.comp.python.numeric.general/28439

>> Anyway, b) above is not good, it raises a lot of questions on semantics
>> (what is int[:] really -- a reference to memory or the memory itself --
>> what happens if one acquires two buffers simultaneously, do they point
>> to the same memory -- etc). So replace b) above with:
>>
>> b) A mechanism for automatically making contiguous copy-in/copy-out on
>> #177-style method calls if necesarry.
> 
> What needs to be done to write up the CEP and to start implementing
> the new syntax, or a subset of it for the GSoC puprposes?  What
> blockers are there that need to be addressed?  Would we need to
> resolve the problem raised in the above section before starting?

I actually think a good starting point right now is to "take a step 
back": We now understand the issues involved, so let's aim for something 
much lower.

Basically, let it suffice to mid-term evaluation to do the Fortran 
integration while requiring any passed arrays to be Fortran-contiguous.

We can use

cdef external_func(object[int] foo)

as a syntax for this and pass the Py_buffer as-is without ever copying. 
(So drop copying until int[:] can be introduced to get rid of the object 
implications in the syntax.)

If a copy is needed: Raise an exception.

This isn't as bad as it sounds, as with NumPy arrays you can just call 
copy('F') manually for now. So it will be usable, though I hope we can 
get further after that is in place.

Then, after mid-term we can see, and there's plenty of time to plan, let 
things mature, etc.

While we're planning: Do you want me to have a look at the G3 f2py 
source, or will you just attempt some first steps and ping me when you 
see what is needed there or have a question?

> 
> Also, presuming the new buffer syntax is fully in place (not as a
> result of my project, which just starts on its implementation) would
> you expect it to be a replacement for the old syntax?  We'd need some
> way to convert a Py_buffer to a PyObject; as you mention we could do
> this in the numpy.pxd with a __frombuffer__ method, or by backporting
> the memoryview to older Python versions.

I'd like it to take over in the end, but for that to happen we would 
need to implement slices and arithmetic operators in Cython sufficiently 
well, which means it might never happen at the current rate.

But given enough developer hours on this, in the end I hope to be able to do

cdef double my_func(double x): return x*x

def f(buf):
     cdef double[:] buf = arg
     return my_func(buf) + 3 + buf # expands to element-wise for-loop

and so on.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Apr 27 22:57:39 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 27 Apr 2009 22:57:39 +0200
Subject: [Cython] Fortran support design issues
In-Reply-To: <49F61BCE.8000809@student.matnat.uio.no>
References: <596107acfcd66ba6aca7512dd3c8299c.squirrel@webmail.uio.no>	<c266a7930904251908w76559b51lf4bd9ae4e9e34f57@mail.gmail.com>	<1eda0ddb60d232e8f855f154db7be1e8.squirrel@webmail.uio.no>	<c266a7930904271236t7c9a59e7j5fffbc37614d1bf3@mail.gmail.com>
	<49F61BCE.8000809@student.matnat.uio.no>
Message-ID: <49F61C43.8010709@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> So the above doesn't work; I just wanted to make a point about the 
> actual shape of the buffer passing not being set in stone. I think 
> benchmarks could help here -- the Py_buffer struct is big and you need 
> pointer lookups to get what you need onto the stack in the reciever. If 
> necesarry, Py_buffer could be allocated on the heap and passed as a 
> pointer and only used for refcounting. Like this:
> 
> struct {
>    Py_buffer buf;
>    size_t refcount; // users of buffer, release buffer on 0
> } __Pyx_buffer;
> 
> And then int[:] would mean passing
> struct { __Pyx_buffer* buf; void* bif; size_t shape0, stride0; }
> 
> (that doesn't add information, but could be faster)

Heh, seems this is already outdated :-)

-- 
Dag Sverre

From stefan_ml at behnel.de  Mon Apr 27 23:04:58 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 27 Apr 2009 23:04:58 +0200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>	<49F4860A.9090106@student.matnat.uio.no>
	<45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>
Message-ID: <49F61DFA.603@behnel.de>


Danilo Freitas wrote:
> Dag wrote:
>> from cython import inc, dec
>>
>> Other "magic" functions are already like in the cython namespace (it's
>> not a real module).
>>
>> Note that for STL iterators we also need *it rather than [0], so add
>> "deref" to that list. Or possibly we just add * to Cython?
>>
>> We could then also move on to define the next builtin for C++ classes,
>> which would translate like this:
>>
>> obj = next(it)
>>
>> would become
>>
>> cython.inc(it)
>> obj = cython.deref(it)
> 
> As I said before, I prefer a Pythonic syntax for Cython.
> So, deref() would be better in this case.
> But, for all this and more, we need to define a default syntax.
> Mixing syntax is a little weird and harder to use.
> 
> If we translate C++ syntax to Pythonic  syntax, I think it will be
> easier to use it.
> 
> So, if we gonna do it, many other functions will be added to the list.
> I have some Ideas. Don't know if they are good. But it sounds like:
> 
> my_list = clist(int) #just an example
> it = cython.get_begin_iterator(my_list)
> while it != cython.get_end_iterator(my_list):
>     if something == cython.deref(it): #just a normal if
>         break
>     cython.inc(it)
> 
> So, we could get iterator fom many ways. Just like in STL.

To me, this looks like the main problem is mapping one protocol (in C++) to
another (in Python). Would it help to define some kind of DSL that allows
us to do that? Say, you could define a mapping from (P|C)ython's iterator
protocol (__iter__ and __next__) to corresponding operations in C++, and
Cython would then just insert the corresponding code into the source at the
right places (and maybe replace some place holders). Then you'd have some
default implementation that works with the 'normal' way C++ iterators are
used, and users could extend that to adapt it to other stuff.

I have no idea how such a DSL would look like, so it's really just an idea
(and it's somewhat close to Dag's string insertions). I'm mainly saying
that not everything needs to be implemented by users in Cython code.

Stefan


From robertwb at math.washington.edu  Mon Apr 27 23:30:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 14:30:17 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F61A9D.3000505@behnel.de>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>	<49F2DF1D.7090403@student.matnat.uio.no>	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>	<49F40B0C.2050707@student.matnat.uio.no>	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>	<49F45D09.6040202@student.matnat.uio.no>	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
Message-ID: <8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>

On Apr 27, 2009, at 1:50 PM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Apr 26, 2009, at 9:04 AM, Dag Sverre Seljebotn wrote:
>>> Note that for STL iterators we also need *it rather than [0], so add
>>> "deref" to that list. Or possibly we just add * to Cython?
>>>
>>> We could then also move on to define the next builtin for C++  
>>> classes,
>>> which would translate like this:
>>>
>>> obj = next(it)
>>>
>>> would become
>>>
>>> cython.inc(it)
>>> obj = cython.deref(it)
>>
>> I could see supporting unary * as a native Cython operator (though
>> one would still want to be able to specify its type when operated on
>> a class overloading it).
>
> Note that there would be an ambiguity with PEP 3132, though.

Argh, I forgot about that PEP.

> We (and Pyrex) did pretty well without unary* for quite a while.  
> I'd be very careful with
> adding it now. Especially for something like iterators, i.e. a  
> concept that
> Python already has (even if it looks and works different in the two
> languages), and for which dereferencing is a totally unrelated  
> concept from
> a Python POV.

That's because x[0] was always the same thing as (*x), even if a bit  
uglier. I'm not not as enthusiastic about it now (though not yet for  
sure set against it either...)

- Robert


From robertwb at math.washington.edu  Mon Apr 27 23:34:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 14:34:45 -0700
Subject: [Cython] About extension types...
In-Reply-To: <c4ab18df0904270812m5968a7bw8f3e9d12a1dfa9df@mail.gmail.com>
References: <c4ab18df0904270812m5968a7bw8f3e9d12a1dfa9df@mail.gmail.com>
Message-ID: <C2FFDF41-9ED0-4ECD-8CB8-078AFE6ECB33@math.washington.edu>

On Apr 27, 2009, at 8:12 AM, Mohamed Lrhazi wrote:

> Extension types are defined as "class", which the python user would
> "instantiate".
>
> Can I assume that every Python instance, using my module,  will have
> its own instance of that "class"?

I'm not sure at all what your asking here, but I believe the answer  
is that things behave just like Python. I.e. if I write

class A:
     pass

cdef class B:
     pass

Then there will be exactly one instance of A and one instance of B  
floating around (just like there's one tuple type, one list type, etc.)

- Robert


From dsurviver at gmail.com  Mon Apr 27 23:44:30 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Mon, 27 Apr 2009 18:44:30 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
Message-ID: <45239150904271444o5f719899ubc1dad9581c4e9db@mail.gmail.com>

Dag, are there more wikis about enchancements  of C++?
I'd like to put all these ideads together, to simplify my work.

And, if possible, could you tell the mainly ideas? The most important of them?
Just to set priorities.

Thanks

From robertwb at math.washington.edu  Mon Apr 27 23:51:15 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 14:51:15 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <45239150904271444o5f719899ubc1dad9581c4e9db@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
	<45239150904271444o5f719899ubc1dad9581c4e9db@mail.gmail.com>
Message-ID: <EF2F97DE-66BE-4566-8FED-D0D21E18E64B@math.washington.edu>

On Apr 27, 2009, at 2:44 PM, Danilo Freitas wrote:

> Dag, are there more wikis about enchancements  of C++?
> I'd like to put all these ideads together, to simplify my work.

That would be *excellent*.

> And, if possible, could you tell the mainly ideas? The most  
> important of them?
> Just to set priorities.

I see (1) C++ class types (including upcasting) (2) function  
overloading (3) templates and (4) operator overloading in the simple  
cases (e.g. binary operators) as the priorities, but not sure about  
the order. I think all are feasible in the given timeframe.

- Robert


From dsurviver at gmail.com  Mon Apr 27 23:52:33 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Mon, 27 Apr 2009 18:52:33 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F61DFA.603@behnel.de>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<45239150904260918l1fe5d7aey2e03b52b8ea9af24@mail.gmail.com>
	<49F61DFA.603@behnel.de>
Message-ID: <45239150904271452s1b28a77bg7d9721421f4e5899@mail.gmail.com>

>
> To me, this looks like the main problem is mapping one protocol (in C++) to
> another (in Python). Would it help to define some kind of DSL that allows
> us to do that? Say, you could define a mapping from (P|C)ython's iterator
> protocol (__iter__ and __next__) to corresponding operations in C++, and
> Cython would then just insert the corresponding code into the source at the
> right places (and maybe replace some place holders). Then you'd have some
> default implementation that works with the 'normal' way C++ iterators are
> used, and users could extend that to adapt it to other stuff.

Are you saying to "substitute" Python __iter__ and __next__ for
iterator operations in C++?
I don't think that's cool. I think it's important to let users still
use Python functions.

Am I right? Or just did a mistake?

From lrhazi at gmail.com  Tue Apr 28 00:57:34 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 18:57:34 -0400
Subject: [Cython] About extension types...
In-Reply-To: <C2FFDF41-9ED0-4ECD-8CB8-078AFE6ECB33@math.washington.edu>
References: <c4ab18df0904270812m5968a7bw8f3e9d12a1dfa9df@mail.gmail.com>
	<C2FFDF41-9ED0-4ECD-8CB8-078AFE6ECB33@math.washington.edu>
Message-ID: <c4ab18df0904271557k73eb9423v713b47ecb77cbf42@mail.gmail.com>

On Mon, Apr 27, 2009 at 5:34 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> class A:
> ? ? pass
>
> cdef class B:
> ? ? pass
>

I meant if this module is called foo, and I have two python instances
running that both import foo, and both instantiate class A, would
there be one such "A" shared by both processes/Pythons?

Thanks,
Mohamed.

From robertwb at math.washington.edu  Tue Apr 28 01:59:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 16:59:46 -0700
Subject: [Cython] About extension types...
In-Reply-To: <c4ab18df0904271557k73eb9423v713b47ecb77cbf42@mail.gmail.com>
References: <c4ab18df0904270812m5968a7bw8f3e9d12a1dfa9df@mail.gmail.com>
	<C2FFDF41-9ED0-4ECD-8CB8-078AFE6ECB33@math.washington.edu>
	<c4ab18df0904271557k73eb9423v713b47ecb77cbf42@mail.gmail.com>
Message-ID: <C6B9484F-A11C-46CD-AF83-52F55052E236@math.washington.edu>

On Apr 27, 2009, at 3:57 PM, Mohamed Lrhazi wrote:

> On Mon, Apr 27, 2009 at 5:34 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> class A:
>>     pass
>>
>> cdef class B:
>>     pass
>>
>
> I meant if this module is called foo, and I have two python instances
> running that both import foo, and both instantiate class A, would
> there be one such "A" shared by both processes/Pythons?

No, Python has no mechanism (other than explicit pickling/message  
passing) to share data across multiple processes.

- Robert


From greg.ewing at canterbury.ac.nz  Tue Apr 28 03:10:34 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 28 Apr 2009 13:10:34 +1200
Subject: [Cython] About extension types...
In-Reply-To: <C2FFDF41-9ED0-4ECD-8CB8-078AFE6ECB33@math.washington.edu>
References: <c4ab18df0904270812m5968a7bw8f3e9d12a1dfa9df@mail.gmail.com>
	<C2FFDF41-9ED0-4ECD-8CB8-078AFE6ECB33@math.washington.edu>
Message-ID: <49F6578A.5030902@canterbury.ac.nz>

Robert Bradshaw wrote:

> class A:
>      pass
> 
> cdef class B:
>      pass
> 
> Then there will be exactly one instance of A and one instance of B  
> floating around

Um, no, there are *no* instances of A or B until you
create some, e.g.

   a = A()
   b = B()

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Apr 28 03:08:45 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 28 Apr 2009 13:08:45 +1200
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
Message-ID: <49F6571D.6060008@canterbury.ac.nz>

Robert Bradshaw wrote:

> That's because x[0] was always the same thing as (*x), even if a bit  
> uglier. I'm not not as enthusiastic about it now (though not yet for  
> sure set against it either...)

The reason for not having a * operator in Pyrex was to
avoid possible ambiguities when parsing arguments to
function calls (since Python already assigns a meaning
to prefix * in that context).

If you want to introduce a * operator you'll have to
deal with that somehow.

-- 
Greg

From robertwb at math.washington.edu  Tue Apr 28 03:57:47 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 18:57:47 -0700
Subject: [Cython] About extension types...
In-Reply-To: <49F6578A.5030902@canterbury.ac.nz>
References: <c4ab18df0904270812m5968a7bw8f3e9d12a1dfa9df@mail.gmail.com>
	<C2FFDF41-9ED0-4ECD-8CB8-078AFE6ECB33@math.washington.edu>
	<49F6578A.5030902@canterbury.ac.nz>
Message-ID: <BA47E790-E858-4ECF-90C9-5E4CCB78C4F4@math.washington.edu>

On Apr 27, 2009, at 6:10 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> class A:
>>      pass
>>
>> cdef class B:
>>      pass
>>
>> Then there will be exactly one instance of A and one instance of B
>> floating around
>
> Um, no, there are *no* instances of A or B until you
> create some, e.g.
>
>    a = A()
>    b = B()

I mean there's only one "A" object in the sense that there's only one  
"tuple" object. "Instance" was arguable a bad word to choose here  
(but it was the one used in the question).

- Robert


From lrhazi at gmail.com  Tue Apr 28 04:03:22 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Mon, 27 Apr 2009 22:03:22 -0400
Subject: [Cython] Dereferencing a pointer....
Message-ID: <c4ab18df0904271903r6f618afamb44190a9fa42ee14@mail.gmail.com>

I have two lib functions declared like this:

void arg_set(handle p, void *arg)
void arg_get(handle p, void **arg)

I call arg_set() from an instance of extension type, like this:
arg_set(handle, <void *>self)

Hoping to access the instance back in arg_get. So in a callback function I do:

cdef void** arg
arg_get(handle, arg)

But I cannot figure out how to use the retrieved arg, I keep getting
segfaults, or object has no such attrib type of errors...

print "call back: config_update: tm_arg_get: ",(<object>(<void*>arg[0])).channel

What would be the right way to dereference such a pointer?

I also tried:
cdef void* arg
arg_get(handle, &arg)
print "call back: config_update: arg_get: ",(<object>arg[0]).channel

But would not compile. gcc dies.

Thanks alot,
Mohamed.

From robertwb at math.washington.edu  Tue Apr 28 04:10:37 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 19:10:37 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <49F6571D.6060008@canterbury.ac.nz>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<9f7dc1ee0738271c79db1608845ea8bf.squirrel@webmail.uio.no>
	<44C30F42-E7DB-42A0-9287-2D2FD4B25360@math.washington.edu>
	<49F2DF1D.7090403@student.matnat.uio.no>
	<FC817A79-F479-4D6C-B3E7-ECAE7B3F2EED@math.washington.edu>
	<d6aefb5b2a708e725060059e9de510b7.squirrel@webmail.uio.no>
	<4168A6CE-8DE7-4B1C-8E7A-1751FE1E4203@math.washington.edu>
	<49F40B0C.2050707@student.matnat.uio.no>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
	<49F6571D.6060008@canterbury.ac.nz>
Message-ID: <46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu>

On Apr 27, 2009, at 6:08 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> That's because x[0] was always the same thing as (*x), even if a bit
>> uglier. I'm not not as enthusiastic about it now (though not yet for
>> sure set against it either...)
>
> The reason for not having a * operator in Pyrex was to
> avoid possible ambiguities when parsing arguments to
> function calls (since Python already assigns a meaning
> to prefix * in that context).
>
> If you want to introduce a * operator you'll have to
> deal with that somehow.

Nice to know the original reasoning. Well, that's another strike  
against it, and in this case "two strikes you're out" is enough for me.

- Robert


From robertwb at math.washington.edu  Tue Apr 28 04:25:30 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 19:25:30 -0700
Subject: [Cython] patch for #295 uploaded
In-Reply-To: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>
References: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>
Message-ID: <0011301C-D364-4509-8D11-361197586653@math.washington.edu>

On Apr 27, 2009, at 7:09 AM, Lisandro Dalcin wrote:

> I've just called the new compiler directive "optimize_simple_methods",
> like the former global option. This should be changed to a better name
> once a decision is taken. A testcase is also provided.

Looks good. Another option for the name would be  
"always_allow_keywords" because that what the usecase is.

- Robert


From dsurviver at gmail.com  Tue Apr 28 04:35:28 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Mon, 27 Apr 2009 23:35:28 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
	<49F6571D.6060008@canterbury.ac.nz>
	<46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu>
Message-ID: <45239150904271935h5be5a414y3b3c71b9d3b13397@mail.gmail.com>

2009/4/27 Robert Bradshaw <robertwb at math.washington.edu>:
> On Apr 27, 2009, at 6:08 PM, Greg Ewing wrote:
>
>> Robert Bradshaw wrote:
>>
>>> That's because x[0] was always the same thing as (*x), even if a bit
>>> uglier. I'm not not as enthusiastic about it now (though not yet for
>>> sure set against it either...)
>>
>> The reason for not having a * operator in Pyrex was to
>> avoid possible ambiguities when parsing arguments to
>> function calls (since Python already assigns a meaning
>> to prefix * in that context).
>>
>> If you want to introduce a * operator you'll have to
>> deal with that somehow.
>
> Nice to know the original reasoning. Well, that's another strike
> against it, and in this case "two strikes you're out" is enough for me.
>
So, we should really use deref()?

From robertwb at math.washington.edu  Tue Apr 28 04:44:29 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 19:44:29 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <45239150904271935h5be5a414y3b3c71b9d3b13397@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<A40D01C2-E7BB-4F02-94BA-CBA88AE54D1E@math.washington.edu>
	<49F45D09.6040202@student.matnat.uio.no>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
	<49F6571D.6060008@canterbury.ac.nz>
	<46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu>
	<45239150904271935h5be5a414y3b3c71b9d3b13397@mail.gmail.com>
Message-ID: <7FB5D572-3A62-4D98-8B82-598BA8B0E408@math.washington.edu>

On Apr 27, 2009, at 7:35 PM, Danilo Freitas wrote:

> 2009/4/27 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Apr 27, 2009, at 6:08 PM, Greg Ewing wrote:
>>
>>> Robert Bradshaw wrote:
>>>
>>>> That's because x[0] was always the same thing as (*x), even if a  
>>>> bit
>>>> uglier. I'm not not as enthusiastic about it now (though not yet  
>>>> for
>>>> sure set against it either...)
>>>
>>> The reason for not having a * operator in Pyrex was to
>>> avoid possible ambiguities when parsing arguments to
>>> function calls (since Python already assigns a meaning
>>> to prefix * in that context).
>>>
>>> If you want to introduce a * operator you'll have to
>>> deal with that somehow.
>>
>> Nice to know the original reasoning. Well, that's another strike
>> against it, and in this case "two strikes you're out" is enough  
>> for me.
>>
> So, we should really use deref()?

Yes, I think so.

We could declare

cdef extern from "foo.h":
     cclass Foo:  # or whatever else we come up with here
         Foo __add__(Foo, Foo)
         Foo __add__(Foo, int)
         int __dereference__(Foo)
         Foo __increment__(Foo)
         ...

This would indicate that Foo supports addition with other Foos and  
ints, incrementing, and that cython.deref(Foo) is an int. The magic  
cython.deref method would see if a __dereference__ method is  
declared, and if not would allow the default behavior for pointer  
types, otherwise raising an error. An actual (*x) would be emitted in  
the code. Likewise with increment/decrement.

- Robert



From dsurviver at gmail.com  Tue Apr 28 05:05:37 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Tue, 28 Apr 2009 00:05:37 -0300
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <7FB5D572-3A62-4D98-8B82-598BA8B0E408@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
	<49F6571D.6060008@canterbury.ac.nz>
	<46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu>
	<45239150904271935h5be5a414y3b3c71b9d3b13397@mail.gmail.com>
	<7FB5D572-3A62-4D98-8B82-598BA8B0E408@math.washington.edu>
Message-ID: <45239150904272005m190be281x46926d477dca3202@mail.gmail.com>

Gettin all together, we want:

cdef extern from "foo.h":
    cclass Foo:  # or whatever else we come up with here
        Foo __add__(Foo, Foo)
        Foo __add__(Foo, int)
        int __dereference__(Foo)
        Foo __increment__(Foo) #different of inc(), right?
        cpp_Iterator __iter__() #or something like this

foo = Foo()
for inc() and dec(), there are many ways to do it;

it = foo.begin()
it = get_begin_iterator(foo) #or cython.get_begin_iterator(foo)

with types:

cdef cpp_Iterator it = foo.begin()
...

and to use in() and dec():

cython.inc(it)
foo.inc(it) #this way, it's possible to customize your own inc() and dec()
it.inc()

we just need to find the better way to use these functions/methods.

I think it would be the same for deref (only on syntax, no custom
implementation)

From robertwb at math.washington.edu  Tue Apr 28 05:21:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Apr 2009 20:21:08 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <45239150904272005m190be281x46926d477dca3202@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<45239150904260855l7a5bcec6qa56d2e0e7a41f6e4@mail.gmail.com>
	<49F4860A.9090106@student.matnat.uio.no>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
	<49F6571D.6060008@canterbury.ac.nz>
	<46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu>
	<45239150904271935h5be5a414y3b3c71b9d3b13397@mail.gmail.com>
	<7FB5D572-3A62-4D98-8B82-598BA8B0E408@math.washington.edu>
	<45239150904272005m190be281x46926d477dca3202@mail.gmail.com>
Message-ID: <C973C222-7230-46BC-905A-2C1826E735A4@math.washington.edu>

On Apr 27, 2009, at 8:05 PM, Danilo Freitas wrote:

> Gettin all together, we want:

Well, it's a proposal.

> cdef extern from "foo.h":
>     cclass Foo:  # or whatever else we come up with here
>         Foo __add__(Foo, Foo)
>         Foo __add__(Foo, int)
>         int __dereference__(Foo)
>         Foo __increment__(Foo) #different of inc(), right?
>         cpp_Iterator __iter__() #or something like this
>
> foo = Foo()
> for inc() and dec(), there are many ways to do it;
>
> it = foo.begin()
> it = get_begin_iterator(foo) #or cython.get_begin_iterator(foo)

I'd much rather have foo.begin(), but here there might be a question  
of allocation...

> with types:
>
> cdef cpp_Iterator it = foo.begin()
> ...
>
> and to use in() and dec():
>
> cython.inc(it)
> foo.inc(it) #this way, it's possible to customize your own inc()  
> and dec()
> it.inc()

The problem with foo.inc() is that there might be a class method  
named inc that is completely orthogonal to ++.

> we just need to find the better way to use these functions/methods.
>
> I think it would be the same for deref (only on syntax, no custom
> implementation)

Yep.

In terms of summarizing the discussion, it would be nice to have a  
page (e.g. http://wiki.cython.org/cpp/operators ) that listed a big  
table all operators in C++ (with grouping, e.g. binary arithmetic  
operators don't need individual treatment) and if/how we'd plan on  
supporting them in Cython (this has two parts--how to declare, and  
how to invoke).

- Robert

From mcfletch at vrplumber.com  Tue Apr 28 06:43:15 2009
From: mcfletch at vrplumber.com (Mike C. Fletcher)
Date: Tue, 28 Apr 2009 00:43:15 -0400
Subject: [Cython] A few suggestions for the numpy tutorial documentation...
Message-ID: <49F68963.60409@vrplumber.com>

Hi all,

I've spent the day writing a Numpy format handler in Cython and thought
there were a few things that the Cython numpy tutorial possibly should
cover:

    * Numpy dtype references appear to be normally "borrowed" references
      in the C API, so when you call numpy array functions that take a
      dtype you need to Py_INCREF the dtype first, this will show up as
      complaints from numpy that you're trying to delete the dtype objects
    * You have to call import_array() in your module or many of the
      Numpy C API entry points will segfault

for the first issue:

cdef extern from "Python.h":
    cdef void Py_INCREF( object )
...
        Py_INCREF( typecode )
        return PyArray_Zeros( c_dims.shape[0],
<np.Py_intptr_t>c_dims.data, typecode, 0 )

for the second issue:

cdef extern from "numpy/arrayobject.h":
    cdef void import_array()
...
import_array()

Anyway, not sure if or where such changes should go, but just thought it
might help others.

Enjoy,
Mike

-- 
________________________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://www.vrplumber.com
  http://blog.vrplumber.com


From stefan_ml at behnel.de  Tue Apr 28 07:16:28 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Apr 2009 07:16:28 +0200
Subject: [Cython] patch for #295 uploaded
In-Reply-To: <0011301C-D364-4509-8D11-361197586653@math.washington.edu>
References: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>
	<0011301C-D364-4509-8D11-361197586653@math.washington.edu>
Message-ID: <49F6912C.7060106@behnel.de>


Robert Bradshaw wrote:
> On Apr 27, 2009, at 7:09 AM, Lisandro Dalcin wrote:
> 
>> I've just called the new compiler directive "optimize_simple_methods",
>> like the former global option. This should be changed to a better name
>> once a decision is taken. A testcase is also provided.
> 
> Looks good. Another option for the name would be  
> "always_allow_keywords" because that what the usecase is.

+1 for "always_allow_keywords"

Stefan


From stefan_ml at behnel.de  Tue Apr 28 07:26:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Apr 2009 07:26:56 +0200
Subject: [Cython] Dereferencing a pointer....
In-Reply-To: <c4ab18df0904271903r6f618afamb44190a9fa42ee14@mail.gmail.com>
References: <c4ab18df0904271903r6f618afamb44190a9fa42ee14@mail.gmail.com>
Message-ID: <49F693A0.7080900@behnel.de>

Hi,

just a general comment on your questions. Your description below provides
some context (which is fine), but please copy error messages exactly. Just
quoting part of them (especially if you do not understand them), or saying
"does not compile" without further background is not helpful to those who
want to help.

This might help, BTW:

http://www.catb.org/~esr/faqs/smart-questions.html


Mohamed Lrhazi wrote:
> I have two lib functions declared like this:
> 
> void arg_set(handle p, void *arg)
> void arg_get(handle p, void **arg)
> 
> I call arg_set() from an instance of extension type, like this:
> arg_set(handle, <void *>self)
> 
> Hoping to access the instance back in arg_get. So in a callback function I do:
> 
> cdef void** arg
> arg_get(handle, arg)
> 
> But I cannot figure out how to use the retrieved arg, I keep getting
> segfaults, or object has no such attrib type of errors...
> 
> print "call back: config_update: tm_arg_get: ",(<object>(<void*>arg[0])).channel
> 
> What would be the right way to dereference such a pointer?
> 
> I also tried:
> cdef void* arg
> arg_get(handle, &arg)
> print "call back: config_update: arg_get: ",(<object>arg[0]).channel
> 
> But would not compile. gcc dies.

There are several issues here. One is that "arg" is a void* in the last
example, not a void**, so saying (<object>arg[0]) will not work, as an
"object" is a kind of pointer, too. Use "<object>arg". If you want to
figure out what's happening, add a "print <int>arg" to see the value of the
pointer, and compare it to what you put in.

Then, the segfaults you get are likely due to the object being no longer
there (or maybe some other reason, can't tell from what you write). Please
provide some background on how and when the callback is called.

Stefan

From dagss at student.matnat.uio.no  Tue Apr 28 08:05:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 08:05:42 +0200 (CEST)
Subject: [Cython] A few suggestions for the numpy tutorial
 documentation...
In-Reply-To: <49F68963.60409@vrplumber.com>
References: <49F68963.60409@vrplumber.com>
Message-ID: <00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>

> Hi all,
>
> I've spent the day writing a Numpy format handler in Cython and thought
> there were a few things that the Cython numpy tutorial possibly should
> cover:

Are you talking about wiki.cython.org/tutorials/numpy? I think that's
targeted to a a different audience who don't immedeately feel comfortable
calling the NumPy C API...

>     * Numpy dtype references appear to be normally "borrowed" references
>       in the C API, so when you call numpy array functions that take a
>       dtype you need to Py_INCREF the dtype first, this will show up as
>       complaints from numpy that you're trying to delete the dtype objects

Good point.

>     * You have to call import_array() in your module or many of the
>       Numpy C API entry points will segfault

Are you sure about this? I never do it. I know you need to do it from C
code, but is it not enough in Cython to do

import numpy

?

Did you actually get a segfault using Cython? Could you send me an
example? I'd really like to make this automatic somehow.

> Anyway, not sure if or where such changes should go, but just thought it
> might help others.

I you feel it fits within the tutorial feel free to add it there,
otherwise you can always create a sub-page about "Using the NumPy C API"
or somesuch.

Dag Sverre


From dagss at student.matnat.uio.no  Tue Apr 28 08:57:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 08:57:41 +0200
Subject: [Cython] New idea for C++ stack allocation
Message-ID: <49F6A8E5.2040006@student.matnat.uio.no>

I got another idea for how using C++ objects without no-arg constructors 
could work. It means enforcing C++ variable scoping through compile-time 
errors, but keeping Python syntax.

a) This gives compiler error: "CppObject has no no-arg constructor, but 
is used before assigned to"

   cdef CppObject obj
   obj.a = 3


b) This works:

   cdef CppObject obj
   if foo:
     obj = bar
     obj.func()
     del obj # note: introducing del, but only in a special case
   else:
     obj = bar2
     obj.asdf()
     del obj

c) Compiler error: "C++ object must be deleted before exiting block"

   cdef CppObject obj
   if foo:
     obj = bar
     obj.func()

d) Compiler error: "C++ object must be deleted within the same block"

   cdef CppObject obj
   if foo:
     obj = bar
     if True:
       del obj

e) All variables are deleted on exit

   cdef CppObject obj = bar
   obj.func()
   # end of function so don't require del

The transform necesarry to enforce these rules should not be too 
complicated. I don't think we need full analysis, just track assignments 
and deletions and usages on a per-name basis.

Finally, the C++ output is made by adding a block, i.e.

   cdef CppObject obj
   print 1
   obj = bar
   obj.func()
   del obj
   print 2

Results in

   print 1
   {
     CppObject obj = bar;
     obj.func()
   } # del
   print 2


Related: One must be able to call two-argument constructors on the stack 
as well. This could e.g happen like this:

cdef CppObject obj
print 1
obj = CppObject(1,2) # or, obj = (1, 2)

which would translate to

print 1
{
   CppObject obj(1,2);
   ...
}


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Apr 28 09:34:34 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 09:34:34 +0200
Subject: [Cython] First round of benchmarks for GSoC fortran project
In-Reply-To: <c266a7930904271351w1cbb5623rfe398f358da10aca@mail.gmail.com>
References: <c266a7930904271351w1cbb5623rfe398f358da10aca@mail.gmail.com>
Message-ID: <49F6B18A.2030407@student.matnat.uio.no>

Kurt Smith wrote:
> Hi All,
> 
> It was suggested off list that I gather some simple benchmarks
> comparing the different ways of passing a struct with a contiguous
> array pointer inside -- through a copy, through a pointer, and through
> explicit unpacking of the struct.
> 
> So far, I find almost no difference whatsoever between the 3 modes,
> which is encouraging.  If these benchmarks hold up, it means we can
> choose the 'best' way to pass Py_buffer structs around without worry
> of a speed hit.
> 
> Comments welcome.

Nice. I do have some comments to refine this:

  - Note that your "simp" vs. "unpack" are likely identical in 
assembler, so you don't need to have them both I thinkk.

  - We also want to compare Py_buffer vs. simpler formats. E.g. if you 
want to pass a simple contiguous buffer, do you pass Py_buffer or your 
"simp"?

  - Py_buffer is much larger than your simp, which could affect the 
benchmark for ptr vs. non-ptr.

  - Make sure ll is Py_ssize_t, this could affect things on 64-bit systems.

  - In Py_buffer, the shape is stored as a pointer, so you actually need 
an extra pointer dereference to get the shape: buf.shape[0] vs. simp.ll.

So the next step would be actually using the Py_buffer struct (which is 
a Cython builtin and is documented in PEP 3118), and see how Py_buffer 
(either by copy or by ptr) holds up against simp.

As Py_buffer would be filled out outside of what you do timings on, you 
don't need to fill in things besides what you need, this will suffice:

cdef Py_buffer buf
buf.shape = &sp.ll # array of lenght 1 :-)
                    # Will cause big problems if sp.ll is not Py_ssize_t!
buf.buf = sp.dta

-- 
Dag Sverre

From hoytak at cs.ubc.ca  Tue Apr 28 09:37:09 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Tue, 28 Apr 2009 00:37:09 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <C973C222-7230-46BC-905A-2C1826E735A4@math.washington.edu>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com> 
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu> 
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu> 
	<49F6571D.6060008@canterbury.ac.nz>
	<46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu> 
	<45239150904271935h5be5a414y3b3c71b9d3b13397@mail.gmail.com> 
	<7FB5D572-3A62-4D98-8B82-598BA8B0E408@math.washington.edu> 
	<45239150904272005m190be281x46926d477dca3202@mail.gmail.com> 
	<C973C222-7230-46BC-905A-2C1826E735A4@math.washington.edu>
Message-ID: <4db580fd0904280037u7edbed4dob63efbd50d5b2139@mail.gmail.com>

Pardon me jumping into the discussion so late, and perhaps missing
something too, but since I've done a lot of c++ coding, here's a few
observations:

1.  It seems that the discussion of iterators and the discussion of
implementing generators are pretty closely linked, at least for basic
iterator types.  For the basic iterator types, I could see having a
special generator type that accepts starting and ending iterator
instances, or methods of the C++ class (such as .begin() and .end()
for vector), as constructor arguments.  At this level they are
conceptual quite similar, at least using the STL iterator concept.  It
does not seem to me like accessing the iterators directly (e.g.
through .inc) is needed if it is provided through fast C-level
generators.  Most of the way the STL is stitched together is so
conceptually similar to python generators, especially in Py3, that it
should be part of this discussion, and the  C++ STL ideal is to
minimize the need for low level operations on the iterators.  (Robert,
I have a book on exactly the concepts of iterators and generic
programming with the C++ STL if you'd like to borrow it for a while;
I'll contribute what I can, but I'm really busy now.)

2. Also, what types in the STL iterator type hierarchy should we
support?  There are forward iterators, backward iterators,
bidirectional iterators which generalize those, and random access
iterators which generalize those.  In addition, there are reading and
writing versions of these iterators.  Plus const/non-const version to
worry about.  Big pain, I know, but seems like good c++ wrapping will
have to deal with all this.  From experience, forward and backward
iterators are the most common, with random access being next, though
these all could perhaps be wrapped in other generator or class
types...

my 2c,
--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From dagss at student.matnat.uio.no  Tue Apr 28 11:27:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 11:27:06 +0200 (CEST)
Subject: [Cython] Fortran support design issues
Message-ID: <0c7e294f10ba11493158f8a6baae36c5.squirrel@webmail.uio.no>

I wrote:
> Kurt Smith wrote:
>> What needs to be done to write up the CEP and to start implementing the
new syntax, or a subset of it for the GSoC puprposes?  What blockers
are there that need to be addressed?  Would we need to resolve the
problem raised in the above section before starting?
>
> I actually think a good starting point right now is to "take a step
back": We now understand the issues involved, so let's aim for something
much lower.
>
> Basically, let it suffice to mid-term evaluation to do the Fortran
integration while requiring any passed arrays to be Fortran-contiguous.
>
> We can use
>
> cdef external_func(object[int] foo)
>
> as a syntax for this and pass the Py_buffer as-is without ever copying.
(So drop copying until int[:] can be introduced to get rid of the object
implications in the syntax.)
>
> If a copy is needed: Raise an exception.
>
> This isn't as bad as it sounds, as with NumPy arrays you can just call
copy('F') manually for now. So it will be usable, though I hope we can
get further after that is in place.
>
> Then, after mid-term we can see, and there's plenty of time to plan, let
things mature, etc.
>
> While we're planning: Do you want me to have a look at the G3 f2py
source, or will you just attempt some first steps and ping me when you
see what is needed there or have a question?

I don't know how fast you'd like to jump into Cython sources, but here's
something which would be a nice start, and which is needed whether we have
the new syntax or not.

It is what will be required to do

extfunc(obj)

rather than having to do

cdef np.ndarray[int] buf = obj
extfunc(buf)

As such, it isn't absolutely top priority, and you might want to work from
the f2py end first instead. (It would be nice to have a Fortran wrapper in
place before actually doing the extfunc(buf) step; but that is next to
trivial anyway, just passed the Py_buffer we already have...)

"Subproject description":

Currently buffer acquistion/release is implemented through something of a
hack (which Robert actually advised against during my GSoC, but I was
stressed about getting things done and less experienced with Cython).
Buffers are only supported on function local variables, and assignments to
those variables are special-cased for acquisition/release.

Instead one should do acquisition/release through the existing type
coercion system in Cython.

I've put the necesarry bits in #299 and #300 in trac.

Also, I've tagged *candidates* for tickets you will do in your GSoC so far
with "kurtgsoc", and added a link on the front page of trac to a query of
those.

Dag Sverre


From robertwb at math.washington.edu  Tue Apr 28 11:53:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 28 Apr 2009 02:53:40 -0700
Subject: [Cython] Operators for ctypedef struct
In-Reply-To: <4db580fd0904280037u7edbed4dob63efbd50d5b2139@mail.gmail.com>
References: <c93c21390904240010l6b1fc172oc2f1717a8ff65a0@mail.gmail.com>
	<5CFDB812-9738-46B1-AC66-33CA12299D14@math.washington.edu>
	<49F61A9D.3000505@behnel.de>
	<8ACD9D8C-712C-446F-A946-310704E4CAD5@math.washington.edu>
	<49F6571D.6060008@canterbury.ac.nz>
	<46A1DFE4-0111-4558-A1B5-6D42F0E7A7F7@math.washington.edu>
	<45239150904271935h5be5a414y3b3c71b9d3b13397@mail.gmail.com>
	<7FB5D572-3A62-4D98-8B82-598BA8B0E408@math.washington.edu>
	<45239150904272005m190be281x46926d477dca3202@mail.gmail.com>
	<C973C222-7230-46BC-905A-2C1826E735A4@math.washington.edu>
	<4db580fd0904280037u7edbed4dob63efbd50d5b2139@mail.gmail.com>
Message-ID: <481439F9-75DA-4823-B9A3-868CBC719BC6@math.washington.edu>

On Apr 28, 2009, at 12:37 AM, Hoyt Koepke wrote:

> Pardon me jumping into the discussion so late,

Never to late to jump in, and the more people we get feedback from  
the better the final result will be.

> and perhaps missing
> something too, but since I've done a lot of c++ coding, here's a few
> observations:
>
> 1.  It seems that the discussion of iterators and the discussion of
> implementing generators are pretty closely linked, at least for basic
> iterator types.  For the basic iterator types, I could see having a
> special generator type that accepts starting and ending iterator
> instances, or methods of the C++ class (such as .begin() and .end()
> for vector), as constructor arguments.  At this level they are
> conceptual quite similar, at least using the STL iterator concept.  It
> does not seem to me like accessing the iterators directly (e.g.
> through .inc) is needed if it is provided through fast C-level
> generators.  Most of the way the STL is stitched together is so
> conceptually similar to python generators, especially in Py3, that it
> should be part of this discussion, and the  C++ STL ideal is to
> minimize the need for low level operations on the iterators.  (Robert,
> I have a book on exactly the concepts of iterators and generic
> programming with the C++ STL if you'd like to borrow it for a while;
> I'll contribute what I can, but I'm really busy now.)

They are conceptually very similar, but very different implementation- 
wise. For generators we need closures, i.e. the ability to store an  
re-instantiate the (virtual) stack frame of a function call.  
Iterators are already implemented and we just need to be able to call  
them. (One can currently use generators just fine in Cython, we just  
can't create them).

Rather than inform Cython about (some subset of) C++ iterator  
semantics, the goal is to extend Cython to add C++ features to make  
it easy to wrap arbitrary C++ code. STL provides a very good test  
case--one should be able to use it in Cython the same way one uses it  
in C++.

Of course, eventually it would be nice to be able to "loop" over an  
iterator with a for-from or for-in loop.

> 2. Also, what types in the STL iterator type hierarchy should we
> support?  There are forward iterators, backward iterators,
> bidirectional iterators which generalize those, and random access
> iterators which generalize those.  In addition, there are reading and
> writing versions of these iterators.  Plus const/non-const version to
> worry about.  Big pain, I know, but seems like good c++ wrapping will
> have to deal with all this.  From experience, forward and backward
> iterators are the most common, with random access being next, though
> these all could perhaps be wrapped in other generator or class
> types...

If we enable using iterators as in C++, rather than trying to provide  
Cython wrappers for iterators, we don't need to know about all of  
these various types in the Cython compiler.

- Robert


From jf.moulin at gmail.com  Tue Apr 28 13:27:51 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 28 Apr 2009 13:27:51 +0200
Subject: [Cython] compiler message
Message-ID: <af6d35810904280427x15f67f75lc0bad606d4da9dd3@mail.gmail.com>

Hi all,

please find in attach the compile errors I got....
this is too cryptic for me too see the link to an actual error in my code...
The only modification I made to a compilable code is to use the numpy buffer
notation for locals in a cdef block which I attach in a second file.

Thanks in advance for your input.
JF
-------------- next part --------------

jfmoulin at jfm:~/My_Progs/refsans/refsans_cython.fix_03_optimize_numpy/read_sweep/
C$ less python setup_delay_line.py build_ext --inplace
running build_ext
cythoning delay_line.pyx to delay_line.c
Traceback (most recent call last):
  File "setup_delay_line.py", line 6, in <module>
    ext_modules = [Extension("delay_line_C", ["delay_line.pyx"])]
  File "/usr/lib/python2.5/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.5/distutils/dist.py", line 974, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.5/distutils/command/build_ext.py", line 290, in run
    self.build_extensions()
  File "/usr/lib/python2.5/site-packages/Cython/Distutils/build_ext.py", line 81, in build_extensions
    ext.sources = self.cython_sources(ext.sources, ext)
  File "/usr/lib/python2.5/site-packages/Cython/Distutils/build_ext.py", line 196, in cython_sources
    full_module_name=module_name)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 690, in compile
    return compile_single(source, options, full_module_name)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 635, in compile_single
    return run_pipeline(source, options, full_module_name)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 524, in run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 183, in run_pipeline
    data = phase(data)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 177, in __call__
    return super(CythonTransform, self).__call__(node)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 163, in __call__
    return self.visit(root)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 43, in visit
    return handler_method(obj)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 588, in visit_ModuleNode
    self.visitchildren(node)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 146, in visitchildren
    result = super(VisitorTransform, self).visitchildren(parent, attrs)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 119, in visitchildren
    childretval = self.visitchild(child, parent, attr, None)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 95, in visitchild
    result = self.visit(child)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 43, in visit
    return handler_method(obj)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 187, in visit_Node
    self.visitchildren(node)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 146, in visitchildren
    result = super(VisitorTransform, self).visitchildren(parent, attrs)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 117, in visitchildren
    childretval = [self.visitchild(x, parent, attr, idx) for idx, x in enumerate(child)]
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 95, in visitchild
    result = self.visit(child)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 43, in visit
    return handler_method(obj)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 187, in visit_Node
    self.visitchildren(node)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 146, in visitchildren
    result = super(VisitorTransform, self).visitchildren(parent, attrs)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 117, in visitchildren
    childretval = [self.visitchild(x, parent, attr, idx) for idx, x in enumerate(child)]
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 95, in visitchild
    result = self.visit(child)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 43, in visit
    return handler_method(obj)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 187, in visit_Node
    self.visitchildren(node)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 146, in visitchildren
    result = super(VisitorTransform, self).visitchildren(parent, attrs)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 119, in visitchildren
    childretval = self.visitchild(child, parent, attr, None)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 95, in visitchild
    result = self.visit(child)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 43, in visit
    return handler_method(obj)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 187, in visit_Node
    self.visitchildren(node)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 146, in visitchildren
    result = super(VisitorTransform, self).visitchildren(parent, attrs)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 117, in visitchildren
    childretval = [self.visitchild(x, parent, attr, idx) for idx, x in enumerate(child)]
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 95, in visitchild
    result = self.visit(child)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Visitor.py", line 43, in visit
    return handler_method(obj)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 592, in visit_FuncDefNode
    node.body.analyse_expressions(node.local_scope)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 317, in analyse_expressions
    stat.analyse_expressions(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 3644, in analyse_expressions
    self.body.analyse_expressions(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 317, in analyse_expressions
    stat.analyse_expressions(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 3399, in analyse_expressions
    self.else_clause.analyse_expressions(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 317, in analyse_expressions
    stat.analyse_expressions(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 3863, in analyse_expressions
    self.body.analyse_expressions(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 317, in analyse_expressions
    stat.analyse_expressions(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 2631, in analyse_expressions
    self.analyse_types(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 2921, in analyse_types
    self.lhs.analyse_target_types(env)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/ExprNodes.py", line 1502, in analyse_target_types
    self.analyse_base_and_index_types(env, setting = 1)
  File "/usr/lib/python2.5/site-packages/Cython/Compiler/ExprNodes.py", line 1527, in analyse_base_and_index_types
    assert not isinstance(self.index, CloneNode)
AssertionError
-------------- next part --------------
A non-text attachment was scrubbed...
Name: code.pyx
Type: application/octet-stream
Size: 2230 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090428/c1b8bb08/attachment.obj 

From dagss at student.matnat.uio.no  Tue Apr 28 13:53:11 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 13:53:11 +0200
Subject: [Cython] compiler message
In-Reply-To: <af6d35810904280427x15f67f75lc0bad606d4da9dd3@mail.gmail.com>
References: <af6d35810904280427x15f67f75lc0bad606d4da9dd3@mail.gmail.com>
Message-ID: <49F6EE27.6030200@student.matnat.uio.no>

Jean-Francois Moulin wrote:
> Hi all,
> 
> please find in attach the compile errors I got....
> this is too cryptic for me too see the link to an actual error in my code...
> The only modification I made to a compilable code is to use the numpy buffer
> notation for locals in a cdef block which I attach in a second file.

Unfortunately this is a bug:

http://trac.cython.org/cython_trac/ticket/155

There's a workaround explained there. I can't tell yet when the bug will 
be fixed.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Apr 28 15:06:36 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 15:06:36 +0200
Subject: [Cython] #155 (compiler message)
In-Reply-To: <af6d35810904280427x15f67f75lc0bad606d4da9dd3@mail.gmail.com>
References: <af6d35810904280427x15f67f75lc0bad606d4da9dd3@mail.gmail.com>
Message-ID: <49F6FF5C.4000309@student.matnat.uio.no>

Jean-Francois Moulin wrote:
> Hi all,
> 
> please find in attach the compile errors I got....
> this is too cryptic for me too see the link to an actual error in my code...
> The only modification I made to a compilable code is to use the numpy buffer
> notation for locals in a cdef block which I attach in a second file.

When I looked at it it was actually something else than I thought and 
much simpler to fix. Fix is now up in the repositories (or apply what I 
attach yourself to avoid waiting for a release).

Thanks to you and Hoyt for reporting this!

-- 
Dag Sverre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: T155.patch
Type: text/x-diff
Size: 1437 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090428/8030bb97/attachment.bin 

From jf.moulin at gmail.com  Tue Apr 28 15:30:53 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 28 Apr 2009 15:30:53 +0200
Subject: [Cython] #155 (compiler message)
In-Reply-To: <49F6FF5C.4000309@student.matnat.uio.no>
References: <af6d35810904280427x15f67f75lc0bad606d4da9dd3@mail.gmail.com>
	<49F6FF5C.4000309@student.matnat.uio.no>
Message-ID: <af6d35810904280630j284cf936g8472110ad7010489@mail.gmail.com>

Thanks for the fast reply... it nevertheless leaves me with a problem ;0)
How do I apply it...?

I tried
jfmoulin at jfm:/usr/lib/python2.5/site-packages/Cython/Compiler$ patch
-p0 </home/jfmoulin/Downloads/T155.patch

and other variations... but it always complains and asks me which file
to patch...
Thanks again
JF

From dagss at student.matnat.uio.no  Tue Apr 28 15:37:11 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 15:37:11 +0200 (CEST)
Subject: [Cython] #155 (compiler message)
In-Reply-To: <af6d35810904280630j284cf936g8472110ad7010489@mail.gmail.com>
References: <af6d35810904280427x15f67f75lc0bad606d4da9dd3@mail.gmail.com>
	<49F6FF5C.4000309@student.matnat.uio.no>
	<af6d35810904280630j284cf936g8472110ad7010489@mail.gmail.com>
Message-ID: <763c8640b02f7554cd9a027a7e455bfc.squirrel@webmail.uio.no>

> Thanks for the fast reply... it nevertheless leaves me with a problem ;0)
> How do I apply it...?
>
> I tried
> jfmoulin at jfm:/usr/lib/python2.5/site-packages/Cython/Compiler$ patch
> -p0 </home/jfmoulin/Downloads/T155.patch

cd ../..
patch -p1 < ...

should work.

If you have hg installed you can also try "hg import thepatch"

If this fails (likely because the development version has drifted too far
from the release) you can either apply it manually by finding the relevant
spot, or download the development trunk (which should be fairly OK now):

Click the download icons next to cython-devel at

http://hg.cython.org/

Dag Sverre


From dalcinl at gmail.com  Tue Apr 28 15:43:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 28 Apr 2009 10:43:46 -0300
Subject: [Cython] patch for #295 uploaded
In-Reply-To: <49F6912C.7060106@behnel.de>
References: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>
	<0011301C-D364-4509-8D11-361197586653@math.washington.edu>
	<49F6912C.7060106@behnel.de>
Message-ID: <e7ba66e40904280643h39178d10nf33fe6cdcbadb63e@mail.gmail.com>

On Tue, Apr 28, 2009 at 2:16 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Robert Bradshaw wrote:
>> On Apr 27, 2009, at 7:09 AM, Lisandro Dalcin wrote:
>>
>>> I've just called the new compiler directive "optimize_simple_methods",
>>> like the former global option. This should be changed to a better name
>>> once a decision is taken. A testcase is also provided.
>>
>> Looks good. Another option for the name would be
>> "always_allow_keywords" because that what the usecase is.
>
> +1 for "always_allow_keywords"
>

Nice, +1 too.

New patch uploaded.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jf.moulin at gmail.com  Tue Apr 28 15:53:07 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 28 Apr 2009 15:53:07 +0200
Subject: [Cython] optimization tips...?
Message-ID: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>

Hi,

I am working on a script which reads rather large amounts of data in a
binary format and then
processes it through different test functions.
I optimized the beast as much as I possibly could: using tuples
instead of lists,
then moving to cython and declaring the types, optimizing the calls to numpy fn
by use of the  buffer notation...

All in all I gain a factor 10 in speed. Not bad but still not really enough...

What I still see as factors slowing me down could be (see my code in attach):
- the use of the file.read() function from python to get a string
which I then process (is an fread call
from c faster... how to implement it?)
- the use of the struct.unpack
- the bit masking technique I use... (is it good or bad)

The above might seem irrelevant but I have millions of events to process...

One more question related to this... I do I profile a cython file (the
info from the
python profiler is no longer split into the different subfunctions...)?


Thanks in advance for your tips,
JF
-------------- next part --------------
A non-text attachment was scrubbed...
Name: next_ev.pyx
Type: application/octet-stream
Size: 1190 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090428/fefbba95/attachment.obj 

From stefan_ml at behnel.de  Tue Apr 28 16:06:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Apr 2009 16:06:41 +0200 (CEST)
Subject: [Cython] optimization tips...?
In-Reply-To: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
References: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
Message-ID: <8ce84119ff3696d1dce26511d74e323b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Hi,

just responding to the last bit for now:

Jean-Francois Moulin wrote:
> One more question related to this... I do I profile a cython file (the
> info from the
> python profiler is no longer split into the different subfunctions...)?

If you're on Linux, use valgrind's callgrind and kcachegrind. It will give
you clickable call graphs with relative timings per branch, and timing
annotated C sources. Perfect for finger pointing.

Stefan


From dagss at student.matnat.uio.no  Tue Apr 28 16:07:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 16:07:50 +0200
Subject: [Cython] optimization tips...?
In-Reply-To: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
References: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
Message-ID: <49F70DB6.5010301@student.matnat.uio.no>

Jean-Francois Moulin wrote:
> Hi,
> 
> I am working on a script which reads rather large amounts of data in a
> binary format and then
> processes it through different test functions.
> I optimized the beast as much as I possibly could: using tuples
> instead of lists,
> then moving to cython and declaring the types, optimizing the calls to numpy fn
> by use of the  buffer notation...
> 
> All in all I gain a factor 10 in speed. Not bad but still not really enough...
> 
> What I still see as factors slowing me down could be (see my code in attach):
> - the use of the file.read() function from python to get a string
> which I then process (is an fread call
> from c faster... how to implement it?)

The real problem is that you read 4 bytes at the time. If you buffer up 
longer stretches somehow it doesn't matter so much which call you use. I.e.:

obj = file.read(400)
cdef char* buf = obj
# hold on to obj, but process buf[0]..buf[399]
buf = NULL
obj = None # do not do this until you no longer use buf

Though if you have a socket rather than a file I suppose you're worse off.

You can use C file handling diretly (the safest thing is to open and 
close the file/socket with C calls as well), just look up Cython 
examples on interfacing with C code and Google for C and file handling.

> - the use of the struct.unpack

As long as you stick to native-endian, you should be able to just cast 
to an int in your case:

cdef char* buf = data
cdef int* buf_as_int = <int*>buf
cdef int value = *buf_as_int

If you need to access more than one int, you can use a struct instead.

> - the bit masking technique I use... (is it good or bad)

For speed it is very fast -- if it has the effect you want there's not 
going to be any faster way.

Consider writing it like this though:

bit30 = data & (1 << 30) != 0

But it is just about readability. (It will be compiled to the same thing.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Apr 28 16:09:52 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 16:09:52 +0200
Subject: [Cython] optimization tips...?
In-Reply-To: <49F70DB6.5010301@student.matnat.uio.no>
References: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
	<49F70DB6.5010301@student.matnat.uio.no>
Message-ID: <49F70E30.9030503@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Jean-Francois Moulin wrote:
>> Hi,
>>
>> I am working on a script which reads rather large amounts of data in a
>> binary format and then
>> processes it through different test functions.
>> I optimized the beast as much as I possibly could: using tuples
>> instead of lists,
>> then moving to cython and declaring the types, optimizing the calls to numpy fn
>> by use of the  buffer notation...
>>
>> All in all I gain a factor 10 in speed. Not bad but still not really enough...
>>
>> What I still see as factors slowing me down could be (see my code in attach):
>> - the use of the file.read() function from python to get a string
>> which I then process (is an fread call
>> from c faster... how to implement it?)
> 
> The real problem is that you read 4 bytes at the time. If you buffer up 
> longer stretches somehow it doesn't matter so much which call you use. I.e.:
> 
> obj = file.read(400)
> cdef char* buf = obj
> # hold on to obj, but process buf[0]..buf[399]
> buf = NULL
> obj = None # do not do this until you no longer use buf
> 
> Though if you have a socket rather than a file I suppose you're worse off.
> 
> You can use C file handling diretly (the safest thing is to open and 
> close the file/socket with C calls as well), just look up Cython 
> examples on interfacing with C code and Google for C and file handling.
> 
>> - the use of the struct.unpack
> 
> As long as you stick to native-endian, you should be able to just cast 
> to an int in your case:
> 
> cdef char* buf = data
> cdef int* buf_as_int = <int*>buf
> cdef int value = *buf_as_int

Argh, this is not Cython :-) (and the irony is we're just having a 
discussion about this on the list)

Do

cdef int value = buf_as_int[0]

instead. Or just value = (<int*>buf)[0]



-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Apr 28 16:11:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Apr 2009 16:11:47 +0200 (CEST)
Subject: [Cython] optimization tips...?
In-Reply-To: <8ce84119ff3696d1dce26511d74e323b.squirrel@groupware.dvs.informatik.tu
	-darmstadt.de>
References: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
	<8ce84119ff3696d1dce26511d74e323b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <bcf417ba28d030ea4b3c8a85de822aed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Stefan Behnel wrote:
> Jean-Francois Moulin wrote:
>> One more question related to this... I do I profile a cython file (the
>> info from the
>> python profiler is no longer split into the different subfunctions...)?
>
> If you're on Linux, use valgrind's callgrind and kcachegrind. It will give
> you clickable call graphs with relative timings per branch, and timing
> annotated C sources. Perfect for finger pointing.

... although, after reading your source, I recommend taking a look at the
generated C source first. It will give you all sorts of ideas what to do
to remove unnecessary Python object creations and to reduce general Python
interaction.

Stefan


From jf.moulin at gmail.com  Tue Apr 28 16:25:10 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 28 Apr 2009 16:25:10 +0200
Subject: [Cython] optimization tips...?
In-Reply-To: <bcf417ba28d030ea4b3c8a85de822aed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
	<8ce84119ff3696d1dce26511d74e323b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<bcf417ba28d030ea4b3c8a85de822aed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <af6d35810904280725w22434aa4udb8055e22c7df22a@mail.gmail.com>

> ... although, after reading your source, I recommend taking a look at the
> generated C source first. It will give you all sorts of ideas what to do
> to remove unnecessary Python object creations and to reduce general Python
> interaction.
>
> Stefan


Yeah... for sure... that means I need to learn C though ;0) That is
exactly to avoid
this that I started playing with cython!
I focussed on a short segment of the script which I thought might be
slow but the
whole thing is around 1000 lines in Python...
I'll give it a look anyway! Thanks for the hint.

JF

From stefan_ml at behnel.de  Tue Apr 28 16:50:26 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Apr 2009 16:50:26 +0200 (CEST)
Subject: [Cython] optimization tips...?
In-Reply-To: <af6d35810904280725w22434aa4udb8055e22c7df22a@mail.gmail.com>
References: <af6d35810904280653o667212e2q14674e9eacefe725@mail.gmail.com>
	<8ce84119ff3696d1dce26511d74e323b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<bcf417ba28d030ea4b3c8a85de822aed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<af6d35810904280725w22434aa4udb8055e22c7df22a@mail.gmail.com>
Message-ID: <4b407087e63f6ba6fc91f71e4a145626.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Jean-Francois Moulin wrote:
> Stefan Behnel wrote:
>> ... although, after reading your source, I recommend taking a look at the
>> generated C source first. It will give you all sorts of ideas what to do
>> to remove unnecessary Python object creations and to reduce general
>> Python interaction.
>
> Yeah... for sure... that means I need to learn C though ;0) That is
> exactly to avoid this that I started playing with cython!
> I focussed on a short segment of the script which I thought might be
> slow but the whole thing is around 1000 lines in Python...
> I'll give it a look anyway! Thanks for the hint.

In that case, try cython's "-a" option. That will generate an annotated
HTML copy of your sources, with dark yellow where things can be improved.

Stefan


From Norbert.Nemec.List at gmx.de  Tue Apr 28 17:00:22 2009
From: Norbert.Nemec.List at gmx.de (Norbert Nemec)
Date: Tue, 28 Apr 2009 17:00:22 +0200
Subject: [Cython] buffer types as members of extension types?
Message-ID: <20090428150022.150360@gmx.net>

Hi there,

currently, the code snippet

-------------
import numpy
cimport numpy
cdef class myclass:
    cdef numpy.ndarray[numpy.float_t] data
-------------

produces the error
    "Buffer types only allowed as function local variables"

Is this a fundamental limitation of Cython or just something that has not yet been implemented?

Is there a workaround that allows me to store and efficiently access buffer data in an extension type?

At the moment, this seems to be the major performance bottleneck in my otherwise straightforwardly cythonized program.

Thanks for any help!

Greetings,
Norbert Nemec
-- 
Neu: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate + Telefonanschluss f?r nur 17,95 Euro/mtl.!* http://dslspecial.gmx.de/freedsl-surfflat/?ac=OM.AD.PD003K11308T4569a

From dagss at student.matnat.uio.no  Tue Apr 28 17:10:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 17:10:40 +0200
Subject: [Cython] buffer types as members of extension types?
In-Reply-To: <20090428150022.150360@gmx.net>
References: <20090428150022.150360@gmx.net>
Message-ID: <49F71C70.8050601@student.matnat.uio.no>

Norbert Nemec wrote:
> Hi there,
> 
> currently, the code snippet
> 
> -------------
> import numpy
> cimport numpy
> cdef class myclass:
>     cdef numpy.ndarray[numpy.float_t] data
> -------------
> 
> produces the error
>     "Buffer types only allowed as function local variables"
> 
> Is this a fundamental limitation of Cython or just something that has not yet been implemented?

It has just not been implemented.

> Is there a workaround that allows me to store and efficiently access buffer data in an extension type?

Is this what you are referring to which is too slow?:

1) Store it as plain numpy.ndarray
2) Copy it to numpy.ndarray[numpy.float_t] inside the local function 
before processing

If that is too slow as well, I'm afraid the only solution right now is 
to store it as numpy.ndarray and access the data field (i.e. 
self.data.data), casted to numpy.float_t*. You must then make sure you 
work with a contiguous array:

     if not data.flags['C_CONTIGUOUS']:
         data = data.copy()
     self.data = data

-- 
Dag Sverre

From mcfletch at vrplumber.com  Tue Apr 28 18:18:24 2009
From: mcfletch at vrplumber.com (Mike C. Fletcher)
Date: Tue, 28 Apr 2009 12:18:24 -0400
Subject: [Cython] A few suggestions for the numpy tutorial
	documentation...
In-Reply-To: <00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>
References: <49F68963.60409@vrplumber.com>
	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>
Message-ID: <49F72C50.50800@vrplumber.com>

Dag Sverre Seljebotn wrote:
>> Hi all,
>>
>> I've spent the day writing a Numpy format handler in Cython and thought
>> there were a few things that the Cython numpy tutorial possibly should
>> cover:
>>     
>
> Are you talking about wiki.cython.org/tutorials/numpy? I think that's
> targeted to a a different audience who don't immedeately feel comfortable
> calling the NumPy C API...
>   
Hmm, good point.
>>     * You have to call import_array() in your module or many of the
>>       Numpy C API entry points will segfault
>>     
>
> Are you sure about this? I never do it. I know you need to do it from C
> code, but is it not enough in Cython to do
>
> import numpy
>
> ?
>
> Did you actually get a segfault using Cython? Could you send me an
> example? I'd really like to make this automatic somehow.
>   
Sure, check out PyOpenGL head:

    bzr branch lp:pyopengl

comment out the import_array() at the end of src/numpy_formathandler.pyx

run:

    python setupaccel.py build_ext --inplace --force

to build the extensions switch to the tests directory and run:

    nosetests -sv test_numpyaccel.py

you'll get a segfault in test_asArray (or at least, I do).

>> Anyway, not sure if or where such changes should go, but just thought it
>> might help others.
>>     
>
> I you feel it fits within the tutorial feel free to add it there,
> otherwise you can always create a sub-page about "Using the NumPy C API"
> or somesuch.
>
> Dag Sverre
>   
Thanks kindly,
Mike

-- 
________________________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://www.vrplumber.com
  http://blog.vrplumber.com


From Norbert.Nemec.list at gmx.de  Tue Apr 28 18:31:37 2009
From: Norbert.Nemec.list at gmx.de (Norbert Nemec)
Date: Tue, 28 Apr 2009 17:31:37 +0100
Subject: [Cython] buffer types as members of extension types?
In-Reply-To: <49F71C70.8050601@student.matnat.uio.no>
References: <20090428150022.150360@gmx.net>
	<49F71C70.8050601@student.matnat.uio.no>
Message-ID: <49F72F69.4020108@gmx.de>

Dag Sverre Seljebotn wrote:
>
>> Is there a workaround that allows me to store and efficiently access buffer data in an extension type?
>>     
>
> Is this what you are referring to which is too slow?:
>
> 1) Store it as plain numpy.ndarray
> 2) Copy it to numpy.ndarray[numpy.float_t] inside the local function 
> before processing
>   
The core routine is called very often, each time, the data is accessed 
just once. Doing the conversion once for every call would therefore not 
gain me anything... :-(

> If that is too slow as well, I'm afraid the only solution right now is 
> to store it as numpy.ndarray and access the data field (i.e. 
> self.data.data), casted to numpy.float_t*. You must then make sure you 
> work with a contiguous array:
>
>      if not data.flags['C_CONTIGUOUS']:
>          data = data.copy()
>      self.data = data
>   
Ok, that's ugly, but it should work. The array is created within the 
class, so I have full control. I could then just store an additional 
class member of pointer type and use that to access the data.

Thanks for the help!


From robertwb at math.washington.edu  Tue Apr 28 20:01:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 28 Apr 2009 11:01:08 -0700
Subject: [Cython] patch for #295 uploaded
In-Reply-To: <e7ba66e40904280643h39178d10nf33fe6cdcbadb63e@mail.gmail.com>
References: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>
	<0011301C-D364-4509-8D11-361197586653@math.washington.edu>
	<49F6912C.7060106@behnel.de>
	<e7ba66e40904280643h39178d10nf33fe6cdcbadb63e@mail.gmail.com>
Message-ID: <FDDDB722-AEF0-433A-B746-3C29A87FD021@math.washington.edu>

On Apr 28, 2009, at 6:43 AM, Lisandro Dalcin wrote:

> On Tue, Apr 28, 2009 at 2:16 AM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>>
>> Robert Bradshaw wrote:
>>> On Apr 27, 2009, at 7:09 AM, Lisandro Dalcin wrote:
>>>
>>>> I've just called the new compiler directive  
>>>> "optimize_simple_methods",
>>>> like the former global option. This should be changed to a  
>>>> better name
>>>> once a decision is taken. A testcase is also provided.
>>>
>>> Looks good. Another option for the name would be
>>> "always_allow_keywords" because that what the usecase is.
>>
>> +1 for "always_allow_keywords"
>>
>
> Nice, +1 too.
>
> New patch uploaded.

Looks good. I updated http://wiki.cython.org/enhancements/ 
compilerdirectives  Please push.

- Robert


From robert.kern at gmail.com  Tue Apr 28 20:01:08 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Apr 2009 13:01:08 -0500
Subject: [Cython] A few suggestions for the numpy tutorial
	documentation...
In-Reply-To: <00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>
References: <49F68963.60409@vrplumber.com>
	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>
Message-ID: <gt7g95$76n$1@ger.gmane.org>

On 2009-04-28 01:05, Dag Sverre Seljebotn wrote:
[Mike Fletcher wrote:]
>>      * You have to call import_array() in your module or many of the
>>        Numpy C API entry points will segfault
>
> Are you sure about this? I never do it. I know you need to do it from C
> code, but is it not enough in Cython to do
>
> import numpy
>
> ?

If you call any of the PyArray_* functions, you will need an import_array() 
call. That does not just import numpy; it sets up the function pointer table in 
your extension module such that the PyArray_* #defines reference the right 
functions.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robertwb at math.washington.edu  Tue Apr 28 20:04:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 28 Apr 2009 11:04:59 -0700
Subject: [Cython] A few suggestions for the numpy tutorial
	documentation...
In-Reply-To: <gt7g95$76n$1@ger.gmane.org>
References: <49F68963.60409@vrplumber.com>
	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>
	<gt7g95$76n$1@ger.gmane.org>
Message-ID: <30A35F95-E6E6-4113-98F3-ADEB6443D4FC@math.washington.edu>

On Apr 28, 2009, at 11:01 AM, Robert Kern wrote:

> On 2009-04-28 01:05, Dag Sverre Seljebotn wrote:
> [Mike Fletcher wrote:]
>>>      * You have to call import_array() in your module or many of the
>>>        Numpy C API entry points will segfault
>>
>> Are you sure about this? I never do it. I know you need to do it  
>> from C
>> code, but is it not enough in Cython to do
>>
>> import numpy
>>
>> ?
>
> If you call any of the PyArray_* functions, you will need an  
> import_array()
> call. That does not just import numpy; it sets up the function  
> pointer table in
> your extension module such that the PyArray_* #defines reference  
> the right
> functions.

So "import numpy" doesn't call import_array() then?

- Robert



From dagss at student.matnat.uio.no  Tue Apr 28 20:10:38 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 20:10:38 +0200
Subject: [Cython] A few suggestions for the numpy
	tutorial	documentation...
In-Reply-To: <30A35F95-E6E6-4113-98F3-ADEB6443D4FC@math.washington.edu>
References: <49F68963.60409@vrplumber.com>	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>	<gt7g95$76n$1@ger.gmane.org>
	<30A35F95-E6E6-4113-98F3-ADEB6443D4FC@math.washington.edu>
Message-ID: <49F7469E.8010509@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 28, 2009, at 11:01 AM, Robert Kern wrote:
> 
>> On 2009-04-28 01:05, Dag Sverre Seljebotn wrote:
>> [Mike Fletcher wrote:]
>>>>      * You have to call import_array() in your module or many of the
>>>>        Numpy C API entry points will segfault
>>> Are you sure about this? I never do it. I know you need to do it  
>>> from C
>>> code, but is it not enough in Cython to do
>>>
>>> import numpy
>>>
>>> ?
>> If you call any of the PyArray_* functions, you will need an  
>> import_array()
>> call. That does not just import numpy; it sets up the function  
>> pointer table in
>> your extension module such that the PyArray_* #defines reference  
>> the right
>> functions.
> 
> So "import numpy" doesn't call import_array() then?

Apparently a plain import it does what is needed for normal/trivial 
usage of NumPy/Cython, but if you in addition need to call low-level 
PyArray_*-functions then you need the call.

This is fairly well documented in the NumPy API, and my tutorial doesn't 
mention PyArray_*-functions, so I consider the issue OK. (Though we 
could certainly have additional documentation on the wiki for using the 
lower-level NumPy API from Cython).

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Apr 28 22:39:45 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Apr 2009 22:39:45 +0200
Subject: [Cython] Path towards Python completeness
In-Reply-To: <2038f7c55d17516c79c8b3b2a9385fb1.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49ED5CE8.6090708@behnel.de>	<80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>	<380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<BFD20AC6-2309-4634-98FB-D30BB66A6B2F@math.washington.edu>
	<2038f7c55d17516c79c8b3b2a9385fb1.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49F76991.5010303@behnel.de>


Stefan Behnel wrote:
> Robert Bradshaw wrote:
>> http://hg.cython.org/cython-closures/file/c4ee199f08a2/tests/run/closures_T82.pyx
>>
>> Before you get your hopes up too much, I want to add a big
>> disclaimer that though I tried to be careful, this is very first-
>> draft code and there are bound to be issues (e.g. I think I've run
>> into some with respect to Python arguments).
> 
> I'll give it a try during the next days.

I exercised it a bit and added more test cases. That revealed a couple of
problems, although I must say that it works pretty well in general.

I added comments to the tests as far as I got with analysing them. One
segfaults pretty badly, which makes it harder to figure out.

Stefan

From robert.kern at gmail.com  Tue Apr 28 22:53:38 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Apr 2009 15:53:38 -0500
Subject: [Cython] A few suggestions for the numpy tutorial
	documentation...
In-Reply-To: <49F7469E.8010509@student.matnat.uio.no>
References: <49F68963.60409@vrplumber.com>	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>	<gt7g95$76n$1@ger.gmane.org>	<30A35F95-E6E6-4113-98F3-ADEB6443D4FC@math.washington.edu>
	<49F7469E.8010509@student.matnat.uio.no>
Message-ID: <gt7qcj$bfn$1@ger.gmane.org>

On 2009-04-28 13:10, Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> On Apr 28, 2009, at 11:01 AM, Robert Kern wrote:
>>
>>> On 2009-04-28 01:05, Dag Sverre Seljebotn wrote:
>>> [Mike Fletcher wrote:]
>>>>>       * You have to call import_array() in your module or many of the
>>>>>         Numpy C API entry points will segfault
>>>> Are you sure about this? I never do it. I know you need to do it
>>>> from C
>>>> code, but is it not enough in Cython to do
>>>>
>>>> import numpy
>>>>
>>>> ?
>>> If you call any of the PyArray_* functions, you will need an
>>> import_array()
>>> call. That does not just import numpy; it sets up the function
>>> pointer table in
>>> your extension module such that the PyArray_* #defines reference
>>> the right
>>> functions.
>> So "import numpy" doesn't call import_array() then?

No, "import numpy" just imports numpy just as in Python. Each numpy-using 
extension module has a variable local to it that points to the global table of 
function pointers. import_array() must be called once per extension module in 
order to make that module-local variable point to the correct location.

> Apparently a plain import it does what is needed for normal/trivial
> usage of NumPy/Cython, but if you in addition need to call low-level
> PyArray_*-functions then you need the call.
>
> This is fairly well documented in the NumPy API, and my tutorial doesn't
> mention PyArray_*-functions, so I consider the issue OK. (Though we
> could certainly have additional documentation on the wiki for using the
> lower-level NumPy API from Cython).

It would be wise to include it in the tutorial with a comment about when it is 
necessary. People have a wide variety of entry points into the material. They 
may start with your tutorial and just need to use a few API functions. They may 
very well skip the parts in the numpy docs about setting up an extension module 
thinking that Cython took care of everything for them.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From dagss at student.matnat.uio.no  Tue Apr 28 23:07:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 28 Apr 2009 23:07:08 +0200
Subject: [Cython] A few suggestions for the numpy
	tutorial	documentation...
In-Reply-To: <gt7qcj$bfn$1@ger.gmane.org>
References: <49F68963.60409@vrplumber.com>	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>	<gt7g95$76n$1@ger.gmane.org>	<30A35F95-E6E6-4113-98F3-ADEB6443D4FC@math.washington.edu>	<49F7469E.8010509@student.matnat.uio.no>
	<gt7qcj$bfn$1@ger.gmane.org>
Message-ID: <49F76FFC.3040304@student.matnat.uio.no>

Robert Kern wrote:
> On 2009-04-28 13:10, Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> On Apr 28, 2009, at 11:01 AM, Robert Kern wrote:
>>>
>>>> On 2009-04-28 01:05, Dag Sverre Seljebotn wrote:
>>>> [Mike Fletcher wrote:]
>>>>>>       * You have to call import_array() in your module or many of the
>>>>>>         Numpy C API entry points will segfault
>>>>> Are you sure about this? I never do it. I know you need to do it
>>>>> from C
>>>>> code, but is it not enough in Cython to do
>>>>>
>>>>> import numpy
>>>>>
>>>>> ?
>>>> If you call any of the PyArray_* functions, you will need an
>>>> import_array()
>>>> call. That does not just import numpy; it sets up the function
>>>> pointer table in
>>>> your extension module such that the PyArray_* #defines reference
>>>> the right
>>>> functions.
>>> So "import numpy" doesn't call import_array() then?
> 
> No, "import numpy" just imports numpy just as in Python. Each numpy-using 
> extension module has a variable local to it that points to the global table of 
> function pointers. import_array() must be called once per extension module in 
> order to make that module-local variable point to the correct location.

Ahh right. Just as intra-Cython-module calls with __pyx_capi :-)

> It would be wise to include it in the tutorial with a comment about when it is 
> necessary. People have a wide variety of entry points into the material. They 
> may start with your tutorial and just need to use a few API functions. They may 
> very well skip the parts in the numpy docs about setting up an extension module 
> thinking that Cython took care of everything for them.

Sounds reasonable to include a C API section prominently there, though I 
won't have time myself (nor do I know that much about the NumPy C API, 
to be honest, all I ever use is the ndarray and dtype struct & macros).

-- 
Dag Sverre

From dalcinl at gmail.com  Tue Apr 28 23:07:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 28 Apr 2009 18:07:37 -0300
Subject: [Cython] patch for #295 uploaded
In-Reply-To: <FDDDB722-AEF0-433A-B746-3C29A87FD021@math.washington.edu>
References: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>
	<0011301C-D364-4509-8D11-361197586653@math.washington.edu>
	<49F6912C.7060106@behnel.de>
	<e7ba66e40904280643h39178d10nf33fe6cdcbadb63e@mail.gmail.com>
	<FDDDB722-AEF0-433A-B746-3C29A87FD021@math.washington.edu>
Message-ID: <e7ba66e40904281407g51e7f766w114af2705f590ced@mail.gmail.com>

On Tue, Apr 28, 2009 at 3:01 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> Looks good. I updated http://wiki.cython.org/enhancements/
> compilerdirectives ?Please push.
>

Pushed to cython-devel. All test pass. Please close the ticket, my
track account does not have enough rights to do that.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robert.kern at gmail.com  Tue Apr 28 23:12:05 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Apr 2009 16:12:05 -0500
Subject: [Cython] A few suggestions for the numpy tutorial
	documentation...
In-Reply-To: <49F76FFC.3040304@student.matnat.uio.no>
References: <49F68963.60409@vrplumber.com>	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>	<gt7g95$76n$1@ger.gmane.org>	<30A35F95-E6E6-4113-98F3-ADEB6443D4FC@math.washington.edu>	<49F7469E.8010509@student.matnat.uio.no>	<gt7qcj$bfn$1@ger.gmane.org>
	<49F76FFC.3040304@student.matnat.uio.no>
Message-ID: <gt7rf5$evk$1@ger.gmane.org>

On 2009-04-28 16:07, Dag Sverre Seljebotn wrote:
> Sounds reasonable to include a C API section prominently there, though I
> won't have time myself (nor do I know that much about the NumPy C API,
> to be honest, all I ever use is the ndarray and dtype struct&  macros).

Just including the import_array() call in the examples with a small comment 
would go a long way.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robertwb at math.washington.edu  Tue Apr 28 23:14:16 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 28 Apr 2009 14:14:16 -0700
Subject: [Cython] patch for #295 uploaded
In-Reply-To: <e7ba66e40904281407g51e7f766w114af2705f590ced@mail.gmail.com>
References: <e7ba66e40904270709k6e7105f5p63e054e7f55ee921@mail.gmail.com>
	<0011301C-D364-4509-8D11-361197586653@math.washington.edu>
	<49F6912C.7060106@behnel.de>
	<e7ba66e40904280643h39178d10nf33fe6cdcbadb63e@mail.gmail.com>
	<FDDDB722-AEF0-433A-B746-3C29A87FD021@math.washington.edu>
	<e7ba66e40904281407g51e7f766w114af2705f590ced@mail.gmail.com>
Message-ID: <14D0127D-7788-4D97-84C9-EE29AD4E9FB8@math.washington.edu>

On Apr 28, 2009, at 2:07 PM, Lisandro Dalcin wrote:

> On Tue, Apr 28, 2009 at 3:01 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> Looks good. I updated http://wiki.cython.org/enhancements/
>> compilerdirectives  Please push.
>>
>
> Pushed to cython-devel. All test pass. Please close the ticket,

Done.

> my track account does not have enough rights to do that.

Sorry. You do now.

- Robert


From mcfletch at vrplumber.com  Wed Apr 29 02:49:06 2009
From: mcfletch at vrplumber.com (Mike C. Fletcher)
Date: Tue, 28 Apr 2009 20:49:06 -0400
Subject: [Cython] A few suggestions for the numpy
	tutorial	documentation...
In-Reply-To: <gt7rf5$evk$1@ger.gmane.org>
References: <49F68963.60409@vrplumber.com>	<00850f9052a7dc3579ba6c0e446d3577.squirrel@webmail.uio.no>	<gt7g95$76n$1@ger.gmane.org>	<30A35F95-E6E6-4113-98F3-ADEB6443D4FC@math.washington.edu>	<49F7469E.8010509@student.matnat.uio.no>	<gt7qcj$bfn$1@ger.gmane.org>	<49F76FFC.3040304@student.matnat.uio.no>
	<gt7rf5$evk$1@ger.gmane.org>
Message-ID: <49F7A402.5020507@vrplumber.com>

Robert Kern wrote:
> On 2009-04-28 16:07, Dag Sverre Seljebotn wrote:
>   
>> Sounds reasonable to include a C API section prominently there, though I
>> won't have time myself (nor do I know that much about the NumPy C API,
>> to be honest, all I ever use is the ndarray and dtype struct&  macros).
>>     
>
> Just including the import_array() call in the examples with a small comment 
> would go a long way.
>   
I've added an initial section on this stuff here:

    http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI

it's a pretty short section, doesn't really seem like it needs a whole
new page.

Enjoy,
Mike

-- 
________________________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://www.vrplumber.com
  http://blog.vrplumber.com


From Norbert.Nemec.List at gmx.de  Wed Apr 29 15:38:01 2009
From: Norbert.Nemec.List at gmx.de (Norbert Nemec)
Date: Wed, 29 Apr 2009 15:38:01 +0200
Subject: [Cython] buffer&numpy bugs&shortcomings
Message-ID: <20090429133801.232270@gmx.net>

Hi there,

thanks for the great work on Cython! It has already helped me to a significant speedup on my code.

Working to port parts of my numpy-based program to Cython, I stumbled over a number of bugs/shortcomings in the implementation of buffer types and the numpy-cython interface that I would like to report in the following. For every snippet, assume a header of

--------
    import numpy
    cimport numpy
--------

==============
1. (reported before) extension types cannot have buffer types as elements. The snippet

--------
cdef class myclass:
    cdef numpy.ndarray[numpy.float_t] data
--------

gives the cython-compiler error "Buffer types only allowed as function local variables". I can work around this by keeping a pointer to the data as an additional member, but that results in a tremendous uglification of the whole code...

==============
2. C functions cannot have buffer types as arguments. The snippet

--------
cdef myfunc(numpy.ndarray[numpy.float] arg):
    pass
--------

produces the cython *syntax error*

cdef myfunc(numpy.ndarray[numpy.float] arg):
                                      ^
------------------------------------------------------------
/home/nobbi/NNlab/src/cython-bugs/tryout2.pyx:6:39: Expected ')'

The same thing works fine with def instead of cdef.

==============
3. On typed variable of type numpy.ndarray, the .shape member does not work

---------
cdef numpy.ndarray var = numpy.zeros((3,))
print var.shape
---------

gives the cython compiler error "Cannot convert 'numpy.npy_intp *' to Python object"

==============
4. The numpy.float_t type is not handled correctly. The snippet

------------
cdef class myclass:
    cdef readonly:
        double var_d
        numpy.float_t var_nf
    def __init__(self):
        self.var_d = 42.0
        self.var_nf = 42.0

c = myclass()

print "double (correctly given as 42.0):",c.var_d
print "numpy.float_t (should be 42.0):",c.var_nf
------------

compiles fine, but prints

    double (correctly given as 42.0): 42.0
    numpy.float_t (should be 42.0): 0.0

so the value of var_nf seems to be lost somewhere on the way.

===============

I am not sure which of these are simply bugs, which are missing features and which might be misunderstandings on my side.

Thanks for any help on this!

Greetings,
Norbert
-- 
Neu: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate + Telefonanschluss f?r nur 17,95 Euro/mtl.!* http://dslspecial.gmx.de/freedsl-surfflat/?ac=OM.AD.PD003K11308T4569a

From dagss at student.matnat.uio.no  Wed Apr 29 15:50:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 29 Apr 2009 15:50:58 +0200
Subject: [Cython] buffer&numpy bugs&shortcomings
In-Reply-To: <20090429133801.232270@gmx.net>
References: <20090429133801.232270@gmx.net>
Message-ID: <49F85B42.8060505@student.matnat.uio.no>

Norbert Nemec wrote:
> Hi there,
> 
> thanks for the great work on Cython! It has already helped me to a significant speedup on my code.
> 
> Working to port parts of my numpy-based program to Cython, I stumbled over a number of bugs/shortcomings in the implementation of buffer types and the numpy-cython interface that I would like to report in the following. For every snippet, assume a header of

Thanks. All in all, we know that there are major shortcomings, the 
problem is simply having resources to pull it off. Ask me if you want me 
to e.g. mentor you for doing a patch or similar.

Still I'll take care to track these problems; see below for individual 
resopnse.

> 
> --------
>     import numpy
>     cimport numpy
> --------
> 
> ==============
> 1. (reported before) extension types cannot have buffer types as elements. The snippet
> 
> --------
> cdef class myclass:
>     cdef numpy.ndarray[numpy.float_t] data
> --------
> 
> gives the cython-compiler error "Buffer types only allowed as function local variables". I can work around this by keeping a pointer to the data as an additional member, but that results in a tremendous uglification of the whole code...

This is now http://trac.cython.org/cython_trac/ticket/301

I don't know when/whether it will be done, but you can try asking again 
after summer when it will likely be easier for me to add it.

> ==============
> 2. C functions cannot have buffer types as arguments. The snippet
> 
> --------
> cdef myfunc(numpy.ndarray[numpy.float] arg):
>     pass
> --------
> 
> produces the cython *syntax error*
> 
> cdef myfunc(numpy.ndarray[numpy.float] arg):
>                                       ^
> ------------------------------------------------------------
> /home/nobbi/NNlab/src/cython-bugs/tryout2.pyx:6:39: Expected ')'
> 
> The same thing works fine with def instead of cdef.

This is http://trac.cython.org/cython_trac/ticket/177

There's a good chance it will be fixed by Kurt Smith in his GSoC over 
summer.

> ==============
> 3. On typed variable of type numpy.ndarray, the .shape member does not work
> 
> ---------
> cdef numpy.ndarray var = numpy.zeros((3,))
> print var.shape
> ---------
> 
> gives the cython compiler error "Cannot convert 'numpy.npy_intp *' to Python object"

Can you file a ticket for this?

(To get access to trac, email an .htpasswd-file to Robert Bradshaw)

This is mostly a question of semantics. shape is defined to be a C 
integer array for speed of var.shape[0]. How can we define a generic 
rule in Cython which makes both possible?

Should the behaviour of "var.shape" e.g. depend on the requested return 
value? This means overloading on return value which is a non-trivial 
language change, though perhaps appropriate here.

Thoughts welcome!


> ==============
> 4. The numpy.float_t type is not handled correctly. The snippet
> 
> ------------
> cdef class myclass:
>     cdef readonly:
>         double var_d
>         numpy.float_t var_nf
>     def __init__(self):
>         self.var_d = 42.0
>         self.var_nf = 42.0
> 
> c = myclass()
> 
> print "double (correctly given as 42.0):",c.var_d
> print "numpy.float_t (should be 42.0):",c.var_nf
> ------------
> 
> compiles fine, but prints
> 
>     double (correctly given as 42.0): 42.0
>     numpy.float_t (should be 42.0): 0.0
> 
> so the value of var_nf seems to be lost somewhere on the way.

This is a (major) bug that I was completely unaware of, and it should be 
fixed soonish. Thanks a lot!

Will you file a ticket for it?

Dag Sverre

From Norbert.Nemec.List at gmx.de  Wed Apr 29 16:47:25 2009
From: Norbert.Nemec.List at gmx.de (Norbert Nemec)
Date: Wed, 29 Apr 2009 16:47:25 +0200
Subject: [Cython] buffer&numpy bugs&shortcomings
In-Reply-To: <49F85B42.8060505@student.matnat.uio.no>
References: <20090429133801.232270@gmx.net>
	<49F85B42.8060505@student.matnat.uio.no>
Message-ID: <20090429144725.316110@gmx.net>

Thanks, Dan, for the comments on the bugs I reported. I just requested a trac account from Robert to produce proper tickets for the latter two.

Good to know that two of the issues will be worked on over the summer. I should have checked the trac tickets more thoroughly...

In fact, I know the problem of limited resources just too well. Just amazing what you guys have produced already. I would certainly like to join in with more than just bug reports, but for the moment, that is all I can offer. :-(

Greetings,
Norbert




-------- Original-Nachricht --------
> Datum: Wed, 29 Apr 2009 15:50:58 +0200
> Von: Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
> An: cython-dev at codespeak.net
> Betreff: Re: [Cython] buffer&numpy bugs&shortcomings

> Norbert Nemec wrote:
> > Hi there,
> > 
> > thanks for the great work on Cython! It has already helped me to a
> significant speedup on my code.
> > 
> > Working to port parts of my numpy-based program to Cython, I stumbled
> over a number of bugs/shortcomings in the implementation of buffer types and
> the numpy-cython interface that I would like to report in the following.
> For every snippet, assume a header of
> 
> Thanks. All in all, we know that there are major shortcomings, the 
> problem is simply having resources to pull it off. Ask me if you want me 
> to e.g. mentor you for doing a patch or similar.
> 
> Still I'll take care to track these problems; see below for individual 
> resopnse.
> 
> > 
> > --------
> >     import numpy
> >     cimport numpy
> > --------
> > 
> > ==============
> > 1. (reported before) extension types cannot have buffer types as
> elements. The snippet
> > 
> > --------
> > cdef class myclass:
> >     cdef numpy.ndarray[numpy.float_t] data
> > --------
> > 
> > gives the cython-compiler error "Buffer types only allowed as function
> local variables". I can work around this by keeping a pointer to the data as
> an additional member, but that results in a tremendous uglification of the
> whole code...
> 
> This is now http://trac.cython.org/cython_trac/ticket/301
> 
> I don't know when/whether it will be done, but you can try asking again 
> after summer when it will likely be easier for me to add it.
> 
> > ==============
> > 2. C functions cannot have buffer types as arguments. The snippet
> > 
> > --------
> > cdef myfunc(numpy.ndarray[numpy.float] arg):
> >     pass
> > --------
> > 
> > produces the cython *syntax error*
> > 
> > cdef myfunc(numpy.ndarray[numpy.float] arg):
> >                                       ^
> > ------------------------------------------------------------
> > /home/nobbi/NNlab/src/cython-bugs/tryout2.pyx:6:39: Expected ')'
> > 
> > The same thing works fine with def instead of cdef.
> 
> This is http://trac.cython.org/cython_trac/ticket/177
> 
> There's a good chance it will be fixed by Kurt Smith in his GSoC over 
> summer.
> 
> > ==============
> > 3. On typed variable of type numpy.ndarray, the .shape member does not
> work
> > 
> > ---------
> > cdef numpy.ndarray var = numpy.zeros((3,))
> > print var.shape
> > ---------
> > 
> > gives the cython compiler error "Cannot convert 'numpy.npy_intp *' to
> Python object"
> 
> Can you file a ticket for this?
> 
> (To get access to trac, email an .htpasswd-file to Robert Bradshaw)
> 
> This is mostly a question of semantics. shape is defined to be a C 
> integer array for speed of var.shape[0]. How can we define a generic 
> rule in Cython which makes both possible?
> 
> Should the behaviour of "var.shape" e.g. depend on the requested return 
> value? This means overloading on return value which is a non-trivial 
> language change, though perhaps appropriate here.
> 
> Thoughts welcome!
> 
> 
> > ==============
> > 4. The numpy.float_t type is not handled correctly. The snippet
> > 
> > ------------
> > cdef class myclass:
> >     cdef readonly:
> >         double var_d
> >         numpy.float_t var_nf
> >     def __init__(self):
> >         self.var_d = 42.0
> >         self.var_nf = 42.0
> > 
> > c = myclass()
> > 
> > print "double (correctly given as 42.0):",c.var_d
> > print "numpy.float_t (should be 42.0):",c.var_nf
> > ------------
> > 
> > compiles fine, but prints
> > 
> >     double (correctly given as 42.0): 42.0
> >     numpy.float_t (should be 42.0): 0.0
> > 
> > so the value of var_nf seems to be lost somewhere on the way.
> 
> This is a (major) bug that I was completely unaware of, and it should be 
> fixed soonish. Thanks a lot!
> 
> Will you file a ticket for it?
> 
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

-- 
Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger01

From jack at yelp.com  Wed Apr 29 18:46:59 2009
From: jack at yelp.com (Jack Stahl)
Date: Wed, 29 Apr 2009 09:46:59 -0700
Subject: [Cython] char* to string conversion
Message-ID: <239f2f640904290946m3954e140l4555b3f3dd97e5ba@mail.gmail.com>

Quick question.  If I want to return a char * from Cython to Python as a
string, should I be freeing the original char * or not?  In other words,
when Cython does the implicit conversion from char * ---> string, does it
create a Python object and strcpy the char * into its data, or does it copy
the pointer?

Here's some code to make this clearer:

cdef class MyStringHolder:

  cdef char *my_str

  ...

  cpdef add_to_set(object set):
    // Is this necessary or is this a memory leak?
    cdef char * str_cpy = <char*>malloc(strlen(my_str) + 1)
    strcpy(str_cpy, my_str)

    // Should this just be set.add(my_str)?
    set.add(str_cpy)


Thanks!
jack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090429/9550f432/attachment.htm 

From stefan_ml at behnel.de  Wed Apr 29 18:55:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Apr 2009 18:55:27 +0200
Subject: [Cython] char* to string conversion
In-Reply-To: <239f2f640904290946m3954e140l4555b3f3dd97e5ba@mail.gmail.com>
References: <239f2f640904290946m3954e140l4555b3f3dd97e5ba@mail.gmail.com>
Message-ID: <49F8867F.8090905@behnel.de>


Jack Stahl wrote:
> Quick question.  If I want to return a char * from Cython to Python as a
> string, should I be freeing the original char * or not?  In other words,
> when Cython does the implicit conversion from char * ---> string, does it
> create a Python object and strcpy the char * into its data, or does it copy
> the pointer?

It creates a copy that is owned by the str/bytes object.


> Here's some code to make this clearer:
> 
> cdef class MyStringHolder:
> 
>   cdef char *my_str
> 
>   ...
> 
>   cpdef add_to_set(object set):
>     // Is this necessary or is this a memory leak?
>     cdef char * str_cpy = <char*>malloc(strlen(my_str) + 1)
>     strcpy(str_cpy, my_str)
> 
>     // Should this just be set.add(my_str)?
>     set.add(str_cpy)

Yes, passing my_str should be enough (unless what you want is actually a
unicode string, in which case you need to decode it).

Stefan


From jack at yelp.com  Wed Apr 29 19:00:48 2009
From: jack at yelp.com (Jack Stahl)
Date: Wed, 29 Apr 2009 10:00:48 -0700
Subject: [Cython] char* to string conversion
In-Reply-To: <49F8867F.8090905@behnel.de>
References: <239f2f640904290946m3954e140l4555b3f3dd97e5ba@mail.gmail.com>
	<49F8867F.8090905@behnel.de>
Message-ID: <239f2f640904291000x48fadfc9v32929c334c08777d@mail.gmail.com>

Thanks Stefan!  That's what I suspected.  The fast reply is much appreciated
(as is Cython in general :))

On Wed, Apr 29, 2009 at 9:55 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

>
> Jack Stahl wrote:
> > Quick question.  If I want to return a char * from Cython to Python as a
> > string, should I be freeing the original char * or not?  In other words,
> > when Cython does the implicit conversion from char * ---> string, does it
> > create a Python object and strcpy the char * into its data, or does it
> copy
> > the pointer?
>
> It creates a copy that is owned by the str/bytes object.
>
>
> > Here's some code to make this clearer:
> >
> > cdef class MyStringHolder:
> >
> >   cdef char *my_str
> >
> >   ...
> >
> >   cpdef add_to_set(object set):
> >     // Is this necessary or is this a memory leak?
> >     cdef char * str_cpy = <char*>malloc(strlen(my_str) + 1)
> >     strcpy(str_cpy, my_str)
> >
> >     // Should this just be set.add(my_str)?
> >     set.add(str_cpy)
>
> Yes, passing my_str should be enough (unless what you want is actually a
> unicode string, in which case you need to decode it).
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090429/071b99af/attachment.htm 

From ondrej at certik.cz  Wed Apr 29 20:46:12 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 29 Apr 2009 11:46:12 -0700
Subject: [Cython] hg repo doesn't seem to work?
Message-ID: <85b5c3130904291146v1395cd1en81ec1415da1b9c58@mail.gmail.com>

Hi,

I did:

$ hg clone http://hg.cython.org/cython/
destination directory: cython
requesting all changes
adding changesets
adding manifests
adding file changes
transaction abort!
rollback completed
abort: 00changelog.i at 696e20737973: no node!



am I doing something wrong?

Ondrej

From ondrej at certik.cz  Wed Apr 29 20:55:11 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 29 Apr 2009 11:55:11 -0700
Subject: [Cython] problem with pure python mode
Message-ID: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>

Hi,

I am having a problem getting the pure mode run (I think this used to
work before). Here is my cx.pyx file:

-------------
import cython

from numpy import array

#@cython.locals(x=cython.int)
def f(x):
    r = cython.declare(cython.int)
    i = cython.declare(cython.int)
    if x > 0:
        r = 1
        for i in range(1, x):
            r *= i
        return r
    else:
        return 0

def f_vectorized(a):
    return array([f(x) for x in a])
----------

and here is my cx.pxd file:
-----
cdef int f(int x)
-------

If I compile it:

$ cython cx.pyx
warning: /home/ondrej/repos/cython_test/cx.pyx:6:0: Overriding cdef
method with def method.


it doesn't build:

gcc -fPIC -O3 -I/usr/include/python2.6 -c -o cx.o cx.c
cx.c:196: warning: ?__pyx_f_2cx_f? used but never defined
gcc -fPIC -O3 -shared -o cx.so cx.o
/usr/bin/ld: cx.o: relocation R_X86_64_PC32 against undefined symbol
`__pyx_f_2cx_f' can not be used when making a shared object; recompile
with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit status
make: *** [cx.so] Error 1


If I delete the pxd file, it works fine (but it's not a cdef function,
so it's slow).

In the wiki:

http://wiki.cython.org/pure

I can read:

Decorators

We have settled on @cython.cclass for the cdef class decorators, and
@cython.cfunc and @cython.ccall for cdef and cpdef functions
(respectively).
http://codespeak.net/pipermail/cython-dev/2008-November/002925.html



But the cfunc doesn't seem to be defined in the cython module.

So what is the way to create a pure python mode cdef functions?


I think I did something stupid above, I am trying to figure it out.

Thanks,
Ondrej

From robertwb at math.washington.edu  Wed Apr 29 21:33:13 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 29 Apr 2009 12:33:13 -0700
Subject: [Cython] hg repo doesn't seem to work?
In-Reply-To: <85b5c3130904291146v1395cd1en81ec1415da1b9c58@mail.gmail.com>
References: <85b5c3130904291146v1395cd1en81ec1415da1b9c58@mail.gmail.com>
Message-ID: <CEF4160A-90F3-4ADC-9509-584F38AF3DAE@math.washington.edu>

On Apr 29, 2009, at 11:46 AM, Ondrej Certik wrote:

> Hi,
>
> I did:
>
> $ hg clone http://hg.cython.org/cython/
> destination directory: cython
> requesting all changes
> adding changesets
> adding manifests
> adding file changes
> transaction abort!
> rollback completed
> abort: 00changelog.i at 696e20737973: no node!
>
> am I doing something wrong?

Hmm... I'm getting that too. I can't look into this now, but will try  
to do so later. You can clone from the others right now.

- Robert


From robertwb at math.washington.edu  Wed Apr 29 21:37:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 29 Apr 2009 12:37:17 -0700
Subject: [Cython] problem with pure python mode
In-Reply-To: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
Message-ID: <B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>

On Apr 29, 2009, at 11:55 AM, Ondrej Certik wrote:

> Hi,
>
> I am having a problem getting the pure mode run (I think this used to
> work before). Here is my cx.pyx file:
>
> -------------
> import cython
>
> from numpy import array
>
> #@cython.locals(x=cython.int)
> def f(x):
>     r = cython.declare(cython.int)
>     i = cython.declare(cython.int)
>     if x > 0:
>         r = 1
>         for i in range(1, x):
>             r *= i
>         return r
>     else:
>         return 0
>
> def f_vectorized(a):
>     return array([f(x) for x in a])
> ----------
>
> and here is my cx.pxd file:
> -----
> cdef int f(int x)
> -------
>
> If I compile it:
>
> $ cython cx.pyx
> warning: /home/ondrej/repos/cython_test/cx.pyx:6:0: Overriding cdef
> method with def method.
>
>
> it doesn't build:
>
> gcc -fPIC -O3 -I/usr/include/python2.6 -c -o cx.o cx.c
> cx.c:196: warning: ?__pyx_f_2cx_f? used but never defined
> gcc -fPIC -O3 -shared -o cx.so cx.o
> /usr/bin/ld: cx.o: relocation R_X86_64_PC32 against undefined symbol
> `__pyx_f_2cx_f' can not be used when making a shared object; recompile
> with -fPIC
> /usr/bin/ld: final link failed: Bad value
> collect2: ld returned 1 exit status
> make: *** [cx.so] Error 1
>
>
> If I delete the pxd file, it works fine (but it's not a cdef function,
> so it's slow).

Not sure--have you tried without the locals decorator in the Python  
source? I hope this isn't a regression.

>
> In the wiki:
>
> http://wiki.cython.org/pure
>
> I can read:
>
> Decorators
>
> We have settled on @cython.cclass for the cdef class decorators, and
> @cython.cfunc and @cython.ccall for cdef and cpdef functions
> (respectively).
> http://codespeak.net/pipermail/cython-dev/2008-November/002925.html
>
>
> But the cfunc doesn't seem to be defined in the cython module.
>
> So what is the way to create a pure python mode cdef functions?

"we have settled on" != it has been implemented, so the .pxd file is  
the only way to do it for now.

- Robert


From ondrej at certik.cz  Wed Apr 29 22:19:19 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 29 Apr 2009 13:19:19 -0700
Subject: [Cython] problem with pure python mode
In-Reply-To: <B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
Message-ID: <85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>

On Wed, Apr 29, 2009 at 12:37 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Apr 29, 2009, at 11:55 AM, Ondrej Certik wrote:
>
>> Hi,
>>
>> I am having a problem getting the pure mode run (I think this used to
>> work before). Here is my cx.pyx file:
>>
>> -------------
>> import cython
>>
>> from numpy import array
>>
>> #@cython.locals(x=cython.int)
>> def f(x):
>> ? ? r = cython.declare(cython.int)
>> ? ? i = cython.declare(cython.int)
>> ? ? if x > 0:
>> ? ? ? ? r = 1
>> ? ? ? ? for i in range(1, x):
>> ? ? ? ? ? ? r *= i
>> ? ? ? ? return r
>> ? ? else:
>> ? ? ? ? return 0
>>
>> def f_vectorized(a):
>> ? ? return array([f(x) for x in a])
>> ----------
>>
>> and here is my cx.pxd file:
>> -----
>> cdef int f(int x)
>> -------
>>
>> If I compile it:
>>
>> $ cython cx.pyx
>> warning: /home/ondrej/repos/cython_test/cx.pyx:6:0: Overriding cdef
>> method with def method.
>>
>>
>> it doesn't build:
>>
>> gcc -fPIC -O3 -I/usr/include/python2.6 -c -o cx.o cx.c
>> cx.c:196: warning: ?__pyx_f_2cx_f? used but never defined
>> gcc -fPIC -O3 -shared -o cx.so cx.o
>> /usr/bin/ld: cx.o: relocation R_X86_64_PC32 against undefined symbol
>> `__pyx_f_2cx_f' can not be used when making a shared object; recompile
>> with -fPIC
>> /usr/bin/ld: final link failed: Bad value
>> collect2: ld returned 1 exit status
>> make: *** [cx.so] Error 1
>>
>>
>> If I delete the pxd file, it works fine (but it's not a cdef function,
>> so it's slow).
>
> Not sure--have you tried without the locals decorator in the Python
> source? I hope this isn't a regression.

Yes, as you can see, the locals is commented out. I'll try to bisect
it to see which commit broke it.

Ondrej

From ondrej at certik.cz  Wed Apr 29 23:49:28 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 29 Apr 2009 14:49:28 -0700
Subject: [Cython] problem with pure python mode
In-Reply-To: <85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
Message-ID: <85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>

On Wed, Apr 29, 2009 at 1:19 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Wed, Apr 29, 2009 at 12:37 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Apr 29, 2009, at 11:55 AM, Ondrej Certik wrote:
>>
>>> Hi,
>>>
>>> I am having a problem getting the pure mode run (I think this used to
>>> work before). Here is my cx.pyx file:
>>>
>>> -------------
>>> import cython
>>>
>>> from numpy import array
>>>
>>> #@cython.locals(x=cython.int)
>>> def f(x):
>>> ? ? r = cython.declare(cython.int)
>>> ? ? i = cython.declare(cython.int)
>>> ? ? if x > 0:
>>> ? ? ? ? r = 1
>>> ? ? ? ? for i in range(1, x):
>>> ? ? ? ? ? ? r *= i
>>> ? ? ? ? return r
>>> ? ? else:
>>> ? ? ? ? return 0
>>>
>>> def f_vectorized(a):
>>> ? ? return array([f(x) for x in a])
>>> ----------
>>>
>>> and here is my cx.pxd file:
>>> -----
>>> cdef int f(int x)
>>> -------
>>>
>>> If I compile it:
>>>
>>> $ cython cx.pyx
>>> warning: /home/ondrej/repos/cython_test/cx.pyx:6:0: Overriding cdef
>>> method with def method.
>>>
>>>
>>> it doesn't build:
>>>
>>> gcc -fPIC -O3 -I/usr/include/python2.6 -c -o cx.o cx.c
>>> cx.c:196: warning: ?__pyx_f_2cx_f? used but never defined
>>> gcc -fPIC -O3 -shared -o cx.so cx.o
>>> /usr/bin/ld: cx.o: relocation R_X86_64_PC32 against undefined symbol
>>> `__pyx_f_2cx_f' can not be used when making a shared object; recompile
>>> with -fPIC
>>> /usr/bin/ld: final link failed: Bad value
>>> collect2: ld returned 1 exit status
>>> make: *** [cx.so] Error 1
>>>
>>>
>>> If I delete the pxd file, it works fine (but it's not a cdef function,
>>> so it's slow).
>>
>> Not sure--have you tried without the locals decorator in the Python
>> source? I hope this isn't a regression.
>
> Yes, as you can see, the locals is commented out. I'll try to bisect
> it to see which commit broke it.

Ok, so it never worked. I am a bit confused, because I thought the
pure python mode worked fine for me before.


In the current cython --- is there any way to keep a pure python code,
and create an accompanying pxd file that would annotate the cdef
functions and classes?

If not, how would you suggest this to be implemented? Ideally, the pxd
file would be either autogenerated, or created from the decorators
(maybe on the fly, or in the background). But as a first step, let's
write it by hand.
I was looking in the code - but I will need some time to dig into it
so that I can implemented the above, so I can do that after my school
finishes in couple weeks.

This summer we will fix assumptions in sympy as part of GSoC, so I
want to prepare the ground for using pure python mode for speeding up
the core, so that we don't have to maintain two codebases.

Thanks,
Ondrej

From robertwb at math.washington.edu  Thu Apr 30 00:00:54 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 29 Apr 2009 15:00:54 -0700
Subject: [Cython] problem with pure python mode
In-Reply-To: <85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
Message-ID: <12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>

On Apr 29, 2009, at 2:49 PM, Ondrej Certik wrote:

>>>
>>> Not sure--have you tried without the locals decorator in the Python
>>> source? I hope this isn't a regression.
>>
>> Yes, as you can see, the locals is commented out. I'll try to bisect
>> it to see which commit broke it.
>
> Ok, so it never worked. I am a bit confused, because I thought the
> pure python mode worked fine for me before.
>
> In the current cython --- is there any way to keep a pure python code,
> and create an accompanying pxd file that would annotate the cdef
> functions and classes?

This should work, that's how I implemented this a long time ago. If  
it's not working now it's because something broke (or, I'm not  
promising that it's completely bug-free, but it should be good--we  
use it to bootstrap Cython itself).

> If not, how would you suggest this to be implemented? Ideally, the pxd
> file would be either autogenerated, or created from the decorators
> (maybe on the fly, or in the background). But as a first step, let's
> write it by hand.
> I was looking in the code - but I will need some time to dig into it
> so that I can implemented the above, so I can do that after my school
> finishes in couple weeks.
>
> This summer we will fix assumptions in sympy as part of GSoC, so I
> want to prepare the ground for using pure python mode for speeding up
> the core, so that we don't have to maintain two codebases.

Yep, that's one of the main objectives.

- Robert


From stefan_ml at behnel.de  Thu Apr 30 18:36:01 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 30 Apr 2009 18:36:01 +0200
Subject: [Cython] Path towards Python completeness
In-Reply-To: <49F76991.5010303@behnel.de>
References: <49ED5CE8.6090708@behnel.de>	<80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu>	<380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<BFD20AC6-2309-4634-98FB-D30BB66A6B2F@math.washington.edu>	<2038f7c55d17516c79c8b3b2a9385fb1.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49F76991.5010303@behnel.de>
Message-ID: <49F9D371.3060209@behnel.de>


Stefan Behnel wrote:
> Stefan Behnel wrote:
>> Robert Bradshaw wrote:
>>> http://hg.cython.org/cython-closures/file/c4ee199f08a2/tests/run/closures_T82.pyx
>>>
>>> Before you get your hopes up too much, I want to add a big
>>> disclaimer that though I tried to be careful, this is very first-
>>> draft code and there are bound to be issues (e.g. I think I've run
>>> into some with respect to Python arguments).
>> I'll give it a try during the next days.
> 
> I exercised it a bit and added more test cases. That revealed a couple of
> problems, although I must say that it works pretty well in general.
> 
> I added comments to the tests as far as I got with analysing them. One
> segfaults pretty badly, which makes it harder to figure out.

Ok, that was due to ref-counting problems. All of these work for me now:

http://hg.cython.org/cython-closures/file/626e754738b6/tests/run/closures_T82.pyx

I find that pretty impressive. Very good work, Robert!

This looks like it could go right into 0.12 rather than waiting for 0.13.

Stefan


From lrhazi at gmail.com  Thu Apr 30 19:04:25 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Thu, 30 Apr 2009 13:04:25 -0400
Subject: [Cython] Newbie: Facilities for handling data
Message-ID: <c4ab18df0904301004i635d0b9emab46ee626aa2c2ec@mail.gmail.com>

Hello all,

I see that Python has Buffer Objects and in 2.7/3.0 Memoryview
Objects, for this purpose...
Are those the facilites I need to learn and use? or does Cython
provide its own tools?

By "data" I mean complex data structures, like arrays of iovec
structs, or long "text" defined by indexes/offsets and lengths.

I need to learn how to provide my Python user with read and write
access to such data.

RTFMs welcome... Maybe I overlook a chapter in the Cython man..

Thanks a lot,
Mohamed.

From lrhazi at gmail.com  Thu Apr 30 19:22:13 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Thu, 30 Apr 2009 13:22:13 -0400
Subject: [Cython] Newbie: Facilities for handling data
In-Reply-To: <c4ab18df0904301004i635d0b9emab46ee626aa2c2ec@mail.gmail.com>
References: <c4ab18df0904301004i635d0b9emab46ee626aa2c2ec@mail.gmail.com>
Message-ID: <c4ab18df0904301022u58482f01mc8814559505977c4@mail.gmail.com>

Oh, I had skipped over numpy, assuming it was for Math only.... Looks
like that's what I need.
Mohamed.


On Thu, Apr 30, 2009 at 1:04 PM, Mohamed Lrhazi <lrhazi at gmail.com> wrote:
> Hello all,
>
> I see that Python has Buffer Objects and in 2.7/3.0 Memoryview
> Objects, for this purpose...
> Are those the facilites I need to learn and use? or does Cython
> provide its own tools?
>
> By "data" I mean complex data structures, like arrays of iovec
> structs, or long "text" defined by indexes/offsets and lengths.
>
> I need to learn how to provide my Python user with read and write
> access to such data.
>
> RTFMs welcome... Maybe I overlook a chapter in the Cython man..
>
> Thanks a lot,
> Mohamed.
>



-- 
" Logic merely sanctions the conquests of the intuition."
Jacques Hadamard

From stefan_ml at behnel.de  Thu Apr 30 19:37:39 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 30 Apr 2009 19:37:39 +0200
Subject: [Cython] Newbie: Facilities for handling data
In-Reply-To: <c4ab18df0904301004i635d0b9emab46ee626aa2c2ec@mail.gmail.com>
References: <c4ab18df0904301004i635d0b9emab46ee626aa2c2ec@mail.gmail.com>
Message-ID: <49F9E1E3.7070306@behnel.de>


Mohamed Lrhazi wrote:
> I see that Python has Buffer Objects and in 2.7/3.0 Memoryview
> Objects, for this purpose...
> Are those the facilites I need to learn and use? or does Cython
> provide its own tools?
> 
> By "data" I mean complex data structures, like arrays of iovec
> structs, or long "text" defined by indexes/offsets and lengths.
> 
> I need to learn how to provide my Python user with read and write
> access to such data.

It's impossible to answer this in general, but it's usually a good idea to
provide some kind of object oriented API to Python users, instead of just
copying the C-API. For example, if you are working on a large memory area
that has some kind of repetitive structure (like an array), you might want
to provide the user with an object that holds this memory and that uses
normal indexing to access each element. Each element might then be
represented by another type of object through which users can manipulate
it. Or, maybe you want to represent the data as a matrix of data entries. Or...

It really just depends on what you want to enable users to do with your
data. Try to come up with good use cases and design your API around those.

Stefan


From dagss at student.matnat.uio.no  Thu Apr 30 21:08:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 30 Apr 2009 21:08:04 +0200
Subject: [Cython] C -- Variable-sized array on stack?
Message-ID: <49F9F714.70907@student.matnat.uio.no>

Will I get away with code like this in Cython?

void foo(int n) {
   int stack[n];
   ...
}

I.e. what C language level are we targeting exactly? Also, does the 
above cause a proper stack allocation?

I get

test.c:15: warning: ISO C90 forbids variable-size array ?arr?

with -ansi -pedantic but otherwise gcc seems happy enough.

(I can get away without doing this as the caller knows the size 
compile-time and can supply the needed stack in the caller function 
stack frame, but this would make the code a lot nicer.)

-- 
Dag Sverre

From dalcinl at gmail.com  Thu Apr 30 21:36:10 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 30 Apr 2009 16:36:10 -0300
Subject: [Cython] C -- Variable-sized array on stack?
In-Reply-To: <49F9F714.70907@student.matnat.uio.no>
References: <49F9F714.70907@student.matnat.uio.no>
Message-ID: <e7ba66e40904301236t49e9419boe252979b977d3b0@mail.gmail.com>

On Thu, Apr 30, 2009 at 4:08 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Will I get away with code like this in Cython?
>
> void foo(int n) {
> ? int stack[n];
> ? ...
> }
>

I would prefer if that feature is not used in Cython code.

> I.e. what C language level are we targeting exactly?

As we currently target Py2.x, we should support C89/C90, and
variable-sized, automatic arrays, are not in C89 AFAIK.

> Also, does the
> above cause a proper stack allocation?
>

I believe is a kind of special allocation, likely not a "normal" stack
allocation (like you get when do "int stack[5]"). If the allocation
fails for any reason ("n" negative? no enough mem?) your process will
likely get signaled (SIGSEGV?).

> I get
>
> test.c:15: warning: ISO C90 forbids variable-size array ?arr?
>
> with -ansi -pedantic but otherwise gcc seems happy enough.
>

Variable-sized, automatic arrays are a C99 feature, but GCC has
provided them as an extension for long time ago... That's the reason
you get warnings with -pedantic (which in fact is a portability
testing flag).

> (I can get away without doing this as the caller knows the size
> compile-time and can supply the needed stack in the caller function
> stack frame, but this would make the code a lot nicer.)
>

Hope my previous arguments help you to make a decision... From my
side, -1. I prefer portability over nicer generated C codes.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jasone at canonware.com  Thu Apr 30 21:36:14 2009
From: jasone at canonware.com (Jason Evans)
Date: Thu, 30 Apr 2009 12:36:14 -0700
Subject: [Cython] C -- Variable-sized array on stack?
In-Reply-To: <49F9F714.70907@student.matnat.uio.no>
References: <49F9F714.70907@student.matnat.uio.no>
Message-ID: <49F9FDAE.20301@canonware.com>

Dag Sverre Seljebotn wrote:
> Will I get away with code like this in Cython?
> 
> void foo(int n) {
>    int stack[n];
>    ...
> }
> 
> I.e. what C language level are we targeting exactly? Also, does the 
> above cause a proper stack allocation?

I think this is a C99 feature that happens to be part of gnu89 as well. 
  I seem to recall that MSVC++ does not support the feature, which is an 
important consideration here.

Jason

From dagss at student.matnat.uio.no  Thu Apr 30 21:39:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 30 Apr 2009 21:39:56 +0200
Subject: [Cython] C -- Variable-sized array on stack?
In-Reply-To: <49F9FDAE.20301@canonware.com>
References: <49F9F714.70907@student.matnat.uio.no>
	<49F9FDAE.20301@canonware.com>
Message-ID: <49F9FE8C.60707@student.matnat.uio.no>

Jason Evans wrote:
> Dag Sverre Seljebotn wrote:
>> Will I get away with code like this in Cython?
>>
>> void foo(int n) {
>>    int stack[n];
>>    ...
>> }
>>
>> I.e. what C language level are we targeting exactly? Also, does the 
>> above cause a proper stack allocation?
> 
> I think this is a C99 feature that happens to be part of gnu89 as well. 
>   I seem to recall that MSVC++ does not support the feature, which is an 
> important consideration here.

Sure, I'm convinced not to use it now.

-- 
Dag Sverre