From dalcinl at gmail.com  Wed Oct  1 00:53:48 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 30 Sep 2008 19:53:48 -0300
Subject: [Cython] cython-devel broken for mpi4py
Message-ID: <e7ba66e40809301553w40360d41l2e0e5a382283e38e@mail.gmail.com>

I just got this, sorry, not time today to figure out what's going on:

Error converting Pyrex file to C:
------------------------------------------------------------
...

    int MPI_Type_get_attr(MPI_Datatype, int, void* , int*)
    int MPI_Type_set_attr(MPI_Datatype, int, void*)
    int MPI_Type_delete_attr(MPI_Datatype, int)

    ctypedef int
(MPI_Type_copy_attr_function)(MPI_Datatype,int,void*,void*,void*,int*)
                ^
------------------------------------------------------------

include/mpi4py/mpi.pxi:182:17: Function cannot return a function
Traceback (most recent call last):
  File "/u/dalcinl/bin/cython", line 8, in <module>
    main(command_line = 1)
  File "/u/dalcinl/lib/python/Cython/Compiler/Main.py", line 701, in main
    result = compile(sources, options)
  File "/u/dalcinl/lib/python/Cython/Compiler/Main.py", line 678, in compile
    return compile_multiple(source, options)
  File "/u/dalcinl/lib/python/Cython/Compiler/Main.py", line 648, in
compile_multiple
    result = run_pipeline(source, options)
  File "/u/dalcinl/lib/python/Cython/Compiler/Main.py", line 510, in
run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
  File "/u/dalcinl/lib/python/Cython/Compiler/Main.py", line 172, in
run_pipeline
    data = phase(data)
  File "/u/dalcinl/lib/python/Cython/Compiler/ParseTreeTransforms.py",
line 491, in __call__
    return super(AnalyseDeclarationsTransform, self).__call__(root)
  File "/u/dalcinl/lib/python/Cython/Compiler/Visitor.py", line 170, in __call__
    return super(CythonTransform, self).__call__(node)
  File "/u/dalcinl/lib/python/Cython/Compiler/Visitor.py", line 156, in __call__
    return self.visit(root)
  File "/u/dalcinl/lib/python/Cython/Compiler/Visitor.py", line 36, in visit
    return m(obj)
  File "/u/dalcinl/lib/python/Cython/Compiler/ParseTreeTransforms.py",
line 494, in visit_ModuleNode
    node.analyse_declarations(self.env_stack[-1])
  File "/u/dalcinl/lib/python/Cython/Compiler/ModuleNode.py", line 57,
in analyse_declarations
    self.body.analyse_declarations(env)
  File "/u/dalcinl/lib/python/Cython/Compiler/Nodes.py", line 312, in
analyse_declarations
    stat.analyse_declarations(env)
  File "/u/dalcinl/lib/python/Cython/Compiler/Nodes.py", line 312, in
analyse_declarations
    stat.analyse_declarations(env)
  File "/u/dalcinl/lib/python/Cython/Compiler/Nodes.py", line 367, in
analyse_declarations
    self.body.analyse_declarations(env)
  File "/u/dalcinl/lib/python/Cython/Compiler/Nodes.py", line 312, in
analyse_declarations
    stat.analyse_declarations(env)
  File "/u/dalcinl/lib/python/Cython/Compiler/Nodes.py", line 864, in
analyse_declarations
    cname = cname, visibility = self.visibility)
  File "/u/dalcinl/lib/python/Cython/Compiler/Symtab.py", line 328, in
declare_typedef
    type.qualified_name = entry.qualified_name
AttributeError: Entry instance has no attribute 'qualified_name'


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From fperez.net at gmail.com  Wed Oct  1 01:59:01 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 30 Sep 2008 16:59:01 -0700
Subject: [Cython] docs.cython.org
In-Reply-To: <E7F6549F-0E9E-4D02-A25F-1A02C0B86357@math.washington.edu>
References: <20080927144913.GA20932@encolpuis>
	<85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com>
	<20080929142813.GA8804@encolpuis>
	<85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com>
	<476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu>
	<20080929215931.GA11292@encolpuis>
	<48E1D66D.7080408@student.matnat.uio.no>
	<E7F6549F-0E9E-4D02-A25F-1A02C0B86357@math.washington.edu>
Message-ID: <db6b5ecc0809301659x225ca128k93da882bf9c5d465@mail.gmail.com>

On Tue, Sep 30, 2008 at 10:35 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:

>> b) A script to build them is shipped with Cython (setup.py could do
>> this
>> I suppose)
>
> If Sphinx is detected, and it's fast. I certainly don't want it to be
> a requirement.

Please feel free to pillage anything from the ipython sources you may
want for this.  We build the docs only when
'sdist' or 'bdist_rpm' are given, so that only package distributors
need to have the toolchain installed:

http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/annotate/1147?file_id=setup.py-20080216095032-xb0is4a97lmosv2z-14

lines 97-120.

There's also a bit of nasty trickery in

http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/annotate/1147?file_id=setupbase.py-20080606221622-xwbqrw3mlwq01vyb-1

line 146, func make_dir_struct() to be able to easily produce the
necessary data layout for distutils to ship the nested doc tree.

You guys already have the makefile and sphinx code, but take any of
this you may find useful.

Anything for better docs :)

Cheers,

f

From uschmitt at mineway.de  Wed Oct  1 13:08:46 2008
From: uschmitt at mineway.de (Uwe Schmitt)
Date: Wed, 01 Oct 2008 13:08:46 +0200
Subject: [Cython] pyximport
Message-ID: <48E35A3E.5080403@mineway.de>

Hi,

I had trouble to import pyximport and found
only two postings about this problem from august.

The problem was that "import pyximport" fails,
which I could fix (as suggested in the postings
above) by making pyximport/ a python packgage,
which means I had to add a "__init__.py" file to
.../site-packages/cython.../pyximport/

I wonder that I found not more about that.
Is nobody using pyximport or did I miss something ?


Greetings, Uwe

-- 
Dr. rer. nat. Uwe Schmitt
F&E Mathematik
 
mineway GmbH
Science Park 2
D-66123 Saarbr?cken
 
Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376
 
uschmitt at mineway.de
www.mineway.de
 
Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339



From dalcinl at gmail.com  Wed Oct  1 15:47:31 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 10:47:31 -0300
Subject: [Cython] pyximport
In-Reply-To: <48E35A3E.5080403@mineway.de>
References: <48E35A3E.5080403@mineway.de>
Message-ID: <e7ba66e40810010647g707314feqb0c2076198b22997@mail.gmail.com>

On Wed, Oct 1, 2008 at 8:08 AM, Uwe Schmitt <uschmitt at mineway.de> wrote:
> The problem was that "import pyximport" fails,
> which I could fix (as suggested in the postings
> above) by making pyximport/ a python packgage,
> which means I had to add a "__init__.py" file to
> .../site-packages/cython.../pyximport/

Yes, add a __init__.py file with the following line inside 'from
pyximport import *'

This still need to be fixed in cython-devel repo. I'll try to make a patch.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Oct  1 15:52:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 10:52:57 -0300
Subject: [Cython] PATCH: fix pyximport installation as a package
Message-ID: <e7ba66e40810010652u77fe0376ud388b549a2ea00e@mail.gmail.com>

This should fix the installation of pyximport as a package. Please,
review and apply.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pyximport.diff
Type: text/x-patch
Size: 382 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081001/a79fa014/attachment.bin 

From dalcinl at gmail.com  Wed Oct  1 20:13:00 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 15:13:00 -0300
Subject: [Cython] cython leaks references to Py_None for cdef'ed globals
Message-ID: <e7ba66e40810011113r653738b7ve21afe5b4803973@mail.gmail.com>

Consider the following code inside a one-line pyx file:

cdef object someint = 7

Then Cython generates the following inside the module init function:

  /*--- Global init code ---*/
  __pyx_v_9refleaks2_someint = Py_None; Py_INCREF(Py_None);

  /* "/u/dalcinl/Devel/Cython/sandbox/refleaks2.pyx":1
 * cdef object someint = 7             # <<<<<<<<<<<<<<
 *
 */
  Py_INCREF(__pyx_int_7);
  __pyx_v_9refleaks2_someint = __pyx_int_7;


Clearly, Py_None references are being leaked. All this is because of
bad interaction between this two methods:

* ModuleNode.generate_global_init_code(...) (in ModuleNode.py)
* FinalOptimizePhase.visit_SingleAssignmentNode(...) (in Optimize.py)



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Oct  1 20:19:53 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 15:19:53 -0300
Subject: [Cython] cython leaks references to Py_None for cdef'ed globals
In-Reply-To: <e7ba66e40810011113r653738b7ve21afe5b4803973@mail.gmail.com>
References: <e7ba66e40810011113r653738b7ve21afe5b4803973@mail.gmail.com>
Message-ID: <e7ba66e40810011119o3b5fbcbi3fff71fd606b71b8@mail.gmail.com>

BTW, this patch fix the issue by not initializing cdef'ed module
globals to None. But this is not the safest approach (I can imagine
some nasty ways to originate segfaults). Perhaps we should try to fix
the isue in FinalOptimizePhase.visit_SingleAssignmentNode

On Wed, Oct 1, 2008 at 3:13 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Consider the following code inside a one-line pyx file:
>
> cdef object someint = 7
>
> Then Cython generates the following inside the module init function:
>
>  /*--- Global init code ---*/
>  __pyx_v_9refleaks2_someint = Py_None; Py_INCREF(Py_None);
>
>  /* "/u/dalcinl/Devel/Cython/sandbox/refleaks2.pyx":1
>  * cdef object someint = 7             # <<<<<<<<<<<<<<
>  *
>  */
>  Py_INCREF(__pyx_int_7);
>  __pyx_v_9refleaks2_someint = __pyx_int_7;
>
>
> Clearly, Py_None references are being leaked. All this is because of
> bad interaction between this two methods:
>
> * ModuleNode.generate_global_init_code(...) (in ModuleNode.py)
> * FinalOptimizePhase.visit_SingleAssignmentNode(...) (in Optimize.py)
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fixinitnoneleaks.diff
Type: text/x-patch
Size: 1166 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081001/8c5e15b7/attachment.bin 

From thomas.e.keller at gmail.com  Wed Oct  1 20:29:02 2008
From: thomas.e.keller at gmail.com (Thomas Keller)
Date: Wed, 1 Oct 2008 13:29:02 -0500
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
Message-ID: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>

This may be a foolish question, but I'm still getting used to to
Cython. I was under the impression that you can cdef all the basic
python types like list, dict,tuple etc. However, when I try to cdef variables
of type set, Cython gives a compile error. Is it simply unsupported at
this time or am I completely missing something? I imagine either way
it wouldn't speed things up that much.
Regards,
Thomas Keller

From dalcinl at gmail.com  Wed Oct  1 21:08:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 16:08:57 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
Message-ID: <e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>

Dear Thomas,

This should be fixed in 'cython-devel' repo. All numeric, string and
container builtin types (plus slice and file) are supported for all
Python versions. Let us know if you give a try to cython-devel and
something does not work.

Of course,  in Py2.3 your C compiler will fail if you use 'set',
perhaps this could be emulated somewhat with sets.Set and
set.InmutableSet, but I do not tried that when I wrote the patch
fixing your problem.

Regards,

On Wed, Oct 1, 2008 at 3:29 PM, Thomas Keller <thomas.e.keller at gmail.com> wrote:
> This may be a foolish question, but I'm still getting used to to
> Cython. I was under the impression that you can cdef all the basic
> python types like list, dict,tuple etc. However, when I try to cdef variables
> of type set, Cython gives a compile error. Is it simply unsupported at
> this time or am I completely missing something? I imagine either way
> it wouldn't speed things up that much.
> Regards,
> Thomas Keller
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Oct  1 21:28:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 01 Oct 2008 21:28:00 +0200
Subject: [Cython] cython leaks references to Py_None for cdef'ed globals
Message-ID: <3305741288.1706043@smtp.netcom.no>

I say let them leak? The only real risk (that I can see at least) is overflowing the refcount, and if I understand and remember an informal chat correctly even that is eliminated in newer Python versions where Py_None and friends have special deallocation which never deallocate...

Dag Sverre Seljebotn
-----Original Message-----
From: "Lisandro Dalcin" <dalcinl at gmail.com>
Date: Wednesday, Oct 1, 2008 8:20 pm
Subject: Re: [Cython] cython leaks references to Py_None for cdef'ed globals
To: cython-dev <cython-dev at codespeak.net>Reply-To: cython-dev at codespeak.net

BTW, this patch fix the issue by not initializing cdef'ed module
>globals to None. But this is not the safest approach (I can imagine
>some nasty ways to originate segfaults). Perhaps we should try to fix
>the isue in FinalOptimizePhase.visit_SingleAssignmentNode
>
>On Wed, Oct 1, 2008 at 3:13 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> Consider the following code inside a one-line pyx file:
>>
>> cdef object someint = 7
>>
>> Then Cython generates the following inside the module init function:
>>
>>  /*--- Global init code ---*/
>>  __pyx_v_9refleaks2_someint = Py_None; Py_INCREF(Py_None);
>>
>>  /* "/u/dalcinl/Devel/Cython/sandbox/refleaks2.pyx":1
>>  * cdef object someint = 7             # <<<<<<<<<<<<<<
>>  *
>>  */
>>  Py_INCREF(__pyx_int_7);
>>  __pyx_v_9refleaks2_someint = __pyx_int_7;
>>
>>
>> Clearly, Py_None references are being leaked. All this is because of
>> bad interaction between this two methods:
>>
>> * ModuleNode.generate_global_init_code(...) (in ModuleNode.py)
>> * FinalOptimizePhase.visit_SingleAssignmentNode(...) (in Optimize.py)
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>>
>
>
>
>-- 
>Lisandro Dalc?n
>---------------
>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>Tel/Fax: +54-(0)342-451.1594
>


From michael.abshoff at googlemail.com  Wed Oct  1 21:34:41 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 01 Oct 2008 12:34:41 -0700
Subject: [Cython] cython leaks references to Py_None for cdef'ed globals
In-Reply-To: <3305741288.1706043@smtp.netcom.no>
References: <3305741288.1706043@smtp.netcom.no>
Message-ID: <48E3D0D1.6050909@gmail.com>

Dag Sverre Seljebotn wrote:

Hi,

> I say let them leak? The only real risk (that I can see at least) is overflowing the refcount,
 > and if I understand and remember an informal chat correctly even that 
is eliminated in newer
 > Python versions where Py_None and friends have special deallocation 
which never deallocate...

I have had the pleasure to debug refcount reference overflows and let me 
say it is not something I would like to repeat since it happened only on 
32 bit test systems (the count would also wraps on a 64 bit box, but 
that would take a while longer). So I would highly suggest to fix any 
reference count leak since you never want code to crash for stupid 
reasons like this.

> Dag Sverre Seljebotn

<SNIP>

Cheers,

Michael

From thomas.e.keller at gmail.com  Wed Oct  1 21:41:15 2008
From: thomas.e.keller at gmail.com (Thomas Keller)
Date: Wed, 1 Oct 2008 19:41:15 +0000 (UTC)
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
Message-ID: <loom.20081001T193537-418@post.gmane.org>

Lisandro Dalcin <dalcinl at ...> writes:

> 
> Dear Thomas,
> 
> This should be fixed in 'cython-devel' repo. All numeric, string and
> container builtin types (plus slice and file) are supported for all
> Python versions. Let us know if you give a try to cython-devel and
> something does not work.
> 

Thanks much for the reply. I realized after the email I provided absolutely no
information about my build. I was using Cython from within SAGE 3.1.1, so I
believe it's not as updated as what is in cython-devel. I'll try that repo and
see how things go.






From dalcinl at gmail.com  Wed Oct  1 21:47:42 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 16:47:42 -0300
Subject: [Cython] cython leaks references to Py_None for cdef'ed globals
In-Reply-To: <48E3D0D1.6050909@gmail.com>
References: <3305741288.1706043@smtp.netcom.no> <48E3D0D1.6050909@gmail.com>
Message-ID: <e7ba66e40810011247j582193a2ma0f2314e336efbfb@mail.gmail.com>

The particular case being discussed, this leak occurs only one time,
at extension module import time.

Anyway, Michael, I'm on your side. In the past, refleaks in numpy made
it very hard for my to make sure in my unittesting that I was not
generating leaks on my code.

So, Dag, I think we should try hard to fix this. My patch fix the
problem, but it's a bit unsafe.  I'm so pedantic (because of bad
experiences in the past) about ref leaks that I always run Cython with
'--cleanup 9' flag. And I always run my full testsuite inside a loop
with debug python builds just to make sure I'm never leaking.


On Wed, Oct 1, 2008 at 4:34 PM, Michael Abshoff
<michael.abshoff at googlemail.com> wrote:
> Dag Sverre Seljebotn wrote:
>
> Hi,
>
>> I say let them leak? The only real risk (that I can see at least) is overflowing the refcount,
>  > and if I understand and remember an informal chat correctly even that
> is eliminated in newer
>  > Python versions where Py_None and friends have special deallocation
> which never deallocate...
>
> I have had the pleasure to debug refcount reference overflows and let me
> say it is not something I would like to repeat since it happened only on
> 32 bit test systems (the count would also wraps on a 64 bit box, but
> that would take a while longer). So I would highly suggest to fix any
> reference count leak since you never want code to crash for stupid
> reasons like this.
>
>> Dag Sverre Seljebotn
>
> <SNIP>
>
> Cheers,
>
> Michael
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Oct  1 22:10:05 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 17:10:05 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <loom.20081001T193537-418@post.gmane.org>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
Message-ID: <e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>

On Wed, Oct 1, 2008 at 4:41 PM, Thomas Keller <thomas.e.keller at gmail.com> wrote:
> Thanks much for the reply. I realized after the email I provided absolutely no
> information about my build. I was using Cython from within SAGE 3.1.1, so I
> believe it's not as updated as what is in cython-devel. I'll try that repo and
> see how things go.

Yep, the suport for your request was pushed a couple of days ago, so
it not availabe even in the latest official release.

BTW, access to the 'set' methods is not currently optimized in any
way. If you can help me a bit, we can optimize this, ie. map
'someset.add(obj)' to a direct Python C-API call. If you make a heavy
use of sets this could give you some speedup.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From thomas.e.keller at gmail.com  Wed Oct  1 23:05:00 2008
From: thomas.e.keller at gmail.com (Thomas Keller)
Date: Wed, 1 Oct 2008 21:05:00 +0000 (UTC)
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
Message-ID: <loom.20081001T210058-967@post.gmane.org>

Lisandro Dalcin <dalcinl at ...> writes:


> 
> BTW, access to the 'set' methods is not currently optimized in any
> way. If you can help me a bit, we can optimize this, ie. map
> 'someset.add(obj)' to a direct Python C-API call. If you make a heavy
> use of sets this could give you some speedup.
> 


I'm not sure how useful I'll be, but I'd be glad to try helping. The program I
am trying to use this with centers around a loop that heavily uses various set
operations, so I am definitely interested. What files should I look at to see
what to follow? As a disclaimer, I've never tried directly interfacing with the
Python C-API.


From robertwb at math.washington.edu  Wed Oct  1 23:13:03 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Oct 2008 14:13:03 -0700
Subject: [Cython] cython leaks references to Py_None for cdef'ed globals
In-Reply-To: <e7ba66e40810011119o3b5fbcbi3fff71fd606b71b8@mail.gmail.com>
References: <e7ba66e40810011113r653738b7ve21afe5b4803973@mail.gmail.com>
	<e7ba66e40810011119o3b5fbcbi3fff71fd606b71b8@mail.gmail.com>
Message-ID: <41440546-AAA6-433E-8FC1-3B6E043B6C93@math.washington.edu>

On Oct 1, 2008, at 11:19 AM, Lisandro Dalcin wrote:

> BTW, this patch fix the issue by not initializing cdef'ed module
> globals to None. But this is not the safest approach (I can imagine
> some nasty ways to originate segfaults). Perhaps we should try to fix
> the isue in FinalOptimizePhase.visit_SingleAssignmentNode

I agree that this should be fixed (I remember debugging a nasty bug  
where the refcount of the integer 0 wrapped around, ugh..). However,  
FinalOptimizePhase is the right place to do it.

>
> On Wed, Oct 1, 2008 at 3:13 PM, Lisandro Dalcin <dalcinl at gmail.com>  
> wrote:
>> Consider the following code inside a one-line pyx file:
>>
>> cdef object someint = 7
>>
>> Then Cython generates the following inside the module init function:
>>
>>  /*--- Global init code ---*/
>>  __pyx_v_9refleaks2_someint = Py_None; Py_INCREF(Py_None);
>>
>>  /* "/u/dalcinl/Devel/Cython/sandbox/refleaks2.pyx":1
>>  * cdef object someint = 7             # <<<<<<<<<<<<<<
>>  *
>>  */
>>  Py_INCREF(__pyx_int_7);
>>  __pyx_v_9refleaks2_someint = __pyx_int_7;
>>
>>
>> Clearly, Py_None references are being leaked. All this is because of
>> bad interaction between this two methods:
>>
>> * ModuleNode.generate_global_init_code(...) (in ModuleNode.py)
>> * FinalOptimizePhase.visit_SingleAssignmentNode(...) (in Optimize.py)
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>>
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0) 
> 342-451.1594<fixinitnoneleaks.diff>___________________________________ 
> ____________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Wed Oct  1 23:17:43 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Oct 2008 14:17:43 -0700
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <loom.20081001T210058-967@post.gmane.org>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
Message-ID: <9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>

On Oct 1, 2008, at 2:05 PM, Thomas Keller wrote:

> Lisandro Dalcin <dalcinl at ...> writes:
>
>
>>
>> BTW, access to the 'set' methods is not currently optimized in any
>> way. If you can help me a bit, we can optimize this, ie. map
>> 'someset.add(obj)' to a direct Python C-API call. If you make a heavy
>> use of sets this could give you some speedup.
>
> I'm not sure how useful I'll be, but I'd be glad to try helping.  
> The program I
> am trying to use this with centers around a loop that heavily uses  
> various set
> operations, so I am definitely interested. What files should I look  
> at to see
> what to follow? As a disclaimer, I've never tried directly  
> interfacing with the
> Python C-API.

You could look at how list and dict are handled at the bottom of

http://hg.cython.org/cython-devel/file/c9f3badef047/Cython/Compiler/ 
Builtin.py

- Robert


From dfdeshom at gmail.com  Wed Oct  1 23:25:51 2008
From: dfdeshom at gmail.com (didier deshommes)
Date: Wed, 1 Oct 2008 17:25:51 -0400
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
Message-ID: <4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>

On Wed, Oct 1, 2008 at 5:17 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> You could look at how list and dict are handled at the bottom of
>
> http://hg.cython.org/cython-devel/file/c9f3badef047/Cython/Compiler/
> Builtin.py

Hi Robert,
What is the meaning of lines like this:
("append", "OO",   "i", "PyList_Append")

I'm especially curious about the "OO" and "i" stuff. Is this the only
code change needed to map these calls?

didier

>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Wed Oct  1 23:29:32 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Oct 2008 14:29:32 -0700
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
Message-ID: <2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>

On Oct 1, 2008, at 2:25 PM, didier deshommes wrote:

> On Wed, Oct 1, 2008 at 5:17 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> You could look at how list and dict are handled at the bottom of
>>
>> http://hg.cython.org/cython-devel/file/c9f3badef047/Cython/Compiler/
>> Builtin.py
>
> Hi Robert,
> What is the meaning of lines like this:
> ("append", "OO",   "i", "PyList_Append")
>
> I'm especially curious about the "OO" and "i" stuff.

"OO" means PyList_Append takes two arguments of type O = object. "i"  
means the return value is an int (for error checking, never actually  
revealed to Python).

> Is this the only code change needed to map these calls?

Yep, it's that easy now that we have support for this kind of thing.  
Looking forward to a patch :).

- Robert


From dalcinl at gmail.com  Wed Oct  1 23:37:45 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 18:37:45 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
Message-ID: <e7ba66e40810011437s4700005at947d4aa0de64fe00@mail.gmail.com>

In Python2.5:
PyAPI_FUNC(int) PySet_Clear(PyObject *set);
PyAPI_FUNC(int) PySet_Contains(PyObject *anyset, PyObject *key);
PyAPI_FUNC(int) PySet_Discard(PyObject *set, PyObject *key);
PyAPI_FUNC(int) PySet_Add(PyObject *set, PyObject *key);
PyAPI_FUNC(int) _PySet_Next(PyObject *set, Py_ssize_t *pos, PyObject **key);
PyAPI_FUNC(int) _PySet_NextEntry(PyObject *set, Py_ssize_t *pos,
PyObject **key, long *hash);
PyAPI_FUNC(PyObject *) PySet_Pop(PyObject *set);
PyAPI_FUNC(int) _PySet_Update(PyObject *set, PyObject *iterable);

then you should use (check it!!!)

"clear", "O", "i", "PySet_Clear"

"discard", "OO", "i", "PySet_Discard"

"add "OO", "i", "PySet_Add"

"pop", "O", "O", "PySet_Pop"



On Wed, Oct 1, 2008 at 6:29 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 1, 2008, at 2:25 PM, didier deshommes wrote:
>
>> On Wed, Oct 1, 2008 at 5:17 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>>
>>> You could look at how list and dict are handled at the bottom of
>>>
>>> http://hg.cython.org/cython-devel/file/c9f3badef047/Cython/Compiler/
>>> Builtin.py
>>
>> Hi Robert,
>> What is the meaning of lines like this:
>> ("append", "OO",   "i", "PyList_Append")
>>
>> I'm especially curious about the "OO" and "i" stuff.
>
> "OO" means PyList_Append takes two arguments of type O = object. "i"
> means the return value is an int (for error checking, never actually
> revealed to Python).
>
>> Is this the only code change needed to map these calls?
>
> Yep, it's that easy now that we have support for this kind of thing.
> Looking forward to a patch :).
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Oct  1 23:44:31 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Oct 2008 14:44:31 -0700
Subject: [Cython] Python-compatible syntax
In-Reply-To: <50542.85.165.152.85.1222448145.squirrel@webmail.uio.no>
References: <3305292230.603409@smtp.netcom.no>
	<85b5c3130809260812u4b9a24bfn84871c1e734a2fe2@mail.gmail.com>
	<50542.85.165.152.85.1222448145.squirrel@webmail.uio.no>
Message-ID: <836CF7B1-3029-4E9E-AE1C-9C17B56B1647@math.washington.edu>

On Sep 26, 2008, at 9:55 AM, Dag Sverre Seljebotn wrote:

> Ondrej wrote:
>> On Fri, Sep 26, 2008 at 4:43 PM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> I can write more later when I'm not on my phone, but: I'd suggest
>>> starting with implementing
>>>
>>> @cython.earlybind
>>> def f(x, y): ...
>>>
>>> and have it turn to
>>>
>>> cpdef f(x, y): ...
>>>
>>> (note on the name below)
>>>
>>> Just getting this working, with arguments and result typed to  
>>> "object",
>>> should get your feet wet. Then I suppose the next step is  
>>> implementing
>>> the Py3 PEP supporting decorators on arguments and return type.  
>>> (I think
>>> one should aim for Py2.6+ here, though I suppose arguments to the
>>> decorator could work too: @cython.earlybind(x=cython.uint,
>>> localvar=cython.float)...opinions?
>>
>> I need it to be working with python2.4+, so I'll choose something  
>> that
>> works in python2.4. Debian just switched to python2.5 recently, so
>> it's a long, long way before python2.6 could become a de facto
>> standard. But generally I agree one should design this so that things
>> are especially easy in python2.6+.
>
> (Writing this just so that you can look it up when you are ready for
> types, ideally when declaring cpdef without any non-object types works
> already).
>
> We've had long discussions about options for this earlier and it is
> something people care strongly about, so please check with the mailing
> list before fixing too hard on a given syntax.
>
> Until something else is proposed then I'm +1 on my following  
> proposal (but
> at least Stefan or Robert should +1 it before it has any kind of  
> official
> status):
>
> @cython.earlybind(cython.uint, x=cython.uint, v=cython.float)
> def f(x):
>     v = x
>     return x
>
> The first positional argument is the return type, while any keyword
> arguments are for either arguments or local variables. Support for the
> Py2.6+ syntax can be added in addition later.
>
> We had a discussion already about the type names, I think we agreed on
> something like (though others should comment too):
> - cython.uint, cython.int, cython.float, cython.longdouble, etc.
> - cython.p_uint for unsigned*. Up to three pointers provided like  
> this:
> cython.ppp_uint is unsigned***. (Hmm, in fact, as this is likely  
> parsed as
> a string and not declared anywhere, one can make the number of p-s
> arbitrary).
> - For people preferring verbosity: cython.ptr(cython.uint), with  
> nesting
> of ptr as many times as one like. Finally, cython.ptr(cython.uint,
> levels=4) for unsigned****...
> - Pointers to functions provided in a similar fashion, i.e.
> cython.func_ptr(cython.int, cython.float) etc.
>
> Notes you can look at again when you get as far (I enjoy thinking  
> about
> how to do stuff, coding it is the boring part :-) of course, if you do
> things in your own way instead there's no harm to it though):
>
> 1) This is already parseable by Parsing.py, so you should be able to
> implement this purely by staying in the transform I wrote about and  
> output
> type nodes in the appropriate places.
>
> 2) It would work by having the transform simply inserting "cdef  
> unsigned
> int v" statements (i.e. CVarDeclNode) at the beginning of the function
> body, or changing info in the argument list of the function  
> (anything in
> the argument list goes there, the rest is inserted at beginning of
> function without checking whether it is used).
>
> 3) As cpdef functions cannot take pointer arguments, one should  
> generate
> cdef functions instead when there are pointer arguments. IMO this  
> should
> be automagic, i.e. cpdef is selected whenever the arguments are  
> coercable
> from objects and cdef otherwise. (An alternative I would like better,
> though, is to have cpdef functions taking pointer arguments be  
> allowed in
> the language, but lead to a "def" version containing a stub raising a
> run-time error! This way things could be more consistent with good  
> error
> reporting. Long term, ctypes pointers could be coercable to Cython
> pointers too.)
>
>>> 2) The ResolveDirectives transform (or something like that)  
>>> already has
>>> code in it to intercept @cython.X-decorators on functions for  
>>> compiler
>>> directives, I think you want to extend that transform, in time  
>>> changing
>>> the name appropriately to give it a wider scope.
>
> Having thought about this, the transform should be renamed and
> redocumented to "Handle the parts of the cython scope that are not  
> normal
> run-time functions, i.e. any 'magic' functions and decorators in the
> cython cimportable module." Something like CythonScopeTransform  
> perhaps.

Some preliminary support for this is up at http://hg.cython.org/ 
cython-devel/rev/cfc5c05e0292 and http://hg.cython.org/cython-devel/ 
rev/c9f3badef047 . There's *lots* of room for improvement (e.g. no  
support for the p*_int notation, etc) but that should be fairly easy  
once the framework is in place. One can now do

import cython

@cython.locals(s=cython.double, i=int, n=int)
def harmonic_sum(n):
     s = 0
     for i in range(1, n+1):
         s += 1.0 / i
     return s

and it will work in Python or Cython (there's a new "shadow" module).  
One can also do

@cython.locals(ptr="void*")
def my_id(x):
     ptr = cython.cast("void*", x)
     return cython.cast(long, ptr)

though of course that won't work in Python. This does not handle def - 
 > c[p]def. Until we have auto-generation of .pxd files, this is  
probably easiest to handle by letting the declarations in the .pxd  
file "mutate" the def nodes in the .py file into the right thing.

- Robert


From robertwb at math.washington.edu  Thu Oct  2 00:02:55 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Oct 2008 15:02:55 -0700
Subject: [Cython] pyximport
In-Reply-To: <48E35A3E.5080403@mineway.de>
References: <48E35A3E.5080403@mineway.de>
Message-ID: <FDF7F4D0-B767-48CD-B6D1-A6BC1EB46D13@math.washington.edu>


On Oct 1, 2008, at 4:08 AM, Uwe Schmitt wrote:

> Hi,
>
> I had trouble to import pyximport and found
> only two postings about this problem from august.
>
> The problem was that "import pyximport" fails,
> which I could fix (as suggested in the postings
> above) by making pyximport/ a python packgage,
> which means I had to add a "__init__.py" file to
> .../site-packages/cython.../pyximport/

Thanks. We are tracking this at http://trac.cython.org/cython_trac/ 
ticket/91

> I wonder that I found not more about that.
> Is nobody using pyximport or did I miss something ?

It is very new, so not many people are using it yet.

- Robert



From greg.ewing at canterbury.ac.nz  Thu Oct  2 02:51:47 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 02 Oct 2008 12:51:47 +1200
Subject: [Cython] quick guide moved ???
In-Reply-To: <C622577D-9A2C-4B47-A215-5440A5FA349B@mineway.de>
References: <C622577D-9A2C-4B47-A215-5440A5FA349B@mineway.de>
Message-ID: <48E41B23.8060509@canterbury.ac.nz>

Uwe Schmitt wrote:

> does anybody know what happened
> to "michaels quick guide to pyrex"
> at http://iopen.net/pyrex-guide/ ???

Not sure about the original, but there's a copy of it available here:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/mpj17-pyrex-guide/

-- 
Greg

From dalcinl at gmail.com  Thu Oct  2 03:42:13 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Oct 2008 22:42:13 -0300
Subject: [Cython] Robert, please help!
Message-ID: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>

Robert, in this changeset

changeset:   1184:515f94dc16f2
user:        Robert Bradshaw <robertwb at math.washington.edu>
date:        Mon Sep 29 19:08:45 2008 -0700
summary:     Refactoring of type parsing

you cannot parse C function typedef like this:

cdef extern from *:
    void (somefunc)(int)

Please note that this declaration is not of the form "void
(*somefuncptr)(int)", and AFAIK the former is valid C code.

I was looking at the changes you pushed, but I was unable to discover
what's going on. So I really need your help here.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Oct  2 04:40:19 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 02 Oct 2008 14:40:19 +1200
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
Message-ID: <48E43493.4040009@canterbury.ac.nz>

Robert Bradshaw wrote:
> On Oct 1, 2008, at 2:25 PM, didier deshommes wrote:
 >
> > I'm especially curious about the "OO" and "i" stuff.
> 
> "OO" means PyList_Append takes two arguments of type O = object. "i"  
> means the return value is an int (for error checking, never actually  
> revealed to Python).

That should really be "r", not "i" ("r" is an int used only
to signal an error, "i" is an actual int to be used).

There's a comment at the top of TypeSlots.py explaining
what all the letters mean.

-- 
Greg

From dfdeshom at gmail.com  Thu Oct  2 05:28:57 2008
From: dfdeshom at gmail.com (didier deshommes)
Date: Wed, 1 Oct 2008 23:28:57 -0400
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
Message-ID: <4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>

On Wed, Oct 1, 2008 at 5:29 PM, Robert Bradshaw
>> Is this the only code change needed to map these calls?
>
> Yep, it's that easy now that we have support for this kind of thing.
> Looking forward to a patch :).

Ok, I've got a patch and I've looked at the generated C code, compiled
the module and verified the behavior of the function. How do I write
tests for cython? Do I just edit tests/run/dict.pyx and look at the
generated code?

didier

From kevinar18 at hotmail.com  Thu Oct  2 05:36:29 2008
From: kevinar18 at hotmail.com (Kevin Ar18)
Date: Wed, 1 Oct 2008 23:36:29 -0400
Subject: [Cython] quick guide moved ???
In-Reply-To: <C622577D-9A2C-4B47-A215-5440A5FA349B@mineway.de>
References: <C622577D-9A2C-4B47-A215-5440A5FA349B@mineway.de>
Message-ID: <BLU110-W453268B61709A4EB013082AA3D0@phx.gbl>



> Hi,
> 
> does anybody know what happened
> to "michaels quick guide to pyrex"
> at http://iopen.net/pyrex-guide/ ???
> 
> Greetings, Uwe
It's my fault.  I asked him for a non-copyleft documentation.  When he said there wasn't one, I thanked him and told him about the super secret docs at http://www.mudskipper.ca/cython-doc/
 I'd say I won't do it again ... but I know I will. :)

/end sarcasm

On a serious note, I might want to work up some kind of Cython reference; I know it will help me, and I suspect it would help others.  So, who can I ask lots of questions? :)
_________________________________________________________________
See how Windows connects the people, information, and fun that are part of your life.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/

From dalcinl at gmail.com  Thu Oct  2 05:50:36 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Oct 2008 00:50:36 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
Message-ID: <e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>

On Thu, Oct 2, 2008 at 12:28 AM, didier deshommes <dfdeshom at gmail.com> wrote:
>
> Ok, I've got a patch and I've looked at the generated C code, compiled
> the module and verified the behavior of the function. How do I write
> tests for cython? Do I just edit tests/run/dict.pyx and look at the
> generated code?

More or less, yes. Write a 'set.pyx' file like this:

__doc__ = u"""
>>> test_set()
True
"""

def test_set():
    cdef set s1 = set()
    cdef set s2 = set()
    # then do stuff with s1 and s2, like s1.add(1)
    # finally return True
    return True


Of course, I do not know if this can be readily incorporated in Cython
test suite, provided that this code is going to fail  in the case of a
Python 2.3 runtime.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Oct  2 05:58:36 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Oct 2008 00:58:36 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <48E43493.4040009@canterbury.ac.nz>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<48E43493.4040009@canterbury.ac.nz>
Message-ID: <e7ba66e40810012058q6053b102y236255c936b965b9@mail.gmail.com>

On Wed, Oct 1, 2008 at 11:40 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Robert Bradshaw wrote:
>> "OO" means PyList_Append takes two arguments of type O = object. "i"
>
> That should really be "r", not "i" ("r" is an int used only
> to signal an error, "i" is an actual int to be used).
>

Mmm.. Perhaps some black magic in Cython? "i" is being used, and the
generated code interprets this as a return flag for errors, with
checks like if(unlikely(__pyx_1 ==  -1))) ...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Oct  2 06:06:42 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 02 Oct 2008 16:06:42 +1200
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <e7ba66e40810012058q6053b102y236255c936b965b9@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<48E43493.4040009@canterbury.ac.nz>
	<e7ba66e40810012058q6053b102y236255c936b965b9@mail.gmail.com>
Message-ID: <48E448D2.1080506@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Mmm.. Perhaps some black magic in Cython? "i" is being used, and the
> generated code interprets this as a return flag for errors, with
> checks like if(unlikely(__pyx_1 ==  -1))) ...

On further investigation, it seems it doesn't make a
difference in this situation. But "r" is conceptually
more correct, and it may become important in a future
version of Pyrex.

-- 
Greg

From ondrej at certik.cz  Thu Oct  2 09:39:10 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 2 Oct 2008 09:39:10 +0200
Subject: [Cython] Does not accept Unicode docstrings
Message-ID: <85b5c3130810020039x4f11e7a6md28e8a747875e2b0@mail.gmail.com>

Hi,

Cython doesn't seem to accept unicode strings:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=500847


Ondrej

From dalcinl at gmail.com  Thu Oct  2 15:11:50 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Oct 2008 10:11:50 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <48E448D2.1080506@canterbury.ac.nz>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<48E43493.4040009@canterbury.ac.nz>
	<e7ba66e40810012058q6053b102y236255c936b965b9@mail.gmail.com>
	<48E448D2.1080506@canterbury.ac.nz>
Message-ID: <e7ba66e40810020611m1a0f380elfcec472b287c465a@mail.gmail.com>

On Thu, Oct 2, 2008 at 1:06 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> Mmm.. Perhaps some black magic in Cython? "i" is being used, and the
>> generated code interprets this as a return flag for errors, with
>> checks like if(unlikely(__pyx_1 ==  -1))) ...
>
> On further investigation, it seems it doesn't make a
> difference in this situation. But "r" is conceptually
> more correct, and it may become important in a future
> version of Pyrex.
>

OK, good to know! Robert, should we fix this in Cython?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Oct  2 15:13:38 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Oct 2008 10:13:38 -0300
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <85b5c3130810020039x4f11e7a6md28e8a747875e2b0@mail.gmail.com>
References: <85b5c3130810020039x4f11e7a6md28e8a747875e2b0@mail.gmail.com>
Message-ID: <e7ba66e40810020613k11f71d26v46c483bbdc61e8ed@mail.gmail.com>

Did you try to write your docstrings as  u"somestuff"", ie, use the u"" prefix?

On Thu, Oct 2, 2008 at 4:39 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> Cython doesn't seem to accept unicode strings:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=500847
>
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Oct  2 15:38:41 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 02 Oct 2008 15:38:41 +0200
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <85b5c3130810020039x4f11e7a6md28e8a747875e2b0@mail.gmail.com>
References: <85b5c3130810020039x4f11e7a6md28e8a747875e2b0@mail.gmail.com>
Message-ID: <48E4CEE1.2060402@student.matnat.uio.no>

Ondrej Certik wrote:
> Hi,
>
> Cython doesn't seem to accept unicode strings:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=500847
>
>   
I just tried it against cython-devel and it worked just fine as long as 
the coding: utf-8 header is declared. This is consistent with how Python 
works (at least Python 2.5).

The user might have used a different encoding etc. but without a better 
bug report (binary file submitted, encoding of system, encoding 
behaviour of Python for the given file etc. etc.) it is difficult to say 
if anything really is wrong here.

Dag Sverre

From uschmitt at mineway.de  Thu Oct  2 17:25:17 2008
From: uschmitt at mineway.de (Uwe Schmitt)
Date: Thu, 02 Oct 2008 17:25:17 +0200
Subject: [Cython] resolving name conflict -- does not work for enums !?
Message-ID: <48E4E7DD.1040906@mineway.de>

Hi,

in order to resolve a name conflict concerning enums I followed the 
description from
http://docs.cython.org/docs/sharing_declarations.html#using-cimport-to-resolve-naming-conflicts

Here comes my code:

    ----- minimal.pxd ------------------------

        cdef extern from "minimal.h":

            cdef enum test:
                A

    ----- minimal.pyx -------------------------

        cimport minimal

        A = minimal.A

    -------------------------------------------


which does not work. Cython says "Assignment to non-lvalue 'A'"
If I modify it to "cdef int A = minimal.A" I get "'A' redeclared".

So, how can I provide the user of my module the same enums
as defined internally ???

Greetings, Uwe

-- 
Dr. rer. nat. Uwe Schmitt
F&E Mathematik
 
mineway GmbH
Science Park 2
D-66123 Saarbr?cken
 
Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376
 
uschmitt at mineway.de
www.mineway.de
 
Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081002/dd889387/attachment.htm 

From pagx13 at yahoo.gr  Thu Oct  2 17:46:14 2008
From: pagx13 at yahoo.gr (P M)
Date: Thu, 2 Oct 2008 18:46:14 +0300
Subject: [Cython] Does not accept Unicode docstrings
Message-ID: <200810021846.14875.pagx13@yahoo.gr>

Hello, I am the original reporter of the bug with description "Does not accept 
Unicode docstrings" in the Debian BTS. I tried using the encoding: UTF-8 
header and using the u'' prefix, but the problem still persists. I'll try to 
post more information with this email.

$ locale
LANG=el_GR.UTF-8
LC_CTYPE="el_GR.UTF-8"
LC_NUMERIC="el_GR.UTF-8"
LC_TIME="el_GR.UTF-8"
LC_COLLATE="el_GR.UTF-8"
LC_MONETARY="el_GR.UTF-8"
LC_MESSAGES="el_GR.UTF-8"
LC_PAPER="el_GR.UTF-8"
LC_NAME="el_GR.UTF-8"
LC_ADDRESS="el_GR.UTF-8"
LC_TELEPHONE="el_GR.UTF-8"
LC_MEASUREMENT="el_GR.UTF-8"
LC_IDENTIFICATION="el_GR.UTF-8"
LC_ALL=
$ head -n1 `which cython`
#!/usr/bin/python
$ python --version
Python 2.5.2
$ cython --version
Cython version 0.9.8
$ file bug.pyx
bug.pyx: UTF-8 Unicode text
$ cat bug.pyx
# -*- encoding: UTF-8 -*-
def hello():
        u"""???? ???, ?????!"""
        print 'Hello, world!'
$ cython bug.pyx
Traceback (most recent call last):
  File "/usr/bin/cython", line 8, in <module>
    main(command_line = 1)
  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 527, 
in main
    result = compile(sources, options)
  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 505, 
in compile
    return compile_multiple(source, options)
  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 472, 
in compile_multiple
    result = context.compile(source, options)
  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 327, 
in compile
    tree.process_implementation(scope, options, result)
  File "/var/lib/python-support/python2.5/Cython/Compiler/ModuleNode.py", line 
59, in process_implementation
    self.generate_c_code(env, options, result)
  File "/var/lib/python-support/python2.5/Cython/Compiler/ModuleNode.py", line 
243, in generate_c_code
    self.body.generate_function_definitions(env, code, options.transforms)
  File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 839, 
in generate_function_definitions
    with_pymethdef = env.is_py_class_scope)
  File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 
1442, in generate_function_header
    self.entry.doc))
  File "/var/lib/python-support/python2.5/Cython/Compiler/Code.py", line 52, 
in putln
    self.put(code)
  File "/var/lib/python-support/python2.5/Cython/Compiler/Code.py", line 69, 
in put
    self._write(code)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-41: 
ordinal not in range(128)
$ echo $?
1
$

I hope this helps :)

From dfdeshom at gmail.com  Thu Oct  2 20:06:48 2008
From: dfdeshom at gmail.com (didier deshommes)
Date: Thu, 2 Oct 2008 14:06:48 -0400
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
Message-ID: <4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>

I've attached a patch for cdef'ed sets. It should work whether you're
running 2.3 or 2.5 (In 2.3 nothing will happen and "cdef set s" should
give you a syntax error). I also had to modify  PyrexTypes.py to make
sure that PyAnySet_CheckExact() was being passed, instead of
PySet_CheckExact() (there are 2 ways to check for the type of a set:
PyAnySet_CheckExact and PyFrozenSet_CheckExact). Reviews welcome!

didier

On Wed, Oct 1, 2008 at 11:50 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On Thu, Oct 2, 2008 at 12:28 AM, didier deshommes <dfdeshom at gmail.com> wrote:
>>
>> Ok, I've got a patch and I've looked at the generated C code, compiled
>> the module and verified the behavior of the function. How do I write
>> tests for cython? Do I just edit tests/run/dict.pyx and look at the
>> generated code?
>
> More or less, yes. Write a 'set.pyx' file like this:
>
> __doc__ = u"""
>>>> test_set()
> True
> """
>
> def test_set():
>    cdef set s1 = set()
>    cdef set s2 = set()
>    # then do stuff with s1 and s2, like s1.add(1)
>    # finally return True
>    return True
>
>
> Of course, I do not know if this can be readily incorporated in Cython
> test suite, provided that this code is going to fail  in the case of a
> Python 2.3 runtime.
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sets.txt
Url: http://codespeak.net/pipermail/cython-dev/attachments/20081002/f4eb378a/attachment-0001.txt 

From dalcinl at gmail.com  Thu Oct  2 20:40:49 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Oct 2008 15:40:49 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
Message-ID: <e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>

n Thu, Oct 2, 2008 at 3:06 PM, didier deshommes <dfdeshom at gmail.com> wrote:
> I've attached a patch for cdef'ed sets. It should work whether you're
> running 2.3 or 2.5 (In 2.3 nothing will happen and "cdef set s" should
> give you a syntax error).

Please, clarify my something: At wich stage the syntax error is shown
for Python 2.3? Do you mean the C compiler fails? Or is Cython
failing?

> I also had to modify  PyrexTypes.py to make
> sure that PyAnySet_CheckExact() was being passed, instead of
> PySet_CheckExact() (there are 2 ways to check for the type of a set:
> PyAnySet_CheckExact and PyFrozenSet_CheckExact). Reviews welcome!

Your patch seems fine, but I believe the fix for 'type_test_code()'
method is wrong for the case of 'frozenset', 'frozenset'.capitalize()
-> 'Frozenset', and you need 'FrozenSet', right? I would add these two
lines

elif type == 'Frozenset'
    type = 'FrozenSet'

As a final note, It would be great if you could also provide a test
(pyx file for compile and run) for all this!




> On Wed, Oct 1, 2008 at 11:50 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> On Thu, Oct 2, 2008 at 12:28 AM, didier deshommes <dfdeshom at gmail.com> wrote:
>>>
>>> Ok, I've got a patch and I've looked at the generated C code, compiled
>>> the module and verified the behavior of the function. How do I write
>>> tests for cython? Do I just edit tests/run/dict.pyx and look at the
>>> generated code?
>>
>> More or less, yes. Write a 'set.pyx' file like this:
>>
>> __doc__ = u"""
>>>>> test_set()
>> True
>> """
>>
>> def test_set():
>>    cdef set s1 = set()
>>    cdef set s2 = set()
>>    # then do stuff with s1 and s2, like s1.add(1)
>>    # finally return True
>>    return True
>>
>>
>> Of course, I do not know if this can be readily incorporated in Cython
>> test suite, provided that this code is going to fail  in the case of a
>> Python 2.3 runtime.
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dfdeshom at gmail.com  Thu Oct  2 22:25:48 2008
From: dfdeshom at gmail.com (didier deshommes)
Date: Thu, 2 Oct 2008 16:25:48 -0400
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
Message-ID: <4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>

On Thu, Oct 2, 2008 at 2:40 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Please, clarify my something: At wich stage the syntax error is shown
> for Python 2.3? Do you mean the C compiler fails? Or is Cython
> failing?
>

Cython fails with a syntax error on "cdef set s". As an aside, if you
have this in py2.3:

"""
#py 2.3
cdef object s
s =set()
"""

Everything will compile fine but the module will fail to load when you
try to import it with this message:
"undefined symbol: PySet_Type". That's because Symtab.py assumes that
sets are available in py2.3 also, and includes it in `builtin_entries`
(a dict).Should we work around that or should we let users shoot
themselves in the foot?

>> I also had to modify  PyrexTypes.py to make
>> sure that PyAnySet_CheckExact() was being passed, instead of
>> PySet_CheckExact() (there are 2 ways to check for the type of a set:
>> PyAnySet_CheckExact and PyFrozenSet_CheckExact). Reviews welcome!
>
> Your patch seems fine, but I believe the fix for 'type_test_code()'
> method is wrong for the case of 'frozenset', 'frozenset'.capitalize()
> -> 'Frozenset', and you need 'FrozenSet', right? I would add these two
> lines
>
> elif type == 'Frozenset'
>    type = 'FrozenSet'

Right.

>
> As a final note, It would be great if you could also provide a test
> (pyx file for compile and run) for all this!

Ok, will do.

didier

From dagss at student.matnat.uio.no  Thu Oct  2 22:58:47 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 02 Oct 2008 22:58:47 +0200
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>	<loom.20081001T210058-967@post.gmane.org>	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
Message-ID: <48E53607.4080908@student.matnat.uio.no>

didier deshommes wrote:
> On Thu, Oct 2, 2008 at 2:40 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> Please, clarify my something: At wich stage the syntax error is shown
>> for Python 2.3? Do you mean the C compiler fails? Or is Cython
>> failing?
>>
> 
> Cython fails with a syntax error on "cdef set s". As an aside, if you
> have this in py2.3:
> 
> """
> #py 2.3
> cdef object s
> s =set()
> """
> 
> Everything will compile fine but the module will fail to load when you
> try to import it with this message:
> "undefined symbol: PySet_Type". That's because Symtab.py assumes that
> sets are available in py2.3 also, and includes it in `builtin_entries`
> (a dict).Should we work around that or should we let users shoot
> themselves in the foot?

My five cents: I'm -1 on making emulating later Python versions in 
earlier ones in most cases, and definitely this one. We are developing 
Cython for the present and future, and Python 2.3 is starting to belong 
to the past.

The Cython compiler should run under Python 2.3 and create Python 
2.3-compatible code, but that is different from backporting builtins 
like "set".

-- 
Dag Sverre

From ggellner at uoguelph.ca  Thu Oct  2 23:00:40 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 2 Oct 2008 17:00:40 -0400
Subject: [Cython] array assignment
Message-ID: <20081002210040.GA15290@encolpuis>

is there any way to assign an cdef C array at assignment time?

I want something like:

    cdef double a[4] = {0.5, 0.3, 0.1, 0.1}

as anything like this possible in Cython? Or do I need to do explicit
initialization.

Gabriel

From ggellner at uoguelph.ca  Thu Oct  2 23:03:06 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 2 Oct 2008 17:03:06 -0400
Subject: [Cython] array assignment
In-Reply-To: <20081002210040.GA15290@encolpuis>
References: <20081002210040.GA15290@encolpuis>
Message-ID: <20081002210306.GA15396@encolpuis>

On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
> is there any way to assign an cdef C array at assignment time?
> 
> I want something like:
> 
>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
> 
> as anything like this possible in Cython? Or do I need to do explicit
> initialization.
> 
And if it is explicit is there any way to not just do (can I do some kind of 
block assignment?)

    a[0] = 0.5
        ...
    a[3] = 0.1

Looking for magic :-)

Gabriel

From dalcinl at gmail.com  Thu Oct  2 23:41:51 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Oct 2008 18:41:51 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
Message-ID: <e7ba66e40810021441l6def9df8w9a635d8f7e40b173@mail.gmail.com>

On Thu, Oct 2, 2008 at 5:25 PM, didier deshommes <dfdeshom at gmail.com> wrote:
>
> Cython fails with a syntax error on "cdef set s". As an aside, if you
> have this in py2.3:
>
> """
> #py 2.3
> cdef object s
> s =set()
> """

OK, that's a bit strange, but should be related to the fact that
Cython access the builtin module (I remember a post from Dag about
this issue). IMHO, a 'cdef set s' should not fail with a syntax error.

> Everything will compile fine but the module will fail to load when you
> try to import it with this message:
> "undefined symbol: PySet_Type". That's because Symtab.py assumes that
> sets are available in py2.3 also, and includes it in `builtin_entries`
> (a dict).Should we work around that or should we let users shoot
> themselves in the foot?

This is expected and we are fine with that.

>>> I also had to modify  PyrexTypes.py to make
>>> sure that PyAnySet_CheckExact() was being passed, instead of
>>> PySet_CheckExact() (there are 2 ways to check for the type of a set:
>>> PyAnySet_CheckExact and PyFrozenSet_CheckExact). Reviews welcome!
>>
>> Your patch seems fine, but I believe the fix for 'type_test_code()'
>> method is wrong for the case of 'frozenset', 'frozenset'.capitalize()
>> -> 'Frozenset', and you need 'FrozenSet', right? I would add these two
>> lines
>>
>> elif type == 'Frozenset'
>>    type = 'FrozenSet'
>
> Right.
>
>>
>> As a final note, It would be great if you could also provide a test
>> (pyx file for compile and run) for all this!
>
> Ok, will do.
>

OK, send the final patch (and possibly the test cases) when you finish it.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Oct  2 23:45:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 2 Oct 2008 18:45:57 -0300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <48E53607.4080908@student.matnat.uio.no>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
	<48E53607.4080908@student.matnat.uio.no>
Message-ID: <e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>

On Thu, Oct 2, 2008 at 5:58 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> My five cents: I'm -1 on making emulating later Python versions in
> earlier ones in most cases, and definitely this one. We are developing
> Cython for the present and future, and Python 2.3 is starting to belong
> to the past.

Me to, -1 on this...

> The Cython compiler should run under Python 2.3 and create Python
> 2.3-compatible code, but that is different from backporting builtins
> like "set".

However, Cython-the-compiler should not fail with syntax error if you
do 'cdef set s', right? This indicates that builtin module is being
used in the Cython codebase, and that is a potential source of
problem. In sort, I believe if you run Cython with python2.3, the
generated source should be valid Py3 C code, even if something like
'set' is being used. I'm missing something here?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dfdeshom at gmail.com  Fri Oct  3 00:00:17 2008
From: dfdeshom at gmail.com (didier deshommes)
Date: Thu, 2 Oct 2008 18:00:17 -0400
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
	<48E53607.4080908@student.matnat.uio.no>
	<e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>
Message-ID: <4db014670810021500p49da5c61w8e54d0378997d699@mail.gmail.com>

On Thu, Oct 2, 2008 at 5:45 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> However, Cython-the-compiler should not fail with syntax error if you
> do 'cdef set s', right? This indicates that builtin module is being
> used in the Cython codebase, and that is a potential source of
> problem. In sort, I believe if you run Cython with python2.3, the
> generated source should be valid Py3 C code, even if something like
> 'set' is being used. I'm missing something here?

To be clear the "syntax error on cdef set s" was introduced by *my*
code changes: I just removed set and frozenset whenever
hasattr(types, 'GetSetDescriptorType') was False (in Builtin.py). This
can be readily corrected and will not break anything. Of course if you
 import this with py2.3, you'll get a missing symbol error. Sorry if
there was any confusion about this.

didier

>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From greg.ewing at canterbury.ac.nz  Fri Oct  3 03:37:14 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 03 Oct 2008 14:37:14 +1300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <e7ba66e40810020611m1a0f380elfcec472b287c465a@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<48E43493.4040009@canterbury.ac.nz>
	<e7ba66e40810012058q6053b102y236255c936b965b9@mail.gmail.com>
	<48E448D2.1080506@canterbury.ac.nz>
	<e7ba66e40810020611m1a0f380elfcec472b287c465a@mail.gmail.com>
Message-ID: <48E5774A.3090704@canterbury.ac.nz>

Lisandro Dalcin wrote:
> On Thu, Oct 2, 2008 at 1:06 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>> But "r" is conceptually
>> more correct, and it may become important in a future
>> version of Pyrex.
> 
> OK, good to know! Robert, should we fix this in Cython?

It occurred to me while investigating this that there's
a slight difference between the optimised version of
list.append et al and the Python version: the optimised
version will return 0 if you use it in a context where
the return value is used, vs. None for the Python
version.

Not sure if anyone will care enough about that to be
worth fixing it, though.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Oct  3 03:47:37 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 03 Oct 2008 14:47:37 +1300
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
Message-ID: <48E579B9.8000408@canterbury.ac.nz>

didier deshommes wrote:

> Cython fails with a syntax error on "cdef set s".

The reason for the syntax error is that the parser doesn't
suspect that 'set' might be a type name.

Not sure about Cython, but in Pyrex the parser's list of
possible type names is seeded from the tables in Builtin.py.
If 'set' is declared there as something of type 'type',
that should fix the syntax error.

(The need for the parser to know about type names is a
bit hacky, but C's screwy declaration syntax makes it
hard to avoid when using an LL(1) parser.)

-- 
Greg

From dfdeshom at gmail.com  Fri Oct  3 06:56:19 2008
From: dfdeshom at gmail.com (didier deshommes)
Date: Fri, 3 Oct 2008 00:56:19 -0400
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <e7ba66e40810021441l6def9df8w9a635d8f7e40b173@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
	<e7ba66e40810021441l6def9df8w9a635d8f7e40b173@mail.gmail.com>
Message-ID: <4db014670810022156m4194e44elc8e8166bb95a6397@mail.gmail.com>

On Thu, Oct 2, 2008 at 5:41 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> OK, send the final patch (and possibly the test cases) when you finish it.

Ok, here is the final patch, with some test cases.

didier
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: set.txt
Url: http://codespeak.net/pipermail/cython-dev/attachments/20081003/bfbbccf8/attachment.txt 

From ondrej at certik.cz  Fri Oct  3 07:31:44 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 3 Oct 2008 07:31:44 +0200
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <200810021846.14875.pagx13@yahoo.gr>
References: <200810021846.14875.pagx13@yahoo.gr>
Message-ID: <85b5c3130810022231l1dfebd29ga6f9370aa0da122e@mail.gmail.com>

On Thu, Oct 2, 2008 at 5:46 PM, P M <pagx13 at yahoo.gr> wrote:
> Hello, I am the original reporter of the bug with description "Does not accept
> Unicode docstrings" in the Debian BTS. I tried using the encoding: UTF-8
> header and using the u'' prefix, but the problem still persists. I'll try to
> post more information with this email.
>
> $ locale
> LANG=el_GR.UTF-8
> LC_CTYPE="el_GR.UTF-8"
> LC_NUMERIC="el_GR.UTF-8"
> LC_TIME="el_GR.UTF-8"
> LC_COLLATE="el_GR.UTF-8"
> LC_MONETARY="el_GR.UTF-8"
> LC_MESSAGES="el_GR.UTF-8"
> LC_PAPER="el_GR.UTF-8"
> LC_NAME="el_GR.UTF-8"
> LC_ADDRESS="el_GR.UTF-8"
> LC_TELEPHONE="el_GR.UTF-8"
> LC_MEASUREMENT="el_GR.UTF-8"
> LC_IDENTIFICATION="el_GR.UTF-8"
> LC_ALL=
> $ head -n1 `which cython`
> #!/usr/bin/python
> $ python --version
> Python 2.5.2
> $ cython --version
> Cython version 0.9.8
> $ file bug.pyx
> bug.pyx: UTF-8 Unicode text
> $ cat bug.pyx
> # -*- encoding: UTF-8 -*-
> def hello():
>        u"""???? ???, ?????!"""
>        print 'Hello, world!'
> $ cython bug.pyx
> Traceback (most recent call last):
>  File "/usr/bin/cython", line 8, in <module>
>    main(command_line = 1)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 527,
> in main
>    result = compile(sources, options)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 505,
> in compile
>    return compile_multiple(source, options)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 472,
> in compile_multiple
>    result = context.compile(source, options)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 327,
> in compile
>    tree.process_implementation(scope, options, result)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/ModuleNode.py", line
> 59, in process_implementation
>    self.generate_c_code(env, options, result)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/ModuleNode.py", line
> 243, in generate_c_code
>    self.body.generate_function_definitions(env, code, options.transforms)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 839,
> in generate_function_definitions
>    with_pymethdef = env.is_py_class_scope)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line
> 1442, in generate_function_header
>    self.entry.doc))
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Code.py", line 52,
> in putln
>    self.put(code)
>  File "/var/lib/python-support/python2.5/Cython/Compiler/Code.py", line 69,
> in put
>    self._write(code)
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-41:
> ordinal not in range(128)
> $ echo $?
> 1
> $
>
> I hope this helps :)

Thanks for the clarification.

Ondrej

From robertwb at math.washington.edu  Fri Oct  3 10:01:50 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 01:01:50 -0700
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <48E448D2.1080506@canterbury.ac.nz>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<e7ba66e40810011208p7bbb3431mb48a52d1e80e9ece@mail.gmail.com>
	<loom.20081001T193537-418@post.gmane.org>
	<e7ba66e40810011310wfdc578bu20b1e0c0045fe7b4@mail.gmail.com>
	<loom.20081001T210058-967@post.gmane.org>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<48E43493.4040009@canterbury.ac.nz>
	<e7ba66e40810012058q6053b102y236255c936b965b9@mail.gmail.com>
	<48E448D2.1080506@canterbury.ac.nz>
Message-ID: <B5903639-A3E6-4730-907B-EFEB3EE432CE@math.washington.edu>

On Oct 1, 2008, at 9:06 PM, Greg Ewing wrote:

> Lisandro Dalcin wrote:
>
>> Mmm.. Perhaps some black magic in Cython? "i" is being used, and the
>> generated code interprets this as a return flag for errors, with
>> checks like if(unlikely(__pyx_1 ==  -1))) ...
>
> On further investigation, it seems it doesn't make a
> difference in this situation. But "r" is conceptually
> more correct, and it may become important in a future
> version of Pyrex.

It seems an "r" means the returned result has no significance other  
than whether or not an error occurred, while an "i" is used to both  
flag an error and return a value. I suppose we should fix this  
(though it is rather low priority).

As you mentioned, it could be used to make the optimized append  
return None as well instead of 0 (if anyone cares to look at it) by  
having the c_returncode_type coerce to an object using None.

- Robert


From robertwb at math.washington.edu  Fri Oct  3 10:24:30 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 01:24:30 -0700
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>
	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>
	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>
	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>
	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>
	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>
	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>
	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>
	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>
	<48E53607.4080908@student.matnat.uio.no>
	<e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>
Message-ID: <4770B9DF-1E5F-41A5-BFE5-BE389A035696@math.washington.edu>

On Oct 2, 2008, at 2:45 PM, Lisandro Dalcin wrote:

> On Thu, Oct 2, 2008 at 5:58 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> My five cents: I'm -1 on making emulating later Python versions in
>> earlier ones in most cases, and definitely this one. We are  
>> developing
>> Cython for the present and future, and Python 2.3 is starting to  
>> belong
>> to the past.
>
> Me to, -1 on this...

Some stuff (for example buffers, the with statement, and the b"byte  
string" syntax) I believe are worth backporting. Python 2.3 will  
continue to be a target for us (at least) as long as it is a target  
for NumPy.

Backporting builtins, however, I don't think is usually worth the  
effort. If one wishes to use more advanced builtins in ones sources,  
then one limits the platform it can run on. This is exactly like .py  
files.

>> The Cython compiler should run under Python 2.3 and create Python
>> 2.3-compatible code, but that is different from backporting builtins
>> like "set".
>
> However, Cython-the-compiler should not fail with syntax error if you
> do 'cdef set s', right? This indicates that builtin module is being
> used in the Cython codebase, and that is a potential source of
> problem. In sort, I believe if you run Cython with python2.3, the
> generated source should be valid Py3 C code, even if something like
> 'set' is being used. I'm missing something here?

Let X <= Y. If you run Cython with Python X, then it should compile  
against and run under Python Y. However, if you run Cython with  
Python Y, it may not compile against or run under Python X (depending  
on if you have used features introduced after X). Without backporting  
every new feature back to 2.3, this is the best we can hope to do.

Letting set be defined no matter what violates this principle. Thus  
if they have a Python 2.3 toolchain, they can (too easily) produce  
code that they cannot use. Furthermore, the error will be some  
obscure (to most people at least) message in the C compile. Much  
better to raise an error during the Cython run. That way if Cython  
succeeds, the C compile will succeed and the module will work (modulo  
any bugs of course).

In this case I think Didier's original patch was the right approach,  
though we will have to adjust the testing framework to account for  
it. I might suggest that testing

import __builtin__
if hasattr(__builtin__, 'set'):
     ....

is a bit more intuitive than the test you had before.

- Robert


From robertwb at math.washington.edu  Fri Oct  3 10:32:55 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 01:32:55 -0700
Subject: [Cython] Robert, please help!
In-Reply-To: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
Message-ID: <D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>

On Oct 1, 2008, at 6:42 PM, Lisandro Dalcin wrote:

> Robert, in this changeset
>
> changeset:   1184:515f94dc16f2
> user:        Robert Bradshaw <robertwb at math.washington.edu>
> date:        Mon Sep 29 19:08:45 2008 -0700
> summary:     Refactoring of type parsing
>
> you cannot parse C function typedef like this:
>
> cdef extern from *:
>     void (somefunc)(int)
>
> Please note that this declaration is not of the form "void
> (*somefuncptr)(int)", and AFAIK the former is valid C code.

Does it have the same meaning, or is there something more subtle  
going on? We don't accept all valid C code, for example

     int foo(void)

is allowed in C but not in Cython.

> I was looking at the changes you pushed, but I was unable to discover
> what's going on. So I really need your help here.

I needed to remove the dependancy the parser had on knowing all type  
names, so in some places it needs to look ahead to determine if it's  
looking at a type or an expression. If this is the only bug  
introduced, than I'm willing to accept that, but are there other  
statements that are now illegal?

- Robert


From robertwb at math.washington.edu  Fri Oct  3 10:36:25 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 01:36:25 -0700
Subject: [Cython] PATCH: fix pyximport installation as a package
In-Reply-To: <e7ba66e40810010652u77fe0376ud388b549a2ea00e@mail.gmail.com>
References: <e7ba66e40810010652u77fe0376ud388b549a2ea00e@mail.gmail.com>
Message-ID: <1A8FA402-AB2F-4F8A-A77A-310555E58250@math.washington.edu>

Looks good, thanks.

On Oct 1, 2008, at 6:52 AM, Lisandro Dalcin wrote:

> This should fix the installation of pyximport as a package. Please,
> review and apply.
>
>
>
> --  
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0) 
> 342-451.1594<pyximport.diff>__________________________________________ 
> _____
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri Oct  3 11:01:58 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 02:01:58 -0700
Subject: [Cython] resolving name conflict -- does not work for enums !?
In-Reply-To: <48E4E7DD.1040906@mineway.de>
References: <48E4E7DD.1040906@mineway.de>
Message-ID: <B227506C-CF43-4C76-BEBC-7A63FF47783E@math.washington.edu>

On Oct 2, 2008, at 8:25 AM, Uwe Schmitt wrote:

> Hi,
>
> in order to resolve a name conflict concerning enums I followed the  
> description from
> http://docs.cython.org/docs/sharing_declarations.html#using-cimport- 
> to-resolve-naming-conflicts
>
> Here comes my code:
>
> ----- minimal.pxd ------------------------
>
>     cdef extern from "minimal.h":
>
>         cdef enum test:
>             A
>
> ----- minimal.pyx -------------------------
>
>     cimport minimal
>
>     A = minimal.A
>
> -------------------------------------------
>
> which does not work. Cython says "Assignment to non-lvalue 'A'"
> If I modify it to "cdef int A = minimal.A" I get "'A' redeclared".

minimal.pxd and minimal.pyx have a special relationship to each  
other, specifically, minimal.pxd defines the interface of the things  
implemented in minimal.pyx (for use from other modules). The  
"imaginary" part of "imaginary module" is important, as once you have  
defined a .pyx file the module is no longer imaginary.

> So, how can I provide the user of my module the same enums
> as defined internally ???

Simply rename minimal.pxd to something else.

As a side note, for non-extern enums, you can write

cdef public enum:
     ...

and it will automatically export the enum variables to your (python- 
level) module scope. Unfortunately, it doesn't yet work for extern  
enums http://trac.cython.org/cython_trac/ticket/92

- Robert

From robertwb at math.washington.edu  Fri Oct  3 11:07:39 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 02:07:39 -0700
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <200810021846.14875.pagx13@yahoo.gr>
References: <200810021846.14875.pagx13@yahoo.gr>
Message-ID: <F7F42678-E0B1-405C-ADBB-EB26DEE72C41@math.washington.edu>

Thanks for the info. I'm not sure why this isn't working for you,  
could you attach the actual bug.pyx file (preferably zipped, to be  
sure that no mail servers mangle it at all)?

- Robert

On Oct 2, 2008, at 8:46 AM, P M wrote:

> Hello, I am the original reporter of the bug with description "Does  
> not accept
> Unicode docstrings" in the Debian BTS. I tried using the encoding:  
> UTF-8
> header and using the u'' prefix, but the problem still persists.  
> I'll try to
> post more information with this email.
>
> $ locale
> LANG=el_GR.UTF-8
> LC_CTYPE="el_GR.UTF-8"
> LC_NUMERIC="el_GR.UTF-8"
> LC_TIME="el_GR.UTF-8"
> LC_COLLATE="el_GR.UTF-8"
> LC_MONETARY="el_GR.UTF-8"
> LC_MESSAGES="el_GR.UTF-8"
> LC_PAPER="el_GR.UTF-8"
> LC_NAME="el_GR.UTF-8"
> LC_ADDRESS="el_GR.UTF-8"
> LC_TELEPHONE="el_GR.UTF-8"
> LC_MEASUREMENT="el_GR.UTF-8"
> LC_IDENTIFICATION="el_GR.UTF-8"
> LC_ALL=
> $ head -n1 `which cython`
> #!/usr/bin/python
> $ python --version
> Python 2.5.2
> $ cython --version
> Cython version 0.9.8
> $ file bug.pyx
> bug.pyx: UTF-8 Unicode text
> $ cat bug.pyx
> # -*- encoding: UTF-8 -*-
> def hello():
>         u"""???? ???, ?????!"""
>         print 'Hello, world!'
> $ cython bug.pyx
> Traceback (most recent call last):
>   File "/usr/bin/cython", line 8, in <module>
>     main(command_line = 1)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py",  
> line 527,
> in main
>     result = compile(sources, options)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py",  
> line 505,
> in compile
>     return compile_multiple(source, options)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py",  
> line 472,
> in compile_multiple
>     result = context.compile(source, options)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py",  
> line 327,
> in compile
>     tree.process_implementation(scope, options, result)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/ 
> ModuleNode.py", line
> 59, in process_implementation
>     self.generate_c_code(env, options, result)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/ 
> ModuleNode.py", line
> 243, in generate_c_code
>     self.body.generate_function_definitions(env, code,  
> options.transforms)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/ 
> Nodes.py", line 839,
> in generate_function_definitions
>     with_pymethdef = env.is_py_class_scope)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/ 
> Nodes.py", line
> 1442, in generate_function_header
>     self.entry.doc))
>   File "/var/lib/python-support/python2.5/Cython/Compiler/Code.py",  
> line 52,
> in putln
>     self.put(code)
>   File "/var/lib/python-support/python2.5/Cython/Compiler/Code.py",  
> line 69,
> in put
>     self._write(code)
> UnicodeEncodeError: 'ascii' codec can't encode characters in  
> position 38-41:
> ordinal not in range(128)
> $ echo $?
> 1
> $
>
> I hope this helps :)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri Oct  3 11:14:14 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 02:14:14 -0700
Subject: [Cython] Asking for review: More result_code refactor etc.
In-Reply-To: <48E11D33.9050004@student.matnat.uio.no>
References: <48E11D33.9050004@student.matnat.uio.no>
Message-ID: <30D13E46-4CC3-44B6-8B39-7422D619B29E@math.washington.edu>

On Sep 29, 2008, at 11:23 AM, Dag Sverre Seljebotn wrote:

> Buffer access towards complex types (treating them as a struct of two
> floats) are now up in -dagss.
>
> In order to get there I had to do a couple of dangerous things  
> regarding
> result_code though. These patches are really small but someone with  
> more
> knowledge may have more or less confidence in them than I:

Sorry to take so long to answer this, but changes like this make me  
wary too. (I think they're *good* changes, they're just scary ones to  
try and pronounce safe).

> http://hg.cython.org/cython-dagss/rev/b2bcc32021fd
> http://hg.cython.org/cython-dagss/rev/a8a9e4c7d5bd

After looking at them (again), I say +1.

> Note that this should fix the CloneNode issue I talked about. (And all
> tests pass btw; yet this is the kind of thing that would cause
> horrendous failures in obscure cases which are likely not tested  
> for...)
>
> Also in -dagss I've introduces NewTempExprNode, which is a  
> transitionary
> class where the temp allocation is moved to code generation.  
> Ideally all
> ExprNodes should descend from this one instead (and then it should be
> folded back into ExprNode to complete the migration).

That sounds like a good idea.

> Some classes fail hard when they are changed to the new scheme. It  
> works
> for BinOpNode which now uses it. This should allow partial migration.
> (One must make sure a class can work with the new temp allocation  
> scheme
> before constructing them in a transform after analysis).

Good point.

- Robert


From robertwb at math.washington.edu  Fri Oct  3 11:26:09 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 02:26:09 -0700
Subject: [Cython] A small change in compiler directive implementation
In-Reply-To: <48D80266.6060701@student.matnat.uio.no>
References: <48D80266.6060701@student.matnat.uio.no>
Message-ID: <ED34485E-E88D-4F01-BB76-D3D519D22413@math.washington.edu>

On Sep 22, 2008, at 1:39 PM, Dag Sverre Seljebotn wrote:

> Just a note that I've changed some internals with compiler directives.
>
> Before each node would get an "options" attribute in a transform that
> ran pretty early. This was getting increasingly cumbersome to deal  
> with
> (as transforms needed to make sure options was copied over to new  
> nodes
> etc.). So now I've created a CompilerDirectivesNode that sit in the  
> tree
>   instead. Each recursive process then stores the directives these  
> sets
> in different places:
>
> - env.directives
> - code.globalstate.directives
> - For CythonTransform descendants, self.current_directives is
> automatically updated as long as you leave the default
> CompilerDirectivesNode visit alone.
>
> If anyone protests I'll take it upon me to back it out.

No, it looks good. I've even put it to use and it works nicely :).

- Robert



From robertwb at math.washington.edu  Fri Oct  3 11:26:44 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 02:26:44 -0700
Subject: [Cython] Issues porting to Cython
In-Reply-To: <1222142045.9872.19.camel@tachyon-laptop>
References: <1222121599.14956.21.camel@tachyon-laptop>
	<9D7C2A64-0D58-4B73-A36D-558A6E05C3B1@math.washington.edu>
	<1222142045.9872.19.camel@tachyon-laptop>
Message-ID: <836B3FDC-9453-4996-8E3F-D7AA4956644D@math.washington.edu>

On Sep 22, 2008, at 8:54 PM, Andrew Collette wrote:

>
>> No, this looks like a bug. http://trac.cython.org/cython_trac/ticket/
>> 73 For now, import your ctypedef types explicitly (functions and
>> classes should cimport fine).
>
> Thanks!  Since hdf5.pxd is 1154 lines and is purely external
> definitions, I'm getting around this by just including it in my
> module .pxd files.  This seems to work for now.

BTW, this is fixed in cython-devel.

- Robert


From dagss at student.matnat.uio.no  Fri Oct  3 12:16:40 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Oct 2008 12:16:40 +0200
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <4770B9DF-1E5F-41A5-BFE5-BE389A035696@math.washington.edu>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>	<48E53607.4080908@student.matnat.uio.no>	<e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>
	<4770B9DF-1E5F-41A5-BFE5-BE389A035696@math.washington.edu>
Message-ID: <48E5F108.50601@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 2, 2008, at 2:45 PM, Lisandro Dalcin wrote:
> 
>> On Thu, Oct 2, 2008 at 5:58 PM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> My five cents: I'm -1 on making emulating later Python versions in
>>> earlier ones in most cases, and definitely this one. We are  
>>> developing
>>> Cython for the present and future, and Python 2.3 is starting to  
>>> belong
>>> to the past.
>> Me to, -1 on this...
> 
> Some stuff (for example buffers, the with statement, and the b"byte  
> string" syntax) I believe are worth backporting. Python 2.3 will  
> continue to be a target for us (at least) as long as it is a target  
> for NumPy.

BTW: This changed a few days ago, when 1.2.0 was released: "Please note 
that NumPy 1.2.0 requires Python 2.4 or greater."

(I'm still +1 on Py2.3 compatability, just noting it.)

> Let X <= Y. If you run Cython with Python X, then it should compile  
> against and run under Python Y. However, if you run Cython with  
> Python Y, it may not compile against or run under Python X (depending  
> on if you have used features introduced after X). Without backporting  
> every new feature back to 2.3, this is the best we can hope to do.

I don't follow the logic here.

Please bear with me, this is important: One consequence is that there is 
currently no way to get access to Py3-only builtins from pyx files 
without using getattr (like "memoryview", unless it was added to 
Python2.6 last minute, I didn't fetch the release yet. But even if it 
was, I don't think there's any rule that all new Python 3.x builtins 
will be backported to 2.x?).

In my mind there's three different Python versions involved:

- A: The version of Python the user targets in the pyx file (when it 
comes to library and builtins; keeping the language part out of this).
- B: The version of Python that imports the compiled module. The user 
just has to make sure that this matches A in a reasonable fashion (the 
rules are non-trivial and not a simple >= or <=, as Py3 removes 
builtins, but Python lays these down for us).
- C: The version that runs Cython.

This discussion was about A and B -- that Cython won't allow users to 
code for Py2.4 and run under Py2.3.

You seem to be talking about C, which is something different. Cython 
just transforms code and could run on any Turing-machine, there's no 
natural reason it has to be connected with A or B.

We do currently require C >= B (which means that any Py3-only builtins 
are plain unavailable).

In my opinion, ideally C should be orthogonal to A and B because that 
seems to be less complicated from a release and bugfinding perspective 
(one less version to worry about in the mix) and it would fix the issue 
with accessing Py3-builtins for free, but I don't care very much.

> 
> Letting set be defined no matter what violates this principle. Thus  
> if they have a Python 2.3 toolchain, they can (too easily) produce  
> code that they cannot use. Furthermore, the error will be some  
> obscure (to most people at least) message in the C compile. Much  
> better to raise an error during the Cython run. That way if Cython  
> succeeds, the C compile will succeed and the module will work (modulo  
> any bugs of course).

You can guard against typing "zpi", but not (currently at least) against 
typing "os.path.exitst". I guess I tend to think of those as about the 
same thing -- i.e. accessing a builtin is just making use of the 
__builtin__ run-time module, it is just that we hacked some of them for 
speed purposes (as we in future may be able to do with os.path.exists 
too through the use of inline code in pxd files).

That was the opposing view. But I see that this is non-trivial, and as I 
said this is not an important matter to me, so I might as well end this 
mail with agreeing with you that what you present is the best course of 
action for now :-)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Oct  3 12:18:55 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Oct 2008 12:18:55 +0200
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <48E5F108.50601@student.matnat.uio.no>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>	<48E53607.4080908@student.matnat.uio.no>	<e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>	<4770B9DF-1E5F-41A5-BFE5-BE389A035696@math.washington.edu>
	<48E5F108.50601@student.matnat.uio.no>
Message-ID: <48E5F18F.20102@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> Let X <= Y. If you run Cython with Python X, then it should compile  
>> against and run under Python Y. However, if you run Cython with  
>> Python Y, it may not compile against or run under Python X (depending  
>> on if you have used features introduced after X). Without backporting  
>> every new feature back to 2.3, this is the best we can hope to do.
> 
> I don't follow the logic here.
> 
> Please bear with me, this is important: One consequence is that there is 
> currently no way to get access to Py3-only builtins from pyx files 
> without using getattr (like "memoryview", unless it was added to 
> Python2.6 last minute, I didn't fetch the release yet. But even if it 
> was, I don't think there's any rule that all new Python 3.x builtins 
> will be backported to 2.x?).
> 
> In my mind there's three different Python versions involved:
> 
> - A: The version of Python the user targets in the pyx file (when it 
> comes to library and builtins; keeping the language part out of this).
> - B: The version of Python that imports the compiled module. The user 
> just has to make sure that this matches A in a reasonable fashion (the 
> rules are non-trivial and not a simple >= or <=, as Py3 removes 
> builtins, but Python lays these down for us).
> - C: The version that runs Cython.
> 
> This discussion was about A and B -- that Cython won't allow users to 
> code for Py2.4 and run under Py2.3.
> 
> You seem to be talking about C, which is something different. Cython 
> just transforms code and could run on any Turing-machine, there's no 
> natural reason it has to be connected with A or B.
> 
> We do currently require C >= B (which means that any Py3-only builtins 
> are plain unavailable).
> 
> In my opinion, ideally C should be orthogonal to A and B because that 
> seems to be less complicated from a release and bugfinding perspective 
> (one less version to worry about in the mix) and it would fix the issue 
> with accessing Py3-builtins for free, but I don't care very much.

To be more concrete, the problems you mention could be fixed with a 
simple compiler flag (or directive) to Cython: --target-platform=3.0 or 
similar, to control the builtins. But, "there's no lack of good ideas".

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Oct  3 12:16:14 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 03:16:14 -0700
Subject: [Cython] array assignment
In-Reply-To: <20081002210306.GA15396@encolpuis>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
Message-ID: <33262D7C-13AF-401D-BF67-3B4E9B9A948C@math.washington.edu>

On Oct 2, 2008, at 2:03 PM, Gabriel Gellner wrote:

> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>> is there any way to assign an cdef C array at assignment time?
>>
>> I want something like:
>>
>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
>>
>> as anything like this possible in Cython? Or do I need to do explicit
>> initialization.
>>
> And if it is explicit is there any way to not just do (can I do  
> some kind of
> block assignment?)
>
>     a[0] = 0.5
>         ...
>     a[3] = 0.1
>
> Looking for magic :-)

Nothing yet, I put up http://trac.cython.org/cython_trac/ticket/93

Also, I would like to be able to use (literal) tuples/dicts to assign  
to structs (and vice-versa). Would non-literal tuples/dicts be going  
too far?

- Robert


From dagss at student.matnat.uio.no  Fri Oct  3 12:35:49 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Oct 2008 12:35:49 +0200
Subject: [Cython] array assignment
In-Reply-To: <33262D7C-13AF-401D-BF67-3B4E9B9A948C@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>
	<33262D7C-13AF-401D-BF67-3B4E9B9A948C@math.washington.edu>
Message-ID: <48E5F585.1040007@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 2, 2008, at 2:03 PM, Gabriel Gellner wrote:
> 
>> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>>> is there any way to assign an cdef C array at assignment time?
>>>
>>> I want something like:
>>>
>>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
>>>
>>> as anything like this possible in Cython? Or do I need to do explicit
>>> initialization.
>>>
>> And if it is explicit is there any way to not just do (can I do  
>> some kind of
>> block assignment?)
>>
>>     a[0] = 0.5
>>         ...
>>     a[3] = 0.1
>>
>> Looking for magic :-)
> 
> Nothing yet, I put up http://trac.cython.org/cython_trac/ticket/93
> 
> Also, I would like to be able to use (literal) tuples/dicts to assign  
> to structs (and vice-versa). Would non-literal tuples/dicts be going  
> too far?

I think a default "literal struct constructor" might be more fitting? 
And I see no reason it would need to be literal except for brevity of C 
code.

A problem is with mutability though, if it is done using C static 
allocation. I.e.:

def f(x):
     cdef int a[3] = [1,2,3]
     if x: a[2] = 1
     print a[2]

f(True) # prints 1...
f(False) # likely prints 1 too!

This is consistent with what you expect from C but very contrary to what 
Python users would expect.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Oct  3 12:44:03 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Oct 2008 12:44:03 +0200
Subject: [Cython] array assignment
In-Reply-To: <48E5F585.1040007@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<33262D7C-13AF-401D-BF67-3B4E9B9A948C@math.washington.edu>
	<48E5F585.1040007@student.matnat.uio.no>
Message-ID: <48E5F773.2030200@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> A problem is with mutability though, if it is done using C static 
> allocation. I.e.:
> 
> def f(x):
>      cdef int a[3] = [1,2,3]
>      if x: a[2] = 1
>      print a[2]
> 
> f(True) # prints 1...
> f(False) # likely prints 1 too!
> 
> This is consistent with what you expect from C but very contrary to what 
> Python users would expect.

No this is apparently plain wrong. Sorry.


-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Oct  3 12:38:17 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 03:38:17 -0700
Subject: [Cython] array assignment
In-Reply-To: <48E5F585.1040007@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>
	<33262D7C-13AF-401D-BF67-3B4E9B9A948C@math.washington.edu>
	<48E5F585.1040007@student.matnat.uio.no>
Message-ID: <F42A2CDC-E1F3-49D0-B0E3-95D2D500F3C4@math.washington.edu>

On Oct 3, 2008, at 3:35 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Oct 2, 2008, at 2:03 PM, Gabriel Gellner wrote:
>>
>>> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>>>> is there any way to assign an cdef C array at assignment time?
>>>>
>>>> I want something like:
>>>>
>>>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
>>>>
>>>> as anything like this possible in Cython? Or do I need to do  
>>>> explicit
>>>> initialization.
>>>>
>>> And if it is explicit is there any way to not just do (can I do
>>> some kind of
>>> block assignment?)
>>>
>>>     a[0] = 0.5
>>>         ...
>>>     a[3] = 0.1
>>>
>>> Looking for magic :-)
>>
>> Nothing yet, I put up http://trac.cython.org/cython_trac/ticket/93
>>
>> Also, I would like to be able to use (literal) tuples/dicts to assign
>> to structs (and vice-versa). Would non-literal tuples/dicts be going
>> too far?
>
> I think a default "literal struct constructor" might be more fitting?
> And I see no reason it would need to be literal except for brevity  
> of C
> code.

The literalness may not be reflected in the code (though it could  
impact more than the brevity of the C code, as assignment one-by-one  
may involve literals in the assembly, as opposed to a memcpy to  
assign an array at once.

> A problem is with mutability though, if it is done using C static
> allocation. I.e.:
>
> def f(x):
>      cdef int a[3] = [1,2,3]
>      if x: a[2] = 1
>      print a[2]
>
> f(True) # prints 1...
> f(False) # likely prints 1 too!
>
> This is consistent with what you expect from C but very contrary to  
> what
> Python users would expect.

No, I believe allocation is done on the stack.

- Robert



From greg.ewing at canterbury.ac.nz  Fri Oct  3 12:46:51 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 03 Oct 2008 22:46:51 +1200
Subject: [Cython] resolving name conflict -- does not work for enums !?
In-Reply-To: <48E4E7DD.1040906@mineway.de>
References: <48E4E7DD.1040906@mineway.de>
Message-ID: <48E5F81B.2000600@canterbury.ac.nz>

Uwe Schmitt wrote:

>     ----- minimal.pxd ------------------------
> 
>         cdef extern from "minimal.h":
> 
>             cdef enum test:
>                 A
> 
>     ----- minimal.pyx -------------------------
> 
>         cimport minimal
> 
>         A = minimal.A
> 
>     -------------------------------------------

You shouldn't be cimporting minimal in minimal.pyx. The files
minimal.pxd and minimal.pyx are two parts of the *same* module,
i.e. minimal. There's only one namespace, and you're trying
to define A as both a C constant and a Python global in that
namespace.

You need to put the definition of enum test into a different
module namespace, e.g.

   # mindefs.pxd
   cdef enum test:
     A

   # minimal.pyx:
   cimport mindefs
   A = mindefs.A

Note that you don't need a mindefs.pyx in this case -- you're
only using mindefs as a compile-time namespace.

-- 
Greg

From robertwb at math.washington.edu  Fri Oct  3 13:06:54 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 04:06:54 -0700
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <48E5F18F.20102@student.matnat.uio.no>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>	<48E53607.4080908@student.matnat.uio.no>	<e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>	<4770B9DF-1E5F-41A5-BFE5-BE389A035696@math.washington.edu>
	<48E5F108.50601@student.matnat.uio.no>
	<48E5F18F.20102@student.matnat.uio.no>
Message-ID: <3D47B019-2A7D-4743-B8E2-EE467DC9FA15@math.washington.edu>


On Oct 3, 2008, at 3:18 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> Let X <= Y. If you run Cython with Python X, then it should compile
>>> against and run under Python Y. However, if you run Cython with
>>> Python Y, it may not compile against or run under Python X  
>>> (depending
>>> on if you have used features introduced after X). Without  
>>> backporting
>>> every new feature back to 2.3, this is the best we can hope to do.
>>
>> I don't follow the logic here.

Sorry I wasn't very clear, thanks for splitting things up more below.

>> Please bear with me, this is important: One consequence is that  
>> there is
>> currently no way to get access to Py3-only builtins from pyx files
>> without using getattr (like "memoryview", unless it was added to
>> Python2.6 last minute, I didn't fetch the release yet. But even if it
>> was, I don't think there's any rule that all new Python 3.x builtins
>> will be backported to 2.x?).
>>
>> In my mind there's three different Python versions involved:
>>
>> - A: The version of Python the user targets in the pyx file (when it
>> comes to library and builtins; keeping the language part out of  
>> this).
>> - B: The version of Python that imports the compiled module. The user
>> just has to make sure that this matches A in a reasonable fashion  
>> (the
>> rules are non-trivial and not a simple >= or <=, as Py3 removes
>> builtins, but Python lays these down for us).
>> - C: The version that runs Cython.
>>
>> This discussion was about A and B -- that Cython won't allow users to
>> code for Py2.4 and run under Py2.3.
>>
>> You seem to be talking about C, which is something different. Cython
>> just transforms code and could run on any Turing-machine, there's no
>> natural reason it has to be connected with A or B.

Theoretically, C is orthogonal. However, for a given user, it is  
likely that there is strong correlation between all of A, B, and C.

>> We do currently require C >= B (which means that any Py3-only  
>> builtins
>> are plain unavailable).

No, we require that C >= A. Often C = B, and obviously B >= A, so  
this is reasonable.

>> In my opinion, ideally C should be orthogonal to A and B because that
>> seems to be less complicated from a release and bugfinding  
>> perspective
>> (one less version to worry about in the mix)

On the other hand, violating C < A results in a nice cython-compile  
error, whereas violating B < A is either a (perhaps not easily hit)  
runtime error or a c-compile error. Which would you rather debug?  
It's also reassuring to say "if it compiles under C, it runs under C 
+" because A is often implicit rather than explicit.

>> and it would fix the issue
>> with accessing Py3-builtins for free, but I don't care very much.

Is this because Cython doesn't run under Py3? (I thought it did.) Or  
is there something I'm not seeing here?

> To be more concrete, the problems you mention could be fixed with a
> simple compiler flag (or directive) to Cython: --target- 
> platform=3.0 or
> similar, to control the builtins. But, "there's no lack of good  
> ideas".

I think inspecting __builtins__ at runtime is a more elegant (and  
safe) approach than manually keeping lists for every version. A  
possible exception is the builtins that disappeared in Py3, in which  
case I would lean towards "forwardporting" them by importing from the  
right place, as they are still there somewhere).

- Robert


From dagss at student.matnat.uio.no  Fri Oct  3 13:32:41 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Oct 2008 13:32:41 +0200
Subject: [Cython] cython doesn't compile cdef'd variables of type set?
In-Reply-To: <3D47B019-2A7D-4743-B8E2-EE467DC9FA15@math.washington.edu>
References: <41d5d14b0810011129o393b4b11nbf64896833981cf6@mail.gmail.com>	<9BA7DDDB-6C85-451E-B8EF-6BDA4D283319@math.washington.edu>	<4db014670810011425l41fce606oba63fcebb9af1c08@mail.gmail.com>	<2A88D46F-77ED-4837-814D-DD8F70CC8237@math.washington.edu>	<4db014670810012028m580a0992u9d57eeaafede3300@mail.gmail.com>	<e7ba66e40810012050w6defa590o4c4bf82bcf3b2d81@mail.gmail.com>	<4db014670810021106w3c338851q85cf0ab6c435ba10@mail.gmail.com>	<e7ba66e40810021140sdf7cc77if718ff4a28d66ac0@mail.gmail.com>	<4db014670810021325p2c76d192pad964314950fe7c0@mail.gmail.com>	<48E53607.4080908@student.matnat.uio.no>	<e7ba66e40810021445t102bfacbhfb08fcf43d52e672@mail.gmail.com>	<4770B9DF-1E5F-41A5-BFE5-BE389A035696@math.washington.edu>	<48E5F108.50601@student.matnat.uio.no>	<48E5F18F.20102@student.matnat.uio.no>
	<3D47B019-2A7D-4743-B8E2-EE467DC9FA15@math.washington.edu>
Message-ID: <48E602D9.9030407@student.matnat.uio.no>

Robert Bradshaw wrote:
> Is this because Cython doesn't run under Py3? (I thought it did.) Or  
> is there something I'm not seeing here?

I haven't heard anything about Cython itself running under Py3. Stefan 
made runtests.py run under Py3, and then you need to first run it under 
Python2 with --no-cleanup, and then under Py3 with --no-cython (so that 
the C files generated using Py2 is compiled under Py3).

So this is where your scheme breaks down currently.

Perhaps this was fixed without me noticing it (if so I've broken it 
again in cython-devel I'm afraid, but that should be easy to fix).

If you want to fix this, then the matter disappears. OTOH I think 
supporting both Python 2 and Python 3 for running Cython could bog us 
down somewhat, at least it is not Guido's recommended porting strategy 
(which is staying with 2.x until you can make a total switch). Users 
will need to keep 2.x around on their systems for years to come anyway 
so I don't think it is something we should worry about at this stage if 
it means much work.

-- 
Dag Sverre

From dalcinl at gmail.com  Fri Oct  3 16:15:43 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 3 Oct 2008 11:15:43 -0300
Subject: [Cython] Robert, please help!
In-Reply-To: <D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
Message-ID: <e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>

On Fri, Oct 3, 2008 at 5:32 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 1, 2008, at 6:42 PM, Lisandro Dalcin wrote:
>
>> Robert, in this changeset
>>
>> changeset:   1184:515f94dc16f2
>> user:        Robert Bradshaw <robertwb at math.washington.edu>
>> date:        Mon Sep 29 19:08:45 2008 -0700
>> summary:     Refactoring of type parsing
>>
>> you cannot parse C function typedef like this:
>>
>> cdef extern from *:
>>     void (somefunc)(int)
>>
>> Please note that this declaration is not of the form "void
>> (*somefuncptr)(int)", and AFAIK the former is valid C code.
>
> Does it have the same meaning, or is there something more subtle
> going on?

Robert, AFAIK they do not have the same meaning. the void
(func1)(args) is function typedef and void (*func2)(args) is a funtion
POINTER typedef. In short, they are really different types. MPI has
many of those (non-pointer) typedef for some callback functions.

I really believe that this is a regression. It could potentially make
invalid some codes. In fact, this change already broke my codes.

> We don't accept all valid C code, for example
>
>     int foo(void)
>
> is allowed in C but not in Cython.

Long ago I discussed that case with Sfefan, and we agreed that there
was not point on supporting 'int foo(void)', as that C declaration was
in fact related to support ancient C and the different meaning of 'int
foo()', hopefully fixed in C++.

But in the fase of function vs. function pointer typedef's, the issues
are different. They really are different types.


>> I was looking at the changes you pushed, but I was unable to discover
>> what's going on. So I really need your help here.
>
> I needed to remove the dependancy the parser had on knowing all type
> names, so in some places it needs to look ahead to determine if it's
> looking at a type or an expression. If this is the only bug
> introduced, than I'm willing to accept that, but are there other
> statements that are now illegal?
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Oct  3 17:51:46 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 03 Oct 2008 17:51:46 +0200
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <F7F42678-E0B1-405C-ADBB-EB26DEE72C41@math.washington.edu>
References: <200810021846.14875.pagx13@yahoo.gr>
	<F7F42678-E0B1-405C-ADBB-EB26DEE72C41@math.washington.edu>
Message-ID: <48E63F92.9020002@behnel.de>

Hi,

Robert Bradshaw wrote:
> Thanks for the info. I'm not sure why this isn't working for you,  
> could you attach the actual bug.pyx file (preferably zipped, to be  
> sure that no mail servers mangle it at all)?

Also, you're using Cython 0.9.8. Could you try with the latest release? IIRC,
there were a couple of unicode related fixes after 0.9.8 was released.

Stefan


From pagx13 at yahoo.gr  Fri Oct  3 18:05:18 2008
From: pagx13 at yahoo.gr (P M)
Date: Fri, 3 Oct 2008 19:05:18 +0300
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <F7F42678-E0B1-405C-ADBB-EB26DEE72C41@math.washington.edu>
References: <200810021846.14875.pagx13@yahoo.gr>
	<F7F42678-E0B1-405C-ADBB-EB26DEE72C41@math.washington.edu>
Message-ID: <200810031905.18194.pagx13@yahoo.gr>

OK, I'm sending the file as an attachment :)

On Friday 03 October 2008 12:07:39 Robert Bradshaw wrote:
> Thanks for the info. I'm not sure why this isn't working for you,
> could you attach the actual bug.pyx file (preferably zipped, to be
> sure that no mail servers mangle it at all)?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bug.pyx.zip
Type: application/x-zip
Size: 217 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081003/a21062ab/attachment.bin 

From pagx13 at yahoo.gr  Fri Oct  3 18:27:09 2008
From: pagx13 at yahoo.gr (P M)
Date: Fri, 3 Oct 2008 19:27:09 +0300
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <48E63F92.9020002@behnel.de>
References: <200810021846.14875.pagx13@yahoo.gr>
	<F7F42678-E0B1-405C-ADBB-EB26DEE72C41@math.washington.edu>
	<48E63F92.9020002@behnel.de>
Message-ID: <200810031927.09646.pagx13@yahoo.gr>

On Friday 03 October 2008 18:51:46 Stefan Behnel wrote:
> Also, you're using Cython 0.9.8. Could you try with the latest release?
> IIRC, there were a couple of unicode related fixes after 0.9.8 was
> released.

I tried with the latest upstream (0.9.8.1.1), and the problem seems to be 
solved, thanks! (I ran cython from within the distribution directory, instead 
of installing it, hope that makes no difference.) The string is translated 
into C code as a series of octal escapes instead of UTF-8 text though... and 
the comments that show the position inside the original file place "?" marks 
in place of UTF-8 characters. But I guess that's OK :)

From ondrej at certik.cz  Fri Oct  3 22:16:17 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 3 Oct 2008 22:16:17 +0200
Subject: [Cython] Does not accept Unicode docstrings
In-Reply-To: <200810031927.09646.pagx13@yahoo.gr>
References: <200810021846.14875.pagx13@yahoo.gr>
	<F7F42678-E0B1-405C-ADBB-EB26DEE72C41@math.washington.edu>
	<48E63F92.9020002@behnel.de> <200810031927.09646.pagx13@yahoo.gr>
Message-ID: <85b5c3130810031316l73329a04pdf0ace2b0f07fe4a@mail.gmail.com>

On Fri, Oct 3, 2008 at 6:27 PM, P M <pagx13 at yahoo.gr> wrote:
> On Friday 03 October 2008 18:51:46 Stefan Behnel wrote:
>> Also, you're using Cython 0.9.8. Could you try with the latest release?
>> IIRC, there were a couple of unicode related fixes after 0.9.8 was
>> released.
>
> I tried with the latest upstream (0.9.8.1.1), and the problem seems to be
> solved, thanks! (I ran cython from within the distribution directory, instead
> of installing it, hope that makes no difference.) The string is translated
> into C code as a series of octal escapes instead of UTF-8 text though... and
> the comments that show the position inside the original file place "?" marks
> in place of UTF-8 characters. But I guess that's OK :)

So just uploading a new upstream version should fix it. Seems like
there is a job for me. :)

Ondrej

From robertwb at math.washington.edu  Fri Oct  3 23:56:17 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 3 Oct 2008 14:56:17 -0700
Subject: [Cython] Robert, please help!
In-Reply-To: <e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
	<e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
Message-ID: <AE445869-E728-4CF5-A2BD-2D2ABDA50BA6@math.washington.edu>

On Oct 3, 2008, at 7:15 AM, Lisandro Dalcin wrote:

> On Fri, Oct 3, 2008 at 5:32 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Oct 1, 2008, at 6:42 PM, Lisandro Dalcin wrote:
>>
>>> Robert, in this changeset
>>>
>>> changeset:   1184:515f94dc16f2
>>> user:        Robert Bradshaw <robertwb at math.washington.edu>
>>> date:        Mon Sep 29 19:08:45 2008 -0700
>>> summary:     Refactoring of type parsing
>>>
>>> you cannot parse C function typedef like this:
>>>
>>> cdef extern from *:
>>>     void (somefunc)(int)
>>>
>>> Please note that this declaration is not of the form "void
>>> (*somefuncptr)(int)", and AFAIK the former is valid C code.
>>
>> Does it have the same meaning, or is there something more subtle
>> going on?
>
> Robert, AFAIK they do not have the same meaning. the void
> (func1)(args) is function typedef and void (*func2)(args) is a funtion
> POINTER typedef. In short, they are really different types. MPI has
> many of those (non-pointer) typedef for some callback functions.

I understand that they are different types, I thought the one could  
be used wherever the other is needed though.

> I really believe that this is a regression. It could potentially make
> invalid some codes. In fact, this change already broke my codes.

OK, I'll look into this some more then. Being able to use "cimport *"  
to import types is why I had to make these changes.

>> We don't accept all valid C code, for example
>>
>>     int foo(void)
>>
>> is allowed in C but not in Cython.
>
> Long ago I discussed that case with Sfefan, and we agreed that there
> was not point on supporting 'int foo(void)', as that C declaration was
> in fact related to support ancient C and the different meaning of 'int
> foo()', hopefully fixed in C++.
>
> But in the fase of function vs. function pointer typedef's, the issues
> are different. They really are different types.
>
>
>>> I was looking at the changes you pushed, but I was unable to  
>>> discover
>>> what's going on. So I really need your help here.
>>
>> I needed to remove the dependancy the parser had on knowing all type
>> names, so in some places it needs to look ahead to determine if it's
>> looking at a type or an expression. If this is the only bug
>> introduced, than I'm willing to accept that, but are there other
>> statements that are now illegal?
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From greg.ewing at canterbury.ac.nz  Sat Oct  4 01:04:31 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 04 Oct 2008 11:04:31 +1200
Subject: [Cython] Robert, please help!
In-Reply-To: <e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
	<e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
Message-ID: <48E6A4FF.1070900@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Long ago I discussed that case with Sfefan, and we agreed that there
> was not point on supporting 'int foo(void)', as that C declaration was
> in fact related to support ancient C and the different meaning of 'int
> foo()', hopefully fixed in C++.

Yes, in C++, all function headers are prototypes, so
there's no need for f(void). It seems to be accepted
for backward compatibility with C, but Pyrex/Cython
doesn't have to be backward compatible with anything,
so there's no need to support it.

As for (f)(), I'm not sure why you would ever have to
write that instead of f(). However, there's a degenerate
case of that when you have an empty declarator, e.g.
in a cast, where ()() means a function, whereas just
() on its own is regarded as a syntax error by C
compilers.

I suspect this isn't important, though, because casting
something to a function (as opposed to a function pointer)
isn't something I can think of a use for.

Nevertheless, (f)() is valid C declaration syntax, and
Pyrex aims to allow any valid C declaration as far as
reasonably possible. So I'd recommend fixing this if
it won't be too difficult.

-- 
Greg

From dalcinl at gmail.com  Sat Oct  4 05:46:03 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 4 Oct 2008 00:46:03 -0300
Subject: [Cython] Robert, please help!
In-Reply-To: <48E6A4FF.1070900@canterbury.ac.nz>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
	<e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
	<48E6A4FF.1070900@canterbury.ac.nz>
Message-ID: <e7ba66e40810032046s40722170o79a522df527323fa@mail.gmail.com>

On Fri, Oct 3, 2008 at 8:04 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> Long ago I discussed that case with Sfefan, and we agreed that there
>> was not point on supporting 'int foo(void)', as that C declaration was
>> in fact related to support ancient C and the different meaning of 'int
>> foo()', hopefully fixed in C++.
>
> Yes, in C++, all function headers are prototypes, so
> there's no need for f(void). It seems to be accepted
> for backward compatibility with C, but Pyrex/Cython
> doesn't have to be backward compatible with anything,
> so there's no need to support it.
>
> As for (f)(), I'm not sure why you would ever have to
> write that instead of f(). However, there's a degenerate
> case of that when you have an empty declarator, e.g.
> in a cast, where ()() means a function, whereas just
> () on its own is regarded as a syntax error by C
> compilers.

Greg, I?m not a C lawyer, but as reference, the MPI standard uses
"typedef int (f)(args)" everywhere, for example:
<http://www.mpi-forum.org/docs/mpi21-report-bw/node147.htm>.

>
> I suspect this isn't important, though, because casting
> something to a function (as opposed to a function pointer)
> isn't something I can think of a use for.
>
> Nevertheless, (f)() is valid C declaration syntax, and
> Pyrex aims to allow any valid C declaration as far as
> reasonably possible. So I'd recommend fixing this if
> it won't be too difficult.
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Sat Oct  4 06:04:38 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 04 Oct 2008 16:04:38 +1200
Subject: [Cython] Robert, please help!
In-Reply-To: <e7ba66e40810032046s40722170o79a522df527323fa@mail.gmail.com>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
	<e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
	<48E6A4FF.1070900@canterbury.ac.nz>
	<e7ba66e40810032046s40722170o79a522df527323fa@mail.gmail.com>
Message-ID: <48E6EB56.9090605@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Greg, I?m not a C lawyer, but as reference, the MPI standard uses
> "typedef int (f)(args)" everywhere

I just tried an experiment, and gcc seems to acccept
either

   typedef int f(void);

or

   typedef int (f)(void);

-- 
Greg

From dalcinl at gmail.com  Sat Oct  4 06:29:22 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 4 Oct 2008 01:29:22 -0300
Subject: [Cython] Robert, please help!
In-Reply-To: <48E6EB56.9090605@canterbury.ac.nz>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
	<e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
	<48E6A4FF.1070900@canterbury.ac.nz>
	<e7ba66e40810032046s40722170o79a522df527323fa@mail.gmail.com>
	<48E6EB56.9090605@canterbury.ac.nz>
Message-ID: <e7ba66e40810032129u8e75f6fy12063ada1c90042e@mail.gmail.com>

OK, at this link http://publications.gbdirect.co.uk/c_book/chapter8/typedef.html

I've found the following example:

/*
* Using typedef, declare 'func' to have type
* 'function taking two int arguments, returning int'
*/
typedef int func(int, int);
/* ERROR */
func func_name{ /*....*/ }
/* Correct. Returns pointer to a type 'func' */
func *func_name(){ /*....*/ }
/*
* Correct if functions could return functions,
* but C can't.
*/
func func_name(){ /*....*/ }


I'm fine with that. So I?ll try tomorrow if Robert?s modifications can
handle this form. Thank's for the point, Greg!


On Sat, Oct 4, 2008 at 1:04 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> Greg, I?m not a C lawyer, but as reference, the MPI standard uses
>> "typedef int (f)(args)" everywhere
>
> I just tried an experiment, and gcc seems to acccept
> either
>
>   typedef int f(void);
>
> or
>
>   typedef int (f)(void);
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sat Oct  4 13:42:20 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 4 Oct 2008 04:42:20 -0700
Subject: [Cython] Pure python mode
Message-ID: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>

I polished off some more of the pure python mode stuff tonight. I  
just pushed it to cython-devel. Now you can do stuff like:

> import cython  # you need a single file to be in your path to use  
> it in pure Python mode, provided (and installed) with Cython
>
> @cython.locals(x=int, y=cython.p_int)
> def foo(x):
>     y = cython.cast(cython.p_int, x)
>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>     if cython.compiled:
>         print "the compiler was run
>     else:
>         print "just being interpreted"
>
> x = cython.declare(int)
> xx = cython.declare(cython.p_int)
> xx = cython.address(x)
>
> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>
> a = cython.declare(MyStruct)
> a.x = 5
>
> T = cython.typedef(MyStruct)
> b = cython.declare(cython.pointer(T))
> b[0].x = 4


On top of this, you can write a .pxd file to accompany your .py file  
and it will coerce your .py file classes and def statements into cdef  
and cpdef versions.

It's still very new and probably highly experimental (though it only  
modifies the cython.* nodes, so it shouldn't cause any regressions).  
Let me know what you think.

- Robert


From dagss at student.matnat.uio.no  Sat Oct  4 14:45:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 04 Oct 2008 14:45:00 +0200
Subject: [Cython] Pure python mode
Message-ID: <3305976321.875339@smtp.netcom.no>

Robert wrote:

> I polished off some more of the pure python mode stuff tonight. I  
>just pushed it to cython-devel. Now you can do stuff like:
>

Excellent! I have some comments to the syntax of course :-)

Do you see this as the way forward for all new Cython code, or just when one has interpreter compatability requirements? I suppose Sage will want to stay consistent with the current code, while for NumPy/numerics one could for instance rewrite the tutorial etc. to this syntax. It probably is more cumbersome for daily usage though...

>> import cython  # you need a single file to be in your path to use  
> it in pure Python mode, provided (and installed) with Cython
>
>> @cython.locals(x=int, y=cython.p_int)
> def foo(x):
>     y = cython.cast(cython.p_int, x)
>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>     if cython.compiled:
>         print "the compiler was run
>     else:
>         print "just being interpreted"
>
>> x = cython.declare(int)
> xx = cython.declare(cython.p_int)
> xx = cython.address(x)

Can you also do something like

from cython import cast

? It doesn't work for directives but I consider that a bug...

As for syntax, I have a problem with the assignment style, i.e. I'd rather see:

cython.declare(x=cython.int)

because it seems more magical, and so one isn't surprised by stuff like this:

x = cython.declare(cython.int)
x = "hello" # illegal? Huh?

On the brainstorming side, I was wondering if perhaps both casting and declaration could be done via constructor:

x = cython.int(3)

It has the same problem as your cython.declare though so I won't support that at this stage (but with inference it could work).

Another alternative:

with cython.locals(..): ...

The thing is, "with" sets up a natural scope block, and so it could be possible to declare variables e.g. inside a for loop with such a construct. A bit verbose though.

>
>> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>
>> a = cython.declare(MyStruct)
> a.x = 5
>
>> T = cython.typedef(MyStruct)
> b = cython.declare(cython.pointer(T))
> b[0].x = 4

Nice that all of this is working!

Brainstorming again:

I feel that this alternative to typedefs may not be too much work, just mentioning it:

cdef class A: pass
B = A # both symbol ass. and typedef

@cython.locals(x=B) # type..
def f():
    x = B() # module var

i.e. B = A be both a variable assignment *and* a typedef. This would only be for top-level module assignments, so

if True:
    C = A
@cython.locals(x=C) #illegal!
def f():
    x = C() # legal 

furthermore,

myint = cython.int

would only trigger a typedef as int is not a variable at runtime.

Take it for what it is (a brainstorming); if the assignment style is changed to statement style for cython.declare and cython.typedef I'm 100 percent happy, and quite happy for all of this even if it stays like now.

Dag Sverre


From robertwb at math.washington.edu  Sat Oct  4 21:17:55 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 4 Oct 2008 12:17:55 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <3305976321.875339@smtp.netcom.no>
References: <3305976321.875339@smtp.netcom.no>
Message-ID: <B9ECD74B-B418-4CC2-AB05-C013BBB5E23E@math.washington.edu>

On Oct 4, 2008, at 5:45 AM, Dag Sverre Seljebotn wrote:

> Robert wrote:
>
>> I polished off some more of the pure python mode stuff tonight. I
>> just pushed it to cython-devel. Now you can do stuff like:
>>
>
> Excellent! I have some comments to the syntax of course :-)

Thanks. Input is good.

> Do you see this as the way forward for all new Cython code, or just  
> when one has interpreter compatability requirements? I suppose Sage  
> will want to stay consistent with the current code, while for NumPy/ 
> numerics one could for instance rewrite the tutorial etc. to this  
> syntax. It probably is more cumbersome for daily usage though...

Both, but primarily for interpreter compatibility. Stuff like SymPy  
is my main target for things like this, and it will facilitate even  
easier migration. It will also be useful for bootstrapping the Cython  
compiler. The current syntax certainly is less cumbersome when  
dealing with a lot of c-ish code, and I don't anticipate existing  
codebases to be changing (not Sage at least, we've got more  
interesting things to work on...).

>>> import cython  # you need a single file to be in your path to use
>> it in pure Python mode, provided (and installed) with Cython
>>
>>> @cython.locals(x=int, y=cython.p_int)
>> def foo(x):
>>     y = cython.cast(cython.p_int, x)
>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>     if cython.compiled:
>>         print "the compiler was run
>>     else:
>>         print "just being interpreted"
>>
>>> x = cython.declare(int)
>> xx = cython.declare(cython.p_int)
>> xx = cython.address(x)
>
> Can you also do something like
>
> from cython import cast
>
> ? It doesn't work for directives but I consider that a bug...

No, it doesn't yet, but it will. I see an easy way to implement that.

> As for syntax, I have a problem with the assignment style, i.e. I'd  
> rather see:
>
> cython.declare(x=cython.int)
>
> because it seems more magical, and so one isn't surprised by stuff  
> like this:
>
> x = cython.declare(cython.int)
> x = "hello" # illegal? Huh?

I could certainly see supporting that form as well. The advantage of  
the x = cython.declare(...) syntax is that it is a declaration in the  
interpreter as well, so you can do stuff like

x = cython.declare(some_struct)
init_struct(x)

Another thing that might be good to support is

x = cython.declare(cython.int, 3)

> On the brainstorming side, I was wondering if perhaps both casting  
> and declaration could be done via constructor:
>
> x = cython.int(3)
>
> It has the same problem as your cython.declare though so I won't  
> support that at this stage (but with inference it could work).

Hmm... I suppose currently when A is a (cython defined) cdef class, A 
(3) *always* returns an A, but this may not be the case in the  
future. Or would it only work for primitive types? I could certainly  
see this working more naturally once type inference is up and running  
(and the need for most declarations would hopefully go away).

> Another alternative:
>
> with cython.locals(..): ...
>
> The thing is, "with" sets up a natural scope block,

This is actually a disadvantage, as neither Cython nor Python have  
the idea of scope blocks (only locals and globals).

> and so it could be possible to declare variables e.g. inside a for  
> loop with such a construct. A bit verbose though.
>
>>
>>> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>>
>>> a = cython.declare(MyStruct)
>> a.x = 5
>>
>>> T = cython.typedef(MyStruct)
>> b = cython.declare(cython.pointer(T))
>> b[0].x = 4
>
> Nice that all of this is working!
>
> Brainstorming again:
>
> I feel that this alternative to typedefs may not be too much work,  
> just mentioning it:
>
> cdef class A: pass
> B = A # both symbol ass. and typedef
>
> @cython.locals(x=B) # type..
> def f():
>     x = B() # module var
>
> i.e. B = A be both a variable assignment *and* a typedef. This  
> would only be for top-level module assignments, so
>
> if True:
>     C = A
> @cython.locals(x=C) #illegal!
> def f():
>     x = C() # legal
>
> furthermore,
>
> myint = cython.int
>
> would only trigger a typedef as int is not a variable at runtime.

The assignment-on-typedef seems a bit too magical for me...  
especially once we support all types as first-level objects (which I  
hope we do). This would also make

x = A
...
x = B

fail for "mysterious" reasons when one of A or B are defined to be  
types. Note that I tried to make it such that every place typing  
information is used, a type is required.

> Take it for what it is (a brainstorming); if the assignment style  
> is changed to statement style for cython.declare and cython.typedef  
> I'm 100 percent happy, and quite happy for all of this even if it  
> stays like now.

Brainstorming is good, thanks for the feedback.

- Robert


From kevinar18 at hotmail.com  Sat Oct  4 22:03:12 2008
From: kevinar18 at hotmail.com (Kevin Ar18)
Date: Sat, 4 Oct 2008 16:03:12 -0400
Subject: [Cython] What to do with modules
Message-ID: <BLU110-W23FDA85037E7F2DE4730F1AA3F0@phx.gbl>


As I understand the Docs, you can't access c types, functions, or classes in a module.  For example, this won't work:


somemodule.pyx

cdef int a = 1
py_get():
    return a

somefile.pyx

import somemodule
cdef int b = somemodule.a
print somemodule.py_get()

or this:

somemodule.pyx

cdef extern from *:
    int a = 1
py_get():
    return a

somefile.pyx

cimport somemodule
cdef int b = somemodule.a



So what do you do with modules that mix Cython and Python?
_________________________________________________________________
See how Windows connects the people, information, and fun that are part of your life.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/

From jasone at canonware.com  Sat Oct  4 23:36:57 2008
From: jasone at canonware.com (Jason Evans)
Date: Sat, 04 Oct 2008 14:36:57 -0700
Subject: [Cython] Namespace semantics differences from Python
Message-ID: <48E7E1F9.7000103@canonware.com>

I am in the process of converting a largish Python program to Cython. 
One thing my software does throughout is to create package-specific 
exception types, so that exception handler code can accurately catch 
only certain types of exceptions.  For example, one subpackage defines 
the following exception types:

----
import Crux

class Exception(Crux.Exception):
     pass

class ValueError(Exception, ValueError):
     def __init__(self, str):
         self._str = str

     def __str__(self):
         return self._str
----

As you can see, this code defines module-specific classes with the same 
names as the top-level Python exception types.  The Cython compiler 
complains with errors like:

/home/jasone/crux/crux/./src/Crux/CTMatrix.pyx:19:0: Assignment to 
non-lvalue 'Exception'

Is this a "feature" or a bug in Cython?

Thanks,
Jason

From jasone at canonware.com  Sun Oct  5 00:46:12 2008
From: jasone at canonware.com (Jason Evans)
Date: Sat, 04 Oct 2008 15:46:12 -0700
Subject: [Cython] Namespace semantics differences from Python
In-Reply-To: <48E7E1F9.7000103@canonware.com>
References: <48E7E1F9.7000103@canonware.com>
Message-ID: <48E7F234.2010800@canonware.com>

Jason Evans wrote:
> [...]
> 
> Is this a "feature" or a bug in Cython?

The answer appears to be "none of the above"; the bug was in my code. 
There was an inconsistency in the .pxd file that I neglected to check 
for before sending email.

Jason

From robertwb at math.washington.edu  Sun Oct  5 06:20:01 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 4 Oct 2008 21:20:01 -0700
Subject: [Cython] What to do with modules
In-Reply-To: <BLU110-W23FDA85037E7F2DE4730F1AA3F0@phx.gbl>
References: <BLU110-W23FDA85037E7F2DE4730F1AA3F0@phx.gbl>
Message-ID: <9E4B94B9-8591-4069-98A0-402957B5BF3A@math.washington.edu>

On Oct 4, 2008, at 1:03 PM, Kevin Ar18 wrote:

> As I understand the Docs, you can't access c types, functions, or  
> classes in a module.

I'm not quite sure I understand your question, but I'll try to answer...

Python stores its module-level in a big dict. The speed gains of  
Cython are usually made by avoiding this dict storage (as well as the  
wrapping everything as an object that is required to do so).

> For example, this won't work:
>
>
> somemodule.pyx
>
> cdef int a = 1
> py_get():
>     return a
>
> somefile.pyx
>
> import somemodule
> cdef int b = somemodule.a

This won't, as a is not a Python object but a c global.

> print somemodule.py_get()

This should.

>
> or this:
>
> somemodule.pyx
>
> cdef extern from *:
>     int a = 1
> py_get():
>     return a
>
> somefile.pyx
>
> cimport somemodule
> cdef int b = somemodule.a

No, that won't either.

> So what do you do with modules that mix Cython and Python?

cdef classes live in both the C and Python namespace. cpdef functions  
are accessible from both C and Python as well. In this case, you  
probably need to just write

a = 1

and deal with it being a Python object so you can get at it from  
Python (and it is still accessible from C). Everything accessible  
from Python is also accessible from Cython.

We probably should support cimporting cdef'd global variables  
(probably with the same trick as we do for functions) and it would be  
nice to be able to declare variables as "public" or "readonly" as one  
can do with class attributes (though the only way I could see how to  
do it is the hack at http://mail.python.org/pipermail/python-list/ 
2007-January/422578.html or making a subtype of the module type)

- Robert


From ondrej at certik.cz  Mon Oct  6 00:46:42 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 6 Oct 2008 00:46:42 +0200
Subject: [Cython] Pure python mode
In-Reply-To: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
Message-ID: <85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>

On Sat, Oct 4, 2008 at 1:42 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> I polished off some more of the pure python mode stuff tonight. I
> just pushed it to cython-devel. Now you can do stuff like:
>
>> import cython  # you need a single file to be in your path to use
>> it in pure Python mode, provided (and installed) with Cython
>>
>> @cython.locals(x=int, y=cython.p_int)
>> def foo(x):
>>     y = cython.cast(cython.p_int, x)
>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>     if cython.compiled:
>>         print "the compiler was run
>>     else:
>>         print "just being interpreted"
>>
>> x = cython.declare(int)
>> xx = cython.declare(cython.p_int)
>> xx = cython.address(x)
>>
>> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>>
>> a = cython.declare(MyStruct)
>> a.x = 5
>>
>> T = cython.typedef(MyStruct)
>> b = cython.declare(cython.pointer(T))
>> b[0].x = 4
>
>
> On top of this, you can write a .pxd file to accompany your .py file
> and it will coerce your .py file classes and def statements into cdef
> and cpdef versions.
>
> It's still very new and probably highly experimental (though it only
> modifies the cython.* nodes, so it shouldn't cause any regressions).
> Let me know what you think.

Thanks a lot for doing this!

I installed cython-devel and then I tried this little example:


--------
import Cython as cython

@cython.locals(x=int, y=cython.p_int)
def foo(x):
    y = cython.cast(cython.p_int, x)
    print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
    if cython.compiled:
        print "the compiler was run"
    else:
        print "just being interpreted"

foo(5)
-----------

When running in it in Python, it gives:

  File "t.py", line 12, in <module>
    foo(5)
  File "t.py", line 5, in foo
    y = cython.cast(cython.p_int, x)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
    return type(arg)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in __init__
    raise ValueError
ValueError


Am I using it incorrectly? Btw, when I compile it with Cython:

$ python -c "import t; t.foo(5)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "t.py", line 12, in t (t.c:515)
    foo(5)
  File "t.py", line 5, in t.foo (t.c:265)
    y = cython.cast(cython.p_int, x)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
    return type(arg)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in __init__
    raise ValueError
ValueError


Looking at the code, I need to pass it Cython p_int integers, right? I
also tried:

x = cython.declare(int)
xx = cython.declare(cython.p_int)
xx = cython.address(x)
foo(xx)

But this also raises the same error as above. But otherwise this is
exactly the way to go, I am really looking forward to just having one
codebase for both compiled and interpreted stuff. Unfortunately, I got
again busy lately to implement this myself, so thanks again for the
work.

Ondrej

From uschmitt at mineway.de  Mon Oct  6 12:38:14 2008
From: uschmitt at mineway.de (Uwe Schmitt)
Date: Mon, 06 Oct 2008 12:38:14 +0200
Subject: [Cython] [mailinglist] Re: resolving name conflict -- does not
 work for enums !?
In-Reply-To: <48E5F81B.2000600@canterbury.ac.nz>
References: <48E4E7DD.1040906@mineway.de> <48E5F81B.2000600@canterbury.ac.nz>
Message-ID: <48E9EA96.2040809@mineway.de>

Greg Ewing schrieb:
> You shouldn't be cimporting minimal in minimal.pyx. The files
> minimal.pxd and minimal.pyx are two parts of the *same* module,
> i.e. minimal. There's only one namespace, and you're trying
> to define A as both a C constant and a Python global in that
> namespace.
>
> You need to put the definition of enum test into a different
> module namespace, e.g.
>
>    # mindefs.pxd
>    cdef enum test:
>      A
>
>    # minimal.pyx:
>    cimport mindefs
>    A = mindefs.A
>
> Note that you don't need a mindefs.pyx in this case -- you're
> only using mindefs as a compile-time namespace.
>   

Thanks, that works now.
Greetings, Uwe

-- 
Dr. rer. nat. Uwe Schmitt
F&E Mathematik
 
mineway GmbH
Science Park 2
D-66123 Saarbr?cken
 
Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376
 
uschmitt at mineway.de
www.mineway.de
 
Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339



From uschmitt at mineway.de  Mon Oct  6 18:03:44 2008
From: uschmitt at mineway.de (Uwe Schmitt)
Date: Mon, 06 Oct 2008 18:03:44 +0200
Subject: [Cython] wrapping c code, classmethod question
Message-ID: <48EA36E0.1010209@mineway.de>

Hi,

I'm trying to wrap an existing C entension as follows:

    cdef struct A:
        double * values
        int n


    cdef class Test:
        cdef A a

        @classmethod
        def buildFromData(cls, filename):

            cdef A a # = foofoo(filename)

            obj = cls()
            obj.a = a
            return obj

Running Cython the "obj.a=a" results in """Cannot convert 'A' to Python 
object""".

So: how can I instantiate extension classes from classmethods ?

Greetings, Uwe


-- 
Dr. rer. nat. Uwe Schmitt
F&E Mathematik
 
mineway GmbH
Science Park 2
D-66123 Saarbr?cken
 
Telefon: +49 (0)681 8390 5334
Telefax: +49 (0)681 830 4376
 
uschmitt at mineway.de
www.mineway.de
 
Gesch?ftsf?hrung: Dr.-Ing. Mathias Bauer
Amtsgericht Saarbr?cken HRB 12339


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081006/e1c0ff8e/attachment.htm 

From dagss at student.matnat.uio.no  Mon Oct  6 18:16:04 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 6 Oct 2008 18:16:04 +0200 (CEST)
Subject: [Cython] wrapping c code, classmethod question
In-Reply-To: <48EA36E0.1010209@mineway.de>
References: <48EA36E0.1010209@mineway.de>
Message-ID: <33372.88.90.248.62.1223309764.squirrel@webmail.uio.no>

Uwe Schmitt wrote:
> Hi,
>
> I'm trying to wrap an existing C entension as follows:
>
>     cdef struct A:
>         double * values
>         int n
>
>
>     cdef class Test:
>         cdef A a
>
>         @classmethod
>         def buildFromData(cls, filename):
>
>             cdef A a # = foofoo(filename)
>
>             obj = cls()
>             obj.a = a
>             return obj
>
> Running Cython the "obj.a=a" results in """Cannot convert 'A' to Python
> object""".

Because "obj.a" is an attribute of the Python object "obj", while a is a C
struct...

I.e. you need to type "cdef Test obj = cls()" instead, so that it is known
compile-time that "obj" will be an instance of Test and thus have a typed
"a" field.

(I don't know myself whether classmethod works as intended for cdef
classes either, but this should be one obstacle less...)

Dag Sverre



From robert.kern at gmail.com  Mon Oct  6 18:47:27 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 11:47:27 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
Message-ID: <gcdfev$8p3$1@ger.gmane.org>

Robert Bradshaw wrote:
> I polished off some more of the pure python mode stuff tonight. I  
> just pushed it to cython-devel. Now you can do stuff like:
> 
>> import cython  # you need a single file to be in your path to use  
>> it in pure Python mode, provided (and installed) with Cython

Hmm, does this cause a problem on systems with case-insensitive files? cython.py 
and Cython/ won't actually conflict on the filesystem, but I suspect the import 
mechanism may get confused.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robertwb at math.washington.edu  Mon Oct  6 19:04:44 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 10:04:44 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
Message-ID: <A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>

On Oct 5, 2008, at 3:46 PM, Ondrej Certik wrote:

> On Sat, Oct 4, 2008 at 1:42 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> I polished off some more of the pure python mode stuff tonight. I
>> just pushed it to cython-devel. Now you can do stuff like:
>>
>>> import cython  # you need a single file to be in your path to use
>>> it in pure Python mode, provided (and installed) with Cython
>>>
>>> @cython.locals(x=int, y=cython.p_int)
>>> def foo(x):
>>>     y = cython.cast(cython.p_int, x)
>>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>>     if cython.compiled:
>>>         print "the compiler was run
>>>     else:
>>>         print "just being interpreted"
>>>
>>> x = cython.declare(int)
>>> xx = cython.declare(cython.p_int)
>>> xx = cython.address(x)
>>>
>>> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>>>
>>> a = cython.declare(MyStruct)
>>> a.x = 5
>>>
>>> T = cython.typedef(MyStruct)
>>> b = cython.declare(cython.pointer(T))
>>> b[0].x = 4
>>
>>
>> On top of this, you can write a .pxd file to accompany your .py file
>> and it will coerce your .py file classes and def statements into cdef
>> and cpdef versions.
>>
>> It's still very new and probably highly experimental (though it only
>> modifies the cython.* nodes, so it shouldn't cause any regressions).
>> Let me know what you think.
>
> Thanks a lot for doing this!

You're welcome.

> I installed cython-devel and then I tried this little example:
>
>
> --------
> import Cython as cython
>
> @cython.locals(x=int, y=cython.p_int)
> def foo(x):
>     y = cython.cast(cython.p_int, x)
>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>     if cython.compiled:
>         print "the compiler was run"
>     else:
>         print "just being interpreted"
>
> foo(5)
> -----------
>
> When running in it in Python, it gives:
>
>   File "t.py", line 12, in <module>
>     foo(5)
>   File "t.py", line 5, in foo
>     y = cython.cast(cython.p_int, x)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
>     return type(arg)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in  
> __init__
>     raise ValueError
> ValueError
>
>
> Am I using it incorrectly?

No, it's just not letting you cast an int into an p_int. I suppose it  
should? It just doesn't make sense to emulate this in Python.

> Btw, when I compile it with Cython:
>
> $ python -c "import t; t.foo(5)"
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "t.py", line 12, in t (t.c:515)
>     foo(5)
>   File "t.py", line 5, in t.foo (t.c:265)
>     y = cython.cast(cython.p_int, x)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
>     return type(arg)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in  
> __init__
>     raise ValueError
> ValueError

You need to import cython, not import Cython as cython.

> Looking at the code, I need to pass it Cython p_int integers, right? I
> also tried:
>
> x = cython.declare(int)
> xx = cython.declare(cython.p_int)
> xx = cython.address(x)
> foo(xx)
>
> But this also raises the same error as above.

This should work, Again, I think it's because you imported Cython  
rather than cython.

> But otherwise this is
> exactly the way to go, I am really looking forward to just having one
> codebase for both compiled and interpreted stuff. Unfortunately, I got
> again busy lately to implement this myself, so thanks again for the
> work.
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Mon Oct  6 19:06:24 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 10:06:24 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <gcdfev$8p3$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
Message-ID: <BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>

On Oct 6, 2008, at 9:47 AM, Robert Kern wrote:

> Robert Bradshaw wrote:
>> I polished off some more of the pure python mode stuff tonight. I
>> just pushed it to cython-devel. Now you can do stuff like:
>>
>>> import cython  # you need a single file to be in your path to use
>>> it in pure Python mode, provided (and installed) with Cython
>
> Hmm, does this cause a problem on systems with case-insensitive  
> files? cython.py
> and Cython/ won't actually conflict on the filesystem, but I  
> suspect the import
> mechanism may get confused.

Currently all the emulating stuff is in Cython/Shadow.py. Both Cython/ 
__init__.py and cython.py import it, thus it should work on both case- 
sensitive and case-insensitive filesystems.

- Robert




From robert.kern at gmail.com  Mon Oct  6 19:18:40 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 12:18:40 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
Message-ID: <gcdh9g$hkh$1@ger.gmane.org>

Robert Bradshaw wrote:
> On Oct 6, 2008, at 9:47 AM, Robert Kern wrote:
> 
>> Robert Bradshaw wrote:
>>> I polished off some more of the pure python mode stuff tonight. I
>>> just pushed it to cython-devel. Now you can do stuff like:
>>>
>>>> import cython  # you need a single file to be in your path to use
>>>> it in pure Python mode, provided (and installed) with Cython
>> Hmm, does this cause a problem on systems with case-insensitive  
>> files? cython.py
>> and Cython/ won't actually conflict on the filesystem, but I  
>> suspect the import
>> mechanism may get confused.
> 
> Currently all the emulating stuff is in Cython/Shadow.py. Both Cython/ 
> __init__.py and cython.py import it, thus it should work on both case- 
> sensitive and case-insensitive filesystems.

Cython/Shadow.py is not what I'm concerned about. It's the presence of both 
cython.py and Cython/__init__.py. I'm wondering if the statement "import cython" 
might actually import Cython/__init__.py on some case-insensitive file system.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ondrej at certik.cz  Mon Oct  6 19:25:53 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 6 Oct 2008 19:25:53 +0200
Subject: [Cython] Pure python mode
In-Reply-To: <A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
Message-ID: <85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>

On Mon, Oct 6, 2008 at 7:04 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 5, 2008, at 3:46 PM, Ondrej Certik wrote:
>
>> On Sat, Oct 4, 2008 at 1:42 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>> I polished off some more of the pure python mode stuff tonight. I
>>> just pushed it to cython-devel. Now you can do stuff like:
>>>
>>>> import cython  # you need a single file to be in your path to use
>>>> it in pure Python mode, provided (and installed) with Cython
>>>>
>>>> @cython.locals(x=int, y=cython.p_int)
>>>> def foo(x):
>>>>     y = cython.cast(cython.p_int, x)
>>>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>>>     if cython.compiled:
>>>>         print "the compiler was run
>>>>     else:
>>>>         print "just being interpreted"
>>>>
>>>> x = cython.declare(int)
>>>> xx = cython.declare(cython.p_int)
>>>> xx = cython.address(x)
>>>>
>>>> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>>>>
>>>> a = cython.declare(MyStruct)
>>>> a.x = 5
>>>>
>>>> T = cython.typedef(MyStruct)
>>>> b = cython.declare(cython.pointer(T))
>>>> b[0].x = 4
>>>
>>>
>>> On top of this, you can write a .pxd file to accompany your .py file
>>> and it will coerce your .py file classes and def statements into cdef
>>> and cpdef versions.
>>>
>>> It's still very new and probably highly experimental (though it only
>>> modifies the cython.* nodes, so it shouldn't cause any regressions).
>>> Let me know what you think.
>>
>> Thanks a lot for doing this!
>
> You're welcome.
>
>> I installed cython-devel and then I tried this little example:
>>
>>
>> --------
>> import Cython as cython
>>
>> @cython.locals(x=int, y=cython.p_int)
>> def foo(x):
>>     y = cython.cast(cython.p_int, x)
>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>     if cython.compiled:
>>         print "the compiler was run"
>>     else:
>>         print "just being interpreted"
>>
>> foo(5)
>> -----------
>>
>> When running in it in Python, it gives:
>>
>>   File "t.py", line 12, in <module>
>>     foo(5)
>>   File "t.py", line 5, in foo
>>     y = cython.cast(cython.p_int, x)
>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
>>     return type(arg)
>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in
>> __init__
>>     raise ValueError
>> ValueError
>>
>>
>> Am I using it incorrectly?
>
> No, it's just not letting you cast an int into an p_int. I suppose it
> should? It just doesn't make sense to emulate this in Python.

No, you are right, I was just trying to get it run.

>
>> Btw, when I compile it with Cython:
>>
>> $ python -c "import t; t.foo(5)"
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "t.py", line 12, in t (t.c:515)
>>     foo(5)
>>   File "t.py", line 5, in t.foo (t.c:265)
>>     y = cython.cast(cython.p_int, x)
>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
>>     return type(arg)
>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in
>> __init__
>>     raise ValueError
>> ValueError
>
> You need to import cython, not import Cython as cython.
>
>> Looking at the code, I need to pass it Cython p_int integers, right? I
>> also tried:
>>
>> x = cython.declare(int)
>> xx = cython.declare(cython.p_int)
>> xx = cython.address(x)
>> foo(xx)
>>
>> But this also raises the same error as above.
>
> This should work, Again, I think it's because you imported Cython
> rather than cython.


I tried that too:

ondra at fuji:~/ext/cython-pure$ cat t.py
import cython

@cython.locals(x=int, y=cython.p_int)
def foo(x):
    y = cython.cast(cython.p_int, x)
    print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
    if cython.compiled:
        print "the compiler was run"
    else:
        print "just being interpreted"

#foo(5)

x = cython.declare(int)
xx = cython.declare(cython.p_int)
xx = cython.address(x)
foo(xx)
ondra at fuji:~/ext/cython-pure$ python t.py
Traceback (most recent call last):
  File "t.py", line 17, in <module>
    foo(xx)
  File "t.py", line 5, in foo
    y = cython.cast(cython.p_int, x)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
    return type(arg)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in __init__
    raise ValueError
ValueError




With the same output. I copied the cython.py from cython-devel, is
that correct? I am probably doing something stupid. The error is
really simple and it fails, but currently I am not sure what it is
supposed to do correctly.

Ondrej

From robert.kern at gmail.com  Mon Oct  6 19:27:14 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 12:27:14 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <gcdh9g$hkh$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org>
Message-ID: <gcdhpi$jsr$1@ger.gmane.org>

Robert Kern wrote:
> Robert Bradshaw wrote:
>> On Oct 6, 2008, at 9:47 AM, Robert Kern wrote:
>>
>>> Robert Bradshaw wrote:
>>>> I polished off some more of the pure python mode stuff tonight. I
>>>> just pushed it to cython-devel. Now you can do stuff like:
>>>>
>>>>> import cython  # you need a single file to be in your path to use
>>>>> it in pure Python mode, provided (and installed) with Cython
>>> Hmm, does this cause a problem on systems with case-insensitive  
>>> files? cython.py
>>> and Cython/ won't actually conflict on the filesystem, but I  
>>> suspect the import
>>> mechanism may get confused.
>> Currently all the emulating stuff is in Cython/Shadow.py. Both Cython/ 
>> __init__.py and cython.py import it, thus it should work on both case- 
>> sensitive and case-insensitive filesystems.
> 
> Cython/Shadow.py is not what I'm concerned about. It's the presence of both 
> cython.py and Cython/__init__.py. I'm wondering if the statement "import cython" 
> might actually import Cython/__init__.py on some case-insensitive file system.

Oh, never mind. I see what you're saying. But then why do we need cython.py if 
everything is provided by Cython/__init__.py?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ondrej at certik.cz  Mon Oct  6 19:35:00 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 6 Oct 2008 19:35:00 +0200
Subject: [Cython] Pure python mode
In-Reply-To: <gcdhpi$jsr$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org> <gcdhpi$jsr$1@ger.gmane.org>
Message-ID: <85b5c3130810061035j214ae166v8768723a0cf02cfd@mail.gmail.com>

On Mon, Oct 6, 2008 at 7:27 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Robert Kern wrote:
>> Robert Bradshaw wrote:
>>> On Oct 6, 2008, at 9:47 AM, Robert Kern wrote:
>>>
>>>> Robert Bradshaw wrote:
>>>>> I polished off some more of the pure python mode stuff tonight. I
>>>>> just pushed it to cython-devel. Now you can do stuff like:
>>>>>
>>>>>> import cython  # you need a single file to be in your path to use
>>>>>> it in pure Python mode, provided (and installed) with Cython
>>>> Hmm, does this cause a problem on systems with case-insensitive
>>>> files? cython.py
>>>> and Cython/ won't actually conflict on the filesystem, but I
>>>> suspect the import
>>>> mechanism may get confused.
>>> Currently all the emulating stuff is in Cython/Shadow.py. Both Cython/
>>> __init__.py and cython.py import it, thus it should work on both case-
>>> sensitive and case-insensitive filesystems.
>>
>> Cython/Shadow.py is not what I'm concerned about. It's the presence of both
>> cython.py and Cython/__init__.py. I'm wondering if the statement "import cython"
>> might actually import Cython/__init__.py on some case-insensitive file system.
>
> Oh, never mind. I see what you're saying. But then why do we need cython.py if
> everything is provided by Cython/__init__.py?

Exactly. Also it behaves identically (fails) on my system.

Ondrej

From dagss at student.matnat.uio.no  Mon Oct  6 19:38:10 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 6 Oct 2008 19:38:10 +0200 (CEST)
Subject: [Cython] Pure python mode
In-Reply-To: <gcdhpi$jsr$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org> <gcdhpi$jsr$1@ger.gmane.org>
Message-ID: <32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>

> Robert Kern wrote:
> Oh, never mind. I see what you're saying. But then why do we need
> cython.py if
> everything is provided by Cython/__init__.py?
>

At least for me, Python cannot import it by the name "cython" if only
Cython/... is available. And I think we want "import cython" (not "import
Cython") to work, as lowercase is more consistent with common Python
naming schemes in general and just "feels more right" for such core
language functionality.

(Also the module name would otherwise need to change in a lot of places --
when compiling under Cython, as opposed to running in Python, "cython" is
a magic, hard-wired module and "Cython" will not work, at least currently
(and my opinion is it never should))

Dag Sverre


From robertwb at math.washington.edu  Mon Oct  6 19:43:24 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 10:43:24 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org> <gcdhpi$jsr$1@ger.gmane.org>
	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>
Message-ID: <0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>

On Oct 6, 2008, at 10:38 AM, Dag Sverre Seljebotn wrote:

>> Robert Kern wrote:
>> Oh, never mind. I see what you're saying. But then why do we need
>> cython.py if
>> everything is provided by Cython/__init__.py?
>>
>
> At least for me, Python cannot import it by the name "cython" if only
> Cython/... is available. And I think we want "import cython" (not  
> "import
> Cython") to work, as lowercase is more consistent with common Python
> naming schemes in general and just "feels more right" for such core
> language functionality.
>
> (Also the module name would otherwise need to change in a lot of  
> places --
> when compiling under Cython, as opposed to running in Python,  
> "cython" is
> a magic, hard-wired module and "Cython" will not work, at least  
> currently
> (and my opinion is it never should))


My thoughts exactly. I think the "cython compiler codebase" and the  
"cython magic module" as completely separate concepts. The only  
reason Cython/__init__.py does anything is to be a workaround for  
case insensitive filesystems.

>

- Robert


From robertwb at math.washington.edu  Mon Oct  6 19:47:15 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 10:47:15 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
Message-ID: <1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>

On Oct 6, 2008, at 10:25 AM, Ondrej Certik wrote:

> On Mon, Oct 6, 2008 at 7:04 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Oct 5, 2008, at 3:46 PM, Ondrej Certik wrote:
>>
>>> On Sat, Oct 4, 2008 at 1:42 PM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>>> I polished off some more of the pure python mode stuff tonight. I
>>>> just pushed it to cython-devel. Now you can do stuff like:
>>>>
>>>>> import cython  # you need a single file to be in your path to use
>>>>> it in pure Python mode, provided (and installed) with Cython
>>>>>
>>>>> @cython.locals(x=int, y=cython.p_int)
>>>>> def foo(x):
>>>>>     y = cython.cast(cython.p_int, x)
>>>>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>>>>     if cython.compiled:
>>>>>         print "the compiler was run
>>>>>     else:
>>>>>         print "just being interpreted"
>>>>>
>>>>> x = cython.declare(int)
>>>>> xx = cython.declare(cython.p_int)
>>>>> xx = cython.address(x)
>>>>>
>>>>> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>>>>>
>>>>> a = cython.declare(MyStruct)
>>>>> a.x = 5
>>>>>
>>>>> T = cython.typedef(MyStruct)
>>>>> b = cython.declare(cython.pointer(T))
>>>>> b[0].x = 4
>>>>
>>>>
>>>> On top of this, you can write a .pxd file to accompany your .py  
>>>> file
>>>> and it will coerce your .py file classes and def statements into  
>>>> cdef
>>>> and cpdef versions.
>>>>
>>>> It's still very new and probably highly experimental (though it  
>>>> only
>>>> modifies the cython.* nodes, so it shouldn't cause any  
>>>> regressions).
>>>> Let me know what you think.
>>>
>>> Thanks a lot for doing this!
>>
>> You're welcome.
>>
>>> I installed cython-devel and then I tried this little example:
>>>
>>>
>>> --------
>>> import Cython as cython
>>>
>>> @cython.locals(x=int, y=cython.p_int)
>>> def foo(x):
>>>     y = cython.cast(cython.p_int, x)
>>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>>     if cython.compiled:
>>>         print "the compiler was run"
>>>     else:
>>>         print "just being interpreted"
>>>
>>> foo(5)
>>> -----------
>>>
>>> When running in it in Python, it gives:
>>>
>>>   File "t.py", line 12, in <module>
>>>     foo(5)
>>>   File "t.py", line 5, in foo
>>>     y = cython.cast(cython.p_int, x)
>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in  
>>> cast
>>>     return type(arg)
>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in
>>> __init__
>>>     raise ValueError
>>> ValueError
>>>
>>>
>>> Am I using it incorrectly?
>>
>> No, it's just not letting you cast an int into an p_int. I suppose it
>> should? It just doesn't make sense to emulate this in Python.
>
> No, you are right, I was just trying to get it run.
>
>>
>>> Btw, when I compile it with Cython:
>>>
>>> $ python -c "import t; t.foo(5)"
>>> Traceback (most recent call last):
>>>   File "<string>", line 1, in <module>
>>>   File "t.py", line 12, in t (t.c:515)
>>>     foo(5)
>>>   File "t.py", line 5, in t.foo (t.c:265)
>>>     y = cython.cast(cython.p_int, x)
>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in  
>>> cast
>>>     return type(arg)
>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in
>>> __init__
>>>     raise ValueError
>>> ValueError
>>
>> You need to import cython, not import Cython as cython.
>>
>>> Looking at the code, I need to pass it Cython p_int integers,  
>>> right? I
>>> also tried:
>>>
>>> x = cython.declare(int)
>>> xx = cython.declare(cython.p_int)
>>> xx = cython.address(x)
>>> foo(xx)
>>>
>>> But this also raises the same error as above.
>>
>> This should work, Again, I think it's because you imported Cython
>> rather than cython.
>
>
> I tried that too:
>
> ondra at fuji:~/ext/cython-pure$ cat t.py
> import cython
>
> @cython.locals(x=int, y=cython.p_int)
> def foo(x):
>     y = cython.cast(cython.p_int, x)
>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>     if cython.compiled:
>         print "the compiler was run"
>     else:
>         print "just being interpreted"
>
> #foo(5)
>
> x = cython.declare(int)
> xx = cython.declare(cython.p_int)
> xx = cython.address(x)
> foo(xx)
> ondra at fuji:~/ext/cython-pure$ python t.py
> Traceback (most recent call last):
>   File "t.py", line 17, in <module>
>     foo(xx)
>   File "t.py", line 5, in foo
>     y = cython.cast(cython.p_int, x)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
>     return type(arg)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in  
> __init__
>     raise ValueError
> ValueError
>
>
> With the same output. I copied the cython.py from cython-devel, is
> that correct? I am probably doing something stupid. The error is
> really simple and it fails, but currently I am not sure what it is
> supposed to do correctly.

To clarify, this is failing for you both interpreted and compiled?  
Try changing the line

> y = cython.cast(cython.p_int, x)


to

> y = cython.address(x)


so it can make sense of it.

- Robert


From ondrej at certik.cz  Mon Oct  6 20:10:56 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 6 Oct 2008 20:10:56 +0200
Subject: [Cython] Pure python mode
In-Reply-To: <1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
Message-ID: <85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>

On Mon, Oct 6, 2008 at 7:47 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 6, 2008, at 10:25 AM, Ondrej Certik wrote:
>
>> On Mon, Oct 6, 2008 at 7:04 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>> On Oct 5, 2008, at 3:46 PM, Ondrej Certik wrote:
>>>
>>>> On Sat, Oct 4, 2008 at 1:42 PM, Robert Bradshaw
>>>> <robertwb at math.washington.edu> wrote:
>>>>> I polished off some more of the pure python mode stuff tonight. I
>>>>> just pushed it to cython-devel. Now you can do stuff like:
>>>>>
>>>>>> import cython  # you need a single file to be in your path to use
>>>>>> it in pure Python mode, provided (and installed) with Cython
>>>>>>
>>>>>> @cython.locals(x=int, y=cython.p_int)
>>>>>> def foo(x):
>>>>>>     y = cython.cast(cython.p_int, x)
>>>>>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>>>>>     if cython.compiled:
>>>>>>         print "the compiler was run
>>>>>>     else:
>>>>>>         print "just being interpreted"
>>>>>>
>>>>>> x = cython.declare(int)
>>>>>> xx = cython.declare(cython.p_int)
>>>>>> xx = cython.address(x)
>>>>>>
>>>>>> MyStruct = cython.struct(x=int, y=int, data=cython.pp_double)
>>>>>>
>>>>>> a = cython.declare(MyStruct)
>>>>>> a.x = 5
>>>>>>
>>>>>> T = cython.typedef(MyStruct)
>>>>>> b = cython.declare(cython.pointer(T))
>>>>>> b[0].x = 4
>>>>>
>>>>>
>>>>> On top of this, you can write a .pxd file to accompany your .py
>>>>> file
>>>>> and it will coerce your .py file classes and def statements into
>>>>> cdef
>>>>> and cpdef versions.
>>>>>
>>>>> It's still very new and probably highly experimental (though it
>>>>> only
>>>>> modifies the cython.* nodes, so it shouldn't cause any
>>>>> regressions).
>>>>> Let me know what you think.
>>>>
>>>> Thanks a lot for doing this!
>>>
>>> You're welcome.
>>>
>>>> I installed cython-devel and then I tried this little example:
>>>>
>>>>
>>>> --------
>>>> import Cython as cython
>>>>
>>>> @cython.locals(x=int, y=cython.p_int)
>>>> def foo(x):
>>>>     y = cython.cast(cython.p_int, x)
>>>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>>>     if cython.compiled:
>>>>         print "the compiler was run"
>>>>     else:
>>>>         print "just being interpreted"
>>>>
>>>> foo(5)
>>>> -----------
>>>>
>>>> When running in it in Python, it gives:
>>>>
>>>>   File "t.py", line 12, in <module>
>>>>     foo(5)
>>>>   File "t.py", line 5, in foo
>>>>     y = cython.cast(cython.p_int, x)
>>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in
>>>> cast
>>>>     return type(arg)
>>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in
>>>> __init__
>>>>     raise ValueError
>>>> ValueError
>>>>
>>>>
>>>> Am I using it incorrectly?
>>>
>>> No, it's just not letting you cast an int into an p_int. I suppose it
>>> should? It just doesn't make sense to emulate this in Python.
>>
>> No, you are right, I was just trying to get it run.
>>
>>>
>>>> Btw, when I compile it with Cython:
>>>>
>>>> $ python -c "import t; t.foo(5)"
>>>> Traceback (most recent call last):
>>>>   File "<string>", line 1, in <module>
>>>>   File "t.py", line 12, in t (t.c:515)
>>>>     foo(5)
>>>>   File "t.py", line 5, in t.foo (t.c:265)
>>>>     y = cython.cast(cython.p_int, x)
>>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in
>>>> cast
>>>>     return type(arg)
>>>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in
>>>> __init__
>>>>     raise ValueError
>>>> ValueError
>>>
>>> You need to import cython, not import Cython as cython.
>>>
>>>> Looking at the code, I need to pass it Cython p_int integers,
>>>> right? I
>>>> also tried:
>>>>
>>>> x = cython.declare(int)
>>>> xx = cython.declare(cython.p_int)
>>>> xx = cython.address(x)
>>>> foo(xx)
>>>>
>>>> But this also raises the same error as above.
>>>
>>> This should work, Again, I think it's because you imported Cython
>>> rather than cython.
>>
>>
>> I tried that too:
>>
>> ondra at fuji:~/ext/cython-pure$ cat t.py
>> import cython
>>
>> @cython.locals(x=int, y=cython.p_int)
>> def foo(x):
>>     y = cython.cast(cython.p_int, x)
>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>     if cython.compiled:
>>         print "the compiler was run"
>>     else:
>>         print "just being interpreted"
>>
>> #foo(5)
>>
>> x = cython.declare(int)
>> xx = cython.declare(cython.p_int)
>> xx = cython.address(x)
>> foo(xx)
>> ondra at fuji:~/ext/cython-pure$ python t.py
>> Traceback (most recent call last):
>>   File "t.py", line 17, in <module>
>>     foo(xx)
>>   File "t.py", line 5, in foo
>>     y = cython.cast(cython.p_int, x)
>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
>>     return type(arg)
>>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in
>> __init__
>>     raise ValueError
>> ValueError
>>
>>
>> With the same output. I copied the cython.py from cython-devel, is
>> that correct? I am probably doing something stupid. The error is
>> really simple and it fails, but currently I am not sure what it is
>> supposed to do correctly.
>
> To clarify, this is failing for you both interpreted and compiled?

In fact, cython refuses to compile this one:

$ cython t.py

Error converting Pyrex file to C:
------------------------------------------------------------
...
#foo(5)

x = cython.declare(int)
xx = cython.declare(cython.p_int)
xx = cython.address(x)
foo(xx)
     ^
------------------------------------------------------------

But when I tried the original foo(5), the cython version failed too.

I mean, this is some artificial example. I was just trying to get it
running to see how it works.

> Try changing the line
>
>> y = cython.cast(cython.p_int, x)
>
>
> to
>
>> y = cython.address(x)
>
>
> so it can make sense of it.

Still the same problem:

Traceback (most recent call last):
  File "t.py", line 17, in <module>
    foo(xx)
  File "t.py", line 5, in foo
    y = cython.address(x)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 22, in address
    return pointer(type(arg))([arg])
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 49, in __init__
    self._items = [cast(self._basetype, a) for a in value]
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
    return type(arg)
  File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in __init__
    raise ValueError
ValueError


But I tried this simpler example:

import cython

@cython.locals(x=int)
def foo(x):
    return x+5

print foo(5)


And this works as expected both interpreted and compiled!  Very nice.
How will you implement something like:

"
def f(x):
    return g(x)

cdef int g(int x):
    return x+5
"

E.g. having the option to write pure C functions (and classes). Maybe
something like:

@cython.locals(g=int, x=int)
def g(x):
    return x+5

?

Ondrej

From robert.kern at gmail.com  Mon Oct  6 20:51:48 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 13:51:48 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>
	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>
	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>
Message-ID: <gcdmo4$6th$1@ger.gmane.org>

Robert Bradshaw wrote:
> On Oct 6, 2008, at 10:38 AM, Dag Sverre Seljebotn wrote:
> 
>>> Robert Kern wrote:
>>> Oh, never mind. I see what you're saying. But then why do we need
>>> cython.py if
>>> everything is provided by Cython/__init__.py?
>>>
>> At least for me, Python cannot import it by the name "cython" if only
>> Cython/... is available. And I think we want "import cython" (not  
>> "import
>> Cython") to work, as lowercase is more consistent with common Python
>> naming schemes in general and just "feels more right" for such core
>> language functionality.
>>
>> (Also the module name would otherwise need to change in a lot of  
>> places --
>> when compiling under Cython, as opposed to running in Python,  
>> "cython" is
>> a magic, hard-wired module and "Cython" will not work, at least  
>> currently
>> (and my opinion is it never should))
> 
> My thoughts exactly. I think the "cython compiler codebase" and the  
> "cython magic module" as completely separate concepts. The only  
> reason Cython/__init__.py does anything is to be a workaround for  
> case insensitive filesystems.

In that case, I would recommend not using "cython" and pick another name that 
won't cause problems on such systems.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From dagss at student.matnat.uio.no  Mon Oct  6 21:20:35 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 6 Oct 2008 21:20:35 +0200 (CEST)
Subject: [Cython] About subclassing the module type
Message-ID: <33186.88.90.248.62.1223320835.squirrel@webmail.uio.no>

(I've mentioned this idea before but it now has more actuality.)

I just read ticket 95. When it comes to subclassing the module type (if
this is possible and easy and doesn't cost too much performance etc etc)
then this has another advantage: We could make early-bound symbols
read-only.

I.e.:

mymod.pyx:

cpdef f(): pass

Python session:
P> import mymod
P> mymod.f = sin
Exception raised:
CythonError("The mymod.f symbol was bound at compile-time and should thus
not be overwritten at runtime")

The main purpose is pedagogical of course, but as we work towards washing
away the differences between "C-level" code and Python-level code, stuff
like this could help getting the point across to users not used to
early-binding...

Same thing applies for "cdef classes" and so on.

Dag Sverre



From kevinar18 at hotmail.com  Mon Oct  6 21:35:24 2008
From: kevinar18 at hotmail.com (Kevin Ar18)
Date: Mon, 6 Oct 2008 15:35:24 -0400
Subject: [Cython] What to do with modules
In-Reply-To: <9E4B94B9-8591-4069-98A0-402957B5BF3A@math.washington.edu>
References: <BLU110-W23FDA85037E7F2DE4730F1AA3F0@phx.gbl>
	<9E4B94B9-8591-4069-98A0-402957B5BF3A@math.washington.edu>
Message-ID: <BLU110-W41D0912DA0573CB391BC43AA390@phx.gbl>



>> So what do you do with modules that mix Cython and Python?
> 
> cdef classes live in both the C and Python namespace. cpdef functions  
> are accessible from both C and Python as well. In this case, you  
> probably need to just write
> 
> a = 1
> 
> and deal with it being a Python object so you can get at it from  
> Python (and it is still accessible from C). Everything accessible  
> from Python is also accessible from Cython.
> 
> We probably should support cimporting cdef'd global variables  
> (probably with the same trick as we do for functions) and it would be  
> nice to be able to declare variables as "public" or "readonly" as one  
> can do with class attributes (though the only way I could see how to  
> do it is the hack at http://mail.python.org/pipermail/python-list/ 
> 2007-January/422578.html or making a subtype of the module type)
So I essentially need to merge several hundred lines of module code into one file.  :(
_________________________________________________________________
Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie.
http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008

From robertwb at math.washington.edu  Mon Oct  6 22:45:52 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 13:45:52 -0700
Subject: [Cython] What to do with modules
In-Reply-To: <BLU110-W41D0912DA0573CB391BC43AA390@phx.gbl>
References: <BLU110-W23FDA85037E7F2DE4730F1AA3F0@phx.gbl>
	<9E4B94B9-8591-4069-98A0-402957B5BF3A@math.washington.edu>
	<BLU110-W41D0912DA0573CB391BC43AA390@phx.gbl>
Message-ID: <CC92CC57-9015-434F-A202-4A4F74C5F989@math.washington.edu>

On Oct 6, 2008, at 12:35 PM, Kevin Ar18 wrote:

>
>
>>> So what do you do with modules that mix Cython and Python?
>>
>> cdef classes live in both the C and Python namespace. cpdef functions
>> are accessible from both C and Python as well. In this case, you
>> probably need to just write
>>
>> a = 1
>>
>> and deal with it being a Python object so you can get at it from
>> Python (and it is still accessible from C). Everything accessible
>> from Python is also accessible from Cython.
>>
>> We probably should support cimporting cdef'd global variables
>> (probably with the same trick as we do for functions) and it would be
>> nice to be able to declare variables as "public" or "readonly" as one
>> can do with class attributes (though the only way I could see how to
>> do it is the hack at http://mail.python.org/pipermail/python-list/
>> 2007-January/422578.html or making a subtype of the module type)
> So I essentially need to merge several hundred lines of module code  
> into one file.  :(

Probably not, that depends on what you're trying to do. Sage has  
hundreds of .pyx and .py files that all work with each other.

- Robert



From robertwb at math.washington.edu  Mon Oct  6 22:53:12 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 13:53:12 -0700
Subject: [Cython] About subclassing the module type
In-Reply-To: <33186.88.90.248.62.1223320835.squirrel@webmail.uio.no>
References: <33186.88.90.248.62.1223320835.squirrel@webmail.uio.no>
Message-ID: <4C076177-0BF0-4C4A-A6CE-BF475A4CFEAB@math.washington.edu>

On Oct 6, 2008, at 12:20 PM, Dag Sverre Seljebotn wrote:

> (I've mentioned this idea before but it now has more actuality.)
>
> I just read ticket 95. When it comes to subclassing the module type  
> (if
> this is possible and easy and doesn't cost too much performance etc  
> etc)

I looked into this a touch, and think it'd be fairly easy and very  
little (if any) performance overhead. The only question is if there  
are thing that access/manipulate the module's __dict__ directly,  
which is of course discouraged.

> then this has another advantage: We could make early-bound symbols
> read-only.

I've been thinking *exactly* the same thing.

- Robert


From robertwb at math.washington.edu  Mon Oct  6 22:58:53 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 13:58:53 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <gcdmo4$6th$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>
	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>
	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>
	<gcdmo4$6th$1@ger.gmane.org>
Message-ID: <9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>

On Oct 6, 2008, at 11:51 AM, Robert Kern wrote:

> Robert Bradshaw wrote:
>> On Oct 6, 2008, at 10:38 AM, Dag Sverre Seljebotn wrote:
>>
>>>> Robert Kern wrote:
>>>> Oh, never mind. I see what you're saying. But then why do we need
>>>> cython.py if
>>>> everything is provided by Cython/__init__.py?
>>>>
>>> At least for me, Python cannot import it by the name "cython" if  
>>> only
>>> Cython/... is available. And I think we want "import cython" (not
>>> "import
>>> Cython") to work, as lowercase is more consistent with common Python
>>> naming schemes in general and just "feels more right" for such core
>>> language functionality.
>>>
>>> (Also the module name would otherwise need to change in a lot of
>>> places --
>>> when compiling under Cython, as opposed to running in Python,
>>> "cython" is
>>> a magic, hard-wired module and "Cython" will not work, at least
>>> currently
>>> (and my opinion is it never should))
>>
>> My thoughts exactly. I think the "cython compiler codebase" and the
>> "cython magic module" as completely separate concepts. The only
>> reason Cython/__init__.py does anything is to be a workaround for
>> case insensitive filesystems.
>
> In that case, I would recommend not using "cython" and pick another  
> name that
> won't cause problems on such systems.

But it doesn't cause a problem on such systems. Personally, I can't  
think of any name that would be more appropriate than "cython."

- Robert

From robertwb at math.washington.edu  Mon Oct  6 23:11:56 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 14:11:56 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
Message-ID: <78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>


>>
>> To clarify, this is failing for you both interpreted and compiled?
>
> In fact, cython refuses to compile this one:
>
> $ cython t.py
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> #foo(5)
>
> x = cython.declare(int)
> xx = cython.declare(cython.p_int)
> xx = cython.address(x)
> foo(xx)
>      ^
> ------------------------------------------------------------
>
> But when I tried the original foo(5), the cython version failed too.

This will fail, because foo takes an int.

> I mean, this is some artificial example. I was just trying to get it
> running to see how it works.

Yeah, my example was totally artificial...

>
>> Try changing the line
>>
>>> y = cython.cast(cython.p_int, x)
>>
>>
>> to
>>
>>> y = cython.address(x)
>>
>>
>> so it can make sense of it.
>
> Still the same problem:
>
> Traceback (most recent call last):
>   File "t.py", line 17, in <module>
>     foo(xx)
>   File "t.py", line 5, in foo
>     y = cython.address(x)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 22, in  
> address
>     return pointer(type(arg))([arg])
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 49, in  
> __init__
>     self._items = [cast(self._basetype, a) for a in value]
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 14, in cast
>     return type(arg)
>   File "/home/ondra/lib/lib/python/Cython/Shadow.py", line 53, in  
> __init__
>     raise ValueError
> ValueError

This works for me:

import cython

@cython.locals(x=int, y=cython.p_int)
def foo(x):
     y = cython.address(x)
     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
     print cython.compiled

foo(34)

> But I tried this simpler example:
>
> import cython
>
> @cython.locals(x=int)
> def foo(x):
>     return x+5
>
> print foo(5)
>
>
> And this works as expected both interpreted and compiled!  Very nice.
> How will you implement something like:
>
> "
> def f(x):
>     return g(x)
>
> cdef int g(int x):
>     return x+5
> "
>
> E.g. having the option to write pure C functions (and classes). Maybe
> something like:
>
> @cython.locals(g=int, x=int)
> def g(x):
>     return x+5

Exactly how to do this with decorators (and/or automatically) is less  
clear (syntax-wise, and there are more technical issues about  
signatures matching, etc.). What you can do is write an  
accompanying .pxd file which will, of course, be ignored by Python,  
but can declare cdef classes/methods/etc for Cython. Then your class  
and def statements will be appropriately coerced when you compile.

- Robert



From dagss at student.matnat.uio.no  Mon Oct  6 23:39:22 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 6 Oct 2008 23:39:22 +0200 (CEST)
Subject: [Cython] Pure python mode
In-Reply-To: <78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
Message-ID: <33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>

Robert wrote:
...
>> E.g. having the option to write pure C functions (and classes). Maybe
>> something like:
>>
>> @cython.locals(g=int, x=int)
>> def g(x):
>>     return x+5
>
> Exactly how to do this with decorators (and/or automatically) is less
> clear (syntax-wise, and there are more technical issues about
> signatures matching, etc.). What you can do is write an
> accompanying .pxd file which will, of course, be ignored by Python,
> but can declare cdef classes/methods/etc for Cython. Then your class
> and def statements will be appropriately coerced when you compile.

Syntax-wise this might be an argument against the name "locals" (and
before a release is done there is still time to change it).

a) With another name, it could be possible to make the return value of the
function the first positional argument. (I.e. a name that means
"@cython.cpdef", rather than only affecting the locals as such, but is
more descriptive).

This is assuming the "cdef/cpdef/def" can be decided automatically
(something like make everything cpdef and for pointer argument/return
types either introduce ctypes coercion or a stub raising a runtime error
when invoked from Python).

b) I think it makes sense to eventually add support for Py3 argument
decorators as well:

@cython.locals
def g(x: cython.int):
    return x + 5

"locals" is a bad fit for this. Of course, one would select another name
for this than the "locals" we have, however that is one more decorator to
learn without any obvious advantages over a concept like "@cython.cpdef"
which could conceptually fit in both situations. One could also drop the
function decorator, but this is rather unfriendly if you use other
decorators in the same program:

def g(x: cython.int):
    return x + 5

def f(x: "The input number"):
    return x + 5

def modify_docstrings_on_module_functions():
    # must now strip cython types...

So I think there should be a decorator on a function when you decide to
Cythonize it. Then you could have chained argument decorators:

@cython.cpdef # or whatever
def g(x: (cython.int, "The input number")):
    return x + 5

and have the cython.cpdef decorator process the argument decorator tuple
and replace it with the second item, which can then be processed by Python
code.

Dag Sverre


From robertwb at math.washington.edu  Mon Oct  6 23:57:38 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 14:57:38 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
Message-ID: <47D1C67E-E763-474C-A6EC-10E9391A4DA0@math.washington.edu>

On Oct 6, 2008, at 2:39 PM, Dag Sverre Seljebotn wrote:

> Robert wrote:
> ...
>>> E.g. having the option to write pure C functions (and classes).  
>>> Maybe
>>> something like:
>>>
>>> @cython.locals(g=int, x=int)
>>> def g(x):
>>>     return x+5
>>
>> Exactly how to do this with decorators (and/or automatically) is less
>> clear (syntax-wise, and there are more technical issues about
>> signatures matching, etc.). What you can do is write an
>> accompanying .pxd file which will, of course, be ignored by Python,
>> but can declare cdef classes/methods/etc for Cython. Then your class
>> and def statements will be appropriately coerced when you compile.
>
> Syntax-wise this might be an argument against the name "locals" (and
> before a release is done there is still time to change it).

The point of locals is to declare the types of local variables,  
orthogonal to the question of whether or not it is a cdef/cpdef/ect.  
function.

> a) With another name, it could be possible to make the return value  
> of the
> function the first positional argument. (I.e. a name that means
> "@cython.cpdef", rather than only affecting the locals as such, but is
> more descriptive).
>
> This is assuming the "cdef/cpdef/def" can be decided automatically
> (something like make everything cpdef and for pointer argument/return
> types either introduce ctypes coercion or a stub raising a runtime  
> error
> when invoked from Python).
>
> b) I think it makes sense to eventually add support for Py3 argument
> decorators as well:
>
> @cython.locals
> def g(x: cython.int):
>     return x + 5

Yes, I certainly want to support this form as well, but it's only for  
Py3+ (or even if it's in 2.6, that's not old enough).

> "locals" is a bad fit for this. Of course, one would select another  
> name
> for this than the "locals" we have, however that is one more  
> decorator to
> learn without any obvious advantages over a concept like  
> "@cython.cpdef"
> which could conceptually fit in both situations.

cython.cpdef would be a bad fit for a def function(that has typed  
local variables). There should be a decorator for calling convention  
(def/cdef/cpdef) and a decorator for locals. Both can declare what  
the argument types are.

> One could also drop the
> function decorator, but this is rather unfriendly if you use other
> decorators in the same program:
>
> def g(x: cython.int):
>     return x + 5
>
> def f(x: "The input number"):
>     return x + 5
>
> def modify_docstrings_on_module_functions():
>     # must now strip cython types...
>
> So I think there should be a decorator on a function when you  
> decide to
> Cythonize it.

Yep.

> Then you could have chained argument decorators:

Yes.

>
> @cython.cpdef # or whatever
> def g(x: (cython.int, "The input number")):
>     return x + 5
>
> and have the cython.cpdef decorator process the argument decorator  
> tuple
> and replace it with the second item, which can then be processed by  
> Python
> code.


??

One issue with calling conventions is that one wants to be able to  
share them across modules/use them in classes (and subclasses), etc.  
Until we have automatic .pxd creation, this will be hard to do with  
decorators alone.

- Robert


From ondrej at certik.cz  Tue Oct  7 00:03:51 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 7 Oct 2008 00:03:51 +0200
Subject: [Cython] Pure python mode
In-Reply-To: <78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
Message-ID: <85b5c3130810061503k4615e4d9oa449b1f655b9b79c@mail.gmail.com>

> This works for me:
>
> import cython
>
> @cython.locals(x=int, y=cython.p_int)
> def foo(x):
>     y = cython.address(x)
>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>     print cython.compiled
>
> foo(34)

Wow, this works too for me (I was trying that xx case above). This is
what I get with pure python:

36
False

and cythonized:

42
True

> Exactly how to do this with decorators (and/or automatically) is less
> clear (syntax-wise, and there are more technical issues about
> signatures matching, etc.). What you can do is write an
> accompanying .pxd file which will, of course, be ignored by Python,
> but can declare cdef classes/methods/etc for Cython. Then your class
> and def statements will be appropriately coerced when you compile.

This brings me the idea, why not to just take the current syntax and
just generate a pure Python one from it? That way one also needs to
maintain just one codebase, only it will by the Cython one, but it
could always be used in Pure python.

For some reason I prefer the pure python syntax approach though.

Ondrej

From dalcinl at gmail.com  Tue Oct  7 00:41:01 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 6 Oct 2008 19:41:01 -0300
Subject: [Cython] PATH: backport pyrex support for 'ctypedef public api'
	declarations
Message-ID: <e7ba66e40810061541v75be0573xfde5c192eee0b540@mail.gmail.com>

This patch backport pyrex support for 'ctypedef public api'
declarations. You also have a testcase (should go to 'tests/compile',
or perhaps should merged in some other testcase).


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ctypedefpubapi.diff
Type: text/x-patch
Size: 1075 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081006/d788cb73/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ctypedefpubapi.pyx
Type: application/octet-stream
Size: 171 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081006/d788cb73/attachment.obj 

From robert.kern at gmail.com  Tue Oct  7 00:54:15 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 17:54:15 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>	<gcdmo4$6th$1@ger.gmane.org>
	<9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>
Message-ID: <gce4un$ou1$1@ger.gmane.org>

Robert Bradshaw wrote:
> On Oct 6, 2008, at 11:51 AM, Robert Kern wrote:
> 
>> Robert Bradshaw wrote:
>>> On Oct 6, 2008, at 10:38 AM, Dag Sverre Seljebotn wrote:
>>>
>>>>> Robert Kern wrote:
>>>>> Oh, never mind. I see what you're saying. But then why do we need
>>>>> cython.py if
>>>>> everything is provided by Cython/__init__.py?
>>>>>
>>>> At least for me, Python cannot import it by the name "cython" if  
>>>> only
>>>> Cython/... is available. And I think we want "import cython" (not
>>>> "import
>>>> Cython") to work, as lowercase is more consistent with common Python
>>>> naming schemes in general and just "feels more right" for such core
>>>> language functionality.
>>>>
>>>> (Also the module name would otherwise need to change in a lot of
>>>> places --
>>>> when compiling under Cython, as opposed to running in Python,
>>>> "cython" is
>>>> a magic, hard-wired module and "Cython" will not work, at least
>>>> currently
>>>> (and my opinion is it never should))
>>> My thoughts exactly. I think the "cython compiler codebase" and the
>>> "cython magic module" as completely separate concepts. The only
>>> reason Cython/__init__.py does anything is to be a workaround for
>>> case insensitive filesystems.
>> In that case, I would recommend not using "cython" and pick another  
>> name that
>> won't cause problems on such systems.
> 
> But it doesn't cause a problem on such systems. Personally, I can't  
> think of any name that would be more appropriate than "cython."

I guess it will probably not cause many problems. It's just a code smell that I 
would avoid. You're repeating code. And making cython.py do double-duty as both 
a module in site-packages and a script in /usr/local/bin adds to the code smell.

Oh, and don't forget to add cython.py to py_modules=, otherwise I don't think it 
will get installed correctly.

While I'm looking at the setup.py script, why are the modules in pyximport being 
listed separately in py_modules= instead of being listed as a package in packages=?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robertwb at math.washington.edu  Tue Oct  7 01:21:48 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 16:21:48 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <gce4un$ou1$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>	<gcdmo4$6th$1@ger.gmane.org>
	<9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>
	<gce4un$ou1$1@ger.gmane.org>
Message-ID: <01F39223-419D-4A32-886C-23C4CB81917B@math.washington.edu>

On Oct 6, 2008, at 3:54 PM, Robert Kern wrote:

> Robert Bradshaw wrote:
>> On Oct 6, 2008, at 11:51 AM, Robert Kern wrote:
>>
>>> Robert Bradshaw wrote:
>>>> On Oct 6, 2008, at 10:38 AM, Dag Sverre Seljebotn wrote:
>>>>
>>>>>> Robert Kern wrote:
>>>>>> Oh, never mind. I see what you're saying. But then why do we need
>>>>>> cython.py if
>>>>>> everything is provided by Cython/__init__.py?
>>>>>>
>>>>> At least for me, Python cannot import it by the name "cython" if
>>>>> only
>>>>> Cython/... is available. And I think we want "import cython" (not
>>>>> "import
>>>>> Cython") to work, as lowercase is more consistent with common  
>>>>> Python
>>>>> naming schemes in general and just "feels more right" for such  
>>>>> core
>>>>> language functionality.
>>>>>
>>>>> (Also the module name would otherwise need to change in a lot of
>>>>> places --
>>>>> when compiling under Cython, as opposed to running in Python,
>>>>> "cython" is
>>>>> a magic, hard-wired module and "Cython" will not work, at least
>>>>> currently
>>>>> (and my opinion is it never should))
>>>> My thoughts exactly. I think the "cython compiler codebase" and the
>>>> "cython magic module" as completely separate concepts. The only
>>>> reason Cython/__init__.py does anything is to be a workaround for
>>>> case insensitive filesystems.
>>> In that case, I would recommend not using "cython" and pick another
>>> name that
>>> won't cause problems on such systems.
>>
>> But it doesn't cause a problem on such systems. Personally, I can't
>> think of any name that would be more appropriate than "cython."
>
> I guess it will probably not cause many problems. It's just a code  
> smell that I
> would avoid. You're repeating code. And making cython.py do double- 
> duty as both
> a module in site-packages and a script in /usr/local/bin adds to  
> the code smell.
>
> Oh, and don't forget to add cython.py to py_modules=, otherwise I  
> don't think it
> will get installed correctly.

For those who want to ship sources without the cython dependancy, I  
they would just copy Shadow.py and name in cython.py in their own  
directory. Maybe there's a cleaner way to do this without renaming  
either "cython" or "Cython." Or should we put it there for testing  
purposes?

> While I'm looking at the setup.py script, why are the modules in  
> pyximport being
> listed separately in py_modules= instead of being listed as a  
> package in packages=?

I'm not sure, other than that I wasn't able to get the "obvious" way  
to work right away. I'm not distutils guru though, so if there's a  
better way I'd welcome a patch.

- Robert


From robert.kern at gmail.com  Tue Oct  7 01:33:11 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 18:33:11 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <01F39223-419D-4A32-886C-23C4CB81917B@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>	<gcdmo4$6th$1@ger.gmane.org>	<9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>	<gce4un$ou1$1@ger.gmane.org>
	<01F39223-419D-4A32-886C-23C4CB81917B@math.washington.edu>
Message-ID: <gce77o$v55$1@ger.gmane.org>

Robert Bradshaw wrote:
> On Oct 6, 2008, at 3:54 PM, Robert Kern wrote:
> 
>> Robert Bradshaw wrote:
>>> On Oct 6, 2008, at 11:51 AM, Robert Kern wrote:
>>>
>>>> Robert Bradshaw wrote:
>>>>> On Oct 6, 2008, at 10:38 AM, Dag Sverre Seljebotn wrote:
>>>>>
>>>>>>> Robert Kern wrote:
>>>>>>> Oh, never mind. I see what you're saying. But then why do we need
>>>>>>> cython.py if
>>>>>>> everything is provided by Cython/__init__.py?
>>>>>>>
>>>>>> At least for me, Python cannot import it by the name "cython" if
>>>>>> only
>>>>>> Cython/... is available. And I think we want "import cython" (not
>>>>>> "import
>>>>>> Cython") to work, as lowercase is more consistent with common  
>>>>>> Python
>>>>>> naming schemes in general and just "feels more right" for such  
>>>>>> core
>>>>>> language functionality.
>>>>>>
>>>>>> (Also the module name would otherwise need to change in a lot of
>>>>>> places --
>>>>>> when compiling under Cython, as opposed to running in Python,
>>>>>> "cython" is
>>>>>> a magic, hard-wired module and "Cython" will not work, at least
>>>>>> currently
>>>>>> (and my opinion is it never should))
>>>>> My thoughts exactly. I think the "cython compiler codebase" and the
>>>>> "cython magic module" as completely separate concepts. The only
>>>>> reason Cython/__init__.py does anything is to be a workaround for
>>>>> case insensitive filesystems.
>>>> In that case, I would recommend not using "cython" and pick another
>>>> name that
>>>> won't cause problems on such systems.
>>> But it doesn't cause a problem on such systems. Personally, I can't
>>> think of any name that would be more appropriate than "cython."
>> I guess it will probably not cause many problems. It's just a code  
>> smell that I
>> would avoid. You're repeating code. And making cython.py do double- 
>> duty as both
>> a module in site-packages and a script in /usr/local/bin adds to  
>> the code smell.
>>
>> Oh, and don't forget to add cython.py to py_modules=, otherwise I  
>> don't think it
>> will get installed correctly.
> 
> For those who want to ship sources without the cython dependancy, I  
> they would just copy Shadow.py and name in cython.py in their own  
> directory.

Ah. I see the intended workflow, now.

> Maybe there's a cleaner way to do this without renaming  
> either "cython" or "Cython." Or should we put it there for testing  
> purposes?

Personally, I'd just rename the thing to "cythonize" and avoid the whole mess. I 
don't see much profit in building multiple workarounds just to keep a name.

>> While I'm looking at the setup.py script, why are the modules in  
>> pyximport being
>> listed separately in py_modules= instead of being listed as a  
>> package in packages=?
> 
> I'm not sure, other than that I wasn't able to get the "obvious" way  
> to work right away. I'm not distutils guru though, so if there's a  
> better way I'd welcome a patch.

The attached should do.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cython-pyximport.diff
Url: http://codespeak.net/pipermail/cython-dev/attachments/20081006/6ae1d1e0/attachment-0001.diff 

From robertwb at math.washington.edu  Tue Oct  7 01:34:50 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 16:34:50 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <85b5c3130810061503k4615e4d9oa449b1f655b9b79c@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<85b5c3130810061503k4615e4d9oa449b1f655b9b79c@mail.gmail.com>
Message-ID: <6DD6B726-68AC-4C41-BABA-74E2E672A714@math.washington.edu>

On Oct 6, 2008, at 3:03 PM, Ondrej Certik wrote:

>> This works for me:
>>
>> import cython
>>
>> @cython.locals(x=int, y=cython.p_int)
>> def foo(x):
>>     y = cython.address(x)
>>     print x + cython.sizeof(y) + cython.sizeof(cython.p_int)
>>     print cython.compiled
>>
>> foo(34)
>
> Wow, this works too for me (I was trying that xx case above). This is
> what I get with pure python:
>
> 36
> False
>
> and cythonized:
>
> 42
> True

Yeah, the cython.sizeof stub always returns 1. The only place I could  
see sizeof being useful is a fake "malloc," etc. that return wrapped  
lists.

For most code the interpreted and compiled should do exactly the same  
thing.

>> Exactly how to do this with decorators (and/or automatically) is less
>> clear (syntax-wise, and there are more technical issues about
>> signatures matching, etc.). What you can do is write an
>> accompanying .pxd file which will, of course, be ignored by Python,
>> but can declare cdef classes/methods/etc for Cython. Then your class
>> and def statements will be appropriately coerced when you compile.
>
> This brings me the idea, why not to just take the current syntax and
> just generate a pure Python one from it? That way one also needs to
> maintain just one codebase, only it will by the Cython one, but it
> could always be used in Pure python.
>
> For some reason I prefer the pure python syntax approach though.

Same here. I'd edit the .py files then force the edit-compile-run  
cycle for that mode as well. Most code already exists (or starts out)  
as pure Python, then gets "cythonoized" which also makes it easier to  
go this direction. I also anticipate being able to point cython at  
any .py file to gain (at least some) improvements. All of these  
reasons make it easier for the .py file to be the "original" rather  
than generated.

Writing a .py writer would be a nontrivial amount of work as  
well...certainly doable but there are more productive things that I  
would rather spend my time doing :).

- Robert

From robertwb at math.washington.edu  Tue Oct  7 01:37:50 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 16:37:50 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <gce77o$v55$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>	<gcdmo4$6th$1@ger.gmane.org>	<9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>	<gce4un$ou1$1@ger.gmane.org>
	<01F39223-419D-4A32-886C-23C4CB81917B@math.washington.edu>
	<gce77o$v55$1@ger.gmane.org>
Message-ID: <F0B60574-535E-4324-9D08-54CE9FB58FEF@math.washington.edu>

On Oct 6, 2008, at 4:33 PM, Robert Kern wrote:

> Robert Bradshaw wrote:
>> On Oct 6, 2008, at 3:54 PM, Robert Kern wrote:
>>>
>>> Oh, and don't forget to add cython.py to py_modules=, otherwise  
>>> I  don't think it
>>> will get installed correctly.
>> For those who want to ship sources without the cython dependancy,  
>> I  they would just copy Shadow.py and name in cython.py in their  
>> own  directory.
>
> Ah. I see the intended workflow, now.
>
>> Maybe there's a cleaner way to do this without renaming  either  
>> "cython" or "Cython." Or should we put it there for testing   
>> purposes?
>
> Personally, I'd just rename the thing to "cythonize" and avoid the  
> whole mess. I don't see much profit in building multiple  
> workarounds just to keep a name.

It's a question of making the (internal) code a bit uglier to make  
the user interface nicer.

>>> While I'm looking at the setup.py script, why are the modules in   
>>> pyximport being
>>> listed separately in py_modules= instead of being listed as a   
>>> package in packages=?
>> I'm not sure, other than that I wasn't able to get the "obvious"  
>> way  to work right away. I'm not distutils guru though, so if  
>> there's a  better way I'd welcome a patch.
>
> The attached should do.

Thanks. Have you tried it with pyximport on a fresh Python install?  
(I think this is what I did first and had issues, but I would be  
happy to be proven wrong.)

>
> -- 
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a  
> harmless enigma
>  that is made terrible by our own mad attempt to interpret it as  
> though it had
>  an underlying truth."
>   -- Umberto Eco
> diff -r f4b39b81687b setup.py
> --- a/setup.py  Wed Sep 03 18:24:45 2008 +0200
> +++ b/setup.py  Mon Oct 06 18:32:29 2008 -0500
> @@ -97,10 +97,8 @@
>
>      'Cython.Tests',
>      'Cython.Compiler.Tests',
> +    'pyximport',
>      ],
> -
> -  # pyximport
> -  py_modules = ["pyximport/pyximport", "pyximport/pyxbuild"],
>
>    **setup_args
>    )_______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ondrej at certik.cz  Tue Oct  7 01:44:58 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 7 Oct 2008 01:44:58 +0200
Subject: [Cython] Pure python mode
In-Reply-To: <6DD6B726-68AC-4C41-BABA-74E2E672A714@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<85b5c3130810061503k4615e4d9oa449b1f655b9b79c@mail.gmail.com>
	<6DD6B726-68AC-4C41-BABA-74E2E672A714@math.washington.edu>
Message-ID: <85b5c3130810061644k9cd60cav8fe00513db3f728e@mail.gmail.com>

>> For some reason I prefer the pure python syntax approach though.
>
> Same here. I'd edit the .py files then force the edit-compile-run
> cycle for that mode as well. Most code already exists (or starts out)
> as pure Python, then gets "cythonoized" which also makes it easier to
> go this direction. I also anticipate being able to point cython at
> any .py file to gain (at least some) improvements. All of these
> reasons make it easier for the .py file to be the "original" rather
> than generated.

Yes, exactly, well written.

>
> Writing a .py writer would be a nontrivial amount of work as
> well...certainly doable but there are more productive things that I
> would rather spend my time doing :).

It's definitely better to concentrate on getting the pure python
syntax working and we'll be fine.

Ondrej

From dalcinl at gmail.com  Tue Oct  7 01:53:24 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 6 Oct 2008 20:53:24 -0300
Subject: [Cython] Pure python mode
In-Reply-To: <gce77o$v55$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org> <gcdhpi$jsr$1@ger.gmane.org>
	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>
	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>
	<gcdmo4$6th$1@ger.gmane.org>
	<9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>
	<gce4un$ou1$1@ger.gmane.org>
	<01F39223-419D-4A32-886C-23C4CB81917B@math.washington.edu>
	<gce77o$v55$1@ger.gmane.org>
Message-ID: <e7ba66e40810061653k3a7212bas9b8ba4262fac962e@mail.gmail.com>

Robert, your change requires a bit more of work. Inside pyximport/ you
have a Setup.py file plus some other stuff, like a README. Perhaps
Setup.py could be removed, and the pyximport/README contents could be
added as a docstring inside pyximport/__init__.py

I believe the distutils stuff (for the sake of Cython build/install,
not talking about Cython.Distutils) needs some cleanups and
enhacements. I'll try to make a good patch for this.

On Mon, Oct 6, 2008 at 8:33 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Robert Bradshaw wrote:
>>
>> On Oct 6, 2008, at 3:54 PM, Robert Kern wrote:
>>
>>> Robert Bradshaw wrote:
>>>>
>>>> On Oct 6, 2008, at 11:51 AM, Robert Kern wrote:
>>>>
>>>>> Robert Bradshaw wrote:
>>>>>>
>>>>>> On Oct 6, 2008, at 10:38 AM, Dag Sverre Seljebotn wrote:
>>>>>>
>>>>>>>> Robert Kern wrote:
>>>>>>>> Oh, never mind. I see what you're saying. But then why do we need
>>>>>>>> cython.py if
>>>>>>>> everything is provided by Cython/__init__.py?
>>>>>>>>
>>>>>>> At least for me, Python cannot import it by the name "cython" if
>>>>>>> only
>>>>>>> Cython/... is available. And I think we want "import cython" (not
>>>>>>> "import
>>>>>>> Cython") to work, as lowercase is more consistent with common  Python
>>>>>>> naming schemes in general and just "feels more right" for such  core
>>>>>>> language functionality.
>>>>>>>
>>>>>>> (Also the module name would otherwise need to change in a lot of
>>>>>>> places --
>>>>>>> when compiling under Cython, as opposed to running in Python,
>>>>>>> "cython" is
>>>>>>> a magic, hard-wired module and "Cython" will not work, at least
>>>>>>> currently
>>>>>>> (and my opinion is it never should))
>>>>>>
>>>>>> My thoughts exactly. I think the "cython compiler codebase" and the
>>>>>> "cython magic module" as completely separate concepts. The only
>>>>>> reason Cython/__init__.py does anything is to be a workaround for
>>>>>> case insensitive filesystems.
>>>>>
>>>>> In that case, I would recommend not using "cython" and pick another
>>>>> name that
>>>>> won't cause problems on such systems.
>>>>
>>>> But it doesn't cause a problem on such systems. Personally, I can't
>>>> think of any name that would be more appropriate than "cython."
>>>
>>> I guess it will probably not cause many problems. It's just a code  smell
>>> that I
>>> would avoid. You're repeating code. And making cython.py do double- duty
>>> as both
>>> a module in site-packages and a script in /usr/local/bin adds to  the
>>> code smell.
>>>
>>> Oh, and don't forget to add cython.py to py_modules=, otherwise I  don't
>>> think it
>>> will get installed correctly.
>>
>> For those who want to ship sources without the cython dependancy, I  they
>> would just copy Shadow.py and name in cython.py in their own  directory.
>
> Ah. I see the intended workflow, now.
>
>> Maybe there's a cleaner way to do this without renaming  either "cython"
>> or "Cython." Or should we put it there for testing  purposes?
>
> Personally, I'd just rename the thing to "cythonize" and avoid the whole
> mess. I don't see much profit in building multiple workarounds just to keep
> a name.
>
>>> While I'm looking at the setup.py script, why are the modules in
>>>  pyximport being
>>> listed separately in py_modules= instead of being listed as a  package in
>>> packages=?
>>
>> I'm not sure, other than that I wasn't able to get the "obvious" way  to
>> work right away. I'm not distutils guru though, so if there's a  better way
>> I'd welcome a patch.
>
> The attached should do.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
>  that is made terrible by our own mad attempt to interpret it as though it
> had
>  an underlying truth."
>  -- Umberto Eco
>
> diff -r f4b39b81687b setup.py
> --- a/setup.py  Wed Sep 03 18:24:45 2008 +0200
> +++ b/setup.py  Mon Oct 06 18:32:29 2008 -0500
> @@ -97,10 +97,8 @@
>
>     'Cython.Tests',
>     'Cython.Compiler.Tests',
> +    'pyximport',
>     ],
> -
> -  # pyximport
> -  py_modules = ["pyximport/pyximport", "pyximport/pyxbuild"],
>
>   **setup_args
>   )
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robert.kern at gmail.com  Tue Oct  7 02:22:40 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 19:22:40 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <F0B60574-535E-4324-9D08-54CE9FB58FEF@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdfev$8p3$1@ger.gmane.org>	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>	<gcdmo4$6th$1@ger.gmane.org>	<9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>	<gce4un$ou1$1@ger.gmane.org>	<01F39223-419D-4A32-886C-23C4CB81917B@math.washington.edu>	<gce77o$v55$1@ger.gmane.org>
	<F0B60574-535E-4324-9D08-54CE9FB58FEF@math.washington.edu>
Message-ID: <gcea4g$640$1@ger.gmane.org>

Robert Bradshaw wrote:
> On Oct 6, 2008, at 4:33 PM, Robert Kern wrote:
> 
>> Robert Bradshaw wrote:
>>> On Oct 6, 2008, at 3:54 PM, Robert Kern wrote:
>>>> Oh, and don't forget to add cython.py to py_modules=, otherwise  
>>>> I  don't think it
>>>> will get installed correctly.
>>> For those who want to ship sources without the cython dependancy,  
>>> I  they would just copy Shadow.py and name in cython.py in their  
>>> own  directory.
>> Ah. I see the intended workflow, now.
>>
>>> Maybe there's a cleaner way to do this without renaming  either  
>>> "cython" or "Cython." Or should we put it there for testing   
>>> purposes?
>> Personally, I'd just rename the thing to "cythonize" and avoid the  
>> whole mess. I don't see much profit in building multiple  
>> workarounds just to keep a name.
> 
> It's a question of making the (internal) code a bit uglier to make  
> the user interface nicer.

Personally, I would say that it's more than a bit. Playing with corner cases in 
the import mechanism is like playing with fire. But that's just my personal 
opinion, and you don't share it, so I'll drop it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robert.kern at gmail.com  Tue Oct  7 02:33:32 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 06 Oct 2008 19:33:32 -0500
Subject: [Cython] Pure python mode
In-Reply-To: <e7ba66e40810061653k3a7212bas9b8ba4262fac962e@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<gcdh9g$hkh$1@ger.gmane.org>
	<gcdhpi$jsr$1@ger.gmane.org>	<32879.88.90.248.62.1223314690.squirrel@webmail.uio.no>	<0566191B-D889-4F65-BF5D-357C6F4D5DC5@math.washington.edu>	<gcdmo4$6th$1@ger.gmane.org>	<9F93735C-3FCE-4877-AE69-FD6B16D0BF57@math.washington.edu>	<gce4un$ou1$1@ger.gmane.org>	<01F39223-419D-4A32-886C-23C4CB81917B@math.washington.edu>	<gce77o$v55$1@ger.gmane.org>
	<e7ba66e40810061653k3a7212bas9b8ba4262fac962e@mail.gmail.com>
Message-ID: <gceaot$82f$1@ger.gmane.org>

Lisandro Dalcin wrote:
> Robert, your change requires a bit more of work. Inside pyximport/ you
> have a Setup.py file plus some other stuff, like a README. Perhaps
> Setup.py could be removed, and the pyximport/README contents could be
> added as a docstring inside pyximport/__init__.py

The README doesn't get installed. The Setup.py does get installed, but is 
harmless. I've removed it, though, in the attached patch.

Cython builds, installs (both as an egg and as a regular install), and pyximport 
works according to my ad hoc testing (pyximport.install(), then importing and 
running some of the modules in tests/run/) with the attached patch.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cython-pyximport.diff
Url: http://codespeak.net/pipermail/cython-dev/attachments/20081006/7d24a78f/attachment.diff 

From greg.ewing at canterbury.ac.nz  Tue Oct  7 02:41:08 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 07 Oct 2008 13:41:08 +1300
Subject: [Cython] Pure python mode
In-Reply-To: <gcdh9g$hkh$1@ger.gmane.org>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org>
Message-ID: <48EAB024.3080208@canterbury.ac.nz>

Robert Kern wrote:

> I'm wondering if the statement "import cython" 
> might actually import Cython/__init__.py on some case-insensitive file system.

On MacOSX (case-insensitive) it seems to be able
to distinguish between them.

-- 
Greg


From kevinar18 at hotmail.com  Tue Oct  7 03:30:46 2008
From: kevinar18 at hotmail.com (Kevin Ar18)
Date: Mon, 6 Oct 2008 21:30:46 -0400
Subject: [Cython] What to do with modules
In-Reply-To: <CC92CC57-9015-434F-A202-4A4F74C5F989@math.washington.edu>
References: <BLU110-W23FDA85037E7F2DE4730F1AA3F0@phx.gbl>
	<9E4B94B9-8591-4069-98A0-402957B5BF3A@math.washington.edu>
	<BLU110-W41D0912DA0573CB391BC43AA390@phx.gbl> 
	<CC92CC57-9015-434F-A202-4A4F74C5F989@math.washington.edu>
Message-ID: <BLU110-W32523F1B7571FBC3A63DFEAA380@phx.gbl>


>> So I essentially need to merge several hundred lines of module code  
>> into one file.  :(
> 
> Probably not, that depends on what you're trying to do. Sage has  
> hundreds of .pyx and .py files that all work with each other.


A summary:
I have a project split among several pyx files/modules.
I want to convert as many vars, functions, and classes to Cython as I can (this will leave a mix of Cython and Python).

I need to be able to import my Cython and Python mixed modules and then access the Cython vars, functions, and classes directly (as well as the Python ones).

I don't know how to do this unless I...
1. Keep Python vars, functions, and classes in the modules that I can call, since I can't call Cython stuff (this is not a good option, since I want to convert as much to Cython as I can)
2. merge all modules into one file
_________________________________________________________________
Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie.
http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008

From robertwb at math.washington.edu  Tue Oct  7 03:39:47 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 18:39:47 -0700
Subject: [Cython] What to do with modules
In-Reply-To: <BLU110-W32523F1B7571FBC3A63DFEAA380@phx.gbl>
References: <BLU110-W23FDA85037E7F2DE4730F1AA3F0@phx.gbl>
	<9E4B94B9-8591-4069-98A0-402957B5BF3A@math.washington.edu>
	<BLU110-W41D0912DA0573CB391BC43AA390@phx.gbl>
	<CC92CC57-9015-434F-A202-4A4F74C5F989@math.washington.edu>
	<BLU110-W32523F1B7571FBC3A63DFEAA380@phx.gbl>
Message-ID: <3590980F-0157-47D2-971E-72426A6167B9@math.washington.edu>

On Oct 6, 2008, at 6:30 PM, Kevin Ar18 wrote:

>
>>> So I essentially need to merge several hundred lines of module code
>>> into one file.  :(
>>
>> Probably not, that depends on what you're trying to do. Sage has
>> hundreds of .pyx and .py files that all work with each other.
>
>
> A summary:
> I have a project split among several pyx files/modules.
> I want to convert as many vars, functions, and classes to Cython as  
> I can (this will leave a mix of Cython and Python).
>
> I need to be able to import my Cython and Python mixed modules and  
> then access the Cython vars, functions, and classes directly (as  
> well as the Python ones).
>
> I don't know how to do this unless I...
> 1. Keep Python vars, functions, and classes in the modules that I  
> can call, since I can't call Cython stuff (this is not a good  
> option, since I want to convert as much to Cython as I can)
> 2. merge all modules into one file

I would take a look at http://docs.cython.org/docs/ 
sharing_declarations.html . The only thing that you cannot do  
(currently) is cimport cython variables (like "cdef int i") from one  
module to another. We would like to support this, but for the moment  
it is a very small overhead, and the import would only need to happen  
once (e.g. the first time you load the module) unless your code  
depends heavily on setting module level variables from other modules,  
this shouldn't be an issue. In fact, if this is the case, you could  
restructure the code to have a cdef class that holds all the  
(previously global) variables.

- Robert


From dalcinl at gmail.com  Tue Oct  7 05:20:08 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Oct 2008 00:20:08 -0300
Subject: [Cython] Pure python mode
In-Reply-To: <48EAB024.3080208@canterbury.ac.nz>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org> <48EAB024.3080208@canterbury.ac.nz>
Message-ID: <e7ba66e40810062020yf8ae4fdrbf9dbbc2a06b1ca8@mail.gmail.com>

On Mon, Oct 6, 2008 at 9:41 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Robert Kern wrote:
>
>> I'm wondering if the statement "import cython"
>> might actually import Cython/__init__.py on some case-insensitive file system.
>
> On MacOSX (case-insensitive) it seems to be able
> to distinguish between them.
>

Perhaps __import__ black magic? Python/import.c is not something easy
to parse, but I bet packages take precedence over modules (or the
other way)... Does this make sense?

BTW, Greg, could you write an abc.py file and other ABC.py (OSX is
also case-preserving, right?), add some (different) stuff in each, and
tell all us what do you get doing 'import abc' and 'import ABC' ? Some
time ago
Brian Granger warned me about using an MPI.pxd file and other mpi.pxd,
because this could be really problematic on OSX.

As R. Kern commented in previous mail, the import machinery is not a
something we should be playing games or assuming 'standard' behavior.
And what would happen if Cython is shipped in something like a
'Portable Python' bundle living on USB stick with FAT ? And perhaps in
the future Guido will change his mind and make Python a
case-insensitive language... (just joking, but I remember people
asking for this in the past on Python-Dev. You know, "Real Programmers
use Fortran" ;-) )



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Tue Oct  7 06:10:22 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 07 Oct 2008 17:10:22 +1300
Subject: [Cython] Pure python mode
In-Reply-To: <e7ba66e40810062020yf8ae4fdrbf9dbbc2a06b1ca8@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org> <48EAB024.3080208@canterbury.ac.nz>
	<e7ba66e40810062020yf8ae4fdrbf9dbbc2a06b1ca8@mail.gmail.com>
Message-ID: <48EAE12E.40208@canterbury.ac.nz>

Lisandro Dalcin wrote:

> BTW, Greg, could you write an abc.py file and other ABC.py

You can't have both of those in the same directory at the
same time on a case-insensitive file system.

However, Python certainly takes notice of the capitalization
when importing, as it fails to find the module if you don't
use the same capitalization as exists in the file system.

My guess is that it's reading the directory itself and
comparing case-sensitively, rather than relying on the
OS to compare file names.

-- 
Greg

From robertwb at math.washington.edu  Tue Oct  7 06:45:52 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Oct 2008 21:45:52 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <e7ba66e40810062020yf8ae4fdrbf9dbbc2a06b1ca8@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<gcdfev$8p3$1@ger.gmane.org>
	<BF55CF4C-C25C-4864-86EE-E956D5BAA0F7@math.washington.edu>
	<gcdh9g$hkh$1@ger.gmane.org> <48EAB024.3080208@canterbury.ac.nz>
	<e7ba66e40810062020yf8ae4fdrbf9dbbc2a06b1ca8@mail.gmail.com>
Message-ID: <63342B1F-CEDE-42CE-9026-F081B5EBAA5F@math.washington.edu>

On Oct 6, 2008, at 8:20 PM, Lisandro Dalcin wrote:

> On Mon, Oct 6, 2008 at 9:41 PM, Greg Ewing  
> <greg.ewing at canterbury.ac.nz> wrote:
>> Robert Kern wrote:
>>
>>> I'm wondering if the statement "import cython"
>>> might actually import Cython/__init__.py on some case-insensitive  
>>> file system.
>>
>> On MacOSX (case-insensitive) it seems to be able
>> to distinguish between them.

MacOSX HSF+ is case preserving, case intensive.

> Perhaps __import__ black magic? Python/import.c is not something easy
> to parse, but I bet packages take precedence over modules (or the
> other way)... Does this make sense?

Yes, that is the case.

> BTW, Greg, could you write an abc.py file and other ABC.py (OSX is
> also case-preserving, right?), add some (different) stuff in each, and
> tell all us what do you get doing 'import abc' and 'import ABC' ? Some
> time ago
> Brian Granger warned me about using an MPI.pxd file and other mpi.pxd,
> because this could be really problematic on OSX.
>
> As R. Kern commented in previous mail, the import machinery is not a
> something we should be playing games or assuming 'standard' behavior.
> And what would happen if Cython is shipped in something like a
> 'Portable Python' bundle living on USB stick with FAT ? And perhaps in
> the future Guido will change his mind and make Python a
> case-insensitive language... (just joking, but I remember people
> asking for this in the past on Python-Dev. You know, "Real Programmers
> use Fortran" ;-) )

The current way works no matter which it chooses first--as long as it  
can find cython.py or Cython (and Python's going to be horribly  
broken if either of those fail) then it just works.

- Robert



From stefan_ml at behnel.de  Wed Oct  8 09:04:47 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 08 Oct 2008 09:04:47 +0200
Subject: [Cython] cython-devel seems broken
Message-ID: <48EC5B8F.6080502@behnel.de>

Hi,

I get this on an "hg pull" from cython-devel:

	searching for changes
	adding changesets
	adding manifests
	adding file changes
	abort: unknown parent 90e366e55871!
	transaction abort!
	rollback completed

Is this fixable somehow?

Stefan

From robertwb at math.washington.edu  Wed Oct  8 09:10:18 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 8 Oct 2008 00:10:18 -0700
Subject: [Cython] cython-devel seems broken
In-Reply-To: <48EC5B8F.6080502@behnel.de>
References: <48EC5B8F.6080502@behnel.de>
Message-ID: <D8D08143-8CA8-41B4-BDE9-F93844F95F51@math.washington.edu>

Never seen that before... can you do hg outgoing or hg incoming?

On Oct 8, 2008, at 12:04 AM, Stefan Behnel wrote:

> Hi,
>
> I get this on an "hg pull" from cython-devel:
>
> 	searching for changes
> 	adding changesets
> 	adding manifests
> 	adding file changes
> 	abort: unknown parent 90e366e55871!
> 	transaction abort!
> 	rollback completed
>
> Is this fixable somehow?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Wed Oct  8 09:39:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 08 Oct 2008 09:39:00 +0200
Subject: [Cython] cython-devel seems broken
Message-ID: <3306303563.315674@smtp.netcom.no>

Does it work to clone it?

I did the last push (as of yesterday evening at least), there was a couple of merges but nothing out of the ordinary...

I was able to clone it fresh from the repo yesterday.

You could also try to pull -dagss, which should contain the same stuff now (for diagnostics purposes).

Dag Sverre Seljebotn
-----Original Message-----
From: Stefan Behnel <stefan_ml at behnel.de>
Date: Wednesday, Oct 8, 2008 9:04 am
Subject: [Cython] cython-devel seems broken
To: Cython-dev <cython-dev at codespeak.net>Reply-To: cython-dev at codespeak.net

Hi,
>
>I get this on an "hg pull" from cython-devel:
>
>	searching for changes
>	adding changesets
>	adding manifests
>	adding file changes
>	abort: unknown parent 90e366e55871!
>	transaction abort!
>	rollback completed
>
>Is this fixable somehow?
>
>Stefan
>_______________________________________________
>Cython-dev mailing list
>Cython-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/cython-dev
>


From dalcinl at gmail.com  Wed Oct  8 15:17:21 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 8 Oct 2008 10:17:21 -0300
Subject: [Cython] cython-devel seems broken
In-Reply-To: <3306303563.315674@smtp.netcom.no>
References: <3306303563.315674@smtp.netcom.no>
Message-ID: <e7ba66e40810080617u3c9b638ft64982bf40c0ecd10@mail.gmail.com>

>From my side, 'hg pull' a fresh 'hg clone' worked just fine.

On Wed, Oct 8, 2008 at 4:39 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Does it work to clone it?
>
> I did the last push (as of yesterday evening at least), there was a couple of merges but nothing out of the ordinary...
>
> I was able to clone it fresh from the repo yesterday.
>
> You could also try to pull -dagss, which should contain the same stuff now (for diagnostics purposes).
>
> Dag Sverre Seljebotn
> -----Original Message-----
> From: Stefan Behnel <stefan_ml at behnel.de>
> Date: Wednesday, Oct 8, 2008 9:04 am
> Subject: [Cython] cython-devel seems broken
> To: Cython-dev <cython-dev at codespeak.net>Reply-To: cython-dev at codespeak.net
>
> Hi,
>>
>>I get this on an "hg pull" from cython-devel:
>>
>>       searching for changes
>>       adding changesets
>>       adding manifests
>>       adding file changes
>>       abort: unknown parent 90e366e55871!
>>       transaction abort!
>>       rollback completed
>>
>>Is this fixable somehow?
>>
>>Stefan
>>_______________________________________________
>>Cython-dev mailing list
>>Cython-dev at codespeak.net
>>http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Oct  8 19:48:19 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 8 Oct 2008 14:48:19 -0300
Subject: [Cython] WARN: isinstace optimization disabled!
Message-ID: <e7ba66e40810081048t726fa994rf7a8d618ccee6c76@mail.gmail.com>

Robert, I believe you forgot to remove an 'if 0 ...' in
Cython/Compiler/Optimize.py . 'isinstance' optimization is then
disabled. Please review!

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From aaron.devore at gmail.com  Wed Oct  8 21:15:59 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Wed, 8 Oct 2008 12:15:59 -0700
Subject: [Cython] list.index(instance) by identity or __eq__?
Message-ID: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>

I'm writing a class/type that will be inserted into a list. After it
is inserted the developer should be able to call
a_list.index(my_type_instance) and get the my_type_instance index by
*identity*, not by a call to my_type.__eq__. I have a my_type.__eq__
function but it doesn't give the correct behavior. Which behavior does
Cython use?

Cheers!
Aaron

From dagss at student.matnat.uio.no  Wed Oct  8 21:33:43 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 08 Oct 2008 21:33:43 +0200
Subject: [Cython] list.index(instance) by identity or __eq__?
In-Reply-To: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
References: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
Message-ID: <48ED0B17.5060400@student.matnat.uio.no>

Aaron DeVore wrote:
> I'm writing a class/type that will be inserted into a list. After it
> is inserted the developer should be able to call
> a_list.index(my_type_instance) and get the my_type_instance index by
> *identity*, not by a call to my_type.__eq__. I have a my_type.__eq__
> function but it doesn't give the correct behavior. Which behavior does
> Cython use?

Cython lists are Python lists, so this should behave in the same way as 
in Python. If something is not working it would help with a code small 
example.

I.e. to do what you describe you should either not implement __eq__ at 
all, or have it return "self is other".

More details would help here (i.e. a working example of what you are 
trying to do and how it works in Python).

If you are trying to do something which cannot be done in Python then 
the answer is no, I suppose. You could consider writing a wrapper like this:

class IdWrapper:
     def __init__(self, x): self.x = x
     def __eq__(self, other): return self.x is other.x

and only insert objects through their wrapper...

-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Oct  9 07:58:14 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 09 Oct 2008 07:58:14 +0200
Subject: [Cython] cython-devel seems broken
In-Reply-To: <e7ba66e40810080617u3c9b638ft64982bf40c0ecd10@mail.gmail.com>
References: <3306303563.315674@smtp.netcom.no>
	<e7ba66e40810080617u3c9b638ft64982bf40c0ecd10@mail.gmail.com>
Message-ID: <48ED9D76.4070307@behnel.de>


Lisandro Dalcin wrote:
>>From my side, 'hg pull' a fresh 'hg clone' worked just fine.

Ok, works for me, too. It's just that I'll have to rescue the non-hg-ed stuff
lying around in my working copy that way, so I hope this won't happen too often.

Thanks,

Stefan

From animator333 at yahoo.com  Thu Oct  9 18:48:03 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Thu, 9 Oct 2008 22:18:03 +0530 (IST)
Subject: [Cython] pickle error when package is used
Message-ID: <595839.6948.qm@web94908.mail.in2.yahoo.com>

hi,

I did a simple test of storing object's instance using pickle and shelve.
In the attached archive you can run "test.py", pickle and shelve both are working fine in read and write mode.
Now change line no. 12 in "test.py" to:

from core.Node import Node

inside core directiory we have the same "Node.pyd".

Run "test.py" again, and here we get an error saying:
cPickle.PicklingError: Can't pickle <class 'Node.Node'>: it's not the same object as Node.Node

In case if you use "Node.pyc", (delete "Node.pyd") everything works fine.

Am I missing something here?

Regards

Prashant


      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081009/3d5dfb6d/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cython_test.zip
Type: application/zip
Size: 19873 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081009/3d5dfb6d/attachment-0001.zip 

From robertwb at math.washington.edu  Thu Oct  9 19:46:17 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 10:46:17 -0700
Subject: [Cython] list.index(instance) by identity or __eq__?
In-Reply-To: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
References: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
Message-ID: <649543CF-09FF-4371-9AB8-BD71C12627FB@math.washington.edu>

On Oct 8, 2008, at 12:15 PM, Aaron DeVore wrote:

> I'm writing a class/type that will be inserted into a list. After it
> is inserted the developer should be able to call
> a_list.index(my_type_instance) and get the my_type_instance index by
> *identity*, not by a call to my_type.__eq__. I have a my_type.__eq__
> function but it doesn't give the correct behavior. Which behavior does
> Cython use?

I think the issue you're running into is that Python extension  
classes use rich comparison rather than __eq__ and friends. See  
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/ 
Manual/special_methods.html .

- Robert



From dalcinl at gmail.com  Thu Oct  9 21:55:32 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 9 Oct 2008 16:55:32 -0300
Subject: [Cython] list.index(instance) by identity or __eq__?
In-Reply-To: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
References: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
Message-ID: <e7ba66e40810091255n230b38eerc2bdb9b9bb5244f@mail.gmail.com>

On Wed, Oct 8, 2008 at 4:15 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
> I have a my_type.__eq__
> function but it doesn't give the correct behavior. Which behavior does
> Cython use?

Could you show us the way you implemented your __eq__ method?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Oct  9 21:59:41 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 12:59:41 -0700
Subject: [Cython] WARN: isinstace optimization disabled!
In-Reply-To: <e7ba66e40810081048t726fa994rf7a8d618ccee6c76@mail.gmail.com>
References: <e7ba66e40810081048t726fa994rf7a8d618ccee6c76@mail.gmail.com>
Message-ID: <34895C88-F448-4557-BF07-AFDDF7AA07D5@math.washington.edu>

Yes, this is an oversight on my part. As we have more optimizations,  
we need to find a way to test if they're being used (maybe a regex  
that must be present in the generated C?)

Thanks for catching this, I'll re-enable.

On Oct 8, 2008, at 10:48 AM, Lisandro Dalcin wrote:

> Robert, I believe you forgot to remove an 'if 0 ...' in
> Cython/Compiler/Optimize.py . 'isinstance' optimization is then
> disabled. Please review!
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Thu Oct  9 22:49:12 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 13:49:12 -0700
Subject: [Cython] Pure python mode
In-Reply-To: <33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
Message-ID: <E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>

I'd like to release this soon. I've changed declare so it can work as  
below (i.e. declare(var=type, ..)). For now there's no "cpdef/ 
cdefclass/earlybind/??" decorator, we'll hammer that one out later.  
Any other comments before we commit to it?

- Robert


On Oct 6, 2008, at 2:39 PM, Dag Sverre Seljebotn wrote:

> Robert wrote:
> ...
>>> E.g. having the option to write pure C functions (and classes).  
>>> Maybe
>>> something like:
>>>
>>> @cython.locals(g=int, x=int)
>>> def g(x):
>>>     return x+5
>>
>> Exactly how to do this with decorators (and/or automatically) is less
>> clear (syntax-wise, and there are more technical issues about
>> signatures matching, etc.). What you can do is write an
>> accompanying .pxd file which will, of course, be ignored by Python,
>> but can declare cdef classes/methods/etc for Cython. Then your class
>> and def statements will be appropriately coerced when you compile.
>
> Syntax-wise this might be an argument against the name "locals" (and
> before a release is done there is still time to change it).
>
> a) With another name, it could be possible to make the return value  
> of the
> function the first positional argument. (I.e. a name that means
> "@cython.cpdef", rather than only affecting the locals as such, but is
> more descriptive).
>
> This is assuming the "cdef/cpdef/def" can be decided automatically
> (something like make everything cpdef and for pointer argument/return
> types either introduce ctypes coercion or a stub raising a runtime  
> error
> when invoked from Python).
>
> b) I think it makes sense to eventually add support for Py3 argument
> decorators as well:
>
> @cython.locals
> def g(x: cython.int):
>     return x + 5
>
> "locals" is a bad fit for this. Of course, one would select another  
> name
> for this than the "locals" we have, however that is one more  
> decorator to
> learn without any obvious advantages over a concept like  
> "@cython.cpdef"
> which could conceptually fit in both situations. One could also  
> drop the
> function decorator, but this is rather unfriendly if you use other
> decorators in the same program:
>
> def g(x: cython.int):
>     return x + 5
>
> def f(x: "The input number"):
>     return x + 5
>
> def modify_docstrings_on_module_functions():
>     # must now strip cython types...
>
> So I think there should be a decorator on a function when you  
> decide to
> Cythonize it. Then you could have chained argument decorators:
>
> @cython.cpdef # or whatever
> def g(x: (cython.int, "The input number")):
>     return x + 5
>
> and have the cython.cpdef decorator process the argument decorator  
> tuple
> and replace it with the second item, which can then be processed by  
> Python
> code.
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Thu Oct  9 23:35:33 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 9 Oct 2008 18:35:33 -0300
Subject: [Cython] cython-devel broken again
Message-ID: <e7ba66e40810091435n604b78b8m615149b106bf8c54@mail.gmail.com>

Trying to install Cython from cython-devel generates the following error:

$ python setup.py install --home=/u/dalcinl
Compiling module Cython.Plex.Scanners ...
ERROR: Internal compiler error: Result forced on non-temp node
Extension module compilation failed, using plain Python implementation
running install
<and more output ...>

Please review...

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Oct  9 23:57:52 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 09 Oct 2008 23:57:52 +0200
Subject: [Cython] Pure python mode
In-Reply-To: <E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
	<E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>
Message-ID: <48EE7E60.7040502@student.matnat.uio.no>

Robert Bradshaw wrote:
> I'd like to release this soon. I've changed declare so it can work as  
> below (i.e. declare(var=type, ..)). For now there's no "cpdef/ 
> cdefclass/earlybind/??" decorator, we'll hammer that one out later.  
> Any other comments before we commit to it?

Hmm.. Well done! :-)

Comments about release schedule though:

For NumPy complex numbers are now in place which was the big lacking 
feature (of course, that makes native complex float support the next big 
lacking feature for numerical use). Structs/record arrays are incredibly 
near the surface (less than a day, easily), OTOH I'm very pressed on 
time at the moment, so I suppose they should not block a release.

There definitely needs to be betas this time around too, the result_code 
refactoring is more dangerous than anything we did in summer. (In fact 
the issue Lisandro just posted may be related to this...)

BTW, are we now anywhere near keeping a feature sync with Pyrex? I'm 
wondering if we could perhaps think again about the versioning scheme -- 
  the version number is getting very long to refer to, and the link to 
Pyrex seems less important than it used to. (Obviously it needs to not 
mess with packaging systems, but both 0.10 and 0.9.9 could work, rather 
than 0.9.8.2).

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Oct 10 00:04:33 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 15:04:33 -0700
Subject: [Cython] Next release
In-Reply-To: <48EE7E60.7040502@student.matnat.uio.no>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>	<85b5c3130810051546h1d953077n3becd7231525448f@mail.gmail.com>	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
	<E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>
	<48EE7E60.7040502@student.matnat.uio.no>
Message-ID: <12A83390-287F-4830-A795-C1A72A97793C@math.washington.edu>

On Oct 9, 2008, at 2:57 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> I'd like to release this soon. I've changed declare so it can work as
>> below (i.e. declare(var=type, ..)). For now there's no "cpdef/
>> cdefclass/earlybind/??" decorator, we'll hammer that one out later.
>> Any other comments before we commit to it?
>
> Hmm.. Well done! :-)
>
> Comments about release schedule though:
>
> For NumPy complex numbers are now in place which was the big lacking
> feature

Yeah, I saw some commits that direction but haven't yet tried it out.  
This should be very nice.

> (of course, that makes native complex float support the next big
> lacking feature for numerical use).

Yep.

> Structs/record arrays are incredibly
> near the surface (less than a day, easily), OTOH I'm very pressed on
> time at the moment, so I suppose they should not block a release.

If you get them done, cool, otherwise we'll continue without them for  
now.

> There definitely needs to be betas this time around too, the  
> result_code
> refactoring is more dangerous than anything we did in summer. (In fact
> the issue Lisandro just posted may be related to this...)

Yep. Any suggestions other than just reporting to the mailing list  
and compiling sage/lxml/etc?

> BTW, are we now anywhere near keeping a feature sync with Pyrex?

I believe the only feature we are lacking is multiple file  
compilation, but I could be wrong. There are also possibly some bug  
fixes in the latest that we haven't caught yet.

> I'm wondering if we could perhaps think again about the versioning  
> scheme --
>   the version number is getting very long to refer to, and the link to
> Pyrex seems less important than it used to. (Obviously it needs to not
> mess with packaging systems, but both 0.10 and 0.9.9 could work,  
> rather
> than 0.9.8.2).

Yes, this is worth revisiting.

- Robert

From dalcinl at gmail.com  Fri Oct 10 00:12:04 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 9 Oct 2008 19:12:04 -0300
Subject: [Cython] Next release
In-Reply-To: <12A83390-287F-4830-A795-C1A72A97793C@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
	<E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>
	<48EE7E60.7040502@student.matnat.uio.no>
	<12A83390-287F-4830-A795-C1A72A97793C@math.washington.edu>
Message-ID: <e7ba66e40810091512x7f881f58mfebf21afd51a26b5@mail.gmail.com>

On Thu, Oct 9, 2008 at 7:04 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 9, 2008, at 2:57 PM, Dag Sverre Seljebotn wrote:
>> BTW, are we now anywhere near keeping a feature sync with Pyrex?
>
> I believe the only feature we are lacking is multiple file
> compilation, but I could be wrong. There are also possibly some bug
> fixes in the latest that we haven't caught yet.
>

BTW, Could someone take a look at the patch I've sent a couple of days
ago about a Pyrex feature backport?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Oct 10 00:14:23 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 15:14:23 -0700
Subject: [Cython] cython-devel broken again
In-Reply-To: <e7ba66e40810091435n604b78b8m615149b106bf8c54@mail.gmail.com>
References: <e7ba66e40810091435n604b78b8m615149b106bf8c54@mail.gmail.com>
Message-ID: <59B15F71-A200-4287-A3C4-F4FA104B6A66@math.washington.edu>

On Oct 9, 2008, at 2:35 PM, Lisandro Dalcin wrote:

> Trying to install Cython from cython-devel generates the following  
> error:
>
> $ python setup.py install --home=/u/dalcinl
> Compiling module Cython.Plex.Scanners ...
> ERROR: Internal compiler error: Result forced on non-temp node
> Extension module compilation failed, using plain Python implementation
> running install
> <and more output ...>
>
> Please review...

Oh, I see what went wrong. Fixed.

- Robert



From robertwb at math.washington.edu  Fri Oct 10 00:15:35 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 15:15:35 -0700
Subject: [Cython] Robert, please help!
In-Reply-To: <e7ba66e40810032129u8e75f6fy12063ada1c90042e@mail.gmail.com>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
	<e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
	<48E6A4FF.1070900@canterbury.ac.nz>
	<e7ba66e40810032046s40722170o79a522df527323fa@mail.gmail.com>
	<48E6EB56.9090605@canterbury.ac.nz>
	<e7ba66e40810032129u8e75f6fy12063ada1c90042e@mail.gmail.com>
Message-ID: <BEC8EDCC-47AA-4172-A0DD-C21AD75097A0@math.washington.edu>


On Oct 3, 2008, at 9:29 PM, Lisandro Dalcin wrote:

> OK, at this link http://publications.gbdirect.co.uk/c_book/chapter8/ 
> typedef.html
>
> I've found the following example:
>
> /*
> * Using typedef, declare 'func' to have type
> * 'function taking two int arguments, returning int'
> */
> typedef int func(int, int);
> /* ERROR */
> func func_name{ /*....*/ }
> /* Correct. Returns pointer to a type 'func' */
> func *func_name(){ /*....*/ }
> /*
> * Correct if functions could return functions,
> * but C can't.
> */
> func func_name(){ /*....*/ }
>
>
> I'm fine with that. So I?ll try tomorrow if Robert?s modifications can
> handle this form. Thank's for the point, Greg!
>

Lisandro,

What was the final resolution on this?

- Robert



From robertwb at math.washington.edu  Fri Oct 10 00:17:40 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 15:17:40 -0700
Subject: [Cython] Next release
In-Reply-To: <e7ba66e40810091512x7f881f58mfebf21afd51a26b5@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<A267C914-808C-45CD-B347-3C00AE9E943C@math.washington.edu>
	<85b5c3130810061025n696ea13xe81af4231eb04ff0@mail.gmail.com>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
	<E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>
	<48EE7E60.7040502@student.matnat.uio.no>
	<12A83390-287F-4830-A795-C1A72A97793C@math.washington.edu>
	<e7ba66e40810091512x7f881f58mfebf21afd51a26b5@mail.gmail.com>
Message-ID: <2A0D365C-106F-4F25-AC9D-29C2B9054151@math.washington.edu>

On Oct 9, 2008, at 3:12 PM, Lisandro Dalcin wrote:

> On Thu, Oct 9, 2008 at 7:04 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Oct 9, 2008, at 2:57 PM, Dag Sverre Seljebotn wrote:
>>> BTW, are we now anywhere near keeping a feature sync with Pyrex?
>>
>> I believe the only feature we are lacking is multiple file
>> compilation, but I could be wrong. There are also possibly some bug
>> fixes in the latest that we haven't caught yet.
>>
>
> BTW, Could someone take a look at the patch I've sent a couple of days
> ago about a Pyrex feature backport?

The typedef one, yes. Also there's the set optimization one that I'd  
like to get in as well (for 2.4+).

Do you have an account on trac? Then we can post these issues for the  
next release to make sure they get done (or consciously postponed,  
though I don't think that's the case with either of these).

- Robert



From dalcinl at gmail.com  Fri Oct 10 00:36:32 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 9 Oct 2008 19:36:32 -0300
Subject: [Cython] Next release
In-Reply-To: <2A0D365C-106F-4F25-AC9D-29C2B9054151@math.washington.edu>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
	<E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>
	<48EE7E60.7040502@student.matnat.uio.no>
	<12A83390-287F-4830-A795-C1A72A97793C@math.washington.edu>
	<e7ba66e40810091512x7f881f58mfebf21afd51a26b5@mail.gmail.com>
	<2A0D365C-106F-4F25-AC9D-29C2B9054151@math.washington.edu>
Message-ID: <e7ba66e40810091536p116434eeq8052889cd972327@mail.gmail.com>

On Thu, Oct 9, 2008 at 7:17 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>> BTW, Could someone take a look at the patch I've sent a couple of days
>> ago about a Pyrex feature backport?
>
> The typedef one, yes. Also there's the set optimization one that I'd
> like to get in as well (for 2.4+).
>
> Do you have an account on trac? Then we can post these issues for the
> next release to make sure they get done (or consciously postponed,
> though I don't think that's the case with either of these).

No, I do not have a Trac account. How do I get one?

BTW, Is there any chance I could get commit permissions for
cython-devel? Do not hessitate to say NO... but that would make far
easier for me to fix trivial bugs (like the previous one related to
isinstance optimization) and push small patches (like the Pyrex
backport, with IMHO do not need to much review).



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Oct 10 00:54:50 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 9 Oct 2008 19:54:50 -0300
Subject: [Cython] Robert, please help!
In-Reply-To: <BEC8EDCC-47AA-4172-A0DD-C21AD75097A0@math.washington.edu>
References: <e7ba66e40810011842o2402b25dk7da32557a1326df3@mail.gmail.com>
	<D2DEA562-72AC-4D54-98A5-9597005A256D@math.washington.edu>
	<e7ba66e40810030715t51546976rb384f615ad70c34@mail.gmail.com>
	<48E6A4FF.1070900@canterbury.ac.nz>
	<e7ba66e40810032046s40722170o79a522df527323fa@mail.gmail.com>
	<48E6EB56.9090605@canterbury.ac.nz>
	<e7ba66e40810032129u8e75f6fy12063ada1c90042e@mail.gmail.com>
	<BEC8EDCC-47AA-4172-A0DD-C21AD75097A0@math.washington.edu>
Message-ID: <e7ba66e40810091554m5292fcd9odcd3dbd8c0e1b295@mail.gmail.com>

You have attached three different patches fixing this in different
ways. But I really believe the #3 is the best, safest one.

You also have a small pyx file in orther you can easily spot the
geneated code and the behavior of patch #3. WARNING: You have to
generate C sources and test with

$ cython --cleanup 9 modglbinit.pyx
$ <gcc modglbinit,c -> modglbinit.so>
$ python -c 'import modglb'

This example is vile hackery, and SHOULD NOT be included in the
testsuite right now (unless the testsuite generates C sources with
--cleanup options?). Perhaps if --cleanup could be passed as a
directive, then it could go in.


On Thu, Oct 9, 2008 at 7:15 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> On Oct 3, 2008, at 9:29 PM, Lisandro Dalcin wrote:
>
>> OK, at this link http://publications.gbdirect.co.uk/c_book/chapter8/
>> typedef.html
>>
>> I've found the following example:
>>
>> /*
>> * Using typedef, declare 'func' to have type
>> * 'function taking two int arguments, returning int'
>> */
>> typedef int func(int, int);
>> /* ERROR */
>> func func_name{ /*....*/ }
>> /* Correct. Returns pointer to a type 'func' */
>> func *func_name(){ /*....*/ }
>> /*
>> * Correct if functions could return functions,
>> * but C can't.
>> */
>> func func_name(){ /*....*/ }
>>
>>
>> I'm fine with that. So I?ll try tomorrow if Robert?s modifications can
>> handle this form. Thank's for the point, Greg!
>>
>
> Lisandro,
>
> What was the final resolution on this?
>
> - Robert
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cleanup_option1.diff
Type: text/x-patch
Size: 1266 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081009/69863e9f/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cleanup_option2.diff
Type: text/x-patch
Size: 743 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081009/69863e9f/attachment-0001.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cleanup_option3.diff
Type: text/x-patch
Size: 1006 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081009/69863e9f/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: modglbinit.pyx
Type: application/octet-stream
Size: 601 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081009/69863e9f/attachment.obj 

From robertwb at math.washington.edu  Fri Oct 10 08:55:13 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Oct 2008 23:55:13 -0700
Subject: [Cython] Next release
In-Reply-To: <e7ba66e40810091536p116434eeq8052889cd972327@mail.gmail.com>
References: <20CF4396-1D84-404F-9BCC-2A2CC986F259@math.washington.edu>
	<1648675C-9FE0-4D45-B9A9-0D70214D6CCC@math.washington.edu>
	<85b5c3130810061110i674c87cfx1fe427d8cde1ddb3@mail.gmail.com>
	<78CB683C-F680-469E-BB4C-F1C771CA4E12@math.washington.edu>
	<33197.88.90.248.62.1223329162.squirrel@webmail.uio.no>
	<E96C653C-E3CA-449D-B12C-613DDD8D5587@math.washington.edu>
	<48EE7E60.7040502@student.matnat.uio.no>
	<12A83390-287F-4830-A795-C1A72A97793C@math.washington.edu>
	<e7ba66e40810091512x7f881f58mfebf21afd51a26b5@mail.gmail.com>
	<2A0D365C-106F-4F25-AC9D-29C2B9054151@math.washington.edu>
	<e7ba66e40810091536p116434eeq8052889cd972327@mail.gmail.com>
Message-ID: <AEC3D4AB-986C-4D72-BE43-1A1A8715D08B@math.washington.edu>

On Oct 9, 2008, at 3:36 PM, Lisandro Dalcin wrote:

> On Thu, Oct 9, 2008 at 7:17 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>> BTW, Could someone take a look at the patch I've sent a couple of  
>>> days
>>> ago about a Pyrex feature backport?
>>
>> The typedef one, yes. Also there's the set optimization one that I'd
>> like to get in as well (for 2.4+).
>>
>> Do you have an account on trac? Then we can post these issues for the
>> next release to make sure they get done (or consciously postponed,
>> though I don't think that's the case with either of these).
>
> No, I do not have a Trac account. How do I get one?

Just send me an .htpasswd file off-list (just trying to prevent spam  
here).

> BTW, Is there any chance I could get commit permissions for
> cython-devel? Do not hessitate to say NO... but that would make far
> easier for me to fix trivial bugs (like the previous one related to
> isinstance optimization) and push small patches (like the Pyrex
> backport, with IMHO do not need to much review).


Yes, I'd be glad to do that. I'll set that up for you (using the same  
passwords as above).

- Robert


From aaron.devore at gmail.com  Fri Oct 10 09:28:01 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Fri, 10 Oct 2008 00:28:01 -0700
Subject: [Cython] list.index(instance) by identity or __eq__?
In-Reply-To: <e7ba66e40810091255n230b38eerc2bdb9b9bb5244f@mail.gmail.com>
References: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
	<e7ba66e40810091255n230b38eerc2bdb9b9bb5244f@mail.gmail.com>
Message-ID: <2ead2fb0810100028u3b1ba737p3edd72845e8c1799@mail.gmail.com>

Tragically, I can't show my implementation of __eq__. However, I will
say this: I absolutely cannot change __eq__ to work with identities.

For a relatively short list (usually 0-20 items, sometimes up to 1000)
would this work and still be blindingly fast? "----" indicates
indentation.:

int index
for index from 0 <= index < len(a_list):
----if a_list[index] is my_type_instance:
--------break

Or would I need to do this to get correct behavior:
int index
for index from 0 <= index < len(a_list):
----if id(a_list[index]) == id(my_type_instance):
--------break

2008/10/9 Lisandro Dalcin <dalcinl at gmail.com>:
> On Wed, Oct 8, 2008 at 4:15 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
>> I have a my_type.__eq__
>> function but it doesn't give the correct behavior. Which behavior does
>> Cython use?
>
> Could you show us the way you implemented your __eq__ method?
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Fri Oct 10 10:02:56 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Oct 2008 01:02:56 -0700
Subject: [Cython] list.index(instance) by identity or __eq__?
In-Reply-To: <2ead2fb0810100028u3b1ba737p3edd72845e8c1799@mail.gmail.com>
References: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
	<e7ba66e40810091255n230b38eerc2bdb9b9bb5244f@mail.gmail.com>
	<2ead2fb0810100028u3b1ba737p3edd72845e8c1799@mail.gmail.com>
Message-ID: <9264823B-EDAB-4BAD-8B6F-BC0F59ABC940@math.washington.edu>

On Oct 10, 2008, at 12:28 AM, Aaron DeVore wrote:

> Tragically, I can't show my implementation of __eq__. However, I will
> say this: I absolutely cannot change __eq__ to work with identities.
>
> For a relatively short list (usually 0-20 items, sometimes up to 1000)
> would this work and still be blindingly fast? "----" indicates
> indentation.:
>
> int index
> for index from 0 <= index < len(a_list):
> ----if a_list[index] is my_type_instance:
> --------break

This will compare on identity, if that's what you want then it should  
be quite fast.

> Or would I need to do this to get correct behavior:
> int index
> for index from 0 <= index < len(a_list):
> ----if id(a_list[index]) == id(my_type_instance):
> --------break

This is essentially how list.index() is implemented.

>
> 2008/10/9 Lisandro Dalcin <dalcinl at gmail.com>:
>> On Wed, Oct 8, 2008 at 4:15 PM, Aaron DeVore  
>> <aaron.devore at gmail.com> wrote:
>>> I have a my_type.__eq__
>>> function but it doesn't give the correct behavior. Which behavior  
>>> does
>>> Cython use?
>>
>> Could you show us the way you implemented your __eq__ method?
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From aaron.devore at gmail.com  Fri Oct 10 10:25:32 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Fri, 10 Oct 2008 01:25:32 -0700
Subject: [Cython] list.index(instance) by identity or __eq__?
In-Reply-To: <9264823B-EDAB-4BAD-8B6F-BC0F59ABC940@math.washington.edu>
References: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
	<e7ba66e40810091255n230b38eerc2bdb9b9bb5244f@mail.gmail.com>
	<2ead2fb0810100028u3b1ba737p3edd72845e8c1799@mail.gmail.com>
	<9264823B-EDAB-4BAD-8B6F-BC0F59ABC940@math.washington.edu>
Message-ID: <2ead2fb0810100125of88811lb3a48c0c1b534861@mail.gmail.com>

I think I'll go for the first code example I gave. It looks like the
clearest version.

On Fri, Oct 10, 2008 at 1:02 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 10, 2008, at 12:28 AM, Aaron DeVore wrote:
>
>> Tragically, I can't show my implementation of __eq__. However, I will
>> say this: I absolutely cannot change __eq__ to work with identities.
>>
>> For a relatively short list (usually 0-20 items, sometimes up to 1000)
>> would this work and still be blindingly fast? "----" indicates
>> indentation.:
>>
>> int index
>> for index from 0 <= index < len(a_list):
>> ----if a_list[index] is my_type_instance:
>> --------break
>
> This will compare on identity, if that's what you want then it should
> be quite fast.
>
>> Or would I need to do this to get correct behavior:
>> int index
>> for index from 0 <= index < len(a_list):
>> ----if id(a_list[index]) == id(my_type_instance):
>> --------break
>
> This is essentially how list.index() is implemented.
>
>>
>> 2008/10/9 Lisandro Dalcin <dalcinl at gmail.com>:
>>> On Wed, Oct 8, 2008 at 4:15 PM, Aaron DeVore
>>> <aaron.devore at gmail.com> wrote:
>>>> I have a my_type.__eq__
>>>> function but it doesn't give the correct behavior. Which behavior
>>>> does
>>>> Cython use?
>>>
>>> Could you show us the way you implemented your __eq__ method?
>>>
>>>
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Fri Oct 10 10:37:01 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Oct 2008 01:37:01 -0700
Subject: [Cython] array assignment
In-Reply-To: <20081002210306.GA15396@encolpuis>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
Message-ID: <EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>

On Oct 2, 2008, at 2:03 PM, Gabriel Gellner wrote:

> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>> is there any way to assign an cdef C array at assignment time?
>>
>> I want something like:
>>
>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
>>
>> as anything like this possible in Cython? Or do I need to do explicit
>> initialization.
>>
> And if it is explicit is there any way to not just do (can I do  
> some kind of
> block assignment?)
>
>     a[0] = 0.5
>         ...
>     a[3] = 0.1

This will now work

     cdef double *a = [0.5, 0.3, 0.1, 0.1]

Note that it is still allocated on the stack, but a pointer type is  
needed as arrays aren't lvalues.

- Robert


From dagss at student.matnat.uio.no  Fri Oct 10 10:40:59 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 10 Oct 2008 10:40:59 +0200
Subject: [Cython] array assignment
In-Reply-To: <EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
Message-ID: <48EF151B.8070005@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 2, 2008, at 2:03 PM, Gabriel Gellner wrote:
>
>   
>> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>>     
>>> is there any way to assign an cdef C array at assignment time?
>>>
>>> I want something like:
>>>
>>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
>>>
>>> as anything like this possible in Cython? Or do I need to do explicit
>>> initialization.
>>>
>>>       
>> And if it is explicit is there any way to not just do (can I do  
>> some kind of
>> block assignment?)
>>
>>     a[0] = 0.5
>>         ...
>>     a[3] = 0.1
>>     
>
> This will now work
>
>      cdef double *a = [0.5, 0.3, 0.1, 0.1]
>
> Note that it is still allocated on the stack, but a pointer type is  
> needed as arrays aren't lvalues.
It looks a bit confusing...

I mean, for a traditional Cython user it looks like a temporary Python 
list is coerced, meaning that a will point to deallocated memory.

Ignoring that issue, what even happens if you reassign a to some other 
pointer? Or is that disallowed?

The initial syntax proposed avoids these problems. I understand that 
this was probably much easier, OTOH it might be better not to have a 
feature rather than having a confusing feature...

Dag Sverre

From robertwb at math.washington.edu  Fri Oct 10 10:49:02 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Oct 2008 01:49:02 -0700
Subject: [Cython] array assignment
In-Reply-To: <48EF151B.8070005@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<48EF151B.8070005@student.matnat.uio.no>
Message-ID: <FC19DD56-E469-49AB-AA59-50098CAA06AD@math.washington.edu>

On Oct 10, 2008, at 1:40 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Oct 2, 2008, at 2:03 PM, Gabriel Gellner wrote:
>>
>>
>>> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>>>
>>>> is there any way to assign an cdef C array at assignment time?
>>>>
>>>> I want something like:
>>>>
>>>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
>>>>
>>>> as anything like this possible in Cython? Or do I need to do  
>>>> explicit
>>>> initialization.
>>>>
>>>>
>>> And if it is explicit is there any way to not just do (can I do
>>> some kind of
>>> block assignment?)
>>>
>>>     a[0] = 0.5
>>>         ...
>>>     a[3] = 0.1
>>>
>>
>> This will now work
>>
>>      cdef double *a = [0.5, 0.3, 0.1, 0.1]
>>
>> Note that it is still allocated on the stack, but a pointer type is
>> needed as arrays aren't lvalues.
> It looks a bit confusing...
>
> I mean, for a traditional Cython user it looks like a temporary Python
> list is coerced, meaning that a will point to deallocated memory.

Yes, it may look like that, but that's not what happens. The  
traditional Cython user shouldn't have to think about allocation, so  
I don't think they'll be worried...

> Ignoring that issue, what even happens if you reassign a to some other
> pointer? Or is that disallowed?

This is allowed, and works as expected.

> The initial syntax proposed avoids these problems. I understand that
> this was probably much easier, OTOH it might be better not to have a
> feature rather than having a confusing feature...

Yes, I'd like to support the original syntax too, but I think this  
will be useful. I don't think it's too confusing.

- Robert



From stefan_ml at behnel.de  Fri Oct 10 11:33:05 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Oct 2008 11:33:05 +0200 (CEST)
Subject: [Cython] array assignment
In-Reply-To: <EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
Message-ID: <62500.213.61.181.86.1223631185.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Oct 2, 2008, at 2:03 PM, Gabriel Gellner wrote:
>
>> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>>> is there any way to assign an cdef C array at assignment time?
>>>
>>> I want something like:
>>>
>>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}

The RHS would be set literal syntax in Py3, which doesn't make sense here
at all.


>>> as anything like this possible in Cython? Or do I need to do explicit
>>> initialization.
>>>
>> And if it is explicit is there any way to not just do (can I do
>> some kind of block assignment?)
>>
>>     a[0] = 0.5
>>         ...
>>     a[3] = 0.1
>
> This will now work
>
>      cdef double *a = [0.5, 0.3, 0.1, 0.1]

Here, the RHS makes sense to me. Would

      cdef double *a = (0.5, 0.3, 0.1, 0.1)

also make sense to others? Or does this contradict 1WTDI? I wonder what
people think when they see both. I could imagine using a list RHS when I
consider changing the array in my function, while a tuple makes more sense
for immutable arrays. OTOH, I'm not sure users will bother with this
detail, they might just assume an assignment by value.

I assume this feature only works for initial assignments, right? Makes
perfect sense to me in the context of Cython.

I would personally prefer making the LHS

      cdef double a[] = [0.5, 0.3, 0.1, 0.1]

but I'm not close to using this anyway, so I'm not bothered with the
details. It would be nice if both worked, since pointers and array names
are pretty much equivalent in C anyway.

Just my two Euro-cents...

Stefan


From dagss at student.matnat.uio.no  Fri Oct 10 11:43:58 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 10 Oct 2008 11:43:58 +0200
Subject: [Cython] array assignment
In-Reply-To: <FC19DD56-E469-49AB-AA59-50098CAA06AD@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>
	<FC19DD56-E469-49AB-AA59-50098CAA06AD@math.washington.edu>
Message-ID: <48EF23DE.10205@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 10, 2008, at 1:40 AM, Dag Sverre Seljebotn wrote:
>
>   
>> The initial syntax proposed avoids these problems. I understand that
>> this was probably much easier, OTOH it might be better not to have a
>> feature rather than having a confusing feature...
>>     
>
> Yes, I'd like to support the original syntax too, but I think this  
> will be useful. I don't think it's too confusing.
>   
OK I'm convinced and +1. But then I'm leaning against ever supporting 
the original syntax, as that means less language complexity to worry 
about everywhere. But that is not the issue at stake here.

(BTW I was referring to the LHS only, I support Robert in using a list 
for the RHS.)

Dag Sverre

From ggellner at uoguelph.ca  Fri Oct 10 13:45:01 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Fri, 10 Oct 2008 07:45:01 -0400
Subject: [Cython] array assignment
In-Reply-To: <EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
Message-ID: <20081010114501.GA13472@encolpuis>

>> On Thu, Oct 02, 2008 at 05:00:40PM -0400, Gabriel Gellner wrote:
>>> is there any way to assign an cdef C array at assignment time?
>>>
>>> I want something like:
>>>
>>>     cdef double a[4] = {0.5, 0.3, 0.1, 0.1}
>>>
>>> as anything like this possible in Cython? Or do I need to do explicit
>>> initialization.
>>>
>> And if it is explicit is there any way to not just do (can I do some 
>> kind of
>> block assignment?)
>>
>>     a[0] = 0.5
>>         ...
>>     a[3] = 0.1
>
> This will now work
>
>     cdef double *a = [0.5, 0.3, 0.1, 0.1]
>
> Note that it is still allocated on the stack, but a pointer type is  
> needed as arrays aren't lvalues.
>
Thank you so much! You are awesome . . .

Gabriel

From ondrej at certik.cz  Fri Oct 10 15:15:58 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 15:15:58 +0200
Subject: [Cython] docs.cython.org fix
Message-ID: <85b5c3130810100615k6df5a979q4b612548eeaceb53@mail.gmail.com>

Hi,

please pull a link fix from here:

http://www.bitbucket.org/certik/cython-doc/

e.g.

hg pull http://www.bitbucket.org/certik/cython-doc/

Ondrej

From dalcinl at gmail.com  Fri Oct 10 15:24:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 10 Oct 2008 10:24:57 -0300
Subject: [Cython] list.index(instance) by identity or __eq__?
In-Reply-To: <2ead2fb0810100028u3b1ba737p3edd72845e8c1799@mail.gmail.com>
References: <2ead2fb0810081215v34a482d8l5dd4a4d18875e481@mail.gmail.com>
	<e7ba66e40810091255n230b38eerc2bdb9b9bb5244f@mail.gmail.com>
	<2ead2fb0810100028u3b1ba737p3edd72845e8c1799@mail.gmail.com>
Message-ID: <e7ba66e40810100624w1b4d651cn8a9ebe8bf94e17ea@mail.gmail.com>

On Fri, Oct 10, 2008 at 4:28 AM, Aaron DeVore <aaron.devore at gmail.com> wrote:
> Tragically, I can't show my implementation of __eq__. However, I will
> say this: I absolutely cannot change __eq__ to work with identities.

Then have you tried to make __eq__ strict about types, and do something like:

cdef MyClass:
  def __richcmp__(self, other, int op):
     if not isinstance(self, MyClass): return NotImplemented
     if not isinstance(other, MyClass): return NotImplemented
     # and now impement the rest:
     if op == 2: # eq
         return True if condition else False
     elif op == 3: #ne
         # like above
     else:
          raise TypeError # only == and !=

>
> For a relatively short list (usually 0-20 items, sometimes up to 1000)
> would this work and still be blindingly fast? "----" indicates
> indentation.:
>
> int index
> for index from 0 <= index < len(a_list):
> ----if a_list[index] is my_type_instance:
> --------break
>
> Or would I need to do this to get correct behavior:
> int index
> for index from 0 <= index < len(a_list):
> ----if id(a_list[index]) == id(my_type_instance):
> --------break
>
> 2008/10/9 Lisandro Dalcin <dalcinl at gmail.com>:
>> On Wed, Oct 8, 2008 at 4:15 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
>>> I have a my_type.__eq__
>>> function but it doesn't give the correct behavior. Which behavior does
>>> Cython use?
>>
>> Could you show us the way you implemented your __eq__ method?
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ggellner at uoguelph.ca  Fri Oct 10 19:44:26 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Fri, 10 Oct 2008 13:44:26 -0400
Subject: [Cython] docs.cython.org fix
In-Reply-To: <85b5c3130810100615k6df5a979q4b612548eeaceb53@mail.gmail.com>
References: <85b5c3130810100615k6df5a979q4b612548eeaceb53@mail.gmail.com>
Message-ID: <20081010174426.GA14315@encolpuis>

On Fri, Oct 10, 2008 at 03:15:58PM +0200, Ondrej Certik wrote:
> Hi,
> 
> please pull a link fix from here:
> 
> http://www.bitbucket.org/certik/cython-doc/
> 
> e.g.
> 
> hg pull http://www.bitbucket.org/certik/cython-doc/
> 
Thanks I will do this tonight, finally having to learn to used mercurial . . .

Gabriel

From ondrej at certik.cz  Fri Oct 10 20:42:29 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 20:42:29 +0200
Subject: [Cython] docs.cython.org fix
In-Reply-To: <20081010174426.GA14315@encolpuis>
References: <85b5c3130810100615k6df5a979q4b612548eeaceb53@mail.gmail.com>
	<20081010174426.GA14315@encolpuis>
Message-ID: <85b5c3130810101142n783021d0nbcac9fa9f2fb598d@mail.gmail.com>

On Fri, Oct 10, 2008 at 7:44 PM, Gabriel Gellner <ggellner at uoguelph.ca> wrote:
> On Fri, Oct 10, 2008 at 03:15:58PM +0200, Ondrej Certik wrote:
>> Hi,
>>
>> please pull a link fix from here:
>>
>> http://www.bitbucket.org/certik/cython-doc/
>>
>> e.g.
>>
>> hg pull http://www.bitbucket.org/certik/cython-doc/
>>
> Thanks I will do this tonight, finally having to learn to used mercurial . . .

That's great, I am sure you'll not regret learning it. :)

Ondrej

From robertwb at math.washington.edu  Fri Oct 10 21:20:07 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Oct 2008 12:20:07 -0700
Subject: [Cython] docs.cython.org fix
In-Reply-To: <20081010174426.GA14315@encolpuis>
References: <85b5c3130810100615k6df5a979q4b612548eeaceb53@mail.gmail.com>
	<20081010174426.GA14315@encolpuis>
Message-ID: <5A4FBCE3-53C3-4142-ABE3-66231C3676C1@math.washington.edu>

On Oct 10, 2008, at 10:44 AM, Gabriel Gellner wrote:

> On Fri, Oct 10, 2008 at 03:15:58PM +0200, Ondrej Certik wrote:
>> Hi,
>>
>> please pull a link fix from here:
>>
>> http://www.bitbucket.org/certik/cython-doc/
>>
>> e.g.
>>
>> hg pull http://www.bitbucket.org/certik/cython-doc/
>>
> Thanks I will do this tonight, finally having to learn to used  
> mercurial . . .

I actually already merged it and pushed to hg.cython.org/cython-docs

- Robert


From greg.ewing at canterbury.ac.nz  Sat Oct 11 08:20:13 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 11 Oct 2008 19:20:13 +1300
Subject: [Cython] array assignment
In-Reply-To: <48EF151B.8070005@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<48EF151B.8070005@student.matnat.uio.no>
Message-ID: <48F0459D.5020200@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
> 
>>     cdef double *a = [0.5, 0.3, 0.1, 0.1]
>
>>Note that it is still allocated on the stack, but a pointer type is  
>>needed as arrays aren't lvalues.
> 
> It looks a bit confusing...

I think it looks confusing, too. Is 'a' a real pointer
here? I.e. can you assign a different pointer value to
it later?

If so, then it's not obvious where the memory for the
initial value is being allocated. A C person would
probably guess that it's either statically allocated
(by analogy with char *p = "literal") or heap allocated.
Allocating it on the stack is neither a C-like nor a
Python-like thing to do.

It would be better if it were written as

   cdef double a[] = [0.5, 0.3, 0.1, 0.1]

the same as a statically initialized array. The fact
that arrays aren't lvalues isn't relevant, because
this isn't an assignment, it's a specification of
an initial value.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat Oct 11 08:25:49 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 11 Oct 2008 19:25:49 +1300
Subject: [Cython] array assignment
In-Reply-To: <62500.213.61.181.86.1223631185.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<62500.213.61.181.86.1223631185.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <48F046ED.8020409@canterbury.ac.nz>

Stefan Behnel wrote:

> Here, the RHS makes sense to me. Would
> 
>       cdef double *a = (0.5, 0.3, 0.1, 0.1)
> 
> also make sense to others?

I think it would be reasonable to allow either list or
tuple notation to be used, but if there is only to be
one, it should probably be a list, on the grounds that
a C array is more of a homogeneous structure than a
heterogeneous one.

-- 
Greg

From robertwb at math.washington.edu  Sat Oct 11 08:54:16 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Oct 2008 23:54:16 -0700
Subject: [Cython] array assignment
In-Reply-To: <48F0459D.5020200@canterbury.ac.nz>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<48EF151B.8070005@student.matnat.uio.no>
	<48F0459D.5020200@canterbury.ac.nz>
Message-ID: <D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>


On Oct 10, 2008, at 11:20 PM, Greg Ewing wrote:

> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>
>>>     cdef double *a = [0.5, 0.3, 0.1, 0.1]
>>
>>> Note that it is still allocated on the stack, but a pointer type is
>>> needed as arrays aren't lvalues.
>>
>> It looks a bit confusing...
>
> I think it looks confusing, too. Is 'a' a real pointer
> here? I.e. can you assign a different pointer value to
> it later?

Yes.

> If so, then it's not obvious where the memory for the
> initial value is being allocated. A C person would
> probably guess that it's either statically allocated
> (by analogy with char *p = "literal") or heap allocated.
> Allocating it on the stack is neither a C-like nor a
> Python-like thing to do.

It's on the stack, just like literals. However, one can write

def foo(x, y):
     cdef double* a
     if x:
         a = [1,2,3,x]
     else:
         a = [1,2,3,y]

> It would be better if it were written as
>
>    cdef double a[] = [0.5, 0.3, 0.1, 0.1]
>
> the same as a statically initialized array. The fact
> that arrays aren't lvalues isn't relevant, because
> this isn't an assignment, it's a specification of
> an initial value.

Yes, I'd like to support this form as well, but it was a lot less  
"local" of a change and less flexible (see above). Also, I'd like to  
have a natural way to get memory-managed double*, etc. built into the  
language (using the allocate a string option). They would be very  
simple--contiguous one-dimensional arrays (anything more, just use  
NumPy), but would save people a lot of pain (especially those coming  
from the Python world who've never heard of malloc, memory leaks,  
segfaults ,etc.).

- Robert



From greg.ewing at canterbury.ac.nz  Sat Oct 11 09:24:20 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 11 Oct 2008 20:24:20 +1300
Subject: [Cython] array assignment
In-Reply-To: <D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<48EF151B.8070005@student.matnat.uio.no>
	<48F0459D.5020200@canterbury.ac.nz>
	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>
Message-ID: <48F054A4.8020300@canterbury.ac.nz>

Robert Bradshaw wrote:

> It's on the stack, just like literals.

What do you mean, "just like literals"? What literal
is ever allocated on the stack?

 > However, one can write
> 
> def foo(x, y):
>      cdef double* a
>      if x:
>          a = [1,2,3,x]
>      else:
>          a = [1,2,3,y]

Now I'm *really* confused. What the heck does that do?
Can you provide a C translation?

-- 
Greg

From robertwb at math.washington.edu  Sat Oct 11 14:06:13 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Oct 2008 05:06:13 -0700
Subject: [Cython] array assignment
In-Reply-To: <48F054A4.8020300@canterbury.ac.nz>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<48EF151B.8070005@student.matnat.uio.no>
	<48F0459D.5020200@canterbury.ac.nz>
	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>
	<48F054A4.8020300@canterbury.ac.nz>
Message-ID: <086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>

On Oct 11, 2008, at 12:24 AM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> It's on the stack, just like literals.
>
> What do you mean, "just like literals"? What literal
> is ever allocated on the stack?

Array literals are allocated on the stack.

>
>> However, one can write
>>
>> def foo(x, y):
>>      cdef double* a
>>      if x:
>>          a = [1,2,3,x]
>>      else:
>>          a = [1,2,3,y]
>
> Now I'm *really* confused. What the heck does that do?
> Can you provide a C translation?


Sure. I'm ignoring the implicit conversions that happen for clarity...

double[4] tmp1;
double[4] tmp2;
if (x) {
     tmp1[0] = 1
     tmp1[1] = 2
     tmp1[2] = 3
     tmp1[3] = x
     a = tmp1
}
else {
     tmp2[0] = 1
     tmp2[1] = 2
     tmp2[2] = 3
     tmp2[3] = y
     a = tmp2
}

- Robert


From ben.aurel at gmail.com  Sat Oct 11 14:27:04 2008
From: ben.aurel at gmail.com (Ben Aurel)
Date: Sat, 11 Oct 2008 05:27:04 -0700
Subject: [Cython] Setup Errors: "error: Python.h: No such file or directory"
Message-ID: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>

hi
I try to build Cython on my ubuntu machine according to the README:
    $ python setup.py install

But I get a bunch of errors, starting with a message that says:
    ...
    /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
error: Python.h: No such file or directory
    ...
    (the first part of the installation log is appended at the end)


The Python.h apparently resides on the following directory on my system:

    /usr/local/python/include/python2.6/Python.h


How can I solve this setup errors? I think helping the installer to
find 'Python.h' is the first step. But how can I do that?

Any ideas?
thanks
ben

///
Setup log:
///
   1 Compiling module Cython.Plex.Scanners ...
   2 running install
   3 running build
   4 running build_py
   5 running build_ext
   6 building 'Cython.Plex.Scanners' extension
   7 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c
/usr/local/src/Cython-0     .9.8.1.1/Cython/Plex/Scanners.c -o
build/temp.linux-i686-2.5/usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.o
   8 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
error: Python.h: No such file or directory
   9 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:5:26:
error: structmember.h: No such file or directory
  10 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:117:
error: expected specifier-qualifier-list before 'PyObject'
  11 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:135:
error: expected ')' before '*' token
  12 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:136:
error: expected '=', ',', ';', 'asm' or '__attribute__' before
'__pyx_PyInt_AsLon     gLong'
  13 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:137:
error: expected '=', ',', ';', 'asm' or '__attribute__' before
'__pyx_PyInt_AsUns     ignedLongLong'
  14 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:138:
error: expected ')' before '*' token
  15 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:143:
error: expected ')' before '*' token
  16 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:144:
error: expected ')' before '*' token
  17 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:145:
error: expected ')' before '*' token
  18 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:146:
error: expected ')' before '*' token
  19 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:147:
error: expected ')' before '*' token
  20 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:148:
error: expected ')' before '*' token
  21 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:149:
error: expected ')' before '*' token
  22 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:150:
error: expected ')' before '*' token
  23 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:151:
error: expected ')' before '*' token
  24 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:152:
error: expected ')' before '*' token
  25 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:153:
error: expected ')' before '*' token
  26 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:168:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token
  27 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:169:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token
  28 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:170:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token
  29 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:179:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token
  30 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:181:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token
  31 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:183:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token
  32 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:185:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token
  33 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:186:
error: expected ')' before '*' token
  34 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:188:
error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
token

From robertwb at math.washington.edu  Sat Oct 11 14:31:25 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Oct 2008 05:31:25 -0700
Subject: [Cython] Setup Errors: "error: Python.h: No such file or
	directory"
In-Reply-To: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>
References: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>
Message-ID: <9AF72710-A208-4906-83F4-B7F102A0AA45@math.washington.edu>

You need the developer version of Python (i.e. the Python header  
files) to use or install Cython.

- Robert

On Oct 11, 2008, at 5:27 AM, Ben Aurel wrote:

> hi
> I try to build Cython on my ubuntu machine according to the README:
>     $ python setup.py install
>
> But I get a bunch of errors, starting with a message that says:
>     ...
>     /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
> error: Python.h: No such file or directory
>     ...
>     (the first part of the installation log is appended at the end)
>
>
> The Python.h apparently resides on the following directory on my  
> system:
>
>     /usr/local/python/include/python2.6/Python.h
>
>
> How can I solve this setup errors? I think helping the installer to
> find 'Python.h' is the first step. But how can I do that?
>
> Any ideas?
> thanks
> ben
>
> ///
> Setup log:
> ///
>    1 Compiling module Cython.Plex.Scanners ...
>    2 running install
>    3 running build
>    4 running build_py
>    5 running build_ext
>    6 building 'Cython.Plex.Scanners' extension
>    7 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c
> /usr/local/src/Cython-0     .9.8.1.1/Cython/Plex/Scanners.c -o
> build/temp.linux-i686-2.5/usr/local/src/Cython-0.9.8.1.1/Cython/ 
> Plex/Scanners.o
>    8 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
> error: Python.h: No such file or directory
>    9 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:5:26:
> error: structmember.h: No such file or directory
>   10 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:117:
> error: expected specifier-qualifier-list before 'PyObject'
>   11 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:135:
> error: expected ')' before '*' token
>   12 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:136:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before
> '__pyx_PyInt_AsLon     gLong'
>   13 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:137:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before
> '__pyx_PyInt_AsUns     ignedLongLong'
>   14 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:138:
> error: expected ')' before '*' token
>   15 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:143:
> error: expected ')' before '*' token
>   16 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:144:
> error: expected ')' before '*' token
>   17 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:145:
> error: expected ')' before '*' token
>   18 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:146:
> error: expected ')' before '*' token
>   19 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:147:
> error: expected ')' before '*' token
>   20 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:148:
> error: expected ')' before '*' token
>   21 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:149:
> error: expected ')' before '*' token
>   22 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:150:
> error: expected ')' before '*' token
>   23 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:151:
> error: expected ')' before '*' token
>   24 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:152:
> error: expected ')' before '*' token
>   25 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:153:
> error: expected ')' before '*' token
>   26 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:168:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
>   27 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:169:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
>   28 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:170:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
>   29 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:179:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
>   30 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:181:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
>   31 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:183:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
>   32 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:185:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
>   33 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:186:
> error: expected ')' before '*' token
>   34 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:188:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
> token
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Sat Oct 11 15:55:47 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 11 Oct 2008 15:55:47 +0200
Subject: [Cython] array assignment
In-Reply-To: <086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>
	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>
Message-ID: <48F0B063.9030401@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 11, 2008, at 12:24 AM, Greg Ewing wrote:
> 
>> Robert Bradshaw wrote:
>>
>>> It's on the stack, just like literals.
>> What do you mean, "just like literals"? What literal
>> is ever allocated on the stack?
> 
> Array literals are allocated on the stack.
> 
>>> However, one can write
>>>
>>> def foo(x, y):
>>>      cdef double* a
>>>      if x:
>>>          a = [1,2,3,x]
>>>      else:
>>>          a = [1,2,3,y]
>> Now I'm *really* confused. What the heck does that do?
>> Can you provide a C translation?
> 
> 
> Sure. I'm ignoring the implicit conversions that happen for clarity...
> 
> double[4] tmp1;
> double[4] tmp2;
> if (x) {
>      tmp1[0] = 1
>      tmp1[1] = 2
>      tmp1[2] = 3
>      tmp1[3] = x
>      a = tmp1
> }
> else {
>      tmp2[0] = 1
>      tmp2[1] = 2
>      tmp2[2] = 3
>      tmp2[3] = y
>      a = tmp2
> }
> 

I can see that this has many practical usecases etc., still I'm back to 
worrying. If the following concern is valid then I'm -1 on this for the 
coming release, it should at least be postponed.

The reason I worry is that one can write (rather contrived) code like this:

cdef int* a
cdef int* b
for i in range(2):
     a = [1,1,1]
     if i == 0:
         b = a
         b[0] = 2

print b[0] # will print 1...

...which will be rather non-obvious if you do not know *exactly* how the 
allocation etc. works.

The reason this is so bad is because it seems like you are pointing "a" 
to *something else* the second time around the loop, and that "b" should 
continue pointing to the "old" array. (But there is only one array. 
Which is especially counter-intuitive given how this code would work in 
Python. Etc.)

To boil it down a bit, the problem is that

a = [1,1,1]
b = a
a = [1,1,1]

has entirely different semantics from

for i in range(2):
     a = [1,1,1]
     if i == 0: b = a

even though the first example looks exactly like just rolling out the 
latter. At least it is very un-Pythonic.

(I did not actually try the code above, but if it doesn't work then I'd 
counter that the mechanism is so magical that I cannot figure it out :-) )

What do I propose instead?
  a) That the reference-counted heap-allocated array stuff in the wiki 
is implemented first (with a list constructor). So the thing gets the 
semantics of a reference counted array, constructed again each time it 
is assigned (very similar to a Python list).

  b) That simple cases (where the pointer is never reassigned or copied, 
only written/read to/from by indexing) are then optimized to use stack 
allocation instead.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Oct 11 16:12:20 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 11 Oct 2008 16:12:20 +0200
Subject: [Cython] array assignment
In-Reply-To: <48F0B063.9030401@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>
	<48F0B063.9030401@student.matnat.uio.no>
Message-ID: <48F0B444.1050007@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:

> What do I propose instead?
>   a) That the reference-counted heap-allocated array stuff in the wiki 
> is implemented first (with a list constructor). So the thing gets the 
> semantics of a reference counted array, constructed again each time it 
> is assigned (very similar to a Python list).
> 
>   b) That simple cases (where the pointer is never reassigned or copied, 
> only written/read to/from by indexing) are then optimized to use stack 
> allocation instead.
> 

First and foremost the point now is to .

But, builing on this proposal, I propose that it is called

cdef list[int] a = [1, 2, 3]

and is not a subtype of list, but close enough:

a) If it is not optimized (by b) above) it is a full Python object and 
emulates most behaviour (but has much more efficient indexing). (If only 
efficient indexing is used then it is a C stack-allocated array.)

b) "a = x", where x is a Python list, automatically converts

c) "cdef list y = someobj" needs to have added auto-conversion from 
list[type]

This allows keeping the pointer types "clean" and without too much 
magic, and also fits well with the buffer syntax etc. Users wouldn't 
have to learn two different worlds for operating with C arrays and NumPy 
arrays.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Oct 11 16:17:35 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 11 Oct 2008 16:17:35 +0200
Subject: [Cython] array assignment
In-Reply-To: <48F0B444.1050007@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>	<48F0B063.9030401@student.matnat.uio.no>
	<48F0B444.1050007@student.matnat.uio.no>
Message-ID: <48F0B57F.7030209@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
> 
>> What do I propose instead?
>>   a) That the reference-counted heap-allocated array stuff in the wiki 
>> is implemented first (with a list constructor). So the thing gets the 
>> semantics of a reference counted array, constructed again each time it 
>> is assigned (very similar to a Python list).
>>
>>   b) That simple cases (where the pointer is never reassigned or copied, 
>> only written/read to/from by indexing) are then optimized to use stack 
>> allocation instead.
>>
> 
> First and foremost the point now is to .

Sorry! I meant: The point now was to argue for postponing the decision 
for later and disable it in the upcoming release.

I'd settle for disabling any "literal assignment" to the pointer outside 
of the declaration though, then I'd be back to "not worrying". (One 
could also disable it only inside loops, but that is too ugly and 
inconsistent.)

-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Oct 11 16:15:56 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Oct 2008 07:15:56 -0700
Subject: [Cython] array assignment
In-Reply-To: <48F0B444.1050007@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>
	<48F0B063.9030401@student.matnat.uio.no>
	<48F0B444.1050007@student.matnat.uio.no>
Message-ID: <4729C46F-D463-44B8-B423-64364998269E@math.washington.edu>

On Oct 11, 2008, at 7:12 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>
>> What do I propose instead?
>>   a) That the reference-counted heap-allocated array stuff in the  
>> wiki
>> is implemented first (with a list constructor). So the thing gets the
>> semantics of a reference counted array, constructed again each  
>> time it
>> is assigned (very similar to a Python list).
>>
>>   b) That simple cases (where the pointer is never reassigned or  
>> copied,
>> only written/read to/from by indexing) are then optimized to use  
>> stack
>> allocation instead.
>>
>
> First and foremost the point now is to .
>
> But, builing on this proposal, I propose that it is called
>
> cdef list[int] a = [1, 2, 3]
>
> and is not a subtype of list, but close enough:
>
> a) If it is not optimized (by b) above) it is a full Python object and
> emulates most behaviour (but has much more efficient indexing). (If  
> only
> efficient indexing is used then it is a C stack-allocated array.)
>
> b) "a = x", where x is a Python list, automatically converts
>
> c) "cdef list y = someobj" needs to have added auto-conversion from
> list[type]
>
> This allows keeping the pointer types "clean" and without too much
> magic, and also fits well with the buffer syntax etc. Users wouldn't
> have to learn two different worlds for operating with C arrays and  
> NumPy
> arrays.

I like this proposal, though I'd propose a new keyword (e.g. array)  
because list seems a bit odd (though it could be good, just not sure  
yet.)

cdef array[int] a = [1, 2, 3]

The question is not whether or not this is a better, more Pythonic  
way to do things, but whether or not anyone is going to hage time to  
implement this anytime soon (let alone by the next release). Yes, the  
arrays as they are now have a few warts (like using them in loops)  
but I think it's too useful to disable altogether.

- Robert


From robertwb at math.washington.edu  Sat Oct 11 16:17:35 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Oct 2008 07:17:35 -0700
Subject: [Cython] array assignment
In-Reply-To: <48F0B57F.7030209@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>	<48F0B063.9030401@student.matnat.uio.no>
	<48F0B444.1050007@student.matnat.uio.no>
	<48F0B57F.7030209@student.matnat.uio.no>
Message-ID: <7C0C6049-2616-449F-9179-588CDAE32219@math.washington.edu>

On Oct 11, 2008, at 7:17 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> Dag Sverre Seljebotn wrote:
>>
>>> What do I propose instead?
>>>   a) That the reference-counted heap-allocated array stuff in the  
>>> wiki
>>> is implemented first (with a list constructor). So the thing gets  
>>> the
>>> semantics of a reference counted array, constructed again each  
>>> time it
>>> is assigned (very similar to a Python list).
>>>
>>>   b) That simple cases (where the pointer is never reassigned or  
>>> copied,
>>> only written/read to/from by indexing) are then optimized to use  
>>> stack
>>> allocation instead.
>>>
>>
>> First and foremost the point now is to .
>
> Sorry! I meant: The point now was to argue for postponing the decision
> for later and disable it in the upcoming release.
>
> I'd settle for disabling any "literal assignment" to the pointer  
> outside
> of the declaration though, then I'd be back to "not worrying". (One
> could also disable it only inside loops, but that is too ugly and
> inconsistent.)

I'm OK with only allowing it outside of declarations (and even  
better, only to an array type), but don't know if I'll have time to  
do that anytime soon.

- Robert


From dagss at student.matnat.uio.no  Sat Oct 11 16:31:50 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 11 Oct 2008 16:31:50 +0200
Subject: [Cython] array assignment
In-Reply-To: <7C0C6049-2616-449F-9179-588CDAE32219@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>	<48F0B063.9030401@student.matnat.uio.no>	<48F0B444.1050007@student.matnat.uio.no>	<48F0B57F.7030209@student.matnat.uio.no>
	<7C0C6049-2616-449F-9179-588CDAE32219@math.washington.edu>
Message-ID: <48F0B8D6.7010502@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 11, 2008, at 7:17 AM, Dag Sverre Seljebotn wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> Dag Sverre Seljebotn wrote:
>>>
>>>> What do I propose instead?
>>>>   a) That the reference-counted heap-allocated array stuff in the  
>>>> wiki
>>>> is implemented first (with a list constructor). So the thing gets  
>>>> the
>>>> semantics of a reference counted array, constructed again each  
>>>> time it
>>>> is assigned (very similar to a Python list).
>>>>
>>>>   b) That simple cases (where the pointer is never reassigned or  
>>>> copied,
>>>> only written/read to/from by indexing) are then optimized to use  
>>>> stack
>>>> allocation instead.
>>>>
>>> First and foremost the point now is to .
>> Sorry! I meant: The point now was to argue for postponing the decision
>> for later and disable it in the upcoming release.
>>
>> I'd settle for disabling any "literal assignment" to the pointer  
>> outside
>> of the declaration though, then I'd be back to "not worrying". (One
>> could also disable it only inside loops, but that is too ugly and
>> inconsistent.)
> 
> I'm OK with only allowing it outside of declarations (and even  
> better, only to an array type), but don't know if I'll have time to  
> do that anytime soon.

I don't understand? Do you mean the opposite, i.e only allowing it 
inside of declarations?

Anyway, the (final and realistic) proposal was:

cdef int* a = [1, 2, 3] # ok!

while

cdef int* a
a = [1, 2, 3] # not ok!

At least for this release (adding features is easier than removing).

This absolutely shouldn't be more than half an hour, it is just about 
throwing up a compiler error in the right transform. If you're +1 to 
this but don't have the time I could have a look during next week. Have 
you decided on a timeframe for the release?

-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Oct 11 16:32:01 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Oct 2008 07:32:01 -0700
Subject: [Cython] array assignment
In-Reply-To: <48F0B8D6.7010502@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>	<48F0B063.9030401@student.matnat.uio.no>	<48F0B444.1050007@student.matnat.uio.no>	<48F0B57F.7030209@student.matnat.uio.no>
	<7C0C6049-2616-449F-9179-588CDAE32219@math.washington.edu>
	<48F0B8D6.7010502@student.matnat.uio.no>
Message-ID: <5EE66792-5EE4-4963-90DB-58143DC01482@math.washington.edu>

On Oct 11, 2008, at 7:31 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Oct 11, 2008, at 7:17 AM, Dag Sverre Seljebotn wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>
>>>>> What do I propose instead?
>>>>>   a) That the reference-counted heap-allocated array stuff in the
>>>>> wiki
>>>>> is implemented first (with a list constructor). So the thing gets
>>>>> the
>>>>> semantics of a reference counted array, constructed again each
>>>>> time it
>>>>> is assigned (very similar to a Python list).
>>>>>
>>>>>   b) That simple cases (where the pointer is never reassigned or
>>>>> copied,
>>>>> only written/read to/from by indexing) are then optimized to use
>>>>> stack
>>>>> allocation instead.
>>>>>
>>>> First and foremost the point now is to .
>>> Sorry! I meant: The point now was to argue for postponing the  
>>> decision
>>> for later and disable it in the upcoming release.
>>>
>>> I'd settle for disabling any "literal assignment" to the pointer
>>> outside
>>> of the declaration though, then I'd be back to "not worrying". (One
>>> could also disable it only inside loops, but that is too ugly and
>>> inconsistent.)
>>
>> I'm OK with only allowing it outside of declarations (and even
>> better, only to an array type), but don't know if I'll have time to
>> do that anytime soon.
>
> I don't understand? Do you mean the opposite, i.e only allowing it
> inside of declarations?

Sorry, jet lag... I meant to say disallowing it outside declarations.

> Anyway, the (final and realistic) proposal was:
>
> cdef int* a = [1, 2, 3] # ok!
>
> while
>
> cdef int* a
> a = [1, 2, 3] # not ok!
>
> At least for this release (adding features is easier than removing).
>
> This absolutely shouldn't be more than half an hour, it is just about
> throwing up a compiler error in the right transform.

Now that you put it that way, it makes it sound much easier--I love  
transformations :). But I actually realize how much more restrictive  
it is, as you can't do

cdef foo(int n, int* a):
     ...

foo(5, [1,2,3,4,5])

which I think could be very useful. So I would maybe say disallow  
array assignment except in declarations, but the above would still work.

> If you're +1 to
> this but don't have the time I could have a look during next week.  
> Have
> you decided on a timeframe for the release?

I was thinking in a week or two. I haven't tried compile Sage for a  
bit, but if that goes smoothly then I'd want an release candidate out  
soon, allowing a week for people to test.

- Robert


From ben.aurel at gmail.com  Sat Oct 11 21:30:29 2008
From: ben.aurel at gmail.com (Ben Aurel)
Date: Sat, 11 Oct 2008 12:30:29 -0700
Subject: [Cython] Setup Errors: "error: Python.h: No such file or
	directory"
In-Reply-To: <9AF72710-A208-4906-83F4-B7F102A0AA45@math.washington.edu>
References: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>
	<9AF72710-A208-4906-83F4-B7F102A0AA45@math.washington.edu>
Message-ID: <74a4f4670810111230g14b02e98v55423a068f1b0165@mail.gmail.com>

 hi robert
I've installed python from source*, and like I've wrote the 'python.h'
header file is there** but I don't know to get the cython installer to
find the file

* http://www.python.org/ftp/python/2.6/Python-2.6.tgz
**  /usr/local/python/include/python2.6/Python.h




On Sat, Oct 11, 2008 at 5:31 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> You need the developer version of Python (i.e. the Python header
> files) to use or install Cython.
>
> - Robert
>
> On Oct 11, 2008, at 5:27 AM, Ben Aurel wrote:
>
>> hi
>> I try to build Cython on my ubuntu machine according to the README:
>>     $ python setup.py install
>>
>> But I get a bunch of errors, starting with a message that says:
>>     ...
>>     /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
>> error: Python.h: No such file or directory
>>     ...
>>     (the first part of the installation log is appended at the end)
>>
>>
>> The Python.h apparently resides on the following directory on my
>> system:
>>
>>     /usr/local/python/include/python2.6/Python.h
>>
>>
>> How can I solve this setup errors? I think helping the installer to
>> find 'Python.h' is the first step. But how can I do that?
>>
>> Any ideas?
>> thanks
>> ben
>>
>> ///
>> Setup log:
>> ///
>>    1 Compiling module Cython.Plex.Scanners ...
>>    2 running install
>>    3 running build
>>    4 running build_py
>>    5 running build_ext
>>    6 building 'Cython.Plex.Scanners' extension
>>    7 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
>> -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c
>> /usr/local/src/Cython-0     .9.8.1.1/Cython/Plex/Scanners.c -o
>> build/temp.linux-i686-2.5/usr/local/src/Cython-0.9.8.1.1/Cython/
>> Plex/Scanners.o
>>    8 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
>> error: Python.h: No such file or directory
>>    9 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:5:26:
>> error: structmember.h: No such file or directory
>>   10 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:117:
>> error: expected specifier-qualifier-list before 'PyObject'
>>   11 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:135:
>> error: expected ')' before '*' token
>>   12 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:136:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before
>> '__pyx_PyInt_AsLon     gLong'
>>   13 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:137:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before
>> '__pyx_PyInt_AsUns     ignedLongLong'
>>   14 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:138:
>> error: expected ')' before '*' token
>>   15 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:143:
>> error: expected ')' before '*' token
>>   16 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:144:
>> error: expected ')' before '*' token
>>   17 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:145:
>> error: expected ')' before '*' token
>>   18 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:146:
>> error: expected ')' before '*' token
>>   19 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:147:
>> error: expected ')' before '*' token
>>   20 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:148:
>> error: expected ')' before '*' token
>>   21 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:149:
>> error: expected ')' before '*' token
>>   22 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:150:
>> error: expected ')' before '*' token
>>   23 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:151:
>> error: expected ')' before '*' token
>>   24 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:152:
>> error: expected ')' before '*' token
>>   25 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:153:
>> error: expected ')' before '*' token
>>   26 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:168:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>>   27 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:169:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>>   28 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:170:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>>   29 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:179:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>>   30 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:181:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>>   31 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:183:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>>   32 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:185:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>>   33 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:186:
>> error: expected ')' before '*' token
>>   34 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:188:
>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>> token
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Sat Oct 11 21:38:47 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Oct 2008 12:38:47 -0700
Subject: [Cython] Setup Errors: "error: Python.h: No such file or
	directory"
In-Reply-To: <74a4f4670810111230g14b02e98v55423a068f1b0165@mail.gmail.com>
References: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>
	<9AF72710-A208-4906-83F4-B7F102A0AA45@math.washington.edu>
	<74a4f4670810111230g14b02e98v55423a068f1b0165@mail.gmail.com>
Message-ID: <13C1828E-23D8-42D4-A205-FC126A3652E7@math.washington.edu>

Can you compile the C file

------------- foo.c ----------------
#include "Python.h"

-------------------------------------

If not, then you probably need to set some environment variables  
somewhere. I'm not expert on that (and there is a huge amount of  
variance depending on exactly what system/os you're running) but I  
believe that is most likely the problem.

- Robert


On Oct 11, 2008, at 12:30 PM, Ben Aurel wrote:

>  hi robert
> I've installed python from source*, and like I've wrote the 'python.h'
> header file is there** but I don't know to get the cython installer to
> find the file
>
> * http://www.python.org/ftp/python/2.6/Python-2.6.tgz
> **  /usr/local/python/include/python2.6/Python.h
>
>
>
>
> On Sat, Oct 11, 2008 at 5:31 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> You need the developer version of Python (i.e. the Python header
>> files) to use or install Cython.
>>
>> - Robert
>>
>> On Oct 11, 2008, at 5:27 AM, Ben Aurel wrote:
>>
>>> hi
>>> I try to build Cython on my ubuntu machine according to the README:
>>>     $ python setup.py install
>>>
>>> But I get a bunch of errors, starting with a message that says:
>>>     ...
>>>     /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
>>> error: Python.h: No such file or directory
>>>     ...
>>>     (the first part of the installation log is appended at the end)
>>>
>>>
>>> The Python.h apparently resides on the following directory on my
>>> system:
>>>
>>>     /usr/local/python/include/python2.6/Python.h
>>>
>>>
>>> How can I solve this setup errors? I think helping the installer to
>>> find 'Python.h' is the first step. But how can I do that?
>>>
>>> Any ideas?
>>> thanks
>>> ben
>>>
>>> ///
>>> Setup log:
>>> ///
>>>    1 Compiling module Cython.Plex.Scanners ...
>>>    2 running install
>>>    3 running build
>>>    4 running build_py
>>>    5 running build_ext
>>>    6 building 'Cython.Plex.Scanners' extension
>>>    7 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
>>> -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c
>>> /usr/local/src/Cython-0     .9.8.1.1/Cython/Plex/Scanners.c -o
>>> build/temp.linux-i686-2.5/usr/local/src/Cython-0.9.8.1.1/Cython/
>>> Plex/Scanners.o
>>>    8 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
>>> error: Python.h: No such file or directory
>>>    9 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:5:26:
>>> error: structmember.h: No such file or directory
>>>   10 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:117:
>>> error: expected specifier-qualifier-list before 'PyObject'
>>>   11 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:135:
>>> error: expected ')' before '*' token
>>>   12 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:136:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before
>>> '__pyx_PyInt_AsLon     gLong'
>>>   13 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:137:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before
>>> '__pyx_PyInt_AsUns     ignedLongLong'
>>>   14 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:138:
>>> error: expected ')' before '*' token
>>>   15 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:143:
>>> error: expected ')' before '*' token
>>>   16 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:144:
>>> error: expected ')' before '*' token
>>>   17 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:145:
>>> error: expected ')' before '*' token
>>>   18 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:146:
>>> error: expected ')' before '*' token
>>>   19 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:147:
>>> error: expected ')' before '*' token
>>>   20 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:148:
>>> error: expected ')' before '*' token
>>>   21 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:149:
>>> error: expected ')' before '*' token
>>>   22 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:150:
>>> error: expected ')' before '*' token
>>>   23 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:151:
>>> error: expected ')' before '*' token
>>>   24 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:152:
>>> error: expected ')' before '*' token
>>>   25 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:153:
>>> error: expected ')' before '*' token
>>>   26 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:168:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>>   27 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:169:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>>   28 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:170:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>>   29 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:179:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>>   30 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:181:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>>   31 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:183:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>>   32 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:185:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>>   33 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:186:
>>> error: expected ')' before '*' token
>>>   34 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:188:
>>> error: expected '=', ',', ';', 'asm' or '__attribute__' before '*'
>>> token
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ondrej at certik.cz  Sat Oct 11 21:55:45 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 11 Oct 2008 21:55:45 +0200
Subject: [Cython] Setup Errors: "error: Python.h: No such file or
	directory"
In-Reply-To: <13C1828E-23D8-42D4-A205-FC126A3652E7@math.washington.edu>
References: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>
	<9AF72710-A208-4906-83F4-B7F102A0AA45@math.washington.edu>
	<74a4f4670810111230g14b02e98v55423a068f1b0165@mail.gmail.com>
	<13C1828E-23D8-42D4-A205-FC126A3652E7@math.washington.edu>
Message-ID: <85b5c3130810111255n527ec52bpb3e37dbf0e22a09c@mail.gmail.com>

On Sat, Oct 11, 2008 at 9:38 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Can you compile the C file
>
> ------------- foo.c ----------------
> #include "Python.h"
>
> -------------------------------------
>
> If not, then you probably need to set some environment variables
> somewhere. I'm not expert on that (and there is a huge amount of
> variance depending on exactly what system/os you're running) but I
> believe that is most likely the problem.

Compile this by hand:

  7 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c
/usr/local/src/Cython-0     .9.8.1.1/Cython/Plex/Scanners.c -o
build/temp.linux-i686-2.5/usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.o
  8 /usr/local/src/Cython-0.9.8.1.1/Cython/Plex/Scanners.c:4:20:
error: Python.h: No such file or directory


And you need to get it working, e.g. change the -I line to point to
your dir with your python2.6 installation. After you get this working,
you need to figure out how to fix the cython's setup.py file.

Ondrej

From michael.abshoff at googlemail.com  Sun Oct 12 00:53:41 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sat, 11 Oct 2008 15:53:41 -0700
Subject: [Cython] Setup Errors: "error: Python.h: No such file
	or	directory"
In-Reply-To: <85b5c3130810111255n527ec52bpb3e37dbf0e22a09c@mail.gmail.com>
References: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>	<9AF72710-A208-4906-83F4-B7F102A0AA45@math.washington.edu>	<74a4f4670810111230g14b02e98v55423a068f1b0165@mail.gmail.com>	<13C1828E-23D8-42D4-A205-FC126A3652E7@math.washington.edu>
	<85b5c3130810111255n527ec52bpb3e37dbf0e22a09c@mail.gmail.com>
Message-ID: <48F12E75.7000501@gmail.com>

Ondrej Certik wrote:
> On Sat, Oct 11, 2008 at 9:38 PM, Robert Bradshaw

<SNIP>

> And you need to get it working, e.g. change the -I line to point to
> your dir with your python2.6 installation. After you get this working,
> you need to figure out how to fix the cython's setup.py file.

One other very likely source of trouble can be when you have more than 
one python binary installed and the wrong on in $PATH. Distutils is 
usually clever enough to find the right one.

> Ondrej

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From greg.ewing at canterbury.ac.nz  Sun Oct 12 02:10:22 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 12 Oct 2008 13:10:22 +1300
Subject: [Cython] array assignment
In-Reply-To: <4729C46F-D463-44B8-B423-64364998269E@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<48EF151B.8070005@student.matnat.uio.no>
	<48F0459D.5020200@canterbury.ac.nz>
	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>
	<48F054A4.8020300@canterbury.ac.nz>
	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>
	<48F0B063.9030401@student.matnat.uio.no>
	<48F0B444.1050007@student.matnat.uio.no>
	<4729C46F-D463-44B8-B423-64364998269E@math.washington.edu>
Message-ID: <48F1406E.9050703@canterbury.ac.nz>

Robert Bradshaw wrote:
> Yes, the  
> arrays as they are now have a few warts (like using them in loops)  
> but I think it's too useful to disable altogether.

You could get the same effect in a much less misleading
way using

    cdef int data[] = [1, 2, 3, 4]
    cdef int *a = data

Now it's perfectly clear that a is pointing to something
stack-allocated. EIBTI.

-- 
Greg

From robertwb at math.washington.edu  Sun Oct 12 03:01:29 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Oct 2008 18:01:29 -0700
Subject: [Cython] array assignment
In-Reply-To: <48F1406E.9050703@canterbury.ac.nz>
References: <20081002210040.GA15290@encolpuis>
	<20081002210306.GA15396@encolpuis>
	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>
	<48EF151B.8070005@student.matnat.uio.no>
	<48F0459D.5020200@canterbury.ac.nz>
	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>
	<48F054A4.8020300@canterbury.ac.nz>
	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>
	<48F0B063.9030401@student.matnat.uio.no>
	<48F0B444.1050007@student.matnat.uio.no>
	<4729C46F-D463-44B8-B423-64364998269E@math.washington.edu>
	<48F1406E.9050703@canterbury.ac.nz>
Message-ID: <600C5066-96D9-49B9-A34F-4C819420F8BC@math.washington.edu>

On Oct 11, 2008, at 5:10 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>> Yes, the
>> arrays as they are now have a few warts (like using them in loops)
>> but I think it's too useful to disable altogether.
>
> You could get the same effect in a much less misleading
> way using
>
>     cdef int data[] = [1, 2, 3, 4]
>     cdef int *a = data
>
> Now it's perfectly clear that a is pointing to something
> stack-allocated. EIBTI.

Yes, but the whole point was to be more concise (specifically, better  
than setting the elements one by one, but even this is somewhat  
redundant). The reason I did it the way I did was because

cdef foo = x

gets translated to

cdef foo
foo = x

and so disallowing assignments in one place, but not in another, and  
allowing assignments to array types just once, etc. was a whole lot  
more complicated that what I wanted to deal with at the time.  
Eventually we will get there.

- Robert


From greg.ewing at canterbury.ac.nz  Sun Oct 12 06:43:19 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 12 Oct 2008 17:43:19 +1300
Subject: [Cython] Setup Errors: "error: Python.h: No such file
	or	directory"
In-Reply-To: <74a4f4670810111230g14b02e98v55423a068f1b0165@mail.gmail.com>
References: <74a4f4670810110527l20565dbfneb6948a3d1a77cc7@mail.gmail.com>
	<9AF72710-A208-4906-83F4-B7F102A0AA45@math.washington.edu>
	<74a4f4670810111230g14b02e98v55423a068f1b0165@mail.gmail.com>
Message-ID: <48F18067.3010203@canterbury.ac.nz>

Ben Aurel wrote:

> **  /usr/local/python/include/python2.6/Python.h

But

>>>   7 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
>>>-Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c

Which looks to me like you're getting a system-installed
Python 2.5 when you run 'python', instead of the Python 2.6
that you installed yourself.

Try this instead:

   python2.6 setup.py install

Also, if you want 'python' to refer to python2.6, you may
need to adjust a symlink somewhere or adjust your PATH.

-- 
Greg

From ondrej at certik.cz  Sun Oct 12 18:54:44 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 12 Oct 2008 18:54:44 +0200
Subject: [Cython] docs.cython.org fix
In-Reply-To: <5A4FBCE3-53C3-4142-ABE3-66231C3676C1@math.washington.edu>
References: <85b5c3130810100615k6df5a979q4b612548eeaceb53@mail.gmail.com>
	<20081010174426.GA14315@encolpuis>
	<5A4FBCE3-53C3-4142-ABE3-66231C3676C1@math.washington.edu>
Message-ID: <85b5c3130810120954wb327ff4rb494c1d8cc22b050@mail.gmail.com>

On Fri, Oct 10, 2008 at 9:20 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 10, 2008, at 10:44 AM, Gabriel Gellner wrote:
>
>> On Fri, Oct 10, 2008 at 03:15:58PM +0200, Ondrej Certik wrote:
>>> Hi,
>>>
>>> please pull a link fix from here:
>>>
>>> http://www.bitbucket.org/certik/cython-doc/
>>>
>>> e.g.
>>>
>>> hg pull http://www.bitbucket.org/certik/cython-doc/
>>>
>> Thanks I will do this tonight, finally having to learn to used
>> mercurial . . .
>
> I actually already merged it and pushed to hg.cython.org/cython-docs

Thanks. Could someone please also update the doc.cython.org?

Ondrej

From animator333 at yahoo.com  Mon Oct 13 10:19:39 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Mon, 13 Oct 2008 13:49:39 +0530 (IST)
Subject: [Cython] pubsub module and cython
Message-ID: <66663.23468.qm@web94913.mail.in2.yahoo.com>

Hi,

I did a simple test of using pubsub module and cython. It seems cython is doing something wrong and not able to properly handle listener.
The code is zipped here.

1. Run "test.py" and everything is working fine.
2. you can compile "test.py" using "setup.bat" on windows. Compiled "test.pyd" is also available.
3. Run "demo.py". Here you have an error.

Regards

Prashant



      Unlimited freedom, unlimited storage. Get it now, on http://help.yahoo.com/l/in/yahoo/mail/yahoomail/tools/tools-08.html/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081013/028baa28/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pubsub_test.zip
Type: application/zip
Size: 17203 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081013/028baa28/attachment.zip 

From stefan_ml at behnel.de  Mon Oct 13 11:53:35 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 13 Oct 2008 11:53:35 +0200 (CEST)
Subject: [Cython] pubsub module and cython
In-Reply-To: <66663.23468.qm@web94913.mail.in2.yahoo.com>
References: <66663.23468.qm@web94913.mail.in2.yahoo.com>
Message-ID: <60556.213.61.181.86.1223891615.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Hi,

it's usually a good idea to provide some more more information than "doing
something wrong" and "an error" in your problem report, especially some
hints on what the error output is. Otherwise, you require everyone to
unpack your test code, compile it and run it, although the error
description might already give some (or even enough) hints on what goes
wrong and who might be able to help you.

Stefan

Prashant Saxena wrote:
> I did a simple test of using pubsub module and cython. It seems cython is
> doing something wrong and not able to properly handle listener.
> The code is zipped here.
>
> 1. Run "test.py" and everything is working fine.
> 2. you can compile "test.py" using "setup.bat" on windows. Compiled
> "test.pyd" is also available.
> 3. Run "demo.py". Here you have an error.
>
> Regards
>
> Prashant


From animator333 at yahoo.com  Mon Oct 13 13:11:13 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Mon, 13 Oct 2008 16:41:13 +0530 (IST)
Subject: [Cython] pubsub module and cython
Message-ID: <665018.6990.qm@web94905.mail.in2.yahoo.com>

Ok, Here are the details

Os win 2k
python 2.5.2
cython  0.9.8.1.1

1. I am using pure python code to convert .py into .c and compile using mingw32.( no cdef stuff)
2. test.py is working fine until unless I am converting into c extension.
3. When using test.pyd, I am getting following error:

Traceback (most recent call last):
  File "demo.py", line 1, in <module>
    import test
  File "test.py", line 33, in test (test..c:831)
    app.MainLoop()
  File "test.py", line 18, in test.MyButtons..__init__ (test.c:413)
    
  File "C:\Python25\Lib\site-packages\pubsubcore\publisher.py", line 86, in subscribe
    topicName, desc=noDesc, usingCallable=listener)
  File "C:\Python25\Lib\site-packages\pubsubcore\topics.py", line 423, in _newTopicFromTemplate_
    allArgsDocs, required = topicArgsFromCallable(usingCallable)
  File "C:\Python25\Lib\site-packages\pubsubcore\topics.py", line 618, in topicArgsFromCallable
    argsInfo = getListenerArgs(_callable)
  File "C:\Python25\lib\site-packages\pubsubcore\callables.py", line 158, in getArgs
    (args, va, vkwa, defaultVals) = getargspec(func)
  File "C:\Python25\lib\inspect.py", line 742, in getargspec
    raise TypeError('arg is not a Python function')
TypeError: arg is not a Python function


In second last line of error it seems the listener(self.OnPubSub) at line no. 17 in test.py:
Publisher.subscribe(self.OnPubSub, "topic")
is changed into something else which "inspect.py" is not able to recognized as a function.
Although subscription method is almost same as wxPython events binding, which is getting compiled and giving no problems.

I gave a try to pydispatch module and wx.lib.pubsub module to do the same thing but none of them are working when test.py is compiled into .pyd.
All of the modules are showing same type of error at line no.17 where we are subscribing our listener. Still I am not able to fugure out where actually the problem is, is it in the modules or is it with cython?

Hope this helps.

Prashant




----- Original Message ----
From: Stefan Behnel <stefan_ml at behnel.de>
To: cython-dev at codespeak.net
Sent: Monday, 13 October, 2008 3:23:35 PM
Subject: Re: [Cython] pubsub module and cython

Hi,

it's usually a good idea to provide some more more information than "doing
something wrong" and "an error" in your problem report, especially some
hints on what the error output is. Otherwise, you require everyone to
unpack your test code, compile it and run it, although the error
description might already give some (or even enough) hints on what goes
wrong and who might be able to help you.

Stefan

Prashant Saxena wrote:
> I did a simple test of using pubsub module and cython. It seems cython is
> doing something wrong and not able to properly handle listener.
> The code is zipped here.
>
> 1. Run "test.py" and everything is working fine.
> 2. you can compile "test.py" using "setup.bat" on windows. Compiled
> "test.pyd" is also available.
> 3. Run "demo.py". Here you have an error.
>
> Regards
>
> Prashant

_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev



      Check out the all-new face of Yahoo! India. Go to http://in.yahoo.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081013/095d78b0/attachment.htm 

From robertwb at math.washington.edu  Mon Oct 13 13:36:20 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 13 Oct 2008 04:36:20 -0700
Subject: [Cython] pubsub module and cython
In-Reply-To: <665018.6990.qm@web94905.mail.in2.yahoo.com>
References: <665018.6990.qm@web94905.mail.in2.yahoo.com>
Message-ID: <F396AD8F-0BA6-439B-A98B-FB770652E4E6@math.washington.edu>

That is correct, arg is not a Python function, it is a compiled  
function. It looks like the listener interface can only accept pure  
Python functions.

You could look at trying to fix wx, but I would recommend writing  
your test.py as an ordinary Python module (it doesn't look like that  
part of the code will benefit from Cython much) and wrapping the  
speed-critical/library interface parts in Cython.

- Robert

On Oct 13, 2008, at 4:11 AM, Prashant Saxena wrote:

> Ok, Here are the details
>
> Os win 2k
> python 2.5.2
> cython 0.9.8.1.1
>
> 1. I am using pure python code to convert .py into .c and compile  
> using mingw32.( no cdef stuff)
> 2. test.py is working fine until unless I am converting into c  
> extension.
> 3. When using test.pyd, I am getting following error:
>
> Traceback (most recent call last):
>   File "demo.py", line 1, in <module>
>     import test
>   File "test.py", line 33, in test (test.c:831)
>     app.MainLoop()
>   File "test.py", line 18, in test.MyButtons.__init__ (test.c:413)
>
>   File "C:\Python25\Lib\site-packages\pubsubcore\publisher.py",  
> line 86, in subscribe
>     topicName, desc=noDesc, usingCallable=listener)
>   File "C:\Python25\Lib\site-packages\pubsubcore\topics.py", line  
> 423, in _newTopicFromTemplate_
>     allArgsDocs, required = topicArgsFromCallable(usingCallable)
>   File "C:\Python25\Lib\site-packages\pubsubcore\topics.py", line  
> 618, in topicArgsFromCallable
>     argsInfo = getListenerArgs(_callable)
>   File "C:\Python25\lib\site-packages\pubsubcore\callables.py",  
> line 158, in getArgs
>     (args, va, vkwa, defaultVals) = getargspec(func)
>   File "C:\Python25\lib\inspect.py", line 742, in getargspec
>     raise TypeError('arg is not a Python function')
> TypeError: arg is not a Python function
>
>
> In second last line of error it seems the listener(self.OnPubSub)  
> at line no. 17 in test.py:
> Publisher.subscribe(self.OnPubSub, "topic")
> is changed into something else which "inspect.py" is not able to  
> recognized as a function.
> Although subscription method is almost same as wxPython events  
> binding, which is getting compiled and giving no problems.
>
> I gave a try to pydispatch module and wx.lib.pubsub module to do  
> the same thing but none of them are working when test.py is  
> compiled into .pyd.
> All of the modules are showing same type of error at line no.17  
> where we are subscribing our listener. Still I am not able to  
> fugure out where actually the problem is, is it in the modules or  
> is it with cython?
>
> Hope this helps.
>
> Prashant
>
>
> ----- Original Message ----
> From: Stefan Behnel <stefan_ml at behnel.de>
> To: cython-dev at codespeak.net
> Sent: Monday, 13 October, 2008 3:23:35 PM
> Subject: Re: [Cython] pubsub module and cython
>
> Hi,
>
> it's usually a good idea to provide some more more information than  
> "doing
> something wrong" and "an error" in your problem report, especially  
> some
> hints on what the error output is. Otherwise, you require everyone to
> unpack your test code, compile it and run it, although the error
> description might already give some (or even enough) hints on what  
> goes
> wrong and who might be able to help you.
>
> Stefan
>
> Prashant Saxena wrote:
> > I did a simple test of using pubsub module and cython. It seems  
> cython is
> > doing something wrong and not able to properly handle listener.
> > The code is zipped here.
> >
> > 1. Run "test.py" and everything is working fine.
> > 2. you can compile "test.py" using "setup.bat" on windows. Compiled
> > "test.pyd" is also available.
> > 3. Run "demo.py". Here you have an error.
> >
> > Regards
> >
> > Prashant
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
> Add more friends to your messenger and enjoy! Invite them now.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ondrej at certik.cz  Mon Oct 13 15:10:07 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 13 Oct 2008 15:10:07 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
Message-ID: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>

Hi,

what is now the canonical way to convert between (python) numpy array and

int *a

or

double *a

?

I looked at the new petsc4py and it uses this method to convert from numpy to C:

cdef inline ndarray iarray_s(object ob, PetscInt* size, PetscScalar** data):
    ob = PyArray_FROM_OTF(ob, NPY_PETSC_SCALAR, NPY_IN_ARRAY)
    cdef ndarray ary = <ndarray> ob
    if size!=NULL: size[0] = <PetscInt>     PyArray_SIZE(ary)
    if data!=NULL: data[0] = <PetscScalar*> PyArray_DATA(ary)
    return ary

The function PyArray_FROM_OTF() is defined in /usr/include/numpy/ndarrayobject.h

And this method to convert from C to numpy:

cdef inline ndarray array_s(PetscInt size, const_PetscScalar* data):
    cdef npy_intp s = <npy_intp> size
    cdef ndarray ary = PyArray_EMPTY(1, &s, NPY_PETSC_SCALAR, 0)
    if data != NULL: memcpy(ary.cdata, data, size*sizeof(PetscScalar))
    return ary


Is this the best way to do things now? When I investigated this half a
year ago, the way to do this was this:

http://wiki.cython.org/WrappingNumpy

however Robert wrote there that now the way is this:

http://wiki.cython.org/tutorials/numpy

unfortunately at the latter page it is only explained how to access
indices of numpy arrays using the buffer interface, but not how to
convert them back and forth to C.

Maybe the canonical methods could be shipped with either numpy or
cython? I can provide a patch, once I learn how to do things properly.

Thanks,
Ondrej

From animator333 at yahoo.com  Mon Oct 13 15:29:30 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Mon, 13 Oct 2008 18:59:30 +0530 (IST)
Subject: [Cython] pubsub module and cython
Message-ID: <22416.81801.qm@web94908.mail.in2.yahoo.com>

I would prefer to do something with publisher module. The example that was included with this topic is for to show the problem only.
I am using publisher in couple of files where I am about to change the code to get the maximum gain using cython(cdef, loops and other stuff). 
Now I have to find a solution for this problem first, later I'll jump for conversion.

Could you please suggest me some hacks that would make this thing work If I would do some changes in pubsub modules or anything?

Prashant



----- Original Message ----
From: Robert Bradshaw <robertwb at math.washington.edu>
To: cython-dev at codespeak.net
Sent: Monday, 13 October, 2008 5:06:20 PM
Subject: Re: [Cython] pubsub module and cython

That is correct, arg is not a Python function, it is a compiled  
function. It looks like the listener interface can only accept pure  
Python functions.

You could look at trying to fix wx, but I would recommend writing  
your test.py as an ordinary Python module (it doesn't look like that  
part of the code will benefit from Cython much) and wrapping the  
speed-critical/library interface parts in Cython.

- Robert

On Oct 13, 2008, at 4:11 AM, Prashant Saxena wrote:

> Ok, Here are the details
>
> Os win 2k
> python 2.5.2
> cython 0.9.8.1.1
>
> 1. I am using pure python code to convert .py into .c and compile  
> using mingw32.( no cdef stuff)
> 2. test.py is working fine until unless I am converting into c  
> extension.
> 3. When using test.pyd, I am getting following error:
>
> Traceback (most recent call last):
>   File "demo.py", line 1, in <module>
>     import test
>   File "test.py", line 33, in test (test.c:831)
>     app.MainLoop()
>   File "test.py", line 18, in test.MyButtons.__init__ (test.c:413)
>
>   File "C:\Python25\Lib\site-packages\pubsubcore\publisher.py",  
> line 86, in subscribe
>     topicName, desc=noDesc, usingCallable=listener)
>   File "C:\Python25\Lib\site-packages\pubsubcore\topics.py", line  
> 423, in _newTopicFromTemplate_
>     allArgsDocs, required = topicArgsFromCallable(usingCallable)
>   File "C:\Python25\Lib\site-packages\pubsubcore\topics.py", line  
> 618, in topicArgsFromCallable
>     argsInfo = getListenerArgs(_callable)
>   File "C:\Python25\lib\site-packages\pubsubcore\callables.py",  
> line 158, in getArgs
>     (args, va, vkwa, defaultVals) = getargspec(func)
>   File "C:\Python25\lib\inspect.py", line 742, in getargspec
>     raise TypeError('arg is not a Python function')
> TypeError: arg is not a Python function
>
>
> In second last line of error it seems the listener(self.OnPubSub)  
> at line no. 17 in test.py:
> Publisher.subscribe(self.OnPubSub, "topic")
> is changed into something else which "inspect.py" is not able to  
> recognized as a function.
> Although subscription method is almost same as wxPython events  
> binding, which is getting compiled and giving no problems.
>
> I gave a try to pydispatch module and wx.lib.pubsub module to do  
> the same thing but none of them are working when test.py is  
> compiled into .pyd.
> All of the modules are showing same type of error at line no.17  
> where we are subscribing our listener. Still I am not able to  
> fugure out where actually the problem is, is it in the modules or  
> is it with cython?
>
> Hope this helps.
>
> Prashant
>
>
> ----- Original Message ----
> From: Stefan Behnel <stefan_ml at behnel.de>
> To: cython-dev at codespeak.net
> Sent: Monday, 13 October, 2008 3:23:35 PM
> Subject: Re: [Cython] pubsub module and cython
>
> Hi,
>
> it's usually a good idea to provide some more more information than  
> "doing
> something wrong" and "an error" in your problem report, especially  
> some
> hints on what the error output is. Otherwise, you require everyone to
> unpack your test code, compile it and run it, although the error
> description might already give some (or even enough) hints on what  
> goes
> wrong and who might be able to help you.
>
> Stefan
>
> Prashant Saxena wrote:
> > I did a simple test of using pubsub module and cython. It seems  
> cython is
> > doing something wrong and not able to properly handle listener.
> > The code is zipped here.
> >
> > 1. Run "test.py" and everything is working fine.
> > 2. you can compile "test.py" using "setup.bat" on windows. Compiled
> > "test.pyd" is also available.
> > 3. Run "demo.py". Here you have an error.
> >
> > Regards
> >
> > Prashant
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
> Add more friends to your messenger and enjoy! Invite them now.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev



      Be the first one to try the new Messenger 9 Beta! Go to http://in.messenger.yahoo.com/win/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081013/ededcf66/attachment-0001.htm 

From dagss at student.matnat.uio.no  Mon Oct 13 15:38:08 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 13 Oct 2008 15:38:08 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>
Message-ID: <48F34F40.7040705@student.matnat.uio.no>

Ondrej Certik wrote:
> Hi,
>
> what is now the canonical way to convert between (python) numpy array and
>
> int *a
>
> or
>
> double *a
>
> ?
>   
Yes, this should probably be fixed/be made more available.

The simplest thing is something like the (untested) code below. It is 
certainly possible to do some C calls instead of Python calls here (like 
petsc4py does); however I tink this definitely belong to the 80% of the 
code that only takes 20% of the time; Cython seems to be more about 
optimizing the other part that tends to get repeated :-)

import numpy as np
arr = some generic numpy array
assert arr.dtype == np.float64 # or whatever
cdef ndarray contarr
if not arr.flags['C_CONTIGUOUS']:
    # Array is not contiguous, need to make contiguous copy
    contarr = arr.copy(order='C')
else:
    contarr = arr
# Get data pointer. Important that contarr is cdef-ed ndarray
cdef npy.float64_t* ptr = <npy.float64_t*>contarr.data
call_c_function(ptr)
# If the C function modifies the data, and the array was not contiguous,
# then the data of contarr must be copied into arr, using standard NumPy 
calls:
arr[:] = contarr[:]

Comment: The only "shady" part here is "contarr.data", which accesses 
implementation details of NumPy arrays. I'm guessing that this will 
never change, but I once planned to make a generic cython function 
"cython.buffer.bufptr" which would return the same pointer but it could 
be acquired through the buffer API.

Note that the code above does not make use of the buffer API, as it is 
written to work regardless of number of dimensions.

Dag Sverre

From ondrej at certik.cz  Mon Oct 13 17:32:51 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 13 Oct 2008 17:32:51 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <48F34F40.7040705@student.matnat.uio.no>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>
	<48F34F40.7040705@student.matnat.uio.no>
Message-ID: <85b5c3130810130832k25d410ddl88a228d378b78e0@mail.gmail.com>

On Mon, Oct 13, 2008 at 3:38 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
>> Hi,
>>
>> what is now the canonical way to convert between (python) numpy array and
>>
>> int *a
>>
>> or
>>
>> double *a
>>
>> ?
>>
> Yes, this should probably be fixed/be made more available.
>
> The simplest thing is something like the (untested) code below. It is
> certainly possible to do some C calls instead of Python calls here (like
> petsc4py does); however I tink this definitely belong to the 80% of the
> code that only takes 20% of the time; Cython seems to be more about
> optimizing the other part that tends to get repeated :-)
>
> import numpy as np
> arr = some generic numpy array
> assert arr.dtype == np.float64 # or whatever
> cdef ndarray contarr
> if not arr.flags['C_CONTIGUOUS']:
>    # Array is not contiguous, need to make contiguous copy
>    contarr = arr.copy(order='C')
> else:
>    contarr = arr
> # Get data pointer. Important that contarr is cdef-ed ndarray
> cdef npy.float64_t* ptr = <npy.float64_t*>contarr.data
> call_c_function(ptr)
> # If the C function modifies the data, and the array was not contiguous,
> # then the data of contarr must be copied into arr, using standard NumPy
> calls:
> arr[:] = contarr[:]
>
> Comment: The only "shady" part here is "contarr.data", which accesses
> implementation details of NumPy arrays. I'm guessing that this will
> never change, but I once planned to make a generic cython function
> "cython.buffer.bufptr" which would return the same pointer but it could
> be acquired through the buffer API.
>
> Note that the code above does not make use of the buffer API, as it is
> written to work regardless of number of dimensions.

Currently I use these methods:

cdef inline ndarray array_d(int size, double *data):
    #cdef ndarray ary2 = PyArray_ZEROS(1, &size, 12, 0)
    cdef ndarray ary = zeros(size, dtype=float64)
    if data != NULL: memcpy(ary.data, data, size*sizeof(double))
    return ary

cdef inline int iarray_d(ndarray a, int *size, double **data) except -1:
    if a.dtype != float64:
        raise TypeError("The array must have the dtype=float64.")
    if size!=NULL: size[0] = a.dimensions[0]
    if data!=NULL: data[0] = <double *> (a.data)



If I uncomment the commented out line (i.e. to call numpy C/API
directly instead of through Python), I'll get a segfault. I haven't
figured out yet why. But nevertheless, the current version works.

Ondrej

From robert.kern at gmail.com  Mon Oct 13 18:02:59 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 13 Oct 2008 11:02:59 -0500
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <48F34F40.7040705@student.matnat.uio.no>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>
	<48F34F40.7040705@student.matnat.uio.no>
Message-ID: <gcvrfj$ibi$1@ger.gmane.org>

Dag Sverre Seljebotn wrote:

> Comment: The only "shady" part here is "contarr.data", which accesses 
> implementation details of NumPy arrays. I'm guessing that this will 
> never change, but I once planned to make a generic cython function 
> "cython.buffer.bufptr" which would return the same pointer but it could 
> be acquired through the buffer API.

The canonical way to avoid accessing .data directly is to use the PyArray_DATA() 
macro (returns a void*). Similarly, dimensions and strides can be accessed via 
PyArray_DIMS() and PyArray_STRIDES().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From dagss at student.matnat.uio.no  Mon Oct 13 18:11:20 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 13 Oct 2008 18:11:20 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <85b5c3130810130832k25d410ddl88a228d378b78e0@mail.gmail.com>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>	<48F34F40.7040705@student.matnat.uio.no>
	<85b5c3130810130832k25d410ddl88a228d378b78e0@mail.gmail.com>
Message-ID: <48F37328.3030205@student.matnat.uio.no>

Ondrej Certik wrote:
> On Mon, Oct 13, 2008 at 3:38 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Ondrej Certik wrote:
>>> Hi,
>>>
>>> what is now the canonical way to convert between (python) numpy array and
>>>
>>> int *a
>>>
>>> or
>>>
>>> double *a
>>>
>>> ?
>>>
>> Yes, this should probably be fixed/be made more available.
>>
>> The simplest thing is something like the (untested) code below. It is
>> certainly possible to do some C calls instead of Python calls here (like
>> petsc4py does); however I tink this definitely belong to the 80% of the
>> code that only takes 20% of the time; Cython seems to be more about
>> optimizing the other part that tends to get repeated :-)
>>
>> import numpy as np
>> arr = some generic numpy array
>> assert arr.dtype == np.float64 # or whatever
>> cdef ndarray contarr
>> if not arr.flags['C_CONTIGUOUS']:
>>    # Array is not contiguous, need to make contiguous copy
>>    contarr = arr.copy(order='C')
>> else:
>>    contarr = arr
>> # Get data pointer. Important that contarr is cdef-ed ndarray
>> cdef npy.float64_t* ptr = <npy.float64_t*>contarr.data
>> call_c_function(ptr)
>> # If the C function modifies the data, and the array was not contiguous,
>> # then the data of contarr must be copied into arr, using standard NumPy
>> calls:
>> arr[:] = contarr[:]
>>
>> Comment: The only "shady" part here is "contarr.data", which accesses
>> implementation details of NumPy arrays. I'm guessing that this will
>> never change, but I once planned to make a generic cython function
>> "cython.buffer.bufptr" which would return the same pointer but it could
>> be acquired through the buffer API.
>>
>> Note that the code above does not make use of the buffer API, as it is
>> written to work regardless of number of dimensions.
> 
> Currently I use these methods:
> 
> cdef inline ndarray array_d(int size, double *data):
>     #cdef ndarray ary2 = PyArray_ZEROS(1, &size, 12, 0)
>     cdef ndarray ary = zeros(size, dtype=float64)
>     if data != NULL: memcpy(ary.data, data, size*sizeof(double))
>     return ary
> 
> cdef inline int iarray_d(ndarray a, int *size, double **data) except -1:
>     if a.dtype != float64:
>         raise TypeError("The array must have the dtype=float64.")
>     if size!=NULL: size[0] = a.dimensions[0]
>     if data!=NULL: data[0] = <double *> (a.data)
> 
> 
> 
> If I uncomment the commented out line (i.e. to call numpy C/API
> directly instead of through Python), I'll get a segfault. I haven't
> figured out yet why. But nevertheless, the current version works.

The issue you are running into is likely that the size passed to 
PyArray_ZEROS must be an "npy_intp" rather than "int", which makes it 
64-bit on 64-bit platforms. As you pass it by its address this is a 
rather crucial point (otherwise you end up trying to allocate a rather 
random number of gigabytes).

This actually came up earlier on this mailing list. It would be great if 
you could make a documentation patch against ndarrayobject.h and submit 
it on the NumPy mailing list since it obviously seems like a common 
problem. Patches for numpy.pxd (including PyArray_ZEROS etc. declared 
with the right pointer types :-) ) are also welcome, currently numpy.pxd 
is geared towards high-level Python use and excludes a lot of C-level 
functions (because of lack of time/laziness only, throwing them in there 
doesn't hurt).

HOWEVER, note that your iarray_d will only work if the array that is 
passed in is contiguous. So you should either a) insert an assertion of 
a.flags['C_CONTIGUOUS'] at the beginning, or b) fix it so that it works 
with all arrays (as by my example).

Any non-trivial NumPy usage is likely to end up with non-contiguous 
arrays, so if the "a" argument could come from the end-user somehow then 
choking on non-contiguous arrays is not going to work well.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Oct 13 18:17:10 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 13 Oct 2008 18:17:10 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <gcvrfj$ibi$1@ger.gmane.org>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>	<48F34F40.7040705@student.matnat.uio.no>
	<gcvrfj$ibi$1@ger.gmane.org>
Message-ID: <48F37486.40304@student.matnat.uio.no>

Robert Kern wrote:
> Dag Sverre Seljebotn wrote:
> 
>> Comment: The only "shady" part here is "contarr.data", which accesses 
>> implementation details of NumPy arrays. I'm guessing that this will 
>> never change, but I once planned to make a generic cython function 
>> "cython.buffer.bufptr" which would return the same pointer but it could 
>> be acquired through the buffer API.
> 
> The canonical way to avoid accessing .data directly is to use the PyArray_DATA() 
> macro (returns a void*). Similarly, dimensions and strides can be accessed via 
> PyArray_DIMS() and PyArray_STRIDES().

Good point. (And I already do this for the buffer access in numpy.pxd, 
so the buffer access stuff is forwards-compatible with any change here.)

The reason I want cython.buffer.bufptr BTW is to have a common way of 
accessing similar pointers for all buffer exporters, not only NumPy arrays.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Oct 13 18:33:06 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 13 Oct 2008 18:33:06 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <48F37328.3030205@student.matnat.uio.no>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>	<48F34F40.7040705@student.matnat.uio.no>	<85b5c3130810130832k25d410ddl88a228d378b78e0@mail.gmail.com>
	<48F37328.3030205@student.matnat.uio.no>
Message-ID: <48F37842.4070306@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Ondrej Certik wrote:
>>
>> cdef inline ndarray array_d(int size, double *data):
>>     #cdef ndarray ary2 = PyArray_ZEROS(1, &size, 12, 0)
>>     cdef ndarray ary = zeros(size, dtype=float64)
>>     if data != NULL: memcpy(ary.data, data, size*sizeof(double))
>>     return ary
>>
>> cdef inline int iarray_d(ndarray a, int *size, double **data) except -1:
>>     if a.dtype != float64:
>>         raise TypeError("The array must have the dtype=float64.")
>>     if size!=NULL: size[0] = a.dimensions[0]
>>     if data!=NULL: data[0] = <double *> (a.data)
>>

> HOWEVER, note that your iarray_d will only work if the array that is 
> passed in is contiguous. So you should either a) insert an assertion of 
> a.flags['C_CONTIGUOUS'] at the beginning, or b) fix it so that it works 
> with all arrays (as by my example).
> 
> Any non-trivial NumPy usage is likely to end up with non-contiguous 
> arrays, so if the "a" argument could come from the end-user somehow then 
> choking on non-contiguous arrays is not going to work well.
> 
> 

Note that this is non-trivial since any copy made must also not be 
deallocated, so that iarray_d would have to return a reference which 
must be held on to for the duration of using the buffer data.

If you need to accept ndarrays coming from the user, a better idiom is 
likely to take a callback function which will be called with the buffer 
as an argument, or similar -- returning the buffer through a double** 
(or np.float64_t**) is rather error-prone when it comes to reference 
counting.

-- 
Dag Sverre

From stefan_ml at behnel.de  Mon Oct 13 19:21:49 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 13 Oct 2008 19:21:49 +0200
Subject: [Cython] pubsub module and cython
In-Reply-To: <22416.81801.qm@web94908.mail.in2.yahoo.com>
References: <22416.81801.qm@web94908.mail.in2.yahoo.com>
Message-ID: <48F383AD.5070204@behnel.de>

Hi,

Prashant Saxena wrote:
> I would prefer to do something with publisher module. The example that was included with this topic is for to show the problem only.
> I am using publisher in couple of files where I am about to change the code to get the maximum gain using cython(cdef, loops and other stuff). 
> Now I have to find a solution for this problem first, later I'll jump for conversion.
> 
> Could you please suggest me some hacks that would make this thing work If I would do some changes in pubsub modules or anything?

As Robert suggested: try to find the parts of your code that are performance
critical and use Cython for them. And find the parts that require Python code
and implement them in plain Python. Cython allows you to mix both pretty
freely, so this will allow you to fine-tune the point of language switch as
you see fit.

Stefan

From animator333 at yahoo.com  Mon Oct 13 21:03:06 2008
From: animator333 at yahoo.com (Prashant Saxena)
Date: Tue, 14 Oct 2008 00:33:06 +0530 (IST)
Subject: [Cython] pubsub module and cython
Message-ID: <943288.16772.qm@web94902.mail.in2.yahoo.com>

After investigating a lot in pubsub module, I have written my own custom dispatcher(50 lines :-)) and it's working like a charm with cython.

cython rocks!!!

Thanks

Prashant



----- Original Message ----
From: Stefan Behnel <stefan_ml at behnel.de>
To: cython-dev at codespeak.net
Sent: Monday, 13 October, 2008 10:51:49 PM
Subject: Re: [Cython] pubsub module and cython

Hi,

Prashant Saxena wrote:
> I would prefer to do something with publisher module. The example that was included with this topic is for to show the problem only.
> I am using publisher in couple of files where I am about to change the code to get the maximum gain using cython(cdef, loops and other stuff). 
> Now I have to find a solution for this problem first, later I'll jump for conversion.
> 
> Could you please suggest me some hacks that would make this thing work If I would do some changes in pubsub modules or anything?

As Robert suggested: try to find the parts of your code that are performance
critical and use Cython for them. And find the parts that require Python code
and implement them in plain Python. Cython allows you to mix both pretty
freely, so this will allow you to fine-tune the point of language switch as
you see fit.

Stefan
_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev



      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081014/f01bcfdd/attachment-0001.htm 

From dagss at student.matnat.uio.no  Mon Oct 13 22:44:40 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 13 Oct 2008 22:44:40 +0200
Subject: [Cython] Literal struct syntax
Message-ID: <48F3B338.4060308@student.matnat.uio.no>

I'm sorry to bring this up as people would rather code and discuss other 
stuff, but, well, once things are released they tend to stick. So I'll 
just raise the concern and you do what you want to do...

My concern is about the dict struct construction syntax -- from 
"struct_conversion.pyx":

cdef Point p = {color: color, x: x, y: y}

That is, without string keys, but rather some magic keys. This feels 
very odd to a Python eye -- especially as the coercion the other way 
(from struct to dict) seems to have normal string keys.

For more brevity without typing all the " there's always the 
Point(color=...), syntax, and I feel being consistent with keeping it a 
normal dict is important because a later feature might be

cdef Point p
d = {"color": color, "x" : x, "y" : y}
p = d

and then these "literal dict vs. Python dict" kind of thing might be 
even more confusing.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Oct 14 09:12:40 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 14 Oct 2008 09:12:40 +0200
Subject: [Cython] array assignment
In-Reply-To: <5EE66792-5EE4-4963-90DB-58143DC01482@math.washington.edu>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>	<48F0B063.9030401@student.matnat.uio.no>	<48F0B444.1050007@student.matnat.uio.no>	<48F0B57F.7030209@student.matnat.uio.no>	<7C0C6049-2616-449F-9179-588CDAE32219@math.washington.edu>	<48F0B8D6.7010502@student.matnat.uio.no>
	<5EE66792-5EE4-4963-90DB-58143DC01482@math.washington.edu>
Message-ID: <48F44668.5040703@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 11, 2008, at 7:31 AM, Dag Sverre Seljebotn wrote:
>> This absolutely shouldn't be more than half an hour, it is just about
>> throwing up a compiler error in the right transform.
> 
> Now that you put it that way, it makes it sound much easier--I love  
> transformations :). But I actually realize how much more restrictive  
> it is, as you can't do
> 
> cdef foo(int n, int* a):
>      ...
> 
> foo(5, [1,2,3,4,5])
> 
> which I think could be very useful. So I would maybe say disallow  
> array assignment except in declarations, but the above would still work.

As you pointed out in another post this was more hairy than I expected 
because of cdef transformation.

But anyway a patch is up doing this now. It is not tied to declaration, 
but what can Cython prove is the first assignment, but that works just 
as well (or better) for this purpose! (And is the same currently).

-- 
Dag Sverre

From robertwb at math.washington.edu  Tue Oct 14 10:12:22 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Oct 2008 01:12:22 -0700
Subject: [Cython] array assignment
In-Reply-To: <48F44668.5040703@student.matnat.uio.no>
References: <20081002210040.GA15290@encolpuis>	<20081002210306.GA15396@encolpuis>	<EAF2E87A-FF8C-41D5-B25D-33BC46E381C9@math.washington.edu>	<48EF151B.8070005@student.matnat.uio.no>	<48F0459D.5020200@canterbury.ac.nz>	<D32E68BE-56A6-4EF1-927D-D5505EAD87EC@math.washington.edu>	<48F054A4.8020300@canterbury.ac.nz>	<086685DA-2F6A-4B59-AC1B-7A40F3CEC8E9@math.washington.edu>	<48F0B063.9030401@student.matnat.uio.no>	<48F0B444.1050007@student.matnat.uio.no>	<48F0B57F.7030209@student.matnat.uio.no>	<7C0C6049-2616-449F-9179-588CDAE32219@math.washington.edu>	<48F0B8D6.7010502@student.matnat.uio.no>
	<5EE66792-5EE4-4963-90DB-58143DC01482@math.washington.edu>
	<48F44668.5040703@student.matnat.uio.no>
Message-ID: <087D00CB-AFF0-4736-A0DC-2D9836CA428E@math.washington.edu>

On Oct 14, 2008, at 12:12 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Oct 11, 2008, at 7:31 AM, Dag Sverre Seljebotn wrote:
>>> This absolutely shouldn't be more than half an hour, it is just  
>>> about
>>> throwing up a compiler error in the right transform.
>>
>> Now that you put it that way, it makes it sound much easier--I love
>> transformations :). But I actually realize how much more restrictive
>> it is, as you can't do
>>
>> cdef foo(int n, int* a):
>>      ...
>>
>> foo(5, [1,2,3,4,5])
>>
>> which I think could be very useful. So I would maybe say disallow
>> array assignment except in declarations, but the above would still  
>> work.
>
> As you pointed out in another post this was more hairy than I expected
> because of cdef transformation.
>
> But anyway a patch is up doing this now. It is not tied to  
> declaration,
> but what can Cython prove is the first assignment, but that works just
> as well (or better) for this purpose! (And is the same currently).

Thanks. A bit more restrictive, but certainly harder for the user to  
invoke confusing behavior.

- Robert



From robertwb at math.washington.edu  Tue Oct 14 10:20:07 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Oct 2008 01:20:07 -0700
Subject: [Cython] Literal struct syntax
In-Reply-To: <48F3B338.4060308@student.matnat.uio.no>
References: <48F3B338.4060308@student.matnat.uio.no>
Message-ID: <E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>

On Oct 13, 2008, at 1:44 PM, Dag Sverre Seljebotn wrote:

> I'm sorry to bring this up as people would rather code and discuss  
> other
> stuff, but, well, once things are released they tend to stick. So I'll
> just raise the concern and you do what you want to do...

No, it is good to bring this up now. In fact, I would encourage  
everyone to look at
http://hg.cython.org/cython-devel/file/5a7f83a639f7/tests/run/ 
struct_conversion.pyx and say what they think.

> My concern is about the dict struct construction syntax -- from
> "struct_conversion.pyx":
>
> cdef Point p = {color: color, x: x, y: y}
>
> That is, without string keys, but rather some magic keys. This feels
> very odd to a Python eye -- especially as the coercion the other way
> (from struct to dict) seems to have normal string keys.

As a (weak) counter argument, I found myself forgetting the quotes  
when testing, and it's not much more magic than the magic string kwds  
arguments keys.

> For more brevity without typing all the " there's always the
> Point(color=...), syntax, and I feel being consistent with keeping  
> it a
> normal dict is important because a later feature might be
>
> cdef Point p
> d = {"color": color, "x" : x, "y" : y}
> p = d

Yes, that's certainly in the plans.

> and then these "literal dict vs. Python dict" kind of thing might be
> even more confusing.

Point taken. I'm not too attached to this syntax--lets see if anyone  
else has feedback.

- Robert
  

From ondrej at certik.cz  Tue Oct 14 14:16:32 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 14 Oct 2008 14:16:32 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <48F37842.4070306@student.matnat.uio.no>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>
	<48F34F40.7040705@student.matnat.uio.no>
	<85b5c3130810130832k25d410ddl88a228d378b78e0@mail.gmail.com>
	<48F37328.3030205@student.matnat.uio.no>
	<48F37842.4070306@student.matnat.uio.no>
Message-ID: <85b5c3130810140516h18756481t2ac8a9f56cdb6c84@mail.gmail.com>

On Mon, Oct 13, 2008 at 6:33 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> Ondrej Certik wrote:
>>>
>>> cdef inline ndarray array_d(int size, double *data):
>>>     #cdef ndarray ary2 = PyArray_ZEROS(1, &size, 12, 0)
>>>     cdef ndarray ary = zeros(size, dtype=float64)
>>>     if data != NULL: memcpy(ary.data, data, size*sizeof(double))
>>>     return ary
>>>
>>> cdef inline int iarray_d(ndarray a, int *size, double **data) except -1:
>>>     if a.dtype != float64:
>>>         raise TypeError("The array must have the dtype=float64.")
>>>     if size!=NULL: size[0] = a.dimensions[0]
>>>     if data!=NULL: data[0] = <double *> (a.data)
>>>
>
>> HOWEVER, note that your iarray_d will only work if the array that is
>> passed in is contiguous. So you should either a) insert an assertion of
>> a.flags['C_CONTIGUOUS'] at the beginning, or b) fix it so that it works
>> with all arrays (as by my example).
>>
>> Any non-trivial NumPy usage is likely to end up with non-contiguous
>> arrays, so if the "a" argument could come from the end-user somehow then
>> choking on non-contiguous arrays is not going to work well.
>>
>>
>
> Note that this is non-trivial since any copy made must also not be
> deallocated, so that iarray_d would have to return a reference which
> must be held on to for the duration of using the buffer data.
>
> If you need to accept ndarrays coming from the user, a better idiom is
> likely to take a callback function which will be called with the buffer
> as an argument, or similar -- returning the buffer through a double**
> (or np.float64_t**) is rather error-prone when it comes to reference
> counting.

Yeah, I thought pretty much along the same lines.

Basically, I am fine if the function raises an exception on
noncontiguous arrays and/or copy the array automatically, so that's
not an issue.

The big problem is with the reference counting, i.e. who is
reposnsible for freeing the double** memory. Basically, the memory is
automatically freeed when the numpy array goes out of scope, right? So
basically, the rule is: either do something with the double** array
immediatelly (in C/C++) on the fly, or make a copy of it if you need
it longer. Another option would be to Py_IncRef/DecRef the numpy array
to make sure the C array is not recycled.


Ondrej

From dagss at student.matnat.uio.no  Tue Oct 14 14:28:56 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 14 Oct 2008 14:28:56 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <85b5c3130810140516h18756481t2ac8a9f56cdb6c84@mail.gmail.com>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>	<48F34F40.7040705@student.matnat.uio.no>	<85b5c3130810130832k25d410ddl88a228d378b78e0@mail.gmail.com>	<48F37328.3030205@student.matnat.uio.no>	<48F37842.4070306@student.matnat.uio.no>
	<85b5c3130810140516h18756481t2ac8a9f56cdb6c84@mail.gmail.com>
Message-ID: <48F49088.30709@student.matnat.uio.no>

Ondrej Certik wrote:
> On Mon, Oct 13, 2008 at 6:33 PM, Dag Sverre Seljebotn
>
>   
>> Note that this is non-trivial since any copy made must also not be
>> deallocated, so that iarray_d would have to return a reference which
>> must be held on to for the duration of using the buffer data.
>>
>> If you need to accept ndarrays coming from the user, a better idiom is
>> likely to take a callback function which will be called with the buffer
>> as an argument, or similar -- returning the buffer through a double**
>> (or np.float64_t**) is rather error-prone when it comes to reference
>> counting.
>>     
>
> Yeah, I thought pretty much along the same lines.
>
> Basically, I am fine if the function raises an exception on
> noncontiguous arrays and/or copy the array automatically, so that's
> not an issue.
>
> The big problem is with the reference counting, i.e. who is
> reposnsible for freeing the double** memory. Basically, the memory is
> automatically freeed when the numpy array goes out of scope, right? So
> basically, the rule is: either do something with the double** array
> immediatelly (in C/C++) on the fly, or make a copy of it if you need
> it longer. Another option would be to Py_IncRef/DecRef the numpy array
> to make sure the C array is not recycled.
>   
Pretty much. To hold a reference, simply assign it to a Cython "object" 
variable, don't bother manually reference counting it.

I think the general rule should be "keep the data in the numpy array for 
as long as possible". I must admit that I cannot imagine many situations 
where it would be good to store the double* for future reference rather 
than the entire Python array object. They're just pointers, it's not 
like it is "heavier" to store the ndarray :-)

I suppose one case is if you have a stateful C library like this:

clib.set_data(mydata)
clib.perform(...)
clib.release_data()

and you want to create a 1:1 wrapper in Cython. Yes, in that case you 
need to mirror the state info on the Cython-side and store a reference 
to the ndarray in the wrapper as well as feeding the pointer to C.

Dag Sverre

From ondrej at certik.cz  Tue Oct 14 14:44:51 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 14 Oct 2008 14:44:51 +0200
Subject: [Cython] canonical way of converting numpy arrays to C arrays
In-Reply-To: <48F49088.30709@student.matnat.uio.no>
References: <85b5c3130810130610j57bcdcd3r6adaa30b513b27cb@mail.gmail.com>
	<48F34F40.7040705@student.matnat.uio.no>
	<85b5c3130810130832k25d410ddl88a228d378b78e0@mail.gmail.com>
	<48F37328.3030205@student.matnat.uio.no>
	<48F37842.4070306@student.matnat.uio.no>
	<85b5c3130810140516h18756481t2ac8a9f56cdb6c84@mail.gmail.com>
	<48F49088.30709@student.matnat.uio.no>
Message-ID: <85b5c3130810140544h3888c2b3oe788d6b21e0d4fc0@mail.gmail.com>

On Tue, Oct 14, 2008 at 2:28 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
>> On Mon, Oct 13, 2008 at 6:33 PM, Dag Sverre Seljebotn
>>
>>
>>> Note that this is non-trivial since any copy made must also not be
>>> deallocated, so that iarray_d would have to return a reference which
>>> must be held on to for the duration of using the buffer data.
>>>
>>> If you need to accept ndarrays coming from the user, a better idiom is
>>> likely to take a callback function which will be called with the buffer
>>> as an argument, or similar -- returning the buffer through a double**
>>> (or np.float64_t**) is rather error-prone when it comes to reference
>>> counting.
>>>
>>
>> Yeah, I thought pretty much along the same lines.
>>
>> Basically, I am fine if the function raises an exception on
>> noncontiguous arrays and/or copy the array automatically, so that's
>> not an issue.
>>
>> The big problem is with the reference counting, i.e. who is
>> reposnsible for freeing the double** memory. Basically, the memory is
>> automatically freeed when the numpy array goes out of scope, right? So
>> basically, the rule is: either do something with the double** array
>> immediatelly (in C/C++) on the fly, or make a copy of it if you need
>> it longer. Another option would be to Py_IncRef/DecRef the numpy array
>> to make sure the C array is not recycled.
>>
> Pretty much. To hold a reference, simply assign it to a Cython "object"
> variable, don't bother manually reference counting it.

Nice trick, didn't occur to me.

>
> I think the general rule should be "keep the data in the numpy array for
> as long as possible". I must admit that I cannot imagine many situations
> where it would be good to store the double* for future reference rather
> than the entire Python array object. They're just pointers, it's not
> like it is "heavier" to store the ndarray :-)
>
> I suppose one case is if you have a stateful C library like this:
>
> clib.set_data(mydata)
> clib.perform(...)
> clib.release_data()
>
> and you want to create a 1:1 wrapper in Cython. Yes, in that case you
> need to mirror the state info on the Cython-side and store a reference
> to the ndarray in the wrapper as well as feeding the pointer to C.

This is exactly my case. I am writting a C++ code that operates over
double* arrays and I have 1:1 Cython wrapper for the classes. But a
nice trick is to hold the reference to the input array in the Cython
object in the Cython class --- that could do the job.

Ondrej

From aaron.devore at gmail.com  Tue Oct 14 22:06:03 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Tue, 14 Oct 2008 13:06:03 -0700
Subject: [Cython] Literal struct syntax
In-Reply-To: <E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
Message-ID: <2ead2fb0810141306v1ef83ff7ra71490ab5af1ea02@mail.gmail.com>

Time for my two cents. When I first saw the {color:color, x:x} syntax
it did seem somewhat jarring. After I thought about it a bit more it
makes more sense. Its advantages over a "string": value syntax:

1. It gives a clear visual distinction between a dict and a struct.
2. dict values are accessed via keys that are often strings (d["k"] ->
v) whereas structs are always referred to like attributes (p.name)
3. Having what appears to be string keys doesn't reflect that the keys
are not, in fact, strings.
4. Some other languages use the same syntax (e.g. objects in JavaScript)
5. It's clear that users are restricted to Python's identifier syntax.

The disadvantage:
1. In {color:color, x:x, y:y} the 'keys' look like identifiers.

I haven't read back through the archives to catch this entire
discussion but I have a suggestion. How about having a character
before the braces to visually indicate that it is a struct? Maybe
something like s{color: color}. That would give a clear visual cue,
not introduce a significant syntax change, and be similar to 1L, u"",
and r"".

Alternatively, the struct member names could be surrounded with a
character that isn't a quote. Just off the top of my head:
- A period/dot like {.color:color}. Pros: matches to the idea of
attribute/member access, fewer characters to type. Cons: Periods
sometimes aren't very visible at the beginning of a word, still looks
somewhat like a variable.
- {|color|: color}. Pros: distinct syntax, Cons: The { and | visually
blend together. The | looks too much like an I in some fonts. Doesn't
match up to any preexisting syntax concepts.
- {/color/: color}. Pros/Cons: Same as pipes, looks like regexp syntax
in Perl and other languages.
- {`color`: color}: Pros: Vaguely looks like a string literal, but
still is distinct. Very visible in email. Cons: Introduces new syntax.



-Aaron

From greg.ewing at canterbury.ac.nz  Wed Oct 15 01:09:50 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 15 Oct 2008 12:09:50 +1300
Subject: [Cython] Literal struct syntax
In-Reply-To: <E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
Message-ID: <48F526BE.7040107@canterbury.ac.nz>

Robert Bradshaw wrote:

> On Oct 13, 2008, at 1:44 PM, Dag Sverre Seljebotn wrote:
> 
>>cdef Point p = {color: color, x: x, y: y}
>>
>>That is, without string keys, but rather some magic keys.
> 
> As a (weak) counter argument, I found myself forgetting the quotes
> when testing, and it's not much more magic than the magic string kwds  
> arguments keys.

But it's inconsistent, because it suggests evaluating
an expression to get the key, as in a dict constructor.

A compromise might be

   cdef Point p = {color = color, x = x, y = y}

-- 
Greg

From stefan_ml at behnel.de  Wed Oct 15 08:44:05 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Oct 2008 08:44:05 +0200 (CEST)
Subject: [Cython] Literal struct syntax
In-Reply-To: <48F526BE.7040107@canterbury.ac.nz>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<48F526BE.7040107@canterbury.ac.nz>
Message-ID: <44461.213.61.181.86.1224053045.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Greg Ewing wrote:
> A compromise might be
>
>    cdef Point p = {color = color, x = x, y = y}

Yes, keywords were my first thought as well. They heavily reduce the
magic. I just wasn't sure if {} is the syntax of choice here, or if
(colour=5, x=y) would match my intuition better. It matches the new
constructor after all.

Another thought I had was that this should somehow match named tuples (new
in Py2.6), but there isn't a literal syntax for them, so this won't help
much. That makes keywords the next best choice, IMHO.

http://code.activestate.com/recipes/500261/
http://docs.python.org/library/collections.html#collections.namedtuple

Stefan


From stefan_ml at behnel.de  Wed Oct 15 09:03:24 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Oct 2008 09:03:24 +0200 (CEST)
Subject: [Cython] Literal struct syntax
In-Reply-To: <E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
Message-ID: <32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Oct 13, 2008, at 1:44 PM, Dag Sverre Seljebotn wrote:
>> For more brevity without typing all the " there's always the
>> Point(color=...), syntax

Which, as I only noticed now, actually matches the way named tuples are used.


>> and I feel being consistent with keeping
>> it a normal dict is important because a later feature might be
>>
>> cdef Point p
>> d = {"color": color, "x" : x, "y" : y}
>> p = d
>
> Yes, that's certainly in the plans.

That certainly matches the opposite conversion way, which, as I see in the
test, you already implemented.

Given this goal, I wonder why the literals are needed in the first place.
Currently, you can just do this:

    p = Point(color=color, x=y)

and once dict unpacking is implemented, you can also do

    p = dict(color=color, x=y)

or

    p = {"color" : color, "x" : y}

where the dict creation could even be optimised away, at least in the
trivial case.

Adding another literal syntax to this, only for the sake of removing one
little name before the parentheses, seems unnecessary to me.

Apart from this little detail, I'm very happy with the general concepts
and I really like what you implemented here. This totally makes sense in
the context of Cython.

Stefan


From dalcinl at gmail.com  Wed Oct 15 18:24:47 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 15 Oct 2008 13:24:47 -0300
Subject: [Cython] Dag, please review this patch
Message-ID: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>

Dag, I was working on a fix for tiket #90
<http://trac.cython.org/cython_trac/ticket/90>, but your latest commit
forced me to rework my patch. Could you please review the attached
patch (you also have it in trac!!)?

 All the tests pass with this patch, but perhaps I'm missing
something... Thanks in advance.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fixnoneleaks.diff
Type: text/x-patch
Size: 854 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081015/41cfb709/attachment.bin 

From dagss at student.matnat.uio.no  Wed Oct 15 19:04:05 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Oct 2008 19:04:05 +0200
Subject: [Cython] Dag, please review this patch
In-Reply-To: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
Message-ID: <48F62285.80309@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Dag, I was working on a fix for tiket #90
> <http://trac.cython.org/cython_trac/ticket/90>, but your latest commit
> forced me to rework my patch. Could you please review the attached
> patch (you also have it in trac!!)?
> 
>  All the tests pass with this patch, but perhaps I'm missing
> something... Thanks in advance.

See <http://trac.cython.org/cython_trac/ticket/90>

"""
The patch should work -- however it seems a bit fragile.

I'm attaching an alternative version, which rather modifies how the 
initalization happens. This should also be very slightly faster.
"""

I don't dare to apply my patch though, a bit unsure if it could break 
something, so somebody please review my patch :-)


-- 
Dag Sverre

From dalcinl at gmail.com  Wed Oct 15 19:14:11 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 15 Oct 2008 14:14:11 -0300
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F62285.80309@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<48F62285.80309@student.matnat.uio.no>
Message-ID: <e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>

Well, what I do not like about your patch is that the cdef module
globals are not initialized to 'None' in the module init function. And
this is a bit unsafe, Cython almost always assumes that PyObect*
pointers are not-NULL. I can imagine some code breaking with your
patch.

On Wed, Oct 15, 2008 at 2:04 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> Dag, I was working on a fix for tiket #90
>> <http://trac.cython.org/cython_trac/ticket/90>, but your latest commit
>> forced me to rework my patch. Could you please review the attached
>> patch (you also have it in trac!!)?
>>
>>  All the tests pass with this patch, but perhaps I'm missing
>> something... Thanks in advance.
>
> See <http://trac.cython.org/cython_trac/ticket/90>
>
> """
> The patch should work -- however it seems a bit fragile.
>
> I'm attaching an alternative version, which rather modifies how the
> initalization happens. This should also be very slightly faster.
> """
>
> I don't dare to apply my patch though, a bit unsure if it could break
> something, so somebody please review my patch :-)
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Oct 15 19:56:18 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Oct 2008 19:56:18 +0200
Subject: [Cython] Dag, please review this patch
In-Reply-To: <e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>	<48F62285.80309@student.matnat.uio.no>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
Message-ID: <48F62EC2.6080907@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Well, what I do not like about your patch is that the cdef module
> globals are not initialized to 'None' in the module init function. And
> this is a bit unsafe, Cython almost always assumes that PyObect*
> pointers are not-NULL. I can imagine some code breaking with your
> patch.
> 

Indeed, it is unsafe. Very much so, I just created a crashing testcase :-)

gtg, brb


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Oct 15 20:13:25 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Oct 2008 20:13:25 +0200
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F62EC2.6080907@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>	<48F62285.80309@student.matnat.uio.no>	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
Message-ID: <48F632C5.6080706@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:
>> Well, what I do not like about your patch is that the cdef module
>> globals are not initialized to 'None' in the module init function. And
>> this is a bit unsafe, Cython almost always assumes that PyObect*
>> pointers are not-NULL. I can imagine some code breaking with your
>> patch.
>>
> 
> Indeed, it is unsafe. Very much so, I just created a crashing testcase :-)

OK, my last attempt is up as a new patch on the ticket:

<http://trac.cython.org/cython_trac/ticket/90>

It works in this way: Assignment nodes which are created from cdef 
statements in the module scope doesn't get the "first" attribute set to 
True (as it is not guaranteed that the cdef statement is indeed the 
first assignment, indeed they are implicitly always set to None first).

I now realize what made my worry about your patch: It works ok, but it 
corrects something which was set wrongly in the first place. This should 
fix the root cause instead.

-- 
Dag Sverre

From dalcinl at gmail.com  Wed Oct 15 20:33:51 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 15 Oct 2008 15:33:51 -0300
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F632C5.6080706@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<48F62285.80309@student.matnat.uio.no>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
	<48F632C5.6080706@student.matnat.uio.no>
Message-ID: <e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>

This look nicer. However, there is still something that does not work.
Try to compile the following:

cdef object foo1
cdef object foo2 = []

and then you will see that the declaration of  the (unused) 'foo1' is
not ever emited in the module declaration part. Some optimization
should be playing games. Or perhaps 'foo1' should be completely
ignored and never declared/initialized/cleaned-up?

On Wed, Oct 15, 2008 at 3:13 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> Lisandro Dalcin wrote:
>>> Well, what I do not like about your patch is that the cdef module
>>> globals are not initialized to 'None' in the module init function. And
>>> this is a bit unsafe, Cython almost always assumes that PyObect*
>>> pointers are not-NULL. I can imagine some code breaking with your
>>> patch.
>>>
>>
>> Indeed, it is unsafe. Very much so, I just created a crashing testcase :-)
>
> OK, my last attempt is up as a new patch on the ticket:
>
> <http://trac.cython.org/cython_trac/ticket/90>
>
> It works in this way: Assignment nodes which are created from cdef
> statements in the module scope doesn't get the "first" attribute set to
> True (as it is not guaranteed that the cdef statement is indeed the
> first assignment, indeed they are implicitly always set to None first).
>
> I now realize what made my worry about your patch: It works ok, but it
> corrects something which was set wrongly in the first place. This should
> fix the root cause instead.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Oct 15 22:05:14 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Oct 2008 22:05:14 +0200
Subject: [Cython] Dag, please review this patch
In-Reply-To: <e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>	<48F62285.80309@student.matnat.uio.no>	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>	<48F62EC2.6080907@student.matnat.uio.no>	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
Message-ID: <48F64CFA.8030100@student.matnat.uio.no>

Lisandro Dalcin wrote:
> This look nicer. However, there is still something that does not work.
> Try to compile the following:
> 
> cdef object foo1
> cdef object foo2 = []
> 
> and then you will see that the declaration of  the (unused) 'foo1' is
> not ever emited in the module declaration part. Some optimization
> should be playing games. Or perhaps 'foo1' should be completely
> ignored and never declared/initialized/cleaned-up?
> 

I believe this is an unrelated and long-standing bug?

Anyway, the following patch should fix it (unless we want the opposite 
behaviour).

Will you commit it together with your testcase? (I.e. try to print 
"foo1" from a doctest, which is only in a string and so will trigger 
this case, and it should be "None"...)

-- 
Dag Sverre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: always_declare_module_objects2.diff
Type: text/x-diff
Size: 621 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081015/d9ae32b0/attachment-0001.bin 

From dalcinl at gmail.com  Wed Oct 15 22:10:22 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 15 Oct 2008 17:10:22 -0300
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F64CFA.8030100@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<48F62285.80309@student.matnat.uio.no>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
	<48F64CFA.8030100@student.matnat.uio.no>
Message-ID: <e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>

On Wed, Oct 15, 2008 at 5:05 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>>
>> This look nicer. However, there is still something that does not work.
>> Try to compile the following:
>>
>> cdef object foo1
>> cdef object foo2 = []
>>
>> and then you will see that the declaration of  the (unused) 'foo1' is
>> not ever emited in the module declaration part. Some optimization
>> should be playing games. Or perhaps 'foo1' should be completely
>> ignored and never declared/initialized/cleaned-up?
>>
>
> I believe this is an unrelated and long-standing bug?
> Anyway, the following patch should fix it (unless we want the opposite
> behaviour).

OK, I'll give a try, but perhaps tomorrow.

>
> Will you commit it together with your testcase? (I.e. try to print "foo1"
> from a doctest, which is only in a string and so will trigger this case, and
> it should be "None"...)

Well, I'll try to make a good test case for all this. But the 'print
foo1' will not work, as 'foo1' is not exported in the module dict, as
it is cdef stuff (print foo2 in a doctest docstring will not work
either).


> --
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Oct 15 22:31:10 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 15 Oct 2008 22:31:10 +0200
Subject: [Cython] Dag, please review this patch
In-Reply-To: <e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>	<48F62285.80309@student.matnat.uio.no>	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>	<48F62EC2.6080907@student.matnat.uio.no>	<48F632C5.6080706@student.matnat.uio.no>	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>	<48F64CFA.8030100@student.matnat.uio.no>
	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
Message-ID: <48F6530E.5080401@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Wed, Oct 15, 2008 at 5:05 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Lisandro Dalcin wrote:
>>> This look nicer. However, there is still something that does not work.
>>> Try to compile the following:
>>>
>>> cdef object foo1
>>> cdef object foo2 = []
>>>
>>> and then you will see that the declaration of  the (unused) 'foo1' is
>>> not ever emited in the module declaration part. Some optimization
>>> should be playing games. Or perhaps 'foo1' should be completely
>>> ignored and never declared/initialized/cleaned-up?
>>>
>> I believe this is an unrelated and long-standing bug?
>> Anyway, the following patch should fix it (unless we want the opposite
>> behaviour).
> 
> OK, I'll give a try, but perhaps tomorrow.
> 
>> Will you commit it together with your testcase? (I.e. try to print "foo1"
>> from a doctest, which is only in a string and so will trigger this case, and
>> it should be "None"...)
> 
> Well, I'll try to make a good test case for all this. But the 'print
> foo1' will not work, as 'foo1' is not exported in the module dict, as
> it is cdef stuff (print foo2 in a doctest docstring will not work
> either).
> 

Ah right. Well my patch is a bad one then; it is the initialization to 
Py_None which should be removed instead.

Well, you seem to have a better grasp on this stuff than me so I'll 
leave it for you :-) You are right, it is optimization stuff going wrong 
-- "entry.used" isn't honoured by the code in ModuleNode.py initializing 
the entries to Py_None. Just search for the comments inserted in the 
.c-file at that point in ModuleNode.py and it will put you right on 
track for where to insert the if-test.

As for the tests, I suppose all of this is really difficult to test for 
(as possibly the doctests would interfer a lot with the refcount of 
Py_None and so on), so actually I'd agree with you if you didn't bother 
with them.

-- 
Dag Sverre

From dalcinl at gmail.com  Wed Oct 15 23:39:55 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 15 Oct 2008 18:39:55 -0300
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F6530E.5080401@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<48F62285.80309@student.matnat.uio.no>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
	<48F64CFA.8030100@student.matnat.uio.no>
	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
	<48F6530E.5080401@student.matnat.uio.no>
Message-ID: <e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>

On Wed, Oct 15, 2008 at 5:31 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Well, you seem to have a better grasp on this stuff than me so I'll
> leave it for you :-) You are right, it is optimization stuff going wrong
> -- "entry.used" isn't honoured by the code in ModuleNode.py initializing
> the entries to Py_None. Just search for the comments inserted in the
> .c-file at that point in ModuleNode.py and it will put you right on
> track for where to insert the if-test.

I believe I will push my original patch with a BIG comment about what
is going one in Optimize.py

> As for the tests, I suppose all of this is really difficult to test for
> (as possibly the doctests would interfer a lot with the refcount of
> Py_None and so on), so actually I'd agree with you if you didn't bother
> with them.

BTW, perhaps if we could support the --clenaup option as a compiler
directive, then I would be able to write a rether good testcase.

Do this request make sense, I mean, adding a 'cleanup' directive
enabling the highest level of cleanup? I ALWAY build my code with
cleanup enabled, so I would love to be hable to use in my preambles::
#cython: cleanup=True


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Oct 16 01:34:07 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 16 Oct 2008 12:34:07 +1300
Subject: [Cython] Literal struct syntax
In-Reply-To: <32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <48F67DEF.9030800@canterbury.ac.nz>

Stefan Behnel wrote:

> Given this goal, I wonder why the literals are needed in the first place.
> Currently, you can just do this:
> 
>     p = Point(color=color, x=y)

I thought about that too, but something I had in mind
for Pyrex in the future was that

   p = Point()

where Point is a struct type would be shorthand for

   p = <Point *>malloc(sizeof(Point))

so I think I'd prefer to reserve the constructor
syntax for that.

-- 
Greg

From dalcinl at gmail.com  Thu Oct 16 03:31:11 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 15 Oct 2008 22:31:11 -0300
Subject: [Cython] Literal struct syntax
In-Reply-To: <48F67DEF.9030800@canterbury.ac.nz>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<48F67DEF.9030800@canterbury.ac.nz>
Message-ID: <e7ba66e40810151831s4ccd3a0ey1aa3e3be160bfc2@mail.gmail.com>

On Wed, Oct 15, 2008 at 8:34 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I thought about that too, but something I had in mind
> for Pyrex in the future was that
>
>   p = Point()
>
> where Point is a struct type would be shorthand for
>
>   p = <Point *>malloc(sizeof(Point))
>
> so I think I'd prefer to reserve the constructor
> syntax for that.

And how are you going to free the memory? Are you thinking about C++
semantics for stack-allocated class instances with
user-defined/Cython-syntetized constructors/destructors?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Oct 16 05:43:54 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 16 Oct 2008 16:43:54 +1300
Subject: [Cython] Literal struct syntax
In-Reply-To: <e7ba66e40810151831s4ccd3a0ey1aa3e3be160bfc2@mail.gmail.com>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<48F67DEF.9030800@canterbury.ac.nz>
	<e7ba66e40810151831s4ccd3a0ey1aa3e3be160bfc2@mail.gmail.com>
Message-ID: <48F6B87A.7040903@canterbury.ac.nz>

Lisandro Dalcin wrote:

> And how are you going to free the memory?

I'm not, that's up to the programmer. It's just a
more convenient way of writing a malloc.

-- 
Greg

From dagss at student.matnat.uio.no  Thu Oct 16 10:00:52 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Oct 2008 10:00:52 +0200
Subject: [Cython] Dag, please review this patch
In-Reply-To: <e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>	<48F62285.80309@student.matnat.uio.no>	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>	<48F62EC2.6080907@student.matnat.uio.no>	<48F632C5.6080706@student.matnat.uio.no>	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>	<48F64CFA.8030100@student.matnat.uio.no>	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>	<48F6530E.5080401@student.matnat.uio.no>
	<e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
Message-ID: <48F6F4B4.1020602@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Wed, Oct 15, 2008 at 5:31 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Well, you seem to have a better grasp on this stuff than me so I'll
>> leave it for you :-) You are right, it is optimization stuff going wrong
>> -- "entry.used" isn't honoured by the code in ModuleNode.py initializing
>> the entries to Py_None. Just search for the comments inserted in the
>> .c-file at that point in ModuleNode.py and it will put you right on
>> track for where to insert the if-test.
> 
> I believe I will push my original patch with a BIG comment about what
> is going one in Optimize.py

Huh? Your patch doesn't solve this problem either? (I just 
tried...fixnoneleaks.diff does nothing about the

cdef object foo1

not being declared issue).

The two should be orthogonal issues. I really believe that my patch is 
how the first problem should be solved. Attaching now how I believe the 
second problem should be solved, for which there seems to be no patch 
yet anyway.

If you agree I can simply make a push of my patches BTW, but there's 
been holes in my patches before so I'm waiting. If you find a way you 
think is better then feel free.

> BTW, perhaps if we could support the --clenaup option as a compiler
> directive, then I would be able to write a rether good testcase.
> 
> Do this request make sense, I mean, adding a 'cleanup' directive
> enabling the highest level of cleanup? I ALWAY build my code with
> cleanup enabled, so I would love to be hable to use in my preambles::
> #cython: cleanup=True

+1

But one should block its use as a decorator and with statement. Perhaps 
a more descriptive name -- module_cleanup or similar?

-- 
Dag Sverre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix_issue2.diff
Type: text/x-diff
Size: 584 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081016/0392f6e9/attachment.bin 

From dalcinl at gmail.com  Thu Oct 16 16:02:46 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Oct 2008 11:02:46 -0300
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F6F4B4.1020602@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
	<48F64CFA.8030100@student.matnat.uio.no>
	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
	<48F6530E.5080401@student.matnat.uio.no>
	<e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
	<48F6F4B4.1020602@student.matnat.uio.no>
Message-ID: <e7ba66e40810160702k60445d37n98a2eba3cc90c3a3@mail.gmail.com>

On Thu, Oct 16, 2008 at 5:00 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>
> Huh? Your patch doesn't solve this problem either? (I just
> tried...fixnoneleaks.diff does nothing about the
>
> cdef object foo1
>
> not being declared issue).

Ups! I wrote that at home last night, I believe it worked. But perhaps
I got confused.

> The two should be orthogonal issues. I really believe that my patch is how
> the first problem should be solved. Attaching now how I believe the second
> problem should be solved, for which there seems to be no patch yet anyway.

OK, I'll give a look right now.

> If you agree I can simply make a push of my patches BTW, but there's been
> holes in my patches before so I'm waiting. If you find a way you think is
> better then feel free.

You are far more experienced with Cython internals than me. So if you
said you way is the way, then I trust you. Just let me play a bit with
the corner cases.

>> BTW, perhaps if we could support the --clenaup option as a compiler
>> directive, then I would be able to write a rether good testcase.
>>
>> Do this request make sense, I mean, adding a 'cleanup' directive
>> enabling the highest level of cleanup? I ALWAY build my code with
>> cleanup enabled, so I would love to be hable to use in my preambles::
>> #cython: cleanup=True
>
> +1
>
> But one should block its use as a decorator and with statement. Perhaps a
> more descriptive name -- module_cleanup or similar?

Well, I would ask for 'modulecleanup' for consistency with the naming
of the other directives.  But I do not know yet how to disable
decorator and with statement. BTW, should this directive be an
integer, supporting various levels of cleanups, like the former
--cleanup option? Or it should mean a full cleanup?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Oct 16 16:20:02 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Oct 2008 11:20:02 -0300
Subject: [Cython] Literal struct syntax
In-Reply-To: <48F6B87A.7040903@canterbury.ac.nz>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<48F67DEF.9030800@canterbury.ac.nz>
	<e7ba66e40810151831s4ccd3a0ey1aa3e3be160bfc2@mail.gmail.com>
	<48F6B87A.7040903@canterbury.ac.nz>
Message-ID: <e7ba66e40810160720t1636c614m82c469ed7766be3b@mail.gmail.com>

On Thu, Oct 16, 2008 at 12:43 AM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> And how are you going to free the memory?
>
> I'm not, that's up to the programmer. It's just a
> more convenient way of writing a malloc.

Then I have to say that I'm -1 in such magic. What about introducing
two new pseudo-keywords cnew/cdel, and use it like this (like in C++):

MyStruct *p = cnew MyStruct(a=1,b=2)
# use p
cdel p # also sets p to NULL
assert p==NULL
cdel p # success, but do nothing?

Such approach would make Cython/Pyrex closer to C++ behavior.
Moreover, this could be extended to support funny features, like
letting user choose the alloc/dealloc routines and perhaps
constructor/destructor routines.

What do you think?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Oct 16 16:27:04 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Oct 2008 07:27:04 -0700
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F6F4B4.1020602@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>	<48F62285.80309@student.matnat.uio.no>	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>	<48F62EC2.6080907@student.matnat.uio.no>	<48F632C5.6080706@student.matnat.uio.no>	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>	<48F64CFA.8030100@student.matnat.uio.no>	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>	<48F6530E.5080401@student.matnat.uio.no>
	<e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
	<48F6F4B4.1020602@student.matnat.uio.no>
Message-ID: <3FD8A375-1691-40D7-A375-2C62C18D53BE@math.washington.edu>


On Oct 16, 2008, at 1:00 AM, Dag Sverre Seljebotn wrote:

> Lisandro Dalcin wrote:
>> On Wed, Oct 15, 2008 at 5:31 PM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> Well, you seem to have a better grasp on this stuff than me so I'll
>>> leave it for you :-) You are right, it is optimization stuff  
>>> going wrong
>>> -- "entry.used" isn't honoured by the code in ModuleNode.py  
>>> initializing
>>> the entries to Py_None. Just search for the comments inserted in the
>>> .c-file at that point in ModuleNode.py and it will put you right on
>>> track for where to insert the if-test.
>> I believe I will push my original patch with a BIG comment about what
>> is going one in Optimize.py
>
> Huh? Your patch doesn't solve this problem either? (I just  
> tried...fixnoneleaks.diff does nothing about the
>
> cdef object foo1
>
> not being declared issue).
>
> The two should be orthogonal issues. I really believe that my patch  
> is how the first problem should be solved. Attaching now how I  
> believe the second problem should be solved, for which there seems  
> to be no patch yet anyway.

If foo1 is never used, it should never be delcared. This avoids a C  
warning in this case, and doesn't impact the operation of the module.

>
> If you agree I can simply make a push of my patches BTW, but  
> there's been holes in my patches before so I'm waiting. If you find  
> a way you think is better then feel free.
>
>> BTW, perhaps if we could support the --clenaup option as a compiler
>> directive, then I would be able to write a rether good testcase.
>> Do this request make sense, I mean, adding a 'cleanup' directive
>> enabling the highest level of cleanup? I ALWAY build my code with
>> cleanup enabled, so I would love to be hable to use in my preambles::
>> #cython: cleanup=True
>
> +1
>
> But one should block its use as a decorator and with statement.  
> Perhaps a more descriptive name -- module_cleanup or similar?
>
> -- 
> Dag Sverre
> diff -r 2dbb7b8520a5 Cython/Compiler/ModuleNode.py
> --- a/Cython/Compiler/ModuleNode.py	Wed Oct 15 21:48:31 2008 +0200
> +++ b/Cython/Compiler/ModuleNode.py	Thu Oct 16 09:56:51 2008 +0200
> @@ -1734,7 +1734,7 @@ class ModuleNode(Nodes.Node, Nodes.Block
>          # variables to None.
>          for entry in env.var_entries:
>              if entry.visibility != 'extern':
> -                if entry.type.is_pyobject:
> +                if entry.type.is_pyobject and entry.used:
>                      code.put_init_var_to_py_none(entry)
>
>      def generate_c_function_export_code(self, env, code):
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Thu Oct 16 16:27:51 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Oct 2008 07:27:51 -0700
Subject: [Cython] Dag, please review this patch
In-Reply-To: <e7ba66e40810160702k60445d37n98a2eba3cc90c3a3@mail.gmail.com>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
	<48F64CFA.8030100@student.matnat.uio.no>
	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
	<48F6530E.5080401@student.matnat.uio.no>
	<e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
	<48F6F4B4.1020602@student.matnat.uio.no>
	<e7ba66e40810160702k60445d37n98a2eba3cc90c3a3@mail.gmail.com>
Message-ID: <8942D37A-E521-4515-8F11-8F8BC2BDC02D@math.washington.edu>


On Oct 16, 2008, at 7:02 AM, Lisandro Dalcin wrote:

> On Thu, Oct 16, 2008 at 5:00 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>>
>> Huh? Your patch doesn't solve this problem either? (I just
>> tried...fixnoneleaks.diff does nothing about the
>>
>> cdef object foo1
>>
>> not being declared issue).
>
> Ups! I wrote that at home last night, I believe it worked. But perhaps
> I got confused.
>
>> The two should be orthogonal issues. I really believe that my  
>> patch is how
>> the first problem should be solved. Attaching now how I believe  
>> the second
>> problem should be solved, for which there seems to be no patch yet  
>> anyway.
>
> OK, I'll give a look right now.
>
>> If you agree I can simply make a push of my patches BTW, but  
>> there's been
>> holes in my patches before so I'm waiting. If you find a way you  
>> think is
>> better then feel free.
>
> You are far more experienced with Cython internals than me. So if you
> said you way is the way, then I trust you. Just let me play a bit with
> the corner cases.
>
>>> BTW, perhaps if we could support the --clenaup option as a compiler
>>> directive, then I would be able to write a rether good testcase.
>>>
>>> Do this request make sense, I mean, adding a 'cleanup' directive
>>> enabling the highest level of cleanup? I ALWAY build my code with
>>> cleanup enabled, so I would love to be hable to use in my  
>>> preambles::
>>> #cython: cleanup=True
>>
>> +1
>>
>> But one should block its use as a decorator and with statement.  
>> Perhaps a
>> more descriptive name -- module_cleanup or similar?
>
> Well, I would ask for 'modulecleanup' for consistency with the naming
> of the other directives.  But I do not know yet how to disable
> decorator and with statement. BTW, should this directive be an
> integer, supporting various levels of cleanups, like the former
> --cleanup option? Or it should mean a full cleanup?

More descriptive is better. I'd prefer cleanupmodule, and it should  
be an integer.

- Robert




From stefan_ml at behnel.de  Thu Oct 16 16:42:21 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Oct 2008 16:42:21 +0200 (CEST)
Subject: [Cython] Literal struct syntax
In-Reply-To: <48F67DEF.9030800@canterbury.ac.nz>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<48F67DEF.9030800@canterbury.ac.nz>
Message-ID: <55661.213.61.181.86.1224168141.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Greg Ewing wrote:
> Stefan Behnel wrote:
>
>> Given this goal, I wonder why the literals are needed in the first
>> place.
>> Currently, you can just do this:
>>
>>     p = Point(color=color, x=y)
>
> I thought about that too, but something I had in mind
> for Pyrex in the future was that
>
>    p = Point()
>
> where Point is a struct type would be shorthand for
>
>    p = <Point *>malloc(sizeof(Point))

IMHO, if we hide the malloc(), we should hide the free(), just as we do
(or Python does) everywhere else. If we can't, it's better to make the
malloc() explicit, too.

Stefan


From robertwb at math.washington.edu  Thu Oct 16 17:33:24 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Oct 2008 08:33:24 -0700
Subject: [Cython] Literal struct syntax
In-Reply-To: <e7ba66e40810160720t1636c614m82c469ed7766be3b@mail.gmail.com>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<48F67DEF.9030800@canterbury.ac.nz>
	<e7ba66e40810151831s4ccd3a0ey1aa3e3be160bfc2@mail.gmail.com>
	<48F6B87A.7040903@canterbury.ac.nz>
	<e7ba66e40810160720t1636c614m82c469ed7766be3b@mail.gmail.com>
Message-ID: <4F4749BD-D22F-4F6E-93DA-CF45557B15C2@math.washington.edu>

On Oct 16, 2008, at 7:20 AM, Lisandro Dalcin wrote:

> On Thu, Oct 16, 2008 at 12:43 AM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
>> Lisandro Dalcin wrote:
>>
>>> And how are you going to free the memory?
>>
>> I'm not, that's up to the programmer. It's just a
>> more convenient way of writing a malloc.
>
> Then I have to say that I'm -1 in such magic. What about introducing
> two new pseudo-keywords cnew/cdel, and use it like this (like in C++):
>
> MyStruct *p = cnew MyStruct(a=1,b=2)
> # use p
> cdel p # also sets p to NULL
> assert p==NULL
> cdel p # success, but do nothing?
>
> Such approach would make Cython/Pyrex closer to C++ behavior.
> Moreover, this could be extended to support funny features, like
> letting user choose the alloc/dealloc routines and perhaps
> constructor/destructor routines.
>
> What do you think?

First, thanks for all the feedback, and thanks Dag for bringing this up.

I'd rather focus on making Cython closer to the Python language than  
introducing new concepts. Also, I'm with Stefan that it is very  
dangerous to hide the malloc and expect the user to explicitly  
provide the free. If the user wants to manage their own memory, they  
can do so with malloc and friends (understanding the associated  
dangers), and any special memory-management stuff we add should be as  
implicit and easy to use as Python's garbage collection (and probably  
piggyback off of it).

I think the MyStruct(...) should result in an actual struct (not a  
pointer), but perhaps on conversion to an object it could be made  
into a memory managed version of the struct (e.g. using the same  
string allocation trick we talked about for arrays) giving C access,  
pass-by-reference, and memory management. This, however, is not going  
to happen in the next release of Cython and when it does it could be  
made backwards compatible with the current conversion to dict.  
(Syntax suggestions for how to declare such an object welcome,  
perhaps "cdef object MyStruct foo" and memory-managed arrays could be  
"cdef object int[] foo")

As for the current proposal, don't want to introduce new, special  
"struct instanciation dictionaries," so I'd opt for allowing the  
{"a": 1, "b": 2} literals with the plan of conversion from a runtime  
dict down the road (including runtime-created keys), but disabling  
the {a: 1, b: 2} as the MyStruct(a=1, b=2) is available (and, even  
better, more explicit).

- Robert


From dalcinl at gmail.com  Thu Oct 16 17:36:17 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Oct 2008 12:36:17 -0300
Subject: [Cython] removing GCC warnings for test suite, please review [part1]
Message-ID: <e7ba66e40810160836u3f3f7357ge317ae6deab83054@mail.gmail.com>

How often do you scan the full output of 'python runtests.py' in order
to look for suspicious C-code generation? I guess never ;-), as Cython
test suite generate many warning with GCC.

As this situation make me feel very uncomfortable, you have here for
review a patch touching many test cases for removing those nasty GCC
warnings. If there are no major objections, I would like to push this
ASAP.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tests.diff
Type: text/x-patch
Size: 14832 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081016/24386980/attachment-0001.bin 

From dalcinl at gmail.com  Thu Oct 16 17:38:51 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Oct 2008 12:38:51 -0300
Subject: [Cython] removing GCC warnings for test suite, please review [part2]
Message-ID: <e7ba66e40810160838t548ab8e3p9559a34ea7cf2209@mail.gmail.com>

You can now see the gains. You have attached a 'runtests.log' file
with the full output of 'python runtest.py'. Please, look carefully at
the warnings. They are pointing real stuff: unused variables in buffer
syntax, docstrings that should never be emitted, and a 'statement with
no effect' in tests/run/pure.py (no time right now to dive the code in
order to see whats going on).

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: runtests.log
Type: application/octet-stream
Size: 18136 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081016/e48a0b11/attachment.obj 

From robertwb at math.washington.edu  Thu Oct 16 17:43:37 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Oct 2008 08:43:37 -0700
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
In-Reply-To: <e7ba66e40810160836u3f3f7357ge317ae6deab83054@mail.gmail.com>
References: <e7ba66e40810160836u3f3f7357ge317ae6deab83054@mail.gmail.com>
Message-ID: <B21AE792-B42E-450B-BBDB-8E36CC5B5A23@math.washington.edu>

On Oct 16, 2008, at 8:36 AM, Lisandro Dalcin wrote:

> How often do you scan the full output of 'python runtests.py' in order
> to look for suspicious C-code generation? I guess never ;-), as Cython
> test suite generate many warning with GCC.

Sometimes I watch it as I go by, but I've gotten used to the unused  
variable errors by now...  (Maybe a better word would be  
desensitized. Not sure it's a good thing.)

> As this situation make me feel very uncomfortable, you have here for
> review a patch touching many test cases for removing those nasty GCC
> warnings. If there are no major objections, I would like to push this
> ASAP.

Yes, it looks good. Ideally valid Cython code would execute with no  
warnings at all (other than those in the users code, like bad  
declarations and using uninitalized data).

- Robert


From dalcinl at gmail.com  Thu Oct 16 18:37:45 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Oct 2008 13:37:45 -0300
Subject: [Cython] Dag, please review this patch
In-Reply-To: <48F6F4B4.1020602@student.matnat.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
	<48F64CFA.8030100@student.matnat.uio.no>
	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
	<48F6530E.5080401@student.matnat.uio.no>
	<e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
	<48F6F4B4.1020602@student.matnat.uio.no>
Message-ID: <e7ba66e40810160937q2c008e4bn9cfa4103c466f425@mail.gmail.com>

On Thu, Oct 16, 2008 at 5:00 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> The two should be orthogonal issues. I really believe that my patch is how
> the first problem should be solved. Attaching now how I believe the second
> problem should be solved, for which there seems to be no patch yet anyway.

I've attached a new patch trying to fix all the issues following your
comments. And you are tight, this patch seems to be far better than my
previous ones.

http://trac.cython.org/cython_trac/attachment/ticket/90/GLOBALS.diff

I could add a testcase, but a really good one would need the
'modulecleanup' directive implemented, as I need to play with the
Py_AtExit() Python C-API call in order to detect that the generated
module cleanup is actually working.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ondrej at certik.cz  Thu Oct 16 18:39:44 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 16 Oct 2008 18:39:44 +0200
Subject: [Cython] docs: cython special methods
Message-ID: <85b5c3130810160939k364f0562h717e8fc7a0c96cfc@mail.gmail.com>

Hi,

here is a stub:

http://docs.cython.org/docs/extension_types.html#special-methods

which says:

There is a separate page devoted to this subject, and you should read
it carefully before attempting to use any special methods in your
extension types.

Where is the page? I only found this:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html

So maybe let's link to it?

Ondrej

From dagss at student.matnat.uio.no  Thu Oct 16 18:47:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 16 Oct 2008 18:47:00 +0200
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
Message-ID: <3307027641.1640445@smtp.netcom.no>

I am very against doing anything with the testcases - the fact that they trigger gcc warnings is a *good* thing.

The ultimate goal of any test suite  is full test coverage. If Cython is capable of producing gcc warnings, but the test suite does not, then that means that some code branches/some cases are never exercised, and hence a reduction in test coverage.

Of course, given resources, the warnings should be eliminated, and where that cannot be done, they should be verified as expected warnings and silenced. But that comes down to priorities...

At least my opinion is that ideally Cython should not produce C warnings if it can help it, instead any warnings should be emitted in Cython, for improved usability.

For instance with the buffer testcase, I definitely want to test *only* declaration and not using it, as a seperate testcase. Currently this leads to generating unused variables. I consider this a bug, but a bug that has been rather low on my list of priorities, especially since they would be automatically fixed by some stuff I'm thinking about down the line.

I refuse to call unused local variables a serious issue, compared to other stuff we could be improving. But, having the testcase there complaining  is much better than pretending the problem doesn't exist though.

(Disclaimer: I didn't actually read the patch as I am on my cell, answering to the contents of your post.)

Dag Sverre Seljebotn
-----Original Message-----
From: "Lisandro Dalcin" <dalcinl at gmail.com>
Date: Thursday, Oct 16, 2008 5:36 pm
Subject: [Cython] removing GCC warnings for test suite, please review [part1]
To: cython-dev <cython-dev at codespeak.net>Reply-To: cython-dev at codespeak.net

How often do you scan the full output of 'python runtests.py' in order
>to look for suspicious C-code generation? I guess never ;-), as Cython
>test suite generate many warning with GCC.
>
>As this situation make me feel very uncomfortable, you have here for
>review a patch touching many test cases for removing those nasty GCC
>warnings. If there are no major objections, I would like to push this
>ASAP.
>
>-- 
>Lisandro Dalc?n
>---------------
>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>Tel/Fax: +54-(0)342-451.1594
>


From ondrej at certik.cz  Thu Oct 16 19:12:11 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 16 Oct 2008 19:12:11 +0200
Subject: [Cython] wrapping C pointers
Message-ID: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>

Hi,

I want to use scipy's quadrature like this:

val, err = quadrature(func2, a, b, args=(f,))

where func2 is my Cython function and "f" is a C function pointer,
that will get executed from func2. The problem is, that "f" is passed
to the Python quadrature function, so it needs to be wrapped.
Currently what I do is:


cdef class MyFunc:
    cdef f2 thisptr

    cdef set_f(MyFunc self, f2 f):
        self.thisptr = f

    cdef f2 get_f(MyFunc self):
        return self.thisptr


cdef MyFunc mf = MyFunc()
mf.set_f(f)
val, err = quadrature(func2, a, b, args=(mf,))

Where func2 is:

def func2(a, MyFunc mf):
    cdef f2 f = mf.get_f()
    return array([f(x) for x in a])

This works nice. Is this the way to do it? Or is there some
better/simpler way. I don't know if it's a good idea to make Cython
clever enough to wrap things like this automatically?

Ondrej

From ondrej at certik.cz  Thu Oct 16 19:13:41 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 16 Oct 2008 19:13:41 +0200
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
Message-ID: <85b5c3130810161013y1d3eab77q66b0a9926becd6c2@mail.gmail.com>

On Thu, Oct 16, 2008 at 7:12 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I want to use scipy's quadrature like this:
>
> val, err = quadrature(func2, a, b, args=(f,))
>
> where func2 is my Cython function and "f" is a C function pointer,
> that will get executed from func2. The problem is, that "f" is passed
> to the Python quadrature function, so it needs to be wrapped.
> Currently what I do is:
>
>
> cdef class MyFunc:
>    cdef f2 thisptr

And here is a declaration of f2:

cdef extern from "cassembly.h":
    ctypedef double (*f2)(double x)

From robertwb at math.washington.edu  Thu Oct 16 19:37:53 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Oct 2008 10:37:53 -0700
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
In-Reply-To: <3307027641.1640445@smtp.netcom.no>
References: <3307027641.1640445@smtp.netcom.no>
Message-ID: <8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>

On Oct 16, 2008, at 9:47 AM, Dag Sverre Seljebotn wrote:

> I am very against doing anything with the testcases - the fact that  
> they trigger gcc warnings is a *good* thing.
>
> The ultimate goal of any test suite  is full test coverage. If  
> Cython is capable of producing gcc warnings, but the test suite  
> does not, then that means that some code branches/some cases are  
> never exercised, and hence a reduction in test coverage.
>
> Of course, given resources, the warnings should be eliminated, and  
> where that cannot be done, they should be verified as expected  
> warnings and silenced. But that comes down to priorities...
>
> At least my opinion is that ideally Cython should not produce C  
> warnings if it can help it, instead any warnings should be emitted  
> in Cython, for improved usability.
>
> For instance with the buffer testcase, I definitely want to test  
> *only* declaration and not using it, as a seperate testcase.  
> Currently this leads to generating unused variables. I consider  
> this a bug, but a bug that has been rather low on my list of  
> priorities, especially since they would be automatically fixed by  
> some stuff I'm thinking about down the line.
>
> I refuse to call unused local variables a serious issue, compared  
> to other stuff we could be improving. But, having the testcase  
> there complaining  is much better than pretending the problem  
> doesn't exist though.

I would rather have a trac tickets about this, and have much better  
signal-to-noise ratio while running the doctest. If a change  
introduces warnings (especially real one) then I'd rather be able to  
see them loud and clear--right now I probably wouldn't notice.

> (Disclaimer: I didn't actually read the patch as I am on my cell,  
> answering to the contents of your post.)

The patch turns unused local vars into used local vars.

- Robert

>
> Dag Sverre Seljebotn
> -----Original Message-----
> From: "Lisandro Dalcin" <dalcinl at gmail.com>
> Date: Thursday, Oct 16, 2008 5:36 pm
> Subject: [Cython] removing GCC warnings for test suite, please  
> review [part1]
> To: cython-dev <cython-dev at codespeak.net>Reply-To: cython- 
> dev at codespeak.net
>
> How often do you scan the full output of 'python runtests.py' in order
>> to look for suspicious C-code generation? I guess never ;-), as  
>> Cython
>> test suite generate many warning with GCC.
>>
>> As this situation make me feel very uncomfortable, you have here for
>> review a patch touching many test cases for removing those nasty GCC
>> warnings. If there are no major objections, I would like to push this
>> ASAP.
>>
>> -- 
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Thu Oct 16 19:47:29 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Oct 2008 10:47:29 -0700
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
Message-ID: <78D2CD58-9759-435D-9FEC-3919865CD9D4@math.washington.edu>

On Oct 16, 2008, at 10:12 AM, Ondrej Certik wrote:

> Hi,
>
> I want to use scipy's quadrature like this:
>
> val, err = quadrature(func2, a, b, args=(f,))
>
> where func2 is my Cython function and "f" is a C function pointer,
> that will get executed from func2. The problem is, that "f" is passed
> to the Python quadrature function, so it needs to be wrapped.
> Currently what I do is:
>
>
> cdef class MyFunc:
>     cdef f2 thisptr
>
>     cdef set_f(MyFunc self, f2 f):
>         self.thisptr = f
>
>     cdef f2 get_f(MyFunc self):
>         return self.thisptr
>
>
> cdef MyFunc mf = MyFunc()
> mf.set_f(f)
> val, err = quadrature(func2, a, b, args=(mf,))
>
> Where func2 is:
>
> def func2(a, MyFunc mf):
>     cdef f2 f = mf.get_f()
>     return array([f(x) for x in a])
>
> This works nice. Is this the way to do it? Or is there some
> better/simpler way. I don't know if it's a good idea to make Cython
> clever enough to wrap things like this automatically?

If I understand right, you want an automatic python wrapper of a C  
function. Perhaps it could be done, as long as the arguments and  
return type could all be converted, but it's certainly not  
implemented currently and easily enough done by the code you have above.

- Robert


From ggellner at uoguelph.ca  Thu Oct 16 20:48:29 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 16 Oct 2008 14:48:29 -0400
Subject: [Cython] docs: cython special methods
In-Reply-To: <85b5c3130810160939k364f0562h717e8fc7a0c96cfc@mail.gmail.com>
References: <85b5c3130810160939k364f0562h717e8fc7a0c96cfc@mail.gmail.com>
Message-ID: <20081016184829.GA20573@encolpuis>

On Thu, Oct 16, 2008 at 06:39:44PM +0200, Ondrej Certik wrote:
> Hi,
> 
> here is a stub:
> 
> http://docs.cython.org/docs/extension_types.html#special-methods
> 
> which says:
> 
> There is a separate page devoted to this subject, and you should read
> it carefully before attempting to use any special methods in your
> extension types.
> 
> Where is the page? I only found this:
> 
> http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html
> 
> So maybe let's link to it?
> 
I am very close to having it ported over, so we should have this at docs soon.

Gabriel

From dagss at student.matnat.uio.no  Thu Oct 16 21:07:26 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Oct 2008 21:07:26 +0200 (CEST)
Subject: [Cython] removing GCC warnings for test suite,
 please review [part1]
In-Reply-To: <8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
References: <3307027641.1640445@smtp.netcom.no>
	<8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
Message-ID: <32769.88.90.248.62.1224184046.squirrel@webmail.uio.no>

Robert wrote:
> On Oct 16, 2008, at 9:47 AM, Dag Sverre Seljebotn wrote:
>> I am very against doing anything with the testcases - the fact that
>> they trigger gcc warnings is a *good* thing.
>
> I would rather have a trac tickets about this, and have much better
> signal-to-noise ratio while running the doctest. If a change
> introduces warnings (especially real one) then I'd rather be able to
> see them loud and clear--right now I probably wouldn't notice.
>
>> (Disclaimer: I didn't actually read the patch as I am on my cell,
>> answering to the contents of your post.)
>
> The patch turns unused local vars into used local vars.

And Lisandro just reported a rather nasty bug where Cython would silently
produce uncompileable code (not warnings, plain errors) for unused
variables. (Global, but still.)

The "entry.used" stuff is non-trivial, and removing the testcases which
exercise that is not a good thing.

OK, time to find some common ground here:

- The patch is not bad in itself, but I strongly feel it should be
accompanied by a new testcase unused_symbols.pyx, where all necesarry,
non-duplicate tests are moved.

- Then this can be moved to tests/broken and a ticket opened for it,
referring to it.

BTW, Lisandro, thanks for doing this and your attention to detail! My
previous rant was a bit too harsh, and I'm slowly coming over to your side
now.

Dag Sverre


From dalcinl at gmail.com  Thu Oct 16 21:08:04 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Oct 2008 16:08:04 -0300
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
In-Reply-To: <8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
References: <3307027641.1640445@smtp.netcom.no>
	<8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
Message-ID: <e7ba66e40810161208o2d8af230icf34b379edf2bfb5@mail.gmail.com>

Well, I'll carefully review my patch and post is to the tracker.

Dag, I'll try to make clear the spirit of this patch. Supose some
Cython test case like this:

cdef void foo():
   int var1, var2
   var1 = var2

Then GCC will emit TWO anoying warnings:
1) 'foo' is never used
2) 'var2' was used uninitialized.

Then my patch will fix test cases like the above as follow:

cdef void foo():
   int var1, var2=0
   var1 = var2

foo()

Note that I'm just initializing 'var2' and using 'foo'. Do you see any
drawback about that?


On Thu, Oct 16, 2008 at 2:37 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 16, 2008, at 9:47 AM, Dag Sverre Seljebotn wrote:
>
>> I am very against doing anything with the testcases - the fact that
>> they trigger gcc warnings is a *good* thing.
>>
>> The ultimate goal of any test suite  is full test coverage. If
>> Cython is capable of producing gcc warnings, but the test suite
>> does not, then that means that some code branches/some cases are
>> never exercised, and hence a reduction in test coverage.
>>
>> Of course, given resources, the warnings should be eliminated, and
>> where that cannot be done, they should be verified as expected
>> warnings and silenced. But that comes down to priorities...
>>
>> At least my opinion is that ideally Cython should not produce C
>> warnings if it can help it, instead any warnings should be emitted
>> in Cython, for improved usability.
>>
>> For instance with the buffer testcase, I definitely want to test
>> *only* declaration and not using it, as a seperate testcase.
>> Currently this leads to generating unused variables. I consider
>> this a bug, but a bug that has been rather low on my list of
>> priorities, especially since they would be automatically fixed by
>> some stuff I'm thinking about down the line.
>>
>> I refuse to call unused local variables a serious issue, compared
>> to other stuff we could be improving. But, having the testcase
>> there complaining  is much better than pretending the problem
>> doesn't exist though.
>
> I would rather have a trac tickets about this, and have much better
> signal-to-noise ratio while running the doctest. If a change
> introduces warnings (especially real one) then I'd rather be able to
> see them loud and clear--right now I probably wouldn't notice.
>
>> (Disclaimer: I didn't actually read the patch as I am on my cell,
>> answering to the contents of your post.)
>
> The patch turns unused local vars into used local vars.
>
> - Robert
>
>>
>> Dag Sverre Seljebotn
>> -----Original Message-----
>> From: "Lisandro Dalcin" <dalcinl at gmail.com>
>> Date: Thursday, Oct 16, 2008 5:36 pm
>> Subject: [Cython] removing GCC warnings for test suite, please
>> review [part1]
>> To: cython-dev <cython-dev at codespeak.net>Reply-To: cython-
>> dev at codespeak.net
>>
>> How often do you scan the full output of 'python runtests.py' in order
>>> to look for suspicious C-code generation? I guess never ;-), as
>>> Cython
>>> test suite generate many warning with GCC.
>>>
>>> As this situation make me feel very uncomfortable, you have here for
>>> review a patch touching many test cases for removing those nasty GCC
>>> warnings. If there are no major objections, I would like to push this
>>> ASAP.
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ggellner at uoguelph.ca  Thu Oct 16 21:09:28 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 16 Oct 2008 15:09:28 -0400
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
Message-ID: <20081016190928.GB20573@encolpuis>

On Thu, Oct 16, 2008 at 07:12:11PM +0200, Ondrej Certik wrote:
> Hi,
> 
> I want to use scipy's quadrature like this:
> 
> val, err = quadrature(func2, a, b, args=(f,))
> 
> where func2 is my Cython function and "f" is a C function pointer,
> that will get executed from func2. The problem is, that "f" is passed
> to the Python quadrature function, so it needs to be wrapped.
> Currently what I do is:
> 
> 
> cdef class MyFunc:
>     cdef f2 thisptr
> 
>     cdef set_f(MyFunc self, f2 f):
>         self.thisptr = f
> 
>     cdef f2 get_f(MyFunc self):
>         return self.thisptr
> 
> 
> cdef MyFunc mf = MyFunc()
> mf.set_f(f)
> val, err = quadrature(func2, a, b, args=(mf,))
> 
> Where func2 is:
> 
> def func2(a, MyFunc mf):
>     cdef f2 f = mf.get_f()
>     return array([f(x) for x in a])
> 
> This works nice. Is this the way to do it? Or is there some
> better/simpler way. I don't know if it's a good idea to make Cython
> clever enough to wrap things like this automatically?
> 
I often just use a closure (I assume your are doing this all in cython), so
you can just have:

    cdef double f(double x):
        return x*x

    def func2(a):
        return array([f(x) for x in a])

    val, err = quadrature(func2, a, b) 

Though I am not sure if this works in your case. When we come to a consensus on
the idiomatic way of doing this we should write up some documentation!

Gabriel

From dagss at student.matnat.uio.no  Thu Oct 16 21:11:22 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Oct 2008 21:11:22 +0200 (CEST)
Subject: [Cython] Dag, please review this patch
In-Reply-To: <e7ba66e40810160937q2c008e4bn9cfa4103c466f425@mail.gmail.com>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<e7ba66e40810151014r2e529d31t41150de5878609e6@mail.gmail.com>
	<48F62EC2.6080907@student.matnat.uio.no>
	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
	<48F64CFA.8030100@student.matnat.uio.no>
	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
	<48F6530E.5080401@student.matnat.uio.no>
	<e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
	<48F6F4B4.1020602@student.matnat.uio.no>
	<e7ba66e40810160937q2c008e4bn9cfa4103c466f425@mail.gmail.com>
Message-ID: <32879.88.90.248.62.1224184282.squirrel@webmail.uio.no>

Lisandro wrote:
> On Thu, Oct 16, 2008 at 5:00 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> The two should be orthogonal issues. I really believe that my patch is
>> how
>> the first problem should be solved. Attaching now how I believe the
>> second
>> problem should be solved, for which there seems to be no patch yet
>> anyway.
>
> I've attached a new patch trying to fix all the issues following your
> comments. And you are tight, this patch seems to be far better than my
> previous ones.
>
> http://trac.cython.org/cython_trac/attachment/ticket/90/GLOBALS.diff
>
> I could add a testcase, but a really good one would need the
> 'modulecleanup' directive implemented, as I need to play with the
> Py_AtExit() Python C-API call in order to detect that the generated
> module cleanup is actually working.

Positive review!

(A testcase that could be added is this one:

"""
cdef object unused
"""

(that's all!) as that one was pretty nasty, resulting in uncompileable
code, and we don't want regression on it if somebody changes what you did
in the future...)

Dag Sverre


From robertwb at math.washington.edu  Thu Oct 16 21:12:16 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Oct 2008 12:12:16 -0700
Subject: [Cython] wrapping C pointers
In-Reply-To: <20081016190928.GB20573@encolpuis>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<20081016190928.GB20573@encolpuis>
Message-ID: <0779941A-94FC-4EF4-ABEF-1A442F1EDFAF@math.washington.edu>

On Oct 16, 2008, at 12:09 PM, Gabriel Gellner wrote:

> On Thu, Oct 16, 2008 at 07:12:11PM +0200, Ondrej Certik wrote:
>> Hi,
>>
>> I want to use scipy's quadrature like this:
>>
>> val, err = quadrature(func2, a, b, args=(f,))
>>
>> where func2 is my Cython function and "f" is a C function pointer,
>> that will get executed from func2. The problem is, that "f" is passed
>> to the Python quadrature function, so it needs to be wrapped.
>> Currently what I do is:
>>
>>
>> cdef class MyFunc:
>>     cdef f2 thisptr
>>
>>     cdef set_f(MyFunc self, f2 f):
>>         self.thisptr = f
>>
>>     cdef f2 get_f(MyFunc self):
>>         return self.thisptr
>>
>>
>> cdef MyFunc mf = MyFunc()
>> mf.set_f(f)
>> val, err = quadrature(func2, a, b, args=(mf,))
>>
>> Where func2 is:
>>
>> def func2(a, MyFunc mf):
>>     cdef f2 f = mf.get_f()
>>     return array([f(x) for x in a])
>>
>> This works nice. Is this the way to do it? Or is there some
>> better/simpler way. I don't know if it's a good idea to make Cython
>> clever enough to wrap things like this automatically?
>>
> I often just use a closure (I assume your are doing this all in  
> cython), so
> you can just have:
>
>     cdef double f(double x):
>         return x*x
>
>     def func2(a):
>         return array([f(x) for x in a])
>
>     val, err = quadrature(func2, a, b)
>
> Though I am not sure if this works in your case. When we come to a  
> consensus on
> the idiomatic way of doing this we should write up some documentation!

I believe the issue is that f needs to be received as a function  
pointer at compile time, but if I'm mistaken then this is certainly  
cleaner.

- Robert



From stefan_ml at behnel.de  Thu Oct 16 21:17:19 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Oct 2008 21:17:19 +0200
Subject: [Cython] removing GCC warnings for test suite,
 please review [part1]
In-Reply-To: <3307027641.1640445@smtp.netcom.no>
References: <3307027641.1640445@smtp.netcom.no>
Message-ID: <48F7933F.1070006@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> I am very against doing anything with the testcases - the fact that they
> trigger gcc warnings is a *good* thing.

It's not intentional, though. The tests that currently trigger gcc compiler
warnings are left-overs from Pyrex's test suite, where tests aren't run and do
not even get compiled in gcc.

So, it's a good thing that Lisandro provides a patch to fix these to free the
view on the things the tests are designed to test. If we use tests to check
for broken code or compiler warnings, they should test for this explicitly,
not by accident.

Stefan

From ondrej at certik.cz  Thu Oct 16 21:18:27 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 16 Oct 2008 21:18:27 +0200
Subject: [Cython] docs: cython special methods
In-Reply-To: <20081016184829.GA20573@encolpuis>
References: <85b5c3130810160939k364f0562h717e8fc7a0c96cfc@mail.gmail.com>
	<20081016184829.GA20573@encolpuis>
Message-ID: <85b5c3130810161218r88ba27br8c85121ea8190496@mail.gmail.com>

On Thu, Oct 16, 2008 at 8:48 PM, Gabriel Gellner <ggellner at uoguelph.ca> wrote:
> On Thu, Oct 16, 2008 at 06:39:44PM +0200, Ondrej Certik wrote:
>> Hi,
>>
>> here is a stub:
>>
>> http://docs.cython.org/docs/extension_types.html#special-methods
>>
>> which says:
>>
>> There is a separate page devoted to this subject, and you should read
>> it carefully before attempting to use any special methods in your
>> extension types.
>>
>> Where is the page? I only found this:
>>
>> http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html
>>
>> So maybe let's link to it?
>>
> I am very close to having it ported over, so we should have this at docs soon.

Excellent, thanks for all the work you do with docs.

Ondrej

From ondrej at certik.cz  Thu Oct 16 21:22:51 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 16 Oct 2008 21:22:51 +0200
Subject: [Cython] wrapping C pointers
In-Reply-To: <0779941A-94FC-4EF4-ABEF-1A442F1EDFAF@math.washington.edu>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<20081016190928.GB20573@encolpuis>
	<0779941A-94FC-4EF4-ABEF-1A442F1EDFAF@math.washington.edu>
Message-ID: <85b5c3130810161222t56501297k9ab4e5144f70949c@mail.gmail.com>

On Thu, Oct 16, 2008 at 9:12 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 16, 2008, at 12:09 PM, Gabriel Gellner wrote:
>
>> On Thu, Oct 16, 2008 at 07:12:11PM +0200, Ondrej Certik wrote:
>>> Hi,
>>>
>>> I want to use scipy's quadrature like this:
>>>
>>> val, err = quadrature(func2, a, b, args=(f,))
>>>
>>> where func2 is my Cython function and "f" is a C function pointer,
>>> that will get executed from func2. The problem is, that "f" is passed
>>> to the Python quadrature function, so it needs to be wrapped.
>>> Currently what I do is:
>>>
>>>
>>> cdef class MyFunc:
>>>     cdef f2 thisptr
>>>
>>>     cdef set_f(MyFunc self, f2 f):
>>>         self.thisptr = f
>>>
>>>     cdef f2 get_f(MyFunc self):
>>>         return self.thisptr
>>>
>>>
>>> cdef MyFunc mf = MyFunc()
>>> mf.set_f(f)
>>> val, err = quadrature(func2, a, b, args=(mf,))
>>>
>>> Where func2 is:
>>>
>>> def func2(a, MyFunc mf):
>>>     cdef f2 f = mf.get_f()
>>>     return array([f(x) for x in a])
>>>
>>> This works nice. Is this the way to do it? Or is there some
>>> better/simpler way. I don't know if it's a good idea to make Cython
>>> clever enough to wrap things like this automatically?
>>>
>> I often just use a closure (I assume your are doing this all in
>> cython), so
>> you can just have:
>>
>>     cdef double f(double x):
>>         return x*x
>>
>>     def func2(a):
>>         return array([f(x) for x in a])
>>
>>     val, err = quadrature(func2, a, b)
>>
>> Though I am not sure if this works in your case. When we come to a
>> consensus on
>> the idiomatic way of doing this we should write up some documentation!
>
> I believe the issue is that f needs to be received as a function
> pointer at compile time, but if I'm mistaken then this is certainly
> cleaner.

Yes, that's the problem, that I get "f" at compile time as a C
function pointer. This arises when I have a C++ program that needs to
call some scipy functions for integration. So I just want to create my
C function and pass it to Cython->scipy->Cython->C.

> If I understand right, you want an automatic python wrapper of a C
> function. Perhaps it could be done, as long as the arguments and
> return type could all be converted, but it's certainly not
> implemented currently and easily enough done by the code you have above.

Yes, but I want it to be explicit, no magic. Maybe there is some nice
way to do that, if not, then we can at least add my way into the
documentation as Gabriel has suggested.

Ondrej

From dagss at student.matnat.uio.no  Thu Oct 16 21:29:05 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Oct 2008 21:29:05 +0200 (CEST)
Subject: [Cython] removing GCC warnings for test suite,
 please review [part1]
In-Reply-To: <e7ba66e40810161208o2d8af230icf34b379edf2bfb5@mail.gmail.com>
References: <3307027641.1640445@smtp.netcom.no>
	<8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
	<e7ba66e40810161208o2d8af230icf34b379edf2bfb5@mail.gmail.com>
Message-ID: <33347.88.90.248.62.1224185345.squirrel@webmail.uio.no>

Lisandro wrote:
> Then my patch will fix test cases like the above as follow:
>
> cdef void foo():
>    int var1, var2=0
>    var1 = var2
>
> foo()
>
> Note that I'm just initializing 'var2' and using 'foo'. Do you see any
> drawback about that?

Well, there are drawbacks, but I'm getting less worried about how serious
they are.

But I'll just explain the drawbacks for the sake of the explanation. We
just had a severe bug espace our attention for months (and which you
found) because

"""
cdef object unused
"""

was not a testcase. Now

"""
cdef object unusded = 3
"""

would not have helped -- in fact this is an entirely different testcase!
The latter one makes "entry.used" be set differently and would trigger
entirely different paths through the code in Cython.

Now, local variables are a different matter. In fact, I don't know how
they work myself. I just know that if they are initialized, the testcase
tests something else than what it originally tested -- some if-tests
within Cython (concerning entry.used and so on) may start taking another
route, and perhaps some code blocks are then left untested.

But this can be countered by creating a new testcase specifically targeted
for covering what you now remove, so I'm getting more friendly towards
your patch.

Dag Sverre


From dagss at student.matnat.uio.no  Thu Oct 16 21:37:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Oct 2008 21:37:00 +0200 (CEST)
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
Message-ID: <32830.88.90.248.62.1224185820.squirrel@webmail.uio.no>

Ondrej Certik wrote:
...
> Where func2 is:
>
> def func2(a, MyFunc mf):
>     cdef f2 f = mf.get_f()
>     return array([f(x) for x in a])
>
> This works nice. Is this the way to do it? Or is there some
> better/simpler way. I don't know if it's a good idea to make Cython
> clever enough to wrap things like this automatically?

I think the way you did it is more than elegant enough :-) As others have
commented, this is fine.

This is just a note on performance: That list comprehension is really
going to kill performance. If you do this instead:

def func2(np.ndarray[right_dtype_t, ndim=1] a, MyFunc mf):
    cdef f2 f = mf.get_f()
    cdef np.ndarray[right_dtype_t, ndim=1] result = np.empty(a.shape,
right_dtype)
    cdef unsigned int i
    for i in range(a.shape[0]):
        result[i] = f(a[i])
    return result

...then it should be much, much faster (avoiding conversion of every
single array element back and forth from/to Python objects).

Dag Sverre


From ondrej at certik.cz  Thu Oct 16 22:44:24 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 16 Oct 2008 22:44:24 +0200
Subject: [Cython] wrapping C pointers
In-Reply-To: <32830.88.90.248.62.1224185820.squirrel@webmail.uio.no>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<32830.88.90.248.62.1224185820.squirrel@webmail.uio.no>
Message-ID: <85b5c3130810161344nc448504g11673877788bf457@mail.gmail.com>

On Thu, Oct 16, 2008 at 9:37 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
> ...
>> Where func2 is:
>>
>> def func2(a, MyFunc mf):
>>     cdef f2 f = mf.get_f()
>>     return array([f(x) for x in a])
>>
>> This works nice. Is this the way to do it? Or is there some
>> better/simpler way. I don't know if it's a good idea to make Cython
>> clever enough to wrap things like this automatically?
>
> I think the way you did it is more than elegant enough :-) As others have
> commented, this is fine.
>
> This is just a note on performance: That list comprehension is really
> going to kill performance. If you do this instead:
>
> def func2(np.ndarray[right_dtype_t, ndim=1] a, MyFunc mf):
>    cdef f2 f = mf.get_f()
>    cdef np.ndarray[right_dtype_t, ndim=1] result = np.empty(a.shape,
> right_dtype)
>    cdef unsigned int i
>    for i in range(a.shape[0]):
>        result[i] = f(a[i])
>    return result
>
> ...then it should be much, much faster (avoiding conversion of every
> single array element back and forth from/to Python objects).

Thanks Dag! This is really useful, I was thinking how to do that
efficiently. This is basically vectorise optimized for C functions.

Maybe stuff like this could go to some numpy Cython file shipped with
numpy? Those are things that are needed over and over again.

Ondrej

From dagss at student.matnat.uio.no  Thu Oct 16 23:05:50 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Oct 2008 23:05:50 +0200 (CEST)
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810161344nc448504g11673877788bf457@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<32830.88.90.248.62.1224185820.squirrel@webmail.uio.no>
	<85b5c3130810161344nc448504g11673877788bf457@mail.gmail.com>
Message-ID: <33553.88.90.248.62.1224191150.squirrel@webmail.uio.no>

Ondrej Certik wrote:
> On Thu, Oct 16, 2008 at 9:37 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Ondrej Certik wrote:
>> ...
>>> Where func2 is:
>>>
>>> def func2(a, MyFunc mf):
>>>     cdef f2 f = mf.get_f()
>>>     return array([f(x) for x in a])
>>>
>>> This works nice. Is this the way to do it? Or is there some
>>> better/simpler way. I don't know if it's a good idea to make Cython
>>> clever enough to wrap things like this automatically?
>>
>> I think the way you did it is more than elegant enough :-) As others
>> have
>> commented, this is fine.
>>
>> This is just a note on performance: That list comprehension is really
>> going to kill performance. If you do this instead:
>>
>> def func2(np.ndarray[right_dtype_t, ndim=1] a, MyFunc mf):
>>    cdef f2 f = mf.get_f()
>>    cdef np.ndarray[right_dtype_t, ndim=1] result = np.empty(a.shape,
>> right_dtype)
>>    cdef unsigned int i
>>    for i in range(a.shape[0]):
>>        result[i] = f(a[i])
>>    return result
>>
>> ...then it should be much, much faster (avoiding conversion of every
>> single array element back and forth from/to Python objects).
>
> Thanks Dag! This is really useful, I was thinking how to do that
> efficiently. This is basically vectorise optimized for C functions.
>
> Maybe stuff like this could go to some numpy Cython file shipped with
> numpy? Those are things that are needed over and over again.

Feel free to propose it for them :-) (I don't have anything to do with
NumPy development myself.) We could probably ship Cython utility libraries
available together with Cython as well. Someone needs to write it though,
and I don't have the time for it.

Note: It is only a 1D vectorise. To work with multi-dimensional arrays
more complicated code is needed (NumPy generic nd-iterators using the
NumPy C API). Probably the NumPy C API has some code one could utilize.

That's the thing with NumPy -- things always get very generic when one
wants to create something that's usable beyond your own specific usecase.

Dag Sverre


From ellisonbg.net at gmail.com  Fri Oct 17 00:16:51 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 16 Oct 2008 15:16:51 -0700
Subject: [Cython] Literal struct syntax
In-Reply-To: <4F4749BD-D22F-4F6E-93DA-CF45557B15C2@math.washington.edu>
References: <48F3B338.4060308@student.matnat.uio.no>
	<E72ABC2B-336C-4249-9D28-58CAE9D795DE@math.washington.edu>
	<32792.213.61.181.86.1224054204.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<48F67DEF.9030800@canterbury.ac.nz>
	<e7ba66e40810151831s4ccd3a0ey1aa3e3be160bfc2@mail.gmail.com>
	<48F6B87A.7040903@canterbury.ac.nz>
	<e7ba66e40810160720t1636c614m82c469ed7766be3b@mail.gmail.com>
	<4F4749BD-D22F-4F6E-93DA-CF45557B15C2@math.washington.edu>
Message-ID: <6ce0ac130810161516y2c9f7472i98a4c36f658de624@mail.gmail.com>

Just watching from the sides....

But, I vote +1 on disallowing the "almost" dict notation {a:1, b:2}.

Brian

On Thu, Oct 16, 2008 at 8:33 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 16, 2008, at 7:20 AM, Lisandro Dalcin wrote:
>
>> On Thu, Oct 16, 2008 at 12:43 AM, Greg Ewing
>> <greg.ewing at canterbury.ac.nz> wrote:
>>> Lisandro Dalcin wrote:
>>>
>>>> And how are you going to free the memory?
>>>
>>> I'm not, that's up to the programmer. It's just a
>>> more convenient way of writing a malloc.
>>
>> Then I have to say that I'm -1 in such magic. What about introducing
>> two new pseudo-keywords cnew/cdel, and use it like this (like in C++):
>>
>> MyStruct *p = cnew MyStruct(a=1,b=2)
>> # use p
>> cdel p # also sets p to NULL
>> assert p==NULL
>> cdel p # success, but do nothing?
>>
>> Such approach would make Cython/Pyrex closer to C++ behavior.
>> Moreover, this could be extended to support funny features, like
>> letting user choose the alloc/dealloc routines and perhaps
>> constructor/destructor routines.
>>
>> What do you think?
>
> First, thanks for all the feedback, and thanks Dag for bringing this up.
>
> I'd rather focus on making Cython closer to the Python language than
> introducing new concepts. Also, I'm with Stefan that it is very
> dangerous to hide the malloc and expect the user to explicitly
> provide the free. If the user wants to manage their own memory, they
> can do so with malloc and friends (understanding the associated
> dangers), and any special memory-management stuff we add should be as
> implicit and easy to use as Python's garbage collection (and probably
> piggyback off of it).
>
> I think the MyStruct(...) should result in an actual struct (not a
> pointer), but perhaps on conversion to an object it could be made
> into a memory managed version of the struct (e.g. using the same
> string allocation trick we talked about for arrays) giving C access,
> pass-by-reference, and memory management. This, however, is not going
> to happen in the next release of Cython and when it does it could be
> made backwards compatible with the current conversion to dict.
> (Syntax suggestions for how to declare such an object welcome,
> perhaps "cdef object MyStruct foo" and memory-managed arrays could be
> "cdef object int[] foo")
>
> As for the current proposal, don't want to introduce new, special
> "struct instanciation dictionaries," so I'd opt for allowing the
> {"a": 1, "b": 2} literals with the plan of conversion from a runtime
> dict down the road (including runtime-created keys), but disabling
> the {a: 1, b: 2} as the MyStruct(a=1, b=2) is available (and, even
> better, more explicit).
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dalcinl at gmail.com  Fri Oct 17 01:20:07 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 16 Oct 2008 20:20:07 -0300
Subject: [Cython] Dag, please review this patch
In-Reply-To: <32879.88.90.248.62.1224184282.squirrel@webmail.uio.no>
References: <e7ba66e40810150924v5c9be975t163061611b09d2d@mail.gmail.com>
	<48F632C5.6080706@student.matnat.uio.no>
	<e7ba66e40810151133r4b2795f6x936b3e556acc11df@mail.gmail.com>
	<48F64CFA.8030100@student.matnat.uio.no>
	<e7ba66e40810151310m303c03dalfda8655b1a0784d5@mail.gmail.com>
	<48F6530E.5080401@student.matnat.uio.no>
	<e7ba66e40810151439k1f7959eam99ff51e67a8e3c0a@mail.gmail.com>
	<48F6F4B4.1020602@student.matnat.uio.no>
	<e7ba66e40810160937q2c008e4bn9cfa4103c466f425@mail.gmail.com>
	<32879.88.90.248.62.1224184282.squirrel@webmail.uio.no>
Message-ID: <e7ba66e40810161620w753b338cq3b28e72b59739e8d@mail.gmail.com>

On Thu, Oct 16, 2008 at 4:11 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>
> Positive review!

OK, then I'll push it.

> (A testcase that could be added is this one:
>
> """
> cdef object unused
> """
>
> (that's all!) as that one was pretty nasty, resulting in uncompileable
> code, and we don't want regression on it if somebody changes what you did
> in the future...)

OK, Then I would add a unusedmoduleglobal.pyx inside tests/compile with

"""
cdef object unused
"""

and I'll try to implement some hackery (at the C level, using some
auxiliar include file) to make completelly sure that Cython actually
do not ever emitted the declaration.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Fri Oct 17 02:17:06 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 17 Oct 2008 13:17:06 +1300
Subject: [Cython] docs: cython special methods
In-Reply-To: <85b5c3130810160939k364f0562h717e8fc7a0c96cfc@mail.gmail.com>
References: <85b5c3130810160939k364f0562h717e8fc7a0c96cfc@mail.gmail.com>
Message-ID: <48F7D982.1060204@canterbury.ac.nz>

Ondrej Certik wrote:

> There is a separate page devoted to this subject, and you should read
> it carefully before attempting to use any special methods in your
> extension types.
> 
> Where is the page?

In the original Pyrex docs, the words "separate page" in the above
paragraph are a link, which seems to have got lost.

Also, the "Special Methods" link in the table of contents of that
page in the Pyrex docs now point to the page instead of the stub,
and there is a link to it in the top-level contents as well. You
may want to update the Cython docs similarly.

-- 
Greg

From ondrej at certik.cz  Fri Oct 17 02:57:29 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 17 Oct 2008 02:57:29 +0200
Subject: [Cython] wrapping C pointers
In-Reply-To: <33553.88.90.248.62.1224191150.squirrel@webmail.uio.no>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<32830.88.90.248.62.1224185820.squirrel@webmail.uio.no>
	<85b5c3130810161344nc448504g11673877788bf457@mail.gmail.com>
	<33553.88.90.248.62.1224191150.squirrel@webmail.uio.no>
Message-ID: <85b5c3130810161757j255ba010q89181ecc4c6e8916@mail.gmail.com>

On Thu, Oct 16, 2008 at 11:05 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
>> On Thu, Oct 16, 2008 at 9:37 PM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> Ondrej Certik wrote:
>>> ...
>>>> Where func2 is:
>>>>
>>>> def func2(a, MyFunc mf):
>>>>     cdef f2 f = mf.get_f()
>>>>     return array([f(x) for x in a])
>>>>
>>>> This works nice. Is this the way to do it? Or is there some
>>>> better/simpler way. I don't know if it's a good idea to make Cython
>>>> clever enough to wrap things like this automatically?
>>>
>>> I think the way you did it is more than elegant enough :-) As others
>>> have
>>> commented, this is fine.
>>>
>>> This is just a note on performance: That list comprehension is really
>>> going to kill performance. If you do this instead:
>>>
>>> def func2(np.ndarray[right_dtype_t, ndim=1] a, MyFunc mf):
>>>    cdef f2 f = mf.get_f()
>>>    cdef np.ndarray[right_dtype_t, ndim=1] result = np.empty(a.shape,
>>> right_dtype)
>>>    cdef unsigned int i
>>>    for i in range(a.shape[0]):
>>>        result[i] = f(a[i])
>>>    return result
>>>
>>> ...then it should be much, much faster (avoiding conversion of every
>>> single array element back and forth from/to Python objects).
>>
>> Thanks Dag! This is really useful, I was thinking how to do that
>> efficiently. This is basically vectorise optimized for C functions.
>>
>> Maybe stuff like this could go to some numpy Cython file shipped with
>> numpy? Those are things that are needed over and over again.
>
> Feel free to propose it for them :-) (I don't have anything to do with
> NumPy development myself.) We could probably ship Cython utility libraries
> available together with Cython as well. Someone needs to write it though,
> and I don't have the time for it.
>
> Note: It is only a 1D vectorise. To work with multi-dimensional arrays
> more complicated code is needed (NumPy generic nd-iterators using the
> NumPy C API). Probably the NumPy C API has some code one could utilize.
>
> That's the thing with NumPy -- things always get very generic when one
> wants to create something that's usable beyond your own specific usecase.

Yes. But on the other hand, I don't really use 2 or more dimensional
arrays anyway. I use sparse matrices, but those are represented but
several 1dim arrays as well, so I think the 1dim case is the most
useful.

Ondrej

From ggellner at uoguelph.ca  Fri Oct 17 03:27:26 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 16 Oct 2008 21:27:26 -0400
Subject: [Cython] docs: cython special methods
In-Reply-To: <48F7D982.1060204@canterbury.ac.nz>
References: <85b5c3130810160939k364f0562h717e8fc7a0c96cfc@mail.gmail.com>
	<48F7D982.1060204@canterbury.ac.nz>
Message-ID: <20081017012726.GA22528@encolpuis>

On Fri, Oct 17, 2008 at 01:17:06PM +1300, Greg Ewing wrote:
> Ondrej Certik wrote:
> 
> > There is a separate page devoted to this subject, and you should read
> > it carefully before attempting to use any special methods in your
> > extension types.
> > 
> > Where is the page?
> 
> In the original Pyrex docs, the words "separate page" in the above
> paragraph are a link, which seems to have got lost.
> 
> Also, the "Special Methods" link in the table of contents of that
> page in the Pyrex docs now point to the page instead of the stub,
> and there is a link to it in the top-level contents as well. You
> may want to update the Cython docs similarly.
> 
Thanks, will do.

Gabriel

From dalcinl at gmail.com  Fri Oct 17 05:28:50 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Oct 2008 00:28:50 -0300
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
Message-ID: <e7ba66e40810162028x68d649dfn5ad50b3c0ca0a75a@mail.gmail.com>

Ondrej, perhaps I'm missing something, but... Why not to implement
__call__ on MyFunc cdef class? IMHO, this way your code would looks
much better. Moreover, it would be faster.

On Thu, Oct 16, 2008 at 2:12 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I want to use scipy's quadrature like this:
>
> val, err = quadrature(func2, a, b, args=(f,))
>
> where func2 is my Cython function and "f" is a C function pointer,
> that will get executed from func2. The problem is, that "f" is passed
> to the Python quadrature function, so it needs to be wrapped.
> Currently what I do is:
>
>
> cdef class MyFunc:
>    cdef f2 thisptr
>
>    cdef set_f(MyFunc self, f2 f):
>        self.thisptr = f
>
>    cdef f2 get_f(MyFunc self):
>        return self.thisptr
>
>
> cdef MyFunc mf = MyFunc()
> mf.set_f(f)
> val, err = quadrature(func2, a, b, args=(mf,))
>
> Where func2 is:
>
> def func2(a, MyFunc mf):
>    cdef f2 f = mf.get_f()
>    return array([f(x) for x in a])
>
> This works nice. Is this the way to do it? Or is there some
> better/simpler way. I don't know if it's a good idea to make Cython
> clever enough to wrap things like this automatically?
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From collette at physics.ucla.edu  Fri Oct 17 05:42:58 2008
From: collette at physics.ucla.edu (Andrew Collette)
Date: Thu, 16 Oct 2008 20:42:58 -0700
Subject: [Cython] Module docstrings
Message-ID: <1224214978.7857.10.camel@tachyon-laptop>

Hi,

I'm a having a weird problem with module docstrings using Cython
0.9.8.1.1.  It seems that if a module contains any comments before the
docstring, it is simply ignored (module.__doc__ is None).  The docstring
doesn't even appear in the .c file.  FWIW, neither Pyrex nor Python
exhibits this behavior.

A workaround is to explicitly assign it to a variable __doc__:

# Header comment
__doc__ = \
"""
    Module docstring.
"""

Thanks,
Andrew Collette


From stefan_ml at behnel.de  Fri Oct 17 09:29:37 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Oct 2008 09:29:37 +0200 (CEST)
Subject: [Cython] Module docstrings
In-Reply-To: <1224214978.7857.10.camel@tachyon-laptop>
References: <1224214978.7857.10.camel@tachyon-laptop>
Message-ID: <42331.213.61.181.86.1224228577.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Andrew Collette wrote:
> I'm a having a weird problem with module docstrings using Cython
> 0.9.8.1.1.  It seems that if a module contains any comments before the
> docstring, it is simply ignored (module.__doc__ is None).  The docstring
> doesn't even appear in the .c file.  FWIW, neither Pyrex nor Python
> exhibits this behavior.

Thanks for the report. Just guessing, but this might have gone lost due to
the comment-based compiler parametrisation (file encoding, compiler
options, ...).

We already special case future imports using a "first statement" hint.
That might be a way to fix this.

Stefan


From dagss at student.matnat.uio.no  Fri Oct 17 09:51:17 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 17 Oct 2008 09:51:17 +0200 (CEST)
Subject: [Cython] Module docstrings
In-Reply-To: <42331.213.61.181.86.1224228577.squirrel@groupware.dvs.informatik.tu-d
	armstadt.de>
References: <1224214978.7857.10.camel@tachyon-laptop>
	<42331.213.61.181.86.1224228577.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <33731.88.90.248.62.1224229877.squirrel@webmail.uio.no>

Stefan wrote:
> Andrew Collette wrote:
>> I'm a having a weird problem with module docstrings using Cython
>> 0.9.8.1.1.  It seems that if a module contains any comments before the
>> docstring, it is simply ignored (module.__doc__ is None).  The docstring
>> doesn't even appear in the .c file.  FWIW, neither Pyrex nor Python
>> exhibits this behavior.
>
> Thanks for the report. Just guessing, but this might have gone lost due to
> the comment-based compiler parametrisation (file encoding, compiler
> options, ...).

Indeed. I'll take the blame for this one. Fix up at
http://hg.cython.org/cython-devel/rev/e597dcac63ac and
http://hg.cython.org/cython-devel/rev/e597dcac63ac

BTW when modifying the testcase (r_doctests.pyx), I encountered a strange
thing: If I made the doctest a unicode string, then Python 2.5 still
wouldn't print it out with a u prefix in the doctest. Is this supposed to
work differently?

Dag Sverre


From ondrej at certik.cz  Fri Oct 17 10:05:16 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 17 Oct 2008 10:05:16 +0200
Subject: [Cython] wrapping C pointers
In-Reply-To: <e7ba66e40810162028x68d649dfn5ad50b3c0ca0a75a@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<e7ba66e40810162028x68d649dfn5ad50b3c0ca0a75a@mail.gmail.com>
Message-ID: <85b5c3130810170105v7cb1bfy9e2f778eb8735a69@mail.gmail.com>

On Fri, Oct 17, 2008 at 5:28 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Ondrej, perhaps I'm missing something, but... Why not to implement
> __call__ on MyFunc cdef class? IMHO, this way your code would looks
> much better. Moreover, it would be faster.

Because __call__ cannot be a cdef function and def function cannot return void*.

Ondrej

From stefan_ml at behnel.de  Fri Oct 17 10:12:15 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Oct 2008 10:12:15 +0200 (CEST)
Subject: [Cython] Module docstrings
In-Reply-To: <33731.88.90.248.62.1224229877.squirrel@webmail.uio.no>
References: <1224214978.7857.10.camel@tachyon-laptop>
	<42331.213.61.181.86.1224228577.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<33731.88.90.248.62.1224229877.squirrel@webmail.uio.no>
Message-ID: <57617.213.61.181.86.1224231135.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> when modifying the testcase (r_doctests.pyx), I encountered a strange
> thing: If I made the doctest a unicode string, then Python 2.5 still
> wouldn't print it out with a u prefix in the doctest. Is this supposed to
> work differently?

No, that's the way docstrings work in Py2 for classes and modules, at
least at the C level (maybe also functions, don't remember). They are
passed as plain encoded byte strings (char*). Only Py3 decodes them from
UTF-8, Py2 takes them as they are.

This is definitely sub-optimal (thus the fix in Py3), and IIRC the current
implementation in Cython even encodes docstrings to UTF-8 to make them
work with Py3, even if they were originally byte strings encoded as, say,
latin-1. Might be worth another work-around for Py2 here...

Stefan


From dalcinl at gmail.com  Fri Oct 17 16:18:39 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Oct 2008 11:18:39 -0300
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810170105v7cb1bfy9e2f778eb8735a69@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<e7ba66e40810162028x68d649dfn5ad50b3c0ca0a75a@mail.gmail.com>
	<85b5c3130810170105v7cb1bfy9e2f778eb8735a69@mail.gmail.com>
Message-ID: <e7ba66e40810170718k702005f4x8bb3a8ff32b50154@mail.gmail.com>

Mmm, I'm not sure if you got the idea. In your code, I'm talking about
removing 'func2', and passing to 'cuadrature' an instance of MyFunc
(and implement __call__ in MyFunc), like this

cdef class MyFunc:
       cdef double (*f)(double)
       def __call__(self, a):
             return array([self.f(x) for x in a])

myfunc = MyFunc()
myfunc.set_f(f)
val, err = quadrature(myfunc, a, b)

See the attached files for a working example. Just run test_functor.py
(uses pyximport, no need to build yourself functor.pyx -> functor.so)
to make sure all is is working, then look at the fine code ;-).


On Fri, Oct 17, 2008 at 5:05 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Fri, Oct 17, 2008 at 5:28 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> Ondrej, perhaps I'm missing something, but... Why not to implement
>> __call__ on MyFunc cdef class? IMHO, this way your code would looks
>> much better. Moreover, it would be faster.
>
> Because __call__ cannot be a cdef function and def function cannot return void*.
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: functor.pyx
Type: application/octet-stream
Size: 714 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081017/0caf11ae/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_functor.py
Type: text/x-python
Size: 526 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081017/0caf11ae/attachment.py 

From dalcinl at gmail.com  Fri Oct 17 17:14:43 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Oct 2008 12:14:43 -0300
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
In-Reply-To: <33347.88.90.248.62.1224185345.squirrel@webmail.uio.no>
References: <3307027641.1640445@smtp.netcom.no>
	<8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
	<e7ba66e40810161208o2d8af230icf34b379edf2bfb5@mail.gmail.com>
	<33347.88.90.248.62.1224185345.squirrel@webmail.uio.no>
Message-ID: <e7ba66e40810170814g2eaddaedva09f72f8e7c48fe0@mail.gmail.com>

OK, I've opened a new ticket (#104) in track.
http://trac.cython.org/cython_trac/ticket/104

If there are no objections, I'll push the patch an close the ticket.




On Thu, Oct 16, 2008 at 4:29 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro wrote:
>> Then my patch will fix test cases like the above as follow:
>>
>> cdef void foo():
>>    int var1, var2=0
>>    var1 = var2
>>
>> foo()
>>
>> Note that I'm just initializing 'var2' and using 'foo'. Do you see any
>> drawback about that?
>
> Well, there are drawbacks, but I'm getting less worried about how serious
> they are.
>
> But I'll just explain the drawbacks for the sake of the explanation. We
> just had a severe bug espace our attention for months (and which you
> found) because
>
> """
> cdef object unused
> """
>
> was not a testcase. Now
>
> """
> cdef object unusded = 3
> """
>
> would not have helped -- in fact this is an entirely different testcase!
> The latter one makes "entry.used" be set differently and would trigger
> entirely different paths through the code in Cython.
>
> Now, local variables are a different matter. In fact, I don't know how
> they work myself. I just know that if they are initialized, the testcase
> tests something else than what it originally tested -- some if-tests
> within Cython (concerning entry.used and so on) may start taking another
> route, and perhaps some code blocks are then left untested.
>
> But this can be countered by creating a new testcase specifically targeted
> for covering what you now remove, so I'm getting more friendly towards
> your patch.
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Oct 17 20:02:26 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Oct 2008 15:02:26 -0300
Subject: [Cython] a quick fix for numpy.pxd, please review
Message-ID: <e7ba66e40810171102j7b7686ear5dc98a3db2d667f0@mail.gmail.com>

Dag, review this, perhaps I'm missing something.

BTW, Did you notice that the 'return' type of the macro
"PyArray_ITEMSIZE" is not 'Py_ssize_t' but 'int' (at least in
numpy-1.2.0)??. Should this be fixed?

diff -r b23f31007194 Cython/Includes/numpy.pxd
--- a/Cython/Includes/numpy.pxd	Thu Oct 16 20:26:34 2008 -0300
+++ b/Cython/Includes/numpy.pxd	Fri Oct 17 14:54:36 2008 -0300
@@ -176,8 +176,8 @@ cdef extern from "numpy/arrayobject.h":
     cdef int PyArray_TYPE(ndarray arr)
     cdef int PyArray_NDIM(ndarray arr)
     cdef int PyArray_ISWRITEABLE(ndarray arr)
-    cdef npy_intp PyArray_STRIDES(ndarray arr)
-    cdef npy_intp PyArray_DIMS(ndarray arr)
+    cdef npy_intp* PyArray_STRIDES(ndarray arr)
+    cdef npy_intp* PyArray_DIMS(ndarray arr)
     cdef Py_ssize_t PyArray_ITEMSIZE(ndarray arr)
     cdef int PyArray_CHKFLAGS(ndarray arr, int flags)
     cdef int PyArray_HASFIELDS(ndarray arr, int flags)

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ondrej at certik.cz  Fri Oct 17 21:06:08 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 17 Oct 2008 21:06:08 +0200
Subject: [Cython] wrapping C pointers
In-Reply-To: <e7ba66e40810170718k702005f4x8bb3a8ff32b50154@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<e7ba66e40810162028x68d649dfn5ad50b3c0ca0a75a@mail.gmail.com>
	<85b5c3130810170105v7cb1bfy9e2f778eb8735a69@mail.gmail.com>
	<e7ba66e40810170718k702005f4x8bb3a8ff32b50154@mail.gmail.com>
Message-ID: <85b5c3130810171206q385b680bj64d481aa7237199@mail.gmail.com>

On Fri, Oct 17, 2008 at 4:18 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Mmm, I'm not sure if you got the idea. In your code, I'm talking about
> removing 'func2', and passing to 'cuadrature' an instance of MyFunc
> (and implement __call__ in MyFunc), like this
>
> cdef class MyFunc:
>       cdef double (*f)(double)
>       def __call__(self, a):
>             return array([self.f(x) for x in a])
>
> myfunc = MyFunc()
> myfunc.set_f(f)
> val, err = quadrature(myfunc, a, b)
>
> See the attached files for a working example. Just run test_functor.py
> (uses pyximport, no need to build yourself functor.pyx -> functor.so)
> to make sure all is is working, then look at the fine code ;-).

Ah, now it looks much better. So why not to combine your code with
Dag's code to make vectorize really fast? :) Then there should be
basically now overhead if you pass it long numpy arrays.

Thanks for the tips and especially about pyximport! I didn't know
about it, it's just awesome. I hated fiddling with makefiles all the
time I did some cython development. Together with pure python mode,
this will be just awesome. I am very excited about all of these tools,
they make life so much easier.

Wow, this brings a lot of new possibilities, like trying to compile
the module first and if it fails, just import it as a python one
(assuming pure python syntax) and in any case you get the job done.

Ondrej

From dalcinl at gmail.com  Fri Oct 17 21:26:50 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Oct 2008 16:26:50 -0300
Subject: [Cython] wrapping C pointers
In-Reply-To: <85b5c3130810171206q385b680bj64d481aa7237199@mail.gmail.com>
References: <85b5c3130810161012r3dbe82aet855f3a2a66788318@mail.gmail.com>
	<e7ba66e40810162028x68d649dfn5ad50b3c0ca0a75a@mail.gmail.com>
	<85b5c3130810170105v7cb1bfy9e2f778eb8735a69@mail.gmail.com>
	<e7ba66e40810170718k702005f4x8bb3a8ff32b50154@mail.gmail.com>
	<85b5c3130810171206q385b680bj64d481aa7237199@mail.gmail.com>
Message-ID: <e7ba66e40810171226p7aa0b7cbi38662a5a382ff91e@mail.gmail.com>

On Fri, Oct 17, 2008 at 4:06 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>
> Ah, now it looks much better. So why not to combine your code with
> Dag's code to make vectorize really fast? :) Then there should be
> basically now overhead if you pass it long numpy arrays.

Just because I wrote it in a hurry in order to make my point ;-).

> Thanks for the tips and especially about pyximport! I didn't know
> about it, it's just awesome. I hated fiddling with makefiles all the
> time I did some cython development. Together with pure python mode,
> this will be just awesome. I am very excited about all of these tools,
> they make life so much easier.
> Wow, this brings a lot of new possibilities, like trying to compile
> the module first and if it fails, just import it as a python one
> (assuming pure python syntax) and in any case you get the job done.

I'm not sure if this is currently supported in pyximport, just check
the docs. Anyway, this should be workable.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Fri Oct 17 22:52:30 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 17 Oct 2008 22:52:30 +0200 (CEST)
Subject: [Cython] a quick fix for numpy.pxd, please review
In-Reply-To: <e7ba66e40810171102j7b7686ear5dc98a3db2d667f0@mail.gmail.com>
References: <e7ba66e40810171102j7b7686ear5dc98a3db2d667f0@mail.gmail.com>
Message-ID: <33754.88.90.248.62.1224276750.squirrel@webmail.uio.no>

Lisandro wrote:
> Dag, review this, perhaps I'm missing something.

Positive review (assuming the test_numpy.pyx testcase still runs of
course; I assume it will, but it is strange that it currently works
without a problem, as these are used in __getbuffer__ ...).

BTW obviously all the available functions/macros should be defined in that
pxd, I just didn't get around to it. Any patches improving numpy.pxd in
this area will be accepted without any discussion (as long as the pointer
types are got right).

> BTW, Did you notice that the 'return' type of the macro
> "PyArray_ITEMSIZE" is not 'Py_ssize_t' but 'int' (at least in
> numpy-1.2.0)??. Should this be fixed?

Please do. It looks nicer (though it doesn't really matter for non-pointer
types, as it is C which is going to do the real typechecking in this case;
Cython only needs to know int vs. float and signed vs. unsigned).

Dag Sverre


From dalcinl at gmail.com  Sat Oct 18 00:53:34 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 17 Oct 2008 19:53:34 -0300
Subject: [Cython] a quick fix for numpy.pxd, please review
In-Reply-To: <33754.88.90.248.62.1224276750.squirrel@webmail.uio.no>
References: <e7ba66e40810171102j7b7686ear5dc98a3db2d667f0@mail.gmail.com>
	<33754.88.90.248.62.1224276750.squirrel@webmail.uio.no>
Message-ID: <e7ba66e40810171553m78c33195h9c871a76946b2efc@mail.gmail.com>

On Fri, Oct 17, 2008 at 5:52 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro wrote:
>> Dag, review this, perhaps I'm missing something.
>
> Positive review (assuming the test_numpy.pyx testcase still runs of
> course; I assume it will, but it is strange that it currently works
> without a problem, as these are used in __getbuffer__ ...).

Well, it wotked because __getbuffer__  has an explicit cast.

> BTW obviously all the available functions/macros should be defined in that
> pxd, I just didn't get around to it. Any patches improving numpy.pxd in
> this area will be accepted without any discussion (as long as the pointer
> types are got right).
>
>> BTW, Did you notice that the 'return' type of the macro
>> "PyArray_ITEMSIZE" is not 'Py_ssize_t' but 'int' (at least in
>> numpy-1.2.0)??. Should this be fixed?
>
> Please do. It looks nicer (though it doesn't really matter for non-pointer
> types, as it is C which is going to do the real typechecking in this case;
> Cython only needs to know int vs. float and signed vs. unsigned).
>

OK, I'll take a look at what can be added/fixed/improved.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From org.andrey at gmail.com  Sat Oct 18 08:15:22 2008
From: org.andrey at gmail.com (Andrey Ivanov)
Date: Sat, 18 Oct 2008 06:15:22 +0000 (UTC)
Subject: [Cython] How do I write C++ shadow classes..
Message-ID: <loom.20081018T053038-609@post.gmane.org>

Hello Cython developers,

recently I started to refactor my C++/Python project, where I used a SWIG tool 
to build extention, and I discovered that Pyrex/Cython tool is much more faster 
and eats less memory. But there are some cons, it cannot create shadow C++ 
classes automaticly, so user must write them by hands. Of course this is needed 
base functionality for such tools, but it is not what developers want (lazy, 
like me :) ).

So I've decided make an addition to Cython tool to provide such functionality. 
Because I'm not very experienced in how Pyrex/Cython works (and there are some 
specific, because tool must parse .h) I've decided to write preprocessor wich 
would roll out simple modified 'extern from' construction. It integrates with 
Cython in Utils.py module (by ihooks) and provides preprocessed files instead 
of original ones.

There are a few constructions parsed by preprocessor (I called it ccython 
because of lack of imagination):

cdef extern from "China.hpp" import Dress, Phone:
    pass
-------------------------
cdef extern from "China.hpp":
    ctypedef struct c_Dress "Dress":
       void tearAtaTouch();
    c_Dress *new_Dress "new Dress" ()
    void del_Dress "delete" (c_Dress * ptr)
    ctypedef struct c_Phone "Phone":
       void ring();
       int break(int days);
    c_Phone *new_Phone "new Phone" ()
    void del_Phone "delete" (c_Phone * ptr)
etc.. all classes specified in import list will be imported, also using of * is 
possible. And:

cdef class Phone import *:
   pass
-------------------------
cdef class Phone:
   cdef c_Phone *thisptr
   def __cinit__(self): new_Phone()
   def __dealloc__(self): del_Phone(self.thisptr)
   def ring(self): self.thisptr.break()
   def break(self, days): return self.thisptr.break(days)

(all like in C++ example on cython.org)
also instead of pass other methods can be specified by hands. Only specified .h 
file and specified class definition parsed, but there is a possibility to use 
'gcc -E' output (all included) and look at all class hierarchy.

I used pyparsing module to parse .pyx and .h files. Now tool is in pre-alpha, 
so I have some question:
- how it corellate with Cython developers goals? Does exist a little chance for 
such features to be implemented in original Cython (my project is only a little 
test).
- is that syntax good ?
- what should I do with arguments of C++ class types? (create shadows for them!)

Now TODO list looks like:
- parse C preprocessed files instead of headers (may be done by pythonic tools 
or just 'gcc -E')
- add import lists for extern structs (I not intends to spam local scope with 
identifiers what I've never seen)
- resolve conflicts with already declared names (how?)
- look through all class tree for public methods and attributes

Thanks for attention!



From robertwb at math.washington.edu  Sun Oct 19 05:52:57 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 18 Oct 2008 20:52:57 -0700
Subject: [Cython] How do I write C++ shadow classes..
In-Reply-To: <loom.20081018T053038-609@post.gmane.org>
References: <loom.20081018T053038-609@post.gmane.org>
Message-ID: <B7DAF9F1-0A65-4CA8-B15E-71CF93F9D743@math.washington.edu>

On Oct 17, 2008, at 11:15 PM, Andrey Ivanov wrote:

> Hello Cython developers,
>
> recently I started to refactor my C++/Python project, where I used  
> a SWIG tool
> to build extention, and I discovered that Pyrex/Cython tool is much  
> more faster
> and eats less memory. But there are some cons, it cannot create  
> shadow C++
> classes automaticly, so user must write them by hands. Of course  
> this is needed
> base functionality for such tools, but it is not what developers  
> want (lazy,
> like me :) ).

Actually, a lot of people do want this, but it's just that there are  
other things we want more, and we have limited time :). Personally, I  
tend to wrap more C than C++ code, and find the interface I want to  
provide to Python is not necessarily the "raw" interface as given by  
the underlying library, but this would certainly come in handy. And  
it could certainly be useful for C as well as C++.

> So I've decided make an addition to Cython tool to provide such  
> functionality.
> Because I'm not very experienced in how Pyrex/Cython works (and  
> there are some
> specific, because tool must parse .h) I've decided to write  
> preprocessor wich
> would roll out simple modified 'extern from' construction. It  
> integrates with
> Cython in Utils.py module (by ihooks) and provides preprocessed  
> files instead
> of original ones.

Hopefully you'll be able to contribute directly to Cython itself. My  
one concern is that I don't want to add pyparsing as a dependancy,  
but the parser we ship with should be flexible enough for our needs.

> There are a few constructions parsed by preprocessor

I'm not quite sure exactly what you're explaining here--is the part  
above the line your input, and the part below autogenerated?

> (I called it ccython because of lack of imagination):
>
> cdef extern from "China.hpp" import Dress, Phone:
>     pass
> -------------------------
> cdef extern from "China.hpp":
>     ctypedef struct c_Dress "Dress":
>        void tearAtaTouch();
>     c_Dress *new_Dress "new Dress" ()
>     void del_Dress "delete" (c_Dress * ptr)
>     ctypedef struct c_Phone "Phone":
>        void ring();
>        int break(int days);
>     c_Phone *new_Phone "new Phone" ()
>     void del_Phone "delete" (c_Phone * ptr)
> etc.. all classes specified in import list will be imported, also  
> using of * is
> possible. And:
>
> cdef class Phone import *:
>    pass
> -------------------------
> cdef class Phone:
>    cdef c_Phone *thisptr
>    def __cinit__(self): new_Phone()

I assume you mean "self. thisptr =  new_Phone()

>    def __dealloc__(self): del_Phone(self.thisptr)
>    def ring(self): self.thisptr.break()

and self.thisptr.ring()

>    def break(self, days): return self.thisptr.break(days)
>
> (all like in C++ example on cython.org)
> also instead of pass other methods can be specified by hands. Only  
> specified .h
> file and specified class definition parsed, but there is a  
> possibility to use
> 'gcc -E' output (all included) and look at all class hierarchy.

Yes, gcc -E would certainly be useful in many cases--we certainly  
don't want to be doing our own macro expansion and all.

>
> I used pyparsing module to parse .pyx and .h files. Now tool is in  
> pre-alpha,
> so I have some question:
> - how it corellate with Cython developers goals? Does exist a  
> little chance for
> such features to be implemented in original Cython (my project is  
> only a little
> test).

The only reason it's not there is because no one's stepped up to do  
it. Now it looks like you have :).

> - is that syntax good ?

I like the "cdef extern from 'file.h' import ..." syntax, though  
perhaps instead of automatically mangling names one should write  
"import Phone as c_Phone." The "cdef class Phone import *:" seems a  
bit odd though... A while back we talked about supporting auto- 
generated methods (e.g. for pickling, comparison, str, ...) and your  
auto-generated class seems like a candidate for the same syntax.  
Should "new" and "delete" be added as

> - what should I do with arguments of C++ class types? (create  
> shadows for them!)

If we end up creating shadows for everything, then that will slow  
things down immensely. This is one of the reasons SWIG is slow, and  
although I think Cython wrappings could be a lot tighter, if we're  
just passing a bunch of Python objects around there's a limit to how  
fast you can be. It could be possible to make cpdef functions and  
only wrap/unwrap things as they pass into Python land (via shadows or  
ctypes or a CObjects)? Would such shadow types be compatible across  
modules?

> Now TODO list looks like:
> - parse C preprocessed files instead of headers (may be done by  
> pythonic tools
> or just 'gcc -E')
> - add import lists for extern structs (I not intends to spam local  
> scope with
> identifiers what I've never seen)
> - resolve conflicts with already declared names (how?)

I think this should just be an error... Eventually, polymorphism  
would be a very nice thing to have in Cython.

> - look through all class tree for public methods and attributes

Yep. How to handle class inheritance (and the ensuing type  
compatibility) is another big question.

- Robert



From robertwb at math.washington.edu  Sun Oct 19 06:38:30 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 18 Oct 2008 21:38:30 -0700
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
In-Reply-To: <e7ba66e40810170814g2eaddaedva09f72f8e7c48fe0@mail.gmail.com>
References: <3307027641.1640445@smtp.netcom.no>
	<8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
	<e7ba66e40810161208o2d8af230icf34b379edf2bfb5@mail.gmail.com>
	<33347.88.90.248.62.1224185345.squirrel@webmail.uio.no>
	<e7ba66e40810170814g2eaddaedva09f72f8e7c48fe0@mail.gmail.com>
Message-ID: <5CD2AED2-D404-40B1-9E66-F91863BC98B7@math.washington.edu>

On Oct 17, 2008, at 8:14 AM, Lisandro Dalcin wrote:

> OK, I've opened a new ticket (#104) in track.
> http://trac.cython.org/cython_trac/ticket/104
>
> If there are no objections, I'll push the patch an close the ticket.

Yes, please do. Did you open a ticket for the unused variables warnings?

>
>
>
>
> On Thu, Oct 16, 2008 at 4:29 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Lisandro wrote:
>>> Then my patch will fix test cases like the above as follow:
>>>
>>> cdef void foo():
>>>    int var1, var2=0
>>>    var1 = var2
>>>
>>> foo()
>>>
>>> Note that I'm just initializing 'var2' and using 'foo'. Do you  
>>> see any
>>> drawback about that?
>>
>> Well, there are drawbacks, but I'm getting less worried about how  
>> serious
>> they are.
>>
>> But I'll just explain the drawbacks for the sake of the  
>> explanation. We
>> just had a severe bug espace our attention for months (and which you
>> found) because
>>
>> """
>> cdef object unused
>> """
>>
>> was not a testcase. Now
>>
>> """
>> cdef object unusded = 3
>> """
>>
>> would not have helped -- in fact this is an entirely different  
>> testcase!
>> The latter one makes "entry.used" be set differently and would  
>> trigger
>> entirely different paths through the code in Cython.
>>
>> Now, local variables are a different matter. In fact, I don't know  
>> how
>> they work myself. I just know that if they are initialized, the  
>> testcase
>> tests something else than what it originally tested -- some if-tests
>> within Cython (concerning entry.used and so on) may start taking  
>> another
>> route, and perhaps some code blocks are then left untested.
>>
>> But this can be countered by creating a new testcase specifically  
>> targeted
>> for covering what you now remove, so I'm getting more friendly  
>> towards
>> your patch.
>>
>> Dag Sverre
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Sun Oct 19 22:10:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 19 Oct 2008 22:10:00 +0200
Subject: [Cython] How do I write C++ shadow classes..
Message-ID: <3307299022.571604@smtp.netcom.no>

Some random thoughts on this:

This is basically two orthogonal things (as given in two seperaate examples):
a) Automatic inclusion of .h-files "as if they were pxd files" (or, making pxds unecessary). This has very wide uses.

b) Automatic generation of wrappers for cpp classes.

I think a) has a very wide circle of potential users, and perhaps getting this integrated first could be a first goal here?

As for syntax, here's another idea (not that I'm against the original proposal):

For a):

from "myheader.h" cimport foo

(or from cython.cheaders.myheader, or similar)

I.e. the .h-file acts exactly like a pxd file. This should integrate very nicely with existing codebases (i.e. some pxds could simply be deleted and the import changed and things would still work the same :-) ).

This has some problems with performance obviously - only the necesarry symbols should go from .h to "implicit pxd" stage. It looks like this is in place though from the other syntax, and now that type symbol resolution is postponed to after parsing (yey!) this could even work with a normal "cimport 'myheader.h' as mylib; mylib.foo(...)"...

For b), I'm thinking something like:

from "mylib.h" cimport MyCppClass

cdef class MyCyClass(MyCppClass):
    pass

i.e. you can "inherit" from the C++ class in Cython. This also provides a nice syntax right away for extending the wrapper with more stuff in addition to the automatic part.

Finally, thoughts on integration with Cython: The first thing to do after a successful integration of what exists now, would be looking at producing a "tree preprocessor" rather than file preprocessor. I.e. in whatever code exists now it should be rather simple to emit the nodes of Nodes.py rather than Cython code strings, which should give a performance boost and make things less complex overall.

As for pyparsing, I note my disagreement with Robert. I don't think it would hurt to add another dependency as long as it is an optional for optional features, at least if switching parser is nontrivial.

Dag Sverre Seljebotn
-----Original Message-----
From: Robert Bradshaw <robertwb at math.washington.edu>
Date: Sunday, Oct 19, 2008 5:53 am
Subject: Re: [Cython] How do I write C++ shadow classes..
To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net

On Oct 17, 2008, at 11:15 PM, Andrey Ivanov wrote:
>
>> Hello Cython developers,
>
>> recently I started to refactor my C++/Python project, where I used  
> a SWIG tool
> to build extention, and I discovered that Pyrex/Cython tool is much  
> more faster
> and eats less memory. But there are some cons, it cannot create  
> shadow C++
> classes automaticly, so user must write them by hands. Of course  
> this is needed
> base functionality for such tools, but it is not what developers  
> want (lazy,
> like me :) ).
>
>Actually, a lot of people do want this, but it's just that there are  
>other things we want more, and we have limited time :). Personally, I  
>tend to wrap more C than C++ code, and find the interface I want to  
>provide to Python is not necessarily the "raw" interface as given by  
>the underlying library, but this would certainly come in handy. And  
>it could certainly be useful for C as well as C++.
>
>> So I've decided make an addition to Cython tool to provide such  
> functionality.
> Because I'm not very experienced in how Pyrex/Cython works (and  
> there are some
> specific, because tool must parse .h) I've decided to write  
> preprocessor wich
> would roll out simple modified 'extern from' construction. It  
> integrates with
> Cython in Utils.py module (by ihooks) and provides preprocessed  
> files instead
> of original ones.
>
>Hopefully you'll be able to contribute directly to Cython itself. My  
>one concern is that I don't want to add pyparsing as a dependancy,  
>but the parser we ship with should be flexible enough for our needs.
>
>> There are a few constructions parsed by preprocessor
>
>I'm not quite sure exactly what you're explaining here--is the part  
>above the line your input, and the part below autogenerated?
>
>> (I called it ccython because of lack of imagination):
>
>> cdef extern from "China.hpp" import Dress, Phone:
>     pass
> -------------------------
> cdef extern from "China.hpp":
>     ctypedef struct c_Dress "Dress":
>        void tearAtaTouch();
>     c_Dress *new_Dress "new Dress" ()
>     void del_Dress "delete" (c_Dress * ptr)
>     ctypedef struct c_Phone "Phone":
>        void ring();
>        int break(int days);
>     c_Phone *new_Phone "new Phone" ()
>     void del_Phone "delete" (c_Phone * ptr)
> etc.. all classes specified in import list will be imported, also  
> using of * is
> possible. And:
>
>> cdef class Phone import *:
>    pass
> -------------------------
> cdef class Phone:
>    cdef c_Phone *thisptr
>    def __cinit__(self): new_Phone()
>
>I assume you mean "self. thisptr =  new_Phone()
>
>>    def __dealloc__(self): del_Phone(self.thisptr)
>    def ring(self): self.thisptr.break()
>
>and self.thisptr.ring()
>
>>    def break(self, days): return self.thisptr.break(days)
>
>> (all like in C++ example on cython.org)
> also instead of pass other methods can be specified by hands. Only  
> specified .h
> file and specified class definition parsed, but there is a  
> possibility to use
> 'gcc -E' output (all included) and look at all class hierarchy.
>
>Yes, gcc -E would certainly be useful in many cases--we certainly  
>don't want to be doing our own macro expansion and all.
>
>>
> I used pyparsing module to parse .pyx and .h files. Now tool is in  
> pre-alpha,
> so I have some question:
> - how it corellate with Cython developers goals? Does exist a  
> little chance for
> such features to be implemented in original Cython (my project is  
> only a little
> test).
>
>The only reason it's not there is because no one's stepped up to do  
>it. Now it looks like you have :).
>
>> - is that syntax good ?
>
>I like the "cdef extern from 'file.h' import ..." syntax, though  
>perhaps instead of automatically mangling names one should write  
>"import Phone as c_Phone." The "cdef class Phone import *:" seems a  
>bit odd though... A while back we talked about supporting auto- 
>generated methods (e.g. for pickling, comparison, str, ...) and your  
>auto-generated class seems like a candidate for the same syntax.  
>Should "new" and "delete" be added as
>
>> - what should I do with arguments of C++ class types? (create  
> shadows for them!)
>
>If we end up creating shadows for everything, then that will slow  
>things down immensely. This is one of the reasons SWIG is slow, and  
>although I think Cython wrappings could be a lot tighter, if we're  
>just passing a bunch of Python objects around there's a limit to how  
>fast you can be. It could be possible to make cpdef functions and  
>only wrap/unwrap things as they pass into Python land (via shadows or  
>ctypes or a CObjects)? Would such shadow types be compatible across  
>modules?
>
>> Now TODO list looks like:
> - parse C preprocessed files instead of headers (may be done by  
> pythonic tools
> or just 'gcc -E')
> - add import lists for extern structs (I not intends to spam local  
> scope with
> identifiers what I've never seen)
> - resolve conflicts with already declared names (how?)
>
>I think this should just be an error... Eventually, polymorphism  
>would be a very nice thing to have in Cython.
>
>> - look through all class tree for public methods and attributes
>
>Yep. How to handle class inheritance (and the ensuing type  
>compatibility) is another big question.
>
>- Robert
>
>
>_______________________________________________
>Cython-dev mailing list
>Cython-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/cython-dev
>


From dalcinl at gmail.com  Mon Oct 20 14:49:12 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 20 Oct 2008 09:49:12 -0300
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
In-Reply-To: <5CD2AED2-D404-40B1-9E66-F91863BC98B7@math.washington.edu>
References: <3307027641.1640445@smtp.netcom.no>
	<8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
	<e7ba66e40810161208o2d8af230icf34b379edf2bfb5@mail.gmail.com>
	<33347.88.90.248.62.1224185345.squirrel@webmail.uio.no>
	<e7ba66e40810170814g2eaddaedva09f72f8e7c48fe0@mail.gmail.com>
	<5CD2AED2-D404-40B1-9E66-F91863BC98B7@math.washington.edu>
Message-ID: <e7ba66e40810200549t18f585c3vc1c2bfec21f0b1b9@mail.gmail.com>

On Sun, Oct 19, 2008 at 1:38 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 17, 2008, at 8:14 AM, Lisandro Dalcin wrote:
>
>> OK, I've opened a new ticket (#104) in track.
>> http://trac.cython.org/cython_trac/ticket/104
>>
>> If there are no objections, I'll push the patch an close the ticket.
>
> Yes, please do.

Done.

> Did you open a ticket for the unused variables warnings?

Ticket #140 could be closed. But some warnings are still there and
need a more close look; they are signaling actual code generation
problems. So, they perhaps deserve new tickets for each case.

So Robert, I'll ask you to run the full test suite, take a fast look
at the few warnings remaining, and close ticket #140 if you consider
that appropriate.


>
>>
>>
>>
>>
>> On Thu, Oct 16, 2008 at 4:29 PM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> Lisandro wrote:
>>>> Then my patch will fix test cases like the above as follow:
>>>>
>>>> cdef void foo():
>>>>    int var1, var2=0
>>>>    var1 = var2
>>>>
>>>> foo()
>>>>
>>>> Note that I'm just initializing 'var2' and using 'foo'. Do you
>>>> see any
>>>> drawback about that?
>>>
>>> Well, there are drawbacks, but I'm getting less worried about how
>>> serious
>>> they are.
>>>
>>> But I'll just explain the drawbacks for the sake of the
>>> explanation. We
>>> just had a severe bug espace our attention for months (and which you
>>> found) because
>>>
>>> """
>>> cdef object unused
>>> """
>>>
>>> was not a testcase. Now
>>>
>>> """
>>> cdef object unusded = 3
>>> """
>>>
>>> would not have helped -- in fact this is an entirely different
>>> testcase!
>>> The latter one makes "entry.used" be set differently and would
>>> trigger
>>> entirely different paths through the code in Cython.
>>>
>>> Now, local variables are a different matter. In fact, I don't know
>>> how
>>> they work myself. I just know that if they are initialized, the
>>> testcase
>>> tests something else than what it originally tested -- some if-tests
>>> within Cython (concerning entry.used and so on) may start taking
>>> another
>>> route, and perhaps some code blocks are then left untested.
>>>
>>> But this can be countered by creating a new testcase specifically
>>> targeted
>>> for covering what you now remove, so I'm getting more friendly
>>> towards
>>> your patch.
>>>
>>> Dag Sverre
>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ggellner at uoguelph.ca  Mon Oct 20 19:48:27 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Mon, 20 Oct 2008 13:48:27 -0400
Subject: [Cython] pxd packages
Message-ID: <20081020174827.GA1570@encolpuis>

So I am trying to get my mind around how the newish pxd packages work,
that is when I have a folder with pxd files and an __init__.py file.

Suppose I want the following structure:

    foo/
       math.pxd
       param.pxd

So that I can do

    cimport foo

    foo.math.func()
    foo.param.PI

    # etc


Can this be done? (I know that I can do cimport foo.math, but can use use the
__init__.py file in some way to be able to use the pxd files like a normal
package?)

Secondly, how to I set the include path for pxd packages in setup.py

I know I do `cython -I .` etc for doing it by hand. Any tips?

Finally our Pyrex difference states that package names and cross directory
imports are just like python, is there anywhere that explains the semantics
for Pyrex?

thanks,
Gabriel

From jek-gmane at kleckner.net  Wed Oct 22 05:29:26 2008
From: jek-gmane at kleckner.net (Jim Kleckner)
Date: Tue, 21 Oct 2008 20:29:26 -0700
Subject: [Cython] doctest wraps stdout so file pointers can't convert
Message-ID: <gdm6mo$rec$1@ger.gmane.org>

I ran across an annoyance and wondered if anyone else hit this.

I have some compiled C code wrapped with Cython that writes to
an ordinary file descriptor when the debug level is high enough.
It directly references sys.stdout and sends output there.  When
using a doctest, however, that object is replaced with _SpoofOut
that subclasses StringIO.  Thus, when executing something like the
following in a doctest:

   ...
   cdef file outFile = sys.stdout  # This fails for the spoofed output
   callDebugCodeInC(PyFile_AsFile(outFile))
   ...

It (understandably) fails on the assignment with a message:
   TypeError: Cannot convert instance to file

Ideas?  At the moment, I just forgo the convenience of turning on
that debug within doctest.


From greg.ewing at canterbury.ac.nz  Wed Oct 22 07:30:51 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 22 Oct 2008 18:30:51 +1300
Subject: [Cython] doctest wraps stdout so file pointers can't convert
In-Reply-To: <gdm6mo$rec$1@ger.gmane.org>
References: <gdm6mo$rec$1@ger.gmane.org>
Message-ID: <48FEBA8B.60306@canterbury.ac.nz>

Jim Kleckner wrote:

>    TypeError: Cannot convert instance to file
> 
> Ideas?  At the moment, I just forgo the convenience of turning on
> that debug within doctest.

You could write your output the Pythonic way instead of
trying to obtain a file descriptor.

-- 
Greg

From dalcinl at gmail.com  Wed Oct 22 08:27:56 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Oct 2008 03:27:56 -0300
Subject: [Cython] doctest wraps stdout so file pointers can't convert
In-Reply-To: <48FEBA8B.60306@canterbury.ac.nz>
References: <gdm6mo$rec$1@ger.gmane.org> <48FEBA8B.60306@canterbury.ac.nz>
Message-ID: <e7ba66e40810212327v793e707bs91dd4afdc715f2e4@mail.gmail.com>

On Wed, Oct 22, 2008 at 2:30 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Jim Kleckner wrote:
>
>>    TypeError: Cannot convert instance to file
>>
>> Ideas?  At the moment, I just forgo the convenience of turning on
>> that debug within doctest.

Well, use 'sys.__stdout__' .  That should always be a 'file' instance
actually associated to standard output.

> You could write your output the Pythonic way instead of
> trying to obtain a file descriptor.
>

Greg, It seems Jim is calling some testing code in pure C and wrapped
with Cython, and it seems such code need FILE* pointer. What are you
suggesting here? I'm missing something?



> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Wed Oct 22 11:17:00 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 22 Oct 2008 22:17:00 +1300
Subject: [Cython] doctest wraps stdout so file pointers can't convert
In-Reply-To: <e7ba66e40810212327v793e707bs91dd4afdc715f2e4@mail.gmail.com>
References: <gdm6mo$rec$1@ger.gmane.org> <48FEBA8B.60306@canterbury.ac.nz>
	<e7ba66e40810212327v793e707bs91dd4afdc715f2e4@mail.gmail.com>
Message-ID: <48FEEF8C.2010803@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Greg, It seems Jim is calling some testing code in pure C and wrapped
> with Cython

Sorry, didn't read the original message carefully enough.

-- 
Greg

From robertwb at math.washington.edu  Wed Oct 22 17:43:48 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Oct 2008 08:43:48 -0700
Subject: [Cython] How do I write C++ shadow classes..
In-Reply-To: <3307299022.571604@smtp.netcom.no>
References: <3307299022.571604@smtp.netcom.no>
Message-ID: <CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>

On Oct 19, 2008, at 1:10 PM, Dag Sverre Seljebotn wrote:

> Some random thoughts on this:
>
> This is basically two orthogonal things (as given in two seperaate  
> examples):
> a) Automatic inclusion of .h-files "as if they were pxd files" (or,  
> making pxds unecessary). This has very wide uses.
>
> b) Automatic generation of wrappers for cpp classes.
>
> I think a) has a very wide circle of potential users, and perhaps  
> getting this integrated first could be a first goal here?

Yes, for sure. I've actually thought about doing this several times,  
but there have been too many other things higher on my list.

> As for syntax, here's another idea (not that I'm against the  
> original proposal):
>
> For a):
>
> from "myheader.h" cimport foo

I like this syntax.

>
> (or from cython.cheaders.myheader, or similar)
>
> I.e. the .h-file acts exactly like a pxd file. This should  
> integrate very nicely with existing codebases (i.e. some pxds could  
> simply be deleted and the import changed and things would still  
> work the same :-) ).
>
> This has some problems with performance obviously - only the  
> necesarry symbols should go from .h to "implicit pxd" stage. It  
> looks like this is in place though from the other syntax, and now  
> that type symbol resolution is postponed to after parsing (yey!)  
> this could even work with a normal "cimport 'myheader.h' as mylib;  
> mylib.foo(...)"...

For both .pxd and (eventually) .h, I think caching will greatly  
mitigate these performance issues. If .h parsing is considerably  
slower though, it might be useful to have a .h -> .pxd utility.

> For b), I'm thinking something like:
>
> from "mylib.h" cimport MyCppClass
>
> cdef class MyCyClass(MyCppClass):
>     pass
>
> i.e. you can "inherit" from the C++ class in Cython. This also  
> provides a nice syntax right away for extending the wrapper with  
> more stuff in addition to the automatic part.

I thought of this too. It's nice syntactic sugar, but it's a bit odd  
in the sense that now MyCyClass can't inherit from something else  
(well, I suppose we could do multiple-inheritance syntax) and we  
might want to expose the inheritance tree of MyCppClass somehow as well.

However, it seems like the intent is to make easy shadow classes, and  
for this purpose it could serve very well. Should all the methods be  
exposed? (One would certainly want to have that as an option.) If  
not, how to select them?

> Finally, thoughts on integration with Cython: The first thing to do  
> after a successful integration of what exists now, would be looking  
> at producing a "tree preprocessor" rather than file preprocessor.  
> I.e. in whatever code exists now it should be rather simple to emit  
> the nodes of Nodes.py rather than Cython code strings, which should  
> give a performance boost and make things less complex overall.

Yep, though the main work is that one doesn't start with at tree  
(especially for (a)). Also, decelerator nodes are such a pain to  
produce. But this is not justification to not do it the right way.  
It'd be awesome to have something that takes a .h file and produces a  
parsed pxd tree.

> As for pyparsing, I note my disagreement with Robert. I don't think  
> it would hurt to add another dependency as long as it is an  
> optional for optional features, at least if switching parser is  
> nontrivial.

I don't see this as an "optional" feature, but something we should  
have in the core of Cython. We already ship with a parser, so if  
possible I'd like to use that one.

- Robert


From robertwb at math.washington.edu  Wed Oct 22 17:49:11 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Oct 2008 08:49:11 -0700
Subject: [Cython] removing GCC warnings for test suite,
	please review [part1]
In-Reply-To: <e7ba66e40810200549t18f585c3vc1c2bfec21f0b1b9@mail.gmail.com>
References: <3307027641.1640445@smtp.netcom.no>
	<8AFFD1C0-F2A4-4A94-AE15-1AA5FA92B9B0@math.washington.edu>
	<e7ba66e40810161208o2d8af230icf34b379edf2bfb5@mail.gmail.com>
	<33347.88.90.248.62.1224185345.squirrel@webmail.uio.no>
	<e7ba66e40810170814g2eaddaedva09f72f8e7c48fe0@mail.gmail.com>
	<5CD2AED2-D404-40B1-9E66-F91863BC98B7@math.washington.edu>
	<e7ba66e40810200549t18f585c3vc1c2bfec21f0b1b9@mail.gmail.com>
Message-ID: <5CB0B0C2-F1D9-40DC-9817-4281661046AA@math.washington.edu>

On Oct 20, 2008, at 5:49 AM, Lisandro Dalcin wrote:

> On Sun, Oct 19, 2008 at 1:38 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Oct 17, 2008, at 8:14 AM, Lisandro Dalcin wrote:
>>
>>> OK, I've opened a new ticket (#104) in track.
>>> http://trac.cython.org/cython_trac/ticket/104
>>>
>>> If there are no objections, I'll push the patch an close the ticket.
>>
>> Yes, please do.
>
> Done.
>
>> Did you open a ticket for the unused variables warnings?
>
> Ticket #140 could be closed. But some warnings are still there and
> need a more close look; they are signaling actual code generation
> problems. So, they perhaps deserve new tickets for each case.
>
> So Robert, I'll ask you to run the full test suite, take a fast look
> at the few warnings remaining, and close ticket #140 if you consider
> that appropriate.

It's great, I can actually see bad warnings now :). Closing #104 (you  
scared me for a bit when I saw #140 :)

- Robert

From dagss at student.matnat.uio.no  Wed Oct 22 18:05:14 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 22 Oct 2008 18:05:14 +0200
Subject: [Cython] How do I write C++ shadow classes..
In-Reply-To: <CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>
References: <3307299022.571604@smtp.netcom.no>
	<CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>
Message-ID: <48FF4F3A.3030509@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Oct 19, 2008, at 1:10 PM, Dag Sverre Seljebotn wrote:
>> Finally, thoughts on integration with Cython: The first thing to do  
>> after a successful integration of what exists now, would be looking  
>> at producing a "tree preprocessor" rather than file preprocessor.  
>> I.e. in whatever code exists now it should be rather simple to emit  
>> the nodes of Nodes.py rather than Cython code strings, which should  
>> give a performance boost and make things less complex overall.
> 
> Yep, though the main work is that one doesn't start with at tree  
> (especially for (a)). Also, decelerator nodes are such a pain to  
> produce. But this is not justification to not do it the right way.  
 > It'd be awesome to have something that takes a .h file and produces a
 > parsed pxd tree.

The way I understood it (though I might be wrong) is that Andrey is 
already using pyparsing to parse the .h file and produce a tree, and 
then emits a string from that tree. Replacing the string-writing 
commands with node-emitting commands shouldn't be such a pain (but it 
depends on the rigour to which the .h files are parsed).

Of course, if the types are just treated as string identifier nodes of 
some kind without any real interpretation (since Cython as a C-like 
syntax and one can just shuffle things around) then one would need to 
chain in some parts of the Cython parser. Difficult to tell without 
seeing any code though.

Andrey: I think this discussion would be easier if we could see any 
code, just to have a look at the approach etc, not necesarrily to use it.

-- 
Dag Sverre

From robertwb at math.washington.edu  Wed Oct 22 18:13:36 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Oct 2008 09:13:36 -0700
Subject: [Cython] pxd packages
In-Reply-To: <20081020174827.GA1570@encolpuis>
References: <20081020174827.GA1570@encolpuis>
Message-ID: <8DDFD3F6-E314-4847-9589-8267F3DA699D@math.washington.edu>

On Oct 20, 2008, at 10:48 AM, Gabriel Gellner wrote:

> So I am trying to get my mind around how the newish pxd packages work,
> that is when I have a folder with pxd files and an __init__.py file.
>
> Suppose I want the following structure:
>
>     foo/
>        math.pxd
>        param.pxd
>
> So that I can do
>
>     cimport foo
>
>     foo.math.func()
>     foo.param.PI
>
>     # etc
>
>
> Can this be done? (I know that I can do cimport foo.math, but can  
> use use the
> __init__.py file in some way to be able to use the pxd files like a  
> normal
> package?)

No, I don't think you can do this. In Python, foo.math is an actual  
object that can be further queried. Perhaps we should support this.

> Secondly, how to I set the include path for pxd packages in setup.py
>
> I know I do `cython -I .` etc for doing it by hand. Any tips?

There's an include_dir option you can pass to the setup command.

> Finally our Pyrex difference states that package names and cross  
> directory
> imports are just like python, is there anywhere that explains the  
> semantics
> for Pyrex?

I'm not sure, but I know they've changed over time. It used to be  
that you had to sit everything in the same folder, and just name the  
files "a.b.c" rather than putting stuff in folders.

- Robert


From org.andrey at gmail.com  Wed Oct 22 19:12:44 2008
From: org.andrey at gmail.com (Andrey Ivanov)
Date: Wed, 22 Oct 2008 17:12:44 +0000 (UTC)
Subject: [Cython] How do I write C++ shadow classes..
References: <3307299022.571604@smtp.netcom.no>
	<CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>
	<48FF4F3A.3030509@student.matnat.uio.no>
Message-ID: <loom.20081022T163936-465@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:

> The way I understood it (though I might be wrong) is that Andrey is 
> already using pyparsing to parse the .h file and produce a tree, and 
> then emits a string from that tree. Replacing the string-writing 
> commands with node-emitting commands shouldn't be such a pain (but it 
> depends on the rigour to which the .h files are parsed).
> 

Yeap I use pyparsing, because I don't know how PLEX works and how to use it 
with C/C++ sources. My program is written very crudely, firstly because of lack 
of skills with writing compilers and such things, secondly just because it 
already suits my needs.

> Of course, if the types are just treated as string identifier nodes of 
> some kind without any real interpretation (since Cython as a C-like 
> syntax and one can just shuffle things around) then one would need to 
> chain in some parts of the Cython parser. Difficult to tell without 
> seeing any code though.
> 
> Andrey: I think this discussion would be easier if we could see any 
> code, just to have a look at the approach etc, not necesarrily to use it.
> 

Sorry for long pause, I've been reading your opinions while seeking hosting for 
project. I've just got hosting on SF and uploaded my project there: http://
sourceforge.net/projects/ccython



From org.andrey at gmail.com  Wed Oct 22 19:44:53 2008
From: org.andrey at gmail.com (Andrey Ivanov)
Date: Wed, 22 Oct 2008 17:44:53 +0000 (UTC)
Subject: [Cython] How do I write C++ shadow classes..
References: <3307299022.571604@smtp.netcom.no>
	<CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>
Message-ID: <loom.20081022T171822-582@post.gmane.org>

Robert Bradshaw <robertwb at ...> writes:

> >
> > I think a) has a very wide circle of potential users, and perhaps  
> > getting this integrated first could be a first goal here?
> 
> Yes, for sure. I've actually thought about doing this several times,  
> but there have been too many other things higher on my list.
> 

Hmm.. almost all users of SWIG just sleep and dream how to get it work faster. 
But they are lazy and cython has no autogeneration...


> > As for syntax, here's another idea (not that I'm against the  
> > original proposal):
> >
> > For a):
> >
> > from "myheader.h" cimport foo
> 
> I like this syntax.

Yeap, it looks much nicer than "cdef extern from.." - horrible bunch of Pyrex, 
C and Python keywords.

 
> However, it seems like the intent is to make easy shadow classes, and  
> for this purpose it could serve very well. Should all the methods be  
> exposed? (One would certainly want to have that as an option.) If  
> not, how to select them?

This is why I started to use "cdef class Foo import *:" - list of exposed 
methods can be just specified like import list. "cdef class Foo import add, 
sub, mul, div:". Fake "inheritance" may only replace common * variant where all 
public methods exposed.

> 
> > Finally, thoughts on integration with Cython: The first thing to do  
> > after a successful integration of what exists now, would be looking  
> > at producing a "tree preprocessor" rather than file preprocessor.  
> > I.e. in whatever code exists now it should be rather simple to emit  
> > the nodes of Nodes.py rather than Cython code strings, which should  
> > give a performance boost and make things less complex overall.
> 
> Yep, though the main work is that one doesn't start with at tree  
> (especially for (a)). Also, decelerator nodes are such a pain to  
> produce. But this is not justification to not do it the right way.  
> It'd be awesome to have something that takes a .h file and produces a  
> parsed pxd tree.

What pxd you mean ? Generated with regard to classes specified in "..extern.." 
statements or entire .h converted to pxd?

> 
> > As for pyparsing, I note my disagreement with Robert. I don't think  
> > it would hurt to add another dependency as long as it is an  
> > optional for optional features, at least if switching parser is  
> > nontrivial.
> 
> I don't see this as an "optional" feature, but something we should  
> have in the core of Cython. We already ship with a parser, so if  
> possible I'd like to use that one.
> 
> - Robert
> 

This is realy not a great point, what parser to use, because I've written very 
raw program and used most simply tool. Let teach PLEX to handle .h and use it.
I've tried not to interfere in cython code at start.



From ellisonbg.net at gmail.com  Wed Oct 22 19:46:38 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 22 Oct 2008 10:46:38 -0700
Subject: [Cython] How do I write C++ shadow classes..
In-Reply-To: <loom.20081022T163936-465@post.gmane.org>
References: <3307299022.571604@smtp.netcom.no>
	<CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>
	<48FF4F3A.3030509@student.matnat.uio.no>
	<loom.20081022T163936-465@post.gmane.org>
Message-ID: <6ce0ac130810221046i127dfdbx7c13860c4dcdb0b4@mail.gmail.com>

This is sort of a separate issue than these others....

I have spent a good amount of time hand wrapping C++ classes using
cython.  The big thing that you quickly run into is the difficulty in
handling inheritance.  More specifically, you would like your c++
class hierarchy to be reflected and mirrored in the Cython class
hierarchy.

Have you looked into these issues yet?  It has been a few months, so
my mind is not fresh on this stuff, but it was quite subtle and I
don't think I ever got to a point of having a solution that we really
satisfactory.  What I do remember:

* There were problems getting superclass methods in Python callable by
sub classes because the pointer types were different.

* Performing dynamic memory allocations, etc. in __cinit__ is
difficult because subclasses might need to do this differently.  For
example a subclass would want to hold a pointer to the C++ subclass,
not a pointer to the subclass.  The calling conventions of __cinit__
in Cython make this really tough to handle this.

I am curious if you or others have made progress in figuring this stuff out?

Cheers,

Brian

From robertwb at math.washington.edu  Wed Oct 22 20:06:07 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Oct 2008 11:06:07 -0700
Subject: [Cython] How do I write C++ shadow classes..
In-Reply-To: <6ce0ac130810221046i127dfdbx7c13860c4dcdb0b4@mail.gmail.com>
References: <3307299022.571604@smtp.netcom.no>
	<CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>
	<48FF4F3A.3030509@student.matnat.uio.no>
	<loom.20081022T163936-465@post.gmane.org>
	<6ce0ac130810221046i127dfdbx7c13860c4dcdb0b4@mail.gmail.com>
Message-ID: <35B71FA9-9573-4B26-9EE8-4C3A990C9DD1@math.washington.edu>

On Oct 22, 2008, at 10:46 AM, Brian Granger wrote:

> This is sort of a separate issue than these others....
>
> I have spent a good amount of time hand wrapping C++ classes using
> cython.  The big thing that you quickly run into is the difficulty in
> handling inheritance.  More specifically, you would like your c++
> class hierarchy to be reflected and mirrored in the Cython class
> hierarchy.
>
> Have you looked into these issues yet?  It has been a few months, so
> my mind is not fresh on this stuff, but it was quite subtle and I
> don't think I ever got to a point of having a solution that we really
> satisfactory.

This is a very good issue to bring up, and though I don't see us  
coming up with a good full-blown solution right away, whatever we do  
end up doing should be extensible to this kind of thing in the future.

> What I do remember:
>
> * There were problems getting superclass methods in Python callable by
> sub classes because the pointer types were different.

On the technical side, the infrastructure is there to say "this  
(pointer) type is a subtype of that" but no way yet to expose it to  
Cython.

> * Performing dynamic memory allocations, etc. in __cinit__ is
> difficult because subclasses might need to do this differently.  For
> example a subclass would want to hold a pointer to the C++ subclass,
> not a pointer to the subclass.  The calling conventions of __cinit__
> in Cython make this really tough to handle this.
>
> I am curious if you or others have made progress in figuring this  
> stuff out?

I personally don't know enough C++ to know the ins and outs of all of  
this, but certainly things could be made easier. If I remember right,  
for some of the C++ wrappings in Sage (e.g. NTL) we were able to  
store the class data itself as part of the struct (rather than  
requiring a second alloc) but this obviously wouldn't play well with  
subclasses (which happened to be less important than speed for that  
particular use case).

- Robert


From org.andrey at gmail.com  Wed Oct 22 20:07:24 2008
From: org.andrey at gmail.com (Andrey Ivanov)
Date: Wed, 22 Oct 2008 18:07:24 +0000 (UTC)
Subject: [Cython] How do I write C++ shadow classes..
References: <3307299022.571604@smtp.netcom.no>
Message-ID: <loom.20081022T174545-230@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:

> 
> Some random thoughts on this:
> 
> This is basically two orthogonal things (as given in two seperaate examples):
> a) Automatic inclusion of .h-files "as if they were pxd files" (or, making 
pxds unecessary). This has very
> wide uses.
> 
> b) Automatic generation of wrappers for cpp classes.
> 
> I think a) has a very wide circle of potential users, and perhaps getting 
this integrated first could be a
> first goal here?
> 

I realy think that b) can be expanded, to "Automatic generation of wrappers to 
anything from C/C++", my humble vision of such tools like cython is to be very 
transparent for base language and to give a feature not to declare things twice 
in cython and C/C++ code. I know that the cython is not just a tool but a 
language, but it must be really helpful in all applications even in simple ones.



From stefan_ml at behnel.de  Wed Oct 22 20:27:19 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 22 Oct 2008 20:27:19 +0200
Subject: [Cython] How do I write C++ shadow classes..
In-Reply-To: <loom.20081022T163936-465@post.gmane.org>
References: <3307299022.571604@smtp.netcom.no>	<CE527AAE-99EF-4B0E-98E5-AE4412A99269@math.washington.edu>	<48FF4F3A.3030509@student.matnat.uio.no>
	<loom.20081022T163936-465@post.gmane.org>
Message-ID: <48FF7087.6020501@behnel.de>

Hi,

Andrey Ivanov wrote:
> I use pyparsing, because I don't know how PLEX works and how to use it 
> with C/C++ sources. My program is written very crudely, firstly because of lack 
> of skills with writing compilers and such things, secondly just because it 
> already suits my needs.

You should never underestimate the power of an incomplete script released to
public. :)

Stefan

From dalcinl at gmail.com  Wed Oct 22 23:31:54 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 22 Oct 2008 18:31:54 -0300
Subject: [Cython] some __Pyx_XXX functions are not emited as 'static'
Message-ID: <e7ba66e40810221431p10cda250j81311d728fc36e2e@mail.gmail.com>

After running 'nm' in my petsc4py extension module, I can see the
following extenal symbols:

$ nm  PETSc.so -D --defined-only
....
00021590 T __Pyx_ErrFetch
000214e0 T __Pyx_ErrRestore
00021650 T __Pyx_ExceptionReset
000215e0 T __Pyx_ExceptionSave
....

IMHO, those symbols should not be exported (I mean, we should emit
them as 'static' in the generated C code). Python normally imports
extension modules with RTLD_LOCAL, but... if some other reason you
have to hack things using sys.setdlopenflags(), or directly open the
extension module with 'ctypes' (I believe it uses RTLD_GLOBAL by
default), then this could cause symbol resolution clashes.

If there are no objections, I'll fix this.  Hope you do not think I'm
being too much pedantic ;-).

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jek-gmane at kleckner.net  Wed Oct 22 23:33:46 2008
From: jek-gmane at kleckner.net (Jim Kleckner)
Date: Wed, 22 Oct 2008 14:33:46 -0700
Subject: [Cython] doctest wraps stdout so file pointers can't convert
In-Reply-To: <e7ba66e40810212327v793e707bs91dd4afdc715f2e4@mail.gmail.com>
References: <gdm6mo$rec$1@ger.gmane.org> <48FEBA8B.60306@canterbury.ac.nz>
	<e7ba66e40810212327v793e707bs91dd4afdc715f2e4@mail.gmail.com>
Message-ID: <gdo67q$rt5$1@ger.gmane.org>

Lisandro Dalcin wrote:
> On Wed, Oct 22, 2008 at 2:30 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>> Jim Kleckner wrote:
>>
>>>    TypeError: Cannot convert instance to file
>>>
>>> Ideas?  At the moment, I just forgo the convenience of turning on
>>> that debug within doctest.
>
> Well, use 'sys.__stdout__' .  That should always be a 'file' instance
> actually associated to standard output.

Thanks for the pointer.

>> You could write your output the Pythonic way instead of
>> trying to obtain a file descriptor.

> Greg, It seems Jim is calling some testing code in pure C and wrapped
> with Cython, and it seems such code need FILE* pointer. What are you
> suggesting here? I'm missing something?

You are both right.

The C code should probably be upgraded to Cython when I get a round tuit.


From robertwb at math.washington.edu  Thu Oct 23 00:05:46 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Oct 2008 15:05:46 -0700
Subject: [Cython] some __Pyx_XXX functions are not emited as 'static'
In-Reply-To: <e7ba66e40810221431p10cda250j81311d728fc36e2e@mail.gmail.com>
References: <e7ba66e40810221431p10cda250j81311d728fc36e2e@mail.gmail.com>
Message-ID: <A84D8371-2C47-410A-9ED2-AD9648887E86@math.washington.edu>

I ran into this too. See http://hg.cython.org/cython-devel/rev/ 
014bf971cfe1

On Oct 22, 2008, at 2:31 PM, Lisandro Dalcin wrote:

> After running 'nm' in my petsc4py extension module, I can see the
> following extenal symbols:
>
> $ nm  PETSc.so -D --defined-only
> ....
> 00021590 T __Pyx_ErrFetch
> 000214e0 T __Pyx_ErrRestore
> 00021650 T __Pyx_ExceptionReset
> 000215e0 T __Pyx_ExceptionSave
> ....
>
> IMHO, those symbols should not be exported (I mean, we should emit
> them as 'static' in the generated C code). Python normally imports
> extension modules with RTLD_LOCAL, but... if some other reason you
> have to hack things using sys.setdlopenflags(), or directly open the
> extension module with 'ctypes' (I believe it uses RTLD_GLOBAL by
> default), then this could cause symbol resolution clashes.
>
> If there are no objections, I'll fix this.  Hope you do not think I'm
> being too much pedantic ;-).
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From aaron.devore at gmail.com  Thu Oct 23 01:41:44 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Wed, 22 Oct 2008 16:41:44 -0700
Subject: [Cython] Using PySet_* with a sets fallback
Message-ID: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>

I have an application that could benefit from using sets. Currently
I'm using lists but I'd like to avoid linear time when I can get
constant time. The set type is the best option for me but my
application has to also work with Python versions 2.3-2.5. I have seen
people use this trick for backward compatibility (---- is used for
indentation):

cdef object set
try:
----set = __builtins__.set
except AttributeError:
----from sets import Set as set

When using this trick will Cython still use PySet_* in the Python API
where available?

As a side note, how about making an adjustment to Cython that
automatically handles the set type/sets module trick?

-Aaron

From greg.ewing at canterbury.ac.nz  Thu Oct 23 02:32:22 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Oct 2008 13:32:22 +1300
Subject: [Cython] pxd packages
In-Reply-To: <20081020174827.GA1570@encolpuis>
References: <20081020174827.GA1570@encolpuis>
Message-ID: <48FFC616.5040902@canterbury.ac.nz>

Gabriel Gellner wrote:

> Suppose I want the following structure:
> 
>     foo/
>        math.pxd
>        param.pxd
> 
> So that I can do
> 
>     cimport foo
> 
>     foo.math.func()
>     foo.param.PI

You should be able to do that by putting an __init__.py
into foo, as long as the directory containing foo is on
Cython's include path.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Oct 23 02:34:38 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Oct 2008 13:34:38 +1300
Subject: [Cython] pxd packages
In-Reply-To: <8DDFD3F6-E314-4847-9589-8267F3DA699D@math.washington.edu>
References: <20081020174827.GA1570@encolpuis>
	<8DDFD3F6-E314-4847-9589-8267F3DA699D@math.washington.edu>
Message-ID: <48FFC69E.50609@canterbury.ac.nz>

Robert Bradshaw wrote:

> I'm not sure, but I know they've changed over time. It used to be  
> that you had to sit everything in the same folder, and just name the  
> files "a.b.c" rather than putting stuff in folders.

Not any more -- Pyrex will recognize a directory as a package
as long as it contains an __init__.py or __init__.pyx file.

-- 
Greg

From ggellner at uoguelph.ca  Thu Oct 23 03:54:32 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Wed, 22 Oct 2008 21:54:32 -0400
Subject: [Cython] pxd packages
In-Reply-To: <48FFC69E.50609@canterbury.ac.nz>
References: <20081020174827.GA1570@encolpuis>
	<8DDFD3F6-E314-4847-9589-8267F3DA699D@math.washington.edu>
	<48FFC69E.50609@canterbury.ac.nz>
Message-ID: <20081023015432.GA17099@encolpuis>

On Thu, Oct 23, 2008 at 01:34:38PM +1300, Greg Ewing wrote:
> Robert Bradshaw wrote:
> 
> > I'm not sure, but I know they've changed over time. It used to be  
> > that you had to sit everything in the same folder, and just name the  
> > files "a.b.c" rather than putting stuff in folders.
> 
> Not any more -- Pyrex will recognize a directory as a package
> as long as it contains an __init__.py or __init__.pyx file.
> 
Great, I will update the docs to reflect this.
Thanks.

Gabriel

From nbspcorp at loper.org  Thu Oct 23 10:12:53 2008
From: nbspcorp at loper.org (Jackson Hoy Loper)
Date: Thu, 23 Oct 2008 04:12:53 -0400
Subject: [Cython] C++ issue
Message-ID: <e474c0c30810230112pb28cb9bo3a9dd22d39fed1e@mail.gmail.com>

stl iterators are still pretty much useless in cython, which seems
rather unfortunate.

one Ravi Lanka brought up the basic issue this april... it's a matter
of initializing classes without default constructors.  as far as i
know the issue is still unresolved (despite Greg Ewing's comment; see
below).  do we expect this guy to get solved any time soon?  it may
seem trivial, but scipy still uses STL, and unordered_map is the best
hash table support around right now...

thoughts:

   1) iterators don't have a constructor that takes no arguments, so
you can't ever say

  cdef STLiterator k # no constructor can be found

   2) cython balks at taking the address of temporaries, so you can't
use pointers...

  cdef STLiterator *k
  k=&(mySTLcontainer.begin()) # can't take address of non-lvalue

   3) I see that Greg Ewing proposed declaring functions like
myhash.begin() "as though" they returned pointers, but I cannot
understand what this means.  The function does not return a pointer,
and c++ notes this with due gravitas.  Unless you have control over
the source code or want to write a shim, this will not work.  It's a
dangerous idea, anyway, if I understand it correctly.  It's a tragedy
that not everything is a pointer in this world, but it is also the
cold, cold reality... (until c++ tr1 goes through, anyway...)

I'd be happy to take this on, as far as the code goes (though I
confess I haven't looked at the source yet).  I'm not sure how
controversial this is, but I'm personally pretty clear that we should
put our declarations in the c code wherever they're declared in the
cython code.  there's a pretty deep sense in which this is the "right"
answer.  mixed code & declarations is officially kosher in C, too, as
of ISO C99. it's true, there are a few oldschool makefiles out there
compiling with -ansi -pedantic-errors, but -std=gnu99 has now nearly
tripled -ansi -pedantic-errors on google.  And most of the hits for
the latter are about how it's causing everybody trouble and they're
going to drop it.  I could well be wrong, but I really don't think
it's an issue, especially with the prevalence of distutils.

I will of course defer to your opinions on this matter.

Obliged,
 Hoy Loper

From dagss at student.matnat.uio.no  Thu Oct 23 11:20:36 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Oct 2008 11:20:36 +0200
Subject: [Cython] C++ issue
In-Reply-To: <e474c0c30810230112pb28cb9bo3a9dd22d39fed1e@mail.gmail.com>
References: <e474c0c30810230112pb28cb9bo3a9dd22d39fed1e@mail.gmail.com>
Message-ID: <490041E4.6040600@student.matnat.uio.no>

Jackson Hoy Loper wrote:
> stl iterators are still pretty much useless in cython, which seems
> rather unfortunate.
>
> one Ravi Lanka brought up the basic issue this april... it's a matter
> of initializing classes without default constructors.  as far as i
> know the issue is still unresolved (despite Greg Ewing's comment; see
> below).  do we expect this guy to get solved any time soon?  it may
> seem trivial, but scipy still uses STL, and unordered_map is the best
> hash table support around right now...
>
> thoughts:
>
>    1) iterators don't have a constructor that takes no arguments, so
> you can't ever say
>
>   cdef STLiterator k # no constructor can be found
>
>    2) cython balks at taking the address of temporaries, so you can't
> use pointers...
>
>   cdef STLiterator *k
>   k=&(mySTLcontainer.begin()) # can't take address of non-lvalue
>
>    3) I see that Greg Ewing proposed declaring functions like
> myhash.begin() "as though" they returned pointers, but I cannot
> understand what this means.  The function does not return a pointer,
> and c++ notes this with due gravitas.  Unless you have control over
> the source code or want to write a shim, this will not work.  It's a
> dangerous idea, anyway, if I understand it correctly.  It's a tragedy
> that not everything is a pointer in this world, but it is also the
> cold, cold reality... (until c++ tr1 goes through, anyway...)
>
> I'd be happy to take this on, as far as the code goes (though I
> confess I haven't looked at the source yet).  I'm not sure how
> controversial this is, but I'm personally pretty clear that we should
> put our declarations in the c code wherever they're declared in the
> cython code.  there's a pretty deep sense in which this is the "right"
> answer.  mixed code & declarations is officially kosher in C, too, as
> of ISO C99. it's true, there are a few oldschool makefiles out there
> compiling with -ansi -pedantic-errors, but -std=gnu99 has now nearly
> tripled -ansi -pedantic-errors on google.  And most of the hits for
> the latter are about how it's causing everybody trouble and they're
> going to drop it.  I could well be wrong, but I really don't think
> it's an issue, especially with the prevalence of distutils.
>   
Hi,

better C++ support is something everybody wants, I think, it just isn't 
high enough on the priority list of the usual list of Cython 
contributors. So if you are willing to take it on as far as code goes 
I'm sure everybody is willing to see if something can be sorted out.

I think the mixed code & declarations issue as such is pretty easy to 
handle (in the sense of conflicting with existing codebases) with a 
compiler directive/option. There already is a --cplus option for Cython 
which could turn this on.

So a first step could thus be to start initializing variables at the 
point of the "cdef" statement if --cplus is turned on. I'm +1 for this 
myself.

There is a related issue though that worries me: Cython doesn't have 
block variable scoping, so syntax like this:

def f(...):
    if should_process:
        cdef myiterator t = items.begin()
        ... process items
        ...

isn't allowed, and that means that writing STL code will be a bit 
cumbersome even if this was supported. To some degree this can be 
mitigated by inner functions though (once those are supported):

def f(...):
    def process_items():
        cdef myiterator t = items.begin()
        ...

    if should_process:
        process_items()


Dag Sverre



From dagss at student.matnat.uio.no  Thu Oct 23 11:25:25 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Oct 2008 11:25:25 +0200
Subject: [Cython] C++ issue
In-Reply-To: <490041E4.6040600@student.matnat.uio.no>
References: <e474c0c30810230112pb28cb9bo3a9dd22d39fed1e@mail.gmail.com>
	<490041E4.6040600@student.matnat.uio.no>
Message-ID: <49004305.7010607@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Jackson Hoy Loper wrote:
>   
>> stl iterators are still pretty much useless in cython, which seems
>> rather unfortunate.
>>
>> one Ravi Lanka brought up the basic issue this april... it's a matter
>> of initializing classes without default constructors.  as far as i
>> know the issue is still unresolved (despite Greg Ewing's comment; see
>> below).  do we expect this guy to get solved any time soon?  it may
>> seem trivial, but scipy still uses STL, and unordered_map is the best
>> hash table support around right now...
>>
>> thoughts:
>>
>>    1) iterators don't have a constructor that takes no arguments, so
>> you can't ever say
>>
>>   cdef STLiterator k # no constructor can be found
>>
>>    2) cython balks at taking the address of temporaries, so you can't
>> use pointers...
>>
>>   cdef STLiterator *k
>>   k=&(mySTLcontainer.begin()) # can't take address of non-lvalue
>>
>>    3) I see that Greg Ewing proposed declaring functions like
>> myhash.begin() "as though" they returned pointers, but I cannot
>> understand what this means.  The function does not return a pointer,
>> and c++ notes this with due gravitas.  Unless you have control over
>> the source code or want to write a shim, this will not work.  It's a
>> dangerous idea, anyway, if I understand it correctly.  It's a tragedy
>> that not everything is a pointer in this world, but it is also the
>> cold, cold reality... (until c++ tr1 goes through, anyway...)
>>
>> I'd be happy to take this on, as far as the code goes (though I
>> confess I haven't looked at the source yet).  I'm not sure how
>> controversial this is, but I'm personally pretty clear that we should
>> put our declarations in the c code wherever they're declared in the
>> cython code.  there's a pretty deep sense in which this is the "right"
>> answer.  mixed code & declarations is officially kosher in C, too, as
>> of ISO C99. it's true, there are a few oldschool makefiles out there
>> compiling with -ansi -pedantic-errors, but -std=gnu99 has now nearly
>> tripled -ansi -pedantic-errors on google.  And most of the hits for
>> the latter are about how it's causing everybody trouble and they're
>> going to drop it.  I could well be wrong, but I really don't think
>> it's an issue, especially with the prevalence of distutils.
>>   
>>     
> Hi,
>
> better C++ support is something everybody wants, I think, it just isn't 
> high enough on the priority list of the usual list of Cython 
> contributors. So if you are willing to take it on as far as code goes 
> I'm sure everybody is willing to see if something can be sorted out.
>
> I think the mixed code & declarations issue as such is pretty easy to 
> handle (in the sense of conflicting with existing codebases) with a 
> compiler directive/option. There already is a --cplus option for Cython 
> which could turn this on.
>
> So a first step could thus be to start initializing variables at the 
> point of the "cdef" statement if --cplus is turned on. I'm +1 for this 
> myself.
>   
I meant "...could be to start _declaring_ variables at the point of...". 
Kind of important to get right in this context :-)

Dag Sverre

From stefan_ml at behnel.de  Thu Oct 23 12:08:36 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Oct 2008 12:08:36 +0200 (CEST)
Subject: [Cython] Using PySet_* with a sets fallback
In-Reply-To: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>
References: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>
Message-ID: <42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Aaron DeVore wrote:
> I have an application that could benefit from using sets. Currently
> I'm using lists but I'd like to avoid linear time when I can get
> constant time. The set type is the best option for me but my
> application has to also work with Python versions 2.3-2.5. I have seen
> people use this trick for backward compatibility (---- is used for
> indentation):
>
> cdef object set
> try:
> ----set = __builtins__.set
> except AttributeError:
> ----from sets import Set as set
>
> When using this trick will Cython still use PySet_* in the Python API
> where available?

It can't use PySet_* anyway, as they are not available in Py2.3, and
Cython code currently runs with 2.3. Although I wouldn't mind restricting
the C code to Py2.4 if "set" is not redefined in the source (maybe that's
already the case, haven't checked).


> As a side note, how about making an adjustment to Cython that
> automatically handles the set type/sets module trick?

Yes, it would be nice to have that. The idea would be to do the Set import
internally, to provide replacement implementations for the generated
PySet_*() calls that work at the Python level, and to enable that
machinery when the C compile-time Python version is Py2.3. However, that's
work that someone has to do.

OTOH, this is a pure legacy feature for code that must run with Py2.3. I
guess people (including myself) can currently live with the little
performance impact regarding the set operations (which are fast anyway),
given the gain that their code runs unchanged in Py2.3.

Stefan


From stefan_ml at behnel.de  Thu Oct 23 12:10:17 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Oct 2008 12:10:17 +0200 (CEST)
Subject: [Cython] some __Pyx_XXX functions are not emited as 'static'
In-Reply-To: <e7ba66e40810221431p10cda250j81311d728fc36e2e@mail.gmail.com>
References: <e7ba66e40810221431p10cda250j81311d728fc36e2e@mail.gmail.com>
Message-ID: <44721.213.61.181.86.1224756617.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> After running 'nm' in my petsc4py extension module, I can see the
> following extenal symbols:
>
> $ nm  PETSc.so -D --defined-only
> ....
> 00021590 T __Pyx_ErrFetch
> 000214e0 T __Pyx_ErrRestore
> 00021650 T __Pyx_ExceptionReset
> 000215e0 T __Pyx_ExceptionSave

My bad, sorry. Thanks for fixing this, Robert.

Stefan


From stefan_ml at behnel.de  Thu Oct 23 12:15:36 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 23 Oct 2008 12:15:36 +0200 (CEST)
Subject: [Cython] doctest wraps stdout so file pointers can't convert
In-Reply-To: <gdm6mo$rec$1@ger.gmane.org>
References: <gdm6mo$rec$1@ger.gmane.org>
Message-ID: <51171.213.61.181.86.1224756936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Hi,

Jim Kleckner wrote:
> I ran across an annoyance and wondered if anyone else hit this.
>
> I have some compiled C code wrapped with Cython that writes to
> an ordinary file descriptor when the debug level is high enough.
> It directly references sys.stdout and sends output there.  When
> using a doctest, however, that object is replaced with _SpoofOut
> that subclasses StringIO.  Thus, when executing something like the
> following in a doctest:
>
>    ...
>    cdef file outFile = sys.stdout  # This fails for the spoofed output
>    callDebugCodeInC(PyFile_AsFile(outFile))
>    ...
>
> It (understandably) fails on the assignment with a message:
>    TypeError: Cannot convert instance to file

Note that PyFile_AsFile() was dropped in Py3 anyway, so your code will not
be future-proof if you use this.

Stefan


From greg.ewing at canterbury.ac.nz  Thu Oct 23 12:12:54 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Oct 2008 23:12:54 +1300
Subject: [Cython] C++ issue
In-Reply-To: <e474c0c30810230112pb28cb9bo3a9dd22d39fed1e@mail.gmail.com>
References: <e474c0c30810230112pb28cb9bo3a9dd22d39fed1e@mail.gmail.com>
Message-ID: <49004E26.50101@canterbury.ac.nz>

Jackson Hoy Loper wrote:

>    3) I see that Greg Ewing proposed declaring functions like
> myhash.begin() "as though" they returned pointers, but I cannot
> understand what this means.

I can't remember what context I made that statement in,
so I'm not sure what it means either. I may have been
talking about dealing with references by treating them
as pointers -- but in code that's compiled as *C* code,
not C++.

As for things like C++ iterators, you can use them as
long as you're willing to heap-allocate them and explicitly
free them afterwards, via the trick of declaring a function
whose cname is "new Foo" or whatever.

-- 
Greg

From marcin.cieslik at gmail.com  Thu Oct 23 14:53:22 2008
From: marcin.cieslik at gmail.com (=?UTF-8?Q?Marcin_Cie=C5=9Blik?=)
Date: Thu, 23 Oct 2008 08:53:22 -0400
Subject: [Cython] pxd packages
In-Reply-To: <48FFC616.5040902@canterbury.ac.nz>
References: <20081020174827.GA1570@encolpuis>
	<48FFC616.5040902@canterbury.ac.nz>
Message-ID: <3f9ed1a70810230553s4e4cf91fu2fd6a4b1586226b1@mail.gmail.com>

> You should be able to do that by putting an __init__.py
> into foo, as long as the directory containing foo is on
> Cython's include path.


I will only add that I had mysterious (run-time) memory corruptions if it
was not the case. What I do is I keep the pyx/pxd/h files seperate from any
__init__.py and copy the result .so to the base dir.

Yours,
Marcin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20081023/3ad1b564/attachment.htm 

From dalcinl at gmail.com  Thu Oct 23 15:13:33 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Oct 2008 10:13:33 -0300
Subject: [Cython] some __Pyx_XXX functions are not emited as 'static'
In-Reply-To: <44721.213.61.181.86.1224756617.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40810221431p10cda250j81311d728fc36e2e@mail.gmail.com>
	<44721.213.61.181.86.1224756617.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40810230613i7a42d845g36ee32bd9af4982a@mail.gmail.com>

BTW, some of these routines are emited as

static void INLINE __Pyx_ .....

For the sake of consistensy, I'll change that to (note the INLINE position)

static INLINE void __Pyx_ .....



On Thu, Oct 23, 2008 at 7:10 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin wrote:
>> After running 'nm' in my petsc4py extension module, I can see the
>> following extenal symbols:
>>
>> $ nm  PETSc.so -D --defined-only
>> ....
>> 00021590 T __Pyx_ErrFetch
>> 000214e0 T __Pyx_ErrRestore
>> 00021650 T __Pyx_ExceptionReset
>> 000215e0 T __Pyx_ExceptionSave
>
> My bad, sorry. Thanks for fixing this, Robert.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ggellner at uoguelph.ca  Thu Oct 23 16:05:26 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 23 Oct 2008 10:05:26 -0400
Subject: [Cython] pxd packages
In-Reply-To: <48FFC616.5040902@canterbury.ac.nz>
References: <20081020174827.GA1570@encolpuis>
	<48FFC616.5040902@canterbury.ac.nz>
Message-ID: <20081023140526.GA21090@encolpuis>

On Thu, Oct 23, 2008 at 01:32:22PM +1300, Greg Ewing wrote:
> Gabriel Gellner wrote:
>
>> Suppose I want the following structure:
>>
>>     foo/
>>        math.pxd
>>        param.pxd
>>
>> So that I can do
>>
>>     cimport foo
>>
>>     foo.math.func()
>>     foo.param.PI
>
> You should be able to do that by putting an __init__.py
> into foo, as long as the directory containing foo is on
> Cython's include path.
>
When I do this I get an error about finding foo.pxd. (Note: I understand
doing: cimport foo.math and then being able to use foo.math.func, but I can't
seem to figure out if it is possible to just import the package name (which
has a __init__.py))

thanks, 
Gabriel

From dagss at student.matnat.uio.no  Thu Oct 23 16:08:41 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Oct 2008 16:08:41 +0200
Subject: [Cython] pxd packages
In-Reply-To: <20081023140526.GA21090@encolpuis>
References: <20081020174827.GA1570@encolpuis>	<48FFC616.5040902@canterbury.ac.nz>
	<20081023140526.GA21090@encolpuis>
Message-ID: <49008569.8010306@student.matnat.uio.no>

Gabriel Gellner wrote:
> On Thu, Oct 23, 2008 at 01:32:22PM +1300, Greg Ewing wrote:
>   
>> Gabriel Gellner wrote:
>>
>>     
>>> Suppose I want the following structure:
>>>
>>>     foo/
>>>        math.pxd
>>>        param.pxd
>>>
>>> So that I can do
>>>
>>>     cimport foo
>>>
>>>     foo.math.func()
>>>     foo.param.PI
>>>       
>> You should be able to do that by putting an __init__.py
>> into foo, as long as the directory containing foo is on
>> Cython's include path.
>>
>>     
> When I do this I get an error about finding foo.pxd. (Note: I understand
> doing: cimport foo.math and then being able to use foo.math.func, but I can't
> seem to figure out if it is possible to just import the package name (which
> has a __init__.py))
>   
I think there simply is a difference between Pyrex and Cython here -- 
from what (respectively) Greg and Robert says, it seems like Pyrex 
supports this but not Cython.

Dag Sverre

From ggellner at uoguelph.ca  Thu Oct 23 16:37:52 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 23 Oct 2008 10:37:52 -0400
Subject: [Cython] pxd packages
In-Reply-To: <49008569.8010306@student.matnat.uio.no>
References: <20081020174827.GA1570@encolpuis>
	<48FFC616.5040902@canterbury.ac.nz>
	<20081023140526.GA21090@encolpuis>
	<49008569.8010306@student.matnat.uio.no>
Message-ID: <20081023143752.GA21226@encolpuis>

On Thu, Oct 23, 2008 at 04:08:41PM +0200, Dag Sverre Seljebotn wrote:
> Gabriel Gellner wrote:
> > On Thu, Oct 23, 2008 at 01:32:22PM +1300, Greg Ewing wrote:
> >   
> >> Gabriel Gellner wrote:
> >>
> >>     
> >>> Suppose I want the following structure:
> >>>
> >>>     foo/
> >>>        math.pxd
> >>>        param.pxd
> >>>
> >>> So that I can do
> >>>
> >>>     cimport foo
> >>>
> >>>     foo.math.func()
> >>>     foo.param.PI
> >>>       
> >> You should be able to do that by putting an __init__.py
> >> into foo, as long as the directory containing foo is on
> >> Cython's include path.
> >>
> >>     
> > When I do this I get an error about finding foo.pxd. (Note: I understand
> > doing: cimport foo.math and then being able to use foo.math.func, but I can't
> > seem to figure out if it is possible to just import the package name (which
> > has a __init__.py))
> >   
> I think there simply is a difference between Pyrex and Cython here -- 
> from what (respectively) Greg and Robert says, it seems like Pyrex 
> supports this but not Cython.
> 
Sorry I should have specified, I get this with both Cython and Pyrex, but if
I am just missing something and Pyrex can do this, I will try to port it.

Gabriel

From mistobaan at gmail.com  Thu Oct 23 17:19:00 2008
From: mistobaan at gmail.com (Mistobaan)
Date: Thu, 23 Oct 2008 08:19:00 -0700
Subject: [Cython] Pxd Packages Libraries
Message-ID: <2E7A462B-D367-4403-A770-88D5996B02C0@gmail.com>

Hi guys,

I am starting noticing how many of the well know C-libraries are  
becoming to be wrapped using cython.

Is there any effort to put all this cython pxd libraries together?

Cheers

Fabrizio


From dagss at student.matnat.uio.no  Thu Oct 23 18:10:45 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 23 Oct 2008 18:10:45 +0200
Subject: [Cython] Pxd Packages Libraries
In-Reply-To: <2E7A462B-D367-4403-A770-88D5996B02C0@gmail.com>
References: <2E7A462B-D367-4403-A770-88D5996B02C0@gmail.com>
Message-ID: <4900A205.4070702@student.matnat.uio.no>

Mistobaan wrote:
> Hi guys,
> 
> I am starting noticing how many of the well know C-libraries are  
> becoming to be wrapped using cython.
> 
> Is there any effort to put all this cython pxd libraries together?

Cython/Includes in the main repo is always in the cimport path for a 
given Cython release -- numpy.pxd is already shipped that way, and I 
suppose it is just a matter of someone requesting that a library is 
added. Perhaps Robert should post a "Call for pxds" email?

If the release cycle is too slow for this, perhaps you could create a 
wiki page with the different pxds attached? "Please add your pxd here". 
New pxds attached to the page could then added to the next release of 
Cython (as Python already comes with "batteries included", it makes 
sense that Cython does the same?)

-- 
Dag Sverre

From ggellner at uoguelph.ca  Thu Oct 23 22:40:47 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Thu, 23 Oct 2008 16:40:47 -0400
Subject: [Cython] pxd packages
In-Reply-To: <8DDFD3F6-E314-4847-9589-8267F3DA699D@math.washington.edu>
References: <20081020174827.GA1570@encolpuis>
	<8DDFD3F6-E314-4847-9589-8267F3DA699D@math.washington.edu>
Message-ID: <20081023204047.GA23333@encolpuis>

On Wed, Oct 22, 2008 at 09:13:36AM -0700, Robert Bradshaw wrote:
> On Oct 20, 2008, at 10:48 AM, Gabriel Gellner wrote:
>
>> So I am trying to get my mind around how the newish pxd packages work,
>> that is when I have a folder with pxd files and an __init__.py file.
>>
>> Suppose I want the following structure:
>>
>>     foo/
>>        math.pxd
>>        param.pxd
>>
>> So that I can do
>>
>>     cimport foo
>>
>>     foo.math.func()
>>     foo.param.PI
>>
>>     # etc
>>
>>
>> Can this be done? (I know that I can do cimport foo.math, but can use 
>> use the
>> __init__.py file in some way to be able to use the pxd files like a  
>> normal
>> package?)
>
> No, I don't think you can do this. In Python, foo.math is an actual  
> object that can be further queried. Perhaps we should support this.
>
>> Secondly, how to I set the include path for pxd packages in setup.py
>>
>> I know I do `cython -I .` etc for doing it by hand. Any tips?
>
> There's an include_dir option you can pass to the setup command.
>
Okay I am still scratching my head on this, using the include_dir option of Extension 
seems to effect gcc not cython, do I give this instead as part of the cmdclass
options?

Thanks for any help with this question . . .

Gabriel

From greg.ewing at canterbury.ac.nz  Fri Oct 24 00:22:44 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 24 Oct 2008 11:22:44 +1300
Subject: [Cython] pxd packages
In-Reply-To: <20081023143752.GA21226@encolpuis>
References: <20081020174827.GA1570@encolpuis>
	<48FFC616.5040902@canterbury.ac.nz> <20081023140526.GA21090@encolpuis>
	<49008569.8010306@student.matnat.uio.no>
	<20081023143752.GA21226@encolpuis>
Message-ID: <4900F934.8000101@canterbury.ac.nz>

Gabriel Gellner wrote:

> Note: I understand
> doing: cimport foo.math and then being able to use foo.math.func, but I can't
> seem to figure out if it is possible to just import the package name

Oh, sorry, I see what you mean now.

No, you can't do that in Pyrex.

Note that you can't do the equivalent thing in Python either.
You have to explicitly import foo.math before using foo.math.func
if foo.math is a module.

It occurs to me that you *should* be able to supply an
__init__.pxd that cimports math into itself, but that's
not supported yet. I'll add it to my list.

-- 
Greg

From nbspcorp at loper.org  Fri Oct 24 01:44:43 2008
From: nbspcorp at loper.org (Jackson Hoy Loper)
Date: Thu, 23 Oct 2008 19:44:43 -0400
Subject: [Cython] C++ issue
In-Reply-To: <e474c0c30810231434q3d59712ek149d946cb660b590@mail.gmail.com>
References: <e474c0c30810231434q3d59712ek149d946cb660b590@mail.gmail.com>
Message-ID: <e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>

On Thu, Oct 23, 2008 at 5:34 PM, Hoy Loper <hloper at gmail.com> wrote:
>> As for things like C++ iterators, you can use them as
>> long as you're willing to heap-allocate them and explicitly
>
> Alas, not strictly true.  STL containers don't return pointers to
> iterators, they return iterators themselves.  And they're not lvalues,
> so you can't take their address.  One sometimes rather needs to be
> able to handle the data itself, rather than a pointer to it.  I know
> it is unpythonic, but that's the way STL is built.  Or possibly I am
> missing something :).
>
> Hoy
>

From greg.ewing at canterbury.ac.nz  Fri Oct 24 06:28:50 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 24 Oct 2008 17:28:50 +1300
Subject: [Cython] C++ issue
In-Reply-To: <e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>
References: <e474c0c30810231434q3d59712ek149d946cb660b590@mail.gmail.com>
	<e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>
Message-ID: <49014F02.5000106@canterbury.ac.nz>

Jackson Hoy Loper wrote:

>>Alas, not strictly true.  STL containers don't return pointers to
>>iterators, they return iterators themselves.

I'm not quite following that. I thought the problem
was dealing with types such that the C++ code

   Foo x;
   x = something;

won't work because Foo requires initialization arguments.
However, it seems to me that wherever you could do

   Foo x(args);

you should equally be able to do

   Foo *x;
   x = new Foo(args);
   ...
   delete x;

However I'm not all that familiar with the STL, so I may
have misunderstood what the problem actually is.

-- 
Greg

From aaron.devore at gmail.com  Fri Oct 24 07:30:19 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Thu, 23 Oct 2008 22:30:19 -0700
Subject: [Cython] Using PySet_* with a sets fallback
In-Reply-To: <42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>
	<42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>

On Thu, Oct 23, 2008 at 3:08 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> It can't use PySet_* anyway, as they are not available in Py2.3, and
> Cython code currently runs with 2.3. Although I wouldn't mind restricting
> the C code to Py2.4 if "set" is not redefined in the source (maybe that's
> already the case, haven't checked).

I just took a peek at the source in cython-dev. I don't really
understand what's happening in the file but Builtin.py seems to have
some support for the set type.

>> As a side note, how about making an adjustment to Cython that
>> automatically handles the set type/sets module trick?
>
> Yes, it would be nice to have that. The idea would be to do the Set import
> internally, to provide replacement implementations for the generated
> PySet_*() calls that work at the Python level, and to enable that
> machinery when the C compile-time Python version is Py2.3. However, that's
> work that someone has to do.
>
> OTOH, this is a pure legacy feature for code that must run with Py2.3. I
> guess people (including myself) can currently live with the little
> performance impact regarding the set operations (which are fast anyway),
> given the gain that their code runs unchanged in Py2.3.

I wish I had the time and expertise for this. :(

Would this work if put at the top of every file (dashes indicate indentation)?
if not hasattr(__builtin__, "set"):
----from sets import Set as set

or this
try:
----set
except NameError:
----from sets import Set as set

It seems like there could be an issue with static typing using the
imported Python class sets.Set (e.g. `cdef set foo`). I don't know
enough about Cython to make a good guess on that one.

-Aaron

From dagss at student.matnat.uio.no  Fri Oct 24 09:33:17 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 24 Oct 2008 09:33:17 +0200
Subject: [Cython] C++ issue
In-Reply-To: <e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>
References: <e474c0c30810231434q3d59712ek149d946cb660b590@mail.gmail.com>
	<e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>
Message-ID: <49017A3D.9010504@student.matnat.uio.no>

Jackson Hoy Loper wrote:
> On Thu, Oct 23, 2008 at 5:34 PM, Hoy Loper <hloper at gmail.com> wrote:
>   
>>> As for things like C++ iterators, you can use them as
>>> long as you're willing to heap-allocate them and explicitly
>>>       
>> Alas, not strictly true.  STL containers don't return pointers to
>> iterators, they return iterators themselves.  And they're not lvalues,
>> so you can't take their address.  One sometimes rather needs to be
>> able to handle the data itself, rather than a pointer to it.  I know
>> it is unpythonic, but that's the way STL is built.  Or possibly I am
>> missing something :).
>>     

Here's a full example:

// Declare at top
std::vector<int>::const_iterator* p_it, * p_end;
...
// Want to loop
p_it = new std::vector<int>::const_iterator(coll.begin()) # Invokes copy 
constructor
p_end = new std::vector<int>::const_iterator(coll.end())
while (*p_it != *p_end) {
    std::cout << **p_it << std::endl;
    (*p_it)++;
}
delete p_it; delete p_end;

Dag Sverre


From stefan_ml at behnel.de  Fri Oct 24 15:22:45 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Oct 2008 15:22:45 +0200
Subject: [Cython] Pxd Packages Libraries
In-Reply-To: <4900A205.4070702@student.matnat.uio.no>
References: <2E7A462B-D367-4403-A770-88D5996B02C0@gmail.com>
	<4900A205.4070702@student.matnat.uio.no>
Message-ID: <4901CC25.3010702@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> I suppose it is just a matter of someone requesting that a library is 
> added. Perhaps Robert should post a "Call for pxds" email?

I don't think this makes sense in general. We can't ship a new copy of a .pxd
file each time a library changes its API. I think it's best to have a library
project maintain the relevant .pxd files, and to distribute them with the
matching version of their releases (or the -dev packages respectively).

The next best solution is to have .pxd maintainers that provide .pxd files as
a separate download apart from the library project itself, and the third best
solution is to push the .pxds into Cython (or it's Wiki) and let them suffer
from bit-rot there.

Stefan

From stefan_ml at behnel.de  Fri Oct 24 15:30:49 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Oct 2008 15:30:49 +0200
Subject: [Cython] Using PySet_* with a sets fallback
In-Reply-To: <2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
References: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>	<42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
Message-ID: <4901CE09.1050902@behnel.de>

Hi,

Aaron DeVore wrote:
> Would this work if put at the top of every file (dashes indicate indentation)?
> if not hasattr(__builtin__, "set"):
> ----from sets import Set as set
> 
> or this
> try:
> ----set
> except NameError:
> ----from sets import Set as set

Just give it a try. :)

None of this can work, as Cython determines at translation time what name
refers to a built-in and what was (re-)defined somewhere. Only built-ins can
be optimised to use straight API calls in the generated C code, everything
else has to go through Python code. In all of the above cases, the name "set"
has been redefined at the global module scope (regardless of any try-except-
or if-blocks), so it is no longer associated with the built-in set type.

Stefan


From dalcinl at gmail.com  Fri Oct 24 17:56:22 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Oct 2008 12:56:22 -0300
Subject: [Cython] Using PySet_* with a sets fallback
In-Reply-To: <2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
References: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>
	<42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
Message-ID: <e7ba66e40810240856r306189c4gf5e92afa2a713db1@mail.gmail.com>

On Fri, Oct 24, 2008 at 2:30 AM, Aaron DeVore <aaron.devore at gmail.com> wrote:
> On Thu, Oct 23, 2008 at 3:08 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> I just took a peek at the source in cython-dev. I don't really
> understand what's happening in the file but Builtin.py seems to have
> some support for the set type.

What's happens is that some support was recently added ;-)

>>> As a side note, how about making an adjustment to Cython that
>>> automatically handles the set type/sets module trick?

Somewhat hard. In Py2.3, sets.Set is Python (i.e. implemented in pure
Python) class, and in Py>2.4 it is an extension (i.e. C-implemented)
class. But perhaps Cython could generate a thin C proxy class for
sets.Set.

>
> or this
> try:
> ----set
> except NameError:
> ----from sets import Set as set

This should work, as long as you use Python 2.3 for runing Cython and
for compliling the generated sources.

> It seems like there could be an issue with static typing using the
> imported Python class sets.Set (e.g. `cdef set foo`). I don't know
> enough about Cython to make a good guess on that one.

And yes, in Python 2.3 you will not be able to do 'cdef set foo'.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From nbspcorp at loper.org  Fri Oct 24 18:42:55 2008
From: nbspcorp at loper.org (Jackson Hoy Loper)
Date: Fri, 24 Oct 2008 12:42:55 -0400
Subject: [Cython] C++ issue
In-Reply-To: <49017A3D.9010504@student.matnat.uio.no>
References: <e474c0c30810231434q3d59712ek149d946cb660b590@mail.gmail.com>
	<e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>
	<49017A3D.9010504@student.matnat.uio.no>
Message-ID: <e474c0c30810240942r61b766bagfb50a5d6937b1547@mail.gmail.com>

lol... good trick!  thanks!

it depends on STL having provided a copy constructor, of course... but
they always do, so life is good.  god, the STL is so weird...

you all have very much made my life better :).  i'd be happy to write
something up on using STL with cython... is there a nice
out-of-the-way place to put something like that?

be well.

h

From dagss at student.matnat.uio.no  Fri Oct 24 19:41:55 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 24 Oct 2008 19:41:55 +0200
Subject: [Cython] C++ issue
In-Reply-To: <e474c0c30810240942r61b766bagfb50a5d6937b1547@mail.gmail.com>
References: <e474c0c30810231434q3d59712ek149d946cb660b590@mail.gmail.com>	<e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>	<49017A3D.9010504@student.matnat.uio.no>
	<e474c0c30810240942r61b766bagfb50a5d6937b1547@mail.gmail.com>
Message-ID: <490208E3.6010405@student.matnat.uio.no>

Jackson Hoy Loper wrote:
> lol... good trick!  thanks!
> 
> it depends on STL having provided a copy constructor, of course... but
> they always do, so life is good.  god, the STL is so weird...

Actually, this is a given -- IIRC,

SomeClass a = something();

is just syntax candy for

SomeClass a(something());

When you call collection.begin(), it actually returns the iterator by 
value, and the only way it is ever going to get into a stack variable is 
by having a copy constructor called.

If you still want to have a look at improving Cython in this area, like 
you indicated initially, adding C++ "new" and "delete" natively to the 
Cython language would be a good place to start, and removing much need 
for some hacks :-) Something like

cimport cython
def f():
     cdef MyClass* pobj = cython.new(MyClass, ctor_arg_1, ctor_arg_2)
     ... do stuff ...
     cython.delete(pobj)

wouldn't be too hard to implement. (Have a look at 
TransformBuiltinMethods in ParseTreeTransforms.py to have a taste. You 
would need to intercept those two methods and replace them by CppNewNode 
and CppDeleteNode which you would need to add in Nodes.py. The "sizeof" 
and "address" functions does similar things; the former takes in a type 
as one of the arguments and the latter returns a pointer, so much of 
what you'd need is contained in those).

> you all have very much made my life better :).  i'd be happy to write
> something up on using STL with cython... is there a nice
> out-of-the-way place to put something like that?

Add a page in the Tips and Tricks section on wiki.cython.org.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Oct 24 16:50:56 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Oct 2008 07:50:56 -0700
Subject: [Cython] Using PySet_* with a sets fallback
In-Reply-To: <2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
References: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>
	<42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
Message-ID: <9A6D8B63-1267-40DF-A18E-618EF570F00B@math.washington.edu>

On Oct 23, 2008, at 10:30 PM, Aaron DeVore wrote:

> On Thu, Oct 23, 2008 at 3:08 AM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>
>> It can't use PySet_* anyway, as they are not available in Py2.3, and
>> Cython code currently runs with 2.3. Although I wouldn't mind  
>> restricting
>> the C code to Py2.4 if "set" is not redefined in the source (maybe  
>> that's
>> already the case, haven't checked).
>
> I just took a peek at the source in cython-dev. I don't really
> understand what's happening in the file but Builtin.py seems to have
> some support for the set type.
>
>>> As a side note, how about making an adjustment to Cython that
>>> automatically handles the set type/sets module trick?
>>
>> Yes, it would be nice to have that. The idea would be to do the  
>> Set import
>> internally, to provide replacement implementations for the generated
>> PySet_*() calls that work at the Python level, and to enable that
>> machinery when the C compile-time Python version is Py2.3.  
>> However, that's
>> work that someone has to do.
>>
>> OTOH, this is a pure legacy feature for code that must run with  
>> Py2.3. I
>> guess people (including myself) can currently live with the little
>> performance impact regarding the set operations (which are fast  
>> anyway),
>> given the gain that their code runs unchanged in Py2.3.
>
> I wish I had the time and expertise for this. :(
>
> Would this work if put at the top of every file (dashes indicate  
> indentation)?
> if not hasattr(__builtin__, "set"):
> ----from sets import Set as set
>
> or this
> try:
> ----set
> except NameError:
> ----from sets import Set as set
>
> It seems like there could be an issue with static typing using the
> imported Python class sets.Set (e.g. `cdef set foo`). I don't know
> enough about Cython to make a good guess on that one.

I think one would have to do

try:
     from __builtin__ import set
except ImportError:
     from sets import Set as set

One couldn't, however, do "cdef set foo" because the symbol set would  
need to be resolved at compile time, whereas the code (as above)  
defers its value to runtime. Consequently the code using the  
PySet_Xxxx api directly wouldn't get generated (which of course only  
works for 2.4+) Hackery could be done in the compiler to emulate the  
builtin set API (via calling the Set module's class), but it should  
be noted that the speed gains we're talking about here for the most  
part are not that large (compared to, say, indexing into a list or  
tuple with an int where there are huge savings).

- Robert


From robertwb at math.washington.edu  Fri Oct 24 17:03:26 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Oct 2008 08:03:26 -0700
Subject: [Cython] Pxd Packages Libraries
In-Reply-To: <4901CC25.3010702@behnel.de>
References: <2E7A462B-D367-4403-A770-88D5996B02C0@gmail.com>
	<4900A205.4070702@student.matnat.uio.no>
	<4901CC25.3010702@behnel.de>
Message-ID: <8B537A59-657C-4015-9C51-CCB44BC4EB13@math.washington.edu>

On Oct 24, 2008, at 6:22 AM, Stefan Behnel wrote:

> Hi,
>
> Dag Sverre Seljebotn wrote:
>> I suppose it is just a matter of someone requesting that a library is
>> added. Perhaps Robert should post a "Call for pxds" email?
>
> I don't think this makes sense in general. We can't ship a new copy  
> of a .pxd
> file each time a library changes its API. I think it's best to have  
> a library
> project maintain the relevant .pxd files, and to distribute them  
> with the
> matching version of their releases (or the -dev packages  
> respectively).

I think it makes sense for some very common, stable libraries, e.g.  
numpy, stl, gmp, and stuff like that. I don't think it makes sense  
for the vast majority of libraries written however.

> The next best solution is to have .pxd maintainers that  
> provide .pxd files as
> a separate download apart from the library project itself, and the  
> third best
> solution is to push the .pxds into Cython (or it's Wiki) and let  
> them suffer
> from bit-rot there.

I really like the wiki idea (would throw up a page now but I can't  
get to the internet at the moment), though to prevent bit-rot dates/ 
versions should be clearly stated. This especially makes sense for  
libraries that may come with the system and/or are most commonly  
obtained via apt-get. However, the most recent .pxd files should be  
shipped with the libraries themselves, probably as part of the  
"python bindings."

Hopefully with parsing of .h files much of the need for this will go  
away, though it can't fulfill all needs and it's a solution now  
rather than pending a future feature.

- Robert



From stefan_ml at behnel.de  Fri Oct 24 20:58:58 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Oct 2008 20:58:58 +0200
Subject: [Cython] C++ issue
In-Reply-To: <490208E3.6010405@student.matnat.uio.no>
References: <e474c0c30810231434q3d59712ek149d946cb660b590@mail.gmail.com>	<e474c0c30810231644r4c5eb441pb5fc8bd8a32cbcdb@mail.gmail.com>	<49017A3D.9010504@student.matnat.uio.no>	<e474c0c30810240942r61b766bagfb50a5d6937b1547@mail.gmail.com>
	<490208E3.6010405@student.matnat.uio.no>
Message-ID: <49021AF2.4060106@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Jackson Hoy Loper wrote:
>> i'd be happy to write
>> something up on using STL with cython... is there a nice
>> out-of-the-way place to put something like that?
> 
> Add a page in the Tips and Tricks section on wiki.cython.org.

*After* reading the dedicated C++ Wiki page.

http://wiki.cython.org/WrappingCPlusPlus

Stefan

From aaron.devore at gmail.com  Fri Oct 24 21:17:57 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Fri, 24 Oct 2008 12:17:57 -0700
Subject: [Cython] Using PySet_* with a sets fallback
In-Reply-To: <9A6D8B63-1267-40DF-A18E-618EF570F00B@math.washington.edu>
References: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>
	<42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
	<9A6D8B63-1267-40DF-A18E-618EF570F00B@math.washington.edu>
Message-ID: <2ead2fb0810241217p6514a6d7j19382ff08572f427@mail.gmail.com>

Could I use a compiler directive to import sets if possible, then use
another compiler directive to pick the type when using cdef? Example:

IF PY_VERSION_HEX < 0x02040000:
----DEF USE_SETS_MODULE 1
----from sets import Set as set

Later in the file...

cdef class Foo:
----IF USE_SETS_MODULE:
--------cdef object member_set
----ELSE:
--------cdef set member_set

I haven't figured out how to import compiler directive constants from
Python.h, hence the lack of me just trying it out on my own.

-Aaron


>>> Yes, it would be nice to have that. The idea would be to do the
>>> Set import
>>> internally, to provide replacement implementations for the generated
>>> PySet_*() calls that work at the Python level, and to enable that
>>> machinery when the C compile-time Python version is Py2.3.
>>> However, that's
>>> work that someone has to do.
>>>
>>> OTOH, this is a pure legacy feature for code that must run with
>>> Py2.3. I
>>> guess people (including myself) can currently live with the little
>>> performance impact regarding the set operations (which are fast
>>> anyway),
>>> given the gain that their code runs unchanged in Py2.3.
>>
>> I wish I had the time and expertise for this. :(
>>
>> Would this work if put at the top of every file (dashes indicate
>> indentation)?
>> if not hasattr(__builtin__, "set"):
>> ----from sets import Set as set
>>
>> or this
>> try:
>> ----set
>> except NameError:
>> ----from sets import Set as set
>>
>> It seems like there could be an issue with static typing using the
>> imported Python class sets.Set (e.g. `cdef set foo`). I don't know
>> enough about Cython to make a good guess on that one.
>
> I think one would have to do
>
> try:
>     from __builtin__ import set
> except ImportError:
>     from sets import Set as set
>
> One couldn't, however, do "cdef set foo" because the symbol set would
> need to be resolved at compile time, whereas the code (as above)
> defers its value to runtime. Consequently the code using the
> PySet_Xxxx api directly wouldn't get generated (which of course only
> works for 2.4+) Hackery could be done in the compiler to emulate the
> builtin set API (via calling the Set module's class), but it should
> be noted that the speed gains we're talking about here for the most
> part are not that large (compared to, say, indexing into a list or
> tuple with an int where there are huge savings).
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From aaron.devore at gmail.com  Fri Oct 24 21:42:08 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Fri, 24 Oct 2008 12:42:08 -0700
Subject: [Cython] Using PySet_* with a sets fallback
In-Reply-To: <2ead2fb0810241217p6514a6d7j19382ff08572f427@mail.gmail.com>
References: <2ead2fb0810221641v15560ea5ne07cac7c69f4b510@mail.gmail.com>
	<42948.213.61.181.86.1224756516.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<2ead2fb0810232230q168f76b7x447adc358c51932c@mail.gmail.com>
	<9A6D8B63-1267-40DF-A18E-618EF570F00B@math.washington.edu>
	<2ead2fb0810241217p6514a6d7j19382ff08572f427@mail.gmail.com>
Message-ID: <2ead2fb0810241242m333d5d73o40c6e659a8eb28da@mail.gmail.com>

Amendment: Could I use a compiler directive to import sets if /needed/.

On Fri, Oct 24, 2008 at 12:17 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
> Could I use a compiler directive to import sets if possible, then use
> another compiler directive to pick the type when using cdef? Example:
>
> IF PY_VERSION_HEX < 0x02040000:
> ----DEF USE_SETS_MODULE 1
> ----from sets import Set as set
>
> Later in the file...
>
> cdef class Foo:
> ----IF USE_SETS_MODULE:
> --------cdef object member_set
> ----ELSE:
> --------cdef set member_set
>
> I haven't figured out how to import compiler directive constants from
> Python.h, hence the lack of me just trying it out on my own.
>
> -Aaron
>
>
>>>> Yes, it would be nice to have that. The idea would be to do the
>>>> Set import
>>>> internally, to provide replacement implementations for the generated
>>>> PySet_*() calls that work at the Python level, and to enable that
>>>> machinery when the C compile-time Python version is Py2.3.
>>>> However, that's
>>>> work that someone has to do.
>>>>
>>>> OTOH, this is a pure legacy feature for code that must run with
>>>> Py2.3. I
>>>> guess people (including myself) can currently live with the little
>>>> performance impact regarding the set operations (which are fast
>>>> anyway),
>>>> given the gain that their code runs unchanged in Py2.3.
>>>
>>> I wish I had the time and expertise for this. :(
>>>
>>> Would this work if put at the top of every file (dashes indicate
>>> indentation)?
>>> if not hasattr(__builtin__, "set"):
>>> ----from sets import Set as set
>>>
>>> or this
>>> try:
>>> ----set
>>> except NameError:
>>> ----from sets import Set as set
>>>
>>> It seems like there could be an issue with static typing using the
>>> imported Python class sets.Set (e.g. `cdef set foo`). I don't know
>>> enough about Cython to make a good guess on that one.
>>
>> I think one would have to do
>>
>> try:
>>     from __builtin__ import set
>> except ImportError:
>>     from sets import Set as set
>>
>> One couldn't, however, do "cdef set foo" because the symbol set would
>> need to be resolved at compile time, whereas the code (as above)
>> defers its value to runtime. Consequently the code using the
>> PySet_Xxxx api directly wouldn't get generated (which of course only
>> works for 2.4+) Hackery could be done in the compiler to emulate the
>> builtin set API (via calling the Set module's class), but it should
>> be noted that the speed gains we're talking about here for the most
>> part are not that large (compared to, say, indexing into a list or
>> tuple with an int where there are huge savings).
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>

From gour at mail.inet.hr  Fri Oct 24 23:33:59 2008
From: gour at mail.inet.hr (Gour)
Date: Fri, 24 Oct 2008 23:33:59 +0200
Subject: [Cython] Is Cython appropriate tool for the task
Message-ID: <87iqrhwud4.fsf@gaura-nitai.no-ip.org>

Hi!

I'm Python noob and got info that SWIG (which I know for quite some time
when playing with Ruby) does not produce best bindings for C libs and
was advised to go with ctype.

However, some research has brought me to Cython and now I'm considering
if it is the right tool for the (potential) task - writing bindings for
Swiss Ephemeris C lib (http://www.astro.com/swisseph/swephprg.htm)

We need it for writing astrological software and although the plan
was/is to use Haskell as the main language, I'm curious if Python with
Cython could provide good-enough performance for the application (GUI
with GTK+, sqlite3 back-end etc.). IOW, Cython could be used not only
for writing Python bindings for C lib, but for some other code as well.

However, first things first, and in order to do any testing, we need
nice Pythonic bindings for C lib which has API looking like follows:

int swe_calc_ut ( double tjd_ut, int ipl, int iflag, double* xx, char* serr),

where

tjd_ut=Julian day, Universal Time

ipl=body number

iflag=a 32 bit integer containing bit flags that indicate what kind of
computation is wanted

xx=array of 6 doubles for longitude, latitude, distance, speed in long.,
speed in lat., and speed in dist.

serr[256] =character string to return error messages in case of error.

In case when return value is < 0, *serr contains error message.

(See http://www.astro.com/swisseph/swephprg.htm#_Toc207617937)

The rest of the API is done in the same way and the whole lib includes
several *.c files and one header swephexp.h.


Anyone can give me some advice whether Cython is good tool for such
task(s)?


p.s. There is already one extension for Swiss Ephemeris lib
(http://pyswisseph.atarax.org/pydoc/index.html) but I believe with
Cython one could produce nicer API.


Sincerely,
Gour

 
-- 

Gour  | Zagreb, Croatia  | GPG key: C6E7162D
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081024/738a8d21/attachment.pgp 

From dalcinl at gmail.com  Sat Oct 25 02:42:21 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Oct 2008 21:42:21 -0300
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3 [Was:
	Using PySet_* with a sets fallback]
Message-ID: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>

I finally found some spare time at home for implementing some good
support. This will work from 2.3 to 3.0. Of course, the sets.Set and
sets.ImmutableSet will only get used in 2.3, for version 2.4 and
above, all this is a no-op, but you can use this in Cython code
targeting all Python versions starting from 2.3.

You have attached the relevant files.

1) You have a 'cython.py' script. This script just add 'set' and
'frozenset' to __builtin__ module for the case of Python 2.3 . If this
is not done, then Cython will not generate the C codes as appropriate,
and will not understand 'cdef set s'. Then, for the case of Python
2.3, you have to run Cython using this script. If you use the
Distutils support for building Cython extensions, then you will have
to add the hackery in 'cython.py' to your 'setup.py'.

2) Place the files 'py23sets.h' and 'py23sets.pxd' as apropriate in
your source tree. Then all what you need to do is the following: near
the beginning of your pyx module sources, add these two lines:

cimport py23sets
py23sets.install()

And now you are ready to use 'set'/'frozenset' !!

3) You have a 'test.pyx' file were the trick above is used in order to
implement some testing.


A nice thing of all this is that Cython does not require any
modification for supporting sets in Py 2.3. But I believe that this
could be more or less safely integrated in Cython.

Dag, would you still object this? Could we provide a compiler
directive perhaps?


PS: Aaron, you  owe me a beer ;-)


On Fri, Oct 24, 2008 at 4:42 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
> Amendment: Could I use a compiler directive to import sets if /needed/.
>
> On Fri, Oct 24, 2008 at 12:17 PM, Aaron DeVore <aaron.devore at gmail.com> wrote:
>> Could I use a compiler directive to import sets if possible, then use
>> another compiler directive to pick the type when using cdef? Example:
>>
>> IF PY_VERSION_HEX < 0x02040000:
>> ----DEF USE_SETS_MODULE 1
>> ----from sets import Set as set
>>
>> Later in the file...
>>
>> cdef class Foo:
>> ----IF USE_SETS_MODULE:
>> --------cdef object member_set
>> ----ELSE:
>> --------cdef set member_set
>>
>> I haven't figured out how to import compiler directive constants from
>> Python.h, hence the lack of me just trying it out on my own.
>>
>> -Aaron
>>
>>
>>>>> Yes, it would be nice to have that. The idea would be to do the
>>>>> Set import
>>>>> internally, to provide replacement implementations for the generated
>>>>> PySet_*() calls that work at the Python level, and to enable that
>>>>> machinery when the C compile-time Python version is Py2.3.
>>>>> However, that's
>>>>> work that someone has to do.
>>>>>
>>>>> OTOH, this is a pure legacy feature for code that must run with
>>>>> Py2.3. I
>>>>> guess people (including myself) can currently live with the little
>>>>> performance impact regarding the set operations (which are fast
>>>>> anyway),
>>>>> given the gain that their code runs unchanged in Py2.3.
>>>>
>>>> I wish I had the time and expertise for this. :(
>>>>
>>>> Would this work if put at the top of every file (dashes indicate
>>>> indentation)?
>>>> if not hasattr(__builtin__, "set"):
>>>> ----from sets import Set as set
>>>>
>>>> or this
>>>> try:
>>>> ----set
>>>> except NameError:
>>>> ----from sets import Set as set
>>>>
>>>> It seems like there could be an issue with static typing using the
>>>> imported Python class sets.Set (e.g. `cdef set foo`). I don't know
>>>> enough about Cython to make a good guess on that one.
>>>
>>> I think one would have to do
>>>
>>> try:
>>>     from __builtin__ import set
>>> except ImportError:
>>>     from sets import Set as set
>>>
>>> One couldn't, however, do "cdef set foo" because the symbol set would
>>> need to be resolved at compile time, whereas the code (as above)
>>> defers its value to runtime. Consequently the code using the
>>> PySet_Xxxx api directly wouldn't get generated (which of course only
>>> works for 2.4+) Hackery could be done in the compiler to emulate the
>>> builtin set API (via calling the Set module's class), but it should
>>> be noted that the speed gains we're talking about here for the most
>>> part are not that large (compared to, say, indexing into a list or
>>> tuple with an int where there are huge savings).
>>>
>>> - Robert
>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cython.py
Type: text/x-python
Size: 269 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081024/cdfd3b2f/attachment-0001.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: py23sets.h
Type: text/x-chdr
Size: 3596 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081024/cdfd3b2f/attachment-0001.h 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: py23sets.pxd
Type: application/octet-stream
Size: 75 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081024/cdfd3b2f/attachment-0001.obj 

From dalcinl at gmail.com  Sat Oct 25 02:55:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Oct 2008 21:55:57 -0300
Subject: [Cython] Is Cython appropriate tool for the task
In-Reply-To: <87iqrhwud4.fsf@gaura-nitai.no-ip.org>
References: <87iqrhwud4.fsf@gaura-nitai.no-ip.org>
Message-ID: <e7ba66e40810241755n6d4812f7w39fc935ddbd458b4@mail.gmail.com>

On Fri, Oct 24, 2008 at 6:33 PM, Gour <gour at mail.inet.hr> wrote:
>
> Anyone can give me some advice whether Cython is good tool for such
> task(s)?
>

IMHO, Cython is the BEST tool out there for such tasks, specially if
you want to get a really robust and good-looking wrapper.

If you know a bit of MPI, you can take a look at mpi4py (the SVN repo
at Google Code). There, you will find who Cython is used for
implementing a OO API (strongly influenced on the C++ bindings for
MPI) but entirely based in the C API of MPI.

If you know about PETSc, then you can also take a look at petsc4py
(Mercurial repo),

$ hg clone http://petsc.cs.iit.edu/petsc4py/petsc4py-dev

In fact, I've originally implemented petsc4py using SWIG, but once I
entered the Cython world, I've sent the former implementation to the
trash.



>
> p.s. There is already one extension for Swiss Ephemeris lib
> (http://pyswisseph.atarax.org/pydoc/index.html) but I believe with
> Cython one could produce nicer API.
>
>
> Sincerely,
> Gour
>
>
> --
>
> Gour  | Zagreb, Croatia  | GPG key: C6E7162D
> ----------------------------------------------------------------
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From aaron.devore at gmail.com  Sat Oct 25 04:25:36 2008
From: aaron.devore at gmail.com (Aaron DeVore)
Date: Fri, 24 Oct 2008 19:25:36 -0700
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3 [Was:
	Using PySet_* with a sets fallback]
In-Reply-To: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
Message-ID: <2ead2fb0810241925ida560a8p9e4ca87faf72a496@mail.gmail.com>

On Fri, Oct 24, 2008 at 5:42 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> PS: Aaron, you  owe me a beer ;-)

I just looked through the code... Would you like a nice microbrew? A
cheap beer doesn't do your work justice. :)

-Aaron

From dagss at student.matnat.uio.no  Sat Oct 25 09:38:20 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 25 Oct 2008 09:38:20 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
Message-ID: <4902CCEC.8010306@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Dag, would you still object this? Could we provide a compiler
> directive perhaps?

Any objections I might have had were probably linked to what developer 
time is used for -- Python 2.3 is phased out, and if I understood the 
issue correctly, being able to do "cdef set x" isn't such a huge boost 
in performance + you can always use a dict for the same thing, storing 
values of True everywhere.

But now that it is done, that argument falls, so I don't object.

It is Robert or Stefan who should accept it into Cython though.

-- 
Dag Sverre

From gour at mail.inet.hr  Sat Oct 25 09:42:41 2008
From: gour at mail.inet.hr (Gour)
Date: Sat, 25 Oct 2008 09:42:41 +0200
Subject: [Cython] Is Cython appropriate tool for the task
References: <87iqrhwud4.fsf@gaura-nitai.no-ip.org>
	<e7ba66e40810241755n6d4812f7w39fc935ddbd458b4@mail.gmail.com>
Message-ID: <87od1986j2.fsf@nitai.hr>

>>>>> "Lisandro" == <dalcinl at gmail.com> writes:

Lisandro> IMHO, Cython is the BEST tool out there for such tasks,
Lisandro> specially if you want to get a really robust and good-looking
Lisandro> wrapper.

Huh, sounds good ;)

Lisandro> If you know a bit of MPI, you can take a look at mpi4py (the
Lisandro> SVN repo at Google Code). There, you will find who Cython is
Lisandro> used for implementing a OO API (strongly influenced on the C++
Lisandro> bindings for MPI) but entirely based in the C API of MPI.

Not at all familiar with MPI :-(


Lisandro> If you know about PETSc, then you can also take a look at
Lisandro> petsc4py (Mercurial repo),

Lisandro> $ hg clone http://petsc.cs.iit.edu/petsc4py/petsc4py-dev

This looks better, although I'd be glad to find some more 'classical'
C lib (similar to the one I referenced here) with Cython bindings...

Lisandro> In fact, I've originally implemented petsc4py using SWIG, but
Lisandro> once I entered the Cython world, I've sent the former
Lisandro> implementation to the trash.

Heh...I like that there is (more) freedom with Cython to provide
interface more suited for host language - that is similar to c2hs tool
which I'd use in Haskell.


Sincerely,
Gour

-- 

Gour  | Zagreb, Croatia  | GPG key: C6E7162D
----------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081025/d6390d3e/attachment.pgp 

From stefan_ml at behnel.de  Sat Oct 25 11:14:29 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Oct 2008 11:14:29 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
Message-ID: <4902E375.2070002@behnel.de>

Hi Lisandro,

thanks for doing this!

Lisandro Dalcin wrote:
> 1) You have a 'cython.py' script. This script just add 'set' and
> 'frozenset' to __builtin__ module for the case of Python 2.3.

IMHO, the compiler shouldn't mingle with __builtins__, so this must be handled
directly in Cython as a special case.


> 2) Place the files 'py23sets.h' and 'py23sets.pxd' as apropriate in
> your source tree. Then all what you need to do is the following: near
> the beginning of your pyx module sources, add these two lines:
> 
> cimport py23sets
> py23sets.install()

I think the #ifdef depending on the Python header file rather than the Python
version makes sense here, although an additional check for Py2.4+ would be
more future proof. Header barriers aren't necessarily what one would call a
public feature.

However, Cython should generate the respective code into the C file body and
the call to the install() function into the module init function - iff the
set/frozenset builtin types were used in the code. This can use the normal
utility code machinery.

I also don't think we need any special __Pyx_*() naming here. Just name the
functions after their Python 2.4+ counterparts and skip the function pointer
assignments in the setup function.

Also, you can replace some more of the functions by plain macros, such as

	#define PySet_Size(anyset) PyObject_Size(anyset)
	#define PySet_Pop(set) PyObject_CallMethod(set, "pop", NULL)

If nothing else, this reduces at least the legacy code overhead in the
generated C file.

Stefan

From stefan_ml at behnel.de  Sat Oct 25 11:43:04 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Oct 2008 11:43:04 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <4902E375.2070002@behnel.de>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
	<4902E375.2070002@behnel.de>
Message-ID: <4902EA28.1010403@behnel.de>

Hi again,

Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>> 1) You have a 'cython.py' script. This script just add 'set' and
>> 'frozenset' to __builtin__ module for the case of Python 2.3.
> 
> IMHO, the compiler shouldn't mingle with __builtins__, so this must be handled
> directly in Cython as a special case.

I just noticed that that special casing code is more or less there already in
Builtin.py. I'll see if I can fix your patch up.

Aaron, given that you requested this feature, could you come up with a good
doctest for sets? See tests/run/*.pyx for examples on how it's done.

Stefan


From stefan_ml at behnel.de  Sat Oct 25 13:02:37 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Oct 2008 13:02:37 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <4902EA28.1010403@behnel.de>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>	<4902E375.2070002@behnel.de>
	<4902EA28.1010403@behnel.de>
Message-ID: <4902FCCD.5000001@behnel.de>

Hi,

Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>> 1) You have a 'cython.py' script. This script just add 'set' and
>>> 'frozenset' to __builtin__ module for the case of Python 2.3.
>> IMHO, the compiler shouldn't mingle with __builtins__, so this must be handled
>> directly in Cython as a special case.
> 
> I just noticed that that special casing code is more or less there already in
> Builtin.py. I'll see if I can fix your patch up.

Ok, I have committed a patch that works for me on Py2.3-2.6.

http://hg.cython.org/cython-devel/rev/303c79051806

One thing I noticed is that at least a part of the set API is already in
Py2.4, so I had to move some of the replacements into a separate Py2.4
section. This is not in line with the Python C-API docs, which state that the
set API is new in Py2.5. I guess it just wasn't officially made public in Py2.4.

Please review the commit for now, I'll also take another look at the Py2.4
issue myself.

> Aaron, given that you requested this feature, could you come up with a good
> doctest for sets? See tests/run/*.pyx for examples on how it's done.

There already was a test for sets: tests/run/set.pyx. It was just disabled
explicitly for Py2.3. It now works for all supported Python versions.

Stefan


From dalcinl at gmail.com  Sat Oct 25 21:00:07 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 25 Oct 2008 16:00:07 -0300
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <4902E375.2070002@behnel.de>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
	<4902E375.2070002@behnel.de>
Message-ID: <e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>

On Sat, Oct 25, 2008 at 6:14 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi Lisandro,
>
> thanks for doing this!
>
> Lisandro Dalcin wrote:
>> 1) You have a 'cython.py' script. This script just add 'set' and
>> 'frozenset' to __builtin__ module for the case of Python 2.3.
>
> IMHO, the compiler shouldn't mingle with __builtins__, so this must be handled
> directly in Cython as a special case.

Yep, but I had to do that for the special casing n Builtins.py

>
>> 2) Place the files 'py23sets.h' and 'py23sets.pxd' as apropriate in
>> your source tree. Then all what you need to do is the following: near
>> the beginning of your pyx module sources, add these two lines:
>>
>> cimport py23sets
>> py23sets.install()
>
> I think the #ifdef depending on the Python header file rather than the Python
> version makes sense here, although an additional check for Py2.4+ would be
> more future proof. Header barriers aren't necessarily what one would call a
> public feature.

Of course. Take into account that I wrote all that in 10 minutes ;-)

> However, Cython should generate the respective code into the C file body and
> the call to the install() function into the module init function - iff the
> set/frozenset builtin types were used in the code. This can use the normal
> utility code machinery.

Of course, this is the right way.

> I also don't think we need any special __Pyx_*() naming here. Just name the
> functions after their Python 2.4+ counterparts and skip the function pointer
> assignments in the setup function.
>
> Also, you can replace some more of the functions by plain macros, such as
>
>        #define PySet_Size(anyset) PyObject_Size(anyset)
>        #define PySet_Pop(set) PyObject_CallMethod(set, "pop", NULL)
>
> If nothing else, this reduces at least the legacy code overhead in the
> generated C file.
>

Well, I always try to avoid function-like macros at all. Why? Just
because a macro is not a function, you cannot take a pointer to the
function. If we also want Cython code to access the C-API for sets,
then we should be able to to

in (*funcptr)(PyObject*) = PySet_Size;

and now call funcptr(anyset).



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Oct 25 21:46:17 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Oct 2008 21:46:17 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>	<4902E375.2070002@behnel.de>
	<e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>
Message-ID: <49037789.8090500@behnel.de>

Hi,

Lisandro Dalcin wrote:
> On Sat, Oct 25, 2008 at 6:14 AM, Stefan Behnel wrote:
>> I also don't think we need any special __Pyx_*() naming here. Just name the
>> functions after their Python 2.4+ counterparts and skip the function pointer
>> assignments in the setup function.
>>
>> Also, you can replace some more of the functions by plain macros, such as
>>
>>        #define PySet_Size(anyset) PyObject_Size(anyset)
>>        #define PySet_Pop(set) PyObject_CallMethod(set, "pop", NULL)
>>
>> If nothing else, this reduces at least the legacy code overhead in the
>> generated C file.
>>
> 
> Well, I always try to avoid function-like macros at all. Why? Just
> because a macro is not a function, you cannot take a pointer to the
> function.

I'm only -0.5 for the macro version anyway, but I doubt that that's a common
use case. Plus, it only affects Py2.3 and partially Py2.4. I'm pretty sure,
anyone who wants to do fancy stuff with the set API will require Py2.5 anyway,
where this API was first made public.


> If we also want Cython code to access the C-API for sets,
> then we should be able to to
> 
> in (*funcptr)(PyObject*) = PySet_Size;
> 
> and now call funcptr(anyset).

I don't see why that's necessary. PySet_Size() will be defined (as function or
macro) for the user code, regardless of the Python version, and it thus
callable from Cython code once it's defined in a cimported .pxd file. BTW,
PySet_Size is a particularly bad example, as it's likely not much faster than
calling len() anyway.

Stefan

From stefan_ml at behnel.de  Sat Oct 25 21:51:29 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Oct 2008 21:51:29 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <49037789.8090500@behnel.de>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>	<4902E375.2070002@behnel.de>	<e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>
	<49037789.8090500@behnel.de>
Message-ID: <490378C1.7030606@behnel.de>


Stefan Behnel wrote:
> I'm only -0.5 for the macro version anyway

I (obviously) meant +0.5 ...

Stefan


From dalcinl at gmail.com  Sat Oct 25 22:22:24 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 25 Oct 2008 17:22:24 -0300
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <490378C1.7030606@behnel.de>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
	<4902E375.2070002@behnel.de>
	<e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>
	<49037789.8090500@behnel.de> <490378C1.7030606@behnel.de>
Message-ID: <e7ba66e40810251322g5eb4bc22o4671c6a33e0e9d0b@mail.gmail.com>

Stefan, if you do not have strong objections, then I would like to
push the following changes:

1) Change the macros to true functions. We already need to emit
functions, so a few more will not harm. I really hate to patching
C-API's with function-like macros. Declaring pointers to functions is
not usual, but IMHO it is a very valid user case. Note that I'm
talking about users that for some rare reason want to access the
PySet_XXX API's directly in Cython codes (or perhaps in a header file
which is 'cdef extern from ..,' included).

2) Add type checking to function arguments, following the actual
implementations in Python 2.5. Again, that would be a bit pointless
for Cython usage of PySet API, but I'm still tinking about direct
calls to those API's.




On Sat, Oct 25, 2008 at 4:51 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Stefan Behnel wrote:
>> I'm only -0.5 for the macro version anyway
>
> I (obviously) meant +0.5 ...
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Oct 25 23:03:24 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Oct 2008 23:03:24 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <e7ba66e40810251322g5eb4bc22o4671c6a33e0e9d0b@mail.gmail.com>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>	<4902E375.2070002@behnel.de>	<e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>	<49037789.8090500@behnel.de>
	<490378C1.7030606@behnel.de>
	<e7ba66e40810251322g5eb4bc22o4671c6a33e0e9d0b@mail.gmail.com>
Message-ID: <4903899C.8070008@behnel.de>

Hi,

Lisandro Dalcin wrote:
> 1) Change the macros to true functions. We already need to emit
> functions, so a few more will not harm.

I'm still not convinced, but I won't object. Note that I don't accept your
second proposed change as a reason to do this.


> Declaring pointers to functions is
> not usual, but IMHO it is a very valid user case.

I'm just saying that it's not likely in this specific case.


> Note that I'm
> talking about users that for some rare reason want to access the
> PySet_XXX API's directly in Cython codes (or perhaps in a header file
> which is 'cdef extern from ..,' included).

... which they won't have to do very often as Cython does most of the work for
them already. In an ideal Cython world, users won't have to call *any* Python
C-API functions on builtin types directly - with the possible exception of the
Check/CheckExact functions (although a parameter type declaration will take
care of this most of the time) and the string processing functions.


> 2) Add type checking to function arguments, following the actual
> implementations in Python 2.5. Again, that would be a bit pointless
> for Cython usage of PySet API, but I'm still tinking about direct
> calls to those API's.

>From my POV, they are plain legacy functions that are only there to enable
code to run on close-to-be-abandoned Python versions. So it's nice to have
them and they definitely help users in writing portable code that works
efficiently on newer Python versions. The only reason I could see why this
additional overhead might be of any good is compatibility with tests that
require the same type error behaviour on different platforms. However, I find
it highly unlikely (and unpythonic) that people actually write tests
specifically for the set type.

So, no, I don't see any real reason to do that.

Stefan


From dalcinl at gmail.com  Sat Oct 25 23:29:25 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 25 Oct 2008 18:29:25 -0300
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <4903899C.8070008@behnel.de>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>
	<4902E375.2070002@behnel.de>
	<e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>
	<49037789.8090500@behnel.de> <490378C1.7030606@behnel.de>
	<e7ba66e40810251322g5eb4bc22o4671c6a33e0e9d0b@mail.gmail.com>
	<4903899C.8070008@behnel.de>
Message-ID: <e7ba66e40810251429n55639561t6587cc0a41268736@mail.gmail.com>

On Sat, Oct 25, 2008 at 6:03 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> >From my POV, they are plain legacy functions that are only there to enable
> code to run on close-to-be-abandoned Python versions.
>
> So, no, I don't see any real reason to do that.
>

OK, I agree with you, all this is just for supporting ancient Py2.3.

A final question: you wrote the utility code like this:


#if PY_VERSION_HEX < 0x02050000
#ifndef PyAnySet_CheckExact

/* COMMON STUFF FOR Py 2.3 and 2.4 */

#endif /* PyAnySet_CheckExact (<= Py2.4) */

.......


Why are you protecting this stuff with the PyAnySet_CheckExact definition ??

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Oct 25 23:40:54 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 25 Oct 2008 23:40:54 +0200
Subject: [Cython] Implemented Set/FrozenSet support for Python 2.3
In-Reply-To: <e7ba66e40810251429n55639561t6587cc0a41268736@mail.gmail.com>
References: <e7ba66e40810241742u76ed75fcq3784de1d57d803db@mail.gmail.com>	<4902E375.2070002@behnel.de>	<e7ba66e40810251200p6cdab11am99d9a11ceceaa4e@mail.gmail.com>	<49037789.8090500@behnel.de>
	<490378C1.7030606@behnel.de>	<e7ba66e40810251322g5eb4bc22o4671c6a33e0e9d0b@mail.gmail.com>	<4903899C.8070008@behnel.de>
	<e7ba66e40810251429n55639561t6587cc0a41268736@mail.gmail.com>
Message-ID: <49039266.4030300@behnel.de>

Hi,

Lisandro Dalcin wrote:
> #if PY_VERSION_HEX < 0x02050000
> #ifndef PyAnySet_CheckExact
> 
> /* COMMON STUFF FOR Py 2.3 and 2.4 */
> 
> #endif /* PyAnySet_CheckExact (<= Py2.4) */
> 
> Why are you protecting this stuff with the PyAnySet_CheckExact definition ??

Just because that's something that was not available in the old Py2.4 set
implementation (which is different from the one in 2.5+, BTW). So that's a way
to distinguish between a 'real' 2.5 and, say, a 2.4 with an updated set
implementation. From the back of my head, I kind of remember murmurs about a
backport at the time, but maybe that was for 2.3...

I admit that it's also a very rare thing to have a non-standard Python build,
so I guess this can be dropped together with the error checks...

Stefan

From dalcinl at gmail.com  Sun Oct 26 01:23:23 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 25 Oct 2008 20:23:23 -0300
Subject: [Cython] test failing
Message-ID: <e7ba66e40810251623m15b08105w98365613f12b3a5b@mail.gmail.com>

This test is failing: tests/run/dictintindex.pyx

Some new optimization is playing bad games?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sun Oct 26 01:46:37 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Oct 2008 01:46:37 +0200
Subject: [Cython] test failing
In-Reply-To: <e7ba66e40810251623m15b08105w98365613f12b3a5b@mail.gmail.com>
References: <e7ba66e40810251623m15b08105w98365613f12b3a5b@mail.gmail.com>
Message-ID: <4903AFDD.5030405@behnel.de>

Hi,

Lisandro Dalcin wrote:
> This test is failing: tests/run/dictintindex.pyx
> 
> Some new optimization is playing bad games?

See the commit comment for that test. It relates to a recent bug report on the
Pyrex list and will continue to fail until someone fixes it.

Stefan

From vel.accel at gmail.com  Sun Oct 26 02:06:50 2008
From: vel.accel at gmail.com (dieter h)
Date: Sat, 25 Oct 2008 21:06:50 -0400
Subject: [Cython] Dynamic Casting For RecArray Buffer?
Message-ID: <1e52e0880810251806i702df37ch76f66ac9dd7904ac@mail.gmail.com>

Hi all,

What would be an 'elegant' way to handle dynamic casting?

For instance, I have a numpy.recarray.data buffer (char*) that I want
to iterate (pointer arithmetic) and manipulate in cython. What would
be an elegant way to dynamically cast the char* to the type of the
heterogeneous elements? Obvious to say, these are unknown at compile
time.

Utilizing tokenization in a C macro is all that I've thought up so
far, considering I'm a self-taught, still studying student of CS. :)

P.S.: Is this one of those cases where preprocessing macros are
invaluable in **some** conditions, as I just read being debated in an
older thread?

Thanks
-dieter

From dalcinl at gmail.com  Sun Oct 26 02:17:48 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 25 Oct 2008 22:17:48 -0300
Subject: [Cython] test failing
In-Reply-To: <4903AFDD.5030405@behnel.de>
References: <e7ba66e40810251623m15b08105w98365613f12b3a5b@mail.gmail.com>
	<4903AFDD.5030405@behnel.de>
Message-ID: <e7ba66e40810251817m1e93a8c6h51523169f97b8a17@mail.gmail.com>

OK, I?ll take a look on Monday. Perhaps trying to use
PyObjec_DelItem() instead of PySequence_DelItem() will fix the issue.

On Sat, Oct 25, 2008 at 8:46 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Lisandro Dalcin wrote:
>> This test is failing: tests/run/dictintindex.pyx
>>
>> Some new optimization is playing bad games?
>
> See the commit comment for that test. It relates to a recent bug report on the
> Pyrex list and will continue to fail until someone fixes it.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sun Oct 26 10:02:39 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Oct 2008 10:02:39 +0100
Subject: [Cython] Dynamic Casting For RecArray Buffer?
In-Reply-To: <1e52e0880810251806i702df37ch76f66ac9dd7904ac@mail.gmail.com>
References: <1e52e0880810251806i702df37ch76f66ac9dd7904ac@mail.gmail.com>
Message-ID: <4904322F.1010003@student.matnat.uio.no>

dieter h wrote:
> Hi all,
> 
> What would be an 'elegant' way to handle dynamic casting?
> 
> For instance, I have a numpy.recarray.data buffer (char*) that I want
> to iterate (pointer arithmetic) and manipulate in cython. What would
> be an elegant way to dynamically cast the char* to the type of the
> heterogeneous elements? Obvious to say, these are unknown at compile
> time.

I would say that is highly unobvious, in most cases you know the type?

Make sure to chech whether you have a contiguous array:

if not arr.flags['C_CONTIGUOUS']: raise ...

> Utilizing tokenization in a C macro is all that I've thought up so
> far, considering I'm a self-taught, still studying student of CS. :)

Not sure if I know what you mean...

All together I know too little about the problem you have to give a good 
answer (is speed or clarity most important? And so on), and also this is 
a generic programming question and not at all Cython-related.

Assuming what you want to do is converting the recarray to a Python 
representation (to make an example -- what do you really do if you do 
not know the types?), what I'd do is create classes for reading each type:

cdef class ReadNext:
     cdef object read_next(char** pptr): pass

cdef class ReadInt:
     cdef object read_next(char** pptr):
         int** typed = pptr
         result = **pptr
         *pptr += 1 # Increment the "reading-head" you use

... repeat for all types ...

cdef class ReadStruct:
     cdef __init__(self, list_of_sub_readers): ...
     cdef object read_next(char** pptr):
         ...iterate subreaders to do your stuff and return a tuple ...


Then, "parse" the dtype of the recarray to construct a tree of these 
readers (that's only done once per array, so consider it O(1)), and then 
you simply feed your buffer to the root reader.

Lots of polymorphic dispatches though, but unless you know the type at 
compile-time... I suppose using if-tests/switches on the type *might* be 
faster, but I'm no C/performance expert on that level, and this looks a 
lot cleaner.

-- 
Dag Sverre

From vel.accel at gmail.com  Sun Oct 26 11:03:27 2008
From: vel.accel at gmail.com (dieter h)
Date: Sun, 26 Oct 2008 06:03:27 -0400
Subject: [Cython] Dynamic Casting For RecArray Buffer?
In-Reply-To: <4904322F.1010003@student.matnat.uio.no>
References: <1e52e0880810251806i702df37ch76f66ac9dd7904ac@mail.gmail.com>
	<4904322F.1010003@student.matnat.uio.no>
Message-ID: <1e52e0880810260303t5de5432fnefa4f67334ed6c90@mail.gmail.com>

Hi Dag,

On Sun, Oct 26, 2008 at 5:02 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> dieter h wrote:
>> Hi all,
>>
>> What would be an 'elegant' way to handle dynamic casting?
>>
>> For instance, I have a numpy.recarray.data buffer (char*) that I want
>> to iterate (pointer arithmetic) and manipulate in cython. What would
>> be an elegant way to dynamically cast the char* to the type of the
>> heterogeneous elements? Obvious to say, these are unknown at compile
>> time.
>
> I would say that is highly unobvious, in most cases you know the type?

Well in terms of generlization, I wouldn't.

>
> Make sure to chech whether you have a contiguous array:
>
> if not arr.flags['C_CONTIGUOUS']: raise ...
>
>> Utilizing tokenization in a C macro is all that I've thought up so
>> far, considering I'm a self-taught, still studying student of CS. :)
>
> Not sure if I know what you mean...

Well I'm trying to abstract out the flow control of the casting, as
decided by the type string given by the
rec.dtype.fields[NAME].dtype.str. This can't be done by a function in
C (i.e., no c++ like templates). So I assumed a C macro that deposits
a type token into the C (cython) source. I'm not well versed in macro
abilities so I could very well be wrong here.

>
> All together I know too little about the problem you have to give a good
> answer (is speed or clarity most important? And so on),

Yes efficiency is the issue, considering I'm mostly working with
iterations for lengths of 10**9 and beyond. Also, I'm avoiding python
objects where possible. I'm loading the recarray's data and metadata
into C types and iterate the buffer via pointer arithmetic; then
manipulating the elements through gnu c functions.

> and also this is
> a generic programming question and not at all Cython-related.

Well kind of, but not necessarily since I'm implementing in cython.
Not to mention it's kind of tough feelin' your way around when your
self taught, I'd say. So I sincerely thank you for your time.

>
> Assuming what you want to do is converting the recarray to a Python
> representation (to make an example -- what do you really do if you do
> not know the types?), what I'd do is create classes for reading each type:
>
> cdef class ReadNext:
>     cdef object read_next(char** pptr): pass
>
> cdef class ReadInt:
>     cdef object read_next(char** pptr):
>         int** typed = pptr
>         result = **pptr
>         *pptr += 1 # Increment the "reading-head" you use
>
> ... repeat for all types ...
>
> cdef class ReadStruct:
>     cdef __init__(self, list_of_sub_readers): ...
>     cdef object read_next(char** pptr):
>         ...iterate subreaders to do your stuff and return a tuple ...
>
>
> Then, "parse" the dtype of the recarray to construct a tree of these
> readers (that's only done once per array, so consider it O(1)), and then
> you simply feed your buffer to the root reader.
>
> Lots of polymorphic dispatches though, but unless you know the type at
> compile-time... I suppose using if-tests/switches on the type *might* be
> faster, but I'm no C/performance expert on that level, and this looks a
> lot cleaner.

Well I've got a feel for the logic in the example above. Thank you very much.

-dieter

>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Sun Oct 26 13:32:54 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 26 Oct 2008 13:32:54 +0100
Subject: [Cython] Dynamic Casting For RecArray Buffer?
In-Reply-To: <1e52e0880810260303t5de5432fnefa4f67334ed6c90@mail.gmail.com>
References: <1e52e0880810251806i702df37ch76f66ac9dd7904ac@mail.gmail.com>	<4904322F.1010003@student.matnat.uio.no>
	<1e52e0880810260303t5de5432fnefa4f67334ed6c90@mail.gmail.com>
Message-ID: <49046376.9080204@student.matnat.uio.no>

dieter h wrote:
> Hi Dag,
> 
> On Sun, Oct 26, 2008 at 5:02 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> dieter h wrote:
>>> Hi all,
>>>
>>> What would be an 'elegant' way to handle dynamic casting?
>>>
>>> For instance, I have a numpy.recarray.data buffer (char*) that I want
>>> to iterate (pointer arithmetic) and manipulate in cython. What would
>>> be an elegant way to dynamically cast the char* to the type of the
>>> heterogeneous elements? Obvious to say, these are unknown at compile
>>> time.
>> I would say that is highly unobvious, in most cases you know the type?
> 
> Well in terms of generlization, I wouldn't.
> 
>> Make sure to chech whether you have a contiguous array:
>>
>> if not arr.flags['C_CONTIGUOUS']: raise ...
>>
>>> Utilizing tokenization in a C macro is all that I've thought up so
>>> far, considering I'm a self-taught, still studying student of CS. :)
>> Not sure if I know what you mean...
> 
> Well I'm trying to abstract out the flow control of the casting, as
> decided by the type string given by the
> rec.dtype.fields[NAME].dtype.str. This can't be done by a function in
> C (i.e., no c++ like templates). So I assumed a C macro that deposits
> a type token into the C (cython) source. I'm not well versed in macro
> abilities so I could very well be wrong here.
> 
>> All together I know too little about the problem you have to give a good
>> answer (is speed or clarity most important? And so on),
> 
> Yes efficiency is the issue, considering I'm mostly working with
> iterations for lengths of 10**9 and beyond. Also, I'm avoiding python
> objects where possible. I'm loading the recarray's data and metadata
> into C types and iterate the buffer via pointer arithmetic; then
> manipulating the elements through gnu c functions.

If speed is critical, nothing can beat

cdef struct CompleteStruct:
    int a
    int b

CompleteStruct* typed = <CompleteStruct*>arr.buf

And when it comes to speed, this /has/ to be somehow bound at 
compile-time, there's simply no way around it (well, you can dynamically 
compile code, i.e. moving the compile-time stage forward a bit). Even 
C++ templates can't help you there -- all your C++ templates would need 
to be instantiated explicitly at compile-time. So something 
super-generic (and at the same time efficient) getting flow control from 
rec.dtype.fields[NAME].dtype.str is diffcult no matter what you do.

The best you can do (with either C macros or C++ templates) is to create 
a generic library function which allows the user to pass in arbitrary 
struct definitions at compile-time.

But yes, the closest thing you will get to C++ templates is using C 
macros. I've posted some proposals to add this functionality to Cython 
in various ways, and the response was postive, but unfortunately I'm not 
close to having the time to follow through on them and actually 
implement it. (And even then it wouldn't buy you more than C++ 
templates, which as I said I do not believe would solve your problem.)

-- 
Dag Sverre


From stefan_ml at behnel.de  Sun Oct 26 18:41:50 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Oct 2008 18:41:50 +0100
Subject: [Cython] Test coverage now at 87%, call for help to push it higher
Message-ID: <4904ABDE.5070609@behnel.de>

Hi,

I just ran the test suite with code coverage enabled (-C), and it turns out
that we now have a statement coverage of 87% (see below for details). That's
much better than the 72% we had in March. I think the remaining bits are about
missing tests, mostly regarding compiler errors (of which we definitely have
too few), as well as some debug/trace code. There's also a little bit of dead
code in there.

It's actually interesting to look at the coverage annotated sources after running:

    python runtests.py -vv -C     # run test suite in tests/*/ directories
    mkdir results
    python -m coverage -a -d results Cython/Compiler/*.py

This requires coverage.py installed (or copied into your PYTHONPATH):

http://nedbatchelder.com/code/modules/coverage.html

CALL FOR HELP: If anyone can spare some time and wants to help making Cython
more robust, please look through the annotated sources and check for uncovered
code sections (lines starting with a "!") for which you can write a test,
especially in the files Parsing.py, Nodes.py, ExprNodes.py and ModuleNode.py.
Sometimes its just low hanging fruit, sometimes it's really hard to figure out
what Cython code triggers this code in the compiler. This can actually be
quite entertaining and training, and is a good starting point for learning how
Cython works internally. Check the hacker guide to see how to write these tests.

http://wiki.cython.org/HackerGuide

Thanks for any help!

Stefan


Current coverage:

Name                                            Stmts   Exec  Cover
-------------------------------------------------------------------
Cython.Compiler.Annotate                          111     93    83%
Cython.Compiler.AutoDocTransforms                  95     86    90%
Cython.Compiler.Buffer                            426    407    95%
Cython.Compiler.Builtin                            47     47   100%
Cython.Compiler.CmdLine                            88      5     5%
Cython.Compiler.Code                              483    457    94%
Cython.Compiler.CodeGeneration                     12     12   100%
Cython.Compiler.ControlFlow                       113     89    78%
Cython.Compiler.CythonScope                        14     14   100%
Cython.Compiler.Errors                             89     79    88%
Cython.Compiler.ExprNodes                        2603   2282    87%
Cython.Compiler.Future                             10     10   100%
Cython.Compiler.Interpreter                        24     22    91%
Cython.Compiler.Lexicon                            42      3     7%
Cython.Compiler.Main                              439    288    65%
Cython.Compiler.ModuleNode                       1314   1167    88%
Cython.Compiler.Naming                             85     84    98%
Cython.Compiler.Nodes                            2698   2371    87%
Cython.Compiler.Optimize                          111    107    96%
Cython.Compiler.ParseTreeTransforms               441    368    83%
Cython.Compiler.Parsing                          1788   1634    91%
Cython.Compiler.PyrexTypes                        771    679    88%
Cython.Compiler.Scanning                          310    251    80%
Cython.Compiler.StringEncoding                     95     92    96%
Cython.Compiler.Symtab                            915    819    89%
Cython.Compiler.Tests                               0      0   100%
Cython.Compiler.Tests.TestBuffer                   66     34    51%
Cython.Compiler.Tests.TestParseTreeTransforms      32     30    93%
Cython.Compiler.Tests.TestTreeFragment             50     48    96%
Cython.Compiler.Tests.__init__                      0      0   100%
Cython.Compiler.TreeFragment                      127    118    92%
Cython.Compiler.TypeSlots                         281    277    98%
Cython.Compiler.UtilNodes                          48     47    97%
Cython.Compiler.Visitor                           129     86    66%
Cython.Compiler.__init__                            0      0   100%
-------------------------------------------------------------------
TOTAL                                           13857  12106    87%


From vel.accel at gmail.com  Mon Oct 27 00:34:17 2008
From: vel.accel at gmail.com (dieter h)
Date: Sun, 26 Oct 2008 19:34:17 -0400
Subject: [Cython] Dynamic Casting For RecArray Buffer?
In-Reply-To: <49046376.9080204@student.matnat.uio.no>
References: <1e52e0880810251806i702df37ch76f66ac9dd7904ac@mail.gmail.com>
	<4904322F.1010003@student.matnat.uio.no>
	<1e52e0880810260303t5de5432fnefa4f67334ed6c90@mail.gmail.com>
	<49046376.9080204@student.matnat.uio.no>
Message-ID: <1e52e0880810261634k3c3123f6m26fdbbf093c31c01@mail.gmail.com>

Dag, I just wanted to follow up with a thank you note for your
assistance as well your great contributions to cython.

For that matter I'll take this opportunity to thank Greg, Robert, et
al for this awesome software concept and its development.

-dieter

On Sun, Oct 26, 2008 at 8:32 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> dieter h wrote:
>> Hi Dag,
>>
>> On Sun, Oct 26, 2008 at 5:02 AM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> dieter h wrote:
>>>> Hi all,
>>>>
>>>> What would be an 'elegant' way to handle dynamic casting?
>>>>
>>>> For instance, I have a numpy.recarray.data buffer (char*) that I want
>>>> to iterate (pointer arithmetic) and manipulate in cython. What would
>>>> be an elegant way to dynamically cast the char* to the type of the
>>>> heterogeneous elements? Obvious to say, these are unknown at compile
>>>> time.
>>> I would say that is highly unobvious, in most cases you know the type?
>>
>> Well in terms of generlization, I wouldn't.
>>
>>> Make sure to chech whether you have a contiguous array:
>>>
>>> if not arr.flags['C_CONTIGUOUS']: raise ...
>>>
>>>> Utilizing tokenization in a C macro is all that I've thought up so
>>>> far, considering I'm a self-taught, still studying student of CS. :)
>>> Not sure if I know what you mean...
>>
>> Well I'm trying to abstract out the flow control of the casting, as
>> decided by the type string given by the
>> rec.dtype.fields[NAME].dtype.str. This can't be done by a function in
>> C (i.e., no c++ like templates). So I assumed a C macro that deposits
>> a type token into the C (cython) source. I'm not well versed in macro
>> abilities so I could very well be wrong here.
>>
>>> All together I know too little about the problem you have to give a good
>>> answer (is speed or clarity most important? And so on),
>>
>> Yes efficiency is the issue, considering I'm mostly working with
>> iterations for lengths of 10**9 and beyond. Also, I'm avoiding python
>> objects where possible. I'm loading the recarray's data and metadata
>> into C types and iterate the buffer via pointer arithmetic; then
>> manipulating the elements through gnu c functions.
>
> If speed is critical, nothing can beat
>
> cdef struct CompleteStruct:
>    int a
>    int b
>
> CompleteStruct* typed = <CompleteStruct*>arr.buf
>
> And when it comes to speed, this /has/ to be somehow bound at
> compile-time, there's simply no way around it (well, you can dynamically
> compile code, i.e. moving the compile-time stage forward a bit). Even
> C++ templates can't help you there -- all your C++ templates would need
> to be instantiated explicitly at compile-time. So something
> super-generic (and at the same time efficient) getting flow control from
> rec.dtype.fields[NAME].dtype.str is diffcult no matter what you do.
>
> The best you can do (with either C macros or C++ templates) is to create
> a generic library function which allows the user to pass in arbitrary
> struct definitions at compile-time.
>
> But yes, the closest thing you will get to C++ templates is using C
> macros. I've posted some proposals to add this functionality to Cython
> in various ways, and the response was postive, but unfortunately I'm not
> close to having the time to follow through on them and actually
> implement it. (And even then it wouldn't buy you more than C++
> templates, which as I said I do not believe would solve your problem.)
>
> --
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dalcinl at gmail.com  Mon Oct 27 15:27:00 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Oct 2008 11:27:00 -0300
Subject: [Cython] PATCH: fix delitem for index nodes
Message-ID: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>

Stefan, please review this patch. This fixes the recently added
tests/run/dictintindex.pyx testcase.

I had to add a new utility code, but I could not figure out how to
emit that utility code for the particular case of 'del obj[i]', I
mean, I do not know how to differentiate a setitem from a delitem.
Apart from that, the patch seems to work and all the testsuite pass.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: delitem_index.diff
Type: text/x-patch
Size: 3290 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081027/bec4249e/attachment.bin 

From dagss at student.matnat.uio.no  Mon Oct 27 15:42:55 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 27 Oct 2008 15:42:55 +0100
Subject: [Cython] PATCH: fix delitem for index nodes
In-Reply-To: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
References: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
Message-ID: <4905D36F.3020904@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Stefan, please review this patch. This fixes the recently added
> tests/run/dictintindex.pyx testcase.
>
> I had to add a new utility code, but I could not figure out how to
> emit that utility code for the particular case of 'del obj[i]', I
> mean, I do not know how to differentiate a setitem from a delitem.
> Apart from that, the patch seems to work and all the testsuite pass.
>   
Since this summer it is also possible to add utility code in the code 
generation phase, so you can just move things there. Just do

code.globalstate.use_utility_code(...)

instead. (Calling env.use_utility_code should probably be refactored 
project-wide to make the code a little bit cleaner and more consistent; 
but, well, as things work as they are...).

Looking at your patch it should then be trivial to fix this.

Dag Sverre

From stefan_ml at behnel.de  Mon Oct 27 16:16:45 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 27 Oct 2008 16:16:45 +0100 (CET)
Subject: [Cython] PATCH: fix delitem for index nodes
In-Reply-To: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
References: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
Message-ID: <56294.213.61.181.86.1225120605.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Hi Lisandro,

thanks for fixing this.

Lisandro Dalcin wrote:
> Stefan, please review this patch. This fixes the recently added
> tests/run/dictintindex.pyx testcase.

Could you add some more test cases to make sure this also works with
negative values and unsigned types? From looking at the code, I'm not sure
if these cases are handled as expected.


> I had to add a new utility code, but I could not figure out how to
> emit that utility code for the particular case of 'del obj[i]', I
> mean, I do not know how to differentiate a setitem from a delitem.
> Apart from that, the patch seems to work and all the testsuite pass.

As Dag already wrote, utility code is generated at need (i.e. when they
are used) during code generation. This avoids duplicating the necessary
decision logic in the analysis and output phase.

Stefan


From dalcinl at gmail.com  Mon Oct 27 16:53:38 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Oct 2008 12:53:38 -0300
Subject: [Cython] PATCH: fix delitem for index nodes
In-Reply-To: <56294.213.61.181.86.1225120605.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
	<56294.213.61.181.86.1225120605.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40810270853u34a2bb23v992a7b4937042eed@mail.gmail.com>

new patch attached, following Dag's advice. Now also fixed
set_utility_code() calls for getitem and setitem.

On Mon, Oct 27, 2008 at 12:16 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Could you add some more test cases to make sure this also works with
> negative values and unsigned types? From looking at the code, I'm not sure
> if these cases are handled as expected.

Done, added checks for [unsigned] char/int/longlong with -1,0,1
values. But the utility codes are still a bit hard to follow ;-)



>
>> I had to add a new utility code, but I could not figure out how to
>> emit that utility code for the particular case of 'del obj[i]', I
>> mean, I do not know how to differentiate a setitem from a delitem.
>> Apart from that, the patch seems to work and all the testsuite pass.
>
> As Dag already wrote, utility code is generated at need (i.e. when they
> are used) during code generation. This avoids duplicating the necessary
> decision logic in the analysis and output phase.

Yes, I know that, what I was missing was the
code.globalstate.use_utility_code() trick. Many thanks, Dag.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: delitem_index.diff
Type: text/x-patch
Size: 6645 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081027/16d2bed5/attachment.bin 

From stephane.drouard at st.com  Mon Oct 27 17:10:32 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Mon, 27 Oct 2008 17:10:32 +0100
Subject: [Cython] Class module names do not respect package hierarchy
Message-ID: <011a01c9384e$8a2f73e0$b9ad810a@gnb.st.com>

Hello,

I've found an issue with class module names, which do not respect the package hierarchy.

With foo.py containing:

  class Foo:
      pass

>>> import foo
>>> foo
<module 'foo' from 'foo.py'>
>>> foo.Foo().__module__
'foo'

Now compiled through Cython:

>>> import foo
>>> foo
<module 'foo' from 'foo.pyd'>
>>> foo.Foo().__module__
'foo'

Same behaviour. Perfect.

Now if I move it into a package:

>>> import pkg.foo
>>> pkg.foo
<module 'pkg.foo' from 'pkg\foo.pyc'>
>>> pkg.foo.Foo().__module__
'pkg.foo'

And through Cython:

>>> import pkg.foo
>>> pkg.foo
<module 'pkg.foo' from 'pkg\foo.pyd'>
>>> pkg.foo.Foo().__module__
'foo'    ### ERROR: should have been 'pkg.foo'


To solve it, I patch the generated code to get the module name through the module:

static PyObject *__Pyx_CreateClass(
    PyObject *bases, PyObject *dict, PyObject *name, char *modname) {
    PyObject *py_modname;
    PyObject *result = 0;

    #if PY_MAJOR_VERSION < 3
    //SD py_modname = PyString_FromString(modname)
    py_modname = PyObject_GetAttrString(__pyx_m, "__name__");
    #else
    py_modname = PyUnicode_FromString(modname);
    #endif
    if (!py_modname)
        goto bad;
    if (PyDict_SetItemString(dict, "__module__", py_modname) < 0)
        goto bad;
    #if PY_MAJOR_VERSION < 3
    result = PyClass_New(bases, dict, name);
    #else
    result = PyObject_CallFunctionObjArgs((PyObject *)&PyType_Type, name, bases, dict, NULL);
    #endif
bad:
    Py_XDECREF(py_modname);
    return result;
}

Notes:
 - I assume the patch is also compatible with 3.x, but I haven't tested it.
 - parameter "modname" is no more needed.
 - I could have patched Cython directly, but it's simpler for me to "sed" the generated code.

Cheers,
Stephane


From stefan_ml at behnel.de  Mon Oct 27 17:19:10 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 27 Oct 2008 17:19:10 +0100 (CET)
Subject: [Cython] PATCH: fix delitem for index nodes
In-Reply-To: <e7ba66e40810270853u34a2bb23v992a7b4937042eed@mail.gmail.com>
References: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
	<56294.213.61.181.86.1225120605.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40810270853u34a2bb23v992a7b4937042eed@mail.gmail.com>
Message-ID: <39921.213.61.181.86.1225124350.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> On Mon, Oct 27, 2008 at 12:16 PM, Stefan Behnel wrote:
>> Could you add some more test cases to make sure this also works with
>> negative values and unsigned types? From looking at the code, I'm not
>> sure if these cases are handled as expected.
>
> Done, added checks for [unsigned] char/int/longlong with -1,0,1
> values.

Looks very good now.


> But the utility codes are still a bit hard to follow ;-)

Definitely. That's why it's good to have tests for them.

Stefan


From robertwb at math.washington.edu  Mon Oct 27 17:24:42 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Oct 2008 09:24:42 -0700
Subject: [Cython] PATCH: fix delitem for index nodes
In-Reply-To: <e7ba66e40810270853u34a2bb23v992a7b4937042eed@mail.gmail.com>
References: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
	<56294.213.61.181.86.1225120605.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40810270853u34a2bb23v992a7b4937042eed@mail.gmail.com>
Message-ID: <52EB56EE-29E5-4C1A-8EAF-6B1247820343@math.washington.edu>

Thanks. It looks good now, I've committed it.

BTW, if you commit then create a patch with "hg export" it will allow  
you to make a changelog statement and also make sure you get credit  
for it :). Also, you can import a patch from an raw email message (in  
which case it pulls the authorship from the from/subject headers).

- Robert

On Oct 27, 2008, at 8:53 AM, Lisandro Dalcin wrote:

> new patch attached, following Dag's advice. Now also fixed
> set_utility_code() calls for getitem and setitem.
>
> On Mon, Oct 27, 2008 at 12:16 PM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>> Could you add some more test cases to make sure this also works with
>> negative values and unsigned types? From looking at the code, I'm  
>> not sure
>> if these cases are handled as expected.
>
> Done, added checks for [unsigned] char/int/longlong with -1,0,1
> values. But the utility codes are still a bit hard to follow ;-)
>
>
>
>>
>>> I had to add a new utility code, but I could not figure out how to
>>> emit that utility code for the particular case of 'del obj[i]', I
>>> mean, I do not know how to differentiate a setitem from a delitem.
>>> Apart from that, the patch seems to work and all the testsuite pass.
>>
>> As Dag already wrote, utility code is generated at need (i.e. when  
>> they
>> are used) during code generation. This avoids duplicating the  
>> necessary
>> decision logic in the analysis and output phase.
>
> Yes, I know that, what I was missing was the
> code.globalstate.use_utility_code() trick. Many thanks, Dag.
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0) 
> 342-451.1594<delitem_index.diff>______________________________________ 
> _________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From jasone at canonware.com  Mon Oct 27 17:27:12 2008
From: jasone at canonware.com (Jason Evans)
Date: Mon, 27 Oct 2008 09:27:12 -0700
Subject: [Cython] Argument type declaration question
Message-ID: <4905EBE0.7000304@canonware.com>

I would like to be able to write code like the following:

     cdef foo(int i, dict d, list l, str s):
	...

But somehow str is treated differently than int, dict, list, etc., which 
means I have to write:

     cdef foo(int i, dict d, list l, s):
         assert type(s) == str
         ...

I don't understand why this limitation exists.  Is it some artifact of 
the support for implicit casting for char * arguments?

Thanks,
Jason

From dalcinl at gmail.com  Mon Oct 27 19:15:13 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Oct 2008 15:15:13 -0300
Subject: [Cython] PATCH: fix delitem for index nodes
In-Reply-To: <52EB56EE-29E5-4C1A-8EAF-6B1247820343@math.washington.edu>
References: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
	<56294.213.61.181.86.1225120605.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40810270853u34a2bb23v992a7b4937042eed@mail.gmail.com>
	<52EB56EE-29E5-4C1A-8EAF-6B1247820343@math.washington.edu>
Message-ID: <e7ba66e40810271115v57b362b5l20d836fd4072be72@mail.gmail.com>

On Mon, Oct 27, 2008 at 1:24 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Thanks. It looks good now, I've committed it.

OK, I've just pushed a few more testcases for big integers (long long
greater than 2**32).

> BTW, if you commit then create a patch with "hg export" it will allow
> you to make a changelog statement and also make sure you get credit
> for it :). Also, you can import a patch from an raw email message (in
> which case it pulls the authorship from the from/subject headers).

Never mind, getting credits is completely unimportant for me...

>
> - Robert
>
> On Oct 27, 2008, at 8:53 AM, Lisandro Dalcin wrote:
>
>> new patch attached, following Dag's advice. Now also fixed
>> set_utility_code() calls for getitem and setitem.
>>
>> On Mon, Oct 27, 2008 at 12:16 PM, Stefan Behnel
>> <stefan_ml at behnel.de> wrote:
>>> Could you add some more test cases to make sure this also works with
>>> negative values and unsigned types? From looking at the code, I'm
>>> not sure
>>> if these cases are handled as expected.
>>
>> Done, added checks for [unsigned] char/int/longlong with -1,0,1
>> values. But the utility codes are still a bit hard to follow ;-)
>>
>>
>>
>>>
>>>> I had to add a new utility code, but I could not figure out how to
>>>> emit that utility code for the particular case of 'del obj[i]', I
>>>> mean, I do not know how to differentiate a setitem from a delitem.
>>>> Apart from that, the patch seems to work and all the testsuite pass.
>>>
>>> As Dag already wrote, utility code is generated at need (i.e. when
>>> they
>>> are used) during code generation. This avoids duplicating the
>>> necessary
>>> decision logic in the analysis and output phase.
>>
>> Yes, I know that, what I was missing was the
>> code.globalstate.use_utility_code() trick. Many thanks, Dag.
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)
>> 342-451.1594<delitem_index.diff>______________________________________
>> _________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Mon Oct 27 19:24:37 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Oct 2008 11:24:37 -0700
Subject: [Cython] PATCH: fix delitem for index nodes
In-Reply-To: <e7ba66e40810271115v57b362b5l20d836fd4072be72@mail.gmail.com>
References: <e7ba66e40810270727h7330e1des7053418bb953ed0e@mail.gmail.com>
	<56294.213.61.181.86.1225120605.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40810270853u34a2bb23v992a7b4937042eed@mail.gmail.com>
	<52EB56EE-29E5-4C1A-8EAF-6B1247820343@math.washington.edu>
	<e7ba66e40810271115v57b362b5l20d836fd4072be72@mail.gmail.com>
Message-ID: <B3930B01-4DA0-499A-A085-96992D6BF7C4@math.washington.edu>

On Oct 27, 2008, at 11:15 AM, Lisandro Dalcin wrote:

> On Mon, Oct 27, 2008 at 1:24 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> Thanks. It looks good now, I've committed it.
>
> OK, I've just pushed a few more testcases for big integers (long long
> greater than 2**32).

Thanks.

>> BTW, if you commit then create a patch with "hg export" it will allow
>> you to make a changelog statement and also make sure you get credit
>> for it :). Also, you can import a patch from an raw email message (in
>> which case it pulls the authorship from the from/subject headers).
>
> Never mind, getting credits is completely unimportant for me...

Well, *giving* correct credit is important to me :). Good changelog  
comments are nice as well.

- Robert


From dalcinl at gmail.com  Mon Oct 27 19:28:06 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Oct 2008 15:28:06 -0300
Subject: [Cython] Argument type declaration question
In-Reply-To: <4905EBE0.7000304@canonware.com>
References: <4905EBE0.7000304@canonware.com>
Message-ID: <e7ba66e40810271128m32c898eer13bb31b3aac86367@mail.gmail.com>

Are you running Python 3.0? In Python 2.X, I do not expect big
differences. Could you elaborate a bit more what you mean by str being
treated differently than list/dict (int is a completely different
thing, as it means a plain C int, not a PyInt_Type instance)?

On Mon, Oct 27, 2008 at 1:27 PM, Jason Evans <jasone at canonware.com> wrote:
> I would like to be able to write code like the following:
>
>     cdef foo(int i, dict d, list l, str s):
>        ...
>
> But somehow str is treated differently than int, dict, list, etc., which
> means I have to write:
>
>     cdef foo(int i, dict d, list l, s):
>         assert type(s) == str
>         ...
>
> I don't understand why this limitation exists.  Is it some artifact of
> the support for implicit casting for char * arguments?
>
> Thanks,
> Jason
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jasone at canonware.com  Mon Oct 27 19:42:05 2008
From: jasone at canonware.com (Jason Evans)
Date: Mon, 27 Oct 2008 11:42:05 -0700
Subject: [Cython] Argument type declaration question
In-Reply-To: <e7ba66e40810271128m32c898eer13bb31b3aac86367@mail.gmail.com>
References: <4905EBE0.7000304@canonware.com>
	<e7ba66e40810271128m32c898eer13bb31b3aac86367@mail.gmail.com>
Message-ID: <49060B7D.3070308@canonware.com>

Lisandro Dalcin wrote:
> Are you running Python 3.0? In Python 2.X, I do not expect big
> differences. Could you elaborate a bit more what you mean by str being
> treated differently than list/dict (int is a completely different
> thing, as it means a plain C int, not a PyInt_Type instance)?

I am using Python 2.6.

> On Mon, Oct 27, 2008 at 1:27 PM, Jason Evans <jasone at canonware.com> wrote:
>> I would like to be able to write code like the following:
>>
>>     cdef foo(int i, dict d, list l, str s):
>>        ...

Here's what I get if I try to compile the above:

============================================================
Error converting Pyrex file to C:
------------------------------------------------------------
...
def foo(int i, dict d, list l, str s):
                                   ^
------------------------------------------------------------

/home/jasone/foo.pyx:1:35: Expected ')'
============================================================

I can declare arguments to be of type dict or list, but not str.  I 
don't understand why str is treated differently.

Thanks,
Jason

From dalcinl at gmail.com  Mon Oct 27 19:46:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Oct 2008 15:46:57 -0300
Subject: [Cython] Argument type declaration question
In-Reply-To: <49060B7D.3070308@canonware.com>
References: <4905EBE0.7000304@canonware.com>
	<e7ba66e40810271128m32c898eer13bb31b3aac86367@mail.gmail.com>
	<49060B7D.3070308@canonware.com>
Message-ID: <e7ba66e40810271146k38c739afqb05144782f541be7@mail.gmail.com>

Well, then it should be your Cython version. In cython-devel, this
works just fine (at least for Py2.X). We recently fixed some stuff
handling builtin types.

On Mon, Oct 27, 2008 at 3:42 PM, Jason Evans <jasone at canonware.com> wrote:
> Lisandro Dalcin wrote:
>> Are you running Python 3.0? In Python 2.X, I do not expect big
>> differences. Could you elaborate a bit more what you mean by str being
>> treated differently than list/dict (int is a completely different
>> thing, as it means a plain C int, not a PyInt_Type instance)?
>
> I am using Python 2.6.
>
>> On Mon, Oct 27, 2008 at 1:27 PM, Jason Evans <jasone at canonware.com> wrote:
>>> I would like to be able to write code like the following:
>>>
>>>     cdef foo(int i, dict d, list l, str s):
>>>        ...
>
> Here's what I get if I try to compile the above:
>
> ============================================================
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> def foo(int i, dict d, list l, str s):
>                                   ^
> ------------------------------------------------------------
>
> /home/jasone/foo.pyx:1:35: Expected ')'
> ============================================================
>
> I can declare arguments to be of type dict or list, but not str.  I
> don't understand why str is treated differently.
>
> Thanks,
> Jason
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jasone at canonware.com  Mon Oct 27 20:13:43 2008
From: jasone at canonware.com (Jason Evans)
Date: Mon, 27 Oct 2008 12:13:43 -0700
Subject: [Cython] Argument type declaration question
In-Reply-To: <e7ba66e40810271146k38c739afqb05144782f541be7@mail.gmail.com>
References: <4905EBE0.7000304@canonware.com>	<e7ba66e40810271128m32c898eer13bb31b3aac86367@mail.gmail.com>	<49060B7D.3070308@canonware.com>
	<e7ba66e40810271146k38c739afqb05144782f541be7@mail.gmail.com>
Message-ID: <490612E7.6010502@canonware.com>

Lisandro Dalcin wrote:
> Well, then it should be your Cython version. In cython-devel, this
> works just fine (at least for Py2.X). We recently fixed some stuff
> handling builtin types.

Okay, I'll switch to developing with the bleeding edge.  Thanks for your 
help.

Jason


From robertwb at math.washington.edu  Mon Oct 27 20:15:58 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Oct 2008 12:15:58 -0700
Subject: [Cython] Argument type declaration question
In-Reply-To: <490612E7.6010502@canonware.com>
References: <4905EBE0.7000304@canonware.com>	<e7ba66e40810271128m32c898eer13bb31b3aac86367@mail.gmail.com>	<49060B7D.3070308@canonware.com>
	<e7ba66e40810271146k38c739afqb05144782f541be7@mail.gmail.com>
	<490612E7.6010502@canonware.com>
Message-ID: <AB21E590-4659-44F0-B171-D75E4820F1AF@math.washington.edu>

On Oct 27, 2008, at 12:13 PM, Jason Evans wrote:

> Lisandro Dalcin wrote:
>> Well, then it should be your Cython version. In cython-devel, this
>> works just fine (at least for Py2.X). We recently fixed some stuff
>> handling builtin types.
>
> Okay, I'll switch to developing with the bleeding edge.  Thanks for  
> your
> help.

I've been hoping to release as the gap between cython-released and  
cython-devel is getting quite large, but I had some issues I need to  
track down that came up when compiling Sage (I don't think it's  
anything major, but I just haven't had time to look into it deeply yet).

- Robert



From robertwb at math.washington.edu  Mon Oct 27 20:16:45 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Oct 2008 12:16:45 -0700
Subject: [Cython] Argument type declaration question
In-Reply-To: <AB21E590-4659-44F0-B171-D75E4820F1AF@math.washington.edu>
References: <4905EBE0.7000304@canonware.com>	<e7ba66e40810271128m32c898eer13bb31b3aac86367@mail.gmail.com>	<49060B7D.3070308@canonware.com>
	<e7ba66e40810271146k38c739afqb05144782f541be7@mail.gmail.com>
	<490612E7.6010502@canonware.com>
	<AB21E590-4659-44F0-B171-D75E4820F1AF@math.washington.edu>
Message-ID: <3ACF9DC9-B392-4293-8E9D-B1D1838A4D90@math.washington.edu>

On Oct 27, 2008, at 12:15 PM, Robert Bradshaw wrote:

> On Oct 27, 2008, at 12:13 PM, Jason Evans wrote:
>
>> Lisandro Dalcin wrote:
>>> Well, then it should be your Cython version. In cython-devel, this
>>> works just fine (at least for Py2.X). We recently fixed some stuff
>>> handling builtin types.
>>
>> Okay, I'll switch to developing with the bleeding edge.  Thanks  
>> for your
>> help.
>
> I've been hoping to release as the gap between cython-released and  
> cython-devel is getting quite large, but I had some issues I need  
> to track down that came up when compiling Sage (I don't think it's  
> anything major, but I just haven't had time to look into it deeply  
> yet).

That being said, the more people who are on the "bleeding edge," the  
more stable our product will be, so I don't want to discourage you :).

-Robert


From greg.ewing at canterbury.ac.nz  Mon Oct 27 23:38:00 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 28 Oct 2008 11:38:00 +1300
Subject: [Cython] Another Cython
Message-ID: <490642C8.5000507@canterbury.ac.nz>

While googling for "cython documentation" I discovered
that there is another project out there called Cython:

http://campbell.nu/oscar/cython/cython-doc.html

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon Oct 27 23:43:45 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 28 Oct 2008 11:43:45 +1300
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <011a01c9384e$8a2f73e0$b9ad810a@gnb.st.com>
References: <011a01c9384e$8a2f73e0$b9ad810a@gnb.st.com>
Message-ID: <49064421.2070002@canterbury.ac.nz>

Stephane DROUARD wrote:

> I've found an issue with class module names, which do not respect the package hierarchy.

I'm not sure exactly how this works in Cython at the moment,
but you may find the following information about Pyrex
useful:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/source_files.html#mozTocId146379

-- 
Greg

From dalcinl at gmail.com  Tue Oct 28 00:07:16 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Oct 2008 20:07:16 -0300
Subject: [Cython] Another Cython
In-Reply-To: <490642C8.5000507@canterbury.ac.nz>
References: <490642C8.5000507@canterbury.ac.nz>
Message-ID: <e7ba66e40810271607w1c8c515fwa37e1c69eadc7032@mail.gmail.com>

Yep. I also noticed that. But this project is not actually related to
Python. It's just about writing C code that looks pythonic. BTW, the
switch statement does not look so bad ...


On Mon, Oct 27, 2008 at 7:38 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> While googling for "cython documentation" I discovered
> that there is another project out there called Cython:
>
> http://campbell.nu/oscar/cython/cython-doc.html
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue Oct 28 00:24:29 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Oct 2008 16:24:29 -0700
Subject: [Cython] Another Cython
In-Reply-To: <e7ba66e40810271607w1c8c515fwa37e1c69eadc7032@mail.gmail.com>
References: <490642C8.5000507@canterbury.ac.nz>
	<e7ba66e40810271607w1c8c515fwa37e1c69eadc7032@mail.gmail.com>
Message-ID: <8C37DFAD-ED40-4466-A21F-F44A8A86D76B@math.washington.edu>

On Oct 27, 2008, at 4:07 PM, Lisandro Dalcin wrote:

> Yep. I also noticed that. But this project is not actually related to
> Python. It's just about writing C code that looks pythonic. BTW, the
> switch statement does not look so bad ...

There's a huge PEP about adding a switch statement to Python we  
should look at if we want to add this to Cython. Of course, the  
optimizer already does this for you when it can.

>
> On Mon, Oct 27, 2008 at 7:38 PM, Greg Ewing  
> <greg.ewing at canterbury.ac.nz> wrote:
>> While googling for "cython documentation" I discovered
>> that there is another project out there called Cython:
>>
>> http://campbell.nu/oscar/cython/cython-doc.html

Yeah, the project looks mostly abandoned though, so I don't think  
there will be too much confusion.

- Robert


From jasone at canonware.com  Tue Oct 28 01:50:49 2008
From: jasone at canonware.com (Jason Evans)
Date: Mon, 27 Oct 2008 17:50:49 -0700
Subject: [Cython] Calls to (non-existent) PyStr_CheckExact
Message-ID: <490661E9.9040403@canonware.com>

Using changeset a71073d1250c of cython-devel, the following program:

----------------
cdef foo():
     cdef str s
     s = "hello"
foo()
----------------

causes the following warning when linking:

lyken:~> cython foo.pyx
lyken:~> gcc -shared -pthread -fPIC -fwrapv -Wall -fno-strict-aliasing 
-I/usr/local/python/dbg/include/python2.6 foo.c -ofoo.so
foo.c: In function ?__pyx_f_3foo_foo?:
foo.c:212: warning: implicit declaration of function ?PyStr_CheckExact?

and the following error when importing the resulting module:

lyken:~> /usr/local/python/dbg/bin/python
Python 2.6 (r26:66714, Oct 21 2008, 10:58:37)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> import foo
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ImportError: ./foo.so: undefined symbol: PyStr_CheckExact
[26355 refs]
 >>>

I've seen Cython generate similar bogus calls to PyStr_CheckExact 
several times today while coding, but in other cases I was able to make 
the problem go away by adding type specifications so that everything was 
clearly a str.  In this case, it looks like the compiler doesn't realize 
that "hello" is a str (nor does it recognize str("hello") as a str).

So, I think there are two problems:

1) Cython doesn't recognize string literals as str.

2) PyStr_CheckExact doesn't exist, at least for Python 2.6.  It does 
have PyString_CheckExact though.

Thanks,
Jason

From jasone at canonware.com  Tue Oct 28 02:03:20 2008
From: jasone at canonware.com (Jason Evans)
Date: Mon, 27 Oct 2008 18:03:20 -0700
Subject: [Cython] Patch (Re:  Calls to (non-existent) PyStr_CheckExact)
In-Reply-To: <490661E9.9040403@canonware.com>
References: <490661E9.9040403@canonware.com>
Message-ID: <490664D8.4010804@canonware.com>

Change Str to String for PyString_CheckExact call.

diff --git a/Cython/Compiler/PyrexTypes.py b/Cython/Compiler/PyrexTypes.py
--- a/Cython/Compiler/PyrexTypes.py
+++ b/Cython/Compiler/PyrexTypes.py
@@ -302,7 +302,9 @@

      def type_test_code(self, arg):
          type = self.name.capitalize()
-        if type == 'Set':
+        if type == 'Str':
+            type = 'String'
+        elif type == 'Set':
              type = 'AnySet'
          elif type == 'Frozenset':
              type = 'FrozenSet'

From robertwb at math.washington.edu  Tue Oct 28 06:22:40 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Oct 2008 22:22:40 -0700
Subject: [Cython] Calls to (non-existent) PyStr_CheckExact
In-Reply-To: <490661E9.9040403@canonware.com>
References: <490661E9.9040403@canonware.com>
Message-ID: <16276E04-D761-4FCA-9045-3B55C619B692@math.washington.edu>

On Oct 27, 2008, at 5:50 PM, Jason Evans wrote:

> Using changeset a71073d1250c of cython-devel, the following program:
>
> ----------------
> cdef foo():
>      cdef str s
>      s = "hello"
> foo()
> ----------------
>
> causes the following warning when linking:
>
> lyken:~> cython foo.pyx
> lyken:~> gcc -shared -pthread -fPIC -fwrapv -Wall -fno-strict-aliasing
> -I/usr/local/python/dbg/include/python2.6 foo.c -ofoo.so
> foo.c: In function ?__pyx_f_3foo_foo?:
> foo.c:212: warning: implicit declaration of function  
> ?PyStr_CheckExact?
>
> and the following error when importing the resulting module:
>
> lyken:~> /usr/local/python/dbg/bin/python
> Python 2.6 (r26:66714, Oct 21 2008, 10:58:37)
> [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import foo
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> ImportError: ./foo.so: undefined symbol: PyStr_CheckExact
> [26355 refs]
>>>>
>
> I've seen Cython generate similar bogus calls to PyStr_CheckExact
> several times today while coding, but in other cases I was able to  
> make
> the problem go away by adding type specifications so that  
> everything was
> clearly a str.  In this case, it looks like the compiler doesn't  
> realize
> that "hello" is a str (nor does it recognize str("hello") as a str).
>
> So, I think there are two problems:
>
> 1) Cython doesn't recognize string literals as str.
>
> 2) PyStr_CheckExact doesn't exist, at least for Python 2.6.  It does
> have PyString_CheckExact though.

Looks like it was (2), from Python 2.2+. Thanks for the fix.

- Robert


From stefan_ml at behnel.de  Tue Oct 28 08:05:26 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 28 Oct 2008 08:05:26 +0100 (CET)
Subject: [Cython] Another Cython
In-Reply-To: <e7ba66e40810271607w1c8c515fwa37e1c69eadc7032@mail.gmail.com>
References: <490642C8.5000507@canterbury.ac.nz>
	<e7ba66e40810271607w1c8c515fwa37e1c69eadc7032@mail.gmail.com>
Message-ID: <36273.213.61.181.86.1225177526.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> the switch statement does not look so bad ...

No, please... ;)

Stefan


From stephane.drouard at st.com  Tue Oct 28 11:52:53 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Tue, 28 Oct 2008 11:52:53 +0100
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <49064421.2070002@canterbury.ac.nz>
Message-ID: <001201c938eb$544248f0$b9ad810a@gnb.st.com>

Greg Ewing <greg.ewing at ...> wrote:

> I'm not sure exactly how this works in Cython at the moment,
> but you may find the following information about Pyrex
> useful:
> 
>
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/
source_files.html#mozTocId146379

It is written:

"If your module is destined to live in a package, the Pyrex compiler needs
to know the fully-qualified name that the module will eventually have.
...
This will ensure that the __name__ properties of the module and any classes
defined in it are set correctly. ..."

The patch I'm proposing avoids having to declare/know the package structure
at compile time, but determines it at runtime.

This is the way Python modules behave.

Stephane


From dalcinl at gmail.com  Tue Oct 28 15:59:09 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 28 Oct 2008 11:59:09 -0300
Subject: [Cython] Another Cython
In-Reply-To: <36273.213.61.181.86.1225177526.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <490642C8.5000507@canterbury.ac.nz>
	<e7ba66e40810271607w1c8c515fwa37e1c69eadc7032@mail.gmail.com>
	<36273.213.61.181.86.1225177526.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40810280759n142282fewff8c514b7885913d@mail.gmail.com>

On Tue, Oct 28, 2008 at 4:05 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin wrote:
>> the switch statement does not look so bad ...
>
> No, please... ;)

I know, it's a nonsense ;-), and more now that Cython optimizes it.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Oct 28 16:08:26 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 28 Oct 2008 12:08:26 -0300
Subject: [Cython] Calls to (non-existent) PyStr_CheckExact
In-Reply-To: <16276E04-D761-4FCA-9045-3B55C619B692@math.washington.edu>
References: <490661E9.9040403@canonware.com>
	<16276E04-D761-4FCA-9045-3B55C619B692@math.washington.edu>
Message-ID: <e7ba66e40810280808q40cf5554y13ebdf820e4e386e@mail.gmail.com>

Rober, I had to fix your last commit about this, please review.

On Tue, Oct 28, 2008 at 2:22 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 27, 2008, at 5:50 PM, Jason Evans wrote:
>
>> Using changeset a71073d1250c of cython-devel, the following program:
>>
>> ----------------
>> cdef foo():
>>      cdef str s
>>      s = "hello"
>> foo()
>> ----------------
>>
>> causes the following warning when linking:
>>
>> lyken:~> cython foo.pyx
>> lyken:~> gcc -shared -pthread -fPIC -fwrapv -Wall -fno-strict-aliasing
>> -I/usr/local/python/dbg/include/python2.6 foo.c -ofoo.so
>> foo.c: In function '__pyx_f_3foo_foo':
>> foo.c:212: warning: implicit declaration of function
>> 'PyStr_CheckExact'
>>
>> and the following error when importing the resulting module:
>>
>> lyken:~> /usr/local/python/dbg/bin/python
>> Python 2.6 (r26:66714, Oct 21 2008, 10:58:37)
>> [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import foo
>> Traceback (most recent call last):
>>    File "<stdin>", line 1, in <module>
>> ImportError: ./foo.so: undefined symbol: PyStr_CheckExact
>> [26355 refs]
>>>>>
>>
>> I've seen Cython generate similar bogus calls to PyStr_CheckExact
>> several times today while coding, but in other cases I was able to
>> make
>> the problem go away by adding type specifications so that
>> everything was
>> clearly a str.  In this case, it looks like the compiler doesn't
>> realize
>> that "hello" is a str (nor does it recognize str("hello") as a str).
>>
>> So, I think there are two problems:
>>
>> 1) Cython doesn't recognize string literals as str.
>>
>> 2) PyStr_CheckExact doesn't exist, at least for Python 2.6.  It does
>> have PyString_CheckExact though.
>
> Looks like it was (2), from Python 2.2+. Thanks for the fix.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Oct 28 16:19:43 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 28 Oct 2008 12:19:43 -0300
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <001201c938eb$544248f0$b9ad810a@gnb.st.com>
References: <49064421.2070002@canterbury.ac.nz>
	<001201c938eb$544248f0$b9ad810a@gnb.st.com>
Message-ID: <e7ba66e40810280819s77614371jcfbf9d38fc845782@mail.gmail.com>

On Tue, Oct 28, 2008 at 7:52 AM, Stephane DROUARD
<stephane.drouard at st.com> wrote:
> This will ensure that the __name__ properties of the module and any classes
> defined in it are set correctly. ..."
>
> The patch I'm proposing avoids having to declare/know the package structure
> at compile time, but determines it at runtime.

Your patch would perhaps work for Python clases, but not for extension
types (I mean, cdef ones).

Why did you run in the need to do this? Could you elaborate a bit more
your needs?




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue Oct 28 18:04:42 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 28 Oct 2008 10:04:42 -0700
Subject: [Cython] Calls to (non-existent) PyStr_CheckExact
In-Reply-To: <e7ba66e40810280808q40cf5554y13ebdf820e4e386e@mail.gmail.com>
References: <490661E9.9040403@canonware.com>
	<16276E04-D761-4FCA-9045-3B55C619B692@math.washington.edu>
	<e7ba66e40810280808q40cf5554y13ebdf820e4e386e@mail.gmail.com>
Message-ID: <5DB503D9-3784-493E-8A57-FE01FA48FD15@math.washington.edu>

On Oct 28, 2008, at 8:08 AM, Lisandro Dalcin wrote:

> Rober, I had to fix your last commit about this, please review.

Thanks. I ran some tests but I guess I didn't run any that used this  
or it would have clearly broken. Should have just run the whole suite.

>
> On Tue, Oct 28, 2008 at 2:22 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Oct 27, 2008, at 5:50 PM, Jason Evans wrote:
>>
>>> Using changeset a71073d1250c of cython-devel, the following program:
>>>
>>> ----------------
>>> cdef foo():
>>>      cdef str s
>>>      s = "hello"
>>> foo()
>>> ----------------
>>>
>>> causes the following warning when linking:
>>>
>>> lyken:~> cython foo.pyx
>>> lyken:~> gcc -shared -pthread -fPIC -fwrapv -Wall -fno-strict- 
>>> aliasing
>>> -I/usr/local/python/dbg/include/python2.6 foo.c -ofoo.so
>>> foo.c: In function '__pyx_f_3foo_foo':
>>> foo.c:212: warning: implicit declaration of function
>>> 'PyStr_CheckExact'
>>>
>>> and the following error when importing the resulting module:
>>>
>>> lyken:~> /usr/local/python/dbg/bin/python
>>> Python 2.6 (r26:66714, Oct 21 2008, 10:58:37)
>>> [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
>>> Type "help", "copyright", "credits" or "license" for more  
>>> information.
>>>>>> import foo
>>> Traceback (most recent call last):
>>>    File "<stdin>", line 1, in <module>
>>> ImportError: ./foo.so: undefined symbol: PyStr_CheckExact
>>> [26355 refs]
>>>>>>
>>>
>>> I've seen Cython generate similar bogus calls to PyStr_CheckExact
>>> several times today while coding, but in other cases I was able to
>>> make
>>> the problem go away by adding type specifications so that
>>> everything was
>>> clearly a str.  In this case, it looks like the compiler doesn't
>>> realize
>>> that "hello" is a str (nor does it recognize str("hello") as a str).
>>>
>>> So, I think there are two problems:
>>>
>>> 1) Cython doesn't recognize string literals as str.
>>>
>>> 2) PyStr_CheckExact doesn't exist, at least for Python 2.6.  It does
>>> have PyString_CheckExact though.
>>
>> Looks like it was (2), from Python 2.2+. Thanks for the fix.
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stephane.drouard at st.com  Tue Oct 28 18:52:06 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Tue, 28 Oct 2008 18:52:06 +0100
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <e7ba66e40810280819s77614371jcfbf9d38fc845782@mail.gmail.com>
Message-ID: <000001c93925$e45a5bf0$b9ad810a@gnb.st.com>

We develop modules (either pure Python or SWIG'ed ones) that are used by
people importing them into their applications. Then can put those modules
where they want.

Some modules need additional data files, so to easily find them and avoid
name clashing, they are located in a sub-directory where the module resides,
with the same name (without the extension).

So to access those data files from an instance:

        path = os.path.splitext(sys.modules[self.__module__].__file__)[0]

We are evaluating the possibility to move to Cython instead of SWIG. But
this statement above may fail, because the name returned by self.__module__
may not exist in sys.modules (when prefixed by package name(s)).

Trying to find the issue, I saw that the module name, at class creation, was
hardcoded, whereas the module's __name__ attribute was correctly reporting
the package hierarchy. So the patch I'm proposing.

Could you provide me with a simple example of "extension types (I mean, cdef
ones)" demonstrating the issue of my patch (sorry I'm new in Cython)?

Thanks,
Stephane


From greg.ewing at canterbury.ac.nz  Tue Oct 28 23:17:21 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 29 Oct 2008 11:17:21 +1300
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <001201c938eb$544248f0$b9ad810a@gnb.st.com>
References: <001201c938eb$544248f0$b9ad810a@gnb.st.com>
Message-ID: <49078F71.2000502@canterbury.ac.nz>

Stephane DROUARD wrote:

> The patch I'm proposing avoids having to declare/know the package structure
> at compile time, but determines it at runtime.

Be careful -- getting the class names right isn't the
only reason Pyrex needs to know the qualified name.
Don't mess with this unless you fully understand all
the implications.

-- 
Greg

From stephane.drouard at st.com  Wed Oct 29 12:24:28 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Wed, 29 Oct 2008 12:24:28 +0100
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <49078F71.2000502@canterbury.ac.nz>
Message-ID: <000401c939b8$e81c8a40$b9ad810a@gnb.st.com>

Greg Ewing <greg.ewing at ...> wrote:

> Be careful -- getting the class names right isn't the
> only reason Pyrex needs to know the qualified name.
> Don't mess with this unless you fully understand all
> the implications.

I did the test with a cdef class:

  cdef class Foo:
      pass

I confirm that it fails when we move foo.pyd among packages.

Same as my original point, it's because the name is hardcoded instead of
being retrieved through the module once created.

If, before PyType_Ready'ing types, we build the type's tp_name at runtime
(for example by the following code), the module can be moved among packages.

  PyObject* name;
  const char* modname;
  int modlen;
  char* classname;

  name = PyObject_GetAttrString(__pyx_m, "__name__");
  PyObject_AsCharBuffer(name, &modname, &modlen);
  classname = (char*)malloc(modlen + 1 + sizeof("Foo"));
  strcpy(classname, modname);
  classname[modlen] = '.';
  strcpy(classname + modlen + 1, "Foo");
  __pyx_type_3foo_Foo.tp_name = classname;
  PY_DECREF(name);


(note that foo.Foo().__module__ is not defined contrary to Python classes.
We have to use foo.Foo().__class__.__module__ instead. I don't know yet what
the issue is).


There are maybe some other situations where Pyrex/Cython hardcodes the
module name in the generated code, but I'm quite confident that they can be
replaced by the module's name retrieved at runtime.


At the end, it's your decision whether you want to stop hardcoding module
name.

Here are my arguments to ask for this change:
 - in our situation, module developpers don't care where their modules will
be located and I don't think we want to add this constraint.
 - moreover, if we add this constraint:
   - module tests will now fail,
   - all users of our modules will have to put them in the same package
name.
 - but maybe the main point is that pure Python modules have no (static)
knowledge of their destination package, why not Pyrex/Cython modules?

Cheers,
Stephane


From stefan_ml at behnel.de  Wed Oct 29 13:11:53 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Oct 2008 13:11:53 +0100 (CET)
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <000401c939b8$e81c8a40$b9ad810a@gnb.st.com>
References: <000401c939b8$e81c8a40$b9ad810a@gnb.st.com>
Message-ID: <41258.213.61.181.86.1225282313.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Stephane DROUARD wrote:
> There are maybe some other situations where Pyrex/Cython hardcodes the
> module name in the generated code, but I'm quite confident that they can
> be replaced by the module's name retrieved at runtime.

I'm not sure this is the case for cimports of external extension types
(which are imported at runtime). This will have to be verified before such
a change can be accepted.

Stefan


From dalcinl at gmail.com  Wed Oct 29 14:48:20 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 29 Oct 2008 10:48:20 -0300
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <41258.213.61.181.86.1225282313.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <000401c939b8$e81c8a40$b9ad810a@gnb.st.com>
	<41258.213.61.181.86.1225282313.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40810290648n58e32c91n5eec2f9e629f3b0d@mail.gmail.com>

Or perhaps we could do thing like this:

#ifndef PYX_MODULE_NAME
#define PYX_MODULE_NAME "mypkg.mymod"
#endif

and use the macro PYX_MODULE_NAME in the generated sources.

And of course, if the user pass the -D flag to the compliler and next
this breaks 'cimport', please do not blame us ;-).



On Wed, Oct 29, 2008 at 9:11 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stephane DROUARD wrote:
>> There are maybe some other situations where Pyrex/Cython hardcodes the
>> module name in the generated code, but I'm quite confident that they can
>> be replaced by the module's name retrieved at runtime.
>
> I'm not sure this is the case for cimports of external extension types
> (which are imported at runtime). This will have to be verified before such
> a change can be accepted.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stephane.drouard at st.com  Wed Oct 29 15:02:28 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Wed, 29 Oct 2008 15:02:28 +0100
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <e7ba66e40810290648n58e32c91n5eec2f9e629f3b0d@mail.gmail.com>
Message-ID: <000e01c939ce$faf3fcf0$b9ad810a@gnb.st.com>

Lisandro Dalcin wrote:

> Or perhaps we could do thing like this:
> 
> #ifndef PYX_MODULE_NAME
> #define PYX_MODULE_NAME "mypkg.mymod"
> #endif
> 
> and use the macro PYX_MODULE_NAME in the generated sources.

It seems you are suggesting to give the possibility to set the module name
at compile time.

This would not solve my issue where I would need that the package hierarchy
is determined at runtime (because I deliver PYD files, not source files).

Now having a macro to say "use a static name" (the default, as of today) or
"determine it at runtime" is fine for me and would possibly avoid breaking
existing code.

Stephane


From dalcinl at gmail.com  Wed Oct 29 17:04:42 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 29 Oct 2008 13:04:42 -0300
Subject: [Cython] PATH: force threads initialization for Py2.3, please review
Message-ID: <e7ba66e40810290904q1d0e9e00uf80bf98d99eb4994@mail.gmail.com>

Some time ago, I've posted some mails about thread issues in Python
2.3. Unfortunately, those post did not receive any attention.

Currently, mpi4py on Python 2.3 makes the interpreter segfault. I
could easily fix this issue myself for mpi4py, but perhaps it do make
sense to alleviate other Py2.3 users of this really low-level issue.

The problem: If at the point PyGILState_Release() is called, the GIL
whas not yet created, then the Python (2.3) interpreter segfautls.

The solution: Call PyEval_InitThread() in the module init function.
But this could have performance impacts for non threaded applications
using a Cython-generated extension module, so this thread
initialization should be done only if needed. A simple approach is to
call PyEval_InitThread() only if the Cython module ever
released/aquired the GIL

The attached patch implement all the ideas discussed above. Any chance
to this being accepted? Any better idea? Note that PyEval_InitThread()
is a no-op if the GIL is already created.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: py23_init_threads.diff
Type: text/x-patch
Size: 2236 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081029/a37b6b40/attachment.bin 

From stefan_ml at behnel.de  Wed Oct 29 18:49:25 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Oct 2008 18:49:25 +0100
Subject: [Cython] PATH: force threads initialization for Py2.3,
 please review
In-Reply-To: <e7ba66e40810290904q1d0e9e00uf80bf98d99eb4994@mail.gmail.com>
References: <e7ba66e40810290904q1d0e9e00uf80bf98d99eb4994@mail.gmail.com>
Message-ID: <4908A225.1070405@behnel.de>

Hi,

just a quick comment on this one:

Lisandro Dalcin wrote:
> The problem: If at the point PyGILState_Release() is called, the GIL
> whas not yet created, then the Python (2.3) interpreter segfautls.

I remember seeing lxml segfault also on Py2.4.1, not sure if it was for the
same reason. I currently use a check for Py < 2.4.2 in a header file to turn
off threading support. This might be needed here, too.

Stefan


From stefan_ml at behnel.de  Wed Oct 29 18:53:54 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 29 Oct 2008 18:53:54 +0100
Subject: [Cython] PATH: force threads initialization for Py2.3,
 please review
In-Reply-To: <e7ba66e40810290904q1d0e9e00uf80bf98d99eb4994@mail.gmail.com>
References: <e7ba66e40810290904q1d0e9e00uf80bf98d99eb4994@mail.gmail.com>
Message-ID: <4908A332.6020409@behnel.de>

Hi again,

Lisandro Dalcin wrote:
> The problem: If at the point PyGILState_Release() is called, the GIL
> whas not yet created, then the Python (2.3) interpreter segfautls.
> 
> The solution: Call PyEval_InitThread() in the module init function.
> But this could have performance impacts for non threaded applications
> using a Cython-generated extension module, so this thread
> initialization should be done only if needed. A simple approach is to
> call PyEval_InitThread() only if the Cython module ever
> released/aquired the GIL
> 
> The attached patch implement all the ideas discussed above. Any chance
> to this being accepted? Any better idea? Note that PyEval_InitThread()
> is a no-op if the GIL is already created.

I'm fine with calling InitThread() when "with nogil" or "with gil" is used
anywhere in the source. And I think it should be left to the users to handle
all other cases themselves.

Also note that UtilityCode() accepts a keyword "init", so you can just put the
call to InitThreads() in there and it will do the right thing. I think the
"proto" can be dropped completely in this case.

Stefan

From dalcinl at gmail.com  Wed Oct 29 21:40:09 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 29 Oct 2008 17:40:09 -0300
Subject: [Cython] PATH: force threads initialization for Py2.3,
	please review
In-Reply-To: <4908A225.1070405@behnel.de>
References: <e7ba66e40810290904q1d0e9e00uf80bf98d99eb4994@mail.gmail.com>
	<4908A225.1070405@behnel.de>
Message-ID: <e7ba66e40810291340q437a7756qcde6408cc2a95576@mail.gmail.com>

On Wed, Oct 29, 2008 at 2:49 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> just a quick comment on this one:
>
> Lisandro Dalcin wrote:
>> The problem: If at the point PyGILState_Release() is called, the GIL
>> whas not yet created, then the Python (2.3) interpreter segfautls.
>
> I remember seeing lxml segfault also on Py2.4.1, not sure if it was for the
> same reason. I currently use a check for Py < 2.4.2 in a header file to turn
> off threading support. This might be needed here, too.
>

AFAIK, the issue I've described is only present in Py2.3 . Moreover, I
mpi4py never experienced any problem on Py2.4


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Oct 29 21:50:29 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 29 Oct 2008 17:50:29 -0300
Subject: [Cython] PATH: force threads initialization for Py2.3,
	please review
In-Reply-To: <4908A332.6020409@behnel.de>
References: <e7ba66e40810290904q1d0e9e00uf80bf98d99eb4994@mail.gmail.com>
	<4908A332.6020409@behnel.de>
Message-ID: <e7ba66e40810291350j5eae7785l8cfc7760ae63318d@mail.gmail.com>

On Wed, Oct 29, 2008 at 2:53 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> I'm fine with calling InitThread() when "with nogil" or "with gil" is used
> anywhere in the source.

OK

> And I think it should be left to the users to handle
> all other cases themselves.

Here I'm not following you. What other cases are you talking about?

> Also note that UtilityCode() accepts a keyword "init", so you can just put the
> call to InitThreads() in there and it will do the right thing. I think the
> "proto" can be dropped completely in this case.

Yes, I reviewed the code you recently pushed. But the "init" (if I got
it right) seems to be intended for initializing global variables. So I
was not sure if using "init" was strictly correct.

Moreover, if you look carefully at my path, you will find that the
"proto" stuff starts with "ifndef __PYX_FORCE_INIT_THREADS", just in
case uses want to pass this flag to the C compiler a
-D__PYX_FORCE_INIT_THREADS=1/0 to force/skip the hardwired call. For
example, if you suspect that Py2.4.1 could have the same problem, then
you could try do pass -D__PYX_FORCE_INIT_THREADS=1 and see what
happens.

But if you think the above stuff is just a nonsense, and we should
just uncondionally make the PyEval_InitThread() call in the Py2.3
case, just let me know and I'll write that like you want.





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Oct 30 00:42:45 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 30 Oct 2008 12:42:45 +1300
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <000001c93925$e45a5bf0$b9ad810a@gnb.st.com>
References: <000001c93925$e45a5bf0$b9ad810a@gnb.st.com>
Message-ID: <4908F4F5.4070809@canterbury.ac.nz>

Stephane DROUARD wrote:

> Could you provide me with a simple example of "extension types (I mean, cdef
> ones)" demonstrating the issue of my patch (sorry I'm new in Cython)?

When a module cimports an extension type from another
module, code is generated to import the type object at
run time. To do this, the Pyrex compiler needs to know
where the other module is in the module namespace. If
it gets moved somewhere else, the import will fail.

-- 
Greg


From casevh at gmail.com  Thu Oct 30 07:08:12 2008
From: casevh at gmail.com (Case Vanhorsen)
Date: Wed, 29 Oct 2008 23:08:12 -0700
Subject: [Cython] Performance question (long)
Message-ID: <99e0b9530810292308n3f8cd738w84b5f711dc2b74a7@mail.gmail.com>

I help maintain the gmpy library and I've been experimenting with a
Cython-based wrapper for GMP. I've done simple performance tests with
an early version of the library and it is slower than the C-based gmpy
library. I compared object creation and addition times for Python
longs, gmpy, and gmpy3 (the experimental library). These are my
results, in seconds (code is below):

create times
long:   2.25385284424
gmpy:   1.70728683472
gmpy3:  2.3947558403

addition times
long:   3.03700900078
gmpy:   2.6448340416
gmpy3:  4.70184206963

Do you have any suggestions for improving the performance?

Thanks,

Case

--------------------------------------------

The test script:

import time
import gmpy
import gmpy3

def create_test(ctor, times):
    start = time.time()
    for i in range(times):
        a=ctor(i)
    return (time.time() - start)

def addition_test(ctor, times):
    b = ctor('12345678901234567890')
    start = time.time()
    for i in range(times):
        a=ctor(i) + b
    return (time.time() - start)

print "create times"
print "long:  ", create_test(long, 10000000)
print "gmpy:  ", create_test(gmpy.mpz, 10000000)
print "gmpy3: ", create_test(gmpy3.mpz, 10000000)
print
print "addition times"
print "long:  ", addition_test(long, 10000000)
print "gmpy:  ", addition_test(gmpy.mpz, 10000000)
print "gmpy3: ", addition_test(gmpy3.mpz, 10000000)

-----------------------------------------------------------------

Excerpts from gmpy3.pyx

cdef class mpz:
    """
    Wrapper for GMP multiple-precision integers.
    """

    cdef mpz_t z

    def __init__(self, val=None, base=10):
        """
        Initialize a variable of type mpz.
        """
        cdef _longobject *l

        if val is None:
            mpz_init(self.z)
        elif PyInt_CheckExact(val):
            mpz_init_set_si(self.z, val)
        elif PyString_CheckExact(val):
            if mpz_init_set_str(self.z, val, base):
                raise ValueError('Invalid literal for mpz() with base
' + str(base))
        elif PyLong_CheckExact(val):
            mpz_init(self.z)
            l = <_longobject *> val
            if l.ob_size < 0:
                negval = True
                length = - l.ob_size
            else:
                negval = False
                length = l.ob_size
            mpz_import(self.z, length, -1, sizeof(l.ob_digit[0]), 0,
sizeof(l.ob_digit[0])*8 - SHIFT, l.ob_digit)
            if negval:
                mpz_neg(self.z, self.z)
        elif PyFloat_CheckExact(val):
            mpz_init_set_d(self.z, val)
        elif isinstance(val, mpz):
            mpz_init_set(self.z, (<mpz>val).z)
        else:
            raise TypeError('mpz() argument must be string, int, long,
or float')

    def __dealloc__(self):
        mpz_clear(self.z)

    def __add__(self, other):
        cdef int temp
        result = mpz()
        cdef mpz Z

        if isinstance(self, mpz):
            if isinstance(other, mpz):
                mpz_add((<mpz>result).z, (<mpz>self).z, (<mpz>other).z)
            elif PyInt_CheckExact(other):
                if other >= 0:
                    mpz_add_ui((<mpz>result).z, (<mpz>self).z, other)
                else:
                    mpz_sub_ui((<mpz>result).z, (<mpz>self).z, abs(other))
            else:
                mpz_add((<mpz>result).z, (<mpz>self).z, (<mpz>mpz(other)).z)
            return result
        else:
            return mpz.__add__(other, self)

From robertwb at math.washington.edu  Thu Oct 30 07:57:17 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 29 Oct 2008 23:57:17 -0700
Subject: [Cython] Performance question (long)
In-Reply-To: <99e0b9530810292308n3f8cd738w84b5f711dc2b74a7@mail.gmail.com>
References: <99e0b9530810292308n3f8cd738w84b5f711dc2b74a7@mail.gmail.com>
Message-ID: <1E178D03-B2CC-488E-B590-4C316A6BD114@math.washington.edu>

On Oct 29, 2008, at 11:08 PM, Case Vanhorsen wrote:

> I help maintain the gmpy library and I've been experimenting with a
> Cython-based wrapper for GMP. I've done simple performance tests with
> an early version of the library and it is slower than the C-based gmpy
> library. I compared object creation and addition times for Python
> longs, gmpy, and gmpy3 (the experimental library). These are my
> results, in seconds (code is below):
>
> create times
> long:   2.25385284424
> gmpy:   1.70728683472
> gmpy3:  2.3947558403
>
> addition times
> long:   3.03700900078
> gmpy:   2.6448340416
> gmpy3:  4.70184206963

Interesting.

> Do you have any suggestions for improving the performance?

It looks like gmpy.c doesn't do any type checking for add, which  
could be a cause for concern (but also make things faster). Is this  
with cython-devel or the release version? The isinstance function  
will be much faster in the next release of Cython, as is the argument  
parsing, so that should help too. Probably the most significant thing  
you could do is construct the mpz class directly rather than call mpz 
() (e.g. see the PY_NEW macro at http://www.sagemath.org/hg/sage-main/ 
file/3859ace86968/c_lib/include/stdsage.h for one way to do that).  
Also, type your "base" parameter to be an int in the __init__ method.

>
> Thanks,
>
> Case
>
> --------------------------------------------
>
> The test script:
>
> import time
> import gmpy
> import gmpy3
>
> def create_test(ctor, times):
>     start = time.time()
>     for i in range(times):
>         a=ctor(i)
>     return (time.time() - start)
>
> def addition_test(ctor, times):
>     b = ctor('12345678901234567890')
>     start = time.time()
>     for i in range(times):
>         a=ctor(i) + b
>     return (time.time() - start)
>
> print "create times"
> print "long:  ", create_test(long, 10000000)
> print "gmpy:  ", create_test(gmpy.mpz, 10000000)
> print "gmpy3: ", create_test(gmpy3.mpz, 10000000)
> print
> print "addition times"
> print "long:  ", addition_test(long, 10000000)
> print "gmpy:  ", addition_test(gmpy.mpz, 10000000)
> print "gmpy3: ", addition_test(gmpy3.mpz, 10000000)
>
> -----------------------------------------------------------------
>
> Excerpts from gmpy3.pyx
>
> cdef class mpz:
>     """
>     Wrapper for GMP multiple-precision integers.
>     """
>
>     cdef mpz_t z
>
>     def __init__(self, val=None, base=10):
>         """
>         Initialize a variable of type mpz.
>         """
>         cdef _longobject *l
>
>         if val is None:
>             mpz_init(self.z)
>         elif PyInt_CheckExact(val):
>             mpz_init_set_si(self.z, val)
>         elif PyString_CheckExact(val):
>             if mpz_init_set_str(self.z, val, base):
>                 raise ValueError('Invalid literal for mpz() with base
> ' + str(base))
>         elif PyLong_CheckExact(val):
>             mpz_init(self.z)
>             l = <_longobject *> val
>             if l.ob_size < 0:
>                 negval = True
>                 length = - l.ob_size
>             else:
>                 negval = False
>                 length = l.ob_size
>             mpz_import(self.z, length, -1, sizeof(l.ob_digit[0]), 0,
> sizeof(l.ob_digit[0])*8 - SHIFT, l.ob_digit)
>             if negval:
>                 mpz_neg(self.z, self.z)
>         elif PyFloat_CheckExact(val):
>             mpz_init_set_d(self.z, val)
>         elif isinstance(val, mpz):
>             mpz_init_set(self.z, (<mpz>val).z)
>         else:
>             raise TypeError('mpz() argument must be string, int, long,
> or float')
>
>     def __dealloc__(self):
>         mpz_clear(self.z)
>
>     def __add__(self, other):
>         cdef int temp
>         result = mpz()
>         cdef mpz Z
>
>         if isinstance(self, mpz):
>             if isinstance(other, mpz):
>                 mpz_add((<mpz>result).z, (<mpz>self).z,  
> (<mpz>other).z)
>             elif PyInt_CheckExact(other):
>                 if other >= 0:
>                     mpz_add_ui((<mpz>result).z, (<mpz>self).z, other)
>                 else:
>                     mpz_sub_ui((<mpz>result).z, (<mpz>self).z, abs 
> (other))
>             else:
>                 mpz_add((<mpz>result).z, (<mpz>self).z, (<mpz>mpz 
> (other)).z)
>             return result
>         else:
>             return mpz.__add__(other, self)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Thu Oct 30 18:43:28 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 30 Oct 2008 10:43:28 -0700
Subject: [Cython] Cython 0.9.8.2 beta
Message-ID: <1B9C1C49-B178-4A67-AE75-A97E38901131@math.washington.edu>

There are lots of new things in cython-devel, and we're overdue for a  
release. I have posed a beta up at http://cython.org/ 
Cython-0.9.8.2.beta.tar.gz based on the current devel. Sage compiles  
and passes all tests, please try it out on your own projects.

- Robert


From stephane.drouard at st.com  Thu Oct 30 19:14:35 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Thu, 30 Oct 2008 19:14:35 +0100
Subject: [Cython] Class module names do not respect package hierarchy
In-Reply-To: <4908F4F5.4070809@canterbury.ac.nz>
Message-ID: <004301c93abb$5da109a0$b9ad810a@gnb.st.com>

Greg Ewing wrote:

> When a module cimports an extension type from another
> module, code is generated to import the type object at
> run time. To do this, the Pyrex compiler needs to know
> where the other module is in the module namespace. If
> it gets moved somewhere else, the import will fail.

Yes, and this is how Python behaves. A module which imports a sub-module
needs to know how to get it.

So having "cimport pkg.foo" that will be translated to "import pkg.foo" is
normal.

My request is different: I would remove the constraint to fix modules'
location (not the sub-modules they eventually import) at compile time.

So when we compile foo.pyx, we don't have to inform Pyrex that foo.pyd will
be in pkg.


From jasone at canonware.com  Fri Oct 31 01:21:00 2008
From: jasone at canonware.com (Jason Evans)
Date: Thu, 30 Oct 2008 17:21:00 -0700
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <48BB43FD.7000907@canterbury.ac.nz>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>	<20080827224353.f641a024.simon@arrowtheory.com>	<e15ccc4d0808271955v30b03f49s83f1c29ae44ed9e6@mail.gmail.com>	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>	<48BA6509.6080905@behnel.de>
	<48BB43FD.7000907@canterbury.ac.nz>
Message-ID: <490A4F6C.8050701@canonware.com>

Greg Ewing wrote:
> The idea of allowing __init__.pyx was so that the main
> code of a package could be written in Pyrex.
> 
> However, I've never actually tested whether Python
> recognises an __init__.so file as a package main
> file, so I'm not sure if this works.

This is an issue for me, since I would like to be able to structure my 
packages/modules something like the following (incomplete) tree:

   Parsing.so
   Crux/
        __init__.so (Cythonized Crux)
        Config.py
        DistMatrix.so
        Tree/
             __init__.so (Cythonized Crux.Tree)
             Parsimony.so

If it were possible to get Python to load __init__.so, everything would 
be great, but Python reports:

   ImportError: No module named Crux

I tried using the dotted file naming convention instead of the directory 
hierarchy:

   Parsing.pyx
   Crux.pyx
   Crux.Config.py
   Crux.DistMatrix.pyx
   Crux.Tree.pyx

Unfortunately, distutils turns that into:

   Parsing.so
   Crux.so
   Crux/
        Config.py
        DistMatrix.so
        Tree.so

This doesn't work because the Crux directory is not a package directory. 
  If I add a Crux/__init__.py, then Crux.so is ignored.

In fact, I haven't been able to find *any* way to get Crux to be a 
Cythonized package.  As a data point, Sage apparently uses a combination 
of empty __init__.py files and all.py files, but never creates 
Cythonized packages.  Also, http://wiki.cython.org/PackageHierarchy 
narrowly avoids what I'm trying to do.  My searches for other examples 
hasn't turned up anything more useful than this email thread.

Is there a way to nest Cythonized modules?  To be clear, I would like to 
be able to create all of the following as Cythonized loadable 
packages/modules:

   Crux
   Crux.Tree
   Crux.Tree.Parsimony

Thanks,
Jason

From jasone at canonware.com  Fri Oct 31 03:11:11 2008
From: jasone at canonware.com (Jason Evans)
Date: Thu, 30 Oct 2008 19:11:11 -0700
Subject: [Cython] Minor code generation problem
Message-ID: <490A693F.2020905@canonware.com>

Using the latest cython-dev, the following program causes a C compiler 
warning:

---------------------------------------------------------------------
cdef foo():
     cdef x = 42

     if x == 42 and x == 43:
         print "foo"

foo()
---------------------------------------------------------------------
lyken:~> cython foo.pyx
lyken:~> gcc -g -shared -pthread -fPIC -fwrapv -Wall 
-fno-strict-aliasing -I/usr/local/python/dbg/include/python2.6 foo.c 
-ofoo.so
foo.c: In function ?__pyx_f_3foo_foo?:
foo.c:214: warning: unused variable ?__pyx_3?
---------------------------------------------------------------------

The conditions under which this code generation problem show up are 
specific to the form of the compound conditional.

Jason

From dalcinl at gmail.com  Fri Oct 31 03:28:28 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 30 Oct 2008 23:28:28 -0300
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <490A4F6C.8050701@canonware.com>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>
	<20080827224353.f641a024.simon@arrowtheory.com>
	<e15ccc4d0808271955v30b03f49s83f1c29ae44ed9e6@mail.gmail.com>
	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>
	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>
	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>
	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>
	<48BA6509.6080905@behnel.de> <48BB43FD.7000907@canterbury.ac.nz>
	<490A4F6C.8050701@canonware.com>
Message-ID: <e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>

If all you can accept some vile hackery, see the attached tarball ;-) .

Python recognizes a PKGNAME directory as a pakage if '__init__.py' is
there, if not, no way. BUT  if '__init__.so' is also there, it loads
'__init__.so' !!! But then the exported module init function in the
dynlib needs to be named initPKGNAME, and "PKGNAME" needs to be passed
to Py_InitModule()

So, Greg, here you have the rules if you want to implement it ;-).

PS: tried only in Py 2.5, this is surely undocumented, and probably it
is in fact some bugy code in CPython's  'import.c' .



On Thu, Oct 30, 2008 at 9:21 PM, Jason Evans <jasone at canonware.com> wrote:
> Greg Ewing wrote:
>> The idea of allowing __init__.pyx was so that the main
>> code of a package could be written in Pyrex.
>>
>> However, I've never actually tested whether Python
>> recognises an __init__.so file as a package main
>> file, so I'm not sure if this works.
>
> This is an issue for me, since I would like to be able to structure my
> packages/modules something like the following (incomplete) tree:
>
>   Parsing.so
>   Crux/
>        __init__.so (Cythonized Crux)
>        Config.py
>        DistMatrix.so
>        Tree/
>             __init__.so (Cythonized Crux.Tree)
>             Parsimony.so
>
> If it were possible to get Python to load __init__.so, everything would
> be great, but Python reports:
>
>   ImportError: No module named Crux
>
> I tried using the dotted file naming convention instead of the directory
> hierarchy:
>
>   Parsing.pyx
>   Crux.pyx
>   Crux.Config.py
>   Crux.DistMatrix.pyx
>   Crux.Tree.pyx
>
> Unfortunately, distutils turns that into:
>
>   Parsing.so
>   Crux.so
>   Crux/
>        Config.py
>        DistMatrix.so
>        Tree.so
>
> This doesn't work because the Crux directory is not a package directory.
>  If I add a Crux/__init__.py, then Crux.so is ignored.
>
> In fact, I haven't been able to find *any* way to get Crux to be a
> Cythonized package.  As a data point, Sage apparently uses a combination
> of empty __init__.py files and all.py files, but never creates
> Cythonized packages.  Also, http://wiki.cython.org/PackageHierarchy
> narrowly avoids what I'm trying to do.  My searches for other examples
> hasn't turned up anything more useful than this email thread.
>
> Is there a way to nest Cythonized modules?  To be clear, I would like to
> be able to create all of the following as Cythonized loadable
> packages/modules:
>
>   Crux
>   Crux.Tree
>   Crux.Tree.Parsimony
>
> Thanks,
> Jason
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Oct 31 03:29:16 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 30 Oct 2008 23:29:16 -0300
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>
	<e15ccc4d0808271955v30b03f49s83f1c29ae44ed9e6@mail.gmail.com>
	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>
	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>
	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>
	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>
	<48BA6509.6080905@behnel.de> <48BB43FD.7000907@canterbury.ac.nz>
	<490A4F6C.8050701@canonware.com>
	<e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
Message-ID: <e7ba66e40810301929rc8c9c2hb27775aa1bd6e131@mail.gmail.com>

Sorry, forgot to attach...

On Thu, Oct 30, 2008 at 11:28 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> If all you can accept some vile hackery, see the attached tarball ;-) .
>
> Python recognizes a PKGNAME directory as a pakage if '__init__.py' is
> there, if not, no way. BUT  if '__init__.so' is also there, it loads
> '__init__.so' !!! But then the exported module init function in the
> dynlib needs to be named initPKGNAME, and "PKGNAME" needs to be passed
> to Py_InitModule()
>
> So, Greg, here you have the rules if you want to implement it ;-).
>
> PS: tried only in Py 2.5, this is surely undocumented, and probably it
> is in fact some bugy code in CPython's  'import.c' .
>
>
>
> On Thu, Oct 30, 2008 at 9:21 PM, Jason Evans <jasone at canonware.com> wrote:
>> Greg Ewing wrote:
>>> The idea of allowing __init__.pyx was so that the main
>>> code of a package could be written in Pyrex.
>>>
>>> However, I've never actually tested whether Python
>>> recognises an __init__.so file as a package main
>>> file, so I'm not sure if this works.
>>
>> This is an issue for me, since I would like to be able to structure my
>> packages/modules something like the following (incomplete) tree:
>>
>>   Parsing.so
>>   Crux/
>>        __init__.so (Cythonized Crux)
>>        Config.py
>>        DistMatrix.so
>>        Tree/
>>             __init__.so (Cythonized Crux.Tree)
>>             Parsimony.so
>>
>> If it were possible to get Python to load __init__.so, everything would
>> be great, but Python reports:
>>
>>   ImportError: No module named Crux
>>
>> I tried using the dotted file naming convention instead of the directory
>> hierarchy:
>>
>>   Parsing.pyx
>>   Crux.pyx
>>   Crux.Config.py
>>   Crux.DistMatrix.pyx
>>   Crux.Tree.pyx
>>
>> Unfortunately, distutils turns that into:
>>
>>   Parsing.so
>>   Crux.so
>>   Crux/
>>        Config.py
>>        DistMatrix.so
>>        Tree.so
>>
>> This doesn't work because the Crux directory is not a package directory.
>>  If I add a Crux/__init__.py, then Crux.so is ignored.
>>
>> In fact, I haven't been able to find *any* way to get Crux to be a
>> Cythonized package.  As a data point, Sage apparently uses a combination
>> of empty __init__.py files and all.py files, but never creates
>> Cythonized packages.  Also, http://wiki.cython.org/PackageHierarchy
>> narrowly avoids what I'm trying to do.  My searches for other examples
>> hasn't turned up anything more useful than this email thread.
>>
>> Is there a way to nest Cythonized modules?  To be clear, I would like to
>> be able to create all of the following as Cythonized loadable
>> packages/modules:
>>
>>   Crux
>>   Crux.Tree
>>   Crux.Tree.Parsimony
>>
>> Thanks,
>> Jason
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mypkg.tar.gz
Type: application/x-gzip
Size: 406 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20081030/e02ee8fa/attachment.bin 

From greg.ewing at canterbury.ac.nz  Fri Oct 31 06:26:10 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 31 Oct 2008 18:26:10 +1300
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <490A4F6C.8050701@canonware.com>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>
	<20080827224353.f641a024.simon@arrowtheory.com>
	<e15ccc4d0808271955v30b03f49s83f1c29ae44ed9e6@mail.gmail.com>
	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>
	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>
	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>
	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>
	<48BA6509.6080905@behnel.de> <48BB43FD.7000907@canterbury.ac.nz>
	<490A4F6C.8050701@canonware.com>
Message-ID: <490A96F2.4070907@canterbury.ac.nz>

Jason Evans wrote:

> If it were possible to get Python to load __init__.so, everything would 
> be great, but Python reports:
> 
>    ImportError: No module named Crux

You might be able to do something with an __init__.py that
explicitly imports the __init__.so and replaces itself in
sys.modules.

In the long run, this is an issue that could perhaps be
put forward as a feature request on python-dev.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Oct 31 06:37:40 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 31 Oct 2008 18:37:40 +1300
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>
	<20080827224353.f641a024.simon@arrowtheory.com>
	<e15ccc4d0808271955v30b03f49s83f1c29ae44ed9e6@mail.gmail.com>
	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>
	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>
	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>
	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>
	<48BA6509.6080905@behnel.de> <48BB43FD.7000907@canterbury.ac.nz>
	<490A4F6C.8050701@canonware.com>
	<e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
Message-ID: <490A99A4.8050502@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Python recognizes a PKGNAME directory as a pakage if '__init__.py' is
> there, if not, no way. BUT  if '__init__.so' is also there, it loads
> '__init__.so' !!! But then the exported module init function in the
> dynlib needs to be named initPKGNAME, and "PKGNAME" needs to be passed
> to Py_InitModule()
> 
> So, Greg, here you have the rules if you want to implement it ;-).

Okay, thanks. Sounds like I need to special-case the module
name when the last component is "__init__", or something like
that. I'll put it on the list of things to look into!

> PS: tried only in Py 2.5, this is surely undocumented, and probably it
> is in fact some bugy code in CPython's  'import.c' .

Yes, it would be better to get it made an officially supported
behaviour -- and to eliminate the need for the __init__.py to
be there as well.

-- 
Greg

From casevh at gmail.com  Fri Oct 31 07:31:15 2008
From: casevh at gmail.com (Case Vanhorsen)
Date: Thu, 30 Oct 2008 23:31:15 -0700
Subject: [Cython] Performance question (long)
In-Reply-To: <1E178D03-B2CC-488E-B590-4C316A6BD114@math.washington.edu>
References: <99e0b9530810292308n3f8cd738w84b5f711dc2b74a7@mail.gmail.com>
	<1E178D03-B2CC-488E-B590-4C316A6BD114@math.washington.edu>
Message-ID: <99e0b9530810302331y68ef6643we901ca4c455f2e06@mail.gmail.com>

Thanks for the fast reply!

On Wed, Oct 29, 2008 at 11:57 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Oct 29, 2008, at 11:08 PM, Case Vanhorsen wrote:
>
>> I help maintain the gmpy library and I've been experimenting with a
>> Cython-based wrapper for GMP. I've done simple performance tests with
>> an early version of the library and it is slower than the C-based gmpy
>> library. I compared object creation and addition times for Python
>> longs, gmpy, and gmpy3 (the experimental library). These are my
>> results, in seconds (code is below):
>>
>> create times
>> long:   2.25385284424
>> gmpy:   1.70728683472
>> gmpy3:  2.3947558403
>>
>> addition times
>> long:   3.03700900078
>> gmpy:   2.6448340416
>> gmpy3:  4.70184206963
>
> Interesting.
>
>> Do you have any suggestions for improving the performance?
>
> It looks like gmpy.c doesn't do any type checking for add, which
> could be a cause for concern (but also make things faster). Is this
> with cython-devel or the release version? The isinstance function
> will be much faster in the next release of Cython, as is the argument
> parsing, so that should help too. Probably the most significant thing
> you could do is construct the mpz class directly rather than call mpz
> () (e.g. see the PY_NEW macro at http://www.sagemath.org/hg/sage-main/
> file/3859ace86968/c_lib/include/stdsage.h for one way to do that).
> Also, type your "base" parameter to be an int in the __init__ method.
>
By changing "result=mpz()" to "result=PY_NEW(mpz)", the running time
for addition improved to ~4.0 seconds. I actually used the PY_NEW
macro from http://lists.copyleft.no/pipermail/pyrex/2007-November/002994.html.

I upgraded to the just posted beta release. No significant change.

>>
>> Thanks,
>>
>> Case

Thanks again,

Case

From stefan_ml at behnel.de  Fri Oct 31 08:46:03 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 31 Oct 2008 08:46:03 +0100 (CET)
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>
	<20080827224353.f641a024.simon@arrowtheory.com>
	<e15ccc4d0808271955v30b03f49s83f1c29ae44ed9e6@mail.gmail.com>
	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>
	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>
	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>
	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>
	<48BA6509.6080905@behnel.de> <48BB43FD.7000907@canterbury.ac.nz>
	<490A4F6C.8050701@canonware.com>
	<e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
Message-ID: <46999.213.61.181.86.1225439163.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> Python recognizes a PKGNAME directory as a pakage if '__init__.py' is
> there, if not, no way. BUT  if '__init__.so' is also there, it loads
> '__init__.so' !!! But then the exported module init function in the
> dynlib needs to be named initPKGNAME, and "PKGNAME" needs to be passed
> to Py_InitModule()
>
> PS: tried only in Py 2.5, this is surely undocumented, and probably it
> is in fact some bugy code in CPython's  'import.c' .

What makes you think that this is different from the normal behaviour of
loading binary modules before loading Python modules?

I think there's a difference between "recognising a package as such when
finding an __init__.py" and "loading the package module". The latter
should use the normal module loading machinery for loading the module
"PKGNAME.__init__" (note the missing .py).

Stefan


From stefan_ml at behnel.de  Fri Oct 31 09:17:02 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 31 Oct 2008 09:17:02 +0100 (CET)
Subject: [Cython] Performance question (long)
In-Reply-To: <99e0b9530810302331y68ef6643we901ca4c455f2e06@mail.gmail.com>
References: <99e0b9530810292308n3f8cd738w84b5f711dc2b74a7@mail.gmail.com>
	<1E178D03-B2CC-488E-B590-4C316A6BD114@math.washington.edu>
	<99e0b9530810302331y68ef6643we901ca4c455f2e06@mail.gmail.com>
Message-ID: <45616.213.61.181.86.1225441022.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Case Vanhorsen wrote:
> On Wed, Oct 29, 2008 at 11:57 PM, Robert Bradshaw
>>> addition times
>>> long:   3.03700900078
>>> gmpy:   2.6448340416
>>> gmpy3:  4.70184206963
>>
>> Probably the most significant thing
>> you could do is construct the mpz class directly rather than call mpz
>> () (e.g. see the PY_NEW macro at http://www.sagemath.org/hg/sage-main/
>> file/3859ace86968/c_lib/include/stdsage.h for one way to do that).
>> Also, type your "base" parameter to be an int in the __init__ method.
>>
> By changing "result=mpz()" to "result=PY_NEW(mpz)", the running time
> for addition improved to ~4.0 seconds.

Have you tried creating a fast path for mpz values by doing a direct type
test instead of calling isinstance()? Cython (like Python 2.6+) provides a
C macro "object Py_TYPE(object)", that you can use with "is". Just cdef it
in your code to use it.

Also note that the special case for int objects in your __add__ method may
not be as fast as you might think. It will generate some additional type
testing code and will do the ">= 0" comparison and the abs() call through
the Python API. Read the generated C code to see what I mean. Try
converting the value to a C int right after the type test, before doing
anything else, and use the abs() function in math.h (although this will
stop working in Py3, where int is long, i.e. no longer compatible with a C
int).


> I actually used the PY_NEW macro from
> http://lists.copyleft.no/pipermail/pyrex/2007-November/002994.html.

That macro constructs a global empty tuple reference on the fly. Cython
already does that for you, so if you want to cheat, use
"__pyx_empty_tuple" instead.

(That's not really an official feature, but your compiler will notice it
if it ever changes...)

Stefan


From stefan_ml at behnel.de  Fri Oct 31 09:22:18 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 31 Oct 2008 09:22:18 +0100 (CET)
Subject: [Cython] Minor code generation problem
In-Reply-To: <490A693F.2020905@canonware.com>
References: <490A693F.2020905@canonware.com>
Message-ID: <52004.213.61.181.86.1225441338.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Jason Evans wrote:
> Using the latest cython-dev, the following program causes a C compiler
> warning:
>
> ---------------------------------------------------------------------
> cdef foo():
>      cdef x = 42
>
>      if x == 42 and x == 43:
>          print "foo"
>
> foo()
> ---------------------------------------------------------------------
> foo.c: In function ?__pyx_f_3foo_foo?:
> foo.c:214: warning: unused variable ?__pyx_3?

I've seen things like this happen when the switch transform optimisation
hits. It can leave already allocated temp variables unused.

Does anyone know if there is an easy way to move the temp allocation to a
later time in cases like this? It would be best to move it after the tree
optimisation transformations.

Stefan


From dagss at student.matnat.uio.no  Fri Oct 31 11:58:33 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 31 Oct 2008 11:58:33 +0100
Subject: [Cython] Minor code generation problem
In-Reply-To: <52004.213.61.181.86.1225441338.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <490A693F.2020905@canonware.com>
	<52004.213.61.181.86.1225441338.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <490AE4D9.4030909@student.matnat.uio.no>

Stefan Behnel wrote:
> Jason Evans wrote:
>> Using the latest cython-dev, the following program causes a C compiler
>> warning:
>>
>> ---------------------------------------------------------------------
>> cdef foo():
>>      cdef x = 42
>>
>>      if x == 42 and x == 43:
>>          print "foo"
>>
>> foo()
>> ---------------------------------------------------------------------
>> foo.c: In function ?__pyx_f_3foo_foo?:
>> foo.c:214: warning: unused variable ?__pyx_3?
> 
> I've seen things like this happen when the switch transform optimisation
> hits. It can leave already allocated temp variables unused.

However x isn't C-typed so the switch transform shouldn't kick in here?

> Does anyone know if there is an easy way to move the temp allocation to a
> later time in cases like this? It would be best to move it after the tree
> optimisation transformations.

First of all, temp allocation is a bit fragile, and I strongly recommend 
waiting until after the release before fixing this one.

There's plans to move all temp allocation to code generation. There's 
two things one can use, depending on the case:

1) Manually allocated temps can be allocated through the code.funcscope 
object instead of the env object

2) I made a transition class "NewTempExprNode" (near top of 
ExprNodes.py) which does temp allocation during code generation, after 
all tranforms have been run.

To start having the temporary result of an expression node be allocated 
later in the pipeline, the class should simply inherit from 
NewTempsExprNode rather than ExprNode.

This does however not work with some expression nodes which makes 
assumptions about earlier availability of the temp expression, which 
have to be refactored and/or rewritten first.

Once all ExprNodes works using this node as ancestor, its alternative 
temp implementation should be folded into ExprNode and the class removed.


-- 
Dag Sverre

From stephane.drouard at st.com  Fri Oct 31 14:00:01 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Fri, 31 Oct 2008 14:00:01 +0100
Subject: [Cython] Scope rules vs. Python behaviour
Message-ID: <003501c93b58$97238ca0$b9ad810a@gnb.st.com>

Hello,

In the documentation it is written that Cython defaults to the built-in scope when it can't determine the scope of a variable, which is different from Python, which defaults to the module scope.

As a consequence, Cython needs module variables to be declared global to behave like under Python, resulting in a compatibility issue.


Would it be possible to implement a compatibility mode, where Cython would behave as much as possible like Python?

Concerning undetermined variables, this could:
 - not cache them,
 - use a version of __Pyx_GetName() like below:

static PyObject *__Pyx_GetName(PyObject *dict, PyObject *name) {
    PyObject *result;
    result = PyObject_GetAttr(dict, name);
    // If not found and the dictionary is the module, try in builtin.
    if (!result && dict == __pyx_m) {
        PyErr_Clear();
        result = PyObject_GetAttr(__pyx_b, name);
    }
    if (!result)
        PyErr_SetObject(PyExc_NameError, name);
    return result;
}


An improvement could be:
 - declare them (static PyObject *__pyx_builtin_xxx = NULL) but do not initialize them through __Pyx_InitCachedBuiltins(),
 - access them through a function that first checks if the variable is initialised (!= NULL), so return it, otherwise do something like above,
 - if the attribute is got from builtin, update the pointer.


During the tests, I remarked that:

import __builtin__
print __builtin__.__name__

__name__ is not cached. A possible improvement ;-)

Cheers,
Stephane


From dalcinl at gmail.com  Fri Oct 31 15:22:28 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 31 Oct 2008 11:22:28 -0300
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <490A99A4.8050502@canterbury.ac.nz>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>
	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>
	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>
	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>
	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>
	<48BA6509.6080905@behnel.de> <48BB43FD.7000907@canterbury.ac.nz>
	<490A4F6C.8050701@canonware.com>
	<e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
	<490A99A4.8050502@canterbury.ac.nz>
Message-ID: <e7ba66e40810310722r4ac7de62kb4d67587cb9c70c4@mail.gmail.com>

On Fri, Oct 31, 2008 at 2:37 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Okay, thanks. Sounds like I need to special-case the module
> name when the last component is "__init__", or something like
> that. I'll put it on the list of things to look into!

If you can point me the place to start looking, perhaps I can help.

>> PS: tried only in Py 2.5, this is surely undocumented, and probably it
>> is in fact some bugy code in CPython's  'import.c' .
>
> Yes, it would be better to get it made an officially supported
> behaviour -- and to eliminate the need for the __init__.py to
> be there as well.

You are right, feel free to discuss this on Python-Dev. Anyway, the
__init__.py file will be still required for backward compatibility
reasons.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Oct 31 15:27:02 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 31 Oct 2008 11:27:02 -0300
Subject: [Cython] 0.9.8.1.1 and .pxd files
In-Reply-To: <46999.213.61.181.86.1225439163.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e15ccc4d0808270930h3328f8ebhfcf216bb5b652dd8@mail.gmail.com>
	<e15ccc4d0808272026l58ea73f6lf1c41f4c0b50bcdf@mail.gmail.com>
	<e7ba66e40808280727r5c746fa0g5a5cbf6fd112e2e4@mail.gmail.com>
	<e15ccc4d0808281505r4bcac6f2qc03591184e8cd77f@mail.gmail.com>
	<e7ba66e40808281540j2aac6d75jbf17cf2410589c17@mail.gmail.com>
	<48BA6509.6080905@behnel.de> <48BB43FD.7000907@canterbury.ac.nz>
	<490A4F6C.8050701@canonware.com>
	<e7ba66e40810301928k454d1b08q3eda881880d3bd71@mail.gmail.com>
	<46999.213.61.181.86.1225439163.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40810310727w7c6689d9j18ad60becee8198b@mail.gmail.com>

On Fri, Oct 31, 2008 at 4:46 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin wrote:
>> Python recognizes a PKGNAME directory as a pakage if '__init__.py' is
>> there, if not, no way. BUT  if '__init__.so' is also there, it loads
>> '__init__.so' !!! But then the exported module init function in the
>> dynlib needs to be named initPKGNAME, and "PKGNAME" needs to be passed
>> to Py_InitModule()
>>
>> PS: tried only in Py 2.5, this is surely undocumented, and probably it
>> is in fact some bugy code in CPython's  'import.c' .
>
> What makes you think that this is different from the normal behaviour of
> loading binary modules before loading Python modules?
>
> I think there's a difference between "recognising a package as such when
> finding an __init__.py" and "loading the package module". The latter
> should use the normal module loading machinery for loading the module
> "PKGNAME.__init__" (note the missing .py).
>

Well, it was just the name of the module init function being
initPKGNAME. But thinking a bit more, perhaps this do really make
sense, and all is fine.

I just was too lize to look at 'import.c', it is a not easy place to
dive into near the midnight ;-).

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Oct 31 17:25:43 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 31 Oct 2008 09:25:43 -0700
Subject: [Cython] Minor code generation problem
In-Reply-To: <490AE4D9.4030909@student.matnat.uio.no>
References: <490A693F.2020905@canonware.com>
	<52004.213.61.181.86.1225441338.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<490AE4D9.4030909@student.matnat.uio.no>
Message-ID: <F2134305-F374-4F85-879D-32F093958366@math.washington.edu>

On Oct 31, 2008, at 3:58 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Jason Evans wrote:
>>> Using the latest cython-dev, the following program causes a C  
>>> compiler
>>> warning:
>>>
>>> -------------------------------------------------------------------- 
>>> -
>>> cdef foo():
>>>      cdef x = 42
>>>
>>>      if x == 42 and x == 43:
>>>          print "foo"
>>>
>>> foo()
>>> -------------------------------------------------------------------- 
>>> -
>>> foo.c: In function ?__pyx_f_3foo_foo?:
>>> foo.c:214: warning: unused variable ?__pyx_3?
>>
>> I've seen things like this happen when the switch transform  
>> optimisation
>> hits. It can leave already allocated temp variables unused.
>
> However x isn't C-typed so the switch transform shouldn't kick in  
> here?

I believe this is because it never needs to store the Python result  
of the and--the expressions are turned into c bints and the and is  
done in pure C. This is an optimization I did a while back.

>
>> Does anyone know if there is an easy way to move the temp  
>> allocation to a
>> later time in cases like this? It would be best to move it after  
>> the tree
>> optimisation transformations.
>
> First of all, temp allocation is a bit fragile, and I strongly  
> recommend
> waiting until after the release before fixing this one.
>
> There's plans to move all temp allocation to code generation. There's
> two things one can use, depending on the case:
>
> 1) Manually allocated temps can be allocated through the  
> code.funcscope
> object instead of the env object
>
> 2) I made a transition class "NewTempExprNode" (near top of
> ExprNodes.py) which does temp allocation during code generation, after
> all tranforms have been run.
>
> To start having the temporary result of an expression node be  
> allocated
> later in the pipeline, the class should simply inherit from
> NewTempsExprNode rather than ExprNode.
>
> This does however not work with some expression nodes which makes
> assumptions about earlier availability of the temp expression, which
> have to be refactored and/or rewritten first.
>
> Once all ExprNodes works using this node as ancestor, its alternative
> temp implementation should be folded into ExprNode and the class  
> removed.

It's going to be a big task, but things will be *much* nicer once  
temps are allocated at this later stage in code generation. Thanks  
Dag for all your work in this direction. Until then I wouldn't worry  
too much about unused variable declarations as long as they're just  
warnings and the code produced is still correct.

- Robert



From robertwb at math.washington.edu  Fri Oct 31 17:32:02 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 31 Oct 2008 09:32:02 -0700
Subject: [Cython] Scope rules vs. Python behaviour
In-Reply-To: <003501c93b58$97238ca0$b9ad810a@gnb.st.com>
References: <003501c93b58$97238ca0$b9ad810a@gnb.st.com>
Message-ID: <DB3D771A-4C03-4B9C-A53B-71E352056DCD@math.washington.edu>

On Oct 31, 2008, at 6:00 AM, Stephane DROUARD wrote:

> Hello,
>
> In the documentation it is written that Cython defaults to the  
> built-in scope when it can't determine the scope of a variable,  
> which is different from Python, which defaults to the module scope.

The documentation needs to be fixed then. If the scope of a variable  
can't be determined, then it actually raises a compile-time error.

> As a consequence, Cython needs module variables to be declared  
> global to behave like under Python, resulting in a compatibility  
> issue.

Could you give an example?

> Would it be possible to implement a compatibility mode, where  
> Cython would behave as much as possible like Python?

This is certainly possible, but would have a significant performance  
hit.

>
> Concerning undetermined variables, this could:
>  - not cache them,
>  - use a version of __Pyx_GetName() like below:
>
> static PyObject *__Pyx_GetName(PyObject *dict, PyObject *name) {
>     PyObject *result;
>     result = PyObject_GetAttr(dict, name);
>     // If not found and the dictionary is the module, try in builtin.
>     if (!result && dict == __pyx_m) {
>         PyErr_Clear();
>         result = PyObject_GetAttr(__pyx_b, name);
>     }
>     if (!result)
>         PyErr_SetObject(PyExc_NameError, name);
>     return result;
> }
>
>
> An improvement could be:
>  - declare them (static PyObject *__pyx_builtin_xxx = NULL) but do  
> not initialize them through __Pyx_InitCachedBuiltins(),
>  - access them through a function that first checks if the variable  
> is initialised (!= NULL), so return it, otherwise do something like  
> above,
>  - if the attribute is got from builtin, update the pointer.

Again, this is a performance hit, and wouldn't handle the case that a  
module-level variable with the name of a module global is assigned at  
some later point in the program.

> During the tests, I remarked that:
>
> import __builtin__
> print __builtin__.__name__
>
> __name__ is not cached. A possible improvement ;-)

But what if I did

import __builtin__
print __builtin__.__name__
foo()
print __builtin__.__name__

where foo (defined elsewhere) is

def foo():
     import __builtin__
     __builtin__.__name__ = "something else"

Thanks for your comments.

- Robert



From stephane.drouard at st.com  Fri Oct 31 18:46:07 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Fri, 31 Oct 2008 18:46:07 +0100
Subject: [Cython] Scope rules vs. Python behaviour
In-Reply-To: <DB3D771A-4C03-4B9C-A53B-71E352056DCD@math.washington.edu>
Message-ID: <00c301c93b80$8db54690$b9ad810a@gnb.st.com>

Robert Bradshaw wrote:

> > In the documentation it is written that Cython defaults to the  
> > built-in scope when it can't determine the scope of a variable,  
> > which is different from Python, which defaults to the module scope.
> 
> The documentation needs to be fixed then. If the scope of a variable  
> can't be determined, then it actually raises a compile-time error.

No, the example given in the documentation does not generate a compile-time
error.

foo.py:
print __name__

But under Python it returns "foo", "__builtin__" when cython'ized.

As the documentation specifies, this can be fixed by:

foo.py:
global __name__
print __name__

Then both return "foo".

> > Would it be possible to implement a compatibility mode, where  
> > Cython would behave as much as possible like Python?
> 
> This is certainly possible, but would have a significant performance  
> hit.

I agree, but would it be better to be faster or compatible? I would be in
favour to be compatible by default and set compiler options to optimize.
This is then the responsibility of the developper to check that the
optimizations are compatible with the use of the module.

> > During the tests, I remarked that:
> >
> > import __builtin__
> > print __builtin__.__name__
> >
> > __name__ is not cached. A possible improvement 
> 
> But what if I did
> 
> import __builtin__
> print __builtin__.__name__
> foo()
> print __builtin__.__name__
> 
> where foo (defined elsewhere) is
> 
> def foo():
>      import __builtin__
>      __builtin__.__name__ = "something else"

OK. But if I do:

print __name__
foo()
print __name__

>>> import __builtin__
>>> __builtin__.__name__
'__builtin__'
>>> __builtin__.__name__ = "blah"
>>> import foo
blah              ## confirms that Cython considers __name__ in  __builtin__
blah              ## update not reflected
>>> __builtin__.__name__
'something else'  ## whereas it is

I assume this is because Cython has cached __name__ at module creation.
(unfortunately I was not able to validate, setting Options.cache_builtins to
0 raises an exception)

Cheers,
Stephane


From robertwb at math.washington.edu  Fri Oct 31 18:58:29 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 31 Oct 2008 10:58:29 -0700
Subject: [Cython] Scope rules vs. Python behaviour
In-Reply-To: <00c301c93b80$8db54690$b9ad810a@gnb.st.com>
References: <00c301c93b80$8db54690$b9ad810a@gnb.st.com>
Message-ID: <CC50E8D8-12EE-457C-91B7-B6049CC163C8@math.washington.edu>

On Oct 31, 2008, at 10:46 AM, Stephane DROUARD wrote:

> Robert Bradshaw wrote:
>
>>> In the documentation it is written that Cython defaults to the
>>> built-in scope when it can't determine the scope of a variable,
>>> which is different from Python, which defaults to the module scope.
>>
>> The documentation needs to be fixed then. If the scope of a variable
>> can't be determined, then it actually raises a compile-time error.
>
> No, the example given in the documentation does not generate a  
> compile-time
> error.
>
> foo.py:
> print __name__
>
> But under Python it returns "foo", "__builtin__" when cython'ized.
>
> As the documentation specifies, this can be fixed by:
>
> foo.py:
> global __name__
> print __name__
>
> Then both return "foo".

Ah, good catch. Cython (and, incidentally I) didn't know that  
__name__ is an already declared module-level variable, hence it  
looked it up as a builtin (and found it there, so no error). This is  
easily fixed by always having __name__ in the module namespace (what  
other magic variables are there?)

>
>>> Would it be possible to implement a compatibility mode, where
>>> Cython would behave as much as possible like Python?
>>
>> This is certainly possible, but would have a significant performance
>> hit.
>
> I agree, but would it be better to be faster or compatible? I would  
> be in
> favour to be compatible by default and set compiler options to  
> optimize.
> This is then the responsibility of the developper to check that the
> optimizations are compatible with the use of the module.

It seems the current community values speed more, otherwise they  
would just be using Python directly. However, this is something we  
should continue to evaluate.

>>> During the tests, I remarked that:
>>>
>>> import __builtin__
>>> print __builtin__.__name__
>>>
>>> __name__ is not cached. A possible improvement
>>
>> But what if I did
>>
>> import __builtin__
>> print __builtin__.__name__
>> foo()
>> print __builtin__.__name__
>>
>> where foo (defined elsewhere) is
>>
>> def foo():
>>      import __builtin__
>>      __builtin__.__name__ = "something else"
>
> OK. But if I do:
>
> print __name__
> foo()
> print __name__
>
>>>> import __builtin__
>>>> __builtin__.__name__
> '__builtin__'
>>>> __builtin__.__name__ = "blah"
>>>> import foo
> blah              ## confirms that Cython considers __name__ in   
> __builtin__
> blah              ## update not reflected
>>>> __builtin__.__name__
> 'something else'  ## whereas it is
>
> I assume this is because Cython has cached __name__ at module  
> creation.
> (unfortunately I was not able to validate, setting  
> Options.cache_builtins to
> 0 raises an exception)

I just tried this, I get

__builtin__
something else

as expected. Cython does not "know" about __builtin__ being special,  
and treats it as any other module (no caching). This allows one to  
get "non-cached" versions of the builtins. The caching only happens  
when builtin names are used directly in the source, not when looked  
up as attributes.

We should delete that option, as now builtins are much more  
integrated into the compiler.

- Robert


From dalcinl at gmail.com  Fri Oct 31 19:17:25 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 31 Oct 2008 15:17:25 -0300
Subject: [Cython] Scope rules vs. Python behaviour
In-Reply-To: <CC50E8D8-12EE-457C-91B7-B6049CC163C8@math.washington.edu>
References: <00c301c93b80$8db54690$b9ad810a@gnb.st.com>
	<CC50E8D8-12EE-457C-91B7-B6049CC163C8@math.washington.edu>
Message-ID: <e7ba66e40810311117k6799d775r33c168b3efbc4300@mail.gmail.com>

On Fri, Oct 31, 2008 at 2:58 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Ah, good catch. Cython (and, incidentally I) didn't know that
> __name__ is an already declared module-level variable, hence it
> looked it up as a builtin (and found it there, so no error). This is
> easily fixed by always having __name__ in the module namespace (what
> other magic variables are there?)

$ touch foo.py
$ python2.5 -c 'import foo; print foo.__dict__.keys()'
['__builtins__', '__name__', '__file__', '__doc__']


> It seems the current community values speed more, otherwise they
> would just be using Python directly.

Indeed

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Oct 31 19:22:26 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 31 Oct 2008 11:22:26 -0700
Subject: [Cython] Scope rules vs. Python behaviour
In-Reply-To: <CC50E8D8-12EE-457C-91B7-B6049CC163C8@math.washington.edu>
References: <00c301c93b80$8db54690$b9ad810a@gnb.st.com>
	<CC50E8D8-12EE-457C-91B7-B6049CC163C8@math.washington.edu>
Message-ID: <6A1CD3C1-0EE9-4B9B-BAF9-49B6FA5EECBB@math.washington.edu>

On Oct 31, 2008, at 10:58 AM, Robert Bradshaw wrote:

> On Oct 31, 2008, at 10:46 AM, Stephane DROUARD wrote:
>
>> Robert Bradshaw wrote:
>>
>>>> In the documentation it is written that Cython defaults to the
>>>> built-in scope when it can't determine the scope of a variable,
>>>> which is different from Python, which defaults to the module scope.
>>>
>>> The documentation needs to be fixed then. If the scope of a variable
>>> can't be determined, then it actually raises a compile-time error.
>>
>> No, the example given in the documentation does not generate a
>> compile-time
>> error.
>>
>> foo.py:
>> print __name__
>>
>> But under Python it returns "foo", "__builtin__" when cython'ized.
>>
>> As the documentation specifies, this can be fixed by:
>>
>> foo.py:
>> global __name__
>> print __name__
>>
>> Then both return "foo".
>
> Ah, good catch. Cython (and, incidentally I) didn't know that
> __name__ is an already declared module-level variable, hence it
> looked it up as a builtin (and found it there, so no error). This is
> easily fixed by always having __name__ in the module namespace (what
> other magic variables are there?)

To clarify, other than these (thanks for the list Lisandro) the only  
time this can cause issues is if one manually adds names to the  
module externally, or if one uses "import *" to overwrite a builtin.

- Robert



From stephane.drouard at st.com  Fri Oct 31 21:38:38 2008
From: stephane.drouard at st.com (Stephane DROUARD)
Date: Fri, 31 Oct 2008 21:38:38 +0100
Subject: [Cython] Scope rules vs. Python behaviour
In-Reply-To: <e7ba66e40810311117k6799d775r33c168b3efbc4300@mail.gmail.com>
Message-ID: <000001c93b98$aac82140$1140fb0a@gnb.st.com>

Lisandro Dalcin wrote:

>> It seems the current community values speed more, otherwise they
>> would just be using Python directly.
>
> Indeed

Me too for sure.

The fact is that I'm currently porting pure Python and SWIG modules to
Cython.
Some issues I'm having are due to a different behaviour between Cython and
Python. And they are not so easy to find, because my first reaction is to
review my code, not to consider the code generated by the tool(s) may be
"guilty".

I would be happy to have switches at Cython level that ensure Python
compatibility, then enable optimizations, ideally one by one, helping me to
fastly identify where to look at.

Moreover, having thoses switches would let me decide whether I fix the issue
now or later on, according to my priorities.

Cheers,
Stephane


From robertwb at math.washington.edu  Fri Oct 31 22:11:01 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 31 Oct 2008 14:11:01 -0700
Subject: [Cython] Scope rules vs. Python behaviour
In-Reply-To: <000001c93b98$aac82140$1140fb0a@gnb.st.com>
References: <000001c93b98$aac82140$1140fb0a@gnb.st.com>
Message-ID: <09423042-A466-4A72-83A9-558830DC2A40@math.washington.edu>

On Oct 31, 2008, at 1:38 PM, Stephane DROUARD wrote:

> Lisandro Dalcin wrote:
>
>>> It seems the current community values speed more, otherwise they
>>> would just be using Python directly.
>>
>> Indeed
>
> Me too for sure.
>
> The fact is that I'm currently porting pure Python and SWIG modules to
> Cython.
> Some issues I'm having are due to a different behaviour between  
> Cython and
> Python. And they are not so easy to find, because my first reaction  
> is to
> review my code, not to consider the code generated by the tool(s)  
> may be
> "guilty".
>
> I would be happy to have switches at Cython level that ensure Python
> compatibility, then enable optimizations, ideally one by one,  
> helping me to
> fastly identify where to look at.
>
> Moreover, having thoses switches would let me decide whether I fix  
> the issue
> now or later on, according to my priorities.

I would be happy to have switches like this too, though for most  
cases the default would probably be set for speed rather than  
correctness.

- Robert