From dagss at student.matnat.uio.no  Sun Feb  1 00:28:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 01 Feb 2009 00:28:55 +0100
Subject: [Cython] char* and string object
In-Reply-To: <EAC9277B-CAE9-40FA-8EFF-DBEDE8521A46@me.com>
References: <EAC9277B-CAE9-40FA-8EFF-DBEDE8521A46@me.com>
Message-ID: <4984DEB7.5090101@student.matnat.uio.no>

Jean-Alexandre Peyroux wrote:
> Hello,
> 
> I think i did not understand the handling character string. This is an  
> example :
> 
> user.pyx :
> 
> cdef class User:
>     def __init__(self, char* name):
>         self.cname = name
> 
>     cpdef printName(self):
>         print "My name is %s" % self.cname
> 
> user.pxd :
> 
> cdef class User:
>     cdef readonly char* cname
> 
>     cpdef printName(self)
> 
> 
<snip>
> In [5]: u.cname
> Out[5]: '_3'
> 
> I do not understand. My string is completely random, as if it was  
> pointing in the wrong location of memory.
> I missed step ?

This is expected. When you do "self.cname = name", then you get a 
pointer to the character buffer inside the "name" string object. Then, 
"name" goes out of scope and is deallocated, along with its memory, and 
perhaps filled with something else.

Moral? Only use conversion to char* when you really need it because you 
call C code or similar, and also then you need to hold on to the 
corresponding Python string as well.

-- 
Dag Sverre

From peyroux at me.com  Sun Feb  1 00:23:36 2009
From: peyroux at me.com (Jean-Alexandre Peyroux)
Date: Sun, 01 Feb 2009 00:23:36 +0100
Subject: [Cython] char* and string object
In-Reply-To: <4984DEB7.5090101@student.matnat.uio.no>
References: <EAC9277B-CAE9-40FA-8EFF-DBEDE8521A46@me.com>
	<4984DEB7.5090101@student.matnat.uio.no>
Message-ID: <94C0DB9D-0AB5-413E-814E-DACD69DBB968@me.com>

Thank you for your advice.

Regards

Le 1 f?vr. 09 ? 00:28, Dag Sverre Seljebotn a ?crit :

> Jean-Alexandre Peyroux wrote:
>> Hello,
>>
>> I think i did not understand the handling character string. This is  
>> an
>> example :
>>
>> user.pyx :
>>
>> cdef class User:
>>    def __init__(self, char* name):
>>        self.cname = name
>>
>>    cpdef printName(self):
>>        print "My name is %s" % self.cname
>>
>> user.pxd :
>>
>> cdef class User:
>>    cdef readonly char* cname
>>
>>    cpdef printName(self)
>>
>>
> <snip>
>> In [5]: u.cname
>> Out[5]: '_3'
>>
>> I do not understand. My string is completely random, as if it was
>> pointing in the wrong location of memory.
>> I missed step ?
>
> This is expected. When you do "self.cname = name", then you get a
> pointer to the character buffer inside the "name" string object. Then,
> "name" goes out of scope and is deallocated, along with its memory,  
> and
> perhaps filled with something else.
>
> Moral? Only use conversion to char* when you really need it because  
> you
> call C code or similar, and also then you need to hold on to the
> corresponding Python string as well.
>
> -- 
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From wstein at gmail.com  Sun Feb  1 01:21:48 2009
From: wstein at gmail.com (William Stein)
Date: Sat, 31 Jan 2009 16:21:48 -0800
Subject: [Cython] http://shootout.alioth.debian.org/
Message-ID: <85e81ba30901311621h397e054cw9477f2d1b84b8f31@mail.gmail.com>

Hi,

Craig Citro just gave a very nice Cython talk (in lieu of Robert
Bradshaw), and there were about 30 questions from the audience, hence
a lot of interest:

   http://sage.math.washington.edu/talks/20090131-seapig/cython.pdf

One question that came up -- could somebody add cython to
http://shootout.alioth.debian.org/
?

It might be fun, since basically C wins every single one of the
language shootouts -- so Cython likely would do very well.

 -- William

-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From casevh at gmail.com  Mon Feb  2 08:38:26 2009
From: casevh at gmail.com (Case Vanhorsen)
Date: Sun, 1 Feb 2009 23:38:26 -0800
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
Message-ID: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>

Hello,

I just tried to install Cython 0.11 beta with Python 3.0. I get the
following error:

case:~/src/Cython-0.11.beta$ py30 setup.py install
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    from Cython.Compiler.Version import version
  File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
    from Shadow import *
ImportError: No module named Shadow

It does install successfully using Python 2.5 and 2.6. Please let me
know if there is any other information you need.

casevh

From dagss at student.matnat.uio.no  Mon Feb  2 10:20:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 10:20:27 +0100
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
In-Reply-To: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
References: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
Message-ID: <4986BADB.50202@student.matnat.uio.no>

Case Vanhorsen wrote:
> Hello,
>
> I just tried to install Cython 0.11 beta with Python 3.0. I get the
> following error:
>
> case:~/src/Cython-0.11.beta$ py30 setup.py install
> Traceback (most recent call last):
>   File "setup.py", line 5, in <module>
>     from Cython.Compiler.Version import version
>   File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
>     from Shadow import *
> ImportError: No module named Shadow
>
> It does install successfully using Python 2.5 and 2.6. Please let me
> know if there is any other information you need.
>   
I had the impression that Cython doesn't support being run under Python 
3 (though Stefan would have the definitive answer on that). I think you 
must install on Python 2.6, and then compile your .pyx to .c under 2.6, 
and THEN you can compile the c file with the Python 3 headers to get a 
Python 3 module.

Dag Sverre

From dalcinl at gmail.com  Mon Feb  2 18:12:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 14:12:19 -0300
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
Message-ID: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>

I'm now getting many warnings like below:

src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
src/petsc4py_PETSc.c:3793: warning: value computed is not used

A possible fix, written following the way core Python Py_XDECREF,
would be the following:


diff -r 83075bb3a319 Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py	Fri Jan 30 23:10:52 2009 +0100
+++ b/Cython/Compiler/ModuleNode.py	Mon Feb 02 14:08:33 2009 -0300
@@ -2334,7 +2334,7 @@
 #define __Pyx_GOTREF(r) __Pyx_Refnanny_GOTREF(__pyx_refchk, r, __LINE__)
 #define __Pyx_GIVEREF(r) __Pyx_Refnanny_GIVEREF(__pyx_refchk, r, __LINE__)
 #define __Pyx_DECREF(r) __Pyx_Refnanny_DECREF(__pyx_refchk, r, __LINE__)
-#define __Pyx_XDECREF(r) (r ? __Pyx_Refnanny_DECREF(__pyx_refchk, r,
__LINE__) : 0)
+#define __Pyx_XDECREF(r) if((r) == NULL) ; else __Pyx_DECREF(r)
 #define __Pyx_SetupRefcountContext(name) \
   void* __pyx_refchk = __Pyx_Refnanny_NewContext(name, __LINE__)
 #define __Pyx_FinishRefcountContext()
__Pyx_Refnanny_FinishContext(__pyx_refchk)
@@ -2347,6 +2347,6 @@
 #define __Pyx_SetupRefcountContext(name)
 #define __Pyx_FinishRefcountContext() 0
 #endif /* CYTHON_REFNANNY */
-#define __Pyx_XGIVEREF(r) (r ? __Pyx_GIVEREF(r) : 0)
-#define __Pyx_XGOTREF(r) (r ? __Pyx_GOTREF(r) : 0)
+#define __Pyx_XGIVEREF(r) if((r) == NULL) ; else __Pyx_GIVEREF(r)
+#define __Pyx_XGOTREF(r) if((r) == NULL) ; else __Pyx_GOTREF(r)
 """)


Can I push this?

Other possible way would be to use "do { ... } while(0)" blocks, I
prefer them much more, but that is not a common pattern in Python C
source code.




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb  2 18:31:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 18:31:54 +0100
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
In-Reply-To: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
References: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
Message-ID: <49872E0A.3090801@student.matnat.uio.no>

Lisandro Dalcin wrote:
> I'm now getting many warnings like below:
> 
> src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
> src/petsc4py_PETSc.c:3793: warning: value computed is not used
> 
> A possible fix, written following the way core Python Py_XDECREF,
> would be the following:
> 
> 
> diff -r 83075bb3a319 Cython/Compiler/ModuleNode.py
> --- a/Cython/Compiler/ModuleNode.py	Fri Jan 30 23:10:52 2009 +0100
> +++ b/Cython/Compiler/ModuleNode.py	Mon Feb 02 14:08:33 2009 -0300
> @@ -2334,7 +2334,7 @@
>  #define __Pyx_GOTREF(r) __Pyx_Refnanny_GOTREF(__pyx_refchk, r, __LINE__)
>  #define __Pyx_GIVEREF(r) __Pyx_Refnanny_GIVEREF(__pyx_refchk, r, __LINE__)
>  #define __Pyx_DECREF(r) __Pyx_Refnanny_DECREF(__pyx_refchk, r, __LINE__)
> -#define __Pyx_XDECREF(r) (r ? __Pyx_Refnanny_DECREF(__pyx_refchk, r,
> __LINE__) : 0)
> +#define __Pyx_XDECREF(r) if((r) == NULL) ; else __Pyx_DECREF(r)
...
> Can I push this?

Go ahead!

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Feb  2 18:34:16 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 18:34:16 +0100
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
In-Reply-To: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
References: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
Message-ID: <49872E98.4050800@student.matnat.uio.no>

Lisandro Dalcin wrote:
> I'm now getting many warnings like below:
> 
> src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
> src/petsc4py_PETSc.c:3793: warning: value computed is not used
> 

Does this mean that you compile with CYTHON_REFNANNY though?

-- 
Dag Sverre

From dalcinl at gmail.com  Mon Feb  2 19:07:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 15:07:21 -0300
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
In-Reply-To: <49872E98.4050800@student.matnat.uio.no>
References: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
	<49872E98.4050800@student.matnat.uio.no>
Message-ID: <e7ba66e40902021007g6b6cd6a6vc23ebdf9f484606c@mail.gmail.com>

On Mon, Feb 2, 2009 at 2:34 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> I'm now getting many warnings like below:
>>
>> src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
>> src/petsc4py_PETSc.c:3793: warning: value computed is not used
>>
>
> Does this mean that you compile with CYTHON_REFNANNY though?
>
> --

No yet. I'm on the process of trying it. But right now I'm having
trouble surely related to temp allocation, and in the process to send
a example for your consideration.

> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb  2 21:28:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 17:28:06 -0300
Subject: [Cython] possible problems with temp allocation?
In-Reply-To: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
References: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
Message-ID: <e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>

See the attached pyx file.

A valgrind run points me to the line where you have
"__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;" basically, DECREF is being
called with __pyx_t_1 being NULL...


  /* "/u/dalcinl/tmp/foo.pyx":10
 *         out = numpy.asarray(out)
 *         out.flat[:] = array
 *     return out             # <<<<<<<<<<<<<<
 *
 *
 */
  __Pyx_INCREF(__pyx_v_out);
  __pyx_r = __pyx_v_out;
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  goto __pyx_L0;


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.pyx
Type: application/octet-stream
Size: 319 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/71442e89/attachment.obj 

From dalcinl at gmail.com  Mon Feb  2 21:41:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 17:41:19 -0300
Subject: [Cython] comments about refnanny implementation...
Message-ID: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>

Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
was mine, but you know... I'm so idiot sometimes... This is going to
be a REAL pain to make it work, specially if you want to make this
easily portable to the many platforms out there.... For example, Do
the current implementation work on Windows?

Would you agree to completelly change the way the refnany helper
routines are imported?. I'm thinking on a ad-hoc way of doing that,
similar but simpler to the way Cython extension modules can export API
functions.

The only requisite would be that the 'refnanny.so' extension module
can be imported. Then, when you build Cython extensions with
CYTHON_REFNANNY defined, some special code in the module init function
imports the 'refnanny' module and get the pointer to a struct having
pointers to the various routines... No more need to dlopen() or even
import refnanny in order to enable it....

What do you think?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb  2 22:01:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 22:01:30 +0100
Subject: [Cython] possible problems with temp allocation?
In-Reply-To: <e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>
References: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
	<e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>
Message-ID: <49875F2A.8040704@student.matnat.uio.no>

Lisandro Dalcin wrote:
> See the attached pyx file.
> 
> A valgrind run points me to the line where you have
> "__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;" basically, DECREF is being
> called with __pyx_t_1 being NULL...

Does this fix it?

http://hg.cython.org/cython-devel/rev/b9fc866182dc


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Feb  2 22:11:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 22:11:26 +0100
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
Message-ID: <4987617E.6070707@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
> was mine, but you know... I'm so idiot sometimes... This is going to
> be a REAL pain to make it work, specially if you want to make this
> easily portable to the many platforms out there.... For example, Do
> the current implementation work on Windows?

I have no idea. I know next to nothing about Windows development (which 
goes beyond pre-2000-era Delphi GUI programming). And it would be a real 
pain for me to find or set up a Windows workstation with enough tools to 
find out.

> Would you agree to completelly change the way the refnany helper
> routines are imported?. I'm thinking on a ad-hoc way of doing that,
> similar but simpler to the way Cython extension modules can export API
> functions.
> 
> The only requisite would be that the 'refnanny.so' extension module
> can be imported. Then, when you build Cython extensions with
> CYTHON_REFNANNY defined, some special code in the module init function
> imports the 'refnanny' module and get the pointer to a struct having
> pointers to the various routines... No more need to dlopen() or even
> import refnanny in order to enable it....

This sounds like the way Cython implements "api", one could build on 
that. (But writing something dedicated in C might be just as well, 
because all the other things can potentially be subject to refnannying).

The reasons I've done what I've done is mostly to get something as quick 
as possible. If what I've done can't work in Windows, I'd be happy to 
accept a patch along the lines you suggest, but it is pretty far down on 
my own list. (It would mean an extra pointer lookup per refcount call I 
think, but speed is not an issue here...)

An alternative is to write runtests.py to disable the refnanny on 
Windows. As long as it is run on Unix prior to a Cython release it 
should work.


-- 
Dag Sverre

From dalcinl at gmail.com  Mon Feb  2 22:07:16 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 18:07:16 -0300
Subject: [Cython] possible problems with temp allocation?
In-Reply-To: <49875F2A.8040704@student.matnat.uio.no>
References: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
	<e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>
	<49875F2A.8040704@student.matnat.uio.no>
Message-ID: <e7ba66e40902021307h41703c72s1aca22323001c5ab@mail.gmail.com>

On Mon, Feb 2, 2009 at 6:01 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> See the attached pyx file.
>>
>> A valgrind run points me to the line where you have
>> "__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;" basically, DECREF is being
>> called with __pyx_t_1 being NULL...
>
> Does this fix it?
>
> http://hg.cython.org/cython-devel/rev/b9fc866182dc
>

Indeed. Many thanks!

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb  2 22:10:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 18:10:21 -0300
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <4987617E.6070707@student.matnat.uio.no>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
	<4987617E.6070707@student.matnat.uio.no>
Message-ID: <e7ba66e40902021310j56fd0797tc376a690971c9ba6@mail.gmail.com>

OK, let me write an test a patch. I'm starting right now...


On Mon, Feb 2, 2009 at 6:11 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
>> was mine, but you know... I'm so idiot sometimes... This is going to
>> be a REAL pain to make it work, specially if you want to make this
>> easily portable to the many platforms out there.... For example, Do
>> the current implementation work on Windows?
>
> I have no idea. I know next to nothing about Windows development (which
> goes beyond pre-2000-era Delphi GUI programming). And it would be a real
> pain for me to find or set up a Windows workstation with enough tools to
> find out.
>
>> Would you agree to completelly change the way the refnany helper
>> routines are imported?. I'm thinking on a ad-hoc way of doing that,
>> similar but simpler to the way Cython extension modules can export API
>> functions.
>>
>> The only requisite would be that the 'refnanny.so' extension module
>> can be imported. Then, when you build Cython extensions with
>> CYTHON_REFNANNY defined, some special code in the module init function
>> imports the 'refnanny' module and get the pointer to a struct having
>> pointers to the various routines... No more need to dlopen() or even
>> import refnanny in order to enable it....
>
> This sounds like the way Cython implements "api", one could build on
> that. (But writing something dedicated in C might be just as well,
> because all the other things can potentially be subject to refnannying).
>
> The reasons I've done what I've done is mostly to get something as quick
> as possible. If what I've done can't work in Windows, I'd be happy to
> accept a patch along the lines you suggest, but it is pretty far down on
> my own list. (It would mean an extra pointer lookup per refcount call I
> think, but speed is not an issue here...)
>
> An alternative is to write runtests.py to disable the refnanny on
> Windows. As long as it is run on Unix prior to a Cython release it
> should work.
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb  2 22:14:32 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 18:14:32 -0300
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <e7ba66e40902021310j56fd0797tc376a690971c9ba6@mail.gmail.com>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
	<4987617E.6070707@student.matnat.uio.no>
	<e7ba66e40902021310j56fd0797tc376a690971c9ba6@mail.gmail.com>
Message-ID: <e7ba66e40902021314x1396f55chbb4092987dc6e500@mail.gmail.com>

BTW, would it be hard for you to disable the generation of refnanny
stuff for functions that do not involve any Python object, just C
types (like those that can be called with the GIL released) ??

On Mon, Feb 2, 2009 at 6:10 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> OK, let me write an test a patch. I'm starting right now...
>
>
> On Mon, Feb 2, 2009 at 6:11 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Lisandro Dalcin wrote:
>>> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
>>> was mine, but you know... I'm so idiot sometimes... This is going to
>>> be a REAL pain to make it work, specially if you want to make this
>>> easily portable to the many platforms out there.... For example, Do
>>> the current implementation work on Windows?
>>
>> I have no idea. I know next to nothing about Windows development (which
>> goes beyond pre-2000-era Delphi GUI programming). And it would be a real
>> pain for me to find or set up a Windows workstation with enough tools to
>> find out.
>>
>>> Would you agree to completelly change the way the refnany helper
>>> routines are imported?. I'm thinking on a ad-hoc way of doing that,
>>> similar but simpler to the way Cython extension modules can export API
>>> functions.
>>>
>>> The only requisite would be that the 'refnanny.so' extension module
>>> can be imported. Then, when you build Cython extensions with
>>> CYTHON_REFNANNY defined, some special code in the module init function
>>> imports the 'refnanny' module and get the pointer to a struct having
>>> pointers to the various routines... No more need to dlopen() or even
>>> import refnanny in order to enable it....
>>
>> This sounds like the way Cython implements "api", one could build on
>> that. (But writing something dedicated in C might be just as well,
>> because all the other things can potentially be subject to refnannying).
>>
>> The reasons I've done what I've done is mostly to get something as quick
>> as possible. If what I've done can't work in Windows, I'd be happy to
>> accept a patch along the lines you suggest, but it is pretty far down on
>> my own list. (It would mean an extra pointer lookup per refcount call I
>> think, but speed is not an issue here...)
>>
>> An alternative is to write runtests.py to disable the refnanny on
>> Windows. As long as it is run on Unix prior to a Cython release it
>> should work.
>>
>>
>> --
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Mon Feb  2 22:25:41 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Mon, 02 Feb 2009 13:25:41 -0800
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <4987617E.6070707@student.matnat.uio.no>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
	<4987617E.6070707@student.matnat.uio.no>
Message-ID: <498764D5.2040105@gmail.com>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:

Hi,

>> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
>> was mine, but you know... I'm so idiot sometimes... This is going to
>> be a REAL pain to make it work, specially if you want to make this
>> easily portable to the many platforms out there.... For example, Do
>> the current implementation work on Windows?
> 
> I have no idea. I know next to nothing about Windows development (which 
> goes beyond pre-2000-era Delphi GUI programming). And it would be a real 
> pain for me to find or set up a Windows workstation with enough tools to 
> find out.

Hmm, judging from http://msdn.microsoft.com/en-us/library/ms684175.aspx 
and various manpages as well as my recollection of the way Python loads 
extension modules I believe that the problem does not exist on Windows, 
i.e. local vs. global symbols. But I obviously have not tested this.

>> Would you agree to completelly change the way the refnany helper
>> routines are imported?. I'm thinking on a ad-hoc way of doing that,
>> similar but simpler to the way Cython extension modules can export API
>> functions.
>>
>> The only requisite would be that the 'refnanny.so' extension module
>> can be imported. Then, when you build Cython extensions with
>> CYTHON_REFNANNY defined, some special code in the module init function
>> imports the 'refnanny' module and get the pointer to a struct having
>> pointers to the various routines... No more need to dlopen() or even
>> import refnanny in order to enable it....
> 
> This sounds like the way Cython implements "api", one could build on 
> that. (But writing something dedicated in C might be just as well, 
> because all the other things can potentially be subject to refnannying).
> 
> The reasons I've done what I've done is mostly to get something as quick 
> as possible. If what I've done can't work in Windows, I'd be happy to 
> accept a patch along the lines you suggest, but it is pretty far down on 
> my own list. (It would mean an extra pointer lookup per refcount call I 
> think, but speed is not an issue here...)
> 
> An alternative is to write runtests.py to disable the refnanny on 
> Windows. As long as it is run on Unix prior to a Cython release it 
> should work.


Well, it doesn't take much imagination to see someone coding Windows 
specific code that cannot be tested via Cygwin or Linux, so making this 
work on any supported platform would be a good thing. I have to port 
some code to MSVC that currently uses dlopen with RTLD_GLOBAL, but if 
Lisandro fixes the problem with Cython I would certainly be interested 
to see the result.

Cheers,

Michael

From dalcinl at gmail.com  Tue Feb  3 01:36:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 21:36:09 -0300
Subject: [Cython] PATCH: new refnanny API
Message-ID: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>

Here you have a preliminary patch for loading the refnanny API
skipping dlopen() hackery.

For this to work, just make Cython/Runtime/refnanny.pyx a extension
module, and put it somewhere in your PYTHONPATH. That's all!! Now
build other extensions with -DCYTHON_REFNANNY and enjoy...

Take into account that I'm not completelly happy with the current
patch, more work is required.

1) As namespaces are a honking great idea, I do not see the point of
using "Refnanny" prefixes on class and function names at refnanny.py

2) I'm a bit uncomfortable with the way I had to initialize stuff at
module initialization function.

3) Many unittests in mpi4py pass, but I'm also gettting some
segfaults. In particular, the refnanny stuff should no be emmited for
"pure" cdef functions (I mean, those with no Python objects at all in
args, retvals and local vars). For example, in mpi4py I have a
finalizer routine that is registered with Py_AtExit(), then it is
called after Py_Finalize(), thus no Python code should run. I'm also
getting some bad interaction between refnanny and exception handling,
but I need to dive deeper to catch the problem, no yet sure what is
going on.


We can continue discussion on cython-dev, but if any of you want to
make this a ticket, go ahead!



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: REFNANNY.diff
Type: application/octet-stream
Size: 6206 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/3d066110/attachment.obj 

From greg.ewing at canterbury.ac.nz  Tue Feb  3 04:44:32 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 03 Feb 2009 16:44:32 +1300
Subject: [Cython] Equivalent of __new__() in python
In-Reply-To: <f70460160808011248p1f5305c6kd57373315cc86f2f@mail.gmail.com>
References: <f70460160808011248p1f5305c6kd57373315cc86f2f@mail.gmail.com>
Message-ID: <4987BDA0.3010504@canterbury.ac.nz>

Arnaud Bergeron wrote:
> I want to have only one instance at a time with a single value of val.
> 
> I already know how to do this in python using the __new__() method and
> a WeakValueDictionary.  I am looking for a way to do the equivalent in
> cython.

This can't be done yet, because you don't get a chance
to directly override the tp_new slot. I intend to fix
this in some future version of Pyrex.

-- 
Greg

From dagss at student.matnat.uio.no  Tue Feb  3 10:16:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 10:16:24 +0100
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
Message-ID: <49880B68.2000102@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Here you have a preliminary patch for loading the refnanny API
> skipping dlopen() hackery.
>
> For this to work, just make Cython/Runtime/refnanny.pyx a extension
> module, and put it somewhere in your PYTHONPATH. That's all!! Now
> build other extensions with -DCYTHON_REFNANNY and enjoy...
>
> Take into account that I'm not completelly happy with the current
> patch, more work is required.
>
> 1) As namespaces are a honking great idea, I do not see the point of
> using "Refnanny" prefixes on class and function names at refnanny.py
>   
By all means.

> 2) I'm a bit uncomfortable with the way I had to initialize stuff at
> module initialization function.
>   
Looks OK with me.
> 3) Many unittests in mpi4py pass, but I'm also gettting some
> segfaults. In particular, the refnanny stuff should no be emmited for
> "pure" cdef functions (I mean, those with no Python objects at all in
> args, retvals and local vars). For example, in mpi4py I have a
> finalizer routine that is registered with Py_AtExit(), then it is
> called after Py_Finalize(), thus no Python code should run. I'm also
> getting some bad interaction between refnanny and exception handling,
> but I need to dive deeper to catch the problem, no yet sure what is
> going on.
>   
Can't you declare the function as "nogil"? I.e.

cdef nogil int myfunc(): ...

I already disable the refnanny if "nogil" is declared, and then Python 
operations are disallowed. Doing this automatically though is a bit 
hard, I'd have to make sure that no Python temporaries are ever 
constructed in the function, which is more than a five-minute job. If 
you give a good reason the nogil modifier doesn't do the trick I can 
look into it.

I've applied your patch, thanks! RTLD_GLOBAL-issues aside, it is a lot 
easier to set up the refnanny on other code than the test suite now.

Please vote: Should we install the refnanny into site-packages when 
doing a Cython install? Then we could just tell users "try compiling 
with the -DCYTHON_REFNANNY flag" to get diagnostics.

Dag Sverre

From muogoro at gmail.com  Tue Feb  3 11:30:26 2009
From: muogoro at gmail.com (Daniele Pianu)
Date: Tue, 3 Feb 2009 11:30:26 +0100
Subject: [Cython] Propagating exceptions with functions which returns void*
Message-ID: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>

Probably I've not completely understood the C-function
exception-propagating mechanism. I've a function with a prototype like
this

void* getItem( void* self, int i )

The function returns the i-th item in self. Some index and bounds
checking are implemented, so if, for example, I specify a negative
index or an index out of self bounds I raise and exception. In cython
the function is declared in this way:

cdef void* getItem( void* self, int i ) except NULL

But when an error occurs and the exception is raised I always receive
a message which says that the exception is ignored. Obviously, the
function continues its execution and terminates whit a segmentation
fault.

The question is: how can I propagate to caller functions the raised exception?
Thanks for your help (and for the great work that you've done with cython :D ),
Daniele

From dagss at student.matnat.uio.no  Tue Feb  3 11:35:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 11:35:06 +0100
Subject: [Cython] Propagating exceptions with functions which returns
 void*
In-Reply-To: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
Message-ID: <49881DDA.5040207@student.matnat.uio.no>

Daniele Pianu wrote:
> Probably I've not completely understood the C-function
> exception-propagating mechanism. I've a function with a prototype like
> this
>
> void* getItem( void* self, int i )
>   
Where is this function declared? What kind of exception is raised? C has 
no exceptions, and C++ exception needs a different form of declaration: 
*http://docs.cython.org/docs/pyrex_differences.html#c-exception-handling*

The "except NULL" is for propagating within-Cython exceptions, see 
http://docs.cython.org/docs/language_basics.html#error-return-values

Is this the reason for your problem? I might be missing the mark here.

Dag Sverre

From muogoro at gmail.com  Tue Feb  3 11:46:52 2009
From: muogoro at gmail.com (Daniele Pianu)
Date: Tue, 3 Feb 2009 11:46:52 +0100
Subject: [Cython] Propagating exceptions with functions which returns
	void*
In-Reply-To: <49881DDA.5040207@student.matnat.uio.no>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
	<49881DDA.5040207@student.matnat.uio.no>
Message-ID: <66a8c45d0902030246q7471da71u25f2f2abc8d909ac@mail.gmail.com>

The function is a cdef extension type's method. Here the part of the
implementation which raises the exceptions:

  cdef void *getItem( self, int i ) except NULL:
    cdef void* tmpPtr

    # Check NULL pointer
    if self.rawPtr == NULL:
      raise ValueError( "Null pointer" )
    # Check specified index
    if i < 0:
      raise IndexError( "Unvalid index" )
    # Bound checking
    if self._layout[0] > 0 and i >= self._layout[0]:
      raise IndexError( "Index out of bounds" )

    # Code used for get the i-th value
    .......

self.rawPtr is the void pointer stored by the extension type.
self._layout is a simple tuple which defines the "shape" of
self.rawPtr, used for bounds checking. The function is called by other
extension types which extend the extension type which defines the
getItem method. What do you main with "The "except NULL" is for
propagating within-Cython exceptions"? Do I need to put the getItem
call in a try-except block in the cython caller?

Daniele

2009/2/3 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
> Daniele Pianu wrote:
>> Probably I've not completely understood the C-function
>> exception-propagating mechanism. I've a function with a prototype like
>> this
>>
>> void* getItem( void* self, int i )
>>
> Where is this function declared? What kind of exception is raised? C has
> no exceptions, and C++ exception needs a different form of declaration:
> *http://docs.cython.org/docs/pyrex_differences.html#c-exception-handling*
>
> The "except NULL" is for propagating within-Cython exceptions, see
> http://docs.cython.org/docs/language_basics.html#error-return-values
>
> Is this the reason for your problem? I might be missing the mark here.
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Tue Feb  3 12:33:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 12:33:29 +0100
Subject: [Cython] Propagating exceptions with functions which returns
 void*
In-Reply-To: <66a8c45d0902030246q7471da71u25f2f2abc8d909ac@mail.gmail.com>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>	<49881DDA.5040207@student.matnat.uio.no>
	<66a8c45d0902030246q7471da71u25f2f2abc8d909ac@mail.gmail.com>
Message-ID: <49882B89.6020209@student.matnat.uio.no>

Daniele Pianu wrote:
> The function is a cdef extension type's method. Here the part of the
> implementation which raises the exceptions:
>
>   cdef void *getItem( self, int i ) except NULL:
>     cdef void* tmpPtr
>
>     # Check NULL pointer
>     if self.rawPtr == NULL:
>       raise ValueError( "Null pointer" )
>     # Check specified index
>     if i < 0:
>       raise IndexError( "Unvalid index" )
>     # Bound checking
>     if self._layout[0] > 0 and i >= self._layout[0]:
>       raise IndexError( "Index out of bounds" )
>
>     # Code used for get the i-th value
>     .......
>
> self.rawPtr is the void pointer stored by the extension type.
> self._layout is a simple tuple which defines the "shape" of
> self.rawPtr, used for bounds checking. The function is called by other
> extension types which extend the extension type which defines the
> getItem method. What do you main with "The "except NULL" is for
> propagating within-Cython exceptions"? Do I need to put the getItem
> call in a try-except block in the cython caller?
>   
I don't think what I said applies in your case. I'm still a bit 
confused, as "self" was typed as "void*" in your original example, while 
here getitem seems to be part of a class.

Here's a minimal testcase, which seems to work OK for me. Can you please 
modify it until it explains your situation?

Dag Sverre

"""
 >>> f()
Traceback (most recent call last):
...
ValueError: foo
"""
cdef class A:
    cdef void* ptr
    def __init__(self):
        self.ptr = NULL
    cdef void* getitem(self) except NULL:
        if self.ptr is NULL:
            raise ValueError("foo")

def f():
    cdef A a = A()
    print a.getitem() == NULL

From dalcinl at gmail.com  Tue Feb  3 14:03:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 11:03:39 -0200
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <49880B68.2000102@student.matnat.uio.no>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
	<49880B68.2000102@student.matnat.uio.no>
Message-ID: <e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>

On Tue, Feb 3, 2009 at 7:16 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>>
> Can't you declare the function as "nogil"? I.e.
>
> cdef nogil int myfunc(): ...
>

I'll give a try. That would be fine with me...

>
> I've applied your patch, thanks! RTLD_GLOBAL-issues aside, it is a lot
> easier to set up the refnanny on other code than the test suite now.
>

Indeed.

> Please vote: Should we install the refnanny into site-packages when
> doing a Cython install? Then we could just tell users "try compiling
> with the -DCYTHON_REFNANNY flag" to get diagnostics.
>

Perhaps betther would be to put iside Cython.Runtime, in such a way
that the full dotted name would be 'Cython.Runtime.refnanny' ?? I'm
not a big fan of populating site-packages directory. Perhaps if the
module is renamed to 'pyxrefnanny'?




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb  3 14:33:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 14:33:44 +0100
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>	<49880B68.2000102@student.matnat.uio.no>
	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
Message-ID: <498847B8.2030600@student.matnat.uio.no>

Lisandro Dalcin wrote:
>
> Perhaps betther would be to put iside Cython.Runtime, in such a way
> that the full dotted name would be 'Cython.Runtime.refnanny' ?? I'm
> not a big fan of populating site-packages directory. Perhaps if the
> module is renamed to 'pyxrefnanny'?
>   
If your code could be modified to first check for just "refnanny", and 
if that is not found, then attempt Cython.Runtime.refnanny?

This would allow runtests.py to compile refnanny outside of the package 
(which may be changed relative to an installed version), while falling 
back to the one in the package otherwise.

An alternative is to have a new toplevel package cythonruntime or 
cython_runtime. The point is that runtests.py must be able to load it 
into sys.path together with the code in the current directory 
(preferably without having to copy the whole Cython tree).

I tried doing this but it didn't work immediately so I'm giving up. Feel 
free :-)

Dag Sverre

From dalcinl at gmail.com  Tue Feb  3 14:45:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 11:45:02 -0200
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <498847B8.2030600@student.matnat.uio.no>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
	<49880B68.2000102@student.matnat.uio.no>
	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
	<498847B8.2030600@student.matnat.uio.no>
Message-ID: <e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>

On Tue, Feb 3, 2009 at 11:33 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>>
> If your code could be modified to first check for just "refnanny", and
> if that is not found, then attempt Cython.Runtime.refnanny?

OK, I'll implement this approach, I like it...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb  3 14:50:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 11:50:22 -0200
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
	<49880B68.2000102@student.matnat.uio.no>
	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
	<498847B8.2030600@student.matnat.uio.no>
	<e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>
Message-ID: <e7ba66e40902030550o5afb7d0dn4c94532192819571@mail.gmail.com>

Dag, I see you pushed my patch, but the refnanny functions are still
'public'... Any good reason for this?

2009/2/3 Lisandro Dalcin <dalcinl at gmail.com>:
> On Tue, Feb 3, 2009 at 11:33 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Lisandro Dalcin wrote:
>>>
>> If your code could be modified to first check for just "refnanny", and
>> if that is not found, then attempt Cython.Runtime.refnanny?
>
> OK, I'll implement this approach, I like it...
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb  3 14:55:20 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 14:55:20 +0100
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902030550o5afb7d0dn4c94532192819571@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>	<49880B68.2000102@student.matnat.uio.no>	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>	<498847B8.2030600@student.matnat.uio.no>	<e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>
	<e7ba66e40902030550o5afb7d0dn4c94532192819571@mail.gmail.com>
Message-ID: <49884CC8.2090607@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Dag, I see you pushed my patch, but the refnanny functions are still
> 'public'... Any good reason for this?
>   
No, that was a slip (I had to merge and didn't notice).

You've obviously got more attention to detail here than me, so feel free 
to push this and other changes on the refnanny yourself if you want to.

Dag Sverre

From dalcinl at gmail.com  Tue Feb  3 18:06:10 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 15:06:10 -0200
Subject: [Cython] review this fix for bad interaction of refnanny + 'with
	gil'
Message-ID: <e7ba66e40902030906l70a55fd4j8bfc6c875098428@mail.gmail.com>

After the patch below (plus some changes in refnanny.pyx, that's for
other mail), I get the full mpi4py testsuite pass....

Basically, the refnanny context needs to be finished BEFORE the GIL
state is released...

Could any of you tell me if this fix would be required at some other
place? I got a bit lost. ;-) ...

diff -r e35f20896675 Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py	Tue Feb 03 14:00:31 2009 -0200
+++ b/Cython/Compiler/Nodes.py	Tue Feb 03 14:49:10 2009 -0200
@@ -1181,8 +1181,7 @@
         for entry in lenv.arg_entries:
             if entry.type.is_pyobject and
lenv.control_flow.get_state((entry.name, 'source')) != 'arg':
                 code.put_var_decref(entry)
-        if acquire_gil:
-            code.putln("PyGILState_Release(_save);")
+
         # code.putln("/* TODO: decref scope object */")
         # ----- Return
         # This code is duplicated in ModuleNode.generate_module_init_func
@@ -1199,6 +1198,9 @@
                                              Naming.retval_cname,
                                              err_val)

+        if acquire_gil:
+            code.putln("PyGILState_Release(_save);")
+
         if not self.return_type.is_void:
             code.putln("return %s;" % Naming.retval_cname)




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb  3 18:31:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 15:31:46 -0200
Subject: [Cython] more enhancement to refnanny API
Message-ID: <e7ba66e40902030931w65722a00n8d299a95c02219df@mail.gmail.com>

Dag, some ideas for your consideration ...

1) What about changing those bare void* context pointer to PyObject*

2) What about making FinishContext(PyObject**), and make it set the
arg to NULL on return?.

3) refnanny.Error (formely refnanny.RefnannyExeception) is raised in
Context.end() in case of errors. Is this fine? What about raising at
the end of FinishContext()?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb  3 20:40:33 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 17:40:33 -0200
Subject: [Cython] a problem with sizeof() return type
Message-ID: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>

Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.

Implementing this the right way would require to make Cython recognize
'size_t' as a native type, more or less in the way it already
recognizes 'Py_ssize_t'. What do you think about this?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Tue Feb  3 21:01:43 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 03 Feb 2009 12:01:43 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
Message-ID: <4988A2A7.8060204@gmail.com>

Lisandro Dalcin wrote:
> Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.

Well, you might be technically correct since that is what the C standard 
asks for, but according to various literature, i.e. for example

http://publications.gbdirect.co.uk/c_book/chapter5/sizeof_and_malloc.html

"The sizeof operator returns the size in bytes of its operand. Whether 
the result of sizeof is unsigned int or unsigned long is implementation 
defined?which is why the declaration of malloc above ducked the issue by 
omitting any parameter information; "


> Implementing this the right way would require to make Cython recognize
> 'size_t' as a native type, more or less in the way it already
> recognizes 'Py_ssize_t'. What do you think about this?
> 

Mmmh, do you want to define size_t in Cython or pull it in from 
somewhere? At least for C code that ought to be rather tricky for VS6 
for example which needs to work since we are still supporting Python 2.3 
on Windows. But you could work around the issue by defining size_t in 
Cython as either unsigned int, unsigned long or unsigned long long 
depending on the platform. But that could potentially clash with various 
C++ runtime libs.

Do you think that people will see overflow issues? I think the tradeoff 
of following the C standard here is rather high since in reality size_t 
in C is rather badly handled. And if you guys decide to follow the C 
standard here someone ought to break out Python 2.6 and VS 6 and verify 
that things are still working :)

Cheers,

Michael

From jasone at canonware.com  Tue Feb  3 21:29:01 2009
From: jasone at canonware.com (Jason Evans)
Date: Tue, 03 Feb 2009 12:29:01 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988A2A7.8060204@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
Message-ID: <4988A90D.30107@canonware.com>

Michael Abshoff wrote:
> Lisandro Dalcin wrote:
>> Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.
> 
> Well, you might be technically correct since that is what the C standard 
> asks for, but according to various literature, i.e. for example
> 
> http://publications.gbdirect.co.uk/c_book/chapter5/sizeof_and_malloc.html
> 
> "The sizeof operator returns the size in bytes of its operand. Whether 
> the result of sizeof is unsigned int or unsigned long is implementation 
> defined?which is why the declaration of malloc above ducked the issue by 
> omitting any parameter information; "

That book appears to have been published in 1991...

Personally, I've found the lack of native size_t knowledge in Cython to 
be a major pain.  In practice though, the following is sufficient for 
every modern system I know of, with the likely exception of Windows:

     cdef extern from "sys/types.h":
         ctypedef unsigned long size_t

Jason

From michael.abshoff at googlemail.com  Tue Feb  3 21:43:50 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 03 Feb 2009 12:43:50 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988A90D.30107@canonware.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>
Message-ID: <4988AC86.5070306@gmail.com>

Jason Evans wrote:
> Michael Abshoff wrote:
>> Lisandro Dalcin wrote:
>>> Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.
>> Well, you might be technically correct since that is what the C standard 
>> asks for, but according to various literature, i.e. for example
>>
>> http://publications.gbdirect.co.uk/c_book/chapter5/sizeof_and_malloc.html
>>
>> "The sizeof operator returns the size in bytes of its operand. Whether 
>> the result of sizeof is unsigned int or unsigned long is implementation 
>> defined?which is why the declaration of malloc above ducked the issue by 
>> omitting any parameter information; "
> 
> That book appears to have been published in 1991...

Yeah, it has been a while, but then Python requires a C89 compiler :)

> Personally, I've found the lack of native size_t knowledge in Cython to 
> be a major pain.  In practice though, the following is sufficient for 
> every modern system I know of, with the likely exception of Windows:
> 
>      cdef extern from "sys/types.h":
>          ctypedef unsigned long size_t


Well, make it unsigned long long on 64 bit Windows and it ought to work 
nearly everywhere. I am not aware of any LLP64 bit OS that Cython 
supports besides 64 bit Windows. Obviously there is no sys/types.h on 
Windows, but that is a different story :)

> Jason

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dalcinl at gmail.com  Tue Feb  3 21:57:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 18:57:09 -0200
Subject: [Cython] range optimization and noqil
Message-ID: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>

Trying to cythonizing this:

cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
        j += 1
    return j


I get this failure... Is this possible and easy to fix ?

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                 ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:18: Accessing Python global or builtin not
allowed without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                   ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:20: Converting to Python object not allowed without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                 ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:18: Constructing Python tuple not allowed without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                 ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:18: Calling gil-requiring function without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
            ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:13: Iterating over Python object not allowed
without gil

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Feb  4 00:30:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Feb 2009 00:30:15 +0100
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988AC86.5070306@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com>
Message-ID: <4988D387.7070304@student.matnat.uio.no>

Michael Abshoff wrote:
> Jason Evans wrote:
>>      cdef extern from "sys/types.h":
>>          ctypedef unsigned long size_t
> 
> 
> Well, make it unsigned long long on 64 bit Windows and it ought to work 
> nearly everywhere. I am not aware of any LLP64 bit OS that Cython 
> supports besides 64 bit Windows. Obviously there is no sys/types.h on 
> Windows, but that is a different story :)

Just a slightly related note:

Note that with this kind of declaration, all Cython uses in 90% of the 
situations is the fact that it is an unsigned integral type. "unsigned 
char size_t" it will do the same in almost all situations -- in the end, 
"size_t" is simply emitted to the .c file.

The 10% is overflow exceptions from coercion from Python integer -- and 
pushing that decision too to the C compiler is on my personal todo-list 
(though far down). In general Cython prefers to leave knowledge of the 
size/capacity of types to the C compiler.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Feb  4 10:49:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Feb 2009 10:49:48 +0100
Subject: [Cython] range optimization and noqil
In-Reply-To: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>
References: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>
Message-ID: <498964BC.8000706@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Trying to cythonizing this:
>
> cdef int f() nogil:
>     cdef int i,j=0, n=3
>     for i in range(n):
>         j += 1
>     return j
>
>
> I get this failure... Is this possible and easy to fix ?
>   

Thanks, good point. This is now

http://trac.cython.org/cython_trac/ticket/205

I think it is definitely 0.12 stuff, so use a for-from construct if you 
need this for 0.11.

If you just want to run the refnanny, perhaps consider just disabling 
the line in the source raising the error on your local copy?

Dag Sverre

From dalcinl at gmail.com  Wed Feb  4 15:06:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 12:06:28 -0200
Subject: [Cython] range optimization and noqil
In-Reply-To: <498964BC.8000706@student.matnat.uio.no>
References: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>
	<498964BC.8000706@student.matnat.uio.no>
Message-ID: <e7ba66e40902040606o27ebc504tbb44ad6bb9106434@mail.gmail.com>

Yes, I changed my code to use for-loops.

But I still think that the refnanny should not be emmited for 'pure'
functions. Declaring them as nogil does the trick, but this can cause
other sorce code changed. For example, in Cython/Include, you have a
pxd with C stdlib stuff. These declarations do not have the 'nogil',
then you cannot use them inside other 'nogil' functions.

I'll try to work on this in the near future...


On Wed, Feb 4, 2009 at 7:49 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> Trying to cythonizing this:
>>
>> cdef int f() nogil:
>>     cdef int i,j=0, n=3
>>     for i in range(n):
>>         j += 1
>>     return j
>>
>>
>> I get this failure... Is this possible and easy to fix ?
>>
>
> Thanks, good point. This is now
>
> http://trac.cython.org/cython_trac/ticket/205
>
> I think it is definitely 0.12 stuff, so use a for-from construct if you
> need this for 0.11.
>
> If you just want to run the refnanny, perhaps consider just disabling
> the line in the source raising the error on your local copy?
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb  4 15:17:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 12:17:11 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988D387.7070304@student.matnat.uio.no>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com> <4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com> <4988D387.7070304@student.matnat.uio.no>
Message-ID: <e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>

Well, then how should we fix the return type in the code below ? (at
Cython/Compiler/ExprNodes.py)

class SizeofNode(ExprNode):
    #  Abstract base class for sizeof(x) expression nodes.

    type = PyrexTypes.c_int_type


I still think that the only sane way to fix that is to make Cython
understand 'size_t' as a already defined C type. That would let us to
properly define the return type of sizeof. Furthermore, that would let
users to forget about having to typedef ssize_t to something.

There are strong objections for my approach? What could be the
potential problems?


On Tue, Feb 3, 2009 at 9:30 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Michael Abshoff wrote:
>> Jason Evans wrote:
>>>      cdef extern from "sys/types.h":
>>>          ctypedef unsigned long size_t
>>
>>
>> Well, make it unsigned long long on 64 bit Windows and it ought to work
>> nearly everywhere. I am not aware of any LLP64 bit OS that Cython
>> supports besides 64 bit Windows. Obviously there is no sys/types.h on
>> Windows, but that is a different story :)
>
> Just a slightly related note:
>
> Note that with this kind of declaration, all Cython uses in 90% of the
> situations is the fact that it is an unsigned integral type. "unsigned
> char size_t" it will do the same in almost all situations -- in the end,
> "size_t" is simply emitted to the .c file.
>
> The 10% is overflow exceptions from coercion from Python integer -- and
> pushing that decision too to the C compiler is on my personal todo-list
> (though far down). In general Cython prefers to leave knowledge of the
> size/capacity of types to the C compiler.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Feb  4 15:37:13 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Feb 2009 15:37:13 +0100
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>
	<4988D387.7070304@student.matnat.uio.no>
	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
Message-ID: <4989A819.9040202@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Well, then how should we fix the return type in the code below ? (at
> Cython/Compiler/ExprNodes.py)
> 
> class SizeofNode(ExprNode):
>     #  Abstract base class for sizeof(x) expression nodes.
> 
>     type = PyrexTypes.c_int_type
> 
> 
> I still think that the only sane way to fix that is to make Cython
> understand 'size_t' as a already defined C type. That would let us to
> properly define the return type of sizeof. Furthermore, that would let
> users to forget about having to typedef ssize_t to something.
> 
> There are strong objections for my approach? What could be the
> potential problems?

The objections as I understand them is that not all C compilers 
understand the "size_t" type. Not all projects (amongst them CPython) 
adhere to the latest C standards.

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Wed Feb  4 15:40:10 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 04 Feb 2009 06:40:10 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4989A819.9040202@student.matnat.uio.no>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>	<4988D387.7070304@student.matnat.uio.no>	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
	<4989A819.9040202@student.matnat.uio.no>
Message-ID: <4989A8CA.1020809@gmail.com>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:
>> Well, then how should we fix the return type in the code below ? (at
>> Cython/Compiler/ExprNodes.py)
>>
>> class SizeofNode(ExprNode):
>>     #  Abstract base class for sizeof(x) expression nodes.
>>
>>     type = PyrexTypes.c_int_type
>>
>>
>> I still think that the only sane way to fix that is to make Cython
>> understand 'size_t' as a already defined C type. That would let us to
>> properly define the return type of sizeof. Furthermore, that would let
>> users to forget about having to typedef ssize_t to something.
>>
>> There are strong objections for my approach? What could be the
>> potential problems?
> 
> The objections as I understand them is that not all C compilers 
> understand the "size_t" type. Not all projects (amongst them CPython) 
> adhere to the latest C standards.
> 

Well, an arguing that those damn C compilers ought to be fixed to follow 
the C standard also is unrealistic. Note that one such broken compiler 
which needs help to understand what a size_t is even in C++ mode is VC6 
which is Python 2.3's official compiler on Windows, so not supporting it 
would be a bad move as long as Cython claims to support Python 2.3. And 
some people have reported bugs in Cython with Python 2.3 and VC6 in the 
last couple months, so it is still being used. That is sad since VC6 
leaves much to be desired, but we live in a world of legacy code :)

Cheers,

Michael

From dalcinl at gmail.com  Wed Feb  4 15:40:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 12:40:25 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4989A819.9040202@student.matnat.uio.no>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com> <4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com> <4988D387.7070304@student.matnat.uio.no>
	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
	<4989A819.9040202@student.matnat.uio.no>
Message-ID: <e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>

On Wed, Feb 4, 2009 at 12:37 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>>
>> There are strong objections for my approach? What could be the
>> potential problems?
>
> The objections as I understand them is that not all C compilers
> understand the "size_t" type. Not all projects (amongst them CPython)
> adhere to the latest C standards.
>

And then could you tell me how core CPython could be build with such a
C compiler...

Please look, at core CPython sources. To be sure, inspect Python-2.3
sources. They use 'size_t' freely after including "Python.h"


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Wed Feb  4 15:58:54 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 04 Feb 2009 06:58:54 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>
	<4988D387.7070304@student.matnat.uio.no>	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>	<4989A819.9040202@student.matnat.uio.no>
	<e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>
Message-ID: <4989AD2E.3080900@gmail.com>

Lisandro Dalcin wrote:
> On Wed, Feb 4, 2009 at 12:37 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>>> There are strong objections for my approach? What could be the
>>> potential problems?
>> The objections as I understand them is that not all C compilers
>> understand the "size_t" type. Not all projects (amongst them CPython)
>> adhere to the latest C standards.
>>
> 
> And then could you tell me how core CPython could be build with such a
> C compiler...
> 
> Please look, at core CPython sources. To be sure, inspect Python-2.3
> sources. They use 'size_t' freely after including "Python.h"
> 
> 

Well, is Python.h guaranteed to be included in every Cython extension? I 
guess it would be. But what about pure C files linked against an 
extension for example? People would be required to either include 
Python.h for size_t or find it somewhere else in the system. Not that 
this is hard, but it will confuse some people :)

I am sure Python 2.3 (and also later releases obviously) defines it 
appropriately if it isn't provided by the system. Overall I think this 
is a worthwhile change, the new code just ought to be verified on some 
of the older known bad compilers.

While poking around I found the following nugget in Sage:

# Memory management
cdef extern from "stdlib.h":
     ctypedef unsigned long size_t


So I would expect some buggy code to be exposed by the suggested change 
above.

Cheers,

Michael

From dalcinl at gmail.com  Wed Feb  4 16:31:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 13:31:01 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4989AD2E.3080900@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com> <4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com> <4988D387.7070304@student.matnat.uio.no>
	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
	<4989A819.9040202@student.matnat.uio.no>
	<e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>
	<4989AD2E.3080900@gmail.com>
Message-ID: <e7ba66e40902040731x17bf511cj6273c38a292d43e4@mail.gmail.com>

On Wed, Feb 4, 2009 at 12:58 PM, Michael Abshoff
<michael.abshoff at googlemail.com> wrote:
>
> Well, is Python.h guaranteed to be included in every Cython extension?
> I guess it would be.
>

AFAIK, yes. See yourself:

class ModuleScope(Scope):
   ....
    def __init__(self, name, parent_module, context):
        ....
        self.python_include_files = ["Python.h", "structmember.h"]

> But what about pure C files linked against an
> extension for example? People would be required to either include
> Python.h for size_t or find it somewhere else in the system. Not that
> this is hard, but it will confuse some people :)
>

Not sure if I understand you. If pure C file want to use 'size_t',
they are in charge of it.

> I am sure Python 2.3 (and also later releases obviously) defines it
> appropriately if it isn't provided by the system. Overall I think this
> is a worthwhile change, the new code just ought to be verified on some
> of the older known bad compilers.
>

Well, then I'll try to write a patch for all you to review and test.

> While poking around I found the following nugget in Sage:
>
> # Memory management
> cdef extern from "stdlib.h":
>     ctypedef unsigned long size_t
>
>
> So I would expect some buggy code to be exposed by the suggested change
> above.
>

Well, such a change would expose buggy code and perhaps be backward
incompatible, right?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Feb  4 20:31:54 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Feb 2009 20:31:54 +0100
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
In-Reply-To: <4986BADB.50202@student.matnat.uio.no>
References: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
	<4986BADB.50202@student.matnat.uio.no>
Message-ID: <4989ED2A.2080303@behnel.de>



Dag Sverre Seljebotn wrote:
> Case Vanhorsen wrote:
>> I just tried to install Cython 0.11 beta with Python 3.0. I get the
>> following error:
>>
>> case:~/src/Cython-0.11.beta$ py30 setup.py install
>> Traceback (most recent call last):
>>   File "setup.py", line 5, in <module>
>>     from Cython.Compiler.Version import version
>>   File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
>>     from Shadow import *
>> ImportError: No module named Shadow
>>
>> It does install successfully using Python 2.5 and 2.6. Please let me
>> know if there is any other information you need.
>>   
> I had the impression that Cython doesn't support being run under Python 
> 3 (though Stefan would have the definitive answer on that). I think you 
> must install on Python 2.6, and then compile your .pyx to .c under 2.6, 
> and THEN you can compile the c file with the Python 3 headers to get a 
> Python 3 module.

That is true. While the generated C source code supports Python 2.3 through
3.0 without changes, the Cython compiler itself requires Python 2.3 through
2.6 to run.

As this does not limit Cython's functionality or applicability in any way,
it's not a major priority for us to migrate Cython's own source code to Py3.

Stefan

From michael.abshoff at googlemail.com  Wed Feb  4 21:15:57 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 04 Feb 2009 12:15:57 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902040731x17bf511cj6273c38a292d43e4@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>
	<4988D387.7070304@student.matnat.uio.no>	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>	<4989A819.9040202@student.matnat.uio.no>	<e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>	<4989AD2E.3080900@gmail.com>
	<e7ba66e40902040731x17bf511cj6273c38a292d43e4@mail.gmail.com>
Message-ID: <4989F77D.2020903@gmail.com>

Lisandro Dalcin wrote:
> On Wed, Feb 4, 2009 at 12:58 PM, Michael Abshoff
> <michael.abshoff at googlemail.com> wrote:
>> Well, is Python.h guaranteed to be included in every Cython extension?
>> I guess it would be.
>>
> 
> AFAIK, yes. See yourself:
> 
> class ModuleScope(Scope):
>    ....
>     def __init__(self, name, parent_module, context):
>         ....
>         self.python_include_files = ["Python.h", "structmember.h"]

Good, I don't venture too deeply into the Cython compiler.

>> But what about pure C files linked against an
>> extension for example? People would be required to either include
>> Python.h for size_t or find it somewhere else in the system. Not that
>> this is hard, but it will confuse some people :)
>>
> 
> Not sure if I understand you. If pure C file want to use 'size_t',
> they are in charge of it.

Yes, it was completely dumb to raise the issue here. I needed some sleep 
and I hope I got caught up on sleep in the last couple hours :)

>> I am sure Python 2.3 (and also later releases obviously) defines it
>> appropriately if it isn't provided by the system. Overall I think this
>> is a worthwhile change, the new code just ought to be verified on some
>> of the older known bad compilers.
>>
> 
> Well, then I'll try to write a patch for all you to review and test.
> 
>> While poking around I found the following nugget in Sage:
>>
>> # Memory management
>> cdef extern from "stdlib.h":
>>     ctypedef unsigned long size_t
>>
>>
>> So I would expect some buggy code to be exposed by the suggested change
>> above.
>>
> 
> Well, such a change would expose buggy code and perhaps be backward
> incompatible, right?

Yes, but code that was buggy and is being exposed by fixing a bug in an 
upstream tool is acceptable since the developer needs to fix that bug 
anyway. It might be painful, but the end result is better code. Hadn't I 
looked around for size_t definitions in Sage we would have hit that bug 
the first time someone does a 4 GB+ allocation on Windows 64 bit mode 
once Sage has been ported to MSVC. That is not in the near future, but 
since I do porting of code all day I can tell you that it is one of the 
best tools to find bugs in code since all those wrong assumptions in 
your code come back to bite you in the ass :)

Cheers,

Michael

From casevh at gmail.com  Wed Feb  4 21:52:52 2009
From: casevh at gmail.com (Case Vanhorsen)
Date: Wed, 4 Feb 2009 12:52:52 -0800
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
In-Reply-To: <4989ED2A.2080303@behnel.de>
References: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
	<4986BADB.50202@student.matnat.uio.no> <4989ED2A.2080303@behnel.de>
Message-ID: <99e0b9530902041252l5d8ceee6g5a9c4118e55c92@mail.gmail.com>

On Wed, Feb 4, 2009 at 11:31 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
>
> Dag Sverre Seljebotn wrote:
>> Case Vanhorsen wrote:
>>> I just tried to install Cython 0.11 beta with Python 3.0. I get the
>>> following error:
>>>
>>> case:~/src/Cython-0.11.beta$ py30 setup.py install
>>> Traceback (most recent call last):
>>>   File "setup.py", line 5, in <module>
>>>     from Cython.Compiler.Version import version
>>>   File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
>>>     from Shadow import *
>>> ImportError: No module named Shadow
>>>
>>> It does install successfully using Python 2.5 and 2.6. Please let me
>>> know if there is any other information you need.
>>>
>> I had the impression that Cython doesn't support being run under Python
>> 3 (though Stefan would have the definitive answer on that). I think you
>> must install on Python 2.6, and then compile your .pyx to .c under 2.6,
>> and THEN you can compile the c file with the Python 3 headers to get a
>> Python 3 module.
>
> That is true. While the generated C source code supports Python 2.3 through
> 3.0 without changes, the Cython compiler itself requires Python 2.3 through
> 2.6 to run.
>
> As this does not limit Cython's functionality or applicability in any way,
> it's not a major priority for us to migrate Cython's own source code to Py3.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Thanks for the information. That's perfectly fine; I just wasn't aware of that.

Case

From jasone at canonware.com  Thu Feb  5 20:00:01 2009
From: jasone at canonware.com (Jason Evans)
Date: Thu, 05 Feb 2009 11:00:01 -0800
Subject: [Cython] Docstrings for 'cdef public' attributes?
Message-ID: <498B3731.6090001@canonware.com>

Is there any reasonable way to attach docstrings to 'cdef public' 
attributes?  The following does not work:

cdef class Spam:
     cdef public int spam:
         """
             Docstring for spam.
         """

spam.pyx:2:9: Suite attached to non-function declaration

Thanks,
Jason

From stefan_ml at behnel.de  Thu Feb  5 20:59:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Feb 2009 20:59:27 +0100
Subject: [Cython] Docstrings for 'cdef public' attributes?
In-Reply-To: <498B3731.6090001@canonware.com>
References: <498B3731.6090001@canonware.com>
Message-ID: <498B451F.7060700@behnel.de>

Hi,

Jason Evans wrote:
> Is there any reasonable way to attach docstrings to 'cdef public' 
> attributes?

I'm not aware of any, but I miss that feature, too.


>  The following does not work:
> 
> cdef class Spam:
>      cdef public int spam:
>          """
>              Docstring for spam.
>          """

This is not a suitable syntax, as we might allow default values for
attributes one day, as in

	cdef public int spam = 1

Putting the docstring before the declaration doesn't work with Python's
syntax either.

I'm more in favour of a decorator as a potential future syntax:

	@cython.docstring("The infamous spam number")
	cdef public int spam = 1

Could you file a wishlist bug for this?

Thanks,

Stefan

From dagss at student.matnat.uio.no  Thu Feb  5 22:15:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 05 Feb 2009 22:15:04 +0100
Subject: [Cython] Docstrings for 'cdef public' attributes?
In-Reply-To: <498B451F.7060700@behnel.de>
References: <498B3731.6090001@canonware.com> <498B451F.7060700@behnel.de>
Message-ID: <498B56D8.2020609@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Jason Evans wrote:
>> Is there any reasonable way to attach docstrings to 'cdef public' 
>> attributes?
> 
> I'm not aware of any, but I miss that feature, too.
> 
> 
>>  The following does not work:
>>
>> cdef class Spam:
>>      cdef public int spam:
>>          """
>>              Docstring for spam.
>>          """
> 
> This is not a suitable syntax, as we might allow default values for
> attributes one day, as in
> 
> 	cdef public int spam = 1
> 
> Putting the docstring before the declaration doesn't work with Python's
> syntax either.
> 
> I'm more in favour of a decorator as a potential future syntax:
> 
> 	@cython.docstring("The infamous spam number")
> 	cdef public int spam = 1


How about introducing a Pythonic syntax for the whole construct, a la 
"property"?

spam = cython.cproperty(type=cython.int, init=1, doc="The infamous spam 
number")


-- 
Dag Sverre

From dalcinl at gmail.com  Thu Feb  5 22:29:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 5 Feb 2009 19:29:25 -0200
Subject: [Cython] Need comments about Python integer -> C size_t
Message-ID: <e7ba66e40902051329h64ed699ak884424aaeeec5048@mail.gmail.com>

Would the chain below be enough?:

PyInt/PyLong -> unsigned PY_LONG_LONG -> size_t


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From magnus at hetland.org  Fri Feb  6 12:17:37 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 12:17:37 +0100
Subject: [Cython] Measuring "C-ness"?
Message-ID: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>

I'm writing a system where I build data structures for searching using  
a blend of Cython and Python code. Ideally, though, the search  
operations (performed after the structures have been built) should be  
initiated from Python, but then stay in as pure C as possible (to keep  
the Python/Cython overhead to an absolute minimum).

Now, I could (and probably will) inspect the code to determine this  
"manually", but I was thinking -- perhaps I can automate it somehow?  
At least partially? And I thought I'd ask if you might have any ideas  
in that respect.

In other words, I'd like to profile the code, to see where it goes  
through Cython/the Python C API (or even Python code), so I can  
eliminate those calls. Could I just use a standard profiler here,  
perhaps? (I'm guessing all C stuff will then just be black-box  
references, but that's OK, I guess.) Any other ideas? Some global  
location where I could insert a counter into the generated code? Some  
existing Cython magic?

- M

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Fri Feb  6 12:41:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 6 Feb 2009 12:41:50 +0100 (CET)
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
Message-ID: <46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Magnus Lie Hetland wrote:
> I'd like to profile the code, to see where it goes
> through Cython/the Python C API (or even Python code), so I can
> eliminate those calls.

Have you looked at Cython's annotation (-a) feature? It will generate an
HTML page of your source code that highlights Python operations.

Stefan


From dagss at student.matnat.uio.no  Fri Feb  6 12:58:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 06 Feb 2009 12:58:15 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <498C25D7.1000803@student.matnat.uio.no>

Stefan Behnel wrote:
> Magnus Lie Hetland wrote:
>> I'd like to profile the code, to see where it goes
>> through Cython/the Python C API (or even Python code), so I can
>> eliminate those calls.
> 
> Have you looked at Cython's annotation (-a) feature? It will generate an
> HTML page of your source code that highlights Python operations.

Also, if you declare functions as "nogil" then Python operations will be 
disallowed.

-- 
Dag Sverre

From magnus at hetland.org  Fri Feb  6 12:51:09 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 12:51:09 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <498C25D7.1000803@student.matnat.uio.no>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<498C25D7.1000803@student.matnat.uio.no>
Message-ID: <2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>

On Feb 6, 2009, at 12:58, Dag Sverre Seljebotn wrote:

> Also, if you declare functions as "nogil" then Python operations  
> will be
> disallowed.

I see. To what degree does that involve Python C API stuff for  
converting objects and the like? (E.g., if I -- by accident -- convert  
between char* and str when calling one of my functions/methods from  
another, etc.)

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Fri Feb  6 13:20:45 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 6 Feb 2009 13:20:45 +0100 (CET)
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<498C25D7.1000803@student.matnat.uio.no>
	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>
Message-ID: <64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Magnus Lie Hetland wrote:
> On Feb 6, 2009, at 12:58, Dag Sverre Seljebotn wrote:
>> Also, if you declare functions as "nogil" then Python operations
>> will be disallowed.
>
> I see. To what degree does that involve Python C API stuff for
> converting objects and the like? (E.g., if I -- by accident -- convert
> between char* and str when calling one of my functions/methods from
> another, etc.)

Any Python operation that involves ref-counting is denied.

Stefan


From magnus at hetland.org  Fri Feb  6 13:37:42 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 13:37:42 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<498C25D7.1000803@student.matnat.uio.no>
	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>
	<64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>

On Feb 6, 2009, at 13:20, Stefan Behnel wrote:

> Any Python operation that involves ref-counting is denied.

Great! :) That, along with the -a switch, ought to take me far, I think.

Thanks,

- M

-- 
Magnus Lie Hetland
http://hetland.org



From dagss at student.matnat.uio.no  Fri Feb  6 14:40:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 06 Feb 2009 14:40:09 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<498C25D7.1000803@student.matnat.uio.no>	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>	<64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>
Message-ID: <498C3DB9.3040408@student.matnat.uio.no>

Magnus Lie Hetland wrote:
> On Feb 6, 2009, at 13:20, Stefan Behnel wrote:
> 
>> Any Python operation that involves ref-counting is denied.
> 
> Great! :) That, along with the -a switch, ought to take me far, I think.

If you think of something better while you do this then please tell us. 
A compiler directive for needing explicit "object" declarations, or a 
directive to disable automatic coercions between C types and objects, or 
perhaps a very smart IDE plugin which colors statements of different 
sorts in different colors? Probably easier to think about while you do 
this in practice.

-- 
Dag Sverre

From dalcinl at gmail.com  Fri Feb  6 15:35:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 12:35:19 -0200
Subject: [Cython] please, review this commit
Message-ID: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>

I had to push this patch.

changeset:   1694:5d63d5481ed5
tag:         tip
user:        Lisandro Dalcin <dalcinl at gmail.com>
date:        Fri Feb 06 12:29:32 2009 -0200
summary:     fix type conversion functions involving PY_LONG_LONG for Python 3

If any of you has a better idea, feel free to rollback it...

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From magnus at hetland.org  Fri Feb  6 16:13:16 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 16:13:16 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <498C3DB9.3040408@student.matnat.uio.no>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<498C25D7.1000803@student.matnat.uio.no>	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>	<64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>
	<498C3DB9.3040408@student.matnat.uio.no>
Message-ID: <C903B26B-BF00-42A0-B70C-B34C81B0766D@hetland.org>

On Feb 6, 2009, at 14:40, Dag Sverre Seljebotn wrote:

> Magnus Lie Hetland wrote:
>> On Feb 6, 2009, at 13:20, Stefan Behnel wrote:
>>
>>> Any Python operation that involves ref-counting is denied.
>>
>> Great! :) That, along with the -a switch, ought to take me far, I  
>> think.
>
> If you think of something better while you do this then please tell  
> us.

[snip]

Can't make any guarantees (of course), but I'll think about it :)

-- 
Magnus Lie Hetland
http://hetland.org



From dalcinl at gmail.com  Fri Feb  6 16:46:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 13:46:37 -0200
Subject: [Cython] negative integer -> unsigned integer,
	TypeError or OverflowError?
Message-ID: <e7ba66e40902060746g205d6114l1312abf691912d17@mail.gmail.com>

Core CPython is not consistent about this. What should we do in
Cython? Should we report this to Python-Dev?

Long story:

In core CPython (svn trunk), at Objects/longobjects.c, you should see
that in almost all cases OverflowError is raise. But see this one:

int
_PyLong_AsByteArray(PyLongObject* v,
		    unsigned char* bytes, size_t n,
		    int little_endian, int is_signed)
{
<...>
		if (!is_signed) {
			PyErr_SetString(PyExc_TypeError,
				"can't convert negative long to unsigned");
			return -1;
		}
<...>
}


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jasone at canonware.com  Fri Feb  6 17:12:48 2009
From: jasone at canonware.com (Jason Evans)
Date: Fri, 06 Feb 2009 08:12:48 -0800
Subject: [Cython] Docstrings for 'cdef public' attributes?
In-Reply-To: <498B451F.7060700@behnel.de>
References: <498B3731.6090001@canonware.com> <498B451F.7060700@behnel.de>
Message-ID: <498C6180.2040600@canonware.com>

Stefan Behnel wrote:
> Jason Evans wrote:
>> Is there any reasonable way to attach docstrings to 'cdef public' 
>> attributes?
> 
> I'm not aware of any, but I miss that feature, too.
> 
> Could you file a wishlist bug for this?

Done: http://trac.cython.org/cython_trac/ticket/206

Thanks,
Jason

From dalcinl at gmail.com  Fri Feb  6 19:21:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 16:21:21 -0200
Subject: [Cython] Bad code generation involving cimport and cdef public
	typed member in cdef classes
Message-ID: <e7ba66e40902061021q61bb9b06q80348f6ef4fbcb09@mail.gmail.com>

Perhaps you noticed warnings like this when building Cython

/u/dalcinl/Devel/cython-devel/Cython/Compiler/Scanning.c:8878:
warning: '__pyx_pf_6Cython_4Plex_8Scanners_7Scanner_5queue___get__'
defined but not used
/u/dalcinl/Devel/cython-devel/Cython/Compiler/Scanning.c:8897:
warning: '__pyx_pf_6Cython_4Plex_8Scanners_7Scanner_5queue___set__'
defined but not used

What happens? Cython.Compiler.Scanning does a 'cimport' or
Cython.Plex.Scanner. The later has a typed 'cdef public list queue'
member. As it is typed (a 'list'), Cython is smart and understand that
assignements needs type-checking, and builds a 'property' with get/set
routines for this. But the get/set routines SHOULD NOT be generated on
Cython/Compiler/Scanning.c.

It seem that the fix should go to Cython/Compiler/Nodes.py, at the
begining of CVarDefNode.analyse_declarations():

        if (dest_scope.is_c_class_scope
                and self.visibility == 'public'
                and base_type.is_pyobject
                and (base_type.is_builtin_type or base_type.is_extension_type)):
            self.need_properties = []
            need_property = True
            visibility = 'private'

The line 'need_property = True' should be 'False' instead  if the
dest_scope is a cimport'ed class.

Any idea about how can I fix this?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb  6 20:00:09 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Feb 2009 20:00:09 +0100
Subject: [Cython] negative integer -> unsigned integer,
	TypeError or OverflowError?
In-Reply-To: <e7ba66e40902060746g205d6114l1312abf691912d17@mail.gmail.com>
References: <e7ba66e40902060746g205d6114l1312abf691912d17@mail.gmail.com>
Message-ID: <498C88B9.4080007@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Core CPython is not consistent about this. What should we do in
> Cython? Should we report this to Python-Dev?
> 
> Long story:
> 
> In core CPython (svn trunk), at Objects/longobjects.c, you should see
> that in almost all cases OverflowError is raise. But see this one:
> 
> int
> _PyLong_AsByteArray(PyLongObject* v,
> 		    unsigned char* bytes, size_t n,
> 		    int little_endian, int is_signed)
> {
> <...>
> 		if (!is_signed) {
> 			PyErr_SetString(PyExc_TypeError,
> 				"can't convert negative long to unsigned");
> 			return -1;
> 		}
> <...>
> }

Raising an OverflowError sounds like the right thing. If CPython raises it
in almost all cases, it's ok if Cython does that, too.

Still, asking on python-dev would allow them to take a decision and maybe
even fix the above discrepancy.

Stefan

From dalcinl at gmail.com  Fri Feb  6 20:48:31 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 17:48:31 -0200
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
Message-ID: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>

What do you think about this?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb  6 20:57:14 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Feb 2009 20:57:14 +0100
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
In-Reply-To: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
References: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
Message-ID: <498C961A.3070403@behnel.de>


Lisandro Dalcin wrote:
> What do you think about this?

Fine with me. None is a very common value and often used in boolean checks like

    if reference_that_may_be_None:
        ...

I bet the C compiler will even be able to figure out that such a variable
is None in some simple cases.

Stefan


From dalcinl at gmail.com  Fri Feb  6 21:51:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 18:51:43 -0200
Subject: [Cython] ready to push changes to make Cython understand 'size_t'
	as a native C type
Message-ID: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>

The only thing that I'm not completely sure about is the
implementation of this, though it works fine in my 32bit Linux box

static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);

Appart, from that, the testsuite seems to run fine for me...


Of course, this change is not backward compatible... Cython will not
compile the code:

cdef extern from "stdlib.h":
    ctypedef unsigned long size_t

However, I've noticed that if I try to ctypedef Py_ssize_t, then
Cython does not even emit a warning !!!

What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
or 'size_t', provided that Cython treats them as pre-defined C types?


Finally, can I push the changes?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb  6 22:11:16 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Feb 2009 22:11:16 +0100
Subject: [Cython] ready to push changes to make Cython understand
 'size_t' as a native C type
In-Reply-To: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
Message-ID: <498CA774.1030506@behnel.de>

Hi Lisandro,

Lisandro Dalcin wrote:
> The only thing that I'm not completely sure about is the
> implementation of this, though it works fine in my 32bit Linux box
> 
> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);

I'd need to see the implementation to comment on this.


> Of course, this change is not backward compatible... Cython will not
> compile the code:
> 
> cdef extern from "stdlib.h":
>     ctypedef unsigned long size_t

Is this because of an explicit error, or just a side effect of the
implementation?


> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
> or 'size_t', provided that Cython treats them as pre-defined C types?

I didn't care at the time when I implemented the Py_ssize_t stuff, but when
you ask now, I prefer getting at least a warning. I doubt that Cython
handles a "ctypedef ... int" gracefully, for example.


> Finally, can I push the changes?

Could you file a bug report for now (if only for documentation purposes)
and post the patch there? We should be a bit careful with major changes by
now, as we are currently trying to stabilise Cython for the 0.11 release.

Thanks,

Stefan

From robertwb at math.washington.edu  Fri Feb  6 22:47:18 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 13:47:18 -0800
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
In-Reply-To: <498C961A.3070403@behnel.de>
References: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
	<498C961A.3070403@behnel.de>
Message-ID: <605AE54D-F7DE-4E47-8B8C-33C88823B504@math.washington.edu>

On Feb 6, 2009, at 11:57 AM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> What do you think about this?
>
> Fine with me. None is a very common value and often used in boolean  
> checks like
>
>     if reference_that_may_be_None:
>         ...
>
> I bet the C compiler will even be able to figure out that such a  
> variable
> is None in some simple cases.

+1 from me too--so it will check for True, False, and None before  
deferring to the generic code then.

- Robert


From robertwb at math.washington.edu  Fri Feb  6 22:49:54 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 13:49:54 -0800
Subject: [Cython] please, review this commit
In-Reply-To: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>
References: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>
Message-ID: <7636BBAB-0C85-4954-AA42-2AF8BB1A5DD1@math.washington.edu>

Looks good to me--do you have any tests to go with it?

On Feb 6, 2009, at 6:35 AM, Lisandro Dalcin wrote:

> I had to push this patch.
>
> changeset:   1694:5d63d5481ed5
> tag:         tip
> user:        Lisandro Dalcin <dalcinl at gmail.com>
> date:        Fri Feb 06 12:29:32 2009 -0200
> summary:     fix type conversion functions involving PY_LONG_LONG  
> for Python 3
>
> If any of you has a better idea, feel free to rollback it...
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Fri Feb  6 22:50:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 19:50:43 -0200
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
In-Reply-To: <605AE54D-F7DE-4E47-8B8C-33C88823B504@math.washington.edu>
References: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
	<498C961A.3070403@behnel.de>
	<605AE54D-F7DE-4E47-8B8C-33C88823B504@math.washington.edu>
Message-ID: <e7ba66e40902061350m6e665148l11f9f75bd5dfcc80@mail.gmail.com>

Pushed...

On Fri, Feb 6, 2009 at 7:47 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 6, 2009, at 11:57 AM, Stefan Behnel wrote:
>
>> Lisandro Dalcin wrote:
>>> What do you think about this?
>>
>> Fine with me. None is a very common value and often used in boolean
>> checks like
>>
>>     if reference_that_may_be_None:
>>         ...
>>
>> I bet the C compiler will even be able to figure out that such a
>> variable
>> is None in some simple cases.
>
> +1 from me too--so it will check for True, False, and None before
> deferring to the generic code then.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Feb  6 22:52:30 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 19:52:30 -0200
Subject: [Cython] please, review this commit
In-Reply-To: <7636BBAB-0C85-4954-AA42-2AF8BB1A5DD1@math.washington.edu>
References: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>
	<7636BBAB-0C85-4954-AA42-2AF8BB1A5DD1@math.washington.edu>
Message-ID: <e7ba66e40902061352jd90693brd073cb6f467ab209@mail.gmail.com>

No yet.... Once I get the 'size_t' patch in, I'll try to write a good
testcase to check Py <-> C integral conversions...

On Fri, Feb 6, 2009 at 7:49 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Looks good to me--do you have any tests to go with it?
>
> On Feb 6, 2009, at 6:35 AM, Lisandro Dalcin wrote:
>
>> I had to push this patch.
>>
>> changeset:   1694:5d63d5481ed5
>> tag:         tip
>> user:        Lisandro Dalcin <dalcinl at gmail.com>
>> date:        Fri Feb 06 12:29:32 2009 -0200
>> summary:     fix type conversion functions involving PY_LONG_LONG
>> for Python 3
>>
>> If any of you has a better idea, feel free to rollback it...
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Feb  6 22:59:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 13:59:03 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988A2A7.8060204@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
Message-ID: <20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>

On Feb 3, 2009, at 12:01 PM, Michael Abshoff wrote:

> Lisandro Dalcin wrote:
>> Cython treats sizeof() as if it returns 'int', but IMHO it should  
>> be 'size_t'.
>
> Well, you might be technically correct since that is what the C  
> standard
> asks for, but according to various literature, i.e. for example
>
> http://publications.gbdirect.co.uk/c_book/chapter5/ 
> sizeof_and_malloc.html
>
> "The sizeof operator returns the size in bytes of its operand. Whether
> the result of sizeof is unsigned int or unsigned long is  
> implementation
> defined?which is why the declaration of malloc above ducked the  
> issue by
> omitting any parameter information; "

I think "int" is the best thing to return--it can always get coerced  
up to a size_t if needed, but the other way may prove problematic.  
Also, "int" is a more abstract (conceptually at least) type (better  
fitting with the ambiguity in the C standard), and we shouldn't have  
to worry about overflow in this case.

>> Implementing this the right way would require to make Cython  
>> recognize
>> 'size_t' as a native type, more or less in the way it already
>> recognizes 'Py_ssize_t'. What do you think about this?
>
> Mmmh, do you want to define size_t in Cython or pull it in from
> somewhere? At least for C code that ought to be rather tricky for VS6
> for example which needs to work since we are still supporting  
> Python 2.3
> on Windows. But you could work around the issue by defining size_t in
> Cython as either unsigned int, unsigned long or unsigned long long
> depending on the platform. But that could potentially clash with  
> various
> C++ runtime libs.

I am -1 on defining size_t ourselves, but would like to pull it from  
somewhere if we could.

> Do you think that people will see overflow issues? I think the  
> tradeoff
> of following the C standard here is rather high since in reality  
> size_t
> in C is rather badly handled. And if you guys decide to follow the C
> standard here someone ought to break out Python 2.6 and VS 6 and  
> verify
> that things are still working :)

For sure.

- Robert


From dalcinl at gmail.com  Fri Feb  6 23:06:31 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 20:06:31 -0200
Subject: [Cython] ready to push changes to make Cython understand
	'size_t' as a native C type
In-Reply-To: <498CA774.1030506@behnel.de>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
	<498CA774.1030506@behnel.de>
Message-ID: <e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>

Opened http://trac.cython.org/cython_trac/ticket/207

On Fri, Feb 6, 2009 at 7:11 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi Lisandro,
>
> Lisandro Dalcin wrote:
>> The only thing that I'm not completely sure about is the
>> implementation of this, though it works fine in my 32bit Linux box
>>
>> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);
>
> I'd need to see the implementation to comment on this.
>

http://trac.cython.org/cython_trac/attachment/ticket/207/SIZE_T.diff#preview

>
>> Of course, this change is not backward compatible... Cython will not
>> compile the code:
>>
>> cdef extern from "stdlib.h":
>>     ctypedef unsigned long size_t
>
> Is this because of an explicit error, or just a side effect of the
> implementation?
>

Cython generates the following error (IMHO, that's a good thing)

Error converting Pyrex file to C:
------------------------------------------------------------
...
#cython: embedsignature=True
ctypedef unsigned long size_t
                            ^
------------------------------------------------------------

/u/dalcinl/Devel/mpi4py-dev/src/mpi4py.MPI.pyx:2:29: Empty declarator


>
>> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
>> or 'size_t', provided that Cython treats them as pre-defined C types?
>
> I didn't care at the time when I implemented the Py_ssize_t stuff, but when
> you ask now, I prefer getting at least a warning. I doubt that Cython
> handles a "ctypedef ... int" gracefully, for example.
>

Sorry for the noise, I got confused.. Cython DO FAIL if you 'ctypedef
... Py_ssize_t'.

>
>> Finally, can I push the changes?
>
> Could you file a bug report for now (if only for documentation purposes)
> and post the patch there? We should be a bit careful with major changes by
> now, as we are currently trying to stabilise Cython for the 0.11 release.
>

OK. Just hope this can get in for the next release.

> Thanks,
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lists at cheimes.de  Fri Feb  6 23:38:19 2009
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 06 Feb 2009 23:38:19 +0100
Subject: [Cython] How access and modify errno?
Message-ID: <gmie4r$vnr$1@ger.gmane.org>

Hi fellow Cython users!

I need to set errno = 0 and later check it for != 0. How can I stop
Cython from mangling the name of my errno variable?

Example code:

cdef extern from "errno.h":
    int errno

def method():
    errno = 0

The Cython compiler insists on creating a new local variable "int
__pyx_v_errno" for errno. Is there a way to stop Cython from doing that
or do I have to use an extra header file with some macro magic?

Christian


From carl.witty at gmail.com  Fri Feb  6 23:42:51 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Fri, 6 Feb 2009 14:42:51 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
Message-ID: <ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>

On Fri, Feb 6, 2009 at 1:59 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 3, 2009, at 12:01 PM, Michael Abshoff wrote:
>
>> Lisandro Dalcin wrote:
>>> Cython treats sizeof() as if it returns 'int', but IMHO it should
>>> be 'size_t'.
>>
>> Well, you might be technically correct since that is what the C
>> standard
>> asks for, but according to various literature, i.e. for example
>>
>> http://publications.gbdirect.co.uk/c_book/chapter5/
>> sizeof_and_malloc.html
>>
>> "The sizeof operator returns the size in bytes of its operand. Whether
>> the result of sizeof is unsigned int or unsigned long is
>> implementation
>> defined?which is why the declaration of malloc above ducked the
>> issue by
>> omitting any parameter information; "
>
> I think "int" is the best thing to return--it can always get coerced
> up to a size_t if needed, but the other way may prove problematic.
> Also, "int" is a more abstract (conceptually at least) type (better
> fitting with the ambiguity in the C standard), and we shouldn't have
> to worry about overflow in this case.

Certainly the result of sizeof can overflow an int.  (Presumably on
64-bit Windows, it can overflow unsigned long, as well.)

Or are you saying that we don't have to worry about it because people
won't define such large types?  That may be true in practice, but it
makes me uncomfortable.

Carl

From robertwb at math.washington.edu  Fri Feb  6 23:50:44 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 14:50:44 -0800
Subject: [Cython] How access and modify errno?
In-Reply-To: <gmie4r$vnr$1@ger.gmane.org>
References: <gmie4r$vnr$1@ger.gmane.org>
Message-ID: <776B5273-748A-46B1-B49A-6140FEA4F635@math.washington.edu>

On Feb 6, 2009, at 2:38 PM, Christian Heimes wrote:

> Hi fellow Cython users!
>
> I need to set errno = 0 and later check it for != 0. How can I stop
> Cython from mangling the name of my errno variable?
>
> Example code:
>
> cdef extern from "errno.h":
>     int errno
>
> def method():

global errno

>     errno = 0
>
> The Cython compiler insists on creating a new local variable "int
> __pyx_v_errno" for errno. Is there a way to stop Cython from doing  
> that
> or do I have to use an extra header file with some macro magic?

No. Cython scoping is just like Python scoping--an assignment creates  
a local in the given scope.

- Robert


From michael.abshoff at googlemail.com  Fri Feb  6 23:52:48 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 06 Feb 2009 14:52:48 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
	<ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>
Message-ID: <498CBF40.8080204@gmail.com>

Carl Witty wrote:
> On Fri, Feb 6, 2009 at 1:59 PM, Robert Bradshaw

<SNIP>

>> I think "int" is the best thing to return--it can always get coerced
>> up to a size_t if needed, but the other way may prove problematic.
>> Also, "int" is a more abstract (conceptually at least) type (better
>> fitting with the ambiguity in the C standard), and we shouldn't have
>> to worry about overflow in this case.
> 
> Certainly the result of sizeof can overflow an int.  (Presumably on
> 64-bit Windows, it can overflow unsigned long, as well.)
> 
> Or are you saying that we don't have to worry about it because people
> won't define such large types?  That may be true in practice, but it
> makes me uncomfortable.

+1

There *is* actually code in the Sage library that has overflow potential 
on 64 bit Windows and it needs to be fixed. Debugging some strange 
segfault caused by the above is hard enough at it is, adding platform 
specific issues doesn't make it any better.

The C and C++ people did define size_t for a reason since all those 
assumptions about int or long being "big enough" does not hold true in 
the real world, so let's do this by the book.

> Carl

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb  6 23:56:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 14:56:07 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <498CBF40.8080204@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
	<ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>
	<498CBF40.8080204@gmail.com>
Message-ID: <C329E07F-C839-495D-AEB6-B89173D7A4EE@math.washington.edu>

On Feb 6, 2009, at 2:52 PM, Michael Abshoff wrote:

> Carl Witty wrote:
>> On Fri, Feb 6, 2009 at 1:59 PM, Robert Bradshaw
>
> <SNIP>
>
>>> I think "int" is the best thing to return--it can always get coerced
>>> up to a size_t if needed, but the other way may prove problematic.
>>> Also, "int" is a more abstract (conceptually at least) type (better
>>> fitting with the ambiguity in the C standard), and we shouldn't have
>>> to worry about overflow in this case.
>>
>> Certainly the result of sizeof can overflow an int.  (Presumably on
>> 64-bit Windows, it can overflow unsigned long, as well.)
>>
>> Or are you saying that we don't have to worry about it because people
>> won't define such large types?

That is what I was implying. But it isn't valid in retrospect because  
one might do, say, malloc(sizeof(T) * 4000000000)

>> That may be true in practice, but it
>> makes me uncomfortable.
>
> +1
>
> There *is* actually code in the Sage library that has overflow  
> potential
> on 64 bit Windows and it needs to be fixed. Debugging some strange
> segfault caused by the above is hard enough at it is, adding platform
> specific issues doesn't make it any better.
>
> The C and C++ people did define size_t for a reason since all those
> assumptions about int or long being "big enough" does not hold true in
> the real world, so let's do this by the book.

I hadn't yet read the rest of this thread where it looks like  
Python.h assumes (or defines?) size_t. Given that's the case, I'm  
very much in favor of making it native to Cython and the sizeof  
return type.

- Robert



From dalcinl at gmail.com  Fri Feb  6 23:57:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 20:57:27 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
Message-ID: <e7ba66e40902061457v1abd84eegfa6debb1a7683497@mail.gmail.com>

On Fri, Feb 6, 2009 at 7:59 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 3, 2009, at 12:01 PM, Michael Abshoff wrote:
>
> I think "int" is the best thing to return--it can always get coerced
> up to a size_t if needed, but the other way may prove problematic.
> Also, "int" is a more abstract (conceptually at least) type (better
> fitting with the ambiguity in the C standard), and we shouldn't have
> to worry about overflow in this case.
>

I have to strongly disagree with you.

Try the output of this Cython code

print sizeof(char[1<<30]) * 3

versus the output of this C code:

#include "stdio.h"
int main() {
  printf("%u\n", sizeof(char[1<<30])*3);
}


For the first you get -1073741824 (clearly wrong), and for the second
3221225472. (Used a 32 bit box)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Sat Feb  7 00:03:27 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 06 Feb 2009 15:03:27 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <C329E07F-C839-495D-AEB6-B89173D7A4EE@math.washington.edu>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>	<ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>	<498CBF40.8080204@gmail.com>
	<C329E07F-C839-495D-AEB6-B89173D7A4EE@math.washington.edu>
Message-ID: <498CC1BF.1010406@gmail.com>

Robert Bradshaw wrote:
> On Feb 6, 2009, at 2:52 PM, Michael Abshoff wrote:

<SNIP>

> I hadn't yet read the rest of this thread where it looks like  
> Python.h assumes (or defines?) size_t. Given that's the case, I'm  
> very much in favor of making it native to Cython and the sizeof  
> return type.

No problem, you must be dealing with a rather large amount of email :)

Anyway, welcome back.

> - Robert

Cheers,

Michael


> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From lists at cheimes.de  Sat Feb  7 00:34:32 2009
From: lists at cheimes.de (Christian Heimes)
Date: Sat, 07 Feb 2009 00:34:32 +0100
Subject: [Cython] How access and modify errno?
In-Reply-To: <776B5273-748A-46B1-B49A-6140FEA4F635@math.washington.edu>
References: <gmie4r$vnr$1@ger.gmane.org>
	<776B5273-748A-46B1-B49A-6140FEA4F635@math.washington.edu>
Message-ID: <gmihe8$8uf$1@ger.gmane.org>

Robert Bradshaw wrote:
> global errno
> 
>>     errno = 0
>>
>> The Cython compiler insists on creating a new local variable "int
>> __pyx_v_errno" for errno. Is there a way to stop Cython from doing  
>> that
>> or do I have to use an extra header file with some macro magic?
> 
> No. Cython scoping is just like Python scoping--an assignment creates  
> a local in the given scope.

Doh! That was too easy ... :(
Why didn't I think of global in the first place?

Thanks Robert!

Christian


From greg.ewing at canterbury.ac.nz  Sat Feb  7 09:24:39 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 07 Feb 2009 21:24:39 +1300
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
Message-ID: <498D4547.8060308@canterbury.ac.nz>

Magnus Lie Hetland wrote:
> Ideally, though, the search  
> operations (performed after the structures have been built) should be  
> initiated from Python, but then stay in as pure C as possible

One way to guarantee that something uses purely C
operations is to actually write it in C, as an
external piece of code, and call it from Cython.

-- 
Greg

From magnus at hetland.org  Sat Feb  7 12:44:55 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sat, 7 Feb 2009 12:44:55 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <498D4547.8060308@canterbury.ac.nz>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
Message-ID: <5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>

On Feb 7, 2009, at 09:24 , Greg Ewing wrote:

> Magnus Lie Hetland wrote:
>> Ideally, though, the search
>> operations (performed after the structures have been built) should be
>> initiated from Python, but then stay in as pure C as possible
>
> One way to guarantee that something uses purely C
> operations is to actually write it in C, as an
> external piece of code, and call it from Cython.


Absolutely an option. It may even be the most sensible thing to do,  
rather than trying to program in C using Cython :-)

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Sat Feb  7 14:08:05 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 07 Feb 2009 14:08:05 +0100
Subject: [Cython] ready to push changes to make Cython understand
 'size_t' as a native C type
In-Reply-To: <e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>	<498CA774.1030506@behnel.de>
	<e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>
Message-ID: <498D87B5.3040208@behnel.de>


Lisandro Dalcin wrote:
>>> The only thing that I'm not completely sure about is the
>>> implementation of this, though it works fine in my 32bit Linux box
>>>
>>> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);
> 
> http://trac.cython.org/cython_trac/attachment/ticket/207/SIZE_T.diff#preview

	static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b) {
 		 unsigned PY_LONG_LONG val =
				 __pyx_PyInt_AsUnsignedLongLong(b);
 		 if (unlikely(val == (unsigned PY_LONG_LONG)-1 &&
				 PyErr_Occurred())) {
 		       return (size_t)-1;

The "(size_t)-1" definitely looks funny. I noticed that "(unsigned
PY_LONG_LONG)-1" is also used in __pyx_PyInt_AsUnsignedLongLong(), so I
just hope it does the right thing.


 		  } else if (unlikely((unsigned PY_LONG_LONG)(size_t)val
							 != val)) {
 		       PyErr_SetString(PyExc_OverflowError, "value too
					 large to convert to size_t");

This should return an error value, so it lacks a "return (size_t)-1".

 		  }
 		  return val;
	}

It would be great if we had tests for the error cases. It's not obvious
that they work as expected. The same applies to the "unsigned long long"
conversion, IMHO.


>>> Of course, this change is not backward compatible... Cython will not
>>> compile the code:
>>>
>>> cdef extern from "stdlib.h":
>>>     ctypedef unsigned long size_t
>> Is this because of an explicit error, or just a side effect of the
>> implementation?
>>
> 
> Cython generates the following error (IMHO, that's a good thing)
> 
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> #cython: embedsignature=True
> ctypedef unsigned long size_t
>                             ^
> ------------------------------------------------------------
> 
> /u/dalcinl/Devel/mpi4py-dev/src/mpi4py.MPI.pyx:2:29: Empty declarator

Not really the most telling error message, but not cryptic either. Fine
with me.


>>> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
>>> or 'size_t', provided that Cython treats them as pre-defined C types?
>> I didn't care at the time when I implemented the Py_ssize_t stuff, but when
>> you ask now, I prefer getting at least a warning. I doubt that Cython
>> handles a "ctypedef ... int" gracefully, for example.
> 
> Sorry for the noise, I got confused.. Cython DO FAIL if you 'ctypedef
> ... Py_ssize_t'.

Same as above, I assume: it thinks that you are still collecting types and
modifiers, so it's missing the new type name at the end.


>>> Finally, can I push the changes?
>> Could you file a bug report for now (if only for documentation purposes)
>> and post the patch there? We should be a bit careful with major changes by
>> now, as we are currently trying to stabilise Cython for the 0.11 release.
>>
> 
> OK. Just hope this can get in for the next release.

Apart from what I said above, it looks ok, so I'd like to see it in as well.

Stefan


From dalcinl at gmail.com  Sat Feb  7 18:39:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 14:39:44 -0300
Subject: [Cython] ready to push changes to make Cython understand
	'size_t' as a native C type
In-Reply-To: <498D87B5.3040208@behnel.de>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
	<498CA774.1030506@behnel.de>
	<e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>
	<498D87B5.3040208@behnel.de>
Message-ID: <e7ba66e40902070939k48194864q9eb5a4719f303304@mail.gmail.com>

On Sat, Feb 7, 2009 at 10:08 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>>>> The only thing that I'm not completely sure about is the
>>>> implementation of this, though it works fine in my 32bit Linux box
>>>>
>>>> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);
>>
>> http://trac.cython.org/cython_trac/attachment/ticket/207/SIZE_T.diff#preview
>
>        static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b) {
>                 unsigned PY_LONG_LONG val =
>                                 __pyx_PyInt_AsUnsignedLongLong(b);
>                 if (unlikely(val == (unsigned PY_LONG_LONG)-1 &&
>                                 PyErr_Occurred())) {
>                       return (size_t)-1;
>
> The "(size_t)-1" definitely looks funny. I noticed that "(unsigned
> PY_LONG_LONG)-1" is also used in __pyx_PyInt_AsUnsignedLongLong(), so I
> just hope it does the right thing.
>

That's the way things are implemented in core CPython, and AFAIK it
should works, at least if Cython-generated C code is compiled with the
same C compiler as core CPython library. I do not know I the C
standard is explicit about what you get when you cast a negative
integer to an unsigned integer.

>
>                  } else if (unlikely((unsigned PY_LONG_LONG)(size_t)val
>                                                         != val)) {
>                       PyErr_SetString(PyExc_OverflowError, "value too
>                                         large to convert to size_t");
>
> This should return an error value, so it lacks a "return (size_t)-1".
>

OK, I'll fix that ...

>
> It would be great if we had tests for the error cases. It's not obvious
> that they work as expected. The same applies to the "unsigned long long"
> conversion, IMHO.
>

I've added a testcase for overflowing 'size_t' conversions. But yes,
such testcases should be implemented for all integral types... I'll
give a try next week in some spare time.


>
>>>> Of course, this change is not backward compatible... Cython will not
>>>> compile the code:
>>>>
>>>> cdef extern from "stdlib.h":
>>>>     ctypedef unsigned long size_t
>>> Is this because of an explicit error, or just a side effect of the
>>> implementation?
>>>
>>
>> Cython generates the following error (IMHO, that's a good thing)
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>> #cython: embedsignature=True
>> ctypedef unsigned long size_t
>>                             ^
>> ------------------------------------------------------------
>>
>> /u/dalcinl/Devel/mpi4py-dev/src/mpi4py.MPI.pyx:2:29: Empty declarator
>
> Not really the most telling error message, but not cryptic either. Fine
> with me.
>
>
>>>> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
>>>> or 'size_t', provided that Cython treats them as pre-defined C types?
>>> I didn't care at the time when I implemented the Py_ssize_t stuff, but when
>>> you ask now, I prefer getting at least a warning. I doubt that Cython
>>> handles a "ctypedef ... int" gracefully, for example.
>>
>> Sorry for the noise, I got confused.. Cython DO FAIL if you 'ctypedef
>> ... Py_ssize_t'.
>
> Same as above, I assume: it thinks that you are still collecting types and
> modifiers, so it's missing the new type name at the end.
>
>
>>>> Finally, can I push the changes?
>>> Could you file a bug report for now (if only for documentation purposes)
>>> and post the patch there? We should be a bit careful with major changes by
>>> now, as we are currently trying to stabilise Cython for the 0.11 release.
>>>
>>
>> OK. Just hope this can get in for the next release.
>
> Apart from what I said above, it looks ok, so I'd like to see it in as well.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sat Feb  7 20:10:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 16:10:01 -0300
Subject: [Cython] problems with C numeric types
Message-ID: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>

After pushing the changes for the 'size_t' stuff, now I believe that
things were a bit broken before the patch, and they still are.

+ Go Cython.Compiler.PyrexTypes

+ Look at the module-level tuple variable 'rank_to_type_name'. You can
see that 'Py_ssize_t' and 'size_t' are AFTER the type 'PY_LONG_LONG'.

+ Now look at the implementation of the function 'widest_numeric_type()'.

Then, this bit of Cython::

cdef f():
    cdef unsigned long long a = 5
    cdef size_t b = 7
    return a + b

generates C code like this::

  /* "/u/dalcinl/Devel/cython-devel/tmp.pyx":5
 *     cdef unsigned long long a = 5
 *     cdef size_t b = 7
 *     return a + b             # <<<<<<<<<<<<<<
 */
  /* __pyx_t_1 allocated */
  __pyx_t_1 = __pyx_PyInt_FromSize_t((__pyx_v_a + __pyx_v_b)); if
(unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 5;
__pyx_clineno = __LINE__; goto __pyx_L1_error;}



This is IMHO totally wrong !!


Could we assume that PY_LONG_LONG is the widest integral?? Should we
assume that 'Py_ssize_t' and 'size_t' are between 'long' and
'PY_LONG_LONG'? In such case, I'll have to fix the 'rank_to_type_name'
tuple, and perhaps the 'sign_and_rank_to_type' dictionary (used as a
fallback in function 'widest_numeric_type()')

However, I'm not sure how to we will handle code like the one below
with the current implementation,

cdef f():
    cdef Py_ssize_t a = 5
    cdef size_t b = 7
    return a + b

I believe the value of 'a+b' should be promoted to 'signed long long',
but it cannot see the way to implent such behavior in the current
Cython codebase.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Feb  7 22:13:26 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 07 Feb 2009 22:13:26 +0100
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
Message-ID: <498DF976.2050403@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Could we assume that PY_LONG_LONG is the widest integral?? Should we
> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
> 'PY_LONG_LONG'?

For starters, Py_ssize_t is defined as a signed size_t according to PEP
353. However, it does not mention a relation to PY_LONG_LONG.

I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the latter being
the largest available type), and my (limited) knowledge about the "long"
type makes me assume that it should be <= Py_ssize_t. So, yes, I would expect

	long <= Py_ssize_t <= PY_LONG_LONG

and

	unsigned long <= size_t <= unsigned PY_LONG_LONG


> However, I'm not sure how to we will handle code like the one below
> with the current implementation,
> 
> cdef f():
>     cdef Py_ssize_t a = 5
>     cdef size_t b = 7
>     return a + b
> 
> I believe the value of 'a+b' should be promoted to 'signed long long',
> but it cannot see the way to implent such behavior in the current
> Cython codebase.

Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's << size_t
for non-negative types, but the missing bit shouldn't matter in almost all
cases. And if users really need to make use of it, I'm fine with requiring
them to write

	return <PY_LONG_LONG>a + <PY_LONG_LONG>b

Stefan

From robertwb at math.washington.edu  Sun Feb  8 00:02:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:02:36 -0800
Subject: [Cython] problems with C numeric types
In-Reply-To: <498DF976.2050403@behnel.de>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
Message-ID: <957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>

On Feb 7, 2009, at 1:13 PM, Stefan Behnel wrote:

> Hi,
>
> Lisandro Dalcin wrote:
>> Could we assume that PY_LONG_LONG is the widest integral?? Should we
>> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
>> 'PY_LONG_LONG'?
>
> For starters, Py_ssize_t is defined as a signed size_t according to  
> PEP
> 353. However, it does not mention a relation to PY_LONG_LONG.
>
> I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the  
> latter being
> the largest available type), and my (limited) knowledge about the  
> "long"
> type makes me assume that it should be <= Py_ssize_t. So, yes, I  
> would expect
>
> 	long <= Py_ssize_t <= PY_LONG_LONG
>
> and
>
> 	unsigned long <= size_t <= unsigned PY_LONG_LONG

I think this is the right ordering for us to make. As we want to  
produce C files often destined to be compiled on other architectures,  
we can't actually test to see the order.

>> However, I'm not sure how to we will handle code like the one below
>> with the current implementation,
>>
>> cdef f():
>>     cdef Py_ssize_t a = 5
>>     cdef size_t b = 7
>>     return a + b
>>
>> I believe the value of 'a+b' should be promoted to 'signed long  
>> long',
>> but it cannot see the way to implent such behavior in the current
>> Cython codebase.
>
> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<  
> size_t
> for non-negative types, but the missing bit shouldn't matter in  
> almost all
> cases. And if users really need to make use of it, I'm fine with  
> requiring
> them to write
>
> 	return <PY_LONG_LONG>a + <PY_LONG_LONG>b

Actually, the target should be size_t, following the C standard that  
operations between signed and unsigned have an unsigned result. But  
we certainly shouldn't be promoting it to a new type.

- Robert


From robertwb at math.washington.edu  Sun Feb  8 00:09:09 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:09:09 -0800
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
Message-ID: <0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>

On Feb 7, 2009, at 3:44 AM, Magnus Lie Hetland wrote:

> On Feb 7, 2009, at 09:24 , Greg Ewing wrote:
>
>> Magnus Lie Hetland wrote:
>>> Ideally, though, the search
>>> operations (performed after the structures have been built)  
>>> should be
>>> initiated from Python, but then stay in as pure C as possible
>>
>> One way to guarantee that something uses purely C
>> operations is to actually write it in C, as an
>> external piece of code, and call it from Cython.
>
> Absolutely an option. It may even be the most sensible thing to do,
> rather than trying to program in C using Cython :-)

Of course this requires one to keep two files in sync, make "cdef  
extern" declarations, and have a more complicated setup.py (i.e. no  
using pyximport), and risk gcc as well as Cython errors. I "program C  
using Cython" all the time--using the annotate option it is easy to  
see when a block of code gets compiled to pure C. But it's a matter  
of personal preference.

- Robert


From dalcinl at gmail.com  Sun Feb  8 00:09:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:09:01 -0300
Subject: [Cython] problems with C numeric types
In-Reply-To: <498DF976.2050403@behnel.de>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
Message-ID: <e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>

On Sat, Feb 7, 2009 at 6:13 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Lisandro Dalcin wrote:
>> Could we assume that PY_LONG_LONG is the widest integral?? Should we
>> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
>> 'PY_LONG_LONG'?
>
> For starters, Py_ssize_t is defined as a signed size_t according to PEP
> 353. However, it does not mention a relation to PY_LONG_LONG.
>
> I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the latter being
> the largest available type), and my (limited) knowledge about the "long"
> type makes me assume that it should be <= Py_ssize_t. So, yes, I would expect
>
>        long <= Py_ssize_t <= PY_LONG_LONG
>
> and
>
>        unsigned long <= size_t <= unsigned PY_LONG_LONG
>

I think the same... Then I'll review this and push fixes.

>
>> However, I'm not sure how to we will handle code like the one below
>> with the current implementation,
>>
>> cdef f():
>>     cdef Py_ssize_t a = 5
>>     cdef size_t b = 7
>>     return a + b
>>
>> I believe the value of 'a+b' should be promoted to 'signed long long',
>> but it cannot see the way to implent such behavior in the current
>> Cython codebase.
>

Well, I was partially wrong in the above comments...

> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's << size_t
> for non-negative types, but the missing bit shouldn't matter in almost all
> cases.

It will not matter until you get a segfault or strange results... ;-)


> And if users really need to make use of it, I'm fine with requiring
> them to write
>
>        return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>

Yes, you are right, but then better for users to not forget the casts...


In short, If we can do it better, go for it and push testcases+fixes...




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sun Feb  8 00:16:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:16:27 -0800
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>
Message-ID: <142903B3-CEDA-4916-BF9E-842CFF6C57E1@math.washington.edu>

On Feb 7, 2009, at 3:09 PM, Lisandro Dalcin wrote:

> On Sat, Feb 7, 2009 at 6:13 PM, Stefan Behnel <stefan_ml at behnel.de>  
> wrote:
>> Hi,
>>
>> Lisandro Dalcin wrote:
>>> Could we assume that PY_LONG_LONG is the widest integral?? Should we
>>> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
>>> 'PY_LONG_LONG'?
>>
>> For starters, Py_ssize_t is defined as a signed size_t according  
>> to PEP
>> 353. However, it does not mention a relation to PY_LONG_LONG.
>>
>> I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the  
>> latter being
>> the largest available type), and my (limited) knowledge about the  
>> "long"
>> type makes me assume that it should be <= Py_ssize_t. So, yes, I  
>> would expect
>>
>>        long <= Py_ssize_t <= PY_LONG_LONG
>>
>> and
>>
>>        unsigned long <= size_t <= unsigned PY_LONG_LONG
>>
>
> I think the same... Then I'll review this and push fixes.

Excellent.

>
>>
>>> However, I'm not sure how to we will handle code like the one below
>>> with the current implementation,
>>>
>>> cdef f():
>>>     cdef Py_ssize_t a = 5
>>>     cdef size_t b = 7
>>>     return a + b
>>>
>>> I believe the value of 'a+b' should be promoted to 'signed long  
>>> long',
>>> but it cannot see the way to implent such behavior in the current
>>> Cython codebase.
>>
>
> Well, I was partially wrong in the above comments...
>
>> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<  
>> size_t
>> for non-negative types, but the missing bit shouldn't matter in  
>> almost all
>> cases.
>
> It will not matter until you get a segfault or strange results... ;-)
>
>
>> And if users really need to make use of it, I'm fine with requiring
>> them to write
>>
>>        return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>>
>
> Yes, you are right, but then better for users to not forget the  
> casts...
>
> In short, If we can do it better, go for it and push testcases 
> +fixes...


This is like asking if someone does

cdef unsigned int a
cdef int b
return a + b

I don't think a + b should be "upgraded" to a long (which may not  
even be bigger), just because it might overflow.

- Robert


From dalcinl at gmail.com  Sun Feb  8 00:17:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:17:25 -0300
Subject: [Cython] problems with C numeric types
In-Reply-To: <957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>
Message-ID: <e7ba66e40902071517i6b7b88er97ebc79e473781aa@mail.gmail.com>

On Sat, Feb 7, 2009 at 8:02 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 7, 2009, at 1:13 PM, Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>>
>>> cdef f():
>>>     cdef Py_ssize_t a = 5
>>>     cdef size_t b = 7
>>>     return a + b
>>>
>>> I believe the value of 'a+b' should be promoted to 'signed long
>>> long',
>>
>> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<
>> size_t
>> for non-negative types, but the missing bit shouldn't matter in
>> almost all
>> cases. And if users really need to make use of it, I'm fine with
>> requiring
>> them to write
>>
>>       return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>
> Actually, the target should be size_t, following the C standard that
> operations between signed and unsigned have an unsigned result. But
> we certainly shouldn't be promoting it to a new type.
>

Yes, you are right, but then this testcase is expected to fail...
Should we do better?

__doc__ = """
>>> f()
-1
"""
def f():
    cdef Py_ssize_t a = -2
    cdef size_t b = 1
    return a + b


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sun Feb  8 00:18:42 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:18:42 -0300
Subject: [Cython] problems with C numeric types
In-Reply-To: <142903B3-CEDA-4916-BF9E-842CFF6C57E1@math.washington.edu>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>
	<142903B3-CEDA-4916-BF9E-842CFF6C57E1@math.washington.edu>
Message-ID: <e7ba66e40902071518w64c1879od634eb105a41c6e8@mail.gmail.com>

On Sat, Feb 7, 2009 at 8:16 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> This is like asking if someone does
>
> cdef unsigned int a
> cdef int b
> return a + b
>
> I don't think a + b should be "upgraded" to a long (which may not
> even be bigger), just because it might overflow.
>

OK.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sun Feb  8 00:21:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:21:17 -0300
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
Message-ID: <e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>

BTW,  a feature I really miss from the times I was a heavy SWIG user
is the ability to include verbatim C code snipets...


On Sat, Feb 7, 2009 at 8:09 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 7, 2009, at 3:44 AM, Magnus Lie Hetland wrote:
>
>> On Feb 7, 2009, at 09:24 , Greg Ewing wrote:
>>
>>> Magnus Lie Hetland wrote:
>>>> Ideally, though, the search
>>>> operations (performed after the structures have been built)
>>>> should be
>>>> initiated from Python, but then stay in as pure C as possible
>>>
>>> One way to guarantee that something uses purely C
>>> operations is to actually write it in C, as an
>>> external piece of code, and call it from Cython.
>>
>> Absolutely an option. It may even be the most sensible thing to do,
>> rather than trying to program in C using Cython :-)
>
> Of course this requires one to keep two files in sync, make "cdef
> extern" declarations, and have a more complicated setup.py (i.e. no
> using pyximport), and risk gcc as well as Cython errors. I "program C
> using Cython" all the time--using the annotate option it is easy to
> see when a block of code gets compiled to pure C. But it's a matter
> of personal preference.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sun Feb  8 00:23:09 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:23:09 -0800
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071517i6b7b88er97ebc79e473781aa@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>
	<e7ba66e40902071517i6b7b88er97ebc79e473781aa@mail.gmail.com>
Message-ID: <7C4EE0C4-897E-417D-B02C-9E001C3FED4E@math.washington.edu>

On Feb 7, 2009, at 3:17 PM, Lisandro Dalcin wrote:

> On Sat, Feb 7, 2009 at 8:02 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Feb 7, 2009, at 1:13 PM, Stefan Behnel wrote:
>>> Lisandro Dalcin wrote:
>>>>
>>>> cdef f():
>>>>     cdef Py_ssize_t a = 5
>>>>     cdef size_t b = 7
>>>>     return a + b
>>>>
>>>> I believe the value of 'a+b' should be promoted to 'signed long
>>>> long',
>>>
>>> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<
>>> size_t
>>> for non-negative types, but the missing bit shouldn't matter in
>>> almost all
>>> cases. And if users really need to make use of it, I'm fine with
>>> requiring
>>> them to write
>>>
>>>       return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>>
>> Actually, the target should be size_t, following the C standard that
>> operations between signed and unsigned have an unsigned result. But
>> we certainly shouldn't be promoting it to a new type.
>>
>
> Yes, you are right, but then this testcase is expected to fail...
> Should we do better?
>
> __doc__ = """
>>>> f()
> -1
> """
> def f():
>     cdef Py_ssize_t a = -2
>     cdef size_t b = 1
>     return a + b

That is because the doctest is wrong. Signed + unsigned = unsigned.  
Consider

def f():
     cdef Py_ssize_t a = -2
     cdef size_t b = 1
     print a + b, a + b < 0

Now the whole "a + b < 0" statement will be evaluated in C land, and  
will always be False, which would make

-1, False

a really inconsistent thing to print out.

- Robert


From wstein at gmail.com  Sun Feb  8 00:46:53 2009
From: wstein at gmail.com (William Stein)
Date: Sat, 7 Feb 2009 15:46:53 -0800
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
	<e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>
Message-ID: <85e81ba30902071546w13fe74ectfc3066301870e0fc@mail.gmail.com>

On Sat, Feb 7, 2009 at 3:21 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> BTW,  a feature I really miss from the times I was a heavy SWIG user
> is the ability to include verbatim C code snipets...
>

Warning: blatant hack below; just for fun.

You can put verbatim C code snippets in Cython by abusing cdef extern.
 Here is an example that illustrates doing this to wrap some C++ code.
 I wanted to put the verbatim C++ code "a[b]=c" in, so I did it like
so.  You
could put any C/C++ code inside that macros though, not just a[b]=c.

cdef extern from "vector.h":
    ctypedef struct intvec "std::vector<unsigned int>":
        int get_entry "operator[]"(int n)
    intvec intvec_factory "std::vector<unsigned int>"(int len)

cdef extern from "":
    void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"()
    void SET(intvec, int, int)

DEFINE_SET()

cdef intvec v = intvec_factory(10)
SET(v,2,399283)
print v.get_entry(2)

Obviously you should never do this :-)

 -- William

From robertwb at math.washington.edu  Sun Feb  8 01:16:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 16:16:39 -0800
Subject: [Cython] Propagating exceptions with functions which returns
	void*
In-Reply-To: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
Message-ID: <48264AC7-4E37-45A4-A87F-F1BEADBB1CB1@math.washington.edu>

On Feb 3, 2009, at 2:30 AM, Daniele Pianu wrote:

> Probably I've not completely understood the C-function
> exception-propagating mechanism. I've a function with a prototype like
> this
>
> void* getItem( void* self, int i )
>
> The function returns the i-th item in self. Some index and bounds
> checking are implemented, so if, for example, I specify a negative
> index or an index out of self bounds I raise and exception. In cython
> the function is declared in this way:
>
> cdef void* getItem( void* self, int i ) except NULL
>
> But when an error occurs and the exception is raised I always receive
> a message which says that the exception is ignored. Obviously, the
> function continues its execution and terminates whit a segmentation
> fault.
>
> The question is: how can I propagate to caller functions the raised  
> exception?
> Thanks for your help (and for the great work that you've done with  
> cython :D ),

Perhaps the function you're calling this from is a cdef function  
without a way to propagate the error? It's really hard to debug  
without some more context though, perhaps you could put a self- 
contained example up at http://pastebin.com/

- Robert

From greg.ewing at canterbury.ac.nz  Sun Feb  8 02:16:13 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 08 Feb 2009 14:16:13 +1300
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
Message-ID: <498E325D.30609@canterbury.ac.nz>

Lisandro Dalcin wrote:

> + Look at the module-level tuple variable 'rank_to_type_name'. You can
> see that 'Py_ssize_t' and 'size_t' are AFTER the type 'PY_LONG_LONG'.
> 
> + Now look at the implementation of the function 'widest_numeric_type()'.

The basic problem here is that the Cython type system has
been stretched a bit too far beyond what was originally
cocceived in Pyrex. In the beginning, Pyrex only knew
about built-in C types, and although it didn't know
their actual sizes, it did know their relative sizes, so
it could calculate a widest type.

But now that we have things like Py_ssize_t that can be
defined differently on different platforms, that's no
longer possible.

I think the only completely correct thing to do would be
to introduce the notion of ints of unknow size and/or
signedness, and to refuse to implicitly convert anything
of such a type. The programmer would have to be explicit
about the conversion wanted in that case, e.g. by casting
it to a standard C type.

-- 
Greg

From dalcinl at gmail.com  Sun Feb  8 02:39:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 22:39:46 -0300
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <85e81ba30902071546w13fe74ectfc3066301870e0fc@mail.gmail.com>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
	<e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>
	<85e81ba30902071546w13fe74ectfc3066301870e0fc@mail.gmail.com>
Message-ID: <e7ba66e40902071739i2ef851as36f7f980ca138d03@mail.gmail.com>

Just hope you never need to build your Cython-generated code with a C
compiler that does not accept C++-style comments  ;-)

Seriously, such a feature would be really useful. I've never made a
serious proposition just because I cannot figure out a "cythonic"
syntax for this... Perhaps somethig like

include verbatim """
<C code>
"""



On Sat, Feb 7, 2009 at 8:46 PM, William Stein <wstein at gmail.com> wrote:
> On Sat, Feb 7, 2009 at 3:21 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> BTW,  a feature I really miss from the times I was a heavy SWIG user
>> is the ability to include verbatim C code snipets...
>>
>
> Warning: blatant hack below; just for fun.
>
> You can put verbatim C code snippets in Cython by abusing cdef extern.
>  Here is an example that illustrates doing this to wrap some C++ code.
>  I wanted to put the verbatim C++ code "a[b]=c" in, so I did it like
> so.  You
> could put any C/C++ code inside that macros though, not just a[b]=c.
>
> cdef extern from "vector.h":
>    ctypedef struct intvec "std::vector<unsigned int>":
>        int get_entry "operator[]"(int n)
>    intvec intvec_factory "std::vector<unsigned int>"(int len)
>
> cdef extern from "":
>    void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"()
>    void SET(intvec, int, int)
>
> DEFINE_SET()
>
> cdef intvec v = intvec_factory(10)
> SET(v,2,399283)
> print v.get_entry(2)
>
> Obviously you should never do this :-)
>
>  -- William
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sun Feb  8 03:56:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 18:56:08 -0800
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <498358B9.20005@behnel.de>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>
	<49830111.7030201@student.matnat.uio.no> <498358B9.20005@behnel.de>
Message-ID: <E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>

On Jan 30, 2009, at 11:44 AM, Stefan Behnel wrote:

> Hi,
>
> Dag Sverre Seljebotn wrote:
>> Magnus Lie Hetland wrote:
>>> For the for-from loop, this may be intended -- as you say, it mimics
>>> the C behavior (although it seems to fly in the face of "Python
>>> intuition"): Even with the existing Cython semantics, getstep() will
>>> be called after *every iteration*.
>>>
>>> It seems to me that in
>>>
>>>    for 0 <= i < n by getstep(): pass
>>>
>>> one would expect (in Python) that the step expression be evaluated
>>> only once. If that is not the case, fine. Then the for-from loop can
>>> safely be left alone as it is.
>>>
>> I think so -- the for-from loop is a C idiom from which one can  
>> expect C
>> behaviour from my perspective, and if this has been the behaviour  
>> so far
>> then backward compatability alone speaks against altering it. The  
>> range
>> optimization has been around for much shorter and clearly should  
>> behave
>> as Python range, so that story is different.
>
> I vote for a) diverging behaviour between for-range and for-from  
> for the
> loop-variable after loop termination, and

+1

> b) the obvious evaluate-once
> semantics for the range() optimisation and the obvious (?)
> evaluate-on-each-step semantics for the for-from loop.

+1

> And while I'm fine with removing the emphases from the for-from  
> loop in the
> docs, I really don't think we should remove it from the language.  
> However,
> it needs to be stated in the docs that it follows the semantics of  
> the C
> for loop as far as possible, and that the usual Python for-range  
> pattern is
> preferred for readability and semantic clarity.

My thoughts exactly.

- Robert



From robertwb at math.washington.edu  Sun Feb  8 04:00:52 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 19:00:52 -0800
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>
	<49830111.7030201@student.matnat.uio.no> <498358B9.20005@behnel.de>
	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
Message-ID: <0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>

On Feb 7, 2009, at 6:56 PM, Robert Bradshaw wrote:

> On Jan 30, 2009, at 11:44 AM, Stefan Behnel wrote:
>
>> Hi,
>>
>> Dag Sverre Seljebotn wrote:
>>> Magnus Lie Hetland wrote:
>>>> For the for-from loop, this may be intended -- as you say, it  
>>>> mimics
>>>> the C behavior (although it seems to fly in the face of "Python
>>>> intuition"): Even with the existing Cython semantics, getstep()  
>>>> will
>>>> be called after *every iteration*.
>>>>
>>>> It seems to me that in
>>>>
>>>>    for 0 <= i < n by getstep(): pass
>>>>
>>>> one would expect (in Python) that the step expression be evaluated
>>>> only once. If that is not the case, fine. Then the for-from loop  
>>>> can
>>>> safely be left alone as it is.
>>>>
>>> I think so -- the for-from loop is a C idiom from which one can
>>> expect C
>>> behaviour from my perspective, and if this has been the behaviour
>>> so far
>>> then backward compatability alone speaks against altering it. The
>>> range
>>> optimization has been around for much shorter and clearly should
>>> behave
>>> as Python range, so that story is different.

I just realized, however, that we are not following Python semantics  
for the empty loop:

sage: i = "foo"
sage: for i in range(10, 0): pass
sage: print i
foo

- Robert

From magnus at hetland.org  Sun Feb  8 08:36:24 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 08:36:24 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
Message-ID: <A91B8970-42DF-4B10-8605-3957A4B38124@hetland.org>

On Feb 8, 2009, at 00:09 , Robert Bradshaw wrote:

> Of course this requires one to keep two files in sync, make "cdef
> extern" declarations, and have a more complicated setup.py (i.e. no
> using pyximport), and risk gcc as well as Cython errors. I "program C
> using Cython" all the time--using the annotate option it is easy to
> see when a block of code gets compiled to pure C. But it's a matter
> of personal preference.

Right. This was how I planned to do it originally (for the reasons you  
state, and because of syntax preferences etc.).

Hm. I guess I'll see how good an overview I get with the annotation  
(and, possibly, the nogil declarations).

-- 
Magnus Lie Hetland
http://hetland.org


From stefan_ml at behnel.de  Sun Feb  8 09:26:18 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Feb 2009 09:26:18 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>
	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
Message-ID: <498E972A.9090205@behnel.de>


Robert Bradshaw wrote:
> I just realized, however, that we are not following Python semantics  
> for the empty loop:
> 
> sage: i = "foo"
> sage: for i in range(10, 0): pass
> sage: print i
> foo

http://trac.cython.org/cython_trac/ticket/208

Stefan


From magnus at hetland.org  Sun Feb  8 10:30:54 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 10:30:54 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <498E972A.9090205@behnel.de>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>
	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
	<498E972A.9090205@behnel.de>
Message-ID: <4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>

On Feb 8, 2009, at 09:26, Stefan Behnel wrote:

>
> Robert Bradshaw wrote:
>> I just realized, however, that we are not following Python semantics
>> for the empty loop:
>>
>> sage: i = "foo"
>> sage: for i in range(10, 0): pass
>> sage: print i
>> foo
>
> http://trac.cython.org/cython_trac/ticket/208

I could have a look.

-- 
Magnus Lie Hetland
http://hetland.org



From magnus at hetland.org  Sun Feb  8 11:52:05 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 11:52:05 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>
	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
	<498E972A.9090205@behnel.de>
	<4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
Message-ID: <F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>

On Feb 8, 2009, at 10:30, Magnus Lie Hetland wrote:

>> http://trac.cython.org/cython_trac/ticket/208
>
> I could have a look.


I tried what seemed to me to be the obvious solution here -- surround  
the loop with an if statement with the same condition that is used for  
ending the loop.

Early in "generate_execution_code":

> from_range = getattr(self, "from_range", False)


Then, before the loop:

> if from_range:
>     code.putln("if (%s%s %s %s) {" % (
>     self.bound1.result(), offset, self.relation2, self.bound2.result()
>     ))

An end brace is output after the "undo" operator (which is also  
conditional on from_range).

I realize we again risk double-evaluating bound1 and bound2 here, but  
I'm guessing that the expressions will have been evaluated before  
they're used here anyway (as with step).

It seems correct, and the tests work without the refnanny (except for  
the NumPy stuff, which all fails because I haven't properly installed  
NumPy at the moment; there also seemed to be a problem when running  
with the refnanny). Should I upload a patch to the tracker?

By the way, while thinking of this, it occurred to me that it might be  
cleaner to just ditch the for loop than to undo its semantics. It  
won't help on the extra check needed to see if we assign to the loop  
variable at all (i.e., what I'm doing here), but by using a while-loop  
with the proper test, we should be able to avoid the spurious  
increment and the "undo" operation. Worth a shot? (Or do C compilers  
somehow rely on the use of for loops for some kind of optimization  
inference or something? Then again, as was pointed out, the extra  
assignment is probably optimized away when not needed -- perhaps  
that's the best solution, after all.)

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Sun Feb  8 12:45:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Feb 2009 12:45:40 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>	<498E972A.9090205@behnel.de>	<4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
	<F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>
Message-ID: <498EC5E4.6010203@behnel.de>

Hi,

Magnus Lie Hetland wrote:
> Should I upload a patch to the tracker?

Sure, that will allow us to take a look at it and compare it to the
while-loop alternative. Just use "hg export" to keep the meta data.


> it occurred to me that it might be
> cleaner to just ditch the for loop than to undo its semantics. It  
> won't help on the extra check needed to see if we assign to the loop  
> variable at all (i.e., what I'm doing here), but by using a while-loop  
> with the proper test, we should be able to avoid the spurious  
> increment and the "undo" operation. Worth a shot?

That's my impression, too. I attached an old (incomplete) patch I had lying
around, don't know if it's still any good. Generating a straight
WhileStatNode instead might already do the right thing (and be less
invasive, given that we are nearing a release). See the methods of the
IterationTransform class in Optimize.py for that.


> Or do C compilers  
> somehow rely on the use of for loops for some kind of optimization  
> inference or something?

No idea. But I doubt that a while loop would be slower than the equivalent
for loop. Plus, we need to fix the semantics here, which is more important
than the last bit of performance.


> there also seemed to be a problem when running  
> with the refnanny.

Could you be a bit more specific here? :)


> the tests work without the refnanny (except for
> the NumPy stuff, which all fails because I haven't properly installed
> NumPy at the moment;

It should actually check for numpy being installed and just skip the test
if not. At least, that's what it does here.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: general_loop_node.patch
Type: text/x-patch
Size: 3149 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090208/f71279d7/attachment.bin 

From magnus at hetland.org  Sun Feb  8 13:03:59 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 13:03:59 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <498EC5E4.6010203@behnel.de>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>	<498E972A.9090205@behnel.de>	<4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
	<F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>
	<498EC5E4.6010203@behnel.de>
Message-ID: <DDE58962-4864-4C05-8249-C7455B1596E1@hetland.org>

On Feb 8, 2009, at 12:45, Stefan Behnel wrote:

> Hi,
>
> Magnus Lie Hetland wrote:
>> Should I upload a patch to the tracker?
>
> Sure, that will allow us to take a look at it and compare it to the
> while-loop alternative. Just use "hg export" to keep the meta data.

Done:

http://trac.cython.org/cython_trac/attachment/ticket/208/if_patch.txt

> That's my impression, too. I attached an old (incomplete) patch I  
> had lying
> around, don't know if it's still any good.

Cool. Don't have time to look at it right now, myself, though.

>> there also seemed to be a problem when running
>> with the refnanny.
>
> Could you be a bit more specific here? :)

My mistake. It seems it was rather some cpp stuff that gave me some  
lost refcount messages. Ran it with --no-cpp and it went away.

>> the tests work without the refnanny (except for
>> the NumPy stuff, which all fails because I haven't properly installed
>> NumPy at the moment;
>
> It should actually check for numpy being installed and just skip the  
> test
> if not. At least, that's what it does here.

Hm. It seems I actually do have numpy installed for the same Python  
that I used for this. But the error I get is:

numpy_test.c:129:31: error: numpy/arrayobject.h: No such file or  
directorynumpy_test.c:129:31:
  error: numpy/arrayobject.h: No such file or directory

Followed by lots of "warning: data definition has no type or storage  
class" and syntax errors and undeclared objects in numpy_test.c.

The errors I end up with when running with no switches:

ERROR: compiling (c) and running numpy_test
ERROR: compiling (cpp) and running numpy_test
ERROR: compiling (cpp) and running withstat
FAIL: Doctest: tryfinally
     Error: References leaked:
       Acquired on lines: 396

(The last one appears twice.)

-- 
Magnus Lie Hetland
http://hetland.org



From ondrej at certik.cz  Mon Feb  9 05:06:52 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 8 Feb 2009 20:06:52 -0800
Subject: [Cython] the LICENSE.txt is unreadable
Message-ID: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>

Hi,

the LICENSE.txt looks like this on my system (Debian) if I open it in vim:

"""
Apache License^M                           Version 2.0, January 2004^M
                       http://www.apache.org/licenses/^M^M   TERMS AND
CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
Definitions.^M^M      "License" shall mean the terms and conditions
for use, reproduction,^M      and distribution as defined by Sections
1 through 9 of this document.^M^M      "Licensor" shall mean the
copyright owner or entity authorized by^M      the copyright owner
that is granting the License.^M^M      "Legal Entity" shall mean the
union of the acting entity and all^M      other entities that cont
[...]
"""

Is it just for me or for everyone?

You can pull a fix from my repository:

http://bitbucket.org/certik/cython-devel/

Ondrej

From sjmachin at lexicon.net  Mon Feb  9 08:54:23 2009
From: sjmachin at lexicon.net (John Machin)
Date: Mon, 09 Feb 2009 18:54:23 +1100
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
Message-ID: <498FE12F.5090903@lexicon.net>

On 9/02/2009 3:06 PM, Ondrej Certik wrote:
> Hi,
> 
> the LICENSE.txt looks like this on my system (Debian) if I open it in vim:
> 
> """
> Apache License^M                           Version 2.0, January 2004^M
>                        http://www.apache.org/licenses/^M^M   TERMS AND
> CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
> Definitions.^M^M      "License" shall mean the terms and conditions
> for use, reproduction,^M      and distribution as defined by Sections
> 1 through 9 of this document.^M^M      "Licensor" shall mean the
> copyright owner or entity authorized by^M      the copyright owner
> that is granting the License.^M^M      "Legal Entity" shall mean the
> union of the acting entity and all^M      other entities that cont
> [...]
> """
> 
> Is it just for me or for everyone?

The file appears to be using the Mac convention i.e. '\r' for line 
terminators. It's a while since I've used a text editor that didn't read 
files silently in whichever of the 3 conventions it found, and write 
them out in the same fashion unless told otherwise. Perhaps you could 
check if vim has an option to do that.

> You can pull a fix from my repository:

A fix for what?

From ondrej at certik.cz  Mon Feb  9 09:19:30 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 9 Feb 2009 00:19:30 -0800
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <498FE12F.5090903@lexicon.net>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
	<498FE12F.5090903@lexicon.net>
Message-ID: <85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>

On Sun, Feb 8, 2009 at 11:54 PM, John Machin <sjmachin at lexicon.net> wrote:
> On 9/02/2009 3:06 PM, Ondrej Certik wrote:
>> Hi,
>>
>> the LICENSE.txt looks like this on my system (Debian) if I open it in vim:
>>
>> """
>> Apache License^M                           Version 2.0, January 2004^M
>>                        http://www.apache.org/licenses/^M^M   TERMS AND
>> CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
>> Definitions.^M^M      "License" shall mean the terms and conditions
>> for use, reproduction,^M      and distribution as defined by Sections
>> 1 through 9 of this document.^M^M      "Licensor" shall mean the
>> copyright owner or entity authorized by^M      the copyright owner
>> that is granting the License.^M^M      "Legal Entity" shall mean the
>> union of the acting entity and all^M      other entities that cont
>> [...]
>> """
>>
>> Is it just for me or for everyone?
>
> The file appears to be using the Mac convention i.e. '\r' for line
> terminators. It's a while since I've used a text editor that didn't read
> files silently in whichever of the 3 conventions it found, and write
> them out in the same fashion unless told otherwise. Perhaps you could
> check if vim has an option to do that.
>
>> You can pull a fix from my repository:
>
> A fix for what?

To use linux convention. It may not be a fix though, depending on the above.

Ondrej

From dalcinl at gmail.com  Mon Feb  9 14:58:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 9 Feb 2009 11:58:21 -0200
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
	<498FE12F.5090903@lexicon.net>
	<85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>
Message-ID: <e7ba66e40902090558g30bd1814h8715ff7b9a10ac4b@mail.gmail.com>

Anyway, IMHO all files should use the same convention.

If the Linux one is used, Windows users would need this:

http://www.selenic.com/mercurial/wiki/index.cgi/Win32TextExtension


On Mon, Feb 9, 2009 at 6:19 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Sun, Feb 8, 2009 at 11:54 PM, John Machin <sjmachin at lexicon.net> wrote:
>> On 9/02/2009 3:06 PM, Ondrej Certik wrote:
>>> Hi,
>>>
>>> the LICENSE.txt looks like this on my system (Debian) if I open it in vim:
>>>
>>> """
>>> Apache License^M                           Version 2.0, January 2004^M
>>>                        http://www.apache.org/licenses/^M^M   TERMS AND
>>> CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
>>> Definitions.^M^M      "License" shall mean the terms and conditions
>>> for use, reproduction,^M      and distribution as defined by Sections
>>> 1 through 9 of this document.^M^M      "Licensor" shall mean the
>>> copyright owner or entity authorized by^M      the copyright owner
>>> that is granting the License.^M^M      "Legal Entity" shall mean the
>>> union of the acting entity and all^M      other entities that cont
>>> [...]
>>> """
>>>
>>> Is it just for me or for everyone?
>>
>> The file appears to be using the Mac convention i.e. '\r' for line
>> terminators. It's a while since I've used a text editor that didn't read
>> files silently in whichever of the 3 conventions it found, and write
>> them out in the same fashion unless told otherwise. Perhaps you could
>> check if vim has an option to do that.
>>
>>> You can pull a fix from my repository:
>>
>> A fix for what?
>
> To use linux convention. It may not be a fix though, depending on the above.
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From broughtj at gmail.com  Mon Feb  9 18:45:28 2009
From: broughtj at gmail.com (Tyler Brough)
Date: Mon, 9 Feb 2009 10:45:28 -0700
Subject: [Cython] Speeding up file io
Message-ID: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>

Hello All,

I am new to Cython, though I heard about it long ago.  I have been
programming in python using scipy and pytables a lot. I am building an HDF5
database using pytables from extremely large files containing tick by tick
stock data (one single day is several GBs). As you can imagine this is very
slow going, even trying to optimize the file io in python. What I am
wondering is will trying to do the file io in cython functions speed up the
process enough to make it worth while?  Can I simply access c FILE pointers
directly in cython and pass them back as python objects?  Your advice is
very much appreciated!

Best,

--TJB

P.S. I am working in Ubuntu 8.10.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090209/6496b9de/attachment.htm 

From stefan_ml at behnel.de  Mon Feb  9 19:14:33 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 9 Feb 2009 19:14:33 +0100 (CET)
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
Message-ID: <45800.213.61.181.86.1234203273.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Tyler Brough wrote:
> I am new to Cython, though I heard about it long ago.  I have been
> programming in python using scipy and pytables a lot. I am building an
> HDF5
> database using pytables from extremely large files containing tick by tick
> stock data (one single day is several GBs). As you can imagine this is
> very
> slow going, even trying to optimize the file io in python. What I am
> wondering is will trying to do the file io in cython functions speed up
> the process enough to make it worth while?

Cython will not magically speed up your I/O operations for you. It does,
however, allow you to work at the C level if you need to, thus avoiding
I/O overhead yourself, at the cost of writing low-level-ish code. Also, if
you keep your code free from Python-level operations, you can free the GIL
and thus leverage full thread parallelism on I/O operations.


> Can I simply access c FILE pointers
> directly in cython and pass them back as python objects?

Cython doesn't convert between FILE and file() automatically. However, the
Python 2.x C-API allows you to access the low-level FILE* from a file()
instance, so you can open a file at the Python level (also from Cython),
and then work with the FILE* in your I/O operations.

I never worked with pytables, so I can't say how that fits together, and
what Cython can do for you here. What makes you think the I/O performance
itself is your problem? What kind of operations are you performing on the
database?

Stefan


From dagss at student.matnat.uio.no  Mon Feb  9 19:28:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 9 Feb 2009 19:28:02 +0100 (CET)
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
Message-ID: <251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>

>Tyler Brough wrote:
 Hello All,
>
> I am new to Cython, though I heard about it long ago.  I have been
> programming in python using scipy and pytables a lot. I am building an
> HDF5
> database using pytables from extremely large files containing tick by tick
> stock data (one single day is several GBs). As you can imagine this is
> very
> slow going, even trying to optimize the file io in python. What I am
> wondering is will trying to do the file io in cython functions speed up
> the
> process enough to make it worth while?  Can I simply access c FILE
> pointers
> directly in cython and pass them back as python objects?  Your advice is
> very much appreciated!

A quick look on the pytables source reveal that they have plenty of C
files there, and also some Pyrex and/or Cython files. The Pytable authors
have likely thought of all the obvious ways to speed things up already...

But to answer your question; from Cython you can call all the C IO
functions directly, but a C FILE pointer cannot be passed into Python. You
might be able to access the underlying handle of Python file objects if
that's what you mean, you should look in the Python headers and CPython
documentation for that.

Please ask again if this missed the mark.

Dag Sverre


From robertwb at math.washington.edu  Mon Feb  9 21:16:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 9 Feb 2009 12:16:36 -0800
Subject: [Cython] Speeding up file io
In-Reply-To: <251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
Message-ID: <46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>

On Feb 9, 2009, at 10:28 AM, Dag Sverre Seljebotn wrote:

>> Tyler Brough wrote:
>  Hello All,
>>
>> I am new to Cython, though I heard about it long ago.  I have been
>> programming in python using scipy and pytables a lot. I am  
>> building an
>> HDF5
>> database using pytables from extremely large files containing tick  
>> by tick
>> stock data (one single day is several GBs). As you can imagine  
>> this is
>> very
>> slow going, even trying to optimize the file io in python. What I am
>> wondering is will trying to do the file io in cython functions  
>> speed up
>> the
>> process enough to make it worth while?  Can I simply access c FILE
>> pointers
>> directly in cython and pass them back as python objects?  Your  
>> advice is
>> very much appreciated!
>
> A quick look on the pytables source reveal that they have plenty of C
> files there, and also some Pyrex and/or Cython files. The Pytable  
> authors
> have likely thought of all the obvious ways to speed things up  
> already...
>
> But to answer your question; from Cython you can call all the C IO
> functions directly, but a C FILE pointer cannot be passed into  
> Python. You
> might be able to access the underlying handle of Python file  
> objects if
> that's what you mean, you should look in the Python headers and  
> CPython
> documentation for that.

You can pass file objects around in Python, which are basically  
wrappers around c FILE pointers (which can easily be extracted and  
used directly from Cython code). As has already been mentioned, it is  
unclear if this will provide speed gains (especially if your  
application is io bound rather than cpu bound).

- Robert


From dalcinl at gmail.com  Mon Feb  9 21:39:41 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 9 Feb 2009 18:39:41 -0200
Subject: [Cython] PATCH: fix reference count problems in tryfinally testcase,
	please 	review & comment
Message-ID: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>

Since Dag's refnanny was integrated in cython-devel,
test/run/tryfinally.pyx was failing because of reference count errors.

The patch below is the only way I can imagine to fix such problem. I
expect the patch could be controversial...


diff -r 3b522098d133 Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py	Sun Feb 08 09:25:28 2009 +0100
+++ b/Cython/Compiler/Nodes.py	Mon Feb 09 18:22:21 2009 -0200
@@ -1054,6 +1054,8 @@
         code.put_var_declarations(lenv.var_entries)
         init = ""
         if not self.return_type.is_void:
+            if self.return_type.is_pyobject:
+                init = " = NULL"
             code.putln(
                 "%s%s;" %
                     (self.return_type.declaration_code(
@@ -3371,6 +3373,9 @@
         if self.value:
             self.value.generate_evaluation_code(code)
             self.value.make_owned_reference(code)
+            if self.return_type.is_pyobject:
+                code.put_xdecref(Naming.retval_cname,
+                                 self.return_type)
             code.putln(
                 "%s = %s;" % (
                     Naming.retval_cname,

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb  9 22:53:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 09 Feb 2009 22:53:01 +0100
Subject: [Cython] PATCH: fix reference count problems in tryfinally
 testcase, please 	review & comment
In-Reply-To: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
Message-ID: <4990A5BD.1030707@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Since Dag's refnanny was integrated in cython-devel,
> test/run/tryfinally.pyx was failing because of reference count errors.
> 
> The patch below is the only way I can imagine to fix such problem. I
> expect the patch could be controversial...

Me and Stefan should be OK with it at least: See #200 in trac. (Whether 
you initialize to Py_None or NULL is a finer point I suppose, I don't 
have a feeling either way).

To avoid the XDECREF in most situations one could make it depend on a 
flag in ReturnStatNode and write a transform in Optimize.py which turns 
on the flag on all return nodes in functions which contain return nodes 
within finally clauses. But that could be done later if this fix is 
pushed now.

Dag Sverre


> 
> 
> diff -r 3b522098d133 Cython/Compiler/Nodes.py
> --- a/Cython/Compiler/Nodes.py	Sun Feb 08 09:25:28 2009 +0100
> +++ b/Cython/Compiler/Nodes.py	Mon Feb 09 18:22:21 2009 -0200
> @@ -1054,6 +1054,8 @@
>          code.put_var_declarations(lenv.var_entries)
>          init = ""
>          if not self.return_type.is_void:
> +            if self.return_type.is_pyobject:
> +                init = " = NULL"
>              code.putln(
>                  "%s%s;" %
>                      (self.return_type.declaration_code(
> @@ -3371,6 +3373,9 @@
>          if self.value:
>              self.value.generate_evaluation_code(code)
>              self.value.make_owned_reference(code)
> +            if self.return_type.is_pyobject:
> +                code.put_xdecref(Naming.retval_cname,
> +                                 self.return_type)
>              code.putln(
>                  "%s = %s;" % (
>                      Naming.retval_cname,
> 


-- 
Dag Sverre

From glenn at tarbox.org  Tue Feb 10 00:29:08 2009
From: glenn at tarbox.org (Glenn Tarbox, PhD)
Date: Mon, 9 Feb 2009 15:29:08 -0800
Subject: [Cython] Speeding up file io
In-Reply-To: <46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
	<46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
Message-ID: <f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>

On Mon, Feb 9, 2009 at 12:16 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On Feb 9, 2009, at 10:28 AM, Dag Sverre Seljebotn wrote:
>
> >> Tyler Brough wrote:
> >  Hello All,
> >>
> >> I am new to Cython, though I heard about it long ago.  I have been
> >> programming in python using scipy and pytables a lot. I am
> >> building an
> >> HDF5
> >> database using pytables from extremely large files containing tick
> >> by tick
> >> stock data (one single day is several GBs). As you can imagine
> >> this is
> >> very
> >> slow going, even trying to optimize the file io in python. What I am
> >> wondering is will trying to do the file io in cython functions
> >> speed up
> >> the
> >> process enough to make it worth while?  Can I simply access c FILE
> >> pointers
> >> directly in cython and pass them back as python objects?  Your
> >> advice is
> >> very much appreciated!
> >
> > A quick look on the pytables source reveal that they have plenty of C
> > files there, and also some Pyrex and/or Cython files. The Pytable
> > authors
> > have likely thought of all the obvious ways to speed things up
> > already...
> >
> > But to answer your question; from Cython you can call all the C IO
> > functions directly, but a C FILE pointer cannot be passed into
> > Python. You
> > might be able to access the underlying handle of Python file
> > objects if
> > that's what you mean, you should look in the Python headers and
> > CPython
> > documentation for that.
>
> You can pass file objects around in Python, which are basically
> wrappers around c FILE pointers (which can easily be extracted and
> used directly from Cython code). As has already been mentioned, it is
> unclear if this will provide speed gains (especially if your
> application is io bound rather than cpu bound).


The theme from Love Story keeps playing in my head as I read...

First, Cython will provide substantial advantages throughout your project if
you're dealing with tick data and python.  You're gonna have places where
numpy / PyTables abstractions aren't sufficient so you'll need your own
code.  If you're focusing on python, you need a mechanism to prototype and
migrate into native code easily... Cython is the answer to that.

However, having worked (and continue to work) with tick data, its not clear
that  PyTables / HDF5 for the lowest levels of your application is the way
to go.  I also started down that path and decided that I needed to go
native...  But that really means "other stuff" that works with the usual
Python subjects.

The real issue with tick data is: "Everything is Huge".  Python and the
library abstractions provided by Numpy, HDF5, PyTables etc are generally
fine if you fit perfectly inside.  For most, when it isn't perfect, a few
lines of python and all is right with the world...

not so much for you...

You should take a quick look at the finance package in Sage.  Inside, you'll
find small set of timeseries classes with support for related operations
written by William Stein.  William and I had spent some time discussing the
issues with tick-data and he was able to produce much of the core capability
in a few days.  That alone should speak volumes about Cython (although
William is, um, lets say "more productive" than most... to put it mildly).

When William wrote those classes, he determined that the abstractions
afforded by Numpy were inappropriate for time-series due to Numpy's
generality, which manifests itself at run time (in compiled C code) with a
number of calculations / logic to make Numpy do Numpy things.  When you're
working with a huge block of tick data, this overhead is gonna hurt.

So, his approach was to facilitate conversion between regular python lists,
numpy arrays, and the tick data structure which is a 1d block of doubles.
(malloc...)

The key to Python (really C-Python) is that it was written to be glue over
C.  Cython is many things, but amongst them is making the language
transition simpler by hiding the details of integrating with the Python
interpreter (you certainly know this).

So, I think you're gonna find that you're better off "going native" at the
tick level because at the Hugeness issue.  You'll likely save development
and certainly execution time with low level code specific to your problem
(maybe mmap'ed blocks of doubles to address the file issues discussed above)

Cython makes this a tolerable experience.

-glenn


>
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
*** Note: New Number
Glenn H. Tarbox, PhD ||  206-274-6919
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090209/57d8fd90/attachment-0001.htm 

From broughtj at gmail.com  Tue Feb 10 01:06:02 2009
From: broughtj at gmail.com (Tyler Brough)
Date: Mon, 9 Feb 2009 17:06:02 -0700
Subject: [Cython] Speeding up file io
In-Reply-To: <f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
	<46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
	<f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>
Message-ID: <ace293fe0902091606k1d1e0aean56002acae6cf4a53@mail.gmail.com>

Thank you all for your input.  I have a lot to think about.

--Tyler

On Mon, Feb 9, 2009 at 4:29 PM, Glenn Tarbox, PhD <glenn at tarbox.org> wrote:

>
>
> On Mon, Feb 9, 2009 at 12:16 PM, Robert Bradshaw <
> robertwb at math.washington.edu> wrote:
>
>> On Feb 9, 2009, at 10:28 AM, Dag Sverre Seljebotn wrote:
>>
>> >> Tyler Brough wrote:
>> >  Hello All,
>> >>
>> >> I am new to Cython, though I heard about it long ago.  I have been
>> >> programming in python using scipy and pytables a lot. I am
>> >> building an
>> >> HDF5
>> >> database using pytables from extremely large files containing tick
>> >> by tick
>> >> stock data (one single day is several GBs). As you can imagine
>> >> this is
>> >> very
>> >> slow going, even trying to optimize the file io in python. What I am
>> >> wondering is will trying to do the file io in cython functions
>> >> speed up
>> >> the
>> >> process enough to make it worth while?  Can I simply access c FILE
>> >> pointers
>> >> directly in cython and pass them back as python objects?  Your
>> >> advice is
>> >> very much appreciated!
>> >
>> > A quick look on the pytables source reveal that they have plenty of C
>> > files there, and also some Pyrex and/or Cython files. The Pytable
>> > authors
>> > have likely thought of all the obvious ways to speed things up
>> > already...
>> >
>> > But to answer your question; from Cython you can call all the C IO
>> > functions directly, but a C FILE pointer cannot be passed into
>> > Python. You
>> > might be able to access the underlying handle of Python file
>> > objects if
>> > that's what you mean, you should look in the Python headers and
>> > CPython
>> > documentation for that.
>>
>> You can pass file objects around in Python, which are basically
>> wrappers around c FILE pointers (which can easily be extracted and
>> used directly from Cython code). As has already been mentioned, it is
>> unclear if this will provide speed gains (especially if your
>> application is io bound rather than cpu bound).
>
>
> The theme from Love Story keeps playing in my head as I read...
>
> First, Cython will provide substantial advantages throughout your project
> if you're dealing with tick data and python.  You're gonna have places where
> numpy / PyTables abstractions aren't sufficient so you'll need your own
> code.  If you're focusing on python, you need a mechanism to prototype and
> migrate into native code easily... Cython is the answer to that.
>
> However, having worked (and continue to work) with tick data, its not clear
> that  PyTables / HDF5 for the lowest levels of your application is the way
> to go.  I also started down that path and decided that I needed to go
> native...  But that really means "other stuff" that works with the usual
> Python subjects.
>
> The real issue with tick data is: "Everything is Huge".  Python and the
> library abstractions provided by Numpy, HDF5, PyTables etc are generally
> fine if you fit perfectly inside.  For most, when it isn't perfect, a few
> lines of python and all is right with the world...
>
> not so much for you...
>
> You should take a quick look at the finance package in Sage.  Inside,
> you'll find small set of timeseries classes with support for related
> operations written by William Stein.  William and I had spent some time
> discussing the issues with tick-data and he was able to produce much of the
> core capability in a few days.  That alone should speak volumes about Cython
> (although William is, um, lets say "more productive" than most... to put it
> mildly).
>
> When William wrote those classes, he determined that the abstractions
> afforded by Numpy were inappropriate for time-series due to Numpy's
> generality, which manifests itself at run time (in compiled C code) with a
> number of calculations / logic to make Numpy do Numpy things.  When you're
> working with a huge block of tick data, this overhead is gonna hurt.
>
> So, his approach was to facilitate conversion between regular python lists,
> numpy arrays, and the tick data structure which is a 1d block of doubles.
> (malloc...)
>
> The key to Python (really C-Python) is that it was written to be glue over
> C.  Cython is many things, but amongst them is making the language
> transition simpler by hiding the details of integrating with the Python
> interpreter (you certainly know this).
>
> So, I think you're gonna find that you're better off "going native" at the
> tick level because at the Hugeness issue.  You'll likely save development
> and certainly execution time with low level code specific to your problem
> (maybe mmap'ed blocks of doubles to address the file issues discussed above)
>
> Cython makes this a tolerable experience.
>
> -glenn
>
>
>>
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> *** Note: New Number
> Glenn H. Tarbox, PhD ||  206-274-6919
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090209/eb3d6c7a/attachment.htm 

From wstein at gmail.com  Tue Feb 10 03:00:53 2009
From: wstein at gmail.com (William Stein)
Date: Mon, 9 Feb 2009 18:00:53 -0800
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902091606k1d1e0aean56002acae6cf4a53@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
	<46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
	<f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>
	<ace293fe0902091606k1d1e0aean56002acae6cf4a53@mail.gmail.com>
Message-ID: <85e81ba30902091800y5702b7cbx9e46aa43d171e683@mail.gmail.com>

On Mon, Feb 9, 2009 at 4:06 PM, Tyler Brough <broughtj at gmail.com> wrote:
> Thank you all for your input.  I have a lot to think about.
>
> --Tyler

>> You should take a quick look at the finance package in Sage.  Inside,
>> you'll find small set of timeseries classes with support for related
>> operations written by William Stein.  William and I had spent some time
>> discussing the issues with tick-data and he was able to produce much of the
>> core capability in a few days.  That alone should speak volumes about Cython
>> (although William is, um, lets say "more productive" than most... to put it
>> mildly).

The code he's talking about is here:

http://www.sagemath.org/hg/sage-main/file/b0aa7ef45b3c/sage/finance/

You can try any of that stuff out live online without having to
install anything by
going to http://sagenb.org and making a free account.

 -- William

From robertwb at math.washington.edu  Tue Feb 10 03:44:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 9 Feb 2009 18:44:28 -0800
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <e7ba66e40902090558g30bd1814h8715ff7b9a10ac4b@mail.gmail.com>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
	<498FE12F.5090903@lexicon.net>
	<85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>
	<e7ba66e40902090558g30bd1814h8715ff7b9a10ac4b@mail.gmail.com>
Message-ID: <ECBB045C-8272-4BFC-AA00-AE364D73A105@math.washington.edu>

On Feb 9, 2009, at 5:58 AM, Lisandro Dalcin wrote:

> Anyway, IMHO all files should use the same convention.

I agree. I just fixed the remaining inconsistencies. OS X uses \n for  
line breaks just fine.

> If the Linux one is used, Windows users would need this:
>
> http://www.selenic.com/mercurial/wiki/index.cgi/Win32TextExtension

Almost all the files already using a simple \n for line breaks, so I  
guess anyone who needs this is already using it. Most editors should  
handle all line endings fine nowadays.

- Robert


From robertwb at math.washington.edu  Tue Feb 10 08:03:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 9 Feb 2009 23:03:26 -0800
Subject: [Cython] PATCH: fix reference count problems in tryfinally
	testcase, please 	review & comment
In-Reply-To: <4990A5BD.1030707@student.matnat.uio.no>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
	<4990A5BD.1030707@student.matnat.uio.no>
Message-ID: <F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>

On Feb 9, 2009, at 1:53 PM, Dag Sverre Seljebotn wrote:

> Lisandro Dalcin wrote:
>> Since Dag's refnanny was integrated in cython-devel,
>> test/run/tryfinally.pyx was failing because of reference count  
>> errors.
>>
>> The patch below is the only way I can imagine to fix such problem. I
>> expect the patch could be controversial...
>
> Me and Stefan should be OK with it at least: See #200 in trac.  
> (Whether
> you initialize to Py_None or NULL is a finer point I suppose, I don't
> have a feeling either way).
>
> To avoid the XDECREF in most situations one could make it depend on a
> flag in ReturnStatNode and write a transform in Optimize.py which  
> turns
> on the flag on all return nodes in functions which contain return  
> nodes
> within finally clauses. But that could be done later if this fix is
> pushed now.

I'm not sure this is the right fix, but it does appear to solve the  
(serious) issue so we should include it and can optimize later.

- Robert



From magnus at hetland.org  Tue Feb 10 12:00:57 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Tue, 10 Feb 2009 12:00:57 +0100
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
Message-ID: <C6D36111-6056-47B3-A738-813AB369BF97@hetland.org>

On Feb 9, 2009, at 18:45, Tyler Brough wrote:

> Hello All,
>
> I am new to Cython, though I heard about it long ago.  I have been  
> programming in python using scipy and pytables a lot. I am building  
> an HDF5 database using pytables from extremely large files  
> containing tick by tick stock data (one single day is several GBs).  
> As you can imagine this is very slow going, even trying to optimize  
> the file io in python. What I am wondering is will trying to do the  
> file io in cython functions speed up the process enough to make it  
> worth while?  Can I simply access c FILE pointers directly in cython  
> and pass them back as python objects?  Your advice is very much  
> appreciated!

I'm doing this sort of thing. I use file descriptors in my  
interactions between C and Python, as they are simply ints, and both C  
and Python have functions for using them (e.g., I use os.close() in  
Python to close a file based on a file descriptor, etc.).

(But, as someone else mentioned, there are already useful wrappers for  
HDF5 in Python that might be worth checking out.)

-- 
Magnus Lie Hetland
http://hetland.org



From dalcinl at gmail.com  Tue Feb 10 14:27:47 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 11:27:47 -0200
Subject: [Cython] PATCH: fix reference count problems in tryfinally
	testcase, please review & comment
In-Reply-To: <F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
	<4990A5BD.1030707@student.matnat.uio.no>
	<F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
Message-ID: <e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>

On Tue, Feb 10, 2009 at 5:03 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> I'm not sure this is the right fix, but it does appear to solve the
> (serious) issue so we should include it and can optimize later.
>

Indeed, the patch did not handle well all cases...

My fix do not work well when cdef functions returning PyObject*, not
sure at this point what to do.

I've pushed two additions to tryfinally testcase, both fail :-(

+
+
+def try_return_none_1():
+    try:
+        return
+    finally:
+        return
+
+cdef extern from *:
+    ctypedef struct PyObject
+    void Py_INCREF(object)
+
+cdef PyObject* _none():
+    ret = None
+    Py_INCREF(ret)
+    return <PyObject*> ret
+
+def try_return_none_2():
+    try:
+        return <object> _none()
+    finally:
+        return <object> _none()



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb 10 14:36:08 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 11:36:08 -0200
Subject: [Cython] failure of withstat
Message-ID: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>

See this:

$ python runtests.py --no-cleanup --no-c withstat
...
OK

$ python runtests.py --no-cleanup --no-cpp withstat
...
OK

$ python runtests.py --no-cleanup --no-cpp withstat
...
withstat.cpp: In function 'PyObject*
__pyx_pf_8withstat_no_as(PyObject*, PyObject*)':
withstat.cpp:801: error: '__pyx_v___tmpvar_2' was not declared in this scope
withstat.cpp:808: error: '__pyx_v___tmpvar_4' was not declared in this scope
withstat.cpp:825: error: '__pyx_v___tmpvar_3' was not declared in this scope
...
CompileError: command 'gcc' failed with exit status 1
...

$ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
< ... lots of output, a bit below >
867,868c838,839
<         __Pyx_DECREF(__pyx_v___tmpvar_1);
<         __pyx_v___tmpvar_1 = Py_None;
---
>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>         __pyx_v___tmpvar_40 = Py_None;


A problem with runtests.py? Perhaps a bad interaction on temp
allocation code and global module/class level variables in the Cython
codebase?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Feb 10 14:49:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Feb 2009 14:49:46 +0100 (CET)
Subject: [Cython] failure of withstat
In-Reply-To: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
Message-ID: <32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> $ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
> < ... lots of output, a bit below >
> 867,868c838,839
> <         __Pyx_DECREF(__pyx_v___tmpvar_1);
> <         __pyx_v___tmpvar_1 = Py_None;
> ---
>>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>>         __pyx_v___tmpvar_40 = Py_None;

Dag already hijacked a bug report for this.

http://trac.cython.org/cython_trac/ticket/151

Stefan


From dagss at student.matnat.uio.no  Tue Feb 10 14:54:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 10 Feb 2009 14:54:28 +0100
Subject: [Cython] failure of withstat
In-Reply-To: <32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49918714.7060809@student.matnat.uio.no>

Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>   
>> $ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
>> < ... lots of output, a bit below >
>> 867,868c838,839
>> <         __Pyx_DECREF(__pyx_v___tmpvar_1);
>> <         __pyx_v___tmpvar_1 = Py_None;
>> ---
>>     
>>>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>>>         __pyx_v___tmpvar_40 = Py_None;
>>>       
>
> Dag already hijacked a bug report for this.
>
> http://trac.cython.org/cython_trac/ticket/151
>   
Heh. Well, it was my fix for #151 which broke things, so that's why...
 
I'm mystified as to what is going on though, and won't have more time to 
look at it for some days. Suggestions welcome.

Dag Sverre

From dalcinl at gmail.com  Tue Feb 10 15:10:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 12:10:39 -0200
Subject: [Cython] failure of withstat
In-Reply-To: <49918714.7060809@student.matnat.uio.no>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49918714.7060809@student.matnat.uio.no>
Message-ID: <e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>

Just to be sure...

Did you realize that a C++ only run (and also a C only run passing
--no-cpp) of the testsuite

$ python runtests.py --no-c withstat

do indeed succeed?


On Tue, Feb 10, 2009 at 11:54 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>
>>> $ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
>>> < ... lots of output, a bit below >
>>> 867,868c838,839
>>> <         __Pyx_DECREF(__pyx_v___tmpvar_1);
>>> <         __pyx_v___tmpvar_1 = Py_None;
>>> ---
>>>
>>>>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>>>>         __pyx_v___tmpvar_40 = Py_None;
>>>>
>>
>> Dag already hijacked a bug report for this.
>>
>> http://trac.cython.org/cython_trac/ticket/151
>>
> Heh. Well, it was my fix for #151 which broke things, so that's why...
>
> I'm mystified as to what is going on though, and won't have more time to
> look at it for some days. Suggestions welcome.
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb 10 15:22:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 10 Feb 2009 15:22:12 +0100
Subject: [Cython] failure of withstat
In-Reply-To: <e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49918714.7060809@student.matnat.uio.no>
	<e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>
Message-ID: <49918D94.20301@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Just to be sure...
>
> Did you realize that a C++ only run (and also a C only run passing
> --no-cpp) of the testsuite
>
> $ python runtests.py --no-c withstat
>
> do indeed succeed?
>   
Ah. Yes, I noticed, but after commenting on the ticket. So apparently it 
is NOT a C++-only issue, but something much more strange going on ...

Dag Sverre

From dalcinl at gmail.com  Tue Feb 10 15:41:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 12:41:21 -0200
Subject: [Cython] failure of withstat
In-Reply-To: <49918D94.20301@student.matnat.uio.no>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49918714.7060809@student.matnat.uio.no>
	<e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>
	<49918D94.20301@student.matnat.uio.no>
Message-ID: <e7ba66e40902100641j2220c856u2ddcbee9b82a8324@mail.gmail.com>

On Tue, Feb 10, 2009 at 12:22 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>>
> Ah. Yes, I noticed, but after commenting on the ticket. So apparently it
> is NOT a C++-only issue, but something much more strange going on ...
>

Indeed. It seems Cython 'memoizes' counters for a module context, then
if you compile twice, the counters are not reinitialized. Or somethig
like that....

You know, module or class level globals could be rally evil :-) ...

$ cd Cython/Compiler/
$ find . -type f -print0 | xargs -0 -e grep -nH -e
TemplateTransform.temp_name_counter
./ParseTreeTransforms.py:549:        TemplateTransform.temp_name_counter += 1
./ParseTreeTransforms.py:550:        handle = "__tmpvar_%d" %
TemplateTransform.temp_name_counter
./TreeFragment.py:122:            TemplateTransform.temp_name_counter += 1
./TreeFragment.py:123:            handle = "__tmpvar_%d" %
TemplateTransform.temp_name_counter

Is the .temp_name_counter counter ever decremented? Could be this the
root of the problem?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb 10 16:07:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 10 Feb 2009 16:07:41 +0100
Subject: [Cython] failure of withstat
In-Reply-To: <e7ba66e40902100641j2220c856u2ddcbee9b82a8324@mail.gmail.com>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49918714.7060809@student.matnat.uio.no>	<e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>	<49918D94.20301@student.matnat.uio.no>
	<e7ba66e40902100641j2220c856u2ddcbee9b82a8324@mail.gmail.com>
Message-ID: <4991983D.90802@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Feb 10, 2009 at 12:22 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>   
>> Ah. Yes, I noticed, but after commenting on the ticket. So apparently it
>> is NOT a C++-only issue, but something much more strange going on ...
>>
>>     
>
> Indeed. It seems Cython 'memoizes' counters for a module context, then
> if you compile twice, the counters are not reinitialized. Or somethig
> like that....
>
> You know, module or class level globals could be rally evil :-) ...
>
> $ cd Cython/Compiler/
> $ find . -type f -print0 | xargs -0 -e grep -nH -e
> TemplateTransform.temp_name_counter
> ./ParseTreeTransforms.py:549:        TemplateTransform.temp_name_counter += 1
> ./ParseTreeTransforms.py:550:        handle = "__tmpvar_%d" %
> TemplateTransform.temp_name_counter
> ./TreeFragment.py:122:            TemplateTransform.temp_name_counter += 1
> ./TreeFragment.py:123:            handle = "__tmpvar_%d" %
> TemplateTransform.temp_name_counter
>
> Is the .temp_name_counter counter ever decremented? Could be this the
> root of the problem?
>   
I tried binding temp_name_counter to Main.Context (which is available as 
self.context in WithTransform), and initializing it to 0. It made no 
difference (and I haven't got around to commiting that).

Dag Sverre


From stefan_ml at behnel.de  Tue Feb 10 18:34:20 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Feb 2009 18:34:20 +0100
Subject: [Cython] PATCH: fix reference count problems in
 tryfinally	testcase, please review & comment
In-Reply-To: <e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>	<4990A5BD.1030707@student.matnat.uio.no>	<F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
	<e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>
Message-ID: <4991BA9C.8000303@behnel.de>

Hi,

Lisandro Dalcin wrote:
> I've pushed two additions to tryfinally testcase, both fail :-(

Please don't add failing tests to the normal test suite (unless you have
the fix ready for immediate commit). Instead, make sure there is a bug
report for it (#200 in this case) and add a test to tests/bugs/, named:
"the_test_name_Tnnn.pyx", "Tnnn" being the ticket number, i.e. "T200" in
your case. The runtests.py script then allows it to be run by passing the
"-T200" at the command line.

Stefan


From dalcinl at gmail.com  Tue Feb 10 18:49:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 15:49:37 -0200
Subject: [Cython] PATCH: fix reference count problems in tryfinally
	testcase, please review & comment
In-Reply-To: <4991BA9C.8000303@behnel.de>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
	<4990A5BD.1030707@student.matnat.uio.no>
	<F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
	<e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>
	<4991BA9C.8000303@behnel.de>
Message-ID: <e7ba66e40902100949n6c58889en2a4402e80a8587cb@mail.gmail.com>

Stefan, the two additions I've made are trivial variations of the
stuff already failing in test/run/tryfinally.pyx . Once the former
failing test pass, my additions should pass for free...

What should I do then? Move all the failing tests (even the former
one) to bugs/test_name_Tnnn.pyx ??



On Tue, Feb 10, 2009 at 3:34 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Lisandro Dalcin wrote:
>> I've pushed two additions to tryfinally testcase, both fail :-(
>
> Please don't add failing tests to the normal test suite (unless you have
> the fix ready for immediate commit). Instead, make sure there is a bug
> report for it (#200 in this case) and add a test to tests/bugs/, named:
> "the_test_name_Tnnn.pyx", "Tnnn" being the ticket number, i.e. "T200" in
> your case. The runtests.py script then allows it to be run by passing the
> "-T200" at the command line.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From glenn at tarbox.org  Tue Feb 10 19:35:32 2009
From: glenn at tarbox.org (Glenn Tarbox, PhD)
Date: Tue, 10 Feb 2009 10:35:32 -0800
Subject: [Cython] IbPy Cross Post...
Message-ID: <f049b35c0902101035k31c8ed37tc72ec08d0dcf707c@mail.gmail.com>

IbPy is a Python interface to the Interactive Brokers API and something I've
been involved with for a while.

A piece of IbPy is a code generator which takes IB's Java client code and
generates Python.  I've had issues with performance... Hence the message
below..

http://groups.google.com/group/ibpy-discuss/msg/1bf318a75376c64f

and pasted here:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

So, I had this weird thought last night about IbPy... and the java to
python generator / compiler...

and something occurred to me.

Cython....

I've been playing around with Cython for a while.  Its pretty amazing
stuff really and the syntax is very much python-like with a few
exceptions... those exceptions are where the Cython compiler does
things to make python code fast (i.e. where you know things about the
data, you can take the load off the Python interpreter and generate
native code)

The speedup can be dramatic... like 200x.  There are a number of
important Cython characteristics but its better to just read about it
(links below)

An important link first:  This to a talk William Stein gave to/at
Microsoft which has a little Cython example which gets executed in
real-time (i.e. C/C++ code gets generated and linked into python
dynamically through the sage workbook which is browser based).  Of
course, just like generating .pyc files, once this happens in real
life, the .dll or .so files are there so just need to be dynamically
linked at runtime unless there is a source code change.

If you're not a math person, just scroll down to the Cython example

http://sagenb.org/home/pub/202/

Note that if you go ahead and get an account on the Sage machines,
you'll be able to run this all yourself... live. (for the more
adventurous, building Sage is amazingly easy and it is completely self
contained... I could expand greatly but those so inclined should check
out Sage (links below)

But, thats not my point.

My point is that since IbPy uses the J2py compiler (called something
like that) its probably "relatively" easy to generate a little extra
python code in key areas to make it Cythonic (couplea type
specifications)... which will make it operate at native code speeds
yet still be completely python at the user level... just that instead
of the library being interpreted, its native...

no diff to the user at all...

and I need to be clear clear cuz it might not be obvious at first
glance.  I'm NOT saying the user interface is Cython, just that the
underlying IB handling library is Cython.  The user still codes in
Python (unless you want to use Cython which is another story
altogether and might make a lot of sense for those of us who run into
compute problems... this is sort of like using cPickle instead of
Pickle.  Same interface, just runs way faster cuz its compiled C code
instead of Python)

This effort might be amost a freebie given the existence of the code
generator used for IbPy.  The java code parsed to generate IbPy has
the information necessary to spew the few key type definitions where
IbPy spends all its cycles (Java is strongly typed), in particular,
the inner loops which do the comms handling and object generation.

Cython homepage: http://www.cython.org/

Sage homepage: http://www.sagemath.org/

-glenn

On Jan 30, 2:22 am, glenn
<gl...<http://groups.google.com/groups/unlock?_done=/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100%3Fshow_docid%3D7b027b93b5472100&msg=1bf318a75376c64f>
@tarbox.org> wrote:

- Hide quoted text -
- Show quoted text -
<http://groups.google.com/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100?hide_quotes=no#msg_1bf318a75376c64f>
> On Jan 29, 4:55 pm, Troy Melhase <troy.melh...<http://groups.google.com/groups/unlock?_done=/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100%3Fshow_docid%3D7b027b93b5472100&msg=1bf318a75376c64f>
@gmail.com> wrote:

> > I once did a bit of looking.  There are python interfaces of various

> > > maturities although I think I remember more about interfaces in Java
> > > and possibly CORBA (not something with which one dabbles in lightly,
> > > particularly with Python)
> > ...
> > > Therefore, you build python remote access to an IbPy process which can

> > > be accessed both from your gnumeric python and your PyQt process over
> > > the wire (pickle is your friend here).

> > What I'm hearing from this thread is that the choice for RPC between
> > an arbitrary process (gnumeric, OO) and an IbPy process (your script,
> > gui, app) is either non-existent or is substandard.  And you really
> > shouldn't have to make a choice, there should be a solid, thought-thru
> > solution for you.

> Wouldn't it be wonderful if everyone thought like you. :-)

> > Every time I've had to construct my own RPC mechanism, I've used a
> > small web server.  The benefits typically outweigh the downsides
> > (stateless nature) of the protocol.  So I think I'll code up an "ibpy
> > server" and an "ibpy command-line client" for the 0.7.7 release.

> I'll write some below, but if you were going to do this, I think it
> would be useful to take a step back and think broadly about the
> architecture.

> If you're gonna expose the interface "over the wire", the server side
> language can be independent of the client if you're willing to avoid
> python specific mechanisms (pickle)

> I had to move away from IbPy on the server side for performance
> reasons... when you pull depth and a bunch of instruments, throw some
> database stuff and maybe an analysis op or two and you quickly hit
> performance limits.

> In my case, I went all the way: I implemented an interface to a Java
> server using Google Protocol Buffers.  The reasons are many and its
> arguable whether its the right choice for an IB only system, but Java,
> C++ and Python are supported natively.

> Unfortunately, another problem quickly became apparent:  Protocol
> buffers are slow in python due to their implementation (exploit a lot
> of reflection... makes it elegant and all, but brings the python up
> again).  So, its great for lots of python apps, but not data at any
> real volume.

> However, my system is quite a bit more performance intensive than most
> need... particularly since I got a live tick feed which requires Java
> or C++ anyways.

> > The server would expose URLs like this:

> >     /ib/cancelOrder/<orderid>
> >     /ib/reqMktData/<tickid>
> >     /ib/reqHistoricalData/<tickid>

> > Those above would be available via HTTP POST, and other required
> > params would have to be sent as part of the request (e.g., in
> > reqMktData we can't have
> > /<tickid>/<contractfield1>/<contractfield2>/<etc>, so we collapse
> > those and only require the key (ticker id) as part of the URL).

> > These calls would be available via HTTP GET:

> >     /ib/order/<orderid>
> >     /ib/marketdata/<tickerid>
> >     /ib/histdata/<tickid>

> > In the case of these, they are data-returning calls, meaning that
> > clients can make requests to fetch the values whenever they like.
> > These calls would return data encoded as CSV, RSS, or JSON.

> So, the proposal is some kind of REST web services interface.  I think
> this is a great idea...

> But, then we're into why is the server in Python?

> Believe me, I do every bit of coding in Python I can (which includes
> Numpy, Matplotlib, Sage) but Python has performance issues.  So, if
> you're gonna build a server using "web stuff", I'm guessing that a
> rethink might be in order.

> It would likely be possible to generate a Python client side interface
> which works similarly to your current "Pythonic" API but with
> enhancements to support synchronous clients (i.e. no callbacks).  Of
> course, the design issues here can get tricky as there are results and
> the client may choose to poll which means the server side needs to
> queue responses. Not nasty technically, but the use case explosion can
> sometimes bite you.

> > As for the command line client, it would operate in one of two modes:
> > as an IbPy HTTP client or as an TWS library client.  In both cases,
> > the interface would be the same, and it would certainly allow for
> > integration with any program that can start other processes.
> > Examples:

> >     $ runibpy --clientid=1 --host=myserver --command=reqMktData
> > --tickerid=832 --rows=50
> >     $ runibpy --http --host=myserver:8080 --command=reqMktData ...

> > The only question here is how to deal with compound data structures
> > like the contract and order objects.

> you betcha... this is where I spent a great deal of time hacking.  It
> was more typing really as I took the IB java code, commented it all
> out, and typed the Protocol Buffer interface definitions kinda in-
> line.  It took a few days but there are constructs in protobuf which
> help with the more complex structures.

> Of course, one could also define an XSD, the schema isn't hugely
> nasty, but now people who use IbPy because its simple and Python are
> wrestling with a different kind of headache.... wrestling with XML can
> be kinda nasty although there are certainly good tools / libraries
> etc.

> > > In previous posts, there are references to the Twisted IbPy
> > > implementation I wrote.  Also, I built / maintain the twisted Qt
> > > reactor.  If you're going to do asynchronous processing, I highly
> > > recommend twisted although its a non-trivial elephant to swallow...
> > > IMHO, its worth it as is Qt (thanks Troy)

> > Does twisted have some kind of WSGI interface or support?  I'd be
> > willing to try that over the basic HTTP servers in the standard
> > library.

> Twisted has a full HTTP server implementation and would also be
> helpful in that it can easily handle multiple clients and IB without
> requiring threads or all the nastiness which comes with it.  Its also
> pretty high performance as its architected well and all... but its a
> non-trivial beast to swallow... as is asynchronous programming in
> general.  I highly recommend it, but I'm not sure you should take my
> recommendation :-)

> > Thoughts?

> As an aside, one of the advantages of the approach I took with a java
> based server, is that I can connect to multiple IB clients all pumping
> a bunch of data without issue.  This is, again, not the kind of
> problem one often finds "in the wild", but it helped me out quite a
> bit.

> Of course, I eventually gave up, wrote a check, and am getting every
> tick for every exchange I trade from another vendor (NxCore).  Next
> step is to go FIX...

> -glenn
> > troy

-- 
*** Note: New Number
Glenn H. Tarbox, PhD ||  206-274-6919
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090210/a9ce77a8/attachment-0001.htm 

From jasone at canonware.com  Wed Feb 11 20:32:31 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 11:32:31 -0800
Subject: [Cython] size_t problem
Message-ID: <499327CF.9040705@canonware.com>

I have a class declaration that looks like:

cdef class DistMatrix:
     # ...
     cdef readonly size_t ntaxa

With the recent size_t-related changes, I now get the following 
compilation error:

	C attribute of type 'size_t' cannot be accessed from Python

I hope this isn't an intentional change... =)

Jason

From jasone at canonware.com  Wed Feb 11 20:41:16 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 11:41:16 -0800
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <499329DC.6040205@canonware.com>

Jason Evans wrote:
> I have a class declaration that looks like:
> 
> cdef class DistMatrix:
>      # ...
>      cdef readonly size_t ntaxa
> 
> With the recent size_t-related changes, I now get the following 
> compilation error:
> 
> 	C attribute of type 'size_t' cannot be accessed from Python

Here's another problem:

     cpdef float distanceGet(self, size_t x, size_t y):
                                            ^
------------------------------------------------------------

/home/jasone/crux/crux/build/run/pkg/Crux/DistMatrix/__init__.pyx:359:44: 
Cannot convert Python object argument to type 'size_t' (when parsing 
input arguments)

Basically, size_t appears to have become an unusable type that is 
relegated to some sort of type purgatory.

Jason

From dagss at student.matnat.uio.no  Wed Feb 11 20:58:47 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 11 Feb 2009 20:58:47 +0100 (CET)
Subject: [Cython] size_t problem
In-Reply-To: <499329DC.6040205@canonware.com>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
Message-ID: <ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>

Jason Evans wrote:
> Jason Evans wrote:
>> I have a class declaration that looks like:
>>
>> cdef class DistMatrix:
>>      # ...
>>      cdef readonly size_t ntaxa
>>
>> With the recent size_t-related changes, I now get the following
>> compilation error:
>>
>> 	C attribute of type 'size_t' cannot be accessed from Python
>
> Here's another problem:
>
>      cpdef float distanceGet(self, size_t x, size_t y):
>                                             ^
> ------------------------------------------------------------
>
> /home/jasone/crux/crux/build/run/pkg/Crux/DistMatrix/__init__.pyx:359:44:
> Cannot convert Python object argument to type 'size_t' (when parsing
> input arguments)
>
> Basically, size_t appears to have become an unusable type that is
> relegated to some sort of type purgatory.

At this stage my vote is for rolling back the size_t changes and file a
ticket for 0.11.1 instead.

Dag Sverre


From dalcinl at gmail.com  Wed Feb 11 21:35:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 18:35:39 -0200
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>

It was not intentional, but indeed this is spotting a portability
problem on your code in previous Cython versions :-).

I'll try to fix this...

On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
> I have a class declaration that looks like:
>
> cdef class DistMatrix:
>     # ...
>     cdef readonly size_t ntaxa
>
> With the recent size_t-related changes, I now get the following
> compilation error:
>
>        C attribute of type 'size_t' cannot be accessed from Python
>
> I hope this isn't an intentional change... =)
>
> Jason
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Feb 11 21:37:56 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 12:37:56 -0800
Subject: [Cython] size_t problem
In-Reply-To: <ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
	<ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
Message-ID: <919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>

On Feb 11, 2009, at 11:58 AM, Dag Sverre Seljebotn wrote:

> Jason Evans wrote:
>> Jason Evans wrote:
>>> I have a class declaration that looks like:
>>>
>>> cdef class DistMatrix:
>>>      # ...
>>>      cdef readonly size_t ntaxa
>>>
>>> With the recent size_t-related changes, I now get the following
>>> compilation error:
>>>
>>> 	C attribute of type 'size_t' cannot be accessed from Python
>>
>> Here's another problem:
>>
>>      cpdef float distanceGet(self, size_t x, size_t y):
>>                                             ^
>> ------------------------------------------------------------
>>
>> /home/jasone/crux/crux/build/run/pkg/Crux/DistMatrix/__init__.pyx: 
>> 359:44:
>> Cannot convert Python object argument to type 'size_t' (when parsing
>> input arguments)
>>
>> Basically, size_t appears to have become an unusable type that is
>> relegated to some sort of type purgatory.
>
> At this stage my vote is for rolling back the size_t changes and  
> file a
> ticket for 0.11.1 instead.

I think the fix should be simple (I'll look at it later today, I'm  
off to catch the bus) but if not, lets roll it back.

- Robert



From dalcinl at gmail.com  Wed Feb 11 21:47:16 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 18:47:16 -0200
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>

Please also note that the code below also fails to cythonize. I bet
this gotcha is not related to my latest patches for 'size_t'

cdef class A:
   cdef readonly Py_ssize_t n


On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
> I have a class declaration that looks like:
>
> cdef class DistMatrix:
>     # ...
>     cdef readonly size_t ntaxa
>
> With the recent size_t-related changes, I now get the following
> compilation error:
>
>        C attribute of type 'size_t' cannot be accessed from Python
>
> I hope this isn't an intentional change... =)
>
> Jason
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Feb 11 22:20:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 13:20:24 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
Message-ID: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>

On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:

> Please also note that the code below also fails to cythonize. I bet
> this gotcha is not related to my latest patches for 'size_t'
>
>
> cdef class A:
>    cdef readonly Py_ssize_t n

Well, yes and no. People used to be able to write "ctypedef unsigned  
long size_t" and have it work.

I pushed a fix and some tests. The public attribute stuff is slightly  
hacky, but I'm going to be revising that to handle complex types  
anyways.

BTW, I didn't see ulong <= size_t <= ulonglong ordering in PyrexTypes  
yet, did we get consensus on this?

- Robert


From michael.abshoff at googlemail.com  Wed Feb 11 22:32:13 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 11 Feb 2009 13:32:13 -0800
Subject: [Cython] size_t problem
In-Reply-To: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
Message-ID: <499343DD.4020609@gmail.com>

Robert Bradshaw wrote:
> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
> 
>> Please also note that the code below also fails to cythonize. I bet
>> this gotcha is not related to my latest patches for 'size_t'
>>
>>
>> cdef class A:
>>    cdef readonly Py_ssize_t n
> 
> Well, yes and no. People used to be able to write "ctypedef unsigned  
> long size_t" and have it work.

Well, not to be a pedantic nit picker, but as we all should know now the 
above is not correct. It tends to work on most systems, but enabling 
people to do dumb things should not have priority over correct code. 
Sage has the same issue that someone defined size_t to be unsigned long 
and I could look up who did that, but I have a good idea who wrote that 
code, so no need to point fingers :)

Maybe the 0.11 release notes should prominently feature this change and 
show how to fix the problem? This might also be something where Pyrex 
and Cython have diverged, i.e. I assume the above is "legal" in Pyrex 
while it won't work in Cython 0.11 or later, so in that case it should 
be added to the wiki page in the Cython wiki where differences are listed.

> I pushed a fix and some tests. The public attribute stuff is slightly  
> hacky, but I'm going to be revising that to handle complex types  
> anyways.
> 
> BTW, I didn't see ulong <= size_t <= ulonglong ordering in PyrexTypes  
> yet, did we get consensus on this?
> 
> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dalcinl at gmail.com  Wed Feb 11 22:44:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 19:44:22 -0200
Subject: [Cython] size_t problem
In-Reply-To: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
Message-ID: <e7ba66e40902111344g3e74054fubeb4166ed7c00562@mail.gmail.com>

On Wed, Feb 11, 2009 at 7:20 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
>
> BTW, I didn't see ulong <= size_t <= ulonglong ordering in PyrexTypes
> yet, did we get consensus on this?
>

I have the patch ready to go mainstream, plus some hackery to make
Cython fail compilation for nonsense 'unsigned Py_ssize_t' or 'signed
Py_ssize_t' (the same for 'size_t').

Can I push this?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 11 22:50:58 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 19:50:58 -0200
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <e7ba66e40902111350j704fb629k3c7d8ae45641d0e6@mail.gmail.com>

On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
> I have a class declaration that looks like:
>
> cdef class DistMatrix:
>     # ...
>     cdef readonly size_t ntaxa
>

I'm now working on a patch to transform that in a property. However,
this should be done with care, regarding the difference between data
and non-data descriptors,
http://users.rcn.com/python/download/Descriptor.htm#id3 . I believe
the __set__ must be generated and it should raise AtributeError,
'can't set attribute'  (like builtin 'property' does when no 'set'
function is passed)

Comments?





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 11 22:59:16 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 19:59:16 -0200
Subject: [Cython] size_t problem
In-Reply-To: <919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
	<ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
	<919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>
Message-ID: <e7ba66e40902111359o158326c5sde4aa524eb015ef1@mail.gmail.com>

On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> I think the fix should be simple (I'll look at it later today, I'm
> off to catch the bus) but if not, lets roll it back.
>

Robert, IMHO, the below code is wrong...

c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) ==
sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")

what would happend if size_t is unsigned int, and sizeof(unsigend int)
< sizeof(unsigend long). You will end-up returning garbage...

As I said in previous mail, I believe the only sane way to fix this is
to use a property...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Feb 12 00:51:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 12 Feb 2009 00:51:00 +0100
Subject: [Cython] size_t problem
Message-ID: <3317244679.737037@smtp.netcom.no>

What about two nested ?-operators?

Dag Sverre Seljebotn
-----Original Message-----
From: Lisandro Dalcin <dalcinl at gmail.com>
Date: Wednesday, Feb 11, 2009 10:59 pm
Subject: Re: [Cython] size_t problem
To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net

On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
><robertwb at math.washington.edu> wrote:
>>
>> I think the fix should be simple (I'll look at it later today, I'm
>> off to catch the bus) but if not, lets roll it back.
>>
>
>Robert, IMHO, the below code is wrong...
>
>c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) =sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")
>
>what would happend if size_t is unsigned int, and sizeof(unsigend int)
>< sizeof(unsigend long). You will end-up returning garbage...
>
>As I said in previous mail, I believe the only sane way to fix this is
>to use a property...
>
>
>
>-- 
>Lisandro Dalc?n
>---------------
>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>Tel/Fax: +54-(0)342-451.1594
>_______________________________________________
>Cython-dev mailing list
>Cython-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/cython-dev
>


From jasone at canonware.com  Thu Feb 12 01:13:38 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 16:13:38 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
Message-ID: <499369B2.1040004@canonware.com>

Lisandro Dalcin wrote:
> [...] this is spotting a portability
> problem on your code in previous Cython versions :-).
> 
> On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
>> I have a class declaration that looks like:
>>
>> cdef class DistMatrix:
>>     # ...
>>     cdef readonly size_t ntaxa

I don't understand what portability problem you're referring to. 
Assuming that size_t was defined to be unsigned long or unsigned long 
long (depending on the platform), why would there be a problem?

Thanks,
Jason

P.S.  The changes Robert pushed today allow my code to build and run 
again.  Thanks Robert!

From michael.abshoff at googlemail.com  Thu Feb 12 01:46:07 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 11 Feb 2009 16:46:07 -0800
Subject: [Cython] size_t problem
In-Reply-To: <499369B2.1040004@canonware.com>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
	<499369B2.1040004@canonware.com>
Message-ID: <4993714F.7050903@gmail.com>

Jason Evans wrote:
> Lisandro Dalcin wrote:
>> [...] this is spotting a portability
>> problem on your code in previous Cython versions :-).
>>
>> On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
>>> I have a class declaration that looks like:
>>>
>>> cdef class DistMatrix:
>>>     # ...
>>>     cdef readonly size_t ntaxa
> 
> I don't understand what portability problem you're referring to. 
> Assuming that size_t was defined to be unsigned long or unsigned long 
> long (depending on the platform), why would there be a problem?

Code created by Cython on one platform ought to work on another 
platform. And since Python.h provides that info about the size of size_t 
for you why bother to implement that independently?

> Thanks,
> Jason

Cheers,

Michael

> P.S.  The changes Robert pushed today allow my code to build and run 
> again.  Thanks Robert!
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From jasone at canonware.com  Thu Feb 12 02:17:12 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 17:17:12 -0800
Subject: [Cython] size_t problem
In-Reply-To: <4993714F.7050903@gmail.com>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>	<499369B2.1040004@canonware.com>
	<4993714F.7050903@gmail.com>
Message-ID: <49937898.9000903@canonware.com>

Michael Abshoff wrote:
> Jason Evans wrote:
>> I don't understand what portability problem you're referring to. 
>> Assuming that size_t was defined to be unsigned long or unsigned long 
>> long (depending on the platform), why would there be a problem?
> 
> Code created by Cython on one platform ought to work on another 
> platform. And since Python.h provides that info about the size of size_t 
> for you why bother to implement that independently?

Ah, okay.  I had to give up on the portability of Cython-generated code 
in order to add the ability to conditionally support mpi4py, so I 
haven't really been thinking in terms of portable Cython output lately. 
  Thanks for the explanation.

Jason

From robertwb at math.washington.edu  Thu Feb 12 03:44:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 18:44:19 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902111359o158326c5sde4aa524eb015ef1@mail.gmail.com>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
	<ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
	<919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>
	<e7ba66e40902111359o158326c5sde4aa524eb015ef1@mail.gmail.com>
Message-ID: <01280D2A-712B-4677-BC6B-07AE7704D166@math.washington.edu>

On Feb 11, 2009, at 1:59 PM, Lisandro Dalcin wrote:

> On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> I think the fix should be simple (I'll look at it later today, I'm
>> off to catch the bus) but if not, lets roll it back.
>>
>
> Robert, IMHO, the below code is wrong...
>
> c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) ==
> sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")
>
> what would happend if size_t is unsigned int, and sizeof(unsigend int)
> < sizeof(unsigend long). You will end-up returning garbage...

Yes, this is the part that I said was a bit hacky, but I am almost  
sure that sizeof(size_t) = sizeof(int) < sizeof(long) is *really*  
rare. It would also break if, say, sizeof(long) < sizeof(size_t) <  
sizeof(long long) on some weird system with, say, a 43-bit size_t.

I actually considered doing a nested if statement, but decided it  
wasn't worth it for now--if anyone on this list is using such a  
system speak up :).

> As I said in previous mail, I believe the only sane way to fix this is
> to use a property...

This was exactly what I was thinking of doing, as well as some other  
changes/simplifications in there, as part of adapting complex numbers  
into the types framework. That's why I figured it wasn't worth  
putting lots of effort into it now, especially as I'd really like to  
push out a release soon. On that note, I've put up a temporary http:// 
hg.cython.org/cython-0.11/ which is for bugfixes only.

- Robert


From lodatom at gmail.com  Thu Feb 12 03:45:30 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Wed, 11 Feb 2009 21:45:30 -0500
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>
References: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>
Message-ID: <ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>

I wrote a small library for creating Numpy ufuncs using Cython.  Just
copy the pxd and pxi linked at
http://wiki.cython.org/MarkLodato/CreatingUfuncs and include the pxi
at the top of your file.  Then you can create 1-input/1-output or
2-input/1-output ufuncs operating of floats, doubles, or long doubles
easily.

For those unfamiliar with ufuncs, these are functions that operate on
NumPy arrays - for example, all of the mathematical operators of NumPy
(add, subtract, multiply, exp, etc) are ufuncs.  They are nice for the
user since they handle broadcasting and all the other stuff you expect
of NumPy.  They are also nice for the writer of the function, since
one needs only write a simple 1-D loop (or even a function that takes
scalars as input and output) and NumPy takes care of the rest.  That
is, the author need not worry about multi-dimensional arrays, error
handling, broadcasting rules, output buffers, etc.  For more details,
see http://docs.scipy.org/doc/numpy/reference/ufuncs.html.

Although NumPy makes it easy to write a ufunc, there's still a fair
amount of boilerplate.  To make the process simpler, I created this
library to do most of the work for you.  Just define a function that
takes a scalar (or two) as input and returns a scalar, then register
the function using register_ufunc_d() (or similar).  The wiki page
shows how to create an "lgamma" ufunc.  Here's another simple example:

{{{
include "numpy_ufuncs.pxi"

cdef double foo_dd(double a, double b):
   return (a-b) / (a+b)

foo = register_ufunc_dd(foo_dd, "foo", "returns (x1-x2)/(x1+x2)", PyUFunc_Zero)
}}}

Now in Python you can do:

{{{
In [1]: from foo import foo

In [2]: import numpy as np

In [3]: a = np.arange(16).reshape(4,4)

In [4]: b = np.arange(30,34)

In [5]: foo(a,b)
Out[5]:
array([[-1.        , -0.9375    , -0.88235294, -0.83333333],
      [-0.76470588, -0.72222222, -0.68421053, -0.65      ],
      [-0.57894737, -0.55      , -0.52380952, -0.5       ],
      [-0.42857143, -0.40909091, -0.39130435, -0.375     ]])

In [6]: foo(32, 16)
Out[6]: 0.33333333333333331
}}}

The great thing is that it's easy to write, easy to use, and fast!

I would like to add more common cases (particularly integer) and more
generic cases.  The NumPy C API only has 1-D loops for floating point
(e.g. PyUFunc_d_d), so I would have to write this myself.  For now
this fits my needs.

I'd be happy to hear any comments or suggestions you have.  Perhaps
this can be distributed with Cython?  It would be really nice to be
able to easily create ufuncs in Cython right out of the box.

~Mark Lodato

From greg.ewing at canterbury.ac.nz  Thu Feb 12 04:27:58 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 12 Feb 2009 16:27:58 +1300
Subject: [Cython] size_t problem
In-Reply-To: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
Message-ID: <4993973E.2030205@canterbury.ac.nz>

Robert Bradshaw wrote:
> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
> 
>>cdef class A:
>>   cdef readonly Py_ssize_t n
> 
> People used to be able to write "ctypedef unsigned  
> long size_t" and have it work.

Only if size_t is actually the size of an unsigned
long on the platform concerned, though. Otherwise
the Python runtime's idea of the struct layout won't
match what is actually there.

-- 
Greg

From robertwb at math.washington.edu  Thu Feb 12 04:35:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 19:35:08 -0800
Subject: [Cython] size_t problem
In-Reply-To: <4993973E.2030205@canterbury.ac.nz>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
	<4993973E.2030205@canterbury.ac.nz>
Message-ID: <7289B038-B4A4-4E13-8CF5-6E57AB3B631A@math.washington.edu>

On Feb 11, 2009, at 7:27 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
>>
>>> cdef class A:
>>>   cdef readonly Py_ssize_t n
>>
>> People used to be able to write "ctypedef unsigned
>> long size_t" and have it work.
>
> Only if size_t is actually the size of an unsigned
> long on the platform concerned, though. Otherwise
> the Python runtime's idea of the struct layout won't
> match what is actually there.

What I meant was to use size_t one would write

cdef extern from *:
     ctypedef unsigned long size_t

which would work perfect as long as sizeof(unsigned long) == sizeof 
(size_t), and would also usually work fine if the sizes were  
different (if one never needed to pass to and from Python-land with a  
value large enough to care (though of course this is a bug waiting to  
happen)). Now that it's a builtin type, it broke the user being able  
to declare it, and we had to make sure to continue to support all use- 
cases.

- Robert


From dagss at student.matnat.uio.no  Thu Feb 12 10:51:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 10:51:25 +0100
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>
References: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>
	<ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>
Message-ID: <4993F11D.2050108@student.matnat.uio.no>

Mark Lodato wrote:
> I wrote a small library for creating Numpy ufuncs using Cython.  Just
> copy the pxd and pxi linked at
> http://wiki.cython.org/MarkLodato/CreatingUfuncs and include the pxi
> at the top of your file.  Then you can create 1-input/1-output or
> 2-input/1-output ufuncs operating of floats, doubles, or long doubles
> easily.
>
>   
Really nice! Excuse my ignorance about ufuncs, but is there a function 
pointer call overhead for every element this way, or is it all macro based?
> I'd be happy to hear any comments or suggestions you have.  Perhaps
> this can be distributed with Cython?  It would be really nice to be
> able to easily create ufuncs in Cython right out of the box.
>   
Definitely. It could be part of numpy.pxd I think; either that, or make 
a pxd package and have numpy.ufunc as a sub-pxd.

I wouldn't like to ship it as "pxi" (because include is kind of 
discouraged these days and it is not good if the Cython standard lib 
requires it). I think all the functions can be put in a pxd if you mark 
them "inline". (An alternative is to create a numpy_cython_ufuncs module 
in Cython/Runtime, but not unless we have to).

It would be best if you would play around in Cython/Includes and submit 
a patch.

Dag Sverre


From dagss at student.matnat.uio.no  Thu Feb 12 10:53:34 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 10:53:34 +0100
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <4993F11D.2050108@student.matnat.uio.no>
References: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>	<ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>
	<4993F11D.2050108@student.matnat.uio.no>
Message-ID: <4993F19E.4050809@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Mark Lodato wrote:
>   
>> I wrote a small library for creating Numpy ufuncs using Cython.  Just
>> copy the pxd and pxi linked at
>> http://wiki.cython.org/MarkLodato/CreatingUfuncs and include the pxi
>> at the top of your file.  Then you can create 1-input/1-output or
>> 2-input/1-output ufuncs operating of floats, doubles, or long doubles
>> easily.
>>
>>   
>>     
> Really nice! Excuse my ignorance about ufuncs, but is there a function 
> pointer call overhead for every element this way, or is it all macro based?
>   
Also feel free to link to your wiki page from tutorials/numpy on the wiki.

Dag Sverre

From dalcinl at gmail.com  Thu Feb 12 14:55:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 11:55:44 -0200
Subject: [Cython] size_t problem
In-Reply-To: <49937898.9000903@canonware.com>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
	<499369B2.1040004@canonware.com> <4993714F.7050903@gmail.com>
	<49937898.9000903@canonware.com>
Message-ID: <e7ba66e40902120555t19e64155v3980bab1bb076e7e@mail.gmail.com>

On Wed, Feb 11, 2009 at 11:17 PM, Jason Evans <jasone at canonware.com> wrote:
> Michael Abshoff wrote:
>
> Ah, okay.  I had to give up on the portability of Cython-generated code
> in order to add the ability to conditionally support mpi4py, so I
> haven't really been thinking in terms of portable Cython output lately.
>  Thanks for the explanation.
>

I do not understand you here. Why is mpi4py a source of problems?
mpi4py should work the same on all platforms. Well, we have MPI_Aint
and MPI_Offset, I'm currently using typedefs, but this is an approach
I have to review and fix ASAP.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Feb 12 14:58:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 11:58:37 -0200
Subject: [Cython] size_t problem
In-Reply-To: <3317244679.737037@smtp.netcom.no>
References: <3317244679.737037@smtp.netcom.no>
Message-ID: <e7ba66e40902120558x348fe6fdy5cadaaf4df4c797@mail.gmail.com>

Yes, too nested ?: operators should work. We have to contemplate at
least on the three cases size_t in (uint, ulong, ulonglong).

 I'll work on it and push a patch.

On Wed, Feb 11, 2009 at 9:51 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> What about two nested ?-operators?
>
> Dag Sverre Seljebotn
> -----Original Message-----
> From: Lisandro Dalcin <dalcinl at gmail.com>
> Date: Wednesday, Feb 11, 2009 10:59 pm
> Subject: Re: [Cython] size_t problem
> To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net
>
> On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
>><robertwb at math.washington.edu> wrote:
>>>
>>> I think the fix should be simple (I'll look at it later today, I'm
>>> off to catch the bus) but if not, lets roll it back.
>>>
>>
>>Robert, IMHO, the below code is wrong...
>>
>>c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) =sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")
>>
>>what would happend if size_t is unsigned int, and sizeof(unsigend int)
>>< sizeof(unsigend long). You will end-up returning garbage...
>>
>>As I said in previous mail, I believe the only sane way to fix this is
>>to use a property...
>>
>>
>>
>>--
>>Lisandro Dalc?n
>>---------------
>>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>Tel/Fax: +54-(0)342-451.1594
>>_______________________________________________
>>Cython-dev mailing list
>>Cython-dev at codespeak.net
>>http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan at sun.ac.za  Thu Feb 12 17:40:30 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 12 Feb 2009 18:40:30 +0200
Subject: [Cython] Incomplete C++ example
In-Reply-To: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>
Message-ID: <9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>

Hi all,

I'm helping a student to wrap some C++ code using Cython, and noticed
that the example in the documentation is incomplete.  A header file,
Rectangle.h, is provided, but no implementation is given, so the
example cannot be compiled.

It would be great if you could add an implementation (see attached)
and modify the setup.py accordingly, i.e.

   Extension(
   "rectangle",
   ["rectangle.pyx", "Rectangle.cc"],
   language="c++")

Cheers
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Rectangle.cc
Type: text/x-c++src
Size: 503 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090212/a419c7ed/attachment.cc 

From jasone at canonware.com  Thu Feb 12 17:47:02 2009
From: jasone at canonware.com (Jason Evans)
Date: Thu, 12 Feb 2009 08:47:02 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902120555t19e64155v3980bab1bb076e7e@mail.gmail.com>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>	<499369B2.1040004@canonware.com>
	<4993714F.7050903@gmail.com>	<49937898.9000903@canonware.com>
	<e7ba66e40902120555t19e64155v3980bab1bb076e7e@mail.gmail.com>
Message-ID: <49945286.40103@canonware.com>

Lisandro Dalcin wrote:
> On Wed, Feb 11, 2009 at 11:17 PM, Jason Evans <jasone at canonware.com> wrote:
>> Ah, okay.  I had to give up on the portability of Cython-generated code
>> in order to add the ability to conditionally support mpi4py, so I
>> haven't really been thinking in terms of portable Cython output lately.
>>  Thanks for the explanation.
> 
> I do not understand you here. Why is mpi4py a source of problems?
> mpi4py should work the same on all platforms. Well, we have MPI_Aint
> and MPI_Offset, I'm currently using typedefs, but this is an approach
> I have to review and fix ASAP.

The issue is that I need to be able to compile my software with the 
mpi4py code disabled (because it isn't a reasonable expectation that my 
users will always have MPI installed).  There is no way I've found to 
make a cimport of a non-existent module safe, except to wrap it in an IF 
block.

So, mpi4py is only indirectly the source of my problems, because it's 
the only third party Cython module I'm using at the moment.  mpi4py 
looks like a great piece of software, and it has worked well for the 
limited testing I've done with it.

Jason

From dalcinl at gmail.com  Thu Feb 12 21:26:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 18:26:05 -0200
Subject: [Cython] more 'size_t' stuff pushed
Message-ID: <e7ba66e40902121226q531eedcbya2d8480f4b5224b2@mail.gmail.com>

After a lot of googling, diving on CPython sources (2.3 to 3.0) and
help from Albert Strasheim and Matthew Knepley, I've pushed more stuff
for 'size_t' (and 'Py_ssize_t') support. Now Cython should be near to
do the right thing, but I cannot be sure until all this is not tested
on Window 32 and Windows 64.

A thing (related to recent fix by Robert) that I was not able to fix
is the unused variable warning below...

size_t.c: In function '__pyx_pf_6size_t_1A___init__':
size_t.c:390: warning: unused variable '__pyx_arg_a'
size_t.cpp: In function 'int __pyx_pf_6size_t_1A___init__(PyObject*,
PyObject*, PyObject*)':
size_t.cpp:390: warning: unused variable '__pyx_arg_a'
tryfinally.c: In function '__pyx_pf_10tryfinally_try_return_temp':

Other missing task is extend Robert's fix to 'long double' ...

BTW, Cython generates this C code... Is this PY_LONG_LONG fallback
definition supposed to be portable?

#define PY_SSIZE_T_CLEAN
#include "Python.h"
#include "structmember.h"
#ifndef PY_LONG_LONG
  #define PY_LONG_LONG LONG_LONG
#endif


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Feb 12 21:42:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 18:42:15 -0200
Subject: [Cython] Adding T_SIZET to structmember.h
Message-ID: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>

After a rather long hacking on Cython in order to support 'Py_ssize_t'
and 'size_t' the right way, I would like to propose the inclusion of a
new T_SIZET in structmember.h in order to suport 'size_t' struct
fields with PyMemberDef. Would such addition be accepted for 2.7 and
3.1?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lodatom at gmail.com  Fri Feb 13 01:48:24 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Thu, 12 Feb 2009 19:48:24 -0500
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
Message-ID: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>

> Really nice! Excuse my ignorance about ufuncs, but is there a function
> pointer call overhead for every element this way, or is it all macro based

Unfortunately, there is a function call for every single element
because I used the PyUFunc_*_* family of wrappers.  For something like
lgamma(), this is perfectly acceptable since you would have to call
the function once per element anyway.  But for your own functions, it
would be nice to eliminate this overhead.  One way around this would
be for me to re-implement the PyUFunc_*_* as static functions, then
have the user declare the element-wise function as inline so that the
compiler does the inlining for us.  Another way around the overhead
would be to build the ufunc wrapping into Cython, perhaps as a
decorator or something.  This would make the user's job really simple,
but I'm not sure if it's worth the effort.

There is also the overhead of using the ufunc machinery, but I think
it's probably safe to say this overhead is minimal since all the NumPy
operators and functions are implemented as ufuncs.

> I wouldn't like to ship it as "pxi" (because include is kind of
> discouraged these days and it is not good if the Cython standard lib
> requires it). I think all the functions can be put in a pxd if you mark
> them "inline". (An alternative is to create a numpy_cython_ufuncs module
> in Cython/Runtime, but not unless we have to).

How can I implement this as a pxd?  I used pxi for two reasons: (1)
import_ufunc() needs to be called before anything is used, so by
including the pxi I can force this to be done automatically, and (2)
the ufunc API functions expect arrays that will stay in existence
forever.  I used globals since this is the easiest route, but you
can't define globals in a pxd.  I guess I can malloc and just create a
memory leak, but this seems sloppy.

Another way to implement all this would be to do it as C macros in a
header file.  The downside to this approach is that the user would
have to add another -I to their CFLAGS.  Is there any way to include
pure C code (well, really CPP macros) into the pxd?  Perhaps a better
solution would be to offer these macros as a patch to NumPy, rather
than Cython?

> It would be best if you would play around in Cython/Includes and submit
> a patch.

I plan to :)

~Mark

From greg.ewing at canterbury.ac.nz  Fri Feb 13 02:18:52 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 13 Feb 2009 14:18:52 +1300
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
References: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
Message-ID: <4994CA7C.2060704@canterbury.ac.nz>

Mark Lodato wrote:
> Another way around the overhead
> would be to build the ufunc wrapping into Cython, perhaps as a
> decorator or something.

I've never looked at the implementation of ufuncs, so this may
be completely nonsensical, but would it be possible to pass
shape/strides info into the function, so that it could do its
own looping over the data?

-- 
Greg


From lodatom at gmail.com  Fri Feb 13 02:44:57 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Thu, 12 Feb 2009 20:44:57 -0500
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <4994CA7C.2060704@canterbury.ac.nz>
References: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
	<4994CA7C.2060704@canterbury.ac.nz>
Message-ID: <ca433830902121744s6823f5b9i2b7a39bdb60a6f45@mail.gmail.com>

On Thu, Feb 12, 2009 at 8:18 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I've never looked at the implementation of ufuncs, so this may
> be completely nonsensical, but would it be possible to pass
> shape/strides info into the function, so that it could do its
> own looping over the data?

Good question.  Not only is that possible, but that's what NumPy
expects.  You define a function that takes one dimensional arrays and
outputs one dimensional arrays (iterating over the single dimension
yourself), and the ufunc machinery takes care of handling the
multi-dimensional aspects, broadcasting, types, error handling, etc.

To make things even easier, NumPy includes a set of functions, such as
PyUFunc_d_d(), which perform the one dimensional loop, calling a
supplied function pointer for each element.  For example, if you
define a function, "double foo(double)", you can register your ufunc
with PyUFunc_d_d() as the "function" and foo() as the "data".  This is
easier, but the downside is the extra function call overhead.

The other thing to note is that you can register multiple functions,
each handling different data types, for a single ufunc.  This allows
NumPy to perform the function directly without having to create a
temporary array of the correct type.  In the NumPy source, there is an
"add" function for every possible data type, while my lgamma example
from the wiki page only defines float, double, and long double
functions.

~Mark

From dagss at student.matnat.uio.no  Fri Feb 13 10:16:11 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 13 Feb 2009 10:16:11 +0100
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
References: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
Message-ID: <49953A5B.2080303@student.matnat.uio.no>

Mark Lodato wrote:
> How can I implement this as a pxd?  I used pxi for two reasons: (1)
> import_ufunc() needs to be called before anything is used, so by

I'm wondering whether this can be done automatically by giving the pxd 
the right name so that Cython imports the right module. If you declare 
external Python extension types in the pxd, at least, this happens. But 
may not work in this case.

> including the pxi I can force this to be done automatically, and (2)
> the ufunc API functions expect arrays that will stay in existence
> forever.  I used globals since this is the easiest route, but you
> can't define globals in a pxd.  I guess I can malloc and just create a
> memory leak, but this seems sloppy.

Yeah, sorry, I overlooked that part when glancing at it. I say just keep 
using a pxi; one can always change it later if we come up with something.

So, a) try to integrate the pxd in a nice way, since it exposes NumPy 
API (i.e. put it either inside the numpy.pxd file, or under it in the 
package hierarchy), and b) also supply a cython_ufunc_utils.pxi (or 
similar). If you put a pxi file in Cython/Includes it will always be in 
the include path as well.

-- 
Dag Sverre

From dalcinl at gmail.com  Fri Feb 13 16:52:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 13 Feb 2009 13:52:43 -0200
Subject: [Cython] import/cimport gotcha
Message-ID: <e7ba66e40902130752j453000b5yc68c9d74c11f1f4e@mail.gmail.com>

Some time ago I reported that the following:

from foo import Bar
from foo cimport Bar

do not put 'Bar' in the module dicti. Commenting the lines below seems
to solve the issue. Am I missing something (I bet the answer is yes)?
Any comments?

diff -r 1771db44498b Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py	Thu Feb 12 18:22:15 2009 -0200
+++ b/Cython/Compiler/Nodes.py	Fri Feb 13 13:42:23 2009 -0200
@@ -4600,8 +4600,9 @@
                         break
             else:
                 entry =  env.lookup(target.name)
-                if entry.is_type and entry.type.name == name and
entry.type.module_name == self.module.module_name.value:
-                    continue # already cimported
+                #if entry.is_type and entry.type.name == name and
entry.type.module_name == self.module.module_name.value:
+                #    continue # already cimported
+                #    pass
                 self.interned_items.append(
                     (env.intern_identifier(name), target))
                 target.analyse_target_expression(env, None)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Feb 13 20:11:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 13 Feb 2009 17:11:02 -0200
Subject: [Cython] possible fix for withstat failure
Message-ID: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>

Please review, if it works, then feel free to reimplement the methods,
as with the current patch is a bit ugly (two calls to
try_substitution()), and then commit! Good weekend.

diff -r 1771db44498b Cython/Compiler/TreeFragment.py
--- a/Cython/Compiler/TreeFragment.py	Thu Feb 12 18:22:15 2009 -0200
+++ b/Cython/Compiler/TreeFragment.py	Fri Feb 13 17:06:00 2009 -0200
@@ -160,6 +160,7 @@
     def visit_NameNode(self, node):
         temphandle = self.tempmap.get(node.name)
         if temphandle:
+            node = self.try_substitution(node, node.name)
             node.name = temphandle
             return node
             # Replace name with temporary


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Fri Feb 13 20:54:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 13 Feb 2009 20:54:30 +0100
Subject: [Cython] possible fix for withstat failure
In-Reply-To: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>
References: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>
Message-ID: <4995CFF6.3030802@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Please review, if it works, then feel free to reimplement the methods,
> as with the current patch is a bit ugly (two calls to
> try_substitution()), and then commit! Good weekend.

I did this (well, it amounts to the same) some hours ago, see

http://hg.cython.org/cython-devel/rev/3304bc6fd2c3


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 13 20:54:47 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 13 Feb 2009 20:54:47 +0100
Subject: [Cython] possible fix for withstat failure
In-Reply-To: <4995CFF6.3030802@student.matnat.uio.no>
References: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>
	<4995CFF6.3030802@student.matnat.uio.no>
Message-ID: <4995D007.2040608@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:
>> Please review, if it works, then feel free to reimplement the methods,
>> as with the current patch is a bit ugly (two calls to
>> try_substitution()), and then commit! Good weekend.
> 
> I did this (well, it amounts to the same) some hours ago, see
> 
> http://hg.cython.org/cython-devel/rev/3304bc6fd2c3
> 
> 

Thanks though! :-)

-- 
Dag Sverre

From greg.ewing at canterbury.ac.nz  Sat Feb 14 02:37:31 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 14 Feb 2009 14:37:31 +1300
Subject: [Cython] [Python-Dev] Adding T_SIZET to structmember.h
In-Reply-To: <499611E6.1020104@v.loewis.de>
References: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>
	<5c6f2a5d0902130715x4d4f4beet7d936f9c549a1335@mail.gmail.com>
	<e7ba66e40902130744s4a14c9a0n5dfed4b5905775d5@mail.gmail.com>
	<499611E6.1020104@v.loewis.de>
Message-ID: <4996205B.8090503@canterbury.ac.nz>

Martin v. L?wis wrote:
>>Mark, the patch is not trivial, I cannot spend time on this until this
>>is accepted. Hope you understand.
> 
> I certainly do understand. So it's likely not going to happen.

Could some platform-dependent #defines be added to
the Cython-generated preamble in the meantime to get
the equivalent of this?

-- 
Greg

From jf.moulin at gmail.com  Sat Feb 14 17:14:46 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Sat, 14 Feb 2009 17:14:46 +0100
Subject: [Cython] function declaration problem
Message-ID: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>

Hi,

my first, probably very naive, question to the list.... I want to
create an extension class with a method whose definition changes as a
function of some test
at the time of creation of the instance.
Something like:

cdef class Test:

    cdef int x, foo

    def __init__(self,foo):

        if foo>4:
            self.f = self.fn1
        else:
            self.f = self.fn2

    cdef fn1(self,x):
        return x

    cdef fn2(self,x):
        return 2 * x


Is this possible, and if yes, how should I declare f?

Thanks in advance for your help.

JF

From greg.ewing at canterbury.ac.nz  Sat Feb 14 20:48:58 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 15 Feb 2009 08:48:58 +1300
Subject: [Cython] function declaration problem
In-Reply-To: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
References: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
Message-ID: <4997202A.7090402@canterbury.ac.nz>

Jean-Francois Moulin wrote:
> I want to
> create an extension class with a method whose definition changes as a
> function of some test

Store a function pointer as a C attribute and have
a fixed f method that calls it.

-- 
Greg

From stefan at sun.ac.za  Mon Feb 16 11:49:36 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 16 Feb 2009 12:49:36 +0200
Subject: [Cython] Optimised data structures
Message-ID: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>

Hi all,

Is the list of optimised data-structures documented somewhere?  For
example, I know that lists are fast to manipulate in Cython, but how
about dictionaries, etc.?

Re: the docs, there's a linking problem in

http://docs.cython.org/docs/tutorial.html#language-details

and the Index and Module files are missing.  I can send a patch, if needed.

Regards
St?fan

From greg.ewing at canterbury.ac.nz  Mon Feb 16 12:02:31 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 17 Feb 2009 00:02:31 +1300
Subject: [Cython] Optimised data structures
In-Reply-To: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
Message-ID: <499947C7.2040902@canterbury.ac.nz>

St?fan van der Walt wrote:

> Is the list of optimised data-structures documented somewhere?

In the Pyrex docs there's a list of all the types it
knows about and the API calls that it will generate
given the right circumstances.

Not sure if it's still up to date with respect to
Cython, though.

-- 
Greg

From stefan_ml at behnel.de  Mon Feb 16 12:00:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Feb 2009 12:00:31 +0100 (CET)
Subject: [Cython] Optimised data structures
In-Reply-To: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
Message-ID: <54297.213.61.181.86.1234782031.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

St?fan van der Walt wrote:
> Is the list of optimised data-structures documented somewhere?  For
> example, I know that lists are fast to manipulate in Cython, but how
> about dictionaries, etc.?

All builtin types should be fast to use. If not, that's a bug.

Stefan


From stefan_ml at behnel.de  Mon Feb 16 13:33:24 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Feb 2009 13:33:24 +0100 (CET)
Subject: [Cython] Optimised data structures
In-Reply-To: <499947C7.2040902@canterbury.ac.nz>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
	<499947C7.2040902@canterbury.ac.nz>
Message-ID: <64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Greg Ewing wrote:
> In the Pyrex docs there's a list of all the types it
> knows about and the API calls that it will generate
> given the right circumstances.
>
> Not sure if it's still up to date with respect to
> Cython, though.

It's definitely not. It's difficult to keep the docs in sync with all
places where optimisations are done in one way or another. Loops are a
good example.

I think it's actually best to just write your code the way you would in
Python, and then check the generated C code to see if it comes out as
expected. That way, you will not only spot places where Cython does not
optimise as expected (in which case a patch is appreciated, although a
wishlist bug report will also be helpful), but you will also find places
where Cython simply didn't know your intention, so that an additional
"cdef" in the right places will bring you a major speedup. That's pretty
orthogonal to type specific optimisations.

Stefan


From stefan at sun.ac.za  Mon Feb 16 13:44:56 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 16 Feb 2009 14:44:56 +0200
Subject: [Cython] Optimised data structures
In-Reply-To: <64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
	<499947C7.2040902@canterbury.ac.nz>
	<64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <9457e7c80902160444j76dce4f7tb4dbaf0ebd8c5cbc@mail.gmail.com>

2009/2/16 Stefan Behnel <stefan_ml at behnel.de>:
> I think it's actually best to just write your code the way you would in
> Python, and then check the generated C code to see if it comes out as
> expected.

Thanks for the advice (and thank you for annotated output)!

Regards
St?fan

From dalcinl at gmail.com  Mon Feb 16 14:01:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 16 Feb 2009 11:01:59 -0200
Subject: [Cython] [Python-Dev] Adding T_SIZET to structmember.h
In-Reply-To: <4996205B.8090503@canterbury.ac.nz>
References: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>
	<5c6f2a5d0902130715x4d4f4beet7d936f9c549a1335@mail.gmail.com>
	<e7ba66e40902130744s4a14c9a0n5dfed4b5905775d5@mail.gmail.com>
	<499611E6.1020104@v.loewis.de> <4996205B.8090503@canterbury.ac.nz>
Message-ID: <e7ba66e40902160501k55fc3a7fvc5154de79c2545ad@mail.gmail.com>

Greg, what you are suggesting is what I've implemented and pushed to
cython-devel. Of couse, the #define can only deal with size_t being
unsigned int/long/longlong. It should be enough, right?

On Fri, Feb 13, 2009 at 11:37 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Martin v. L?wis wrote:
>>>Mark, the patch is not trivial, I cannot spend time on this until this
>>>is accepted. Hope you understand.
>>
>> I certainly do understand. So it's likely not going to happen.
>
> Could some platform-dependent #defines be added to
> the Cython-generated preamble in the meantime to get
> the equivalent of this?
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb 16 16:55:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 16 Feb 2009 13:55:22 -0200
Subject: [Cython] [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
In-Reply-To: <E2F78134-ABBD-4822-A843-96CDB2127D77@mcs.anl.gov>
References: <760F2670-F665-45DC-8D34-CBDD3493EBDB@stanford.edu>
	<E2F78134-ABBD-4822-A843-96CDB2127D77@mcs.anl.gov>
Message-ID: <e7ba66e40902160755l3058678duc09b9cc7ccb997d2@mail.gmail.com>

Two separate issues here...

1) Regarding mpi4py, as it is based on the C API for MPI, it does not
have complex support natively. Of course, you could use

COMPLEX = MPI.DOUBLE.Create_contiguous(2)

and then the new 'COMPLEX' datatype will likely work for communicating
arrays with (double precision) complex entries.

1) Regarding petsc4py, the support for complex numbers is not ready
yet. The big issue is that Cython does not natively support them.
Rather than hacking petsc4py for complex support, I would prefer to
spend my time on hacking Cython to enable good complex support.
However, if there is a strong need for petsc4py+complex, I could work
on providing a patch to make it work.



Now a request to the Cython folks,

Could we create a branch 'cython-devel-complex' for start working on
good complex support?
This way I could push my preliminary work on this, all you can review
my progress, and other users can pull and build petsc4py with complex
support...



On Mon, Feb 16, 2009 at 1:17 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Begin forwarded message:
>
> From: Wonseok Shin <wsshin at stanford.edu>
> Date: February 15, 2009 9:44:08 PM CST
> To: petsc-maint Maintenance <petsc-maint at mcs.anl.gov>
> Cc: petsc-maint at mcs.anl.gov
> Subject: Fwd: [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
> Dear PETSc developers,
> I have received the following quoted email from the PETSc installer of
> BIgben, PSC.
> Is it true that mpi4py and petsc4py cannot be used with PETSc built for
> complex numbers?  Is there any workaround?
>
> Regards,
> Wonseok Shin
> Ph.D candidate
> Department of Electrical Engineering, Stanford University
> wsshin at stanford.edu
>
> Begin forwarded message:
>
> From: "Rick Costa via RT" <remarks at psc.edu>
> Date: February 15, 2009 8:22:21 AM PST
> To: wsshin at stanford.edu
> Subject: Re: [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
> Reply-To: remarks at psc.edu
>
>
> ---------- Forwarded message ----------
> Date: Fri, 13 Feb 2009 01:50:27 -0500
> From: David O'Neal <oneal at psc.edu>
> To: R.F. Costa <costa at psc.edu>
> Subject: Re: PETsc additions
>
> Hi Rick.
>
> I've run into a problem with the build requested by the user. Support for
> Python
> features is incompatible with the complex data type...
>
> *********************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for
> details):
> -----------------------------------------------------------------------------------
> Cannot use mpi4py with complex numbers it is not coded for this capability
> *********************************************************************************
>
> At 08:21 AM 2/10/2009 -0500, you wrote:
>
> Dave
>
> Is it possible to make these additions to PETSc? Have you had a chance
>
> to install the bug fix to PETSc? Thanks.
>
> Rick
>
> ---------- Forwarded message ----------
>
> Date: Tue, 10 Feb 2009 07:58:55 -0500 (EST)
>
> From: Wonseok Shin via RT <remarks at psc.edu>
>
> Subject: [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
>
> Tue Feb 10 07:58:55 2009: Request 16673 was acted upon.
>
> Transaction: Ticket created by wsshin at stanford.edu
>
>      Queue: Remarks
>
>    Subject: Fwd: [PETSC #19028] PETSc and Python
>
>      Owner: Nobody
>
> Requestors: wsshin at stanford.edu
>
>     Status: new
>
> Ticket <URL: https://tickets.psc.edu/Ticket/Display.html?id=16673 >
>
>
>
> Dear Rick,
>
> I am Wonseok Shin using bigben.
>
> I would like to distribute input data to my PETSc program using
>
> Python.  I asked PETSc developers how to do that, and they sent me the
>
> email quoted below.
>
> Basically, the Python packages called mpi4py and petsc4py should be
>
> installed, and they said that this can be done by configuring PETSc
>
> with the options written in the quoted email.
>
> Is it possible to install mpi4py and petsc4py?  Could you forward this
>
> email to the PETSc or Python installer of bigben?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> wsshin at stanford.edu
>
>
>
> Begin forwarded message:
>
> From: Matthew Knepley <petsc-maint at mcs.anl.gov>
>
> Date: February 10, 2009 4:39:57 AM PST
>
> To: Wonseok Shin <wsshin at stanford.edu>
>
> Cc: petsc-maint Maintenance <petsc-maint at mcs.anl.gov>
>
> Subject: Re: [PETSC #19028] PETSc and Python
>
> Reply-To: petsc-maint at mcs.anl.gov
>
> You can use mpi4py and petsc4py. They both can be automatically
>
> installed by configure:
>
> --download-petsc4py --download-mpi4py
>
> They have a lot of documentation associated with them and examples.
>
>  Matt
>
> On Tue, Feb 10, 2009 at 5:42 AM, Wonseok Shin <wsshin at stanford.edu>
>
> wrote:
>
> Dear PETSc developers,
>
> I am interested in the interaction of PETSc and Python.  Is it
>
> possible to
>
> distribute data from Python to PETSc and vice versa?  I know how to
>
> do this
>
> for a single process, but don't know for multiple processes.
>
> If I need some Python MPI module, could you recommend a specific
>
> one that
>
> works well with PETSc?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> wsshin at stanford.edu
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
>
> experiments is infinitely more interesting than any results to which
>
> their experiments lead.
>
> -- Norbert Wiener
>
>
>
> Dear Rick,
>
> I am Wonseok Shin using bigben.
>
> I would like to distribute input data to my PETSc program using Python.  I
> asked PETSc developers how to do that, and they sent me the email quoted
> below.
>
> Basically, the Python packages called mpi4py and petsc4py should be
> installed, and they said that this can be done by configuring PETSc with the
> options written in the quoted email.
>
> Is it possible to install mpi4py and petsc4py?  Could you forward this email
> to the PETSc or Python installer of bigben?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> <mailto:wsshin at stanford.edu>wsshin at stanford.edu
>
>
>
> Begin forwarded message:
>
> From: Matthew Knepley
> <<mailto:petsc-maint at mcs.anl.gov>petsc-maint at mcs.anl.gov>
>
> Date: February 10, 2009 4:39:57 AM PST
>
> To: Wonseok Shin <<mailto:wsshin at stanford.edu>wsshin at stanford.edu>
>
> Cc: petsc-maint Maintenance
> <<mailto:petsc-maint at mcs.anl.gov>petsc-maint at mcs.anl.gov>
>
> Subject: Re: [PETSC #19028] PETSc and Python
>
> Reply-To: <mailto:petsc-maint at mcs.anl.gov>petsc-maint at mcs.anl.gov
>
> You can use mpi4py and petsc4py. They both can be automatically
>
> installed by configure:
>
> --download-petsc4py --download-mpi4py
>
> They have a lot of documentation associated with them and examples.
>
>  Matt
>
> On Tue, Feb 10, 2009 at 5:42 AM, Wonseok Shin
> <<mailto:wsshin at stanford.edu>wsshin at stanford.edu> wrote:
>
> Dear PETSc developers,
>
> I am interested in the interaction of PETSc and Python.  Is it possible to
>
> distribute data from Python to PETSc and vice versa?  I know how to do this
>
> for a single process, but don't know for multiple processes.
>
> If I need some Python MPI module, could you recommend a specific one that
>
> works well with PETSc?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> <mailto:wsshin at stanford.edu>wsshin at stanford.edu
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
>
> experiments is infinitely more interesting than any results to which
>
> their experiments lead.
>
> -- Norbert Wiener
>
>
>
> Our PETSc installer has created a new version of PETSc. You can run it
> by issuing the command
>
>   module load petsc/3.0.0-p3.complex
>
> I am not sure what it includes. I do not think it includes the Python
> additions you requested, because they are incompatible with the complex
> data type. I have asked for clarification on what this new version
> includes.
>
> Rick Costa
> PSC User Services
>
>
>
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Mon Feb 16 20:10:33 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Feb 2009 11:10:33 -0800
Subject: [Cython] [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
In-Reply-To: <e7ba66e40902160755l3058678duc09b9cc7ccb997d2@mail.gmail.com>
References: <760F2670-F665-45DC-8D34-CBDD3493EBDB@stanford.edu>
	<E2F78134-ABBD-4822-A843-96CDB2127D77@mcs.anl.gov>
	<e7ba66e40902160755l3058678duc09b9cc7ccb997d2@mail.gmail.com>
Message-ID: <84066170-DE9C-4B0A-AFC1-AD32672EA4B0@math.washington.edu>

On Feb 16, 2009, at 7:55 AM, Lisandro Dalcin wrote:

> Two separate issues here...
>
> 1) Regarding mpi4py, as it is based on the C API for MPI, it does not
> have complex support natively. Of course, you could use
>
> COMPLEX = MPI.DOUBLE.Create_contiguous(2)
>
> and then the new 'COMPLEX' datatype will likely work for communicating
> arrays with (double precision) complex entries.
>
> 1) Regarding petsc4py, the support for complex numbers is not ready
> yet. The big issue is that Cython does not natively support them.
> Rather than hacking petsc4py for complex support, I would prefer to
> spend my time on hacking Cython to enable good complex support.
> However, if there is a strong need for petsc4py+complex, I could work
> on providing a patch to make it work.
>
>
>
> Now a request to the Cython folks,
>
> Could we create a branch 'cython-devel-complex' for start working on
> good complex support?
> This way I could push my preliminary work on this, all you can review
> my progress, and other users can pull and build petsc4py with complex
> support...

I have started doing a bit of work on this too, but first priority  
should be to get 0.11 released. Then we can focus on new features again.

- Robert

From robertwb at math.washington.edu  Mon Feb 16 20:18:41 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Feb 2009 11:18:41 -0800
Subject: [Cython] Optimised data structures
In-Reply-To: <64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
	<499947C7.2040902@canterbury.ac.nz>
	<64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <9C4AB959-6070-4B83-9565-170928F1CE46@math.washington.edu>

On Feb 16, 2009, at 4:33 AM, Stefan Behnel wrote:

> Greg Ewing wrote:
>> In the Pyrex docs there's a list of all the types it
>> knows about and the API calls that it will generate
>> given the right circumstances.
>>
>> Not sure if it's still up to date with respect to
>> Cython, though.
>
> It's definitely not. It's difficult to keep the docs in sync with all
> places where optimisations are done in one way or another. Loops are a
> good example.

It would be nice to have this in the documentation, but there are so  
many little optimizations that it would be a lot to write them all  
down. And it's not very well defined to say "type X is optimized"  
because some operations may be optimized, and some not.

> I think it's actually best to just write your code the way you  
> would in
> Python, and then check the generated C code to see if it comes out as
> expected. That way, you will not only spot places where Cython does  
> not
> optimise as expected (in which case a patch is appreciated, although a
> wishlist bug report will also be helpful), but you will also find  
> places
> where Cython simply didn't know your intention, so that an additional
> "cdef" in the right places will bring you a major speedup. That's  
> pretty
> orthogonal to type specific optimisations.

I heartily second this.

- Robert


From greg.ewing at canterbury.ac.nz  Mon Feb 16 22:28:04 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 17 Feb 2009 10:28:04 +1300
Subject: [Cython] [Python-Dev] Adding T_SIZET to structmember.h
In-Reply-To: <e7ba66e40902160501k55fc3a7fvc5154de79c2545ad@mail.gmail.com>
References: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>
	<5c6f2a5d0902130715x4d4f4beet7d936f9c549a1335@mail.gmail.com>
	<e7ba66e40902130744s4a14c9a0n5dfed4b5905775d5@mail.gmail.com>
	<499611E6.1020104@v.loewis.de> <4996205B.8090503@canterbury.ac.nz>
	<e7ba66e40902160501k55fc3a7fvc5154de79c2545ad@mail.gmail.com>
Message-ID: <4999DA64.8020907@canterbury.ac.nz>

Lisandro Dalcin wrote:
> the #define can only deal with size_t being
> unsigned int/long/longlong. It should be enough, right?

Probably handles all the CPUs we're likely to
encounter at the moment.

-- 
Greg

From ondrej at certik.cz  Tue Feb 17 00:16:01 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 15:16:01 -0800
Subject: [Cython] where should .pxd files be installed
Message-ID: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>

Hi,

where should the .pxd files for a package be installed?
/usr/include/package_name/, /usr/lib/python/package_name/include, or
somewhere else?

Is there some place that cython looks at for those .pyx and .pxd
files? Is there some consensus/standard for this?

Ondrej

From robertwb at math.washington.edu  Tue Feb 17 01:37:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Feb 2009 16:37:38 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
Message-ID: <7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>

They should be installed in the tree alongside the .py files, and  
will be discovered just like normal Python modules.

On Feb 16, 2009, at 3:16 PM, Ondrej Certik wrote:

> Hi,
>
> where should the .pxd files for a package be installed?
> /usr/include/package_name/, /usr/lib/python/package_name/include, or
> somewhere else?
>
> Is there some place that cython looks at for those .pyx and .pxd
> files? Is there some consensus/standard for this?
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ondrej at certik.cz  Tue Feb 17 01:52:09 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 16:52:09 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
Message-ID: <85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>

On Mon, Feb 16, 2009 at 4:37 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> They should be installed in the tree alongside the .py files, and
> will be discovered just like normal Python modules.

Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
some C++ library, then it has some .pyx and .pxd files and some .so
files as a result of cythonzing it, then:

C++ .so should go to /usr/lib/
C++ .h files should go to /usr/include
.pxd and other .py files and the .so as a result of cythonizing should
go along the .py files

Ondrej

From ondrej at certik.cz  Tue Feb 17 01:55:41 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 16:55:41 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
	<85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
Message-ID: <85b5c3130902161655i3745e8f9v2d4231bfb4360350@mail.gmail.com>

On Mon, Feb 16, 2009 at 4:52 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Mon, Feb 16, 2009 at 4:37 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> They should be installed in the tree alongside the .py files, and
>> will be discovered just like normal Python modules.
>
> Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
> some C++ library, then it has some .pyx and .pxd files and some .so
> files as a result of cythonzing it, then:
>
> C++ .so should go to /usr/lib/
> C++ .h files should go to /usr/include
> .pxd and other .py files and the .so as a result of cythonizing should
> go along the .py files

I am asking because I am using cmake, I made it work with Cython and
numpy nicely and I like it. When I get more experience with it, I post
here the files for other people to use.

My repository is here, see the master-ondrej branch:

http://github.com/certik/hermes2d/tree/master-ondrej


Ondrej

From ellisonbg.net at gmail.com  Tue Feb 17 02:04:42 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 16 Feb 2009 17:04:42 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
	<85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
Message-ID: <6ce0ac130902161704wb18549bs446e22337babd47f@mail.gmail.com>

> Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
> some C++ library, then it has some .pyx and .pxd files and some .so
> files as a result of cythonzing it, then:
>
> C++ .so should go to /usr/lib/
> C++ .h files should go to /usr/include

This is only if you want the library available outside of Python.
Otherwise, these things can go with the .py files in the package as
well.

> .pxd and other .py files and the .so as a result of cythonizing should
> go along the .py files

As I understand it, there are two things that are relevant about the
location of the .pxd files:

* When you are building your own project, the .pxd files need to be
"findable" by Cython.  In your case, this finding happens when cmake
is used to build call Cython and build extensions to begin with.

* When external projects want to use your .pxd files in a cimport
statement, they need to know where they are.  For that, it is really
helpful if your package can provide a get_pxd_include() function that
returns the include directory for your  pxd files.  Something like
(for your hermes2d project):

hermes2d.get_pxd_include()

Another issue for your project, is that people who want to link to it
will want to know where the .so and .h files are installed (the actual
C++ library).  Again, I would simply provide top-level functions for
this:

hermes2d.get_lib()
hermes2d.get_include()

With these 3 functions, anyone building Cython package based on your
code can quickly find out where everything is installed.

The only other thing that could complicate this is if there are any
other dependencies that end users need to build against your project.

Cheers,

Brian

ps - I know this is answering more than your initial question and is
slightly off topic for this list.

> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ondrej at certik.cz  Tue Feb 17 02:54:15 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 17:54:15 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <6ce0ac130902161704wb18549bs446e22337babd47f@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
	<85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
	<6ce0ac130902161704wb18549bs446e22337babd47f@mail.gmail.com>
Message-ID: <85b5c3130902161754g31da5aecua049eec933c30ab4@mail.gmail.com>

On Mon, Feb 16, 2009 at 5:04 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
>> some C++ library, then it has some .pyx and .pxd files and some .so
>> files as a result of cythonzing it, then:
>>
>> C++ .so should go to /usr/lib/
>> C++ .h files should go to /usr/include
>
> This is only if you want the library available outside of Python.
> Otherwise, these things can go with the .py files in the package as
> well.

Yes, I think I want my C++ library to behave like a regular C++ library.

>
>> .pxd and other .py files and the .so as a result of cythonizing should
>> go along the .py files
>
> As I understand it, there are two things that are relevant about the
> location of the .pxd files:
>
> * When you are building your own project, the .pxd files need to be
> "findable" by Cython.  In your case, this finding happens when cmake
> is used to build call Cython and build extensions to begin with.

My own project builds fine, there is no problem with that.

>
> * When external projects want to use your .pxd files in a cimport
> statement, they need to know where they are.  For that, it is really
> helpful if your package can provide a get_pxd_include() function that
> returns the include directory for your  pxd files.  Something like
> (for your hermes2d project):
>
> hermes2d.get_pxd_include()
>
> Another issue for your project, is that people who want to link to it
> will want to know where the .so and .h files are installed (the actual
> C++ library).  Again, I would simply provide top-level functions for
> this:
>
> hermes2d.get_lib()
> hermes2d.get_include()
>
> With these 3 functions, anyone building Cython package based on your
> code can quickly find out where everything is installed.

That's a good idea, I'll do that.

> The only other thing that could complicate this is if there are any
> other dependencies that end users need to build against your project.
>
> Cheers,
>
> Brian
>
> ps - I know this is answering more than your initial question and is
> slightly off topic for this list.

Thanks a lot for answering, in fact, this is exactly what I wanted to know. :)

I'll implement the functions above, that way it will be easy for
anyone to build against my project. There should be no other
dependencies, after I fix everything.

Ondrej

From ondrej at certik.cz  Tue Feb 17 05:26:24 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 20:26:24 -0800
Subject: [Cython] how to initialize a class without calling it's constructor
Message-ID: <85b5c3130902162026g7a2bfaa7qe4af928f105b9e02@mail.gmail.com>

Hi,

I have the following class:

cdef class H1Space:

    def __init__(self, Mesh m, H1Shapeset s):
        self.thisptr = new_H1Space(m.thisptr, s.thisptr)

and it is initialized from 2 other classes. Sometimes however, I would
like to initialize it directly from a C++ instance of c_H1Space, e.g.
I would like to create the H1Space class *without* calling it's
constructor and just set the self.thisptr object to the C++ instance.

Currently I am using the following code and everything works for me:


cdef H1Space H1Space_from_c_H1Space(c_H1Space *h):
    cdef H1Space n
    n = <H1Space>PY_NEW(H1Space)
    n.thisptr = h
    return n


But it uses the PY_NEW macro:


#define PY_NEW(zzz_type) \
    (((PyTypeObject*)(zzz_type))->tp_new((PyTypeObject*)(zzz_type),
global_empty_tuple, NULL))

and global_empty_tuple() that I had to initialize, etc. I copied this
from Sage. Is there some better way to do the above, so that I don't
have to include the special macro above and the corresponding .h and
.c files in my project?

Thanks,
Ondrej

From wstein at gmail.com  Tue Feb 17 06:25:30 2009
From: wstein at gmail.com (William Stein)
Date: Mon, 16 Feb 2009 21:25:30 -0800
Subject: [Cython] project web servers
Message-ID: <85e81ba30902162125s6e0e33a7k3743896aa72019ce@mail.gmail.com>

Hi,

I had to migrate all the cython, Sage and mpir websites and related
servers to another machine, due to some hardware failures on the
server that was hosting them.    Please let me know if any of the
cython, Sage, or MPIR websites are suddenly broken.

 -- William

-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From wstein at gmail.com  Tue Feb 17 07:39:05 2009
From: wstein at gmail.com (William Stein)
Date: Mon, 16 Feb 2009 22:39:05 -0800
Subject: [Cython] [sage-devel] Re: project web servers
In-Reply-To: <ee247281-3b84-4c3b-82c0-e82dfcd330d1@d32g2000yqe.googlegroups.com>
References: <85e81ba30902162125s6e0e33a7k3743896aa72019ce@mail.gmail.com>
	<ee247281-3b84-4c3b-82c0-e82dfcd330d1@d32g2000yqe.googlegroups.com>
Message-ID: <85e81ba30902162239j14b1fc00o4b83fc6f85d308c1@mail.gmail.com>

On Mon, Feb 16, 2009 at 9:45 PM, mabshoff <mabshoff at googlemail.com> wrote:
>
>
>
> On Feb 16, 9:25 pm, William Stein <wst... at gmail.com> wrote:
>> Hi,
>>
>> I had to migrate all the cython, Sage and mpir websites and related
>> servers to another machine, due to some hardware failures on the
>> server that was hosting them.    Please let me know if any of the
>> cython, Sage, or MPIR websites are suddenly broken.
>
> Attaching sub-1kb patches is broken in Sage's trac with
>
>
> [9:39pm] mabs: wstein2: I cannot attach patches to trac all the
> sudden. It worked 10 minutes ago:
> [9:39pm] mabs: Traceback (most recent call last):
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/web/main.py", line 387, in
> dispatch_request
> [9:39pm] mabs:     dispatcher.dispatch(req)
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/web/main.py", line 237, in dispatch
> [9:39pm] mabs:     resp = chosen_handler.process_request(req)
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/attachment.py", line 361, in
> process_request
> [9:39pm] mabs:     self._do_save(req, attachment)
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/attachment.py", line 450, in _do_save
> [9:39pm] mabs:     size = upload.file.len
>
> I tested and as soon as a patch is larger than 1023 bytes it works
> again.
>

Thanks for pointing this out.

I've upgraded the trac servers for sage, cython, and mpir all to
trac-0.11.3.  Again, let me know if you have any problems as a result.

Note that trac-0.11* has much better support for "workflows", but I
haven't done anything with that -- I just migrated the old installs.
Mike Hansen will likely incorporate better workflow support, at least
for the sage project.

William

From robertwb at math.washington.edu  Tue Feb 17 10:32:30 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 01:32:30 -0800
Subject: [Cython] how to initialize a class without calling it's
	constructor
In-Reply-To: <85b5c3130902162026g7a2bfaa7qe4af928f105b9e02@mail.gmail.com>
References: <85b5c3130902162026g7a2bfaa7qe4af928f105b9e02@mail.gmail.com>
Message-ID: <26B8DFAA-3CB3-4D73-9735-32709CA76C24@math.washington.edu>

On Feb 16, 2009, at 8:26 PM, Ondrej Certik wrote:

> Currently I am using the following code and everything works for me:

[...]

> But it uses the PY_NEW macro:
>
> #define PY_NEW(zzz_type) \
>     (((PyTypeObject*)(zzz_type))->tp_new((PyTypeObject*)(zzz_type),
> global_empty_tuple, NULL))
>
> and global_empty_tuple() that I had to initialize, etc. I copied this
> from Sage. Is there some better way to do the above, so that I don't
> have to include the special macro above and the corresponding .h and
> .c files in my project?

How about

cdef extern from *:
     ctypedef struct RichPyTypeObject "PyTypeObject":
         object tp_new(object, object, PyObject*)

cdef inline PY_NEW(T):
     return (<RichPyTypeObject*>x).tp_new(T, (), NULL)


- Robert


From robertwb at math.washington.edu  Tue Feb 17 10:47:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 01:47:25 -0800
Subject: [Cython] function declaration problem
In-Reply-To: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
References: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
Message-ID: <3D9D09CD-1214-46ED-8D24-B25F40C91F98@math.washington.edu>

On Feb 14, 2009, at 8:14 AM, Jean-Francois Moulin wrote:

> Hi,
>
> my first, probably very naive, question to the list.... I want to
> create an extension class with a method whose definition changes as a
> function of some test
> at the time of creation of the instance.
> Something like:
>
> cdef class Test:
>
>     cdef int x, foo

I don't think this does what you think it does--it makes Test have  
two attributes "x" and "foo"

>
>     def __init__(self,foo):
>
>         if foo>4:
>             self.f = self.fn1
>         else:
>             self.f = self.fn2
>
>     cdef fn1(self,x):
>         return x
>
>     cdef fn2(self,x):
>         return 2 * x
>
>
> Is this possible, and if yes, how should I declare f?

Declare it as a c function pointer


cdef class Test:

     cdef int (*f)(Test, int)

     def __init__(self, foo):

         if foo > 4:
             self.f = self.fn1
         else:
             self.f = self.fn2

     cdef int fn1(self, int x):
         return x

     cdef int fn2(self, int x):
         return 2 * x

     def test_f(self, x):
         return self.f(self, x)

 >>> a = Test(1)
 >>> a.test_f(10)
20
 >>> a = Test(100)
 >>> a.test_f(10)
10

- Robert

From robertwb at math.washington.edu  Tue Feb 17 10:56:48 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 01:56:48 -0800
Subject: [Cython] import/cimport gotcha
In-Reply-To: <e7ba66e40902130752j453000b5yc68c9d74c11f1f4e@mail.gmail.com>
References: <e7ba66e40902130752j453000b5yc68c9d74c11f1f4e@mail.gmail.com>
Message-ID: <1BBB9640-C0CE-41F5-82DE-25F4E031E03C@math.washington.edu>

On Feb 13, 2009, at 7:52 AM, Lisandro Dalcin wrote:

> Some time ago I reported that the following:
>
> from foo import Bar
> from foo cimport Bar
>
> do not put 'Bar' in the module dicti. Commenting the lines below seems
> to solve the issue. Am I missing something (I bet the answer is yes)?
> Any comments?

Hmm... those lines were added in changeset 1309 because of issues  
with compiling .py files. I think it may change the name to be an  
ordinary name, rather than a cimported one, but I'm not sure.

Does Cython compile itself after this patch? Also, rather than  
commenting stuff out, if it's not needed it should just be deleted.

> diff -r 1771db44498b Cython/Compiler/Nodes.py
> --- a/Cython/Compiler/Nodes.py	Thu Feb 12 18:22:15 2009 -0200
> +++ b/Cython/Compiler/Nodes.py	Fri Feb 13 13:42:23 2009 -0200
> @@ -4600,8 +4600,9 @@
>                          break
>              else:
>                  entry =  env.lookup(target.name)
> -                if entry.is_type and entry.type.name == name and
> entry.type.module_name == self.module.module_name.value:
> -                    continue # already cimported
> +                #if entry.is_type and entry.type.name == name and
> entry.type.module_name == self.module.module_name.value:
> +                #    continue # already cimported
> +                #    pass
>                  self.interned_items.append(
>                      (env.intern_identifier(name), target))
>                  target.analyse_target_expression(env, None)
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From jf.moulin at gmail.com  Tue Feb 17 11:01:40 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 17 Feb 2009 11:01:40 +0100
Subject: [Cython] function declaration problem
In-Reply-To: <3D9D09CD-1214-46ED-8D24-B25F40C91F98@math.washington.edu>
References: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
	<3D9D09CD-1214-46ED-8D24-B25F40C91F98@math.washington.edu>
Message-ID: <af6d35810902170201x446d55f7u90b29f641a6aed66@mail.gmail.com>

Thanks a bunch for the informative answers!
One of these days I'll have to read a bit about C, for sure. I've been
spoiled by python ;0)
JF

From robertwb at math.washington.edu  Tue Feb 17 11:04:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 02:04:28 -0800
Subject: [Cython] Incomplete C++ example
In-Reply-To: <9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>
	<9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
Message-ID: <B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>

On Feb 12, 2009, at 8:40 AM, St?fan van der Walt wrote:

> Hi all,
>
> I'm helping a student to wrap some C++ code using Cython, and noticed
> that the example in the documentation is incomplete.  A header file,
> Rectangle.h, is provided, but no implementation is given, so the
> example cannot be compiled.
>
> It would be great if you could add an implementation (see attached)

There is already one attached (see the bottom of the page) but it  
should probably be made more prominent.

> and modify the setup.py accordingly, i.e.
>
>    Extension(
>    "rectangle",
>    ["rectangle.pyx", "Rectangle.cc"],
>    language="c++")


The example setup.py assumes Rectangle is part of an externally  
linked library (which is the most common use case). However, the  
example Setup.py is more explicit. It is a wiki, so it would be nice  
if someone (you?) improved the presentation.

- Robert


From dagss at student.matnat.uio.no  Tue Feb 17 11:23:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 17 Feb 2009 11:23:40 +0100
Subject: [Cython] Incomplete C++ example
In-Reply-To: <B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>	<9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
	<B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>
Message-ID: <499A902C.3060604@student.matnat.uio.no>

Robert Bradshaw wrote:
> The example setup.py assumes Rectangle is part of an externally  
> linked library (which is the most common use case). However, the  
> example Setup.py is more explicit. It is a wiki, so it would be nice  
> if someone (you?) improved the presentation.

I think Gabrial has also pulled it into the main documentation though, 
where the attachment is not present:

http://docs.cython.org/docs/wrapping_CPlusPlus.html


-- 
Dag Sverre

From robertwb at math.washington.edu  Tue Feb 17 11:15:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 02:15:04 -0800
Subject: [Cython] Incomplete C++ example
In-Reply-To: <499A902C.3060604@student.matnat.uio.no>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>	<9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
	<B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>
	<499A902C.3060604@student.matnat.uio.no>
Message-ID: <E67CA6BB-8012-45A2-822D-2FF5E4C4A938@math.washington.edu>

On Feb 17, 2009, at 2:23 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> The example setup.py assumes Rectangle is part of an externally
>> linked library (which is the most common use case). However, the
>> example Setup.py is more explicit. It is a wiki, so it would be nice
>> if someone (you?) improved the presentation.
>
> I think Gabrial has also pulled it into the main documentation though,
> where the attachment is not present:
>
> http://docs.cython.org/docs/wrapping_CPlusPlus.html

Ah, good point. Yes, we need to have it there.

- Robert


From dalcinl at gmail.com  Wed Feb 18 21:39:50 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 18 Feb 2009 18:39:50 -0200
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
Message-ID: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>

Supose this fragment:

cdef extern from *:
    ctypedef long Py_ssize_t
    ctypedef unsigned long size_t

With this patch, these ctypedef's seems to be just ignored. This is
good for the sake of Cython-backward and Pyrex compatibility. However,
it would be nice to make Cython generate a warning. But the right
place to do that is not at the point of parsing basic C types, but on
parsing a ctypedef statement itself.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SPECIAL_C_TYPES.diff
Type: application/octet-stream
Size: 2943 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090218/fbdf4935/attachment.obj 

From robertwb at math.washington.edu  Thu Feb 19 09:22:10 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 00:22:10 -0800
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
Message-ID: <ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>

I think it makes sense to include these as basic_c_type_names, the  
same as long and char. However, I'm not to particular on the  
implementation. They should not be able to accept modifiers.

In terms of the behavior for code below, I agree an warning should be  
emitted and the ctypedef ignored.

- Robert


On Feb 18, 2009, at 12:39 PM, Lisandro Dalcin wrote:

> Supose this fragment:
>
> cdef extern from *:
>     ctypedef long Py_ssize_t
>     ctypedef unsigned long size_t
>
> With this patch, these ctypedef's seems to be just ignored. This is
> good for the sake of Cython-backward and Pyrex compatibility. However,
> it would be nice to make Cython generate a warning. But the right
> place to do that is not at the point of parsing basic C types, but on
> parsing a ctypedef statement itself.
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0) 
> 342-451.1594<SPECIAL_C_TYPES.diff>____________________________________ 
> ___________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Thu Feb 19 13:03:07 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 19 Feb 2009 10:03:07 -0200
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
Message-ID: <e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>

On Thu, Feb 19, 2009 at 6:22 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> I think it makes sense to include these as basic_c_type_names, the
> same as long and char. However, I'm not to particular on the
> implementation. They should not be able to accept modifiers.

Well, if Py_ssize_t and size_t are listed in basic_c_types_names, I
cannot see a simple way to make the parser reject a modifier. That's
the reason I want to treat them in a slightly special fashion.

BTW, did you mean 'int' instead of 'long'? AFAIK, 'long' is actually
'signed long int'.

> In terms of the behavior for code below, I agree an warning should be
> emitted and the ctypedef ignored.
>

Then I'll try to make this work, but this would still need to
special-case when parsing basic C types. If not, the mix of
<modifiers> <Py_ssize_t | size_t> get consumed and a the ctypedef
parsing will fail because of a empty typedef name (the same if you
wrote "typedef double" instead of "typedef double MyDouble").


Just a question: if the ctypedef is ignored, what should be the return
of p_ctypedef_statement() ??? None? a PassNode?



> - Robert
>
>
> On Feb 18, 2009, at 12:39 PM, Lisandro Dalcin wrote:
>
>> Supose this fragment:
>>
>> cdef extern from *:
>>     ctypedef long Py_ssize_t
>>     ctypedef unsigned long size_t
>>
>> With this patch, these ctypedef's seems to be just ignored. This is
>> good for the sake of Cython-backward and Pyrex compatibility. However,
>> it would be nice to make Cython generate a warning. But the right
>> place to do that is not at the point of parsing basic C types, but on
>> parsing a ctypedef statement itself.
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)
>> 342-451.1594<SPECIAL_C_TYPES.diff>____________________________________
>> ___________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Feb 19 13:29:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Feb 2009 13:29:40 +0100 (CET)
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
	<e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>
Message-ID: <43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> Then I'll try to make this work, but this would still need to
> special-case when parsing basic C types. If not, the mix of
> <modifiers> <Py_ssize_t | size_t> get consumed and a the ctypedef
> parsing will fail because of a empty typedef name (the same if you
> wrote "typedef double" instead of "typedef double MyDouble").

You can let the parser parse them as usual and once that succeeded, have
it check for the two names to drop the declaration with a warning. From my
side, it's ok to do that in the parser.


> Just a question: if the ctypedef is ignored, what should be the return
> of p_ctypedef_statement() ??? None? a PassNode?

A PassNode should be safe and allowed wherever we allow a ctypedef statement.

Stefan


From dalcinl at gmail.com  Thu Feb 19 14:44:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 19 Feb 2009 11:44:25 -0200
Subject: [Cython] I cannot push
Message-ID: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>

Yesterdey I was getting 404 errors and then changed the repo url in
.hg/hgrc. Is this the right url to push? Or perhaps my http password
was not added to the new server?

$ hg push
pushing to http://hg.cython.org/cython-devel
searching for changes
abort: authorization failed


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From wstein at gmail.com  Thu Feb 19 20:46:07 2009
From: wstein at gmail.com (William Stein)
Date: Thu, 19 Feb 2009 11:46:07 -0800
Subject: [Cython] I cannot push
In-Reply-To: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>
Message-ID: <85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>

On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Yesterdey I was getting 404 errors and then changed the repo url in
> .hg/hgrc. Is this the right url to push? Or perhaps my http password
> was not added to the new server?
>
> $ hg push
> pushing to http://hg.cython.org/cython-devel
> searching for changes
> abort: authorization failed

Can you try again now?

William

>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From robertwb at math.washington.edu  Thu Feb 19 21:03:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 12:03:04 -0800
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
	<e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>
	<43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <D528A6E4-FD4F-4FA8-A5A6-AFC85F940679@math.washington.edu>

On Feb 19, 2009, at 4:29 AM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> Then I'll try to make this work, but this would still need to
>> special-case when parsing basic C types. If not, the mix of
>> <modifiers> <Py_ssize_t | size_t> get consumed and a the ctypedef
>> parsing will fail because of a empty typedef name (the same if you
>> wrote "typedef double" instead of "typedef double MyDouble").
>
> You can let the parser parse them as usual and once that succeeded,  
> have
> it check for the two names to drop the declaration with a warning.  
> From my
> side, it's ok to do that in the parser.
>
>> Just a question: if the ctypedef is ignored, what should be the  
>> return
>> of p_ctypedef_statement() ??? None? a PassNode?
>
> A PassNode should be safe and allowed wherever we allow a ctypedef  
> statement.

I am actually in favor of moving the logic out of the parser as much  
as possible. For example, I think the logic for rejecting "signed  
short size_t" belongs in PyrexTypes.simple_c_type, and raising an  
warning on "ctypedef int size_t" belongs in  
CTypeDefNode.analyse_declarations.

Does anyone else find it odd "ctypedef int short" is OK?

- Robert


From robertwb at math.washington.edu  Thu Feb 19 21:06:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 12:06:24 -0800
Subject: [Cython] I cannot push
In-Reply-To: <85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>
	<85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
Message-ID: <58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>

On Feb 19, 2009, at 11:46 AM, William Stein wrote:

> On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin  
> <dalcinl at gmail.com> wrote:
>> Yesterdey I was getting 404 errors and then changed the repo url in
>> .hg/hgrc. Is this the right url to push? Or perhaps my http password
>> was not added to the new server?
>>
>> $ hg push
>> pushing to http://hg.cython.org/cython-devel
>> searching for changes
>> abort: authorization failed
>
> Can you try again now?

The permissions on sage-devel/.hg didn't copy over. I fixed them, it  
should work now.

- Robert


From stefan_ml at behnel.de  Thu Feb 19 21:08:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Feb 2009 21:08:11 +0100
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <D528A6E4-FD4F-4FA8-A5A6-AFC85F940679@math.washington.edu>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>	<e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>	<43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<D528A6E4-FD4F-4FA8-A5A6-AFC85F940679@math.washington.edu>
Message-ID: <499DBC2B.1080009@behnel.de>


Robert Bradshaw wrote:
> Does anyone else find it odd "ctypedef int short" is OK?

Ah, good point. Fine, let's make that a general feature.

Stefan


From dagss at student.matnat.uio.no  Fri Feb 20 01:56:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 01:56:06 +0100
Subject: [Cython] I cannot push
In-Reply-To: <58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>	<85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
	<58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>
Message-ID: <499DFFA6.2090202@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 19, 2009, at 11:46 AM, William Stein wrote:
> 
>> On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin  
>> <dalcinl at gmail.com> wrote:
>>> Yesterdey I was getting 404 errors and then changed the repo url in
>>> .hg/hgrc. Is this the right url to push? Or perhaps my http password
>>> was not added to the new server?
>>>
>>> $ hg push
>>> pushing to http://hg.cython.org/cython-devel
>>> searching for changes
>>> abort: authorization failed
>> Can you try again now?
> 
> The permissions on sage-devel/.hg didn't copy over. I fixed them, it  
> should work now.

I still cannot push to either cython-devel or cython-0.11.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 01:59:03 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 01:59:03 +0100
Subject: [Cython] Status for 0.11
Message-ID: <499E0057.9040504@student.matnat.uio.no>

I have a fix for #200 ready, waiting to be pushed when I have access. 
(I'll attach it to the ticket shortly as well.)

#143: I could take upon me to disable the current method without 
introducing a new one within a week. Robert, do you want me to do that, 
or do you have plans to do it properly?

#106 can be moved to 0.11.1 as Robert says.

That should leave running the refnanny on lxml and Sage before one can 
do a release(?). Has there been any attempts on that? I may also be able 
to spend some time (not too much) on refnannying Sage if Robert hasn't 
started on that (if so, I'd be grateful to know how I can add 
CYTHON_REFNANNY to CFLAGS in the Sage build easily)

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Feb 20 02:51:35 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 17:51:35 -0800
Subject: [Cython] I cannot push
In-Reply-To: <499DFFA6.2090202@student.matnat.uio.no>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>	<85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
	<58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>
	<499DFFA6.2090202@student.matnat.uio.no>
Message-ID: <5089743F-4665-4EA3-985C-53D9E7B90F56@math.washington.edu>

On Feb 19, 2009, at 4:56 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 19, 2009, at 11:46 AM, William Stein wrote:
>>
>>> On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin
>>> <dalcinl at gmail.com> wrote:
>>>> Yesterdey I was getting 404 errors and then changed the repo url in
>>>> .hg/hgrc. Is this the right url to push? Or perhaps my http  
>>>> password
>>>> was not added to the new server?
>>>>
>>>> $ hg push
>>>> pushing to http://hg.cython.org/cython-devel
>>>> searching for changes
>>>> abort: authorization failed
>>> Can you try again now?
>>
>> The permissions on sage-devel/.hg didn't copy over. I fixed them, it
>> should work now.
>
> I still cannot push to either cython-devel or cython-0.11.

Me either ;).

OK, it should be good now.

- Robert



From robertwb at math.washington.edu  Fri Feb 20 02:58:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 17:58:03 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E0057.9040504@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>
Message-ID: <98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>

On Feb 19, 2009, at 4:59 PM, Dag Sverre Seljebotn wrote:

> I have a fix for #200 ready, waiting to be pushed when I have access.
> (I'll attach it to the ticket shortly as well.)

OK, I pushed this. I'm going to do more testing before I close it.

> #143: I could take upon me to disable the current method without
> introducing a new one within a week. Robert, do you want me to do  
> that,
> or do you have plans to do it properly?



> #106 can be moved to 0.11.1 as Robert says.

Moved.

> That should leave running the refnanny on lxml and Sage before one can
> do a release(?). Has there been any attempts on that? I may also be  
> able
> to spend some time (not too much) on refnannying Sage if Robert hasn't
> started on that (if so, I'd be grateful to know how I can add
> CYTHON_REFNANNY to CFLAGS in the Sage build easily)

I haven't done that yet. I'll give it a go and let you know if  
anything comes up (I can do the Sage side of things easily...).

- Robert


From michael.abshoff at googlemail.com  Fri Feb 20 05:17:57 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 19 Feb 2009 20:17:57 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>
	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>
Message-ID: <499E2EF5.2090907@gmail.com>

Robert Bradshaw wrote:
> On Feb 19, 2009, at 4:59 PM, Dag Sverre Seljebotn wrote:

<SNIP>

Dad, Robert,

>> That should leave running the refnanny on lxml and Sage before one can
>> do a release(?). Has there been any attempts on that? I may also be  
>> able
>> to spend some time (not too much) on refnannying Sage if Robert hasn't
>> started on that (if so, I'd be grateful to know how I can add
>> CYTHON_REFNANNY to CFLAGS in the Sage build easily)
> 
> I haven't done that yet. I'll give it a go and let you know if  
> anything comes up (I can do the Sage side of things easily...).

I got busy with other things, so refresh my mind: All I need to do is to 
add

  -DCYTHON_REFNANNY

to CFLAGS and after rebuilding the library run the test suite?


> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb 20 06:33:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 21:33:39 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E2EF5.2090907@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>
	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>
	<499E2EF5.2090907@gmail.com>
Message-ID: <4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>

On Feb 19, 2009, at 8:17 PM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 19, 2009, at 4:59 PM, Dag Sverre Seljebotn wrote:
>
> <SNIP>
>
> Dad, Robert,
>
>>> That should leave running the refnanny on lxml and Sage before  
>>> one can
>>> do a release(?). Has there been any attempts on that? I may also be
>>> able
>>> to spend some time (not too much) on refnannying Sage if Robert  
>>> hasn't
>>> started on that (if so, I'd be grateful to know how I can add
>>> CYTHON_REFNANNY to CFLAGS in the Sage build easily)
>>
>> I haven't done that yet. I'll give it a go and let you know if
>> anything comes up (I can do the Sage side of things easily...).
>
> I got busy with other things, so refresh my mind: All I need to do  
> is to
> add
>
>   -DCYTHON_REFNANNY
>
> to CFLAGS and after rebuilding the library run the test suite?

Yes, I think that would do the trick. You can also use Extension(...,  
define_macros=[('CYTHON_REFNANNY', None)]) in setup.py, but that'd  
probably be less simple.

- Robert


From michael.abshoff at googlemail.com  Fri Feb 20 06:47:07 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 19 Feb 2009 21:47:07 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>	<499E2EF5.2090907@gmail.com>
	<4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>
Message-ID: <499E43DB.4070702@gmail.com>

Robert Bradshaw wrote:
> On Feb 19, 2009, at 8:17 PM, Michael Abshoff wrote:

<SNIP>

Hi Robert,

>> I got busy with other things, so refresh my mind: All I need to do  
>> is to
>> add
>>
>>   -DCYTHON_REFNANNY
>>
>> to CFLAGS and after rebuilding the library run the test suite?
> 
> Yes, I think that would do the trick. You can also use Extension(...,  
> define_macros=[('CYTHON_REFNANNY', None)]) in setup.py, but that'd  
> probably be less simple.

Well, I will just patch it into the Python CFLAGS :). But a clean, 
on-demand solution in setup.py would be the way to go. I am not known 
for my elegant coding, so I will someone else do that.

Do you have a recent cython.spkg to test or should I just pull from devel?

> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb 20 06:49:20 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 21:49:20 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E43DB.4070702@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>	<499E2EF5.2090907@gmail.com>
	<4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>
	<499E43DB.4070702@gmail.com>
Message-ID: <498CB5D1-3809-45F6-9351-185ED90CFCF2@math.washington.edu>

On Feb 19, 2009, at 9:47 PM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 19, 2009, at 8:17 PM, Michael Abshoff wrote:
>
> <SNIP>
>
> Hi Robert,
>
>>> I got busy with other things, so refresh my mind: All I need to do
>>> is to
>>> add
>>>
>>>   -DCYTHON_REFNANNY
>>>
>>> to CFLAGS and after rebuilding the library run the test suite?
>>
>> Yes, I think that would do the trick. You can also use Extension(...,
>> define_macros=[('CYTHON_REFNANNY', None)]) in setup.py, but that'd
>> probably be less simple.
>
> Well, I will just patch it into the Python CFLAGS :). But a clean,
> on-demand solution in setup.py would be the way to go. I am not known
> for my elegant coding, so I will someone else do that.

Yeah, I think we iterate over extensions to add gmp, etc. to the  
libraries anyways.

> Do you have a recent cython.spkg to test or should I just pull from  
> devel?

Just pull from devel.

- Robert

From stefan_ml at behnel.de  Fri Feb 20 08:33:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 08:33:50 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E0057.9040504@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>
Message-ID: <499E5CDE.6000804@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> #106 can be moved to 0.11.1 as Robert says.

+1. Could anyone still give some feedback if this still applies to current
cython-devel?


> That should leave running the refnanny on lxml and Sage before one can 
> do a release(?). Has there been any attempts on that?

I ran lxml's test suite with the refnanny and it looks mostly good, except
for a number of dropped AssertErrors and dropped unnamed(?) exceptions. It
doesn't say where they come from and I was unable to add any traceable
information to the output. There are two plain assert statements in the
refnanny code, but adding text to them doesn't change anything.

I'll post the output here as soon as I get to it.

Stefan

From stefan_ml at behnel.de  Fri Feb 20 08:42:21 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 08:42:21 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E5CDE.6000804@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de>
Message-ID: <499E5EDD.2070800@behnel.de>


Stefan Behnel wrote:
> I ran lxml's test suite with the refnanny and it looks mostly good, except
> for a number of dropped AssertErrors and dropped unnamed(?) exceptions. It
> doesn't say where they come from and I was unable to add any traceable
> information to the output. There are two plain assert statements in the
> refnanny code, but adding text to them doesn't change anything.
> 
> I'll post the output here as soon as I get to it.

Quicker than I though. Looks like only two separate problems. There are two
'void' methods in _ParserContext that do not propagate exceptions, but
enabling that doesn't change anything. Any idea what this might mean?

Stefan



Exception AssertionError in <lxml.etree._ParserContext object at 0x96010d4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960109c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9601064>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960102c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621fb4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621f7c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621f44>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621e9c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621e64>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621f0c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621ed4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621e2c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621df4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621dbc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621d84>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621d4c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621d14>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621bc4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621bfc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621c34>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9623684>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960efb4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ef7c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ef44>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ef0c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eed4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ee9c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ee64>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ee2c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960edf4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960edbc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ed84>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ed4c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ed14>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ecdc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eca4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ec6c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ec34>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ebfc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ebc4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eb8c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eb54>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eb1c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eae4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eaac>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ea74>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ea3c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ea04>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e9cc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e994>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e95c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e924>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e8ec>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e8b4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e87c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e844>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e684>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e41c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e144>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e0d4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e09c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f6f4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f80c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f4c4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f684>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987fcdc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f6bc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f56c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f79c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987fdf4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f614>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f5dc>
ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x94426bc>
ignored


From dagss at student.matnat.uio.no  Fri Feb 20 09:45:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 09:45:30 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E5EDD.2070800@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>
Message-ID: <499E6DAA.7020209@student.matnat.uio.no>

Stefan Behnel wrote:
> Stefan Behnel wrote:
>> I ran lxml's test suite with the refnanny and it looks mostly good, except
>> for a number of dropped AssertErrors and dropped unnamed(?) exceptions. It
>> doesn't say where they come from and I was unable to add any traceable
>> information to the output. There are two plain assert statements in the
>> refnanny code, but adding text to them doesn't change anything.
>>
>> I'll post the output here as soon as I get to it.
> 
> Quicker than I though. Looks like only two separate problems. There are two
> 'void' methods in _ParserContext that do not propagate exceptions, but
> enabling that doesn't change anything. Any idea what this might mean?

Robert, Michael, this applies to you as well.

I think I have an idea. The refnanny raises normal exceptions when 
something is violated; this is great for the smaller tests in the Cython 
test suite but might become a problem in programs.

There might be more situations where what you see happens, but this is 
one I can think of:

def f():
     # this function raises AssertError, and then
     # during exit of the function, a this exception is replaced
     # (overwritten) with a refnanny exception

try:
     f()
except:
     pass

The solution may be to change the way Cython/Runtime/refnanny.pyx 
reports errors, for instance change Context.delref to do a 
print/sys.exit(1) thing instead of raising an exception, and then 
reinstall Cython.

(Note that in f() above, the refcount error could happen before the 
AssertError is raised -- refnanny.pyx keeps track of all errors that 
happen in a function, and then raises/replaces an exception on function 
return.)

-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Feb 20 10:25:36 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 10:25:36 +0100 (CET)
Subject: [Cython] Status for 0.11
In-Reply-To: <499E6DAA.7020209@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
Message-ID: <38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> The solution may be to change the way Cython/Runtime/refnanny.pyx
> reports errors, for instance change Context.delref to do a
> print/sys.exit(1) thing instead of raising an exception

I would prefer a simple sys.__stderr__.write() with some helpful
information in a dedicated code line that you could point gdb to. Killing
the process sounds everything but helpful. Note that sys.exit() also just
raises an exception, as opposed to os._exit().

Stefan


From robertwb at math.washington.edu  Fri Feb 20 10:31:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 01:31:43 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>

On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:

> Dag Sverre Seljebotn wrote:
>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>> reports errors, for instance change Context.delref to do a
>> print/sys.exit(1) thing instead of raising an exception
>
> I would prefer a simple sys.__stderr__.write() with some helpful
> information in a dedicated code line that you could point gdb to.  
> Killing
> the process sounds everything but helpful. Note that sys.exit()  
> also just
> raises an exception, as opposed to os._exit().

+1. If memory is leaking, I'd rather have it tell me and move on. Of  
course if there are reference errors later on it may segfault, but at  
least I'd have useful information already.

- Robert



From dagss at student.matnat.uio.no  Fri Feb 20 10:50:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 10:50:27 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <499E7CE3.7010302@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>> reports errors, for instance change Context.delref to do a
>> print/sys.exit(1) thing instead of raising an exception
> 
> I would prefer a simple sys.__stderr__.write() with some helpful
> information in a dedicated code line that you could point gdb to. Killing
> the process sounds everything but helpful. Note that sys.exit() also just
> raises an exception, as opposed to os._exit().

I was referring to doing this for lxml and Sage testing only -- in which 
case you get to do whatever you like :-) So just go ahead and don't use 
sys.exit(), it was just an example, so that you could figure out what is 
going on, I didn't mean to change it like this permanently.

I would like to continue with the current behaviour for the Cython test 
suite, in order to get the nice test failures (though if you think it is 
a problem, one could have runtests.py inspect global variable in the 
refnanny module instead).

One could always have refnanny.pyx check sys.environ for some variable 
to see whether it should print to stderr or raise an exception? Or it 
could be controlled by a global variable in refnanny which runtests.py 
could change...


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 10:53:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 10:53:15 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E7CE3.7010302@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<499E7CE3.7010302@student.matnat.uio.no>
Message-ID: <499E7D8B.3060800@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>> reports errors, for instance change Context.delref to do a
>>> print/sys.exit(1) thing instead of raising an exception
>> I would prefer a simple sys.__stderr__.write() with some helpful
>> information in a dedicated code line that you could point gdb to. Killing
>> the process sounds everything but helpful. Note that sys.exit() also just
>> raises an exception, as opposed to os._exit().
> 
> I was referring to doing this for lxml and Sage testing only -- in which 
> case you get to do whatever you like :-) So just go ahead and don't use 

Huhhmm. Actually, you get to do whatever you like with anything in 
Cython. But you know what I meant... :-)

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Fri Feb 20 10:59:11 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 01:59:11 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
Message-ID: <499E7EEF.2020304@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>> reports errors, for instance change Context.delref to do a
>>> print/sys.exit(1) thing instead of raising an exception
>> I would prefer a simple sys.__stderr__.write() with some helpful
>> information in a dedicated code line that you could point gdb to.  
>> Killing
>> the process sounds everything but helpful. Note that sys.exit()  
>> also just
>> raises an exception, as opposed to os._exit().
> 
> +1. If memory is leaking, I'd rather have it tell me and move on. Of  
> course if there are reference errors later on it may segfault, but at  
> least I'd have useful information already.

Not surprisingly we get bitten by the widely discussed ctypedef sizt_t 
thingy:

# Memory management
cdef extern from "stdlib.h":
     ctypedef unsigned long size_t


> - Robert

Before I go off and make a fool of myself fixing various potential 
Cython issues in 3.3.rc2: Robert - do you have a patch or a list of 
known issues? If the above is the only problem I can probably handle it :)

Cheers,

Michael

> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From michael.abshoff at googlemail.com  Fri Feb 20 11:11:35 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 02:11:35 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
Message-ID: <499E81D7.4000906@gmail.com>

Ok, after commenting out all the various size_t declarations (in *5* 
places) I am stuck with this:

python2.5 `which cython` --embed-positions --incref-local-binop 
-I/scratch/mabshoff/cython-0.11-nanny/sage-3.3.rc2-sage.math-only-x86_64-Linux/devel/sage-main 
-o sage/matrix/matrix_integer_dense.c sage/matrix/matrix_integer_dense.pyx

Error converting Pyrex file to C:
------------------------------------------------------------
...
                 raise MemoryError, "out of memory allocating 
multi-modular coefficent list"
             for j from 0 <= j < self._ncols:
                 matrix[i][j] = mpz_fdiv_ui(self._matrix[i][j], n)

         cdef Linbox_modn_dense L = Linbox_modn_dense()
         L.set(n, matrix, self._nrows, self._ncols)
                       ^
------------------------------------------------------------

/scratch/mabshoff/cython-0.11-nanny/sage-3.3.rc2-sage.math-only-x86_64-Linux/devel/sage-main/sage/matrix/matrix_integer_dense.pyx:1161:23: 
Cannot assign type 'sage.matrix.matrix_integer_dense.mod_int **' to 
'sage.libs.linbox.linbox.mod_int **'


But I guess at this point this should move off the Cython list :)

Cheers,

Michael

PS: my patch so far, but I might have done something very stupid to 
cause the above issue :)


diff -r 43f267532b7a sage/combinat/matrices/dancing_links.pyx
--- a/sage/combinat/matrices/dancing_links.pyx	Tue Feb 17 17:00:05 2009 
-0800
+++ b/sage/combinat/matrices/dancing_links.pyx	Fri Feb 20 02:09:18 2009 
-0800
@@ -25,8 +25,8 @@
  include "../../ext/python_int.pxi"
  include "../../ext/python_ref.pxi"

-cdef extern from "stdlib.h":
-     ctypedef unsigned long size_t
+#cdef extern from "stdlib.h":
+#     ctypedef unsigned long size_t

  cdef extern from "dancing_links_c.h":
      ctypedef struct vector_int "std::vector<int>":
diff -r 43f267532b7a sage/ext/cdefs.pxi
--- a/sage/ext/cdefs.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/ext/cdefs.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -2,7 +2,7 @@
  include "python.pxi"

  cdef extern from "stdlib.h":
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t
      void free(void *ptr)
      void *malloc(size_t size)
      void *realloc(void *ptr, size_t size)
diff -r 43f267532b7a sage/ext/python_mem.pxi
--- a/sage/ext/python_mem.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/ext/python_mem.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -1,5 +1,5 @@
  cdef extern from "Python.h":
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t

      #####################################################################
      # 9.2 Memory Interface
diff -r 43f267532b7a sage/ext/stdsage.pxi
--- a/sage/ext/stdsage.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/ext/stdsage.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -39,8 +39,8 @@


  # Memory management
-cdef extern from "stdlib.h":
-    ctypedef unsigned long size_t
+#cdef extern from "stdlib.h":
+#    ctypedef unsigned long size_t

  cdef extern from "stdsage.h":
      void  sage_free(void *p)
diff -r 43f267532b7a sage/libs/gmp/types.pxd
--- a/sage/libs/gmp/types.pxd	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/libs/gmp/types.pxd	Fri Feb 20 02:09:18 2009 -0800
@@ -3,7 +3,7 @@
      ### Type Declarations ###

      # Underlying typedefs
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t
      ctypedef unsigned long mp_limb_t
      ctypedef long mp_size_t
      ctypedef long mp_exp_t
diff -r 43f267532b7a sage/libs/pari/decl.pxi
--- a/sage/libs/pari/decl.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/libs/pari/decl.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -1,7 +1,7 @@
  include '../../ext/cdefs.pxi'

  cdef extern from "stdlib.h":
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t
      void free(void *ptr)
      void *malloc(size_t size)
      void *realloc(void *ptr, size_t size)


From robertwb at math.washington.edu  Fri Feb 20 11:45:02 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 02:45:02 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E7EEF.2020304@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
Message-ID: <E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>

On Feb 20, 2009, at 1:59 AM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>>> reports errors, for instance change Context.delref to do a
>>>> print/sys.exit(1) thing instead of raising an exception
>>> I would prefer a simple sys.__stderr__.write() with some helpful
>>> information in a dedicated code line that you could point gdb to.
>>> Killing
>>> the process sounds everything but helpful. Note that sys.exit()
>>> also just
>>> raises an exception, as opposed to os._exit().
>>
>> +1. If memory is leaking, I'd rather have it tell me and move on. Of
>> course if there are reference errors later on it may segfault, but at
>> least I'd have useful information already.
>
> Not surprisingly we get bitten by the widely discussed ctypedef sizt_t
> thingy:
>
> # Memory management
> cdef extern from "stdlib.h":
>      ctypedef unsigned long size_t
>
>
>> - Robert
>
> Before I go off and make a fool of myself fixing various potential
> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
> known issues? If the above is the only problem I can probably  
> handle it :)

I'll make a 0.11 ticket and post a patch. I got sage to compile, but  
it fails to start up.

- Robert


From robertwb at math.washington.edu  Fri Feb 20 11:48:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 02:48:38 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
Message-ID: <703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>

On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:

> On Feb 20, 2009, at 1:59 AM, Michael Abshoff wrote:
>
>> Robert Bradshaw wrote:
>>> On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:
>>>
>>>> Dag Sverre Seljebotn wrote:
>>>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>>>> reports errors, for instance change Context.delref to do a
>>>>> print/sys.exit(1) thing instead of raising an exception
>>>> I would prefer a simple sys.__stderr__.write() with some helpful
>>>> information in a dedicated code line that you could point gdb to.
>>>> Killing
>>>> the process sounds everything but helpful. Note that sys.exit()
>>>> also just
>>>> raises an exception, as opposed to os._exit().
>>>
>>> +1. If memory is leaking, I'd rather have it tell me and move on. Of
>>> course if there are reference errors later on it may segfault,  
>>> but at
>>> least I'd have useful information already.
>>
>> Not surprisingly we get bitten by the widely discussed ctypedef  
>> sizt_t
>> thingy:
>>
>> # Memory management
>> cdef extern from "stdlib.h":
>>      ctypedef unsigned long size_t
>>
>>
>>> - Robert
>>
>> Before I go off and make a fool of myself fixing various potential
>> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
>> known issues? If the above is the only problem I can probably
>> handle it :)
>
> I'll make a 0.11 ticket and post a patch. I got sage to compile, but
> it fails to start up.

http://trac.sagemath.org/sage_trac/ticket/4987



From michael.abshoff at googlemail.com  Fri Feb 20 12:00:21 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 03:00:21 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
Message-ID: <499E8D45.6020605@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:

<SNIP>

Hi Robert,

>>> Before I go off and make a fool of myself fixing various potential
>>> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
>>> known issues? If the above is the only problem I can probably
>>> handle it :)
>> I'll make a 0.11 ticket and post a patch. I got sage to compile, but
>> it fails to start up.
> 
> http://trac.sagemath.org/sage_trac/ticket/4987

Ahh, I see there was another size_t hiding in LinBox, I guess I should 
have at least looked into that file before giving up.

I have to get 3.3.rc3 out the door and will take your patch later today 
and see of I can figure out what the problem is.

Cheers,

Michael

> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb 20 12:06:21 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 03:06:21 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E8D45.6020605@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
Message-ID: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>

On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:
>
> <SNIP>
>
> Hi Robert,
>
>>>> Before I go off and make a fool of myself fixing various potential
>>>> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
>>>> known issues? If the above is the only problem I can probably
>>>> handle it :)
>>> I'll make a 0.11 ticket and post a patch. I got sage to compile, but
>>> it fails to start up.
>>
>> http://trac.sagemath.org/sage_trac/ticket/4987
>
> Ahh, I see there was another size_t hiding in LinBox, I guess I should
> have at least looked into that file before giving up.

Yeah. I'm really happy to have native size_t now.

> I have to get 3.3.rc3 out the door and will take your patch later  
> today
> and see of I can figure out what the problem is.

Here's the log. (I made it print rather than raise an error). I'm  
going to try and make it print the source file as well as line.

----------------------------------------------------------------------
| Sage Version 3.2.3, Release Date: 2009-01-05                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
References leaked:
   Acquired on lines: 4278, 4287, 4345, 4354, 4412, 4421
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 7564
Too many decrefs on line 7577, reference acquired on lines []
NULL argument on line 42791
Too many decrefs on line 42813, reference acquired on lines []
NULL argument on line 1596
Too many decrefs on line 1617, reference acquired on lines []
NULL argument on line 1924
Too many decrefs on line 1948, reference acquired on lines []
NULL argument on line 819
Too many decrefs on line 846, reference acquired on lines []
NULL argument on line 1596
Too many decrefs on line 1617, reference acquired on lines []
NULL argument on line 819
Too many decrefs on line 846, reference acquired on lines []
NULL argument on line 1596
Too many decrefs on line 1617, reference acquired on lines []
NULL argument on line 1924
Too many decrefs on line 1940, reference acquired on lines []
NULL argument on line 819
Too many decrefs on line 846, reference acquired on lines []
References leaked:
   Acquired on lines: 1048, 1114
   Acquired on lines: 1049, 1115
   Acquired on lines: 1050, 1116
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8400, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 11296
Too many decrefs on line 11317, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8540
Too many decrefs on line 8561, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8562
Too many decrefs on line 8585, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 7421
Too many decrefs on line 7434, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8406, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10869
Too many decrefs on line 10882, reference acquired on lines []
NULL argument on line 5211
Too many decrefs on line 5233, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10869
Too many decrefs on line 10882, reference acquired on lines []
NULL argument on line 5211
Too many decrefs on line 5233, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10869
Too many decrefs on line 10882, reference acquired on lines []
NULL argument on line 5211
Too many decrefs on line 5233, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 7103
Too many decrefs on line 7116, reference acquired on lines []
NULL argument on line 42088
Too many decrefs on line 42110, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8562
Too many decrefs on line 8585, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8406, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 7421
Too many decrefs on line 7434, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8406, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
sage:
Exiting SAGE (CPU time 0m0.06s, Wall time 0m1.19s).
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/ 
local/bin/sage-sage: line 210:  6932 Bus error               sage- 
ipython "$@" -i


From michael.abshoff at googlemail.com  Fri Feb 20 12:21:29 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 03:21:29 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
Message-ID: <499E9239.5030205@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:

<SNIP>

Hi Robert,

>> Ahh, I see there was another size_t hiding in LinBox, I guess I should
>> have at least looked into that file before giving up.
> 
> Yeah. I'm really happy to have native size_t now.

Yep, I never expected that there were six different definitions of 
size_t in Sage.

>> I have to get 3.3.rc3 out the door and will take your patch later  
>> today
>> and see of I can figure out what the problem is.
> 
> Here's the log. (I made it print rather than raise an error). I'm  
> going to try and make it print the source file as well as line.

Ok, that would be *extremely* nice.

> ----------------------------------------------------------------------
> | Sage Version 3.2.3, Release Date: 2009-01-05                       |
> | Type notebook() for the GUI, and license() for information.        |
> ----------------------------------------------------------------------

<SNIP>

> NULL argument on line 819
> Too many decrefs on line 846, reference acquired on lines []
> References leaked:
>    Acquired on lines: 1048, 1114
>    Acquired on lines: 1049, 1115
>    Acquired on lines: 1050, 1116

Excellent, this is catching real bugs.

Dag: Next time I meet you in person dinner is on me for the refanny :)

<SNIP>

> sage:
> Exiting SAGE (CPU time 0m0.06s, Wall time 0m1.19s).
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/ 
> local/bin/sage-sage: line 210:  6932 Bus error               sage- 
> ipython "$@" -i

I am afraid what we will find when we run this code on the whole doctest 
suite, but hopefully this will just kill a whole class of bugs in Sage :)

I can even see this happening per default as long as I merge patches.

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dagss at student.matnat.uio.no  Fri Feb 20 12:51:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 12:51:09 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
Message-ID: <499E992D.6040600@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>> I have to get 3.3.rc3 out the door and will take your patch later  
>> today
>> and see of I can figure out what the problem is.
> 
> Here's the log. (I made it print rather than raise an error). I'm  
> going to try and make it print the source file as well as line.

Hehe, not much use without :-)

> Too many decrefs on line 4775, reference acquired on lines []

Be especially careful that errors like this may easily come from missing 
GOTREFs rather than a refcount error. I don't know whether I have 
GOTREF/GIVEREF everywhere they are needed, just that they are everywhere 
were needed within what we have coverage for.

> NULL argument on line 4760

Furthermore, if this is a GOTREF or GIVEREF, it may be that the refnanny 
part should be altered to XGOTREF/XGIVEREF for that situation.

> References leaked:
>    Acquired on lines: 1048, 1114
>    Acquired on lines: 1049, 1115
>    Acquired on lines: 1050, 1116

This kind of output could be clearer, but it means that three different 
objects were leaked in the function, each of which were incref-ed or 
gotref-ed two times.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 12:57:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 12:57:01 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E9239.5030205@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E9239.5030205@gmail.com>
Message-ID: <499E9A8D.6030003@student.matnat.uio.no>

Michael Abshoff wrote:
> Robert Bradshaw wrote:
>> References leaked:
>>    Acquired on lines: 1048, 1114
>>    Acquired on lines: 1049, 1115
>>    Acquired on lines: 1050, 1116
> 
> Excellent, this is catching real bugs.

Well, the error message is not that clear -- it may also be that the 
function legitimately handed off a reference to somewhere else, but 
didn't declare that it did with a GIVEREF.

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Fri Feb 20 12:56:51 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 03:56:51 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E9A8D.6030003@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E9239.5030205@gmail.com>
	<499E9A8D.6030003@student.matnat.uio.no>
Message-ID: <499E9A83.2030106@gmail.com>

Dag Sverre Seljebotn wrote:
> Michael Abshoff wrote:

Hi Dag,

>> Robert Bradshaw wrote:
>>> References leaked:
>>>    Acquired on lines: 1048, 1114
>>>    Acquired on lines: 1049, 1115
>>>    Acquired on lines: 1050, 1116
>> Excellent, this is catching real bugs.
> 
> Well, the error message is not that clear -- it may also be that the 
> function legitimately handed off a reference to somewhere else, but 
> didn't declare that it did with a GIVEREF.
> 

Sure. Note that technically I didn't claim those were bugs in Sage :)

I just think that the refnanny and Sage will be good tests for each other.

Cheers,

Michael

From robertwb at math.washington.edu  Fri Feb 20 13:16:30 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 04:16:30 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E992D.6040600@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E992D.6040600@student.matnat.uio.no>
Message-ID: <F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>

On Feb 20, 2009, at 3:51 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>>> I have to get 3.3.rc3 out the door and will take your patch later
>>> today
>>> and see of I can figure out what the problem is.
>>
>> Here's the log. (I made it print rather than raise an error). I'm
>> going to try and make it print the source file as well as line.
>
> Hehe, not much use without :-)

Full log at http://trac.sagemath.org/sage_trac/ticket/4987  
Fortunately, lots are the same error over and over. BTW, I don't know  
if printing them out like this could simplify a lot of your manual  
exception handling code.

>> Too many decrefs on line 4775, reference acquired on lines []
>
> Be especially careful that errors like this may easily come from  
> missing
> GOTREFs rather than a refcount error. I don't know whether I have
> GOTREF/GIVEREF everywhere they are needed, just that they are  
> everywhere
> were needed within what we have coverage for.

Yep. Either way, we should track them down.

>> NULL argument on line 4760
>
> Furthermore, if this is a GOTREF or GIVEREF, it may be that the  
> refnanny
> part should be altered to XGOTREF/XGIVEREF for that situation.

Yeah, it could be the case.

>> References leaked:
>>    Acquired on lines: 1048, 1114
>>    Acquired on lines: 1049, 1115
>>    Acquired on lines: 1050, 1116
>
> This kind of output could be clearer, but it means that three  
> different
> objects were leaked in the function, each of which were incref-ed or
> gotref-ed two times.

Thanks for the explanation.

- Robert

From dagss at student.matnat.uio.no  Fri Feb 20 14:01:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 14:01:53 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>
	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
Message-ID: <499EA9C1.9000308@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 3:51 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>>>> I have to get 3.3.rc3 out the door and will take your patch later
>>>> today
>>>> and see of I can figure out what the problem is.
>>> Here's the log. (I made it print rather than raise an error). I'm
>>> going to try and make it print the source file as well as line.
>> Hehe, not much use without :-)
> 
> Full log at http://trac.sagemath.org/sage_trac/ticket/4987  
> Fortunately, lots are the same error over and over. BTW, I don't know  
> if printing them out like this could simplify a lot of your manual  
> exception handling code.

Not really. One may enter refnanny.pyx during exception cleanup while an 
exception is set, and then refnanny.pyx goes on to doing lots of Python 
operations -- the first error flag check within refnanny.pyx will cause 
the execution stack to pop even if there wasn't a problem within 
refnanny.pyx itself.

It could however simplify the four lines at the end of every function in 
Cython-generated C code. Personally I prefer to have the testcases 
listed as "failed" when run from runtests.py (and even be able to flag 
refcount errors as expected in doctests, if we ever need that), but I 
don't have a strong opinion either way.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 14:14:19 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 14:14:19 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499EA9C1.9000308@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
Message-ID: <499EACAB.4010502@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> It could however simplify the four lines at the end of every function in 
> Cython-generated C code. Personally I prefer to have the testcases 
> listed as "failed" when run from runtests.py (and even be able to flag 
> refcount errors as expected in doctests, if we ever need that), but I 
> don't have a strong opinion either way.

OK I see where you are coming from now and have come to agree with you 
instead.

http://trac.cython.org/cython_trac/ticket/212

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 14:31:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 14:31:12 +0100
Subject: [Cython] size_t/#207
Message-ID: <499EB0A0.60602@student.matnat.uio.no>

Is this now fixed?

http://trac.cython.org/cython_trac/ticket/207

If not, I guess it should get 0.11 milestone as size_t is now introduced?


-- 
Dag Sverre

From dalcinl at gmail.com  Fri Feb 20 14:57:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 20 Feb 2009 11:57:28 -0200
Subject: [Cython] size_t/#207
In-Reply-To: <499EB0A0.60602@student.matnat.uio.no>
References: <499EB0A0.60602@student.matnat.uio.no>
Message-ID: <e7ba66e40902200557w1cbc36d2ie54ac80646229e85@mail.gmail.com>

Wait me a bit more of time, I'll push a fix in a few minutes to enable
backward compatibility. Next we can discuss how to emit warnings about
'ctypeded unsigned long size_t'

On Fri, Feb 20, 2009 at 11:31 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Is this now fixed?
>
> http://trac.cython.org/cython_trac/ticket/207
>
> If not, I guess it should get 0.11 milestone as size_t is now introduced?
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Feb 20 15:42:56 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 20 Feb 2009 12:42:56 -0200
Subject: [Cython] please close ticket #207,
	'size_t' support finished (and backward 	compatible)
Message-ID: <e7ba66e40902200642m520b9db5lf1f8459d150ecd68@mail.gmail.com>

After my last push, ticket #207 can be closed. Now Cython treats
Py_ssize_t/size_t as basic (but special) C types.

Moreover, the former practice of doing:

cdef extern from *:
    ctypedef unsigned long size_t

would be simply ignored. This is IMHO good for backward compatibility
with previous Cython released and also Pyrex. Of course, in the near
future we issue warnings at CTypedefNode.analize_declarations() (an
not only for size_t !!!).


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb 20 20:22:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 20:22:52 +0100
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
Message-ID: <499F030C.4060400@behnel.de>

Hi Lisandro,

I get a couple of these with your latest change when compiling Cython
itself. Could you try to come up with a fix?

Thanks,

Stefan


Compiling module Cython.Plex.Scanners ...

Compiling module Cython.Compiler.Scanning ...

Compiling module Cython.Compiler.Parsing ...


Error converting Pyrex file to C:
------------------------------------------------------------
...

    bint PyObject_TypeCheck(object o, PyTypeObject *type)
    # Return true if the object o is of type type or a subtype of
    # type. Both parameters must be non-NULL.

    Py_ssize_t PyObject_Length(object o)
   ^
------------------------------------------------------------

/home/sbehnel/source/Python/cython/cython-work/Cython/Includes/python_object.pxd:240:4:
Unrecognised type modifier combination

Error converting Pyrex file to C:
------------------------------------------------------------
...
    bint PyObject_TypeCheck(object o, PyTypeObject *type)
    # Return true if the object o is of type type or a subtype of
    # type. Both parameters must be non-NULL.

    Py_ssize_t PyObject_Length(object o)
    Py_ssize_t PyObject_Size(object o)
   ^
------------------------------------------------------------

/home/sbehnel/source/Python/cython/cython-work/Cython/Includes/python_object.pxd:241:4:
Unrecognised type modifier combination
Compilation failed



From robertwb at math.washington.edu  Fri Feb 20 21:12:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 12:12:46 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499EACAB.4010502@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
	<499EACAB.4010502@student.matnat.uio.no>
Message-ID: <C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>


On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> It could however simplify the four lines at the end of every  
>> function in
>> Cython-generated C code. Personally I prefer to have the testcases
>> listed as "failed" when run from runtests.py (and even be able to  
>> flag
>> refcount errors as expected in doctests, if we ever need that), but I
>> don't have a strong opinion either way.

If it prints stuff out, it will still be listed as failed, right?

> OK I see where you are coming from now and have come to agree with you
> instead.
>
> http://trac.cython.org/cython_trac/ticket/212

I did push a bit of a patch in this direction

http://hg.cython.org/cython-devel/rev/dbb1a4421f5d

- Robert


From stefan_ml at behnel.de  Fri Feb 20 21:40:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 21:40:50 +0100
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902201230k494b2d1fn9266fdc999d7f012@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>	
	<499F030C.4060400@behnel.de>
	<e7ba66e40902201230k494b2d1fn9266fdc999d7f012@mail.gmail.com>
Message-ID: <499F1552.8040504@behnel.de>


Lisandro Dalcin wrote:
> On Fri, Feb 20, 2009 at 5:22 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> I get a couple of these with your latest change when compiling Cython
>> itself. Could you try to come up with a fix?
>>
> Stefan, I sincerely do not know what's going on here... I DO NOT get
> these errors!! Cython builds itself just fine in my box ...

Argh, sorry, my fault. I forgot to say "make clean" before building, so it
tried to build using outdated binary modules.

Well done, just keep going, nothing to see here ...

Stefan

From stefan_ml at behnel.de  Fri Feb 20 21:52:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 21:52:11 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>	<499EA9C1.9000308@student.matnat.uio.no>	<499EACAB.4010502@student.matnat.uio.no>
	<C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
Message-ID: <499F17FB.10908@behnel.de>

Hi,

Robert Bradshaw wrote:
> On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> It could however simplify the four lines at the end of every  
>>> function in
>>> Cython-generated C code. Personally I prefer to have the testcases
>>> listed as "failed" when run from runtests.py (and even be able to  
>>> flag
>>> refcount errors as expected in doctests, if we ever need that), but I
>>> don't have a strong opinion either way.
> 
> If it prints stuff out, it will still be listed as failed, right?

Sure, she just needs to keep her mouth shut when all is ok, and burst stuff
into Python's stdout when things go wrong. That will make the doctests fail
for free. That way it's trivial to test for output in the doctests as well,
e.g. you could enforce a leak on purpose and expect the nanny's cry as output.

Stefan


From michael.f.ellis at gmail.com  Fri Feb 20 22:02:26 2009
From: michael.f.ellis at gmail.com (Mike Ellis)
Date: Fri, 20 Feb 2009 21:02:26 +0000 (UTC)
Subject: [Cython] C API declarations
References: <4974E631.10309@simula.no>
Message-ID: <loom.20090220T205230-146@post.gmane.org>


Ilmar Wilbers <ilmarw at ...> writes:

> 
> Hi,
> 
> I am having some serious problems with the C API declarations, chapter 
> 7.2.2 i n the manual.
> 
> Using the exact two example files does not work for me, I keep getting:
> marty.c:3: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? 
> before ?car?
> marty.c: In function ?main?:
> marty.c:7: error: ?car? undeclared (first use in this function)
> marty.c:7: error: (Each undeclared identifier is reported only once
> marty.c:7: error: for each function it appears in.)
> 
<snip>
> Is there a problem with the header files that I don't understand? I run 
> Cython on delorean.pyx before running
> gcc -c -I/usr/include/python2.5 -lpython2.5 marty.c
> 
> Has anyone tried these examples lately?
> 
> Sincerly, Ilmar
> 

I ran into the same problem with the delorean example.  I can get marty.c to
compile by changing

Vehicle car;

to 

struct Vehicle car;

Getting it to run is another story. It dies deep inside import_delorean(). 
Heres a trace from gdb:

Program received signal SIGSEGV, Segmentation fault.
0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0xb7ea7ac7 in PyEval_GetGlobals () from /usr/lib/libpython2.5.so.1.0
#2  0xb7ec9187 in PyImport_Import () from /usr/lib/libpython2.5.so.1.0
#3  0x08048797 in __Pyx_ImportModule (name=0x8048bc7 "delorean")
    at delorean_api.h:24
#4  0x08048984 in import_delorean () at delorean_api.h:73
#5  0x08048a3e in main (argc=3, argv=0xbfdbfa34) at marty.c:6

I'm building with gcc (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu3) using the same command
line as Ilmar.

Anyone know what might be going wrong?

Thanks,
Mike




From dagss at student.matnat.uio.no  Fri Feb 20 22:21:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 22:21:05 +0100 (CET)
Subject: [Cython] Status for 0.11
In-Reply-To: <499F17FB.10908@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E992D.6040600@student.matnat.uio.no>
	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
	<499EACAB.4010502@student.matnat.uio.no>
	<C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
	<499F17FB.10908@behnel.de>
Message-ID: <690acc13b787caf0bdb159655a5863e3.squirrel@webmail.uio.no>

Stefan wrote:
> Hi,
>
> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> It could however simplify the four lines at the end of every
>>>> function in
>>>> Cython-generated C code. Personally I prefer to have the testcases
>>>> listed as "failed" when run from runtests.py (and even be able to
>>>> flag
>>>> refcount errors as expected in doctests, if we ever need that), but I
>>>> don't have a strong opinion either way.
>>
>> If it prints stuff out, it will still be listed as failed, right?
>
> Sure, she just needs to keep her mouth shut when all is ok, and burst
> stuff
> into Python's stdout when things go wrong. That will make the doctests
> fail
> for free. That way it's trivial to test for output in the doctests as

Well, not if you expect exceptions in the doctests... (I hate that aspect
of doctest myself; I'

Dag Sverre


From dagss at student.matnat.uio.no  Fri Feb 20 22:24:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 22:24:10 +0100 (CET)
Subject: [Cython] Status for 0.11
In-Reply-To: <499F17FB.10908@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E992D.6040600@student.matnat.uio.no>
	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
	<499EACAB.4010502@student.matnat.uio.no>
	<C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
	<499F17FB.10908@behnel.de>
Message-ID: <50f1fe2a55831242182c36b6e0fc6408.squirrel@webmail.uio.no>

Stefan wrote:
> Hi,
>
> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> It could however simplify the four lines at the end of every
>>>> function in
>>>> Cython-generated C code. Personally I prefer to have the testcases
>>>> listed as "failed" when run from runtests.py (and even be able to
>>>> flag
>>>> refcount errors as expected in doctests, if we ever need that), but I
>>>> don't have a strong opinion either way.
>>
>> If it prints stuff out, it will still be listed as failed, right?
>
> Sure, she just needs to keep her mouth shut when all is ok, and burst
> stuff
> into Python's stdout when things go wrong. That will make the doctests

If an exception is expected from the function, doctest will not compare
the output printed before the exception happens (and I really dislike that
aspect of doctest). So in those cases, one could have the refnanny
complain without a failure.

Of course, you'll usually pay enough attention to see the output anyway,
so it's not a big deal.

Dag Sverre


From robertwb at math.washington.edu  Sat Feb 21 00:08:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 15:08:40 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
Message-ID: <A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>

On Feb 20, 2009, at 3:06 AM, Robert Bradshaw wrote:

> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>
>> Robert Bradshaw wrote:
>>> On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:
>>
>> <SNIP>
>>
>> Hi Robert,
>>
>>>>> Before I go off and make a fool of myself fixing various potential
>>>>> Cython issues in 3.3.rc2: Robert - do you have a patch or a  
>>>>> list of
>>>>> known issues? If the above is the only problem I can probably
>>>>> handle it :)
>>>> I'll make a 0.11 ticket and post a patch. I got sage to compile,  
>>>> but
>>>> it fails to start up.
>>>
>>> http://trac.sagemath.org/sage_trac/ticket/4987
>>
>> Ahh, I see there was another size_t hiding in LinBox, I guess I  
>> should
>> have at least looked into that file before giving up.
>
> Yeah. I'm really happy to have native size_t now.
>
>> I have to get 3.3.rc3 out the door and will take your patch later
>> today
>> and see of I can figure out what the problem is.
>
> Here's the log. (I made it print rather than raise an error). I'm
> going to try and make it print the source file as well as line.

One little fix in cpdef functions fixed almost all the issues. We're  
down to

----------------------------------------------------------------------
| Sage Version 3.2.3, Release Date: 2009-01-05                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage/rings/padics/padic_printing.cpp: __init__()
References leaked:
   Acquired on lines: 4278, 4287, 4345, 4354, 4412, 4421
sage/structure/factory.c: get_object()
References leaked:
   Acquired on lines: 1048, 1114
   Acquired on lines: 1049, 1115
   Acquired on lines: 1050, 1116
sage:
Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/ 
local/bin/sage-sage: line 587: 16443 Bus error               sage- 
ipython "$@" -i

- Robert


From robertwb at math.washington.edu  Sat Feb 21 07:44:14 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 22:44:14 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
Message-ID: <6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>

I found and fixed two memory leaks. This refcounting nanny sure is  
nice. Sage now starts up without any warnings, but upon exiting I  
still get

> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/
> local/bin/sage-sage: line 587: 16443 Bus error               sage-
> ipython "$@" -i

Next up is running the test suite.

- Robert

From michael.abshoff at googlemail.com  Sat Feb 21 07:56:14 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 22:56:14 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>
Message-ID: <499FA58E.9010303@gmail.com>

Robert Bradshaw wrote:

Hi Robert,

> I found and fixed two memory leaks. 

Got a patch for me? I could still get that one into 3.3 :)

> This refcounting nanny sure is  
> nice. Sage now starts up without any warnings, but upon exiting I  
> still get
> 
>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>> Exception exceptions.AssertionError in  ignored
>> Exception exceptions.AssertionError in  ignored
>> Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/
>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>> ipython "$@" -i
> 
> Next up is running the test suite.

Cool, looking forward to all the fixes coming out of that.

> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Sat Feb 21 07:57:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 22:57:45 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499FA58E.9010303@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>
	<499FA58E.9010303@gmail.com>
Message-ID: <431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>

On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>
> Hi Robert,
>
>> I found and fixed two memory leaks.
>
> Got a patch for me? I could still get that one into 3.3 :)

Memory leaks in Cython, not in Sage. (Actually, probably not  
noticeable ones, one was leaking an exception, and the other was  
"leaking" the values True and False in most cases.

>> This refcounting nanny sure is
>> nice. Sage now starts up without any warnings, but upon exiting I
>> still get
>>
>>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>>> Exception exceptions.AssertionError in  ignored
>>> Exception exceptions.AssertionError in  ignored
>>> Exception exceptions.AssertionError in /Users/robert/sage/sage- 
>>> cython/
>>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>>> ipython "$@" -i
>>
>> Next up is running the test suite.
>
> Cool, looking forward to all the fixes coming out of that.

I'm going to run it overnight. Given that Sage doesn't exit cleanly  
(not sure what's up with that), I expect everything to fail, but  
hopefully it'll still be useful.

- Robert


From michael.abshoff at googlemail.com  Sat Feb 21 08:17:36 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 23:17:36 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>
	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
Message-ID: <499FAA90.5020505@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:

Hi Robert,

>>> I found and fixed two memory leaks.
>> Got a patch for me? I could still get that one into 3.3 :)
> 
> Memory leaks in Cython, not in Sage. (Actually, probably not  
> noticeable ones, one was leaking an exception, and the other was  
> "leaking" the values True and False in most cases.

Well, we had 8 byte leaks in Sage that ate up all available RAM on the 
old sage.math in about two or three days, i.e. leaking about 1 to 2 
GB/hour, so if you hit them often enough even tiny leaks have a giant 
impact. And I have some cases where I do hope that the refnanny will 
point out ref count issues and explain leaks where I have been totally 
unable to pin point a cause in Sage code itself. Getting Sage to start 
up and exit clean will be great, but the test suite will likely expose 
many more issues :)

<SNIP>

> I'm going to run it overnight. Given that Sage doesn't exit cleanly  
> (not sure what's up with that), I expect everything to fail, but  
> hopefully it'll still be useful.

Sure. I took today's cython spkg and I am valgrinding Sage with it, so 
maybe I can give you some pointer what is going wrong.

> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dalcinl at gmail.com  Mon Feb  2 19:20:30 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 02 Feb 2009 18:20:30 -0000
Subject: [Cython] possible problems with temp allocation?
Message-ID: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>

See the attached files. The line 452 in generated foo.c (attached just
for your convenience) seems to be wrong. Furthermore, valgrind pointed
me to that line.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.pyx
Type: application/octet-stream
Size: 319 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/ade53f48/attachment-0001.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.c
Type: text/x-csrc
Size: 47242 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/ade53f48/attachment-0001.c 

From dagss at student.matnat.uio.no  Sat Feb 21 16:26:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 16:26:26 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>
	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
Message-ID: <49A01D22.3070504@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:
> 
>> Robert Bradshaw wrote:
>>
>> Hi Robert,
>>
>>> I found and fixed two memory leaks.
>> Got a patch for me? I could still get that one into 3.3 :)
> 
> Memory leaks in Cython, not in Sage. (Actually, probably not  
> noticeable ones, one was leaking an exception, and the other was  
> "leaking" the values True and False in most cases.
> 
>>> This refcounting nanny sure is
>>> nice. Sage now starts up without any warnings, but upon exiting I
>>> still get
>>>
>>>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>>>> Exception exceptions.AssertionError in  ignored
>>>> Exception exceptions.AssertionError in  ignored
>>>> Exception exceptions.AssertionError in /Users/robert/sage/sage- 
>>>> cython/
>>>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>>>> ipython "$@" -i
>>> Next up is running the test suite.
>> Cool, looking forward to all the fixes coming out of that.
> 
> I'm going to run it overnight. Given that Sage doesn't exit cleanly  
> (not sure what's up with that), I expect everything to fail, but  
> hopefully it'll still be useful.

Don't work too hard on it yet -- there's some bugs in the refnanny now 
which I should fix, and which may produce these kinds of errors.

One suspect is here:

http://hg.cython.org/cython-devel/rev/4e19b8a61272

There are some more as well which could lead to similar symptoms which 
I'll have to wait with pushing -- ironically, the fix itself seems to 
produce a temp handling refcount bug on the refnanny.c code itself, so I 
must fix that simultaneously.

-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Feb 21 16:30:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 21 Feb 2009 07:30:53 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <49A01D22.3070504@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>
	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
	<49A01D22.3070504@student.matnat.uio.no>
Message-ID: <AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>

On Feb 21, 2009, at 7:26 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:
>>
>>> Robert Bradshaw wrote:
>>>
>>> Hi Robert,
>>>
>>>> I found and fixed two memory leaks.
>>> Got a patch for me? I could still get that one into 3.3 :)
>>
>> Memory leaks in Cython, not in Sage. (Actually, probably not
>> noticeable ones, one was leaking an exception, and the other was
>> "leaking" the values True and False in most cases.
>>
>>>> This refcounting nanny sure is
>>>> nice. Sage now starts up without any warnings, but upon exiting I
>>>> still get
>>>>
>>>>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>>>>> Exception exceptions.AssertionError in  ignored
>>>>> Exception exceptions.AssertionError in  ignored
>>>>> Exception exceptions.AssertionError in /Users/robert/sage/sage-
>>>>> cython/
>>>>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>>>>> ipython "$@" -i
>>>> Next up is running the test suite.
>>> Cool, looking forward to all the fixes coming out of that.
>>
>> I'm going to run it overnight. Given that Sage doesn't exit cleanly
>> (not sure what's up with that), I expect everything to fail, but
>> hopefully it'll still be useful.
>
> Don't work too hard on it yet -- there's some bugs in the refnanny now
> which I should fix, and which may produce these kinds of errors.

I'm running the tests, but probably won't be able to work on it much  
today anyways. So far I have a 7MB log (and it's less than half way  
done).

> One suspect is here:
>
> http://hg.cython.org/cython-devel/rev/4e19b8a61272
>
> There are some more as well which could lead to similar symptoms which
> I'll have to wait with pushing -- ironically, the fix itself seems to
> produce a temp handling refcount bug on the refnanny.c code itself,  
> so I
> must fix that simultaneously.

Cool, let us know when you smash this one.

- Robert



From dagss at student.matnat.uio.no  Sat Feb 21 17:00:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 17:00:25 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>	<49A01D22.3070504@student.matnat.uio.no>
	<AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>
Message-ID: <49A02519.8030409@student.matnat.uio.no>

Robert Bradshaw wrote:
> I'm running the tests, but probably won't be able to work on it much  
> today anyways. So far I have a 7MB log (and it's less than half way  
> done).

Well, as you showed us with your last patch, one bug in this area tends 
to be repeated a lot of time. So IMHO anything but the top 20 lines in 
that log is worthless, because you'd like to rerun after fixing the 
first issue anyway to remove the duplicates...

(Of course, statistics is fun.)

>> One suspect is here:
>>
>> http://hg.cython.org/cython-devel/rev/4e19b8a61272
>>
>> There are some more as well which could lead to similar symptoms which
>> I'll have to wait with pushing -- ironically, the fix itself seems to
>> produce a temp handling refcount bug on the refnanny.c code itself,  
>> so I
>> must fix that simultaneously.
> 
> Cool, let us know when you smash this one.

http://hg.cython.org/cython-devel/rev/6692a0f0e87a

I won't get into a discussion on whether a fault like this is due to me 
over-relying on the test suite or the pathetic low coverage it has. It's 
fixed now; but I wonder how many more there are...

Anyway, having a refcount fault on every single "del a.b[c]" is 
something I can easily see accounting for some of the megabytes in that log.

-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Feb 21 16:57:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 21 Feb 2009 07:57:36 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <49A02519.8030409@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>	<49A01D22.3070504@student.matnat.uio.no>
	<AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>
	<49A02519.8030409@student.matnat.uio.no>
Message-ID: <4EFE7985-C038-4B28-A2E8-8F4A3A1C5D6E@math.washington.edu>

On Feb 21, 2009, at 8:00 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> I'm running the tests, but probably won't be able to work on it much
>> today anyways. So far I have a 7MB log (and it's less than half way
>> done).
>
> Well, as you showed us with your last patch, one bug in this area  
> tends
> to be repeated a lot of time. So IMHO anything but the top 20 lines in
> that log is worthless, because you'd like to rerun after fixing the
> first issue anyway to remove the duplicates...
>
> (Of course, statistics is fun.)

Yes, there's a lot of duplicates

Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
test.log | grep "Acquired on lines" | sort | uniq | wc
       25     367    2431

Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
test.log | grep "sage/.*: " | sort | uniq
     sage/libs/pari/gen.c: _normalize_slice()
     sage/libs/singular/singular.cpp: sa2si_ZZ()
     sage/matrix/misc.c: matrix_rational_echelon_form_multimodular()
     sage/rings/polynomial/multi_polynomial_libsingular.cpp: subs()

Which, not surprisingly, are the "usual suspects" for memory leaks. I  
haven't hit sage/rings yet though, which may turn up a lot more.

>>> One suspect is here:
>>>
>>> http://hg.cython.org/cython-devel/rev/4e19b8a61272
>>>
>>> There are some more as well which could lead to similar symptoms  
>>> which
>>> I'll have to wait with pushing -- ironically, the fix itself  
>>> seems to
>>> produce a temp handling refcount bug on the refnanny.c code itself,
>>> so I
>>> must fix that simultaneously.
>>
>> Cool, let us know when you smash this one.
>
> http://hg.cython.org/cython-devel/rev/6692a0f0e87a
>
> I won't get into a discussion on whether a fault like this is due  
> to me
> over-relying on the test suite or the pathetic low coverage it has.  
> It's
> fixed now; but I wonder how many more there are...

Yeah. At least Sage provides pretty good coverage :).

> Anyway, having a refcount fault on every single "del a.b[c]" is
> something I can easily see accounting for some of the megabytes in  
> that log.

I don't think this is common, but happening just once in a lot of  
places could result in a big log.

- Robert

From michael.abshoff at googlemail.com  Sat Feb 21 17:13:09 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sat, 21 Feb 2009 08:13:09 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <4EFE7985-C038-4B28-A2E8-8F4A3A1C5D6E@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>	<49A01D22.3070504@student.matnat.uio.no>	<AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>	<49A02519.8030409@student.matnat.uio.no>
	<4EFE7985-C038-4B28-A2E8-8F4A3A1C5D6E@math.washington.edu>
Message-ID: <49A02815.3000403@gmail.com>

Robert Bradshaw wrote:
> On Feb 21, 2009, at 8:00 AM, Dag Sverre Seljebotn wrote:

Hi Robert,

>>
>> (Of course, statistics is fun.)
> 
> Yes, there's a lot of duplicates
> 
> Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
> test.log | grep "Acquired on lines" | sort | uniq | wc
>        25     367    2431
> 
> Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
> test.log | grep "sage/.*: " | sort | uniq
>      sage/libs/pari/gen.c: _normalize_slice()
>      sage/libs/singular/singular.cpp: sa2si_ZZ()
>      sage/matrix/misc.c: matrix_rational_echelon_form_multimodular()
>      sage/rings/polynomial/multi_polynomial_libsingular.cpp: subs()
> 
> Which, not surprisingly, are the "usual suspects" for memory leaks. I  
> haven't hit sage/rings yet though, which may turn up a lot more.


Yep, the best strategy here after having run the complete test suite 
once for the "horrible before-picure" is likely to start with coercion 
and structure and then work your way through rings before then looking 
at what is left over. There are some other leaks not in the test suite I 
would like to examine with this once the noise from the low level stuff 
has been quited. Please let me know what I can do on my end to make 
Cython 0.11 happen faster since I cannot wait for it to be in an 
official Sage release. I am expecting similar miracles as from the 
0.10.3 Cython release which already fixed on bad, bad reference count 
related leak issue :)

Note that Sage 3.3 even with the patch from Sage's Cython 0.11 trac 
ticket applied does not compile with yesterday's Cython devel head and I 
did put the failure into a comment on that ticket. It was basically that 
Cython did throw an error for

  typedef unsigned long ulong

in the FLINT header.

Cheers,

Michael

From stefan_ml at behnel.de  Sat Feb 21 17:57:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 17:57:00 +0100
Subject: [Cython] Problems in Py 3.0.1
Message-ID: <49A0325C.9040901@behnel.de>

Hi,

just a quick note that I'm currently trying to get the test suite running
under Python 3.0.1. This release features a couple of fixes for quirks that
were found in the 3.0 release. It is now a lot stricter regarding Py3
concepts. I'll file a couple of new tickets as I move along, and I'll
nominate (most of) them for 0.11, especially those that make Cython
unusable on that target. I expect them to be easy to fix, but I'll
concentrate on finding them for now. So don't be surprised when the bug
list for 0.11 starts getting longer again.

I also noticed tons of warnings in the C++ "compile" tests regarding
useless code. I'm not sure why they pop up in Py3 and not in Py2.

Stefan

From stefan_ml at behnel.de  Sat Feb 21 18:06:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 18:06:04 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0325C.9040901@behnel.de>
References: <49A0325C.9040901@behnel.de>
Message-ID: <49A0347C.8000208@behnel.de>



Stefan Behnel wrote:
> I also noticed tons of warnings in the C++ "compile" tests regarding
> useless code. I'm not sure why they pop up in Py3 and not in Py2.

Those are actually due to the refnanny code. I had to disable the refnanny
in Py3, as we cannot currently compile Cython on that platform anyway, and
the refnanny is built as part of the normal Cython build.

Stefan


From dagss at student.matnat.uio.no  Sat Feb 21 18:24:22 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:24:22 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0347C.8000208@behnel.de>
References: <49A0325C.9040901@behnel.de> <49A0347C.8000208@behnel.de>
Message-ID: <49A038C6.4000007@student.matnat.uio.no>

Stefan Behnel wrote:
> 
> Stefan Behnel wrote:
>> I also noticed tons of warnings in the C++ "compile" tests regarding
>> useless code. I'm not sure why they pop up in Py3 and not in Py2.
> 
> Those are actually due to the refnanny code. I had to disable the refnanny
> in Py3, as we cannot currently compile Cython on that platform anyway, and
> the refnanny is built as part of the normal Cython build.

Have you done a pull recently? I expect some certain useless code to be 
removed now.

The refnanny can be as useful in Python 3 as in Python 2, though if it 
doesn't work that's certainly not a priority, as the issues it checks 
should be the same under both.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Feb 21 18:26:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:26:53 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A038C6.4000007@student.matnat.uio.no>
References: <49A0325C.9040901@behnel.de> <49A0347C.8000208@behnel.de>
	<49A038C6.4000007@student.matnat.uio.no>
Message-ID: <49A0395D.4060207@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Stefan Behnel wrote:
>>> I also noticed tons of warnings in the C++ "compile" tests regarding
>>> useless code. I'm not sure why they pop up in Py3 and not in Py2.
>> Those are actually due to the refnanny code. I had to disable the refnanny
>> in Py3, as we cannot currently compile Cython on that platform anyway, and
>> the refnanny is built as part of the normal Cython build.

To be clearer, runtests.py works with the refnanny without the normal 
Cython build happening first (runtests.py uses pyximport to build it). 
To test e.g. lxml with the refnanny under Python 3 you would need to 
manually compile and install Cython/Runtime/refnanny.pyx

(You probably already know this; just to be sure)

-- 
Dag Sverre

From jasone at canonware.com  Sat Feb 21 18:13:52 2009
From: jasone at canonware.com (Jason Evans)
Date: Sat, 21 Feb 2009 09:13:52 -0800
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
Message-ID: <49A03650.7050803@canonware.com>

The following line in ModuleNode.py causes hundreds of compiler warnings 
about statements with no effect when I compile Crux:

	#define __Pyx_FinishRefcountContext() 0

IMO, the 0 should simply be removed, but this would also be okay if you 
want a non-empty definition:

	#define __Pyx_FinishRefcountContext() do {} while (0)

Jason

From dagss at student.matnat.uio.no  Sat Feb 21 18:29:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:29:53 +0100
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
In-Reply-To: <49A03650.7050803@canonware.com>
References: <49A03650.7050803@canonware.com>
Message-ID: <49A03A11.7000907@student.matnat.uio.no>

Jason Evans wrote:
> The following line in ModuleNode.py causes hundreds of compiler warnings 
> about statements with no effect when I compile Crux:
> 
> 	#define __Pyx_FinishRefcountContext() 0
> 
> IMO, the 0 should simply be removed, but this would also be okay if you 
> want a non-empty definition:
> 
> 	#define __Pyx_FinishRefcountContext() do {} while (0)
> 

Ahh, right. Thanks!

http://hg.cython.org/cython-devel/rev/b5723a7a044a


-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Feb 21 18:16:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 18:16:47 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0395D.4060207@student.matnat.uio.no>
References: <49A0325C.9040901@behnel.de>
	<49A0347C.8000208@behnel.de>	<49A038C6.4000007@student.matnat.uio.no>
	<49A0395D.4060207@student.matnat.uio.no>
Message-ID: <49A036FF.8060003@behnel.de>


Dag Sverre Seljebotn wrote:
> To be clearer, runtests.py works with the refnanny without the normal 
> Cython build happening first (runtests.py uses pyximport to build it). 
> To test e.g. lxml with the refnanny under Python 3 you would need to 
> manually compile and install Cython/Runtime/refnanny.pyx

Then the next step is to port pyximport to Py3, right? :)

Stefan


From stefan_ml at behnel.de  Sat Feb 21 18:29:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 18:29:56 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A036FF.8060003@behnel.de>
References: <49A0325C.9040901@behnel.de>	<49A0347C.8000208@behnel.de>	<49A038C6.4000007@student.matnat.uio.no>	<49A0395D.4060207@student.matnat.uio.no>
	<49A036FF.8060003@behnel.de>
Message-ID: <49A03A14.8050206@behnel.de>


Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> To be clearer, runtests.py works with the refnanny without the normal 
>> Cython build happening first (runtests.py uses pyximport to build it). 
>> To test e.g. lxml with the refnanny under Python 3 you would need to 
>> manually compile and install Cython/Runtime/refnanny.pyx
> 
> Then the next step is to port pyximport to Py3, right? :)

ah, and: make it work even if Cython cannot be imported due to SyntaxErrors.

http://trac.cython.org/cython_trac/ticket/214

Stefan

From dalcinl at gmail.com  Sat Feb 21 18:39:08 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 14:39:08 -0300
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
In-Reply-To: <49A03A11.7000907@student.matnat.uio.no>
References: <49A03650.7050803@canonware.com>
	<49A03A11.7000907@student.matnat.uio.no>
Message-ID: <e7ba66e40902210939u72a6c30ej4dcc281d81f4ad81@mail.gmail.com>

A "(void)0" shoud also do the trick, right?

On Sat, Feb 21, 2009 at 2:29 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Jason Evans wrote:
>> The following line in ModuleNode.py causes hundreds of compiler warnings
>> about statements with no effect when I compile Crux:
>>
>>       #define __Pyx_FinishRefcountContext() 0
>>
>> IMO, the 0 should simply be removed, but this would also be okay if you
>> want a non-empty definition:
>>
>>       #define __Pyx_FinishRefcountContext() do {} while (0)
>>
>
> Ahh, right. Thanks!
>
> http://hg.cython.org/cython-devel/rev/b5723a7a044a
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sat Feb 21 18:58:46 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:58:46 +0100
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
In-Reply-To: <e7ba66e40902210939u72a6c30ej4dcc281d81f4ad81@mail.gmail.com>
References: <49A03650.7050803@canonware.com>	<49A03A11.7000907@student.matnat.uio.no>
	<e7ba66e40902210939u72a6c30ej4dcc281d81f4ad81@mail.gmail.com>
Message-ID: <49A040D6.8040600@student.matnat.uio.no>

It was a remnant after when the refnanny raised exceptions which I 
forgot to remove; the correct thing is to be a noop now. (It needed to 
return 0 earlier on and was used in an expression then.)

Dag Sverre

Lisandro Dalcin wrote:
> A "(void)0" shoud also do the trick, right?
> 
> On Sat, Feb 21, 2009 at 2:29 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Jason Evans wrote:
>>> The following line in ModuleNode.py causes hundreds of compiler warnings
>>> about statements with no effect when I compile Crux:
>>>
>>>       #define __Pyx_FinishRefcountContext() 0
>>>
>>> IMO, the 0 should simply be removed, but this would also be okay if you
>>> want a non-empty definition:
>>>
>>>       #define __Pyx_FinishRefcountContext() do {} while (0)
>>>
>> Ahh, right. Thanks!
>>
>> http://hg.cython.org/cython-devel/rev/b5723a7a044a
>>
>>
>> --
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> 
> 
> 


-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Feb 21 19:40:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 19:40:00 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0325C.9040901@behnel.de>
References: <49A0325C.9040901@behnel.de>
Message-ID: <49A04A80.5020705@behnel.de>


Stefan Behnel wrote:
> just a quick note that I'm currently trying to get the test suite running
> under Python 3.0.1. This release features a couple of fixes for quirks that
> were found in the 3.0 release. It is now a lot stricter regarding Py3
> concepts.

Interesting enough, the size_t tests fail for me with a TypeError:

---------------------------------------
Failed example:
    a.foo(-1) #doctest: +ELLIPSIS
Expected:
    Traceback (most recent call last):
        ...
    OverflowError: ...
Got:
    Traceback (most recent call last):
      File "/usr/local/python3.0.1-debug/lib/python3.0/doctest.py", line
1234, in __run
        compileflags, 1), test.globs)
      File "<doctest size_t[10]>", line 1, in <module>
        a.foo(-1) #doctest: +ELLIPSIS
      File "size_t.pyx", line 53, in size_t.A.foo (size_t.cpp:569)
    TypeError: can't convert negative int to unsigned
---------------------------------------

Wasn't there supposed to be an OverflowError instead?

Stefan

From stefan_ml at behnel.de  Sat Feb 21 21:26:55 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 21:26:55 +0100
Subject: [Cython] string target type of struct-to-dict conversion
Message-ID: <49A0638F.5060409@behnel.de>

Hi,

I found this code in the struct_conversion.pyx test:

----------------------------------
    >>> test_constructor_kwds(1.25, 2.5, 128)
    {'y': 2.5, 'x': 1.25, 'color': 128}

...


def test_dict_construction(x, y, color):
    cdef Point p = {'color': color, 'x': x, 'y': y}
    return p

def test_constructor_kwds(x, y, color):
    cdef Point p = Point(x=x, y=y, color=color)
    return p
----------------------------------

Should we really base this feature on byte strings instead of unicode strings?

Note that the doctest at the top fails in Py3 anyway, as the repr() of a
byte string has a leading 'b'.

Stefan

From dalcinl at gmail.com  Sat Feb 21 22:26:24 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 18:26:24 -0300
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A04A80.5020705@behnel.de>
References: <49A0325C.9040901@behnel.de> <49A04A80.5020705@behnel.de>
Message-ID: <e7ba66e40902211326p757acd6biedaafb7b7bdf2c7c@mail.gmail.com>

Stefan, in Py3, PyInt is gone and we only have PyLong. Then the code
follows other path.

Some days ago I filed an issue and provided a patch wich was accepted,
now core Python (2.7 and 3.1) will raise OverflowError...

If you want, I can implement some vile hackery, I just need to check
myself if the PyLong is negative (PyLong implementation abuses the
tp_size member of the object struct to carry the sign)



On Sat, Feb 21, 2009 at 3:40 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Stefan Behnel wrote:
>> just a quick note that I'm currently trying to get the test suite running
>> under Python 3.0.1. This release features a couple of fixes for quirks that
>> were found in the 3.0 release. It is now a lot stricter regarding Py3
>> concepts.
>
> Interesting enough, the size_t tests fail for me with a TypeError:
>
> ---------------------------------------
> Failed example:
>    a.foo(-1) #doctest: +ELLIPSIS
> Expected:
>    Traceback (most recent call last):
>        ...
>    OverflowError: ...
> Got:
>    Traceback (most recent call last):
>      File "/usr/local/python3.0.1-debug/lib/python3.0/doctest.py", line
> 1234, in __run
>        compileflags, 1), test.globs)
>      File "<doctest size_t[10]>", line 1, in <module>
>        a.foo(-1) #doctest: +ELLIPSIS
>      File "size_t.pyx", line 53, in size_t.A.foo (size_t.cpp:569)
>    TypeError: can't convert negative int to unsigned
> ---------------------------------------
>
> Wasn't there supposed to be an OverflowError instead?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Feb 21 22:35:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 22:35:17 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <e7ba66e40902211326p757acd6biedaafb7b7bdf2c7c@mail.gmail.com>
References: <49A0325C.9040901@behnel.de> <49A04A80.5020705@behnel.de>
	<e7ba66e40902211326p757acd6biedaafb7b7bdf2c7c@mail.gmail.com>
Message-ID: <49A07395.4030806@behnel.de>


Lisandro Dalcin wrote:
> Stefan, in Py3, PyInt is gone and we only have PyLong. Then the code
> follows other path.
> 
> Some days ago I filed an issue and provided a patch wich was accepted,
> now core Python (2.7 and 3.1) will raise OverflowError...

Right, I found that in the back of my head after writing the mail.


> If you want, I can implement some vile hackery, I just need to check
> myself if the PyLong is negative (PyLong implementation abuses the
> tp_size member of the object struct to carry the sign)

No, for now, I'm fine with whatever Python does here. I pushed a fix for
the test case.

Stefan

From dalcinl at gmail.com  Sat Feb 21 23:04:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 19:04:28 -0300
Subject: [Cython] On refnanny commit
In-Reply-To: <49A079F3.8020108@student.matnat.uio.no>
References: <49A079F3.8020108@student.matnat.uio.no>
Message-ID: <e7ba66e40902211404m9fc50fbo9867891e2e93574@mail.gmail.com>

Dag, this change was required for the case you pass --cleanup to Cython.

This flag is surely not used in runtests.py, so any of us never
noticed that the generated code (when --cleanup is used) do not
compile...

Remember that cleanup funtions are registered with "atexit" to run at
Python shutdown... I do not think that is a good place to be tracking
references....

BTW, I'll try to enable refnanny for the module init function.
Basically, we should GIVEREF on module globals before finishing the
refcount context. I'll give a try.



On Sat, Feb 21, 2009 at 7:02 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Hi, I just noticed your commit. I'm fine with it, I was just wondering if
> you could write a line or two about the reasons behind it?
>
> Also, I'm worried that user code in module scope will still be refnannied in
> that function, so you cannot remove the refnanny initialization for that
> function altogether... Did you work around that? (Didn't look at it in close
> detail.)
>
> --
> Dag Sverre
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sun Feb 22 00:19:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 00:19:41 +0100
Subject: [Cython] Py3 exception handling and chaining
Message-ID: <49A08C0D.2020508@behnel.de>

Hi,

I fixed most of the Py3 problems in the test suite. The last one relates to
exception chaining.

http://www.python.org/dev/peps/pep-3134/

Currently, when you write this:

    try: ...
    except SomeException:
        raise NewException

SomeException will remain in sys.exc_info(), or rather in
PyThreadState->exc_type/value/tb. Subsequently raised exceptions will find
it there and make it their "context". A way to fix this is to put this into
Pyx_ErrRestore() to clear the context exception when the current exception
is set:

    tmp_type = tstate->exc_type;
    tmp_value = tstate->exc_value;
    tmp_tb = tstate->exc_traceback;
    tstate->exc_type = 0;
    tstate->exc_value = 0;
    tstate->exc_traceback = 0;
    Py_XDECREF(tmp_type);
    Py_XDECREF(tmp_value);
    Py_XDECREF(tmp_tb);

While doing that, I also noticed that ReRaise is currently implemented like
this:

  static void __Pyx_ReRaise(void) {
    PyThreadState *tstate = PyThreadState_Get();
    PyObject *type = tstate->exc_type;
    PyObject *value = tstate->exc_value;
    PyObject *tb = tstate->exc_traceback;
    Py_XINCREF(type);
    Py_XINCREF(value);
    Py_XINCREF(tb);
    __Pyx_ErrRestore(type, value, tb);
  }

I wonder why this doesn't set exc_* to NULL either.

Both changes make the test suite pass in Py 3.0.1, and it still works in
Py2.6. Maybe Greg can tell us why it was originally implemented differently
and if there are potential issues?

In any case, this is more of a quick fix rather than the final solution.
The real fix would be to handle exception chaining correctly in Py3, i.e.
to set the context of an exception depending on where it is raised, and to
support the explicit "raise ... from ..." syntax to allow users to set the
cause.

http://trac.cython.org/cython_trac/ticket/217

Stefan

From dalcinl at gmail.com  Sun Feb 22 00:40:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 20:40:37 -0300
Subject: [Cython] enabling refnanny in module init function
Message-ID: <e7ba66e40902211540g2ca9c7cfyb0f35bb7fc1608cf@mail.gmail.com>

Try the trivial patch below, the problems with globals get fixed...
However, enabling the refnanny will spot other reference count
problems, at least I get them in mpi4py ....


diff -r 3628c9268dfe Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py	Sat Feb 21 22:34:25 2009 +0100
+++ b/Cython/Compiler/ModuleNode.py	Sat Feb 21 20:32:14 2009 -0300
@@ -1651,11 +1651,9 @@
         env.use_utility_code(Nodes.traceback_utility_code)
         code.putln("%s = NULL;" % Naming.retval_cname)
         code.put_label(code.return_label)
-        # Disabled because of confusion with refcount of global
variables -- run ass2cglobal testcase to see
-        #code.put_finish_refcount_context()
-        code.putln("#if CYTHON_REFNANNY")
-        code.putln("if (__pyx_refchk) {};")
-        code.putln("#endif")
+
+        code.put_finish_refcount_context()
+
         code.putln("#if PY_MAJOR_VERSION < 3")
         code.putln("return;")
         code.putln("#else")
@@ -1793,7 +1791,7 @@
         for entry in env.var_entries:
             if entry.visibility != 'extern':
                 if entry.type.is_pyobject and entry.used:
-                    code.put_init_var_to_py_none(entry)
+                    code.put_init_var_to_py_none(entry, nanny=False)

     def generate_c_function_export_code(self, env, code):
         # Generate code to create PyCFunction wrappers for exported C
functions.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sun Feb 22 11:27:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 11:27:31 +0100
Subject: [Cython] enabling refnanny in module init function
In-Reply-To: <e7ba66e40902211540g2ca9c7cfyb0f35bb7fc1608cf@mail.gmail.com>
References: <e7ba66e40902211540g2ca9c7cfyb0f35bb7fc1608cf@mail.gmail.com>
Message-ID: <49A12893.5040401@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Try the trivial patch below, the problems with globals get fixed...
> However, enabling the refnanny will spot other reference count
> problems, at least I get them in mpi4py ....

Yes, I've looked into some of them and they really need fixing. I've pushed
your patch.

Stefan


From stefan_ml at behnel.de  Sun Feb 22 12:01:28 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 12:01:28 +0100
Subject: [Cython] module cleanup (was: On refnanny commit)
In-Reply-To: <e7ba66e40902211404m9fc50fbo9867891e2e93574@mail.gmail.com>
References: <49A079F3.8020108@student.matnat.uio.no>
	<e7ba66e40902211404m9fc50fbo9867891e2e93574@mail.gmail.com>
Message-ID: <49A13088.8030201@behnel.de>


Lisandro Dalcin wrote:
> Remember that cleanup funtions are registered with "atexit" to run at
> Python shutdown... I do not think that is a good place to be tracking
> references....

This is OK for Py2.x. However, for Python 3, the right solution would be to
put the cleanup code into the official module clear function m_clear(), as
described in PEP 3121.

http://www.python.org/dev/peps/pep-3121/

Then, the test runner could simply unload the module after running its
doctests, but still as part of the test itself.

There already is a ticket for general PEP 3121 support (#173), but I
extracted this as a new ticket #218, as I find it the most important part,
which can be done much sooner.

http://trac.cython.org/cython_trac/ticket/173
http://trac.cython.org/cython_trac/ticket/218

Stefan

From stefan_ml at behnel.de  Sun Feb 22 12:56:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 12:56:30 +0100
Subject: [Cython] embedded signatures and function annotations (PEP 3107)
Message-ID: <49A13D6E.1090003@behnel.de>

Hi,

I wonder why this function declaration

    cpdef str cf_sd(str s='spam'):
        return s

leads to this embedded signature:

    >>> cf_sd.__doc__
    "cf_sd(str s='spam') -> str"

The return value follows PEP 3107 style annotations, but wouldn't it make
sense to use PEP 3107 style annotations also for the parameter types like this:

    >>> cf_sd.__doc__
    "cf_sd(s : str = 'spam') -> str"

?

Stefan

From stefan_ml at behnel.de  Sun Feb 22 15:13:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 15:13:11 +0100
Subject: [Cython] Py3 exception handling and chaining
In-Reply-To: <49A08C0D.2020508@behnel.de>
References: <49A08C0D.2020508@behnel.de>
Message-ID: <49A15D77.4080905@behnel.de>

Hi again,

Stefan Behnel wrote:
> Currently, when you write this:
> 
>     try: ...
>     except SomeException:
>         raise NewException
> 
> SomeException will remain in sys.exc_info(), or rather in
> PyThreadState->exc_type/value/tb. Subsequently raised exceptions will find
> it there and make it their "context".
> [...]
> Both changes make the test suite pass in Py 3.0.1, and it still works in
> Py2.6.

They also seem to be the fix for an extremely long standing crash bug that
I experienced with lxml under Py3. So, for me, that means that the final
release of lxml 2.2 will finally support Py 3.0 completely and reliably.

Filed and closed as ticket 219.

http://trac.cython.org/cython_trac/ticket/219
http://hg.cython.org/cython-devel/rev/24bb0456d787

I actually found these things using a debug build of Py 3.0.1. It's a very
valuable source of information when it comes to tracking down crashes and a
number of other problems with extensions.

Stefan

From sturla at molden.no  Mon Feb 23 05:37:36 2009
From: sturla at molden.no (Sturla Molden)
Date: Mon, 23 Feb 2009 05:37:36 +0100 (CET)
Subject: [Cython] Fortran types in C
Message-ID: <70bfb5794dc767535a8679334bc64028.squirrel@webmail.uio.no>


>What is the best way of declaring Fortran types in C? I understand there
>is likely no perfect cross-platform solution here

ISO Fortran 2003 has standard bindings to ISO C. Most Fortran compilers
support them, including gfortran. These C bindings make C callable from
Fortran, but they also make Fortran callable from C in a standardized and
portable way.

The short receipe is: you don't declare any Fortran types in C. You
declare your Fortran functions with C types. This even includes
pass-by-value calling convention. Then the Fortran compiler will do the
correct conversion.

>Furthermore it would be very sweet to
>automatically be able to pass Python buffer objects as F90 assumed-shape
>arrays :-)

Take a look at the array-descriptor library of Chasm.

http://chasm-interop.sourceforge.net/


Regards,
Sturla Molden


From anatoly at gnu.org  Mon Feb 23 10:19:51 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Mon, 23 Feb 2009 15:19:51 +0600
Subject: [Cython] Wrapping callbacks
Message-ID: <20090223151951.5960dce1.anatoly@gnu.org>

Hello!

I'm trying to wrap function with callback parameter:

error_t fshelp_start_translator (fshelp_open_fn_t underlying_open_fn, void *cookie, char *name, char *argz, int argz_len, int timeout, fsys_t *control)

where fshelp_open_fn_t is

typedef error_t (*fshelp_open_fn_t) (int flags, file_t *node, mach_msg_type_name_t *node_type, task_t, void *cookie)

Is it possible?

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090223/300fab87/attachment.pgp 

From stefan_ml at behnel.de  Mon Feb 23 10:55:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 10:55:23 +0100 (CET)
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090223151951.5960dce1.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
Message-ID: <62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Anatoly A. Kazantsev wrote:
> I'm trying to wrap function with callback parameter:
>
> error_t fshelp_start_translator (fshelp_open_fn_t underlying_open_fn, void
> *cookie, char *name, char *argz, int argz_len, int timeout, fsys_t
> *control)
>
> where fshelp_open_fn_t is
>
> typedef error_t (*fshelp_open_fn_t) (int flags, file_t *node,
> mach_msg_type_name_t *node_type, task_t, void *cookie)

Have you looked at the callback example in the Cython distribution?

http://hg.cython.org/cython-devel/file/tip/Demos/callback

Stefan


From anatoly at gnu.org  Mon Feb 23 11:56:25 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Mon, 23 Feb 2009 16:56:25 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <20090223165625.5deaffe9.anatoly@gnu.org>

On Mon, 23 Feb 2009 10:55:23 +0100 (CET)
"Stefan Behnel" <stefan_ml at behnel.de> wrote:

> Anatoly A. Kazantsev wrote:
> > I'm trying to wrap function with callback parameter:
> >
> > error_t fshelp_start_translator (fshelp_open_fn_t underlying_open_fn, void
> > *cookie, char *name, char *argz, int argz_len, int timeout, fsys_t
> > *control)
> >
> > where fshelp_open_fn_t is
> >
> > typedef error_t (*fshelp_open_fn_t) (int flags, file_t *node,
> > mach_msg_type_name_t *node_type, task_t, void *cookie)
> 
> Have you looked at the callback example in the Cython distribution?

Yes, of course.

But I can not understand how to pass pointer of Python callback-function.
Additional parameter 'void *user_data' is used in the example for this purpose.

But I can't modify 'fshelp_open_fn_t' function type.

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090223/e13e8962/attachment-0001.pgp 

From greg.ewing at canterbury.ac.nz  Mon Feb 23 12:43:33 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Feb 2009 00:43:33 +1300
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090223165625.5deaffe9.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
Message-ID: <49A28BE5.1020709@canterbury.ac.nz>

Anatoly A. Kazantsev wrote:

> Additional parameter 'void *user_data' is used in the example for this purpose.
> 
> But I can't modify 'fshelp_open_fn_t' function type.

What's the 'void *cookie'? Is it an arbitrary pointer
that you can use for your own purposes? If so, use
it the same way as the user_data in the example.

-- 
Greg

From dalcinl at gmail.com  Mon Feb 23 20:09:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 17:09:27 -0200
Subject: [Cython] unicode can bite us ...
Message-ID: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>

Stefan, in this commit

changeset:   1743:58406a9db77b
user:        Stefan Behnel <scoder at users.berlios.de>
date:        Sat Feb 21 21:08:46 2009 +0100
summary:     unicode fixes for numpy test (untested, hopefully not
breaking everything)

your broke everything ;-)


BTW, I still want Cython to let me specify in an easy way that a
string literal should match the one Python normally uses for
identifier...

For example:

np.arange(10, dtype=s'i')


Then s'i' literal whould be PyString in Py2 and PyUnicode in Py3... We
could even make this syntax sugar really strict and reject
non-pure-ASCII literals.

We cannot assume all Python packages out there (specially those
targeted for scientific users)  are unicode-aware... My numpy
installation (numpy-1.2.0 as packaged in Fedora 10) is a good
example...

Stefan, we really need this... IMHO this is a clear case where
practicality beats purity....

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Mon Feb 23 21:24:20 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 21:24:20 +0100
Subject: [Cython] unicode can bite us ...
In-Reply-To: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
Message-ID: <49A305F4.3080107@behnel.de>


Lisandro Dalcin wrote:
> Stefan, in this commit
> 
> changeset:   1743:58406a9db77b
> user:        Stefan Behnel <scoder at users.berlios.de>
> date:        Sat Feb 21 21:08:46 2009 +0100
> summary:     unicode fixes for numpy test (untested, hopefully not
> breaking everything)
> 
> your broke everything ;-)

Sorry 'bout that! :)

I fixed that, changing the NumPy parameters from u'...' to b'...' should
make it work. I'll push it as soon as I get the conflict resolved that I
just got from you. :o)


> BTW, I still want Cython to let me specify in an easy way that a
> string literal should match the one Python normally uses for
> identifier...
> 
> For example:
> 
> np.arange(10, dtype=s'i')
> 
> Then s'i' literal whould be PyString in Py2 and PyUnicode in Py3...

However, that makes it sound like it's NumPy that's broken here. There's
absolutely no reason why the parameters should be byte strings.

I still doubt that such a feature is really needed. Any API that requires
byte strings where unicode strings would be appropriate is broken. The
above will become useless (and even semantically wrong) once NumPy is
fixed. Other libraries that currently require this will also be fixed in
the not so far future.


> Stefan, we really need this... IMHO this is a clear case where
> practicality beats purity....

Is NumPy actually Py3 compatible? If not, it's definitely not a case for a
decision either way.

Stefan

From stefan_ml at behnel.de  Mon Feb 23 21:35:15 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 21:35:15 +0100
Subject: [Cython] rev 5863fde464a0
Message-ID: <49A30883.3090603@behnel.de>

Hi Lisandro,

could you explain why this is needed? My fix would have been to add a
GIVEREF, as the default argument is being made global. I also liked the
code a lot better the old way. Is there a case where the original code
didn't work?

Stefan


@@ -1231,15 +1233,14 @@
             if default:
                 if not default.is_literal:
                     default.generate_evaluation_code(code)
-                    default.make_owned_reference(code)
-                    code.putln(
-                        "%s = %s;" % (
-                            arg.default_entry.cname,
-                            default.result_as(arg.default_entry.type)))
-                    if default.is_temp and default.type.is_pyobject:
-                        code.putln(
-                            "%s = 0;" %
-                                default.result())
+                    assign_code = "%s = %s;" % (
+                        arg.default_entry.cname,
+                        default.result_as(arg.default_entry.type))
+                    if default.type.is_pyobject:
+                        assign_code += " Py_INCREF(%s);" % \
+                            arg.type.as_pyobject(arg.default_entry.cname)
+                    code.putln(assign_code)
+                    default.generate_disposal_code(code)
                     default.free_temps(code)

From robert.kern at gmail.com  Mon Feb 23 22:03:38 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Feb 2009 15:03:38 -0600
Subject: [Cython] unicode can bite us ...
In-Reply-To: <49A305F4.3080107@behnel.de>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
Message-ID: <gnv2va$tc8$1@ger.gmane.org>

On 2009-02-23 14:24, Stefan Behnel wrote:

> Is NumPy actually Py3 compatible?

Not even close.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From stefan_ml at behnel.de  Mon Feb 23 22:07:42 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 22:07:42 +0100
Subject: [Cython] rev 5863fde464a0
In-Reply-To: <49A30883.3090603@behnel.de>
References: <49A30883.3090603@behnel.de>
Message-ID: <49A3101E.4030501@behnel.de>

Hi again,

> @@ -1231,15 +1233,14 @@
>              if default:
>                  if not default.is_literal:
>                      default.generate_evaluation_code(code)
> -                    default.make_owned_reference(code)
> -                    code.putln(
> -                        "%s = %s;" % (
> -                            arg.default_entry.cname,
> -                            default.result_as(arg.default_entry.type)))
> -                    if default.is_temp and default.type.is_pyobject:
> -                        code.putln(
> -                            "%s = 0;" %
> -                                default.result())
> +                    assign_code = "%s = %s;" % (
> +                        arg.default_entry.cname,
> +                        default.result_as(arg.default_entry.type))
> +                    if default.type.is_pyobject:
> +                        assign_code += " Py_INCREF(%s);" % \
> +                            arg.type.as_pyobject(arg.default_entry.cname)
> +                    code.putln(assign_code)
> +                    default.generate_disposal_code(code)
>                      default.free_temps(code)

Here's what I pushed now (Nodes.py):

            if default:
                if not default.is_literal:
                    default.generate_evaluation_code(code)
                    default.make_owned_reference(code)
                    code.putln(
                        "%s = %s;" % (
                            arg.default_entry.cname,
                            default.result_as(arg.default_entry.type)))
                    if default.is_temp and default.type.is_pyobject:
                        code.putln("%s = 0;" % default.result())
                    default.free_temps(code)
                    code.put_giveref(arg.default_entry.cname)

Note that generate_evaluation_code() already cleans up the temps of the RHS
subexpressions, so there is no need to do anything but stealing the reference.

BTW, note that a doctest must be a unicode string. Otherwise, it will not
be executed in Py3.

Stefan


From robertwb at math.washington.edu  Mon Feb 23 22:27:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 23 Feb 2009 13:27:07 -0800
Subject: [Cython] unicode can bite us ...
In-Reply-To: <49A305F4.3080107@behnel.de>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
Message-ID: <72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>

On Feb 23, 2009, at 12:24 PM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> Stefan, in this commit
>>
>> changeset:   1743:58406a9db77b
>> user:        Stefan Behnel <scoder at users.berlios.de>
>> date:        Sat Feb 21 21:08:46 2009 +0100
>> summary:     unicode fixes for numpy test (untested, hopefully not
>> breaking everything)
>>
>> your broke everything ;-)
>
> Sorry 'bout that! :)
>
> I fixed that, changing the NumPy parameters from u'...' to b'...'  
> should
> make it work. I'll push it as soon as I get the conflict resolved  
> that I
> just got from you. :o)
>
>> BTW, I still want Cython to let me specify in an easy way that a
>> string literal should match the one Python normally uses for
>> identifier...
>>
>> For example:
>>
>> np.arange(10, dtype=s'i')
>>
>> Then s'i' literal whould be PyString in Py2 and PyUnicode in Py3...
>
> However, that makes it sound like it's NumPy that's broken here.  
> There's
> absolutely no reason why the parameters should be byte strings.

This is how unmarked string literals should work, right? Just like  
the Python interpreter.

> I still doubt that such a feature is really needed. Any API that  
> requires
> byte strings where unicode strings would be appropriate is broken. The
> above will become useless (and even semantically wrong) once NumPy is
> fixed. Other libraries that currently require this will also be  
> fixed in
> the not so far future.

I think you underestimate how long broken libraries will be out there.

>> Stefan, we really need this... IMHO this is a clear case where
>> practicality beats purity....
>
> Is NumPy actually Py3 compatible? If not, it's definitely not a  
> case for a
> decision either way.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Mon Feb 23 22:36:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 19:36:06 -0200
Subject: [Cython] Stefan, please fix this
Message-ID: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>

In commit df54195ce73a, you changed this bit of my original push.

------------------------ Cython/Compiler/ModuleNode.py ------------------------
@@ -1706,6 -1706,10 +1706,9 @@
                 code.put_decref_clear(entry.pystring_cname,
                                       PyrexTypes.py_object_type,
                                       nanny=False)
+        for entry in env.default_entries:
+            if entry.type.is_pyobject and entry.used:
-                code.putln("Py_DECREF(%s); %s = 0;" % (
-                        code.entry_as_pyobject(entry), entry.cname))
+                code.put_var_decref_clear(entry)
         code.putln("Py_INCREF(Py_None); return Py_None;")
         code.putln('}')


Please revert to my implementation (bare Py_DECREF and nullify), or
any other that does not break C compilation... IMHO, The module
cleanup function is not a place to play with refnanny. Moreover, the
module cleanup funcion is not refnanny-managed...

BTW, I believe that cleanup code should be generated when running the
testsuite...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb 23 22:57:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 23 Feb 2009 22:57:57 +0100
Subject: [Cython] rev 5863fde464a0
In-Reply-To: <49A3101E.4030501@behnel.de>
References: <49A30883.3090603@behnel.de> <49A3101E.4030501@behnel.de>
Message-ID: <49A31BE5.9070406@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi again,
> 
>> @@ -1231,15 +1233,14 @@
>>              if default:
>>                  if not default.is_literal:
>>                      default.generate_evaluation_code(code)
>> -                    default.make_owned_reference(code)
>> -                    code.putln(
>> -                        "%s = %s;" % (
>> -                            arg.default_entry.cname,
>> -                            default.result_as(arg.default_entry.type)))
>> -                    if default.is_temp and default.type.is_pyobject:
>> -                        code.putln(
>> -                            "%s = 0;" %
>> -                                default.result())
>> +                    assign_code = "%s = %s;" % (
>> +                        arg.default_entry.cname,
>> +                        default.result_as(arg.default_entry.type))
>> +                    if default.type.is_pyobject:
>> +                        assign_code += " Py_INCREF(%s);" % \
>> +                            arg.type.as_pyobject(arg.default_entry.cname)
>> +                    code.putln(assign_code)
>> +                    default.generate_disposal_code(code)
>>                      default.free_temps(code)
> 
> Here's what I pushed now (Nodes.py):
> 
>             if default:
>                 if not default.is_literal:
>                     default.generate_evaluation_code(code)
>                     default.make_owned_reference(code)
>                     code.putln(
>                         "%s = %s;" % (
>                             arg.default_entry.cname,
>                             default.result_as(arg.default_entry.type)))
>                     if default.is_temp and default.type.is_pyobject:
>                         code.putln("%s = 0;" % default.result())
>                     default.free_temps(code)
>                     code.put_giveref(arg.default_entry.cname)
> 
> Note that generate_evaluation_code() already cleans up the temps of the RHS
> subexpressions, so there is no need to do anything but stealing the reference.

+1; at least dropping that call to make_owned_reference was asking for 
trouble (as I suppose the object could or could not own the reference, 
and that could be a noop or not a noop, depending on circumstances)

-- 
Dag Sverre

From dalcinl at gmail.com  Mon Feb 23 22:48:38 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 19:48:38 -0200
Subject: [Cython] unicode can bite us ...
In-Reply-To: <72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
Message-ID: <e7ba66e40902231348y740d3d57gfd8fd91baabfc90d@mail.gmail.com>

On Mon, Feb 23, 2009 at 7:27 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> This is how unmarked string literals should work, right? Just like
> the Python interpreter.
>

I'm on your side, Robert. At least, I would like to have a easy way to
specify a string literal matching whatever Python uses for
identifiers...

>
> I think you underestimate how long broken libraries will be out there.
>

Indeed... And I still thing we should provide facilities to alleviate
the str/unicode difference and the Py2/Py3 difference...




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb 23 23:53:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 19:53:09 -0300
Subject: [Cython] patch, func arg default vals, please review
Message-ID: <e7ba66e40902231453q1d1853bfn24db63882ad90629@mail.gmail.com>

Take a look, I believe the fix is valid, in such case please push it.
Today I lost my self-confidence, don't like to mess the codebase again
:-)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PATCH.diff
Type: application/octet-stream
Size: 1151 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090223/bd41b5cd/attachment.obj 

From anatoly at gnu.org  Tue Feb 24 07:11:16 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Tue, 24 Feb 2009 12:11:16 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <49A28BE5.1020709@canterbury.ac.nz>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
Message-ID: <20090224121116.4e732c2a.anatoly@gnu.org>

On Tue, 24 Feb 2009 00:43:33 +1300
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Anatoly A. Kazantsev wrote:
> 
> > Additional parameter 'void *user_data' is used in the example for this purpose.
> > 
> > But I can't modify 'fshelp_open_fn_t' function type.
> 
> What's the 'void *cookie'? Is it an arbitrary pointer
> that you can use for your own purposes? If so, use
> it the same way as the user_data in the example.

Yes, I can use this pointer :-)

But how can I wrap callbacks without such pointers. Or this pointers is a 'good'
practise in designing of callback functions?

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/0b4cd83b/attachment.pgp 

From robertwb at math.washington.edu  Tue Feb 24 08:24:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 23 Feb 2009 23:24:32 -0800
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090224121116.4e732c2a.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
Message-ID: <C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>

On Feb 23, 2009, at 10:11 PM, Anatoly A. Kazantsev wrote:

> On Tue, 24 Feb 2009 00:43:33 +1300
> Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
>> Anatoly A. Kazantsev wrote:
>>
>>> Additional parameter 'void *user_data' is used in the example for  
>>> this purpose.
>>>
>>> But I can't modify 'fshelp_open_fn_t' function type.
>>
>> What's the 'void *cookie'? Is it an arbitrary pointer
>> that you can use for your own purposes? If so, use
>> it the same way as the user_data in the example.
>
> Yes, I can use this pointer :-)
>
> But how can I wrap callbacks without such pointers. Or this  
> pointers is a 'good'
> practise in designing of callback functions?

Pointers are the only way to pass around functions in C. Pointers  
aren't bad, they're just powerful and potentially unforgiving.

- Robert


From anatoly at gnu.org  Tue Feb 24 08:42:18 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Tue, 24 Feb 2009 13:42:18 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
Message-ID: <20090224134218.6d65f7ce.anatoly@gnu.org>

On Mon, 23 Feb 2009 23:24:32 -0800
Robert Bradshaw <robertwb at math.washington.edu> wrote:

> Pointers are the only way to pass around functions in C. Pointers  
> aren't bad, they're just powerful and potentially unforgiving.

Yeah, I know. But what to do in a case when there's no such 'cookie' pointer in
callback-function definition.

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/1a877e05/attachment.pgp 

From robertwb at math.washington.edu  Tue Feb 24 08:48:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 23 Feb 2009 23:48:57 -0800
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090224134218.6d65f7ce.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
	<20090224134218.6d65f7ce.anatoly@gnu.org>
Message-ID: <3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>

On Feb 23, 2009, at 11:42 PM, Anatoly A. Kazantsev wrote:

> On Mon, 23 Feb 2009 23:24:32 -0800
> Robert Bradshaw <robertwb at math.washington.edu> wrote:
>
>> Pointers are the only way to pass around functions in C. Pointers
>> aren't bad, they're just powerful and potentially unforgiving.
>
> Yeah, I know. But what to do in a case when there's no such  
> 'cookie' pointer in
> callback-function definition.

It's standard practice to have such void* arguments for callback  
functions. Otherwise you'd have to store something globally and not  
use two concurrently (which is obviously not ideal).

- Robert


From stefan_ml at behnel.de  Tue Feb 24 09:29:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 09:29:07 +0100 (CET)
Subject: [Cython] unicode can bite us ...
In-Reply-To: <72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
Message-ID: <49698.213.61.181.86.1235464147.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Feb 23, 2009, at 12:24 PM, Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>> np.arange(10, dtype=s'i')
>> There's
>> absolutely no reason why the parameters should be byte strings.
>
> This is how unmarked string literals should work, right? Just like
> the Python interpreter.

Well, at least, that's what's written in the code: a byte string. What I'm
saying is that /requiring/ a byte string at the interface level is wrong.


> I think you underestimate how long broken libraries will be out there.

Let's wait and see. It didn't take me very long to fix up the Py3 unicode
problems of lxml's API (those that were independent of Cython), so I would
expect that any library can be fixed in a couple of weeks - as long as the
tools play nicely. I have no idea how hard it is to fix these things in a
SWIG wrapper, for example, but SWIG will also be fixed one day (and Cython
code will likely not interfere with a SWIG wrapper anyway).

Anyway, the NumPy case is not a use case for a new syntax feature, as it
requires plain byte strings for now, not identifiers. Lisandro, if you can
come up with a compelling use case that cannot easily be fixed in user
code, I'll rethink my opposition. For now, I cannot see how this feature
can lead to anything but even more broken interfaces.

Stefan


From stefan_ml at behnel.de  Tue Feb 24 09:33:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 09:33:52 +0100 (CET)
Subject: [Cython] unicode can bite us ...
In-Reply-To: <e7ba66e40902231348y740d3d57gfd8fd91baabfc90d@mail.gmail.com>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
	<e7ba66e40902231348y740d3d57gfd8fd91baabfc90d@mail.gmail.com>
Message-ID: <54907.213.61.181.86.1235464432.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> I'm on your side, Robert. At least, I would like to have a easy way to
> specify a string literal matching whatever Python uses for
> identifiers...
>
> And I still thing we should provide facilities to alleviate
> the str/unicode difference and the Py2/Py3 difference...

Well, then show me a case where this is required or at least helpful. My
gut feeling is that this feature will just push the problem we have on one
side of wrapper code over to the other side, without solving anything.

Anyway, this is all just bike-shedding unless we have real code to discuss.

Stefan


From stefan_ml at behnel.de  Tue Feb 24 09:38:18 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 09:38:18 +0100 (CET)
Subject: [Cython] Stefan, please fix this
In-Reply-To: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
Message-ID: <59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> +        for entry in env.default_entries:
> +            if entry.type.is_pyobject and entry.used:
> -                code.putln("Py_DECREF(%s); %s = 0;" % (
> -                        code.entry_as_pyobject(entry), entry.cname))
> +                code.put_var_decref_clear(entry)
>
> Please revert to my implementation (bare Py_DECREF and nullify), or
> any other that does not break C compilation... IMHO, The module
> cleanup function is not a place to play with refnanny. Moreover, the
> module cleanup funcion is not refnanny-managed...

I agree.


> BTW, I believe that cleanup code should be generated when running the
> testsuite...

I'd even go one step further.

http://trac.cython.org/cython_trac/ticket/218

Stefan


From greg.ewing at canterbury.ac.nz  Tue Feb 24 09:42:44 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Feb 2009 21:42:44 +1300
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090224121116.4e732c2a.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
Message-ID: <49A3B304.4070404@canterbury.ac.nz>

Anatoly A. Kazantsev wrote:

> Yes, I can use this pointer :-)
> 
> But how can I wrap callbacks without such pointers.

You can't, except in a very limited way. For example,
you could store a reference to your Python callback
in a global variable, but then you would only be able
to have one callback of a given type installed at
a time.

> Or this pointers is a 'good'
> practise in designing of callback functions?

Yes. Any competently-designed C-level callback
mechanism will provide some way for the user to have
data of his choosing passed to the callback function,
on a per-callback basis.

-- 
Greg

From robertwb at math.washington.edu  Tue Feb 24 09:48:12 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 00:48:12 -0800
Subject: [Cython] unicode can bite us ...
In-Reply-To: <49698.213.61.181.86.1235464147.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
	<49698.213.61.181.86.1235464147.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <5A78ADED-D2AF-433A-B362-0C9DEBAA7C58@math.washington.edu>

On Feb 24, 2009, at 12:29 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Feb 23, 2009, at 12:24 PM, Stefan Behnel wrote:
>>> Lisandro Dalcin wrote:
>>>> np.arange(10, dtype=s'i')
>>> There's
>>> absolutely no reason why the parameters should be byte strings.
>>
>> This is how unmarked string literals should work, right? Just like
>> the Python interpreter.
>
> Well, at least, that's what's written in the code: a byte string.  
> What I'm
> saying is that /requiring/ a byte string at the interface level is  
> wrong.

I agree on this point. I'm not as convinced that accepting a byte  
string is wrong though.

>> I think you underestimate how long broken libraries will be out  
>> there.
>
> Let's wait and see. It didn't take me very long to fix up the Py3  
> unicode
> problems of lxml's API (those that were independent of Cython), so  
> I would
> expect that any library can be fixed in a couple of weeks - as long  
> as the
> tools play nicely. I have no idea how hard it is to fix these  
> things in a
> SWIG wrapper, for example, but SWIG will also be fixed one day (and  
> Cython
> code will likely not interfere with a SWIG wrapper anyway).

I don't doubt most libraries could be made Py3 unicode compliant if  
someone were willing to spend "a couple of weeks" fixing it--I think  
it's more a question of motivation. Unicode support is something you  
care a lot about (and I'm glad you do, it's thanks to you we support  
it so well in Cython) and is also a very natural and important issue  
to deal with for an xml parser. People writing scientific libraries  
(for example) are probably more worried about endianness issues and  
fortran compatibility than unicode support, though for open source  
projects hopefully someone steps up and does it.

> Anyway, the NumPy case is not a use case for a new syntax feature,  
> as it
> requires plain byte strings for now, not identifiers. Lisandro, if  
> you can
> come up with a compelling use case that cannot easily be fixed in user
> code, I'll rethink my opposition. For now, I cannot see how this  
> feature
> can lead to anything but even more broken interfaces.

We don't need a new syntax.

def foo():
     return "Something."

should return a str object: bytes under Py2, and unicode under Py3.

- Robert


From robertwb at math.washington.edu  Tue Feb 24 09:50:41 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 00:50:41 -0800
Subject: [Cython] Stefan, please fix this
In-Reply-To: <59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>

On Feb 24, 2009, at 12:38 AM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> +        for entry in env.default_entries:
>> +            if entry.type.is_pyobject and entry.used:
>> -                code.putln("Py_DECREF(%s); %s = 0;" % (
>> -                        code.entry_as_pyobject(entry), entry.cname))
>> +                code.put_var_decref_clear(entry)
>>
>> Please revert to my implementation (bare Py_DECREF and nullify), or
>> any other that does not break C compilation... IMHO, The module
>> cleanup function is not a place to play with refnanny. Moreover, the
>> module cleanup funcion is not refnanny-managed...
>
> I agree.
>
>
>> BTW, I believe that cleanup code should be generated when running the
>> testsuite...
>
> I'd even go one step further.
>
> http://trac.cython.org/cython_trac/ticket/218

Note that the cleanup code is not (and cannot) always be safe across  
multiple modules. When objects such as interned strings, numerical  
literals, etc. are released, the module code becomes unsafe to use  
(e.g. even a __del__ method of an object laying around somewhere else.)

- Robert



From robertwb at math.washington.edu  Tue Feb 24 09:57:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 00:57:59 -0800
Subject: [Cython] string target type of struct-to-dict conversion
In-Reply-To: <49A0638F.5060409@behnel.de>
References: <49A0638F.5060409@behnel.de>
Message-ID: <6EBE2B73-CD32-4581-8741-EB9AE8B74DB8@math.washington.edu>

On Feb 21, 2009, at 12:26 PM, Stefan Behnel wrote:

> Hi,
>
> I found this code in the struct_conversion.pyx test:
>
> ----------------------------------
>>>> test_constructor_kwds(1.25, 2.5, 128)
>     {'y': 2.5, 'x': 1.25, 'color': 128}
>
> ...
>
>
> def test_dict_construction(x, y, color):
>     cdef Point p = {'color': color, 'x': x, 'y': y}
>     return p
>
> def test_constructor_kwds(x, y, color):
>     cdef Point p = Point(x=x, y=y, color=color)
>     return p
> ----------------------------------
>
> Should we really base this feature on byte strings instead of  
> unicode strings?

They should be str, i.e. byte strings in Py2, and unicode strings in  
Py3. (Basically, it's like the attribute dict of a struct).

- Robert


From anatoly at gnu.org  Tue Feb 24 09:58:39 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Tue, 24 Feb 2009 14:58:39 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
	<20090224134218.6d65f7ce.anatoly@gnu.org>
	<3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>
Message-ID: <20090224145839.6639d57b.anatoly@gnu.org>

On Mon, 23 Feb 2009 23:48:57 -0800
Robert Bradshaw <robertwb at math.washington.edu> wrote:

 
> It's standard practice to have such void* arguments for callback  
> functions. Otherwise you'd have to store something globally and not  
> use two concurrently (which is obviously not ideal).

Good practise :-)

What about such:

cpdef python_foo(python_callback):
  cdef c_callback():
    python_callback()

  c_foo(c_callback)

where c_foo and c_callback from some C lib.

I know in cython function nesting is not implemented. But if it was, is it possible
to wrap callbacks this way?
-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/141cd6fb/attachment.pgp 

From stefan_ml at behnel.de  Tue Feb 24 09:59:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 09:59:17 +0100 (CET)
Subject: [Cython] module cleanup
In-Reply-To: <3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>
Message-ID: <52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Feb 24, 2009, at 12:38 AM, Stefan Behnel wrote:
>> http://trac.cython.org/cython_trac/ticket/218
>
> Note that the cleanup code is not (and cannot) always be safe across
> multiple modules. When objects such as interned strings, numerical
> literals, etc. are released, the module code becomes unsafe to use
> (e.g. even a __del__ method of an object laying around somewhere else.)

Hmm, we're in a ref-counted environment. Why would these objects be
released when the module itself is still in use?

Stefan


From stefan_ml at behnel.de  Tue Feb 24 10:01:20 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 10:01:20 +0100 (CET)
Subject: [Cython] patch, func arg default vals, please review
In-Reply-To: <e7ba66e40902231453q1d1853bfn24db63882ad90629@mail.gmail.com>
References: <e7ba66e40902231453q1d1853bfn24db63882ad90629@mail.gmail.com>
Message-ID: <55092.213.61.181.86.1235466080.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> Take a look, I believe the fix is valid, in such case please push it.

Looks ok to me. I can't push it now, but I will do it tonight, unless
someone else is faster.


> Today I lost my self-confidence, don't like to mess the codebase again

You're still pretty new as a Cython developer. I've been working on the
code for a few years now, and I'm still learning new things about old code
from time to time.

As long as we keep reviewing code (and as long as the test suite keeps
growing), the harmfulness of these things will stay within bounds.

Stefan


From robertwb at math.washington.edu  Tue Feb 24 10:07:34 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 01:07:34 -0800
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090224145839.6639d57b.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
	<20090224134218.6d65f7ce.anatoly@gnu.org>
	<3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>
	<20090224145839.6639d57b.anatoly@gnu.org>
Message-ID: <7F8CD1E7-497F-4B83-90A1-BD0533A88EEE@math.washington.edu>

On Feb 24, 2009, at 12:58 AM, Anatoly A. Kazantsev wrote:

> On Mon, 23 Feb 2009 23:48:57 -0800
> Robert Bradshaw <robertwb at math.washington.edu> wrote:
>
>
>> It's standard practice to have such void* arguments for callback
>> functions. Otherwise you'd have to store something globally and not
>> use two concurrently (which is obviously not ideal).
>
> Good practise :-)
>
> What about such:
>
> cpdef python_foo(python_callback):
>   cdef c_callback():
>     python_callback()
>
>   c_foo(c_callback)
>
> where c_foo and c_callback from some C lib.
>
> I know in cython function nesting is not implemented. But if it  
> was, is it possible
> to wrap callbacks this way?

Yes and no. The problem is that usually the callback needs to have a  
specified signature, and a closure created like that wouldn't (it  
would need its environment to be passed in when called as well).

- Robert



From robertwb at math.washington.edu  Tue Feb 24 10:14:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 01:14:32 -0800
Subject: [Cython] module cleanup
In-Reply-To: <52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>
	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>

On Feb 24, 2009, at 12:59 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Feb 24, 2009, at 12:38 AM, Stefan Behnel wrote:
>>> http://trac.cython.org/cython_trac/ticket/218
>>
>> Note that the cleanup code is not (and cannot) always be safe across
>> multiple modules. When objects such as interned strings, numerical
>> literals, etc. are released, the module code becomes unsafe to use
>> (e.g. even a __del__ method of an object laying around somewhere  
>> else.)
>
> Hmm, we're in a ref-counted environment. Why would these objects be
> released when the module itself is still in use?


That is what the cleanup functions do--they try and release the  
memory in use by the module. The problem is that it isn't easy to  
tell when a module is still in use (as far as I can tell). It sounds  
like things have improved with Py3's unloading of modules.

A concrete example might help explain things. Say I have a Cython class

class A:
     def __del__(self):
         print 1

Now when this is compiled, a python int 1 will be created and stored  
in the module dictionary. If the module defining A is cleaned up  
before every instance of A is deallocated, then the cached Python 1  
will be collected and then at some later point A.__del__ could be  
called, which may result in a segfault.

- Robert



From stefan_ml at behnel.de  Tue Feb 24 10:34:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 10:34:47 +0100 (CET)
Subject: [Cython] unicode can bite us ...
In-Reply-To: <5A78ADED-D2AF-433A-B362-0C9DEBAA7C58@math.washington.edu>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
	<49698.213.61.181.86.1235464147.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<5A78ADED-D2AF-433A-B362-0C9DEBAA7C58@math.washington.edu>
Message-ID: <61683.213.61.181.86.1235468087.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Feb 24, 2009, at 12:29 AM, Stefan Behnel wrote:
>> Well, at least, that's what's written in the code: a byte string.
>> What I'm
>> saying is that /requiring/ a byte string at the interface level is
>> wrong.
>
> I agree on this point. I'm not as convinced that accepting a byte
> string is wrong though.

It's not wrong to /accept/ one. But it's wrong to have your code fail when
someone passes a unicode string.

I normally vote for requiring unicode strings in APIs under Py3, but
that's something that needs to be decided for each case separately.


>>> I think you underestimate how long broken libraries will be out
>>> there.
>>
>> Let's wait and see. It didn't take me very long to fix up the Py3
>> unicode
>> problems of lxml's API (those that were independent of Cython), so
>> I would
>> expect that any library can be fixed in a couple of weeks
>
> I don't doubt most libraries could be made Py3 unicode compliant if
> someone were willing to spend "a couple of weeks" fixing it

I was actually speaking in terms of spare-time weeks rather than full-time
weeks.

It's all about fixing APIs. What I'd advocate (for Cython code at least)
is to pass all API string input through a helper function that does the
right thing, and to do the same for string output. That gives you a single
place for fixing things, and it only needs to be done once. Then there's
evil things like file name handling, but that's about it.

Even ParseArgs() and friends help you by accepting unicode strings for a
char* ("s"), as long as the ASCII codec can decode them. Which is
definitely the case for NumPy arguments, for example.


> I think
> it's more a question of motivation. Unicode support is something you
> care a lot about (and I'm glad you do, it's thanks to you we support
> it so well in Cython) and is also a very natural and important issue
> to deal with for an xml parser. People writing scientific libraries
> (for example) are probably more worried about endianness issues and
> fortran compatibility than unicode support, though for open source
> projects hopefully someone steps up and does it.

I think that Py3 makes programmers a lot more aware of these issues, and
provides a growing motivation for them to fix their code. It's not
uncommon for release announcements these days to contain at least a
comment on support for Py3, if not a success story. That really makes me
confident that these problems will go away rather sooner than later.

In the specific case of NumPy, there's also the IronClad project that
attempts a port to IronPython. They'll come up with their own set of fixes
anyway. I wouldn't be surprised if some of them were related to unicode
handling.


> We don't need a new syntax.
>
> def foo():
>      return "Something."
>
> should return a str object: bytes under Py2, and unicode under Py3.

:) didn't we have this discussion already?

What if you wanted to pass the result of that function into C code?

Stefan


From anatoly at gnu.org  Tue Feb 24 10:34:53 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Tue, 24 Feb 2009 15:34:53 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <7F8CD1E7-497F-4B83-90A1-BD0533A88EEE@math.washington.edu>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
	<20090224134218.6d65f7ce.anatoly@gnu.org>
	<3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>
	<20090224145839.6639d57b.anatoly@gnu.org>
	<7F8CD1E7-497F-4B83-90A1-BD0533A88EEE@math.washington.edu>
Message-ID: <20090224153453.f7417ed1.anatoly@gnu.org>

On Tue, 24 Feb 2009 01:07:34 -0800
Robert Bradshaw <robertwb at math.washington.edu> wrote:

> Yes and no. The problem is that usually the callback needs to have a  
> specified signature, and a closure created like that wouldn't (it  
> would need its environment to be passed in when called as well).

OK, understand :-)

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/3dc7564f/attachment.pgp 

From stefan_ml at behnel.de  Tue Feb 24 14:33:10 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 14:33:10 +0100
Subject: [Cython] Stefan, please fix this
In-Reply-To: <59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49A3F716.3030503@behnel.de>


Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>> +        for entry in env.default_entries:
>> +            if entry.type.is_pyobject and entry.used:
>> -                code.putln("Py_DECREF(%s); %s = 0;" % (
>> -                        code.entry_as_pyobject(entry), entry.cname))
>> +                code.put_var_decref_clear(entry)
>>
>> Please revert to my implementation (bare Py_DECREF and nullify), or
>> any other that does not break C compilation... IMHO, The module
>> cleanup function is not a place to play with refnanny. Moreover, the
>> module cleanup funcion is not refnanny-managed...

Ok, reverted.

Stefan


From stefan_ml at behnel.de  Tue Feb 24 14:36:51 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 14:36:51 +0100
Subject: [Cython] module cleanup
In-Reply-To: <8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>
Message-ID: <49A3F7F3.5060409@behnel.de>

Hi,

Robert Bradshaw wrote:
> That is what the cleanup functions do--they try and release the  
> memory in use by the module. The problem is that it isn't easy to  
> tell when a module is still in use (as far as I can tell). It sounds  
> like things have improved with Py3's unloading of modules.

This would be a Py3-only feature. The cleanup code for Py2 would still be
optional.


> A concrete example might help explain things. Say I have a Cython class
> 
> class A:
>      def __del__(self):
>          print 1
> 
> Now when this is compiled, a python int 1 will be created and stored  
> in the module dictionary. If the module defining A is cleaned up  
> before every instance of A is deallocated, then the cached Python 1  
> will be collected and then at some later point A.__del__ could be  
> called, which may result in a segfault.

Ok, but all this means is that the class needs to hold a reference to the
module instance that created it, right?

Stefan


From stefan_ml at behnel.de  Tue Feb 24 14:39:54 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 14:39:54 +0100
Subject: [Cython] string target type of struct-to-dict conversion
In-Reply-To: <6EBE2B73-CD32-4581-8741-EB9AE8B74DB8@math.washington.edu>
References: <49A0638F.5060409@behnel.de>
	<6EBE2B73-CD32-4581-8741-EB9AE8B74DB8@math.washington.edu>
Message-ID: <49A3F8AA.40203@behnel.de>


Robert Bradshaw wrote:
> On Feb 21, 2009, at 12:26 PM, Stefan Behnel wrote:
>> I found this code in the struct_conversion.pyx test:
>>
>> ----------------------------------
>>>>> test_constructor_kwds(1.25, 2.5, 128)
>>     {'y': 2.5, 'x': 1.25, 'color': 128}
>>
>> ...
>>
>>
>> def test_dict_construction(x, y, color):
>>     cdef Point p = {'color': color, 'x': x, 'y': y}
>>     return p
>>
>> def test_constructor_kwds(x, y, color):
>>     cdef Point p = Point(x=x, y=y, color=color)
>>     return p
>> ----------------------------------
>>
>> Should we really base this feature on byte strings instead of  
>> unicode strings?
> 
> They should be str, i.e. byte strings in Py2, and unicode strings in  
> Py3. (Basically, it's like the attribute dict of a struct).

I thought about this a bit. This makes sense, as struct member names should
behave like class attributes for the above feature, i.e. they are identifiers.

+1

Stefan

From dalcinl at gmail.com  Tue Feb 24 15:01:18 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 12:01:18 -0200
Subject: [Cython] module cleanup
In-Reply-To: <49A3F7F3.5060409@behnel.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>
	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>
	<49A3F7F3.5060409@behnel.de>
Message-ID: <e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>

On Tue, Feb 24, 2009 at 11:36 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Robert Bradshaw wrote:
>> That is what the cleanup functions do--they try and release the
>> memory in use by the module. The problem is that it isn't easy to
>> tell when a module is still in use (as far as I can tell). It sounds
>> like things have improved with Py3's unloading of modules.
>
> This would be a Py3-only feature. The cleanup code for Py2 would still be
> optional.
>

However, some people could argue that cleanup code will slow-down
Python process termination.

>
>> A concrete example might help explain things. Say I have a Cython class
>>
>> class A:
>> ? ? ?def __del__(self):
>> ? ? ? ? ?print 1
>>
>> Now when this is compiled, a python int 1 will be created and stored
>> in the module dictionary. If the module defining A is cleaned up
>> before every instance of A is deallocated, then the cached Python 1
>> will be collected and then at some later point A.__del__ could be
>> called, which may result in a segfault.
>
> Ok, but all this means is that the class needs to hold a reference to the
> module instance that created it, right?
>

That would work, but such approach will introduce reference cycles
that I would like to avoid. And of course, you need a extra slot in
object struct of cdef classes...

In any case, the only way to write safe code is to avoid the usage of
module-level globals in __del__() and __dealloc__() methods... For
example, in petsc4py and slepc4py (the second depend on the first, and
the second cimport types from the first) I'm able to generate
full-cleanup code for both. But this work as long as cimported types
are not nullified (I've just pushed a fix for this, I made a mistake
in a previous commit), and just because core CPython does not
dlclose() extension modules and extension types are not heap-types
(i.e. the typeobject struct is always available on static storage).



> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Feb 24 18:58:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 18:58:12 +0100
Subject: [Cython] module cleanup
In-Reply-To: <e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>
	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>
Message-ID: <49A43534.6090905@behnel.de>

Hi,

Lisandro Dalcin wrote:
> On Tue, Feb 24, 2009 at 11:36 AM, Stefan Behnel wrote:
>> This would be a Py3-only feature. The cleanup code for Py2 would still be
>> optional.
> 
> However, some people could argue that cleanup code will slow-down
> Python process termination.

We are much too early in the design process to think about this kind of
performance issues. We can always make it optional in Py3, too, but on by
default (if it works out well, that is). Being able to cleanly unload and
reload C modules is a plain great feature that's very valuable in it's own
right. Honestly, the cleanup code is mostly there, it just needs to be
adapted to the normal GC interface.


>>> A concrete example might help explain things. Say I have a Cython class
>>>
>>> class A:
>>>      def __del__(self):
>>>          print 1
>>>
>>> Now when this is compiled, a python int 1 will be created and stored
>>> in the module dictionary. If the module defining A is cleaned up
>>> before every instance of A is deallocated, then the cached Python 1
>>> will be collected and then at some later point A.__del__ could be
>>> called, which may result in a segfault.
>> Ok, but all this means is that the class needs to hold a reference to the
>> module instance that created it, right?
> 
> That would work, but such approach will introduce reference cycles
> that I would like to avoid.

Hmm, true. When each object references its type, which in turn references
the module instance, which in turn references the type, that means that
*all* objects created in a module will end up requiring the GC due to
reference cycles.

I guess we need to raise these issues on python-dev. They'll be happy to
dig into the real-life issues anyway. I'm pretty sure we'd be the first to
actually use the new module features.


> And of course, you need a extra slot in
> object struct of cdef classes...

But that's plain cheap. It just means that each type will be copied on the
heap in the module init function (there's a simple helper function for
that), and then receive a reference to the module instance. So it's one
copy of the type per copy of the module. And we could even decide if we
really need to make a copy of the type, based on its declaration.


> In any case, the only way to write safe code is to avoid the usage of
> module-level globals in __del__() and __dealloc__() methods...

Which is a rare thing anyway. Most of the time, you'd just call some kind
of free() function and be done.


> For
> example, in petsc4py and slepc4py (the second depend on the first, and
> the second cimport types from the first) I'm able to generate
> full-cleanup code for both. But this work as long as cimported types
> are not nullified (I've just pushed a fix for this, I made a mistake
> in a previous commit), and just because core CPython does not
> dlclose() extension modules and extension types are not heap-types
> (i.e. the typeobject struct is always available on static storage).

I didn't say it would be trivial. I'm just saying that this is a valuable
goal. And Cython is definitely the right tool to achieve it.

Stefan

From dagss at student.matnat.uio.no  Tue Feb 24 19:19:21 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 24 Feb 2009 19:19:21 +0100
Subject: [Cython] module cleanup
In-Reply-To: <49A43534.6090905@behnel.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>
	<49A43534.6090905@behnel.de>
Message-ID: <49A43A29.2010503@student.matnat.uio.no>

Stefan Behnel wrote:
> Hmm, true. When each object references its type, which in turn references
> the module instance, which in turn references the type, that means that
> *all* objects created in a module will end up requiring the GC due to
> reference cycles.
> 
> I guess we need to raise these issues on python-dev. They'll be happy to
> dig into the real-life issues anyway. I'm pretty sure we'd be the first to
> actually use the new module features.
> 
>> In any case, the only way to write safe code is to avoid the usage of
>> module-level globals in __del__() and __dealloc__() methods...
> 
> Which is a rare thing anyway. Most of the time, you'd just call some kind
> of free() function and be done.

If it turns out these reference cycles will be a problem, I think we 
could always change Cython to *not* use initialized constants in __del__ 
and __dealloc__? (i.e. construct them on need; meaning a small 
performance penalty).

The global constants system could need a cleanup anyway/move to code 
generation time (low priority in itself though). If that was done, 
change the behaviour of constants in __del__ etc. would be rather easy.

http://trac.cython.org/cython_trac/ticket/144

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Feb 24 19:26:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 19:26:47 +0100
Subject: [Cython] module cleanup
In-Reply-To: <49A43A29.2010503@student.matnat.uio.no>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>
	<49A43A29.2010503@student.matnat.uio.no>
Message-ID: <49A43BE7.3080107@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Hmm, true. When each object references its type, which in turn references
>> the module instance, which in turn references the type, that means that
>> *all* objects created in a module will end up requiring the GC due to
>> reference cycles.
>>
>> I guess we need to raise these issues on python-dev. They'll be happy to
>> dig into the real-life issues anyway. I'm pretty sure we'd be the first to
>> actually use the new module features.
>>
>>> In any case, the only way to write safe code is to avoid the usage of
>>> module-level globals in __del__() and __dealloc__() methods...
>> Which is a rare thing anyway. Most of the time, you'd just call some kind
>> of free() function and be done.
> 
> If it turns out these reference cycles will be a problem, I think we 
> could always change Cython to *not* use initialized constants in __del__ 
> and __dealloc__? (i.e. construct them on need; meaning a small 
> performance penalty).

Plus, we could generate an external closure for a type (not for the
instance, just the type object) that keeps all referenced global values.
The type would then reference that instead of the module.

I so like the idea of implementing these things under the hood, instead of
requiring users to rewrite their code.


> The global constants system could need a cleanup anyway/move to code
> generation time (low priority in itself though). If that was done,
> change the behaviour of constants in __del__ etc. would be rather easy.
>
> http://trac.cython.org/cython_trac/ticket/144

Yep, I guess it makes sense to let 218 depend on this.

Stefan

From dagss at student.matnat.uio.no  Tue Feb 24 20:02:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 24 Feb 2009 20:02:50 +0100
Subject: [Cython] module cleanup
In-Reply-To: <49A43BE7.3080107@behnel.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>	<49A43A29.2010503@student.matnat.uio.no>
	<49A43BE7.3080107@behnel.de>
Message-ID: <49A4445A.9090507@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> Hmm, true. When each object references its type, which in turn references
>>> the module instance, which in turn references the type, that means that
>>> *all* objects created in a module will end up requiring the GC due to
>>> reference cycles.
>>>
>>> I guess we need to raise these issues on python-dev. They'll be happy to
>>> dig into the real-life issues anyway. I'm pretty sure we'd be the first to
>>> actually use the new module features.
>>>
>>>> In any case, the only way to write safe code is to avoid the usage of
>>>> module-level globals in __del__() and __dealloc__() methods...
>>> Which is a rare thing anyway. Most of the time, you'd just call some kind
>>> of free() function and be done.
>> If it turns out these reference cycles will be a problem, I think we 
>> could always change Cython to *not* use initialized constants in __del__ 
>> and __dealloc__? (i.e. construct them on need; meaning a small 
>> performance penalty).
> 
> Plus, we could generate an external closure for a type (not for the
> instance, just the type object) that keeps all referenced global values.
> The type would then reference that instead of the module.

Ah, nice idea; that can be done right away (= right after 0.11) without 
#144, can it not?

I.e. let all global values be referenced by a big tuple; then have each 
type and the module reference the tuple, but the tuple itself doesn't 
reference anything but the values themselves => no cycles.

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Feb 24 19:57:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 19:57:56 +0100
Subject: [Cython] module cleanup
In-Reply-To: <49A4445A.9090507@student.matnat.uio.no>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>	<49A43A29.2010503@student.matnat.uio.no>	<49A43BE7.3080107@behnel.de>
	<49A4445A.9090507@student.matnat.uio.no>
Message-ID: <49A44334.9040204@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> we could generate an external closure for a type (not for the
>> instance, just the type object) that keeps all referenced global values.
>> The type would then reference that instead of the module.
> 
> Ah, nice idea; that can be done right away (= right after 0.11) without 
> #144, can it not?

Sure. As you can see, I nominated #218 for 0.12 anyway. This would be a
perfect first step.


> I.e. let all global values be referenced by a big tuple; then have each 
> type and the module reference the tuple, but the tuple itself doesn't 
> reference anything but the values themselves => no cycles.

Although "globals" would also include types, right? So there's our
reference cycle back... (although that doesn't hit us until we really start
moving types to the heap)

Stefan

From dalcinl at gmail.com  Tue Feb 24 20:14:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 17:14:21 -0200
Subject: [Cython] module cleanup
In-Reply-To: <49A43534.6090905@behnel.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>
	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>
	<49A3F7F3.5060409@behnel.de>
	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>
	<49A43534.6090905@behnel.de>
Message-ID: <e7ba66e40902241114y462d2741i158fc16c474f5f8d@mail.gmail.com>

On Tue, Feb 24, 2009 at 3:58 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Lisandro Dalcin wrote:
>> On Tue, Feb 24, 2009 at 11:36 AM, Stefan Behnel wrote:
>>> This would be a Py3-only feature. The cleanup code for Py2 would still be
>>> optional.
>>
>> However, some people could argue that cleanup code will slow-down
>> Python process termination.
>
> We are much too early in the design process to think about this kind of
> performance issues. We can always make it optional in Py3, too, but on by
> default (if it works out well, that is).

I'm on your side here. I've just fixed my projects to generate the
module cleanup function...

> Being able to cleanly unload and
> reload C modules is a plain great feature that's very valuable in it's own
> right. Honestly, the cleanup code is mostly there, it just needs to be
> adapted to the normal GC interface.
>

Well, I believe the module init+cleanup works pretty well, see this code:

http://code.google.com/p/mpi4py/source/browse/trunk/demo/embedding/helloworld.c

As the code is run between Py_Initialize()/Py_Finalize() pairs, in
each step of the loop you have calls to the module init and cleanup
functions. Of course, this requieres some care about how Python and C
globals are initalized and managed. That's the reason you will see
that I've pushed some fixes to explicit initialize to NULL some global
stuff.


>>
>> That would work, but such approach will introduce reference cycles
>> that I would like to avoid.
>
> Hmm, true. When each object references its type, which in turn references
> the module instance, which in turn references the type, that means that
> *all* objects created in a module will end up requiring the GC due to
> reference cycles.
>
> I guess we need to raise these issues on python-dev. They'll be happy to
> dig into the real-life issues anyway. I'm pretty sure we'd be the first to
> actually use the new module features.
>

Hey! Please read this thread...  Even Guido seems to give-up on the issue ;-)

http://mail.python.org/pipermail/python-dev/2009-February/086358.html


>> And of course, you need a extra slot in
>> object struct of cdef classes...
>
> But that's plain cheap. It just means that each type will be copied on the
> heap in the module init function (there's a simple helper function for
> that), and then receive a reference to the module instance. So it's one
> copy of the type per copy of the module. And we could even decide if we
> really need to make a copy of the type, based on its declaration.
>

OK, I'll need to learn how that type-copying works...

>
>> In any case, the only way to write safe code is to avoid the usage of
>> module-level globals in __del__() and __dealloc__() methods...
>
> Which is a rare thing anyway. Most of the time, you'd just call some kind
> of free() function and be done.
>

Mmm... not sure if it is rare in all cases... It is rare when you are
wrapping libs, in other apps, perhaps it is not so rare... Again, see
the thread in Python-Dev I've mentioned above.

>
> I didn't say it would be trivial. I'm just saying that this is a valuable
> goal. And Cython is definitely the right tool to achieve it.
>

OK, let's go for it... but please, let's try hard to avoid reference cycles...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb 24 20:19:58 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 17:19:58 -0200
Subject: [Cython] module cleanup
In-Reply-To: <49A43BE7.3080107@behnel.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>
	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>
	<49A3F7F3.5060409@behnel.de>
	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>
	<49A43534.6090905@behnel.de> <49A43A29.2010503@student.matnat.uio.no>
	<49A43BE7.3080107@behnel.de>
Message-ID: <e7ba66e40902241119g26086e47g9469f834bf11ecfa@mail.gmail.com>

On Tue, Feb 24, 2009 at 4:26 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn wrote:
>>
>> If it turns out these reference cycles will be a problem, I think we
>> could always change Cython to *not* use initialized constants in __del__
>> and __dealloc__? (i.e. construct them on need; meaning a small
>> performance penalty).
>
> Plus, we could generate an external closure for a type (not for the
> instance, just the type object) that keeps all referenced global values.
> The type would then reference that instead of the module.
>

Where would you hold a reference to such closure?

> I so like the idea of implementing these things under the hood, instead of
> requiring users to rewrite their code.
>

Unless these things are related to bytes/unicode management, right?
Just joking ;-) ...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb 24 21:19:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 18:19:09 -0200
Subject: [Cython] PATCH: make kw args unpacking code compatible with C89
Message-ID: <e7ba66e40902241219qd07b0cen642ab07489dbc217@mail.gmail.com>

After GCC-compiling mpi4py with flags '-std=c89 -pedantic
-Wno-long-long', I've got a bunch of warnings like this:

warning: initializer element is not computable at load time

After googling a bit to understand the issue, I wrote the attached
patch. Please review and apply if it is OK.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: KW_UNPACK.diff
Type: application/octet-stream
Size: 1260 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/aaee6b34/attachment.obj 

From dalcinl at gmail.com  Tue Feb 24 22:44:29 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 19:44:29 -0200
Subject: [Cython] PATCH: use function pointer arguments in import/export api
	functions
Message-ID: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>

Just to prevent this:

warning: ISO C forbids conversion of function pointer to object pointer type


- Please note that I've changed the names of some local variables to
make {Import|Export}Function implementations more symmetric.

- Additionaly, note the Py_DECREF(cobj) I've added at ExportFunction,
it seems we should refnanny our "trusted" utility code ;-) ...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMP_EXP_FUNC.diff
Type: application/octet-stream
Size: 4265 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/09c3a044/attachment.obj 

From greg.ewing at canterbury.ac.nz  Tue Feb 24 23:24:25 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 25 Feb 2009 11:24:25 +1300
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090224145839.6639d57b.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
	<20090224134218.6d65f7ce.anatoly@gnu.org>
	<3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>
	<20090224145839.6639d57b.anatoly@gnu.org>
Message-ID: <49A47399.3070009@canterbury.ac.nz>

Anatoly A. Kazantsev wrote:

> cpdef python_foo(python_callback):
>   cdef c_callback():
>     python_callback()
> 
>   c_foo(c_callback)
> 
> where c_foo and c_callback from some C lib.
> 
> I know in cython function nesting is not implemented. But if it was, is it possible
> to wrap callbacks this way?

No, this would not be possible, at least not without some
very platform-dependent hackery[1]. The callback needs to be
a plain C function, and there is no such thing as a closure
in plain C.

Cython might support nested C functions one day, but if
they behave like closures, they won't be acceptable to
something expecting a standard C function pointer.

Footnotes:

[1] It could be done if you're willing to dynamically
generate machine code for a new function each time you
want a new instance of the callback, but that would
hardly be portable.

-- 
Greg

From robertwb at math.washington.edu  Tue Feb 24 23:32:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 14:32:45 -0800
Subject: [Cython] module cleanup
In-Reply-To: <49A43BE7.3080107@behnel.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>
	<49A43A29.2010503@student.matnat.uio.no>
	<49A43BE7.3080107@behnel.de>
Message-ID: <0C456D71-9369-42C2-9C07-E19784A7D5FD@math.washington.edu>

On Feb 24, 2009, at 10:26 AM, Stefan Behnel wrote:

> Hi,
>
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> Hmm, true. When each object references its type, which in turn  
>>> references
>>> the module instance, which in turn references the type, that  
>>> means that
>>> *all* objects created in a module will end up requiring the GC  
>>> due to
>>> reference cycles.
>>>
>>> I guess we need to raise these issues on python-dev. They'll be  
>>> happy to
>>> dig into the real-life issues anyway. I'm pretty sure we'd be the  
>>> first to
>>> actually use the new module features.
>>>
>>>> In any case, the only way to write safe code is to avoid the  
>>>> usage of
>>>> module-level globals in __del__() and __dealloc__() methods...
>>> Which is a rare thing anyway. Most of the time, you'd just call  
>>> some kind
>>> of free() function and be done.

A more common case is that is uses an interned string (e.g. to lookup  
something in a dictionary).

>> If it turns out these reference cycles will be a problem, I think we
>> could always change Cython to *not* use initialized constants in  
>> __del__
>> and __dealloc__? (i.e. construct them on need; meaning a small
>> performance penalty).

Arbitrary code may be called from a __del__ and/or __dealloc__, how  
far would one go? Sometimes it's better to not special case just part  
of the problem.

> Plus, we could generate an external closure for a type (not for the
> instance, just the type object) that keeps all referenced global  
> values.
> The type would then reference that instead of the module.
>
> I so like the idea of implementing these things under the hood,  
> instead of
> requiring users to rewrite their code.

I am solidly against adding another slot to cdef classes that  
reference their type/module, especially as it requires the class to  
be GC'd and creates cyclic references (which can have significant  
performance penalties in Python).

Remember, this only happens right before the whole heap is  
deallocated, so the main use cases are to reduce noise during  
valgrinding and similar tools.

- Robert


From robertwb at math.washington.edu  Tue Feb 24 23:36:02 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 14:36:02 -0800
Subject: [Cython] unicode can bite us ...
In-Reply-To: <61683.213.61.181.86.1235468087.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
	<49698.213.61.181.86.1235464147.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<5A78ADED-D2AF-433A-B362-0C9DEBAA7C58@math.washington.edu>
	<61683.213.61.181.86.1235468087.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <1906BE75-BCC2-4171-83F6-18601F7BFEEF@math.washington.edu>

On Feb 24, 2009, at 1:34 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Feb 24, 2009, at 12:29 AM, Stefan Behnel wrote:
>>> Well, at least, that's what's written in the code: a byte string.
>>> What I'm
>>> saying is that /requiring/ a byte string at the interface level is
>>> wrong.
>>
>> I agree on this point. I'm not as convinced that accepting a byte
>> string is wrong though.
>
> It's not wrong to /accept/ one. But it's wrong to have your code  
> fail when
> someone passes a unicode string.

Yes, this is what I was agreeing to.

> I normally vote for requiring unicode strings in APIs under Py3, but
> that's something that needs to be decided for each case separately.
>
>
>>>> I think you underestimate how long broken libraries will be out
>>>> there.
>>>
>>> Let's wait and see. It didn't take me very long to fix up the Py3
>>> unicode
>>> problems of lxml's API (those that were independent of Cython), so
>>> I would
>>> expect that any library can be fixed in a couple of weeks
>>
>> I don't doubt most libraries could be made Py3 unicode compliant if
>> someone were willing to spend "a couple of weeks" fixing it
>
> I was actually speaking in terms of spare-time weeks rather than  
> full-time
> weeks.
>
> It's all about fixing APIs. What I'd advocate (for Cython code at  
> least)
> is to pass all API string input through a helper function that does  
> the
> right thing, and to do the same for string output. That gives you a  
> single
> place for fixing things, and it only needs to be done once. Then  
> there's
> evil things like file name handling, but that's about it.
>
> Even ParseArgs() and friends help you by accepting unicode strings  
> for a
> char* ("s"), as long as the ASCII codec can decode them. Which is
> definitely the case for NumPy arguments, for example.


>> We don't need a new syntax.
>>
>> def foo():
>>      return "Something."
>>
>> should return a str object: bytes under Py2, and unicode under Py3.
>
> :) didn't we have this discussion already?

Yep, so I'll try not to belabor the point.

> What if you wanted to pass the result of that function into C code?

Personally, I would either (1) declare for once and for all that  
PyUnicode objects <-> char* conversion always happens via UTF-8 or  
(2) allow implicit conversion, raising an error for non 7-bit ascii  
either way.

- Robert


From michael.abshoff at googlemail.com  Tue Feb 24 23:41:20 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 24 Feb 2009 14:41:20 -0800
Subject: [Cython] module cleanup
In-Reply-To: <0C456D71-9369-42C2-9C07-E19784A7D5FD@math.washington.edu>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>	<49A43A29.2010503@student.matnat.uio.no>	<49A43BE7.3080107@behnel.de>
	<0C456D71-9369-42C2-9C07-E19784A7D5FD@math.washington.edu>
Message-ID: <49A47790.4040804@gmail.com>

Robert Bradshaw wrote:
> On Feb 24, 2009, at 10:26 AM, Stefan Behnel wrote:
> 

<SNIP>

>> I so like the idea of implementing these things under the hood,  
>> instead of
>> requiring users to rewrite their code.
> 
> I am solidly against adding another slot to cdef classes that  
> reference their type/module, especially as it requires the class to  
> be GC'd and creates cyclic references (which can have significant  
> performance penalties in Python).
> 
> Remember, this only happens right before the whole heap is  
> deallocated, so the main use cases are to reduce noise during  
> valgrinding and similar tools.

+1

> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From carl.witty at gmail.com  Wed Feb 25 00:03:42 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Tue, 24 Feb 2009 15:03:42 -0800
Subject: [Cython] Wrapping callbacks
In-Reply-To: <49A47399.3070009@canterbury.ac.nz>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
	<20090224134218.6d65f7ce.anatoly@gnu.org>
	<3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>
	<20090224145839.6639d57b.anatoly@gnu.org>
	<49A47399.3070009@canterbury.ac.nz>
Message-ID: <ac1e4c420902241503l52ac491clab4d19d1e9546bb7@mail.gmail.com>

On Tue, Feb 24, 2009 at 2:24 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> > [1] It could be done if you're willing
to dynamically
> generate machine code for a new function each time you
> want a new instance of the callback, but that would
> hardly be portable.

Well, it's somewhat portable, if you rely on somebody else to write the code :)

The libffi library has routines to write the required machine language
stub that work on a lot of different machines (but probably less
places than Python works).  The Python ctypes library (a standard part
of Python since Python 2.5) depends on (and includes) libffi to let
you create C function pointers that call an arbitrary Python callable,
so this should work for Cython functions.  (I've never tested any of
this, only read the documentation.)

This isn't a good idea for general programming; if it's at all
possible to change the callback interface, that would be better.
Using ctypes this way will definitely reduce the portability of your
code (to Python 2.5+, and to machines that have libffi callback
support).

Carl

From dalcinl at gmail.com  Wed Feb 25 01:32:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 21:32:17 -0300
Subject: [Cython] PATCH: use function pointer arguments in import/export
	api functions
In-Reply-To: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>
References: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>
Message-ID: <e7ba66e40902241632p2600988ay1c0f20b0dbcec8ff@mail.gmail.com>

Even if the patch is not accepted, I've pushed the refcount fix,
Cython was leaking a reference to PyCObject at every export of a cdef
api function.

BTW, I've added a testcase for this, using sys.getrefcount... any
objections regarding its implementation?


On Tue, Feb 24, 2009 at 6:44 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Just to prevent this:
>
> warning: ISO C forbids conversion of function pointer to object pointer type
>
>
> - Please note that I've changed the names of some local variables to
> make {Import|Export}Function implementations more symmetric.
>
> - Additionaly, note the Py_DECREF(cobj) I've added at ExportFunction,
> it seems we should refnanny our "trusted" utility code ;-) ...
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 25 03:59:34 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Feb 2009 23:59:34 -0300
Subject: [Cython] __dealloc__ generation, please comment this ...
Message-ID: <e7ba66e40902241859s5c5a2b47v1a0f2b1f88205b98@mail.gmail.com>

AFAIK, cdef classes can only have one base, right? This would let us
get rid of module-global types at deallocators... Am I missing
something?

diff -r edbbb70369d7 Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py	Tue Feb 24 21:31:02 2009 -0300
+++ b/Cython/Compiler/ModuleNode.py	Tue Feb 24 23:53:26 2009 -0300
@@ -943,7 +943,7 @@
         if base_type:
             tp_dealloc = TypeSlots.get_base_slot_function(scope, tp_slot)
             if tp_dealloc is None:
-                tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname
+                tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc"
             code.putln(
                     "%s(o);" % tp_dealloc)
         else:


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Feb 25 04:49:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Feb 2009 19:49:39 -0800
Subject: [Cython] __dealloc__ generation, please comment this ...
In-Reply-To: <e7ba66e40902241859s5c5a2b47v1a0f2b1f88205b98@mail.gmail.com>
References: <e7ba66e40902241859s5c5a2b47v1a0f2b1f88205b98@mail.gmail.com>
Message-ID: <E19919B8-5221-43B7-A878-91E19A8E8C7A@math.washington.edu>

On Feb 24, 2009, at 6:59 PM, Lisandro Dalcin wrote:

> AFAIK, cdef classes can only have one base, right? This would let us
> get rid of module-global types at deallocators... Am I missing
> something?

Yes, you're missing two things. First, the deallocation needs to call  
up the chain. Suppose We have classes A -> B -> C. Then if o is of  
type C, Py_TYPE(o)->tp_base will always be B. What one needs to do is  
have C's deallocator call B's, and B's deallocator call A.

Second, this would slow down deallocation of every object (which  
happens a lot), which IMHO is more important.

Also, it's not enough to make the deallocator safe, one would need to  
make any code that can be called by any deallocator safe.

> diff -r edbbb70369d7 Cython/Compiler/ModuleNode.py
> --- a/Cython/Compiler/ModuleNode.py	Tue Feb 24 21:31:02 2009 -0300
> +++ b/Cython/Compiler/ModuleNode.py	Tue Feb 24 23:53:26 2009 -0300
> @@ -943,7 +943,7 @@
>          if base_type:
>              tp_dealloc = TypeSlots.get_base_slot_function(scope,  
> tp_slot)
>              if tp_dealloc is None:
> -                tp_dealloc = "%s->tp_dealloc" %  
> base_type.typeptr_cname
> +                tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc"
>              code.putln(
>                      "%s(o);" % tp_dealloc)
>          else:
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ondrej at certik.cz  Wed Feb 25 05:32:41 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 24 Feb 2009 20:32:41 -0800
Subject: [Cython] Fortran types in C
In-Reply-To: <A6DE077C-30DF-4807-B390-CFB7B7AFEE09@math.washington.edu>
References: <49705534.2030003@student.matnat.uio.no>
	<A6DE077C-30DF-4807-B390-CFB7B7AFEE09@math.washington.edu>
Message-ID: <85b5c3130902242032w319ce355yd9891ff29f7c02cf@mail.gmail.com>

On Fri, Jan 16, 2009 at 11:43 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Jan 16, 2009, at 1:36 AM, Dag Sverre Seljebotn wrote:
>
>> As I have to interface a lot with some Fortran code these days, and a
>> full-fledged Cython<->Fortran-binding *may* come out of it (if I
>> continue to manage using Cython for this task it would be a likely
>> result in about a year, at least I structure my hacky scripts in that
>> direction),
>
> I know virtually nothing about Fortran, but this sure sounds cool and
> I'm sure the number-crunching communities would really like it.

I was just thinking about this too. So far I use f2py, but I would
much prefer the Cython approach.

Could you make your scripts public?

Ondrej

From stefan_ml at behnel.de  Wed Feb 25 08:19:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Feb 2009 08:19:50 +0100
Subject: [Cython] module cleanup
In-Reply-To: <e7ba66e40902241119g26086e47g9469f834bf11ecfa@mail.gmail.com>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>
	<49A43A29.2010503@student.matnat.uio.no>	<49A43BE7.3080107@behnel.de>
	<e7ba66e40902241119g26086e47g9469f834bf11ecfa@mail.gmail.com>
Message-ID: <49A4F116.6020604@behnel.de>


Lisandro Dalcin wrote:
> On Tue, Feb 24, 2009 at 4:26 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Dag Sverre Seljebotn wrote:
>>> If it turns out these reference cycles will be a problem, I think we
>>> could always change Cython to *not* use initialized constants in __del__
>>> and __dealloc__? (i.e. construct them on need; meaning a small
>>> performance penalty).
>> Plus, we could generate an external closure for a type (not for the
>> instance, just the type object) that keeps all referenced global values.
>> The type would then reference that instead of the module.
> 
> Where would you hold a reference to such closure?

The type itself would hold it. During module creation, all types that
require GC will be copied to the heap, and their closure will be
initialised. That way, the module would hold a reference to a complete set
of all globals, types and functions, and the types would hold a reference
only to their own closure. No reference cycles so far.

Module-level functions would not need a closure as they get their module
reference passed as self argument and can use it to reference the right
globals. Cdef module functions are a problem, though, as they don't have a
self argument. Not sure how to deal with that.


>> I so like the idea of implementing these things under the hood, instead of
>> requiring users to rewrite their code.
> 
> Unless these things are related to bytes/unicode management, right?
> Just joking ;-) ...

I hope so. Unicode handling isn't about additional functionality, it's
about correctness and safety.

Stefan


From stefan_ml at behnel.de  Wed Feb 25 08:34:42 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Feb 2009 08:34:42 +0100
Subject: [Cython] PATCH: make kw args unpacking code compatible with C89
In-Reply-To: <e7ba66e40902241219qd07b0cen642ab07489dbc217@mail.gmail.com>
References: <e7ba66e40902241219qd07b0cen642ab07489dbc217@mail.gmail.com>
Message-ID: <49A4F492.9060500@behnel.de>


Lisandro Dalcin wrote:
> After GCC-compiling mpi4py with flags '-std=c89 -pedantic
> -Wno-long-long', I've got a bunch of warnings like this:
> 
> warning: initializer element is not computable at load time
> 
> After googling a bit to understand the issue, I wrote the attached
> patch. Please review and apply if it is OK.

Would it work to write this kind of code:

    PyObject* values[N] = {0,0,...,0}; // all 0!
    values[3] = non_NULL_default_value3;
    values[8] = non_NULL_default_value8;

instead of setting /all/ values separately? I would like to avoid
unnecessary code here, but seeing the default values directly could even
make things more readable.

Stefan

From dagss at student.matnat.uio.no  Wed Feb 25 09:10:49 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 25 Feb 2009 09:10:49 +0100
Subject: [Cython] Fortran types in C
In-Reply-To: <85b5c3130902242032w319ce355yd9891ff29f7c02cf@mail.gmail.com>
References: <49705534.2030003@student.matnat.uio.no>	<A6DE077C-30DF-4807-B390-CFB7B7AFEE09@math.washington.edu>
	<85b5c3130902242032w319ce355yd9891ff29f7c02cf@mail.gmail.com>
Message-ID: <49A4FD09.8050306@student.matnat.uio.no>

Ondrej Certik wrote:
> On Fri, Jan 16, 2009 at 11:43 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Jan 16, 2009, at 1:36 AM, Dag Sverre Seljebotn wrote:
>>
>>> As I have to interface a lot with some Fortran code these days, and a
>>> full-fledged Cython<->Fortran-binding *may* come out of it (if I
>>> continue to manage using Cython for this task it would be a likely
>>> result in about a year, at least I structure my hacky scripts in that
>>> direction),
>> I know virtually nothing about Fortran, but this sure sounds cool and
>> I'm sure the number-crunching communities would really like it.
> 
> I was just thinking about this too. So far I use f2py, but I would
> much prefer the Cython approach.
> 
> Could you make your scripts public?

I'm afraid I ended up postponing it and wrapped only the Fortran parts I 
really needed right away manually instead (by writing Cython wrappers 
which pass the arguments by address and does the necesarry NumPy things; 
and by creating an f90 wrapper for calling functions with assumed-shape 
array arguments). So there's nothing usable in the sense of code, I've 
mostly been thinking about how to do it.

So if it is me who's doing it, my wild guess is that this will happen 
either in summer or during autumn (so within the end of 2009, but 
perhaps not very much before).

-- 
Dag Sverre

From moguillansky at gmx.de  Wed Feb 25 09:11:54 2009
From: moguillansky at gmx.de (Eduardo Moguillansky)
Date: Wed, 25 Feb 2009 09:11:54 +0100
Subject: [Cython] C parameter to __cinit__
Message-ID: <49A4FD4A.9030804@gmx.de>

Hallo.

I am working on python bindings for the SDIF library (sdif stands for 
sound description interchange format) via Cython.

I am creating a class in cython which should wrap a C-struct called 
SdifMatrixDataT. Instances of this class can be created either 
internally in Cython or in Python. When creating them in Cython I would 
like to have either a constructor or a class method, so that I can write 
code like


cdef SdifMatrixDataT *m = SdifFReadMatrixData(sdif_file)
return Matrix(m)

where Matrix should be:

cdef class Matrix:
    cdef SdifMatrixDataT *this
    def __cinit__(self, SdifMatrixDataT *matrix):
        self.this = matrix

but Cython does not allow that, complaining that it cannot convert m to 
a python object. It does not allow also for cdefs methods to be 
classmethods or staticmethods.
The only way around was to create a c function outside of this class to 
instantiate the Matrix:

cdef Matrix Matrix_new(SdifMatrixDataT *matrix):
    cdef Matrix m = Matrix()
    m.this = matrix
    return m

This works but it is not so elegant.
Am I overseeing something obvious or is there no other way around this?

best,
eduardo

From stefan_ml at behnel.de  Wed Feb 25 09:36:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Feb 2009 09:36:46 +0100 (CET)
Subject: [Cython] C parameter to __cinit__
In-Reply-To: <49A4FD4A.9030804@gmx.de>
References: <49A4FD4A.9030804@gmx.de>
Message-ID: <55916.213.61.181.86.1235551006.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Eduardo Moguillansky wrote:
> I am working on python bindings for the SDIF library (sdif stands for
> sound description interchange format) via Cython.
>
> I am creating a class in cython which should wrap a C-struct called
> SdifMatrixDataT. Instances of this class can be created either
> internally in Cython or in Python. When creating them in Cython I would
> like to have either a constructor or a class method

It's good practice to use a factory instead that initialises the C
attributes of the instance manually.

http://wiki.cython.org/FAQ#head-9d29572410f6953f692486641dc555ac184cf70b

Note that the PY_NEW() macro will not call __init__(), so this is a very
good way to distinguish between a user originated class instantiation and
one from within your own internal module code.

Stefan


From dagss at student.matnat.uio.no  Wed Feb 25 10:02:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 25 Feb 2009 10:02:00 +0100
Subject: [Cython] Fortran types in C
In-Reply-To: <70bfb5794dc767535a8679334bc64028.squirrel@webmail.uio.no>
References: <70bfb5794dc767535a8679334bc64028.squirrel@webmail.uio.no>
Message-ID: <49A50908.3070303@student.matnat.uio.no>

Sturla Molden wrote:
>> What is the best way of declaring Fortran types in C? I understand there
>> is likely no perfect cross-platform solution here
> 
> ISO Fortran 2003 has standard bindings to ISO C. Most Fortran compilers
> support them, including gfortran. These C bindings make C callable from
> Fortran, but they also make Fortran callable from C in a standardized and
> portable way.
> 
> The short receipe is: you don't declare any Fortran types in C. You
> declare your Fortran functions with C types. This even includes
> pass-by-value calling convention. Then the Fortran compiler will do the
> correct conversion.

Thanks! (I'm rather new to Fortran...)

Ideally one should be able to point Cython to a Fortran library and 
start using it automatically. A good solution then seems to be to have 
Cython generate a Fortran 2003 wrapper around Fortran code (of any 
dialect), retyping the arguments as the ISO C types.

However one would still need some hint about which C types correspond to 
which Fortran types (in size, though they won't be the same type) to be 
able to generate such a wrapper automatically? Any ideas? (I suppose 
worst-case is that we can ship a configure script which generates an 
include-file with the right definitions)

-- 
Dag Sverre

From moguillansky at gmx.de  Wed Feb 25 09:49:31 2009
From: moguillansky at gmx.de (Eduardo Moguillansky)
Date: Wed, 25 Feb 2009 09:49:31 +0100
Subject: [Cython] C parameter to __cinit__
In-Reply-To: <55916.213.61.181.86.1235551006.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <49A4FD4A.9030804@gmx.de>
	<55916.213.61.181.86.1235551006.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49A5061B.8050601@gmx.de>

Stefan Behnel wrote:
> Eduardo Moguillansky wrote:
>   
>> I am working on python bindings for the SDIF library (sdif stands for
>> sound description interchange format) via Cython.
>>
>> I am creating a class in cython which should wrap a C-struct called
>> SdifMatrixDataT. Instances of this class can be created either
>> internally in Cython or in Python. When creating them in Cython I would
>> like to have either a constructor or a class method
>>     
>
> It's good practice to use a factory instead that initialises the C
> attributes of the instance manually.
>
> http://wiki.cython.org/FAQ#head-9d29572410f6953f692486641dc555ac184cf70b
>
> Note that the PY_NEW() macro will not call __init__(), so this is a very
> good way to distinguish between a user originated class instantiation and
> one from within your own internal module code.
>
> Stefan
>   
Ok, thanks for clarifying that.

From dagss at student.matnat.uio.no  Wed Feb 25 10:59:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 25 Feb 2009 10:59:02 +0100
Subject: [Cython] Fortran types in C
In-Reply-To: <49A50908.3070303@student.matnat.uio.no>
References: <70bfb5794dc767535a8679334bc64028.squirrel@webmail.uio.no>
	<49A50908.3070303@student.matnat.uio.no>
Message-ID: <49A51666.2070401@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Sturla Molden wrote:
>>> What is the best way of declaring Fortran types in C? I understand there
>>> is likely no perfect cross-platform solution here
>> ISO Fortran 2003 has standard bindings to ISO C. Most Fortran compilers
>> support them, including gfortran. These C bindings make C callable from
>> Fortran, but they also make Fortran callable from C in a standardized and
>> portable way.
>>
>> The short receipe is: you don't declare any Fortran types in C. You
>> declare your Fortran functions with C types. This even includes
>> pass-by-value calling convention. Then the Fortran compiler will do the
>> correct conversion.
> 
> Thanks! (I'm rather new to Fortran...)
> 
> Ideally one should be able to point Cython to a Fortran library and 
> start using it automatically. A good solution then seems to be to have 
> Cython generate a Fortran 2003 wrapper around Fortran code (of any 
> dialect), retyping the arguments as the ISO C types.

Actually, I suppose chasm might help me there and do much of this. I'll 
have to look closer at it. Thanks again.

-- 
Dag Sverre

From sturla at molden.no  Wed Feb 25 12:33:45 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 25 Feb 2009 12:33:45 +0100
Subject: [Cython] Fortran types in C
In-Reply-To: <49A50908.3070303@student.matnat.uio.no>
References: <70bfb5794dc767535a8679334bc64028.squirrel@webmail.uio.no>
	<49A50908.3070303@student.matnat.uio.no>
Message-ID: <49A52C99.3010907@molden.no>

On 2/25/2009 10:02 AM, Dag Sverre Seljebotn wrote:

> However one would still need some hint about which C types correspond to 
> which Fortran types (in size, though they won't be the same type) to be 
> able to generate such a wrapper automatically? Any ideas?

Fortran has the intrinsic methods "selected_int_kind" and 
"selected_real_kind" for obtaining the compiler's "kind numbers" for 
given values of resolution or precision.

module kindtypes
    use iso_c_binding
    intrinsic :: kind, selected_int_kind, selected_real_kind
    integer(c_int) ::  i8 = selected_int_kind(2)
    integer(c_int) :: i16 = selected_int_kind(4)
    integer(c_int) :: i32 = selected_int_kind(9)
    integer(c_int) :: i64 = selected_int_kind(16)
    integer(c_int) :: r32 = selected_real_kind(p=5)
    integer(c_int) :: r64 = selected_real_kind(p=13)
    integer(c_int) :: c32 = selected_real_kind(p=5)
    integer(c_int) :: c64 = selected_real_kind(p=13)
    bind(C) :: i8, i16, i32, i64, r32, r64, c32, c64
end module

And now in C:

extern int i32; /* the kind of a 32-bit integer in Fortran */

In Cython you could create dictionaries mapping "kind to size" and "size 
to kind".

If you write a Fortran 2003 for your Fortran 77 or 90 library, you do 
not have to worry about compiler specific details when calling from C. 
This includes functions for converting between C pointers and Fortran 
pointers. You can e.g. pass a Fortran pointer in place of an 
"assumed-shape array" in Fortran 90. Chasm will also give you the "dope 
array struct" if it supports the Fortran compiler (vendor and version). 
But with Fortran 2003, you do not have to worry about this, you can 
leave this mess to the Fortran compiler.



Sturla Molden




















From dalcinl at gmail.com  Wed Feb 25 14:36:48 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 25 Feb 2009 11:36:48 -0200
Subject: [Cython] PATCH: make kw args unpacking code compatible with C89
In-Reply-To: <49A4F492.9060500@behnel.de>
References: <e7ba66e40902241219qd07b0cen642ab07489dbc217@mail.gmail.com>
	<49A4F492.9060500@behnel.de>
Message-ID: <e7ba66e40902250536u3b007da2yd2efacbb06cdc4ee@mail.gmail.com>

On Wed, Feb 25, 2009 at 5:34 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>> After GCC-compiling mpi4py with flags '-std=c89 -pedantic
>> -Wno-long-long', I've got a bunch of warnings like this:
>>
>> warning: initializer element is not computable at load time
>>
>> After googling a bit to understand the issue, I wrote the attached
>> patch. Please review and apply if it is OK.
>
> Would it work to write this kind of code:
>
> ? ?PyObject* values[N] = {0,0,...,0}; // all 0!
> ? ?values[3] = non_NULL_default_value3;
> ? ?values[8] = non_NULL_default_value8;
>

Yes, it would work ...

> instead of setting /all/ values separately? I would like to avoid
> unnecessary code here, but seeing the default values directly could even
> make things more readable.

I am +1 on your new way... If you agree,  I'll implement that and push ...


>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Feb 25 15:09:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Feb 2009 15:09:17 +0100 (CET)
Subject: [Cython] PATCH: make kw args unpacking code compatible with C89
In-Reply-To: <e7ba66e40902250536u3b007da2yd2efacbb06cdc4ee@mail.gmail.com>
References: <e7ba66e40902241219qd07b0cen642ab07489dbc217@mail.gmail.com>
	<49A4F492.9060500@behnel.de>
	<e7ba66e40902250536u3b007da2yd2efacbb06cdc4ee@mail.gmail.com>
Message-ID: <58231.213.61.181.86.1235570957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> I am +1 on your new way... If you agree,  I'll implement that and push ...

I already implemented it that way, just wait until it's up.

Stefan


From robertwb at math.washington.edu  Wed Feb 25 21:18:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 25 Feb 2009 12:18:36 -0800
Subject: [Cython] PATCH: make kw args unpacking code compatible with C89
In-Reply-To: <e7ba66e40902250536u3b007da2yd2efacbb06cdc4ee@mail.gmail.com>
References: <e7ba66e40902241219qd07b0cen642ab07489dbc217@mail.gmail.com>
	<49A4F492.9060500@behnel.de>
	<e7ba66e40902250536u3b007da2yd2efacbb06cdc4ee@mail.gmail.com>
Message-ID: <0911749B-4D90-4B4D-9222-394FB931E7B7@math.washington.edu>

On Feb 25, 2009, at 5:36 AM, Lisandro Dalcin wrote:

> On Wed, Feb 25, 2009 at 5:34 AM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>>
>> Lisandro Dalcin wrote:
>>> After GCC-compiling mpi4py with flags '-std=c89 -pedantic
>>> -Wno-long-long', I've got a bunch of warnings like this:
>>>
>>> warning: initializer element is not computable at load time
>>>
>>> After googling a bit to understand the issue, I wrote the attached
>>> patch. Please review and apply if it is OK.
>>
>> Would it work to write this kind of code:
>>
>>    PyObject* values[N] = {0,0,...,0}; // all 0!
>>    values[3] = non_NULL_default_value3;
>>    values[8] = non_NULL_default_value8;
>>
>
> Yes, it would work ...
>
>> instead of setting /all/ values separately? I would like to avoid
>> unnecessary code here, but seeing the default values directly  
>> could even
>> make things more readable.

I would assume the compiler optimizes the statement to be as good as  
before?

- Robert



From stefan_ml at behnel.de  Wed Feb 25 21:33:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Feb 2009 21:33:23 +0100
Subject: [Cython] PATCH: make kw args unpacking code compatible with C89
In-Reply-To: <0911749B-4D90-4B4D-9222-394FB931E7B7@math.washington.edu>
References: <e7ba66e40902241219qd07b0cen642ab07489dbc217@mail.gmail.com>	<49A4F492.9060500@behnel.de>	<e7ba66e40902250536u3b007da2yd2efacbb06cdc4ee@mail.gmail.com>
	<0911749B-4D90-4B4D-9222-394FB931E7B7@math.washington.edu>
Message-ID: <49A5AB13.6070007@behnel.de>


Robert Bradshaw wrote:
>> On Wed, Feb 25, 2009 at 5:34 AM, Stefan Behnel wrote:
>>>    PyObject* values[N] = {0,0,...,0}; // all 0!
>>>    values[3] = non_NULL_default_value3;
>>>    values[8] = non_NULL_default_value8;
> 
> I would assume the compiler optimizes the statement to be as good as  
> before?

That was my intuition, too. And my disassembler tells me that it avoids
setting anything more than once.

Stefan


From dalcinl at gmail.com  Thu Feb 26 16:03:12 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 26 Feb 2009 13:03:12 -0200
Subject: [Cython] about commit 1774:7dd195513552
Message-ID: <e7ba66e40902260703l9e22914gaad24e8801fcd2@mail.gmail.com>

Robert, about this commit...

changeset:   1774:7dd195513552
user:        Robert Bradshaw <robertwb at math.washington.edu>
date:        Wed Feb 25 12:25:16 2009 -0800
summary:     Avoid crashes in refnanny when PyFinalize gets messy,
Sage now exits cleanly.

Could you point me to the pieces of Cython code where you had these
problems at Py_Finalize() ?

I'm fine with the patch, but IMHO correct usage of atexit module or
Py_AtExit() should not need the precautions you took.

1) Python functions registered with atexit module are called before
any cleanup. Moreover, Py_IsInitialized() should return true on any of
them.

2) C functions registered with Py_AtExit() are called when all Python
state has gone. Then, they should be pure C functions, no Python C-API
calls or Python objects involved. The only safe way for you to ensure
this is to declare them using 'nogil'. And in such case, the code in
not refnanny managed...

Any chance that some code in Sage does not takes (2) into account??

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From mikael.arvids at enea.com  Thu Feb 26 16:52:26 2009
From: mikael.arvids at enea.com (Mikael Arvids)
Date: Thu, 26 Feb 2009 16:52:26 +0100
Subject: [Cython] ImportError with cimport within package
Message-ID: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>

Hi all,

I'm trying to create package compiled with Cython. It compiles fine but when I try to import it I get an ImportError exception. Here is a simplifed version which generates the same error:

The structure of the compiled package:
  package/
    __init__.py    - Empty
    module_A.pyd
    module_B.pyd
  
Content of the Cython modules:
  module_B.pyx

    cdef class BaseClass:
      ...
      
  module_B.pxd

    cdef class BaseClass:
      pass
    
  module_A.pyx

    cimport module_B

    cdef class SubClass(module_B.BaseClass):
      ...

Content of my test script:
  from package.module_A import Subclass

When I run the test script I get "ImportError: No module named module_B". I works if I put module_B.pyd in the same folder as the test script. I don't understand why since module_A which contains the subclass of the class in module_B is in the same folder as module_B, and there exist no Python imports from the module (which I thought would work as well).

Any suggestions?

(I use Cython 0.10.3, Python 2.5.1 and MSVS C++ 7.1)

Best regards,
Mikael Arvids
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090226/6ce38c75/attachment.htm 

From stefan_ml at behnel.de  Thu Feb 26 13:23:16 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Feb 2009 13:23:16 +0100
Subject: [Cython] PATCH: use function pointer arguments in import/export
 api	functions
In-Reply-To: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>
References: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>
Message-ID: <49A689B4.1000104@behnel.de>


Lisandro Dalcin wrote:
> Just to prevent this:
> 
> warning: ISO C forbids conversion of function pointer to object pointer type
> 
> - Please note that I've changed the names of some local variables to
> make {Import|Export}Function implementations more symmetric.

Fine with me. If this makes things more portable, and it doesn't look like
it breaks anything - just go and push.

BTW, it's good that that you split the patch into two.

Stefan


From stefan_ml at behnel.de  Thu Feb 26 16:37:29 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Feb 2009 16:37:29 +0100
Subject: [Cython] module cleanup
In-Reply-To: <0C456D71-9369-42C2-9C07-E19784A7D5FD@math.washington.edu>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>	<49A43A29.2010503@student.matnat.uio.no>	<49A43BE7.3080107@behnel.de>
	<0C456D71-9369-42C2-9C07-E19784A7D5FD@math.washington.edu>
Message-ID: <49A6B739.30109@behnel.de>


Robert Bradshaw wrote:
> On Feb 24, 2009, at 10:26 AM, Stefan Behnel wrote:
>>>>> In any case, the only way to write safe code is to avoid the  
>>>>> usage of
>>>>> module-level globals in __del__() and __dealloc__() methods...
>>>> Which is a rare thing anyway. Most of the time, you'd just call  
>>>> some kind of free() function and be done.
> 
> A more common case is that is uses an interned string (e.g. to lookup  
> something in a dictionary).

Could we disable things like this at least for __dealloc__() methods? I.e.
is there a reason __dealloc__() should not have nogil restrictions?

I just read up on the official specification of __del__().

http://docs.python.org/reference/datamodel.html#basic-customization

The specification contains a clear warning that module globals may no
longer exist when __del__() is called on an object. IMHO, that makes a
closure for __del__() less of a requirement, although it makes more sense
in the context of Cython than in Python. It would definitely help to keep
users from writing innocently looking code that crashes (such as "1+1").

Stefan


From dalcinl at gmail.com  Thu Feb 26 18:08:04 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 26 Feb 2009 15:08:04 -0200
Subject: [Cython] PATCH: use function pointer arguments in import/export
	api functions
In-Reply-To: <49A689B4.1000104@behnel.de>
References: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>
	<49A689B4.1000104@behnel.de>
Message-ID: <e7ba66e40902260908p29d13287t43bc9220b91ff230@mail.gmail.com>

On Thu, Feb 26, 2009 at 10:23 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>> Just to prevent this:
>>
>> warning: ISO C forbids conversion of function pointer to object pointer type
>>
>> - Please note that I've changed the names of some local variables to
>> make {Import|Export}Function implementations more symmetric.
>
> Fine with me. If this makes things more portable, and it doesn't look like
> it breaks anything - just go and push.
>
> BTW, it's good that that you split the patch into two.
>

Sorry, I did not undestand you. What should I split? Two commits, one
for each of the functions?

> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Feb 26 19:52:52 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 26 Feb 2009 16:52:52 -0200
Subject: [Cython] improving Py <-> C conversion of numeric types
Message-ID: <e7ba66e40902261052t8f535cdtbebc1efa6f6e2d7f@mail.gmail.com>

I'm working on a patch related to de subject of this mail, but I will
need any of you to make some coments on the above ....

In [1]: from Cython.Compiler import PyrexTypes

In [2]: PyrexTypes.c_anon_enum_type.rank
Out[2]: -1

In [3]: PyrexTypes.c_anon_enum_type.sign_and_name()
Out[3]: 'long double'


[2] seems fine, but what about [3] ?? Should we specialize the method
and make it return 'int' ??

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Feb 26 20:57:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 26 Feb 2009 11:57:25 -0800
Subject: [Cython] ImportError with cimport within package
In-Reply-To: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>
References: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>
Message-ID: <18DA0DBF-5C82-4892-8388-EA55C7686D69@math.washington.edu>

On Feb 26, 2009, at 7:52 AM, Mikael Arvids wrote:
> Hi all,
>
> I'm trying to create package compiled with Cython. It compiles fine  
> but when I try to import it I get an ImportError exception. Here is  
> a simplifed version which generates the same error:
>
> The structure of the compiled package:
>   package/
>     __init__.py    - Empty
>     module_A.pyd
>     module_B.pyd
>
[...]
>
> Any suggestions?
>

Are module_A.pyx, module_A.pxd, module_B.pyx, module_B.pxd inside the  
package directory as well?

- Robert


From dalcinl at gmail.com  Fri Feb 27 00:41:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 26 Feb 2009 21:41:37 -0200
Subject: [Cython] better handling of numeric types
Message-ID: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>

Before going on, I would like to know your opinion... patch and
testcase attached ...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numerictypes.pyx
Type: application/octet-stream
Size: 8835 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090226/0bea30bf/attachment-0002.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUMERICTYPES.diff
Type: application/octet-stream
Size: 11516 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090226/0bea30bf/attachment-0003.obj 

From robertwb at math.washington.edu  Fri Feb 27 01:21:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 26 Feb 2009 16:21:42 -0800
Subject: [Cython] better handling of numeric types
In-Reply-To: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
Message-ID: <12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>

It mostly looks good to me. To summarize, you basically made a macro  
that's the function for converting to/from the various types, and  
applied that everywhere, right?

In your tests, "1..." could catch 100 or anything like that, so I'm  
not sure if that's a concern.

- Robert

On Feb 26, 2009, at 3:41 PM, Lisandro Dalcin wrote:

> Before going on, I would like to know your opinion... patch and
> testcase attached ...
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0) 
> 342-451.1594<numerictypes.pyx><NUMERICTYPES.diff>_____________________ 
> __________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Fri Feb 27 02:05:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 26 Feb 2009 22:05:05 -0300
Subject: [Cython] better handling of numeric types
In-Reply-To: <12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
	<12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>
Message-ID: <e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>

On Thu, Feb 26, 2009 at 9:21 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> It mostly looks good to me. To summarize, you basically made a macro
> that's the function for converting to/from the various types, and
> applied that everywhere, right?

Yes, though this is preliminary...

Perhaps I should make Cython emit the code directly?

Moreover, I need to review the Py_ssize_t/size_t converters..

However, I would like to point a gotcha... The some converters use
PyNumber_{Int|Long} ... This is good as they would exercise general
Python types implementing the special methods "__int__()" or
"__long__()" ... However, they will also accept strings, and even
(char) buffer objects !!!! What to do here ?? This makes me feel a bit
uncomfortable, seems too much ...

>
> In your tests, "1..." could catch 100 or anything like that, so I'm
> not sure if that's a concern.
>

Yes, know.. that was a quick hack for making the tests pass on Py2/Py3
(because of the "L" suffix on Py2's "long" instances)... I just forgot
to use "print" ....


> - Robert
>
> On Feb 26, 2009, at 3:41 PM, Lisandro Dalcin wrote:
>
>> Before going on, I would like to know your opinion... patch and
>> testcase attached ...
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)
>> 342-451.1594<numerictypes.pyx><NUMERICTYPES.diff>_____________________
>> __________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Feb 27 03:23:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 26 Feb 2009 18:23:27 -0800
Subject: [Cython] better handling of numeric types
In-Reply-To: <e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
	<12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>
	<e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>
Message-ID: <6433709C-3BF9-49F1-9579-A71EBCD0528B@math.washington.edu>

On Feb 26, 2009, at 5:05 PM, Lisandro Dalcin wrote:

> On Thu, Feb 26, 2009 at 9:21 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> It mostly looks good to me. To summarize, you basically made a macro
>> that's the function for converting to/from the various types, and
>> applied that everywhere, right?
>
> Yes, though this is preliminary...
>
> Perhaps I should make Cython emit the code directly?

I think that's what was there originally. I'm OK with it either way.

> Moreover, I need to review the Py_ssize_t/size_t converters..

For Py_ssize_t we should be using PyNumber_Index--probably should use  
the same for size_t.

> However, I would like to point a gotcha... The some converters use
> PyNumber_{Int|Long} ... This is good as they would exercise general
> Python types implementing the special methods "__int__()" or
> "__long__()" ... However, they will also accept strings, and even
> (char) buffer objects !!!! What to do here ?? This makes me feel a bit
> uncomfortable, seems too much ...

I'm confused, I don't think PyNumber_(Int|Long) accept strings, etc.  
They only accept objects that have implemented the __int__ method.  
Note that I wrote an inline function to extract it directly if it's a  
Python int (as this is an extremely common case).

>> In your tests, "1..." could catch 100 or anything like that, so I'm
>> not sure if that's a concern.
>>
>
> Yes, know.. that was a quick hack for making the tests pass on Py2/Py3
> (because of the "L" suffix on Py2's "long" instances)... I just forgot
> to use "print" ....

Yeah, that's what I figured.

- Robert



From stefan_ml at behnel.de  Thu Feb 26 17:21:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Feb 2009 17:21:50 +0100
Subject: [Cython] module cleanup
In-Reply-To: <49A44334.9040204@behnel.de>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>	<59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<3E444205-8D32-4970-86A7-378181415B48@math.washington.edu>	<52786.213.61.181.86.1235465957.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<8EDB19D4-78EF-4194-B817-21776D8E473F@math.washington.edu>	<49A3F7F3.5060409@behnel.de>	<e7ba66e40902240601q594cc4faueb1baec5262bf33c@mail.gmail.com>	<49A43534.6090905@behnel.de>	<49A43A29.2010503@student.matnat.uio.no>	<49A43BE7.3080107@behnel.de>	<49A4445A.9090507@student.matnat.uio.no>
	<49A44334.9040204@behnel.de>
Message-ID: <49A6C19E.2070209@behnel.de>


Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> let all global values be referenced by a big tuple; then have each 
>> type and the module reference the tuple, but the tuple itself doesn't 
>> reference anything but the values themselves => no cycles.
> 
> Although "globals" would also include types, right? So there's our
> reference cycle back... (although that doesn't hit us until we really start
> moving types to the heap)

BTW, a tuple isn't enough here. I sometimes write things like this in my code:

	cdef str NAMESPACE = "http://sdffsadf.org/jhshdkflh/fsdfs"
	cdef char* _NAMESPACE = NAMESPACE

i.e. you'd need an extension class to also reference the module-local C
values, which may depend on module-local Python values.

Also note that the PEP has a way to store module-local state. Since this is
a Py3-only feature, we might just as well use that.

Stefan



From stefan_ml at behnel.de  Fri Feb 27 06:35:32 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 06:35:32 +0100
Subject: [Cython] PATCH: use function pointer arguments in import/export
 api functions
In-Reply-To: <e7ba66e40902260908p29d13287t43bc9220b91ff230@mail.gmail.com>
References: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>	<49A689B4.1000104@behnel.de>
	<e7ba66e40902260908p29d13287t43bc9220b91ff230@mail.gmail.com>
Message-ID: <49A77BA4.7000100@behnel.de>


Lisandro Dalcin wrote:
> On Thu, Feb 26, 2009 at 10:23 AM, Stefan Behnel wrote:
>> BTW, it's good that that you split the patch into two.
> 
> Sorry, I did not undestand you. What should I split? Two commits, one
> for each of the functions?

I was just commenting the fact that you had committed the ref-count fix
separately.

Never mind, I already pushed the rest of your fix.

Stefan


From stefan_ml at behnel.de  Fri Feb 27 07:05:42 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 07:05:42 +0100
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
Message-ID: <49A782B6.20005@behnel.de>

Hi Robert,

this change:

+                if self.base.type is list_type:
+                    function = "__Pyx_GetItemInt_List"

...

+static INLINE PyObject *__Pyx_GetItemInt_List(
                     PyObject *o, Py_ssize_t i, int is_unsigned) {
+    if (likely(0 <= i && i < PyList_GET_SIZE(o))) {
+        PyObject *r = PyList_GET_ITEM(o, i);
+        Py_INCREF(r);
+        return r;
+    }
+    else return __Pyx_GetItemInt_Generic(o, i, is_unsigned);
+}

does not handle the case that the 'list' might be None. Ticket 166 already
deals with this, but I think this is actually worth discussing, as I would
assume that it leads to a crash here.

http://trac.cython.org/cython_trac/ticket/166

I think that a little set of short inline helper functions that check a
value for Py_None and raise the appropriate exception for a given context
would be nice to have.

I doubt that a check for Py_None will kill performance in this kind of
optimisation, especially since the compiler will be able to see the value
in many cases (e.g. after the first check for None that exits for an
exception). Even in loops, most processors can remember the result of such
a check and adjust their branch prediction accordingly, so that this will
cost us a cycle or something in that order.

Removing these None checks later on is another optimisation, but it's a
feature compared to a potential crash. Maybe we could add a tristate
attribute to entries that tells us if a variable (a) is not, (b) may be or
(c) is definitely None, all but (a) needing such a check, and (c) being
worth knowing in some cases. Even without complete flow analysis, this can
be decided for single-assignment cases (which are definitely common,
especially for container types).

Stefan

From stefan_ml at behnel.de  Fri Feb 27 08:17:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 08:17:44 +0100
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
Message-ID: <49A79398.4000809@behnel.de>


Robert Bradshaw wrote:
> On Feb 26, 2009, at 10:05 PM, Stefan Behnel wrote:
>> this change:
>>[...]
>> does not handle the case that the 'list' might be None.
> 
> Good point. I'm pushing a fix.

Regarding this bit:

	likely(o != Py_None && 0 <= i && i < PyList_GET_SIZE(o))

Does anyone know if gcc handles this as desired? Does this mean that the
whole expression is likely, and does that make each part likely, or would
it be better to wrap each part of the expression in its own likely()?

(BTW, likely() refers to GCC's __builtin_expect() here)


> I'd love for someone to implement a control flow analysis--I started to
> but never had time to finish (or do it right).

I guess the "someone" bit is the main problem here. ;)

The "do it right" can also refer to complexity, BTW. There was a post on
c.l.py a couple of months ago where someone presented some heavy algorithm
(don't remember what it did) and asked for ways to make it faster. I just
went: "ah, I can use Cython for that", but I didn't even manage to compile
it. It contained an impressively long list of "if" statements, just one
after the other (no nesting), and that let the control flow analysis die
from stack exhaustion.

Stefan


From stefan_ml at behnel.de  Fri Feb 27 09:00:55 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 09:00:55 +0100
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
	<49A79398.4000809@behnel.de>
	<E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
Message-ID: <49A79DB7.2040702@behnel.de>


Robert Bradshaw wrote:
> I was actually surprised how little of a difference the __builtin_expect
> macros made in most cases, I guess gcc is good at guessing right most of
> the time. (Still worth it I think, as we *know* the common code path.)

I think the problem is that both of us are benchmarking these things on
Intel 32-bit platforms that don't support branch prediction hints and
usually require a certain opcode chain anyway. Plus, these processors are
pretty smart about branch prediction anyway, so running tight loops in
benchmarks will usually result in perfect branch prediction.

There might be a difference on other architectures like Itanium where
compilers are smart and the metal is stupid.


>>> I'd love for someone to implement a control flow analysis--I started to
>>> but never had time to finish (or do it right).
>>
>> The "do it right" can also refer to complexity, BTW. There was a post on
>> c.l.py a couple of months ago where someone presented some heavy
>> algorithm
>> (don't remember what it did) and asked for ways to make it faster. I just
>> went: "ah, I can use Cython for that", but I didn't even manage to
>> compile
>> it. It contained an impressively long list of "if" statements, just one
>> after the other (no nesting), and that let the control flow analysis die
>> from stack exhaustion.
> 
> Hmm... did you file a trac ticket?

Ah, you're right, I will.

Stefan


From robertwb at math.washington.edu  Fri Feb 27 09:05:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 00:05:51 -0800
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <49A79DB7.2040702@behnel.de>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
	<49A79398.4000809@behnel.de>
	<E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
	<49A79DB7.2040702@behnel.de>
Message-ID: <60444235-FB04-466B-B1DC-5EC595965E7B@math.washington.edu>

On Feb 27, 2009, at 12:00 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> I was actually surprised how little of a difference the  
>> __builtin_expect
>> macros made in most cases, I guess gcc is good at guessing right  
>> most of
>> the time. (Still worth it I think, as we *know* the common code  
>> path.)
>
> I think the problem is that both of us are benchmarking these  
> things on
> Intel 32-bit platforms that don't support branch prediction hints and
> usually require a certain opcode chain anyway. Plus, these  
> processors are
> pretty smart about branch prediction anyway, so running tight loops in
> benchmarks will usually result in perfect branch prediction.

Sometimes it did make a difference though, but I found no logic as to  
when and can't remember the details now (probably because I wasn't  
able to fit it to a pattern). IIRC, the first time a branch is taken  
it makes a difference whether or not it's jumping forward or  
backward, and gcc sometimes uses this to rearrange code blocks even  
on processors that don't support hints.

> There might be a difference on other architectures like Itanium where
> compilers are smart and the metal is stupid.

For sure.

- Robert


From robertwb at math.washington.edu  Fri Feb 27 09:09:56 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 00:09:56 -0800
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <49A79930.80706@behnel.de>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
	<49A79398.4000809@behnel.de>
	<E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
	<49A79930.80706@behnel.de>
Message-ID: <6DD00FC1-69A9-455A-919C-669773F6A693@math.washington.edu>

On Feb 26, 2009, at 11:41 PM, Stefan Behnel wrote:

> Hi,
>
> you noticed that you are not responding to the list, didn't you? ;)

Oops...

> BTW, I pushed a fix to avoid code duplication in the index functions.

Nice.

> I'd like to also optimise negative indices (at least -1 is  
> extremely common),
> that'll definitely make the List/Tuple functions too long for their  
> minor
> differences.

Good idea, doing that right now.

- Robert


From stefan_ml at behnel.de  Fri Feb 27 09:18:09 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 09:18:09 +0100
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <6DD00FC1-69A9-455A-919C-669773F6A693@math.washington.edu>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
	<49A79398.4000809@behnel.de>
	<E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
	<49A79930.80706@behnel.de>
	<6DD00FC1-69A9-455A-919C-669773F6A693@math.washington.edu>
Message-ID: <49A7A1C1.20702@behnel.de>


Robert Bradshaw wrote:
> On Feb 26, 2009, at 11:41 PM, Stefan Behnel wrote:
>> I'd like to also optimise negative indices (at least -1 is extremely
>> common),
>> that'll definitely make the List/Tuple functions too long for their minor
>> differences.
> 
> Good idea, doing that right now.

What about this? I checked the assembly, and it looks not too bloated to me.

Stefan


static INLINE PyObject *__Pyx_GetItemInt_%(type)s(PyObject *o, Py_ssize_t
i, int is_unsigned) {
    if (likely(o != Py_None)) {
        if (likely((i < Py%(type)s_GET_SIZE(o)) &
                                 (i >= -Py%(type)s_GET_SIZE(o)))) {
            if (unlikely(i < 0))
                i = Py%(type)s_GET_SIZE(o) + i;
            PyObject *r = Py%(type)s_GET_ITEM(o, i);
            Py_INCREF(r);
            return r;
        }
    }
    return __Pyx_GetItemInt_Generic(o, i, is_unsigned);
}


From robertwb at math.washington.edu  Fri Feb 27 09:23:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 00:23:50 -0800
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <49A7A1C1.20702@behnel.de>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
	<49A79398.4000809@behnel.de>
	<E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
	<49A79930.80706@behnel.de>
	<6DD00FC1-69A9-455A-919C-669773F6A693@math.washington.edu>
	<49A7A1C1.20702@behnel.de>
Message-ID: <F3EA5AA9-10CA-46CD-8C0F-F416C57D90D0@math.washington.edu>

On Feb 27, 2009, at 12:18 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Feb 26, 2009, at 11:41 PM, Stefan Behnel wrote:
>>> I'd like to also optimise negative indices (at least -1 is extremely
>>> common),
>>> that'll definitely make the List/Tuple functions too long for  
>>> their minor
>>> differences.
>>
>> Good idea, doing that right now.
>
> What about this? I checked the assembly, and it looks not too  
> bloated to me.
>
> Stefan
>
>
> static INLINE PyObject *__Pyx_GetItemInt_%(type)s(PyObject *o,  
> Py_ssize_t
> i, int is_unsigned) {
>     if (likely(o != Py_None)) {
>         if (likely((i < Py%(type)s_GET_SIZE(o)) &
>                                  (i >= -Py%(type)s_GET_SIZE(o)))) {
>             if (unlikely(i < 0))
>                 i = Py%(type)s_GET_SIZE(o) + i;
>             PyObject *r = Py%(type)s_GET_ITEM(o, i);
>             Py_INCREF(r);
>             return r;
>         }
>     }
>     return __Pyx_GetItemInt_Generic(o, i, is_unsigned);
> }

This doesn't handle i < -len(o). I actually already implemented this,  
pushing now.

- Robert


From robertwb at math.washington.edu  Fri Feb 27 09:31:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 00:31:23 -0800
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <F3EA5AA9-10CA-46CD-8C0F-F416C57D90D0@math.washington.edu>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
	<49A79398.4000809@behnel.de>
	<E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
	<49A79930.80706@behnel.de>
	<6DD00FC1-69A9-455A-919C-669773F6A693@math.washington.edu>
	<49A7A1C1.20702@behnel.de>
	<F3EA5AA9-10CA-46CD-8C0F-F416C57D90D0@math.washington.edu>
Message-ID: <E17C3CE7-69C3-4D01-8ED9-7AFFFD78B212@math.washington.edu>

On Feb 27, 2009, at 12:23 AM, Robert Bradshaw wrote:

> On Feb 27, 2009, at 12:18 AM, Stefan Behnel wrote:
>
>> Robert Bradshaw wrote:
>>> On Feb 26, 2009, at 11:41 PM, Stefan Behnel wrote:
>>>> I'd like to also optimise negative indices (at least -1 is  
>>>> extremely
>>>> common),
>>>> that'll definitely make the List/Tuple functions too long for
>>>> their minor
>>>> differences.
>>>
>>> Good idea, doing that right now.
>>
>> What about this? I checked the assembly, and it looks not too
>> bloated to me.
>>
>> Stefan
>>
>>
>> static INLINE PyObject *__Pyx_GetItemInt_%(type)s(PyObject *o,
>> Py_ssize_t
>> i, int is_unsigned) {
>>     if (likely(o != Py_None)) {
>>         if (likely((i < Py%(type)s_GET_SIZE(o)) &
>>                                  (i >= -Py%(type)s_GET_SIZE(o)))) {
>>             if (unlikely(i < 0))
>>                 i = Py%(type)s_GET_SIZE(o) + i;
>>             PyObject *r = Py%(type)s_GET_ITEM(o, i);
>>             Py_INCREF(r);
>>             return r;
>>         }
>>     }
>>     return __Pyx_GetItemInt_Generic(o, i, is_unsigned);
>> }
>
> This doesn't handle i < -len(o).

Never-mind, didn't read carefully enough.

Another optimization I'd like to do is not increfing the result. Then  
it could even be used in a nogil context (maybe?). Somehow, we'd have  
to know the reference was borrowed though, which will be a bit more  
invasive.

- Robert


From mikael.arvids at enea.com  Fri Feb 27 09:31:10 2009
From: mikael.arvids at enea.com (Mikael Arvids)
Date: Fri, 27 Feb 2009 08:31:10 +0000 (UTC)
Subject: [Cython] ImportError with cimport within package
References: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>
	<18DA0DBF-5C82-4892-8388-EA55C7686D69@math.washington.edu>
Message-ID: <loom.20090227T082755-709@post.gmane.org>

Robert Bradshaw <robertwb at ...> writes:

> 
> Are module_A.pyx, module_A.pxd, module_B.pyx, module_B.pxd inside the  
> package directory as well?
> 
> - Robert
> 
> 

I had the Cython files in a separate directory earlier, but I tried with
everything in the same folder, recompiled, and still the same problem.



From stefan_ml at behnel.de  Fri Feb 27 09:36:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 09:36:41 +0100
Subject: [Cython] None checks for variables of builtin types (rev
	ffeb158969e)
In-Reply-To: <F3EA5AA9-10CA-46CD-8C0F-F416C57D90D0@math.washington.edu>
References: <49A782B6.20005@behnel.de>
	<A6EBB005-B6DD-494F-B3B4-BAF1DEBE80B4@math.washington.edu>
	<49A79398.4000809@behnel.de>
	<E674138F-EDD0-4E9D-A654-4D1DED17277C@math.washington.edu>
	<49A79930.80706@behnel.de>
	<6DD00FC1-69A9-455A-919C-669773F6A693@math.washington.edu>
	<49A7A1C1.20702@behnel.de>
	<F3EA5AA9-10CA-46CD-8C0F-F416C57D90D0@math.washington.edu>
Message-ID: <49A7A619.9070301@behnel.de>


Robert Bradshaw wrote:
> I actually already implemented this, pushing now.

Ah, yes. Your version definitely looks better for a constant value of i.
Making it clear to the compiler that both cases are exclusive leads to very
efficient code here.

Stefan


From stefan_ml at behnel.de  Fri Feb 27 09:50:09 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 09:50:09 +0100
Subject: [Cython] ImportError with cimport within package
In-Reply-To: <loom.20090227T082755-709@post.gmane.org>
References: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>	<18DA0DBF-5C82-4892-8388-EA55C7686D69@math.washington.edu>
	<loom.20090227T082755-709@post.gmane.org>
Message-ID: <49A7A941.8090609@behnel.de>


Mikael Arvids wrote:
> Robert Bradshaw writes:
> 
>> Are module_A.pyx, module_A.pxd, module_B.pyx, module_B.pxd inside the  
>> package directory as well?
> 
> I had the Cython files in a separate directory earlier, but I tried with
> everything in the same folder, recompiled, and still the same problem.

What Robert mean, was: Did you compile the module from source files that
are inside the package directory? Cython needs to know the package
structure at compile time, it's not enough to move the shared libraries around.

Stefan


From mikael.arvids at enea.com  Fri Feb 27 09:56:57 2009
From: mikael.arvids at enea.com (Mikael Arvids)
Date: Fri, 27 Feb 2009 08:56:57 +0000 (UTC)
Subject: [Cython] ImportError with cimport within package
References: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>	<18DA0DBF-5C82-4892-8388-EA55C7686D69@math.washington.edu>
	<loom.20090227T082755-709@post.gmane.org>
	<49A7A941.8090609@behnel.de>
Message-ID: <loom.20090227T085134-462@post.gmane.org>

Stefan Behnel <stefan_ml at ...> writes:

> 
> 
> What Robert mean, was: Did you compile the module from source files that
> are inside the package directory? Cython needs to know the package
> structure at compile time, it's not enough to move the shared libraries around.
> 
> Stefan
> 
> 

Yes, I moved the .pyx and .pxd files to the same folder as the shared libraries
are built and changed my setup.py so that those files were built.

My current folder structure:
  package/
    __init__.py    - Empty
    module_A.pyd
    module_A.pxd
    module_A.pyx
    module_B.pyd
    module_B.pxd
    module_B.pyx
    setup.py

My setup.py:

  from distutils.core import setup
  from distutils.extension import Extension
  from Cython.Distutils import build_ext

  A_module = Extension('module_A',
                      sources = ['module_A.pyx'])
                    
  B_module = Extension('module_B',
                      sources = ['module_B.pyx'])

  setup(
      name='package',
      packages=['package'],
      cmdclass = {'build_ext': build_ext},
      ext_modules = [A_module, B_module]
  )



From robertwb at math.washington.edu  Fri Feb 27 10:15:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 01:15:27 -0800
Subject: [Cython] ImportError with cimport within package
In-Reply-To: <loom.20090227T085134-462@post.gmane.org>
References: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>	<18DA0DBF-5C82-4892-8388-EA55C7686D69@math.washington.edu>
	<loom.20090227T082755-709@post.gmane.org>
	<49A7A941.8090609@behnel.de>
	<loom.20090227T085134-462@post.gmane.org>
Message-ID: <D74DB309-2734-43A6-B159-E80FDD85901F@math.washington.edu>

On Feb 27, 2009, at 12:56 AM, Mikael Arvids wrote:

> Stefan Behnel <stefan_ml at ...> writes:
>
>>
>>
>> What Robert mean, was: Did you compile the module from source  
>> files that
>> are inside the package directory? Cython needs to know the package
>> structure at compile time, it's not enough to move the shared  
>> libraries around.
>>
>> Stefan
>>
>>
>
> Yes, I moved the .pyx and .pxd files to the same folder as the  
> shared libraries
> are built and changed my setup.py so that those files were built.
>
> My current folder structure:
>   package/
>     __init__.py    - Empty
>     module_A.pyd
>     module_A.pxd
>     module_A.pyx
>     module_B.pyd
>     module_B.pxd
>     module_B.pyx
>     setup.py
>
> My setup.py:
>
>   from distutils.core import setup
>   from distutils.extension import Extension
>   from Cython.Distutils import build_ext
>
>   A_module = Extension('module_A',
>                       sources = ['module_A.pyx'])
>
>   B_module = Extension('module_B',
>                       sources = ['module_B.pyx'])
>
>   setup(
>       name='package',
>       packages=['package'],
>       cmdclass = {'build_ext': build_ext},
>       ext_modules = [A_module, B_module]
>   )

Try moving setup.py up out of the package directory, and then building

   A_module = Extension('package.module_A',
                       sources = ['package/module_A.pyx'])
...

- Robert


From mikael.arvids at enea.com  Fri Feb 27 10:20:35 2009
From: mikael.arvids at enea.com (Mikael Arvids)
Date: Fri, 27 Feb 2009 09:20:35 +0000 (UTC)
Subject: [Cython] ImportError with cimport within package
References: <DC6F86A94046A64094043D44A3A708F723BDC9@sestoex02.enea.se>	<18DA0DBF-5C82-4892-8388-EA55C7686D69@math.washington.edu>
	<loom.20090227T082755-709@post.gmane.org>
	<49A7A941.8090609@behnel.de>
	<loom.20090227T085134-462@post.gmane.org>
	<D74DB309-2734-43A6-B159-E80FDD85901F@math.washington.edu>
Message-ID: <loom.20090227T091947-345@post.gmane.org>

Robert Bradshaw <robertwb at ...> writes:

> 
> Try moving setup.py up out of the package directory, and then building
> 
>    A_module = Extension('package.module_A',
>                        sources = ['package/module_A.pyx'])
> ...
> 
> - Robert
> 
> 

Thanks! Solved the problem.

/ Mikael




From dalcinl at gmail.com  Fri Feb 27 12:59:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 27 Feb 2009 09:59:27 -0200
Subject: [Cython] better handling of numeric types
In-Reply-To: <6433709C-3BF9-49F1-9579-A71EBCD0528B@math.washington.edu>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
	<12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>
	<e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>
	<6433709C-3BF9-49F1-9579-A71EBCD0528B@math.washington.edu>
Message-ID: <e7ba66e40902270359j671673efg278d954429b2c64f@mail.gmail.com>

On Fri, Feb 27, 2009 at 12:23 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 26, 2009, at 5:05 PM, Lisandro Dalcin wrote:
>
>> On Thu, Feb 26, 2009 at 9:21 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>> It mostly looks good to me. To summarize, you basically made a macro
>>> that's the function for converting to/from the various types, and
>>> applied that everywhere, right?
>>
>> Yes, though this is preliminary...
>>
>> Perhaps I should make Cython emit the code directly?
>
> I think that's what was there originally. I'm OK with it either way.
>

IMHO, the only benefit of macro-based would be to save disk space...
These functions are generated for every ext module.

>> Moreover, I need to review the Py_ssize_t/size_t converters..
>
> For Py_ssize_t we should be using PyNumber_Index--probably should use
> the same for size_t.
>

OK.. that was what I meant...

>> However, I would like to point a gotcha... The some converters use
>> PyNumber_{Int|Long} ... This is good as they would exercise general
>> Python types implementing the special methods "__int__()" or
>> "__long__()" ... However, they will also accept strings, and even
>> (char) buffer objects !!!! What to do here ?? This makes me feel a bit
>> uncomfortable, seems too much ...
>
> I'm confused, I don't think PyNumber_(Int|Long) accept strings, etc.
> They only accept objects that have implemented the __int__ method.
> Note that I wrote an inline function to extract it directly if it's a
> Python int (as this is an extremely common case).
>

Unfortunately, you are wrong. Plese check PyNumber_(Int|Long)
implementation at core CPython sources...

Anyway, I'll try first to make this work right, signaling overflows
where appropiate...

IMHO, we cannot use things like PyInt_AsUnsignedLongMask()... Do you agree here?


Next we can discuss how to optimize this....

BTW, can I asume that if I write

if ( sizeof(c_type) < sizeof(long) ) {

} else if ( sizeof(c_type) == sizeof(long) ) {

} else {

}

then C compiler will be smart enough to eliminate the branching? (I
know, I have to learn how to read disasemblies....)

I'm thinking on extending all this to extern ctypedef'ed integral,
this way Cython will stop using the base-type's longness of the
typedef.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb 27 13:06:48 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 13:06:48 +0100
Subject: [Cython] better handling of numeric types
In-Reply-To: <e7ba66e40902270359j671673efg278d954429b2c64f@mail.gmail.com>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>	<12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>	<e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>	<6433709C-3BF9-49F1-9579-A71EBCD0528B@math.washington.edu>
	<e7ba66e40902270359j671673efg278d954429b2c64f@mail.gmail.com>
Message-ID: <49A7D758.9060407@behnel.de>



Lisandro Dalcin wrote:
> BTW, can I asume that if I write
> 
> if ( sizeof(c_type) < sizeof(long) ) {
> 
> } else if ( sizeof(c_type) == sizeof(long) ) {
> 
> } else {
> 
> }
> 
> then C compiler will be smart enough to eliminate the branching? (I
> know, I have to learn how to read disasemblies....)

Absolutely. The result of sizeof() is known at compile time and the C
compiler will only generate code for the first branch that evaluates to true.

Stefan


From robertwb at math.washington.edu  Fri Feb 27 13:15:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 04:15:59 -0800
Subject: [Cython] better handling of numeric types
In-Reply-To: <e7ba66e40902270359j671673efg278d954429b2c64f@mail.gmail.com>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
	<12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>
	<e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>
	<6433709C-3BF9-49F1-9579-A71EBCD0528B@math.washington.edu>
	<e7ba66e40902270359j671673efg278d954429b2c64f@mail.gmail.com>
Message-ID: <91C70965-680F-46D8-BBFB-495496011E1D@math.washington.edu>


On Feb 27, 2009, at 3:59 AM, Lisandro Dalcin wrote:

> On Fri, Feb 27, 2009 at 12:23 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Feb 26, 2009, at 5:05 PM, Lisandro Dalcin wrote:
>>
>>> On Thu, Feb 26, 2009 at 9:21 PM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>>> It mostly looks good to me. To summarize, you basically made a  
>>>> macro
>>>> that's the function for converting to/from the various types, and
>>>> applied that everywhere, right?
>>>
>>> Yes, though this is preliminary...
>>>
>>> Perhaps I should make Cython emit the code directly?
>>
>> I think that's what was there originally. I'm OK with it either way.
>>
>
> IMHO, the only benefit of macro-based would be to save disk space...
> These functions are generated for every ext module.

OK, makes sense. It might actually be nice to have a flag which uses  
a (common) .h file for lots of the boilerplate code. Using .h files  
to share declarations across modules could save a lot of space too  
(and probably make compilation faster when you cimport a lot of  
stuff, like Sage).

>
>>> Moreover, I need to review the Py_ssize_t/size_t converters..
>>
>> For Py_ssize_t we should be using PyNumber_Index--probably should use
>> the same for size_t.
>>
>
> OK.. that was what I meant...
>
>>> However, I would like to point a gotcha... The some converters use
>>> PyNumber_{Int|Long} ... This is good as they would exercise general
>>> Python types implementing the special methods "__int__()" or
>>> "__long__()" ... However, they will also accept strings, and even
>>> (char) buffer objects !!!! What to do here ?? This makes me feel  
>>> a bit
>>> uncomfortable, seems too much ...
>>
>> I'm confused, I don't think PyNumber_(Int|Long) accept strings, etc.
>> They only accept objects that have implemented the __int__ method.
>> Note that I wrote an inline function to extract it directly if it's a
>> Python int (as this is an extremely common case).
>
> Unfortunately, you are wrong. Plese check PyNumber_(Int|Long)
> implementation at core CPython sources...

You're right, in that case we should definitely be avoiding them  
PyInt_AsLong looks like the right thing to use here.

> Anyway, I'll try first to make this work right, signaling overflows
> where appropiate...
>
> IMHO, we cannot use things like PyInt_AsUnsignedLongMask()... Do  
> you agree here?

I'd certainly like to avoid this.

> Next we can discuss how to optimize this....

As long as we don't slow down the (very common) Python int -> int/ 
long code, I'm OK with deferring optimization.

>
> BTW, can I asume that if I write
>
> if ( sizeof(c_type) < sizeof(long) ) {
>
> } else if ( sizeof(c_type) == sizeof(long) ) {
>
> } else {
>
> }
>
> then C compiler will be smart enough to eliminate the branching? (I
> know, I have to learn how to read disasemblies....)

Yes, the C compiler will kill all but the correct branch.

> I'm thinking on extending all this to extern ctypedef'ed integral,
> this way Cython will stop using the base-type's longness of the
> typedef.

Yes, for sure we should be doing this.

- Robert



From dalcinl at gmail.com  Fri Feb 27 14:13:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 27 Feb 2009 11:13:46 -0200
Subject: [Cython] better handling of numeric types
In-Reply-To: <91C70965-680F-46D8-BBFB-495496011E1D@math.washington.edu>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
	<12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>
	<e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>
	<6433709C-3BF9-49F1-9579-A71EBCD0528B@math.washington.edu>
	<e7ba66e40902270359j671673efg278d954429b2c64f@mail.gmail.com>
	<91C70965-680F-46D8-BBFB-495496011E1D@math.washington.edu>
Message-ID: <e7ba66e40902270513m4abe28ffk3a51a71b785185a3@mail.gmail.com>

On Fri, Feb 27, 2009 at 10:15 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> On Feb 27, 2009, at 3:59 AM, Lisandro Dalcin wrote:
>
>> Next we can discuss how to optimize this....
>
> As long as we don't slow down the (very common) Python int -> int/
> long code, I'm OK with deferring optimization.
>

However, do you realize that for Py3 these optimizations are a bit
pointless? Moreover, if you look at core CPython 2.X sources, you will
find many places were were optimizations could be made, but they are
not there...

Anyway, I'll try to do my best...

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb 27 17:28:49 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 17:28:49 +0100
Subject: [Cython] strict aliasing problem with subclass method
Message-ID: <49A814C1.5050302@behnel.de>

Hi,

I have a problem with a method in a subclass that overrides the parent
method. The setup is basically as follows:

-------------------------------
# base class:
cdef class _BaseContext:
    cdef xpath.xmlXPathContext* _xpathCtxt
    cdef _Document _doc
    cdef dict _extensions
    ... # other attributes here

    cdef _BaseContext _copy(self):
        cdef _BaseContext context
        if self._namespaces is not None:
            namespaces = self._namespaces[:]
        context = self.__class__(namespaces, None, False,
                                 self._build_smart_strings)
        if self._extensions is not None:
            context._extensions = self._extensions.copy()
        return context

# subclass:
cdef class _XSLTContext(_BaseContext):
    cdef xslt.xsltTransformContext* _xsltCtxt
    cdef _ReadOnlyElementProxy _extension_element_proxy
    cdef dict _extension_elements
    ...
    cdef _BaseContext _copy(self):
        cdef _XSLTContext context
        context = <_XSLTContext>_BaseContext._copy(self)
        context._extension_elements = self._extension_elements
        return context

# using class:
cdef class XSLT:
    cdef _XSLTContext _context
    cdef xslt.xsltStylesheet* _c_style
    ... # other attributes here
    def __call__(self, _input, *, profile_run=False, **kw):
        ...
        try:
            self._error_log.connect()
            context = self._context._copy()
            context.register_context(transform_ctxt, input_doc)
-------------------------------

For the method call to ._copy(), Cython generates this code:

-------------------------------
    /* __pyx_t_1 allocated */
    __pyx_t_1 = ((PyObject *)((struct
__pyx_vtabstruct_4lxml_5etree__XSLTContext *)((struct
__pyx_obj_4lxml_5etree_XSLT
*)__pyx_v_self)->_context->__pyx_base.__pyx_vtab)->__pyx_base._copy(((struct
__pyx_obj_4lxml_5etree__BaseContext *)((struct __pyx_obj_4lxml_5etree_XSLT
*)__pyx_v_self)->_context))); if (unlikely(!__pyx_t_1)) /* ... */

    __Pyx_GOTREF(__pyx_t_1);

    if (!(__Pyx_TypeTest(__pyx_t_1,
__pyx_ptype_4lxml_5etree__XSLTContext))) /* ... */

    __Pyx_DECREF(((PyObject *)__pyx_v_context));

    __pyx_v_context = ((struct __pyx_obj_4lxml_5etree__XSLTContext
*)__pyx_t_1);

    __pyx_t_1 = 0;
    /* __pyx_t_1 released */
-------------------------------

The problem with this is that gcc 4.3.2(-1ubuntu12) generates a call to the
baseclass's _copy() method instead of the one in _XSLTContext, which
prevents the "_extension_elements" field from being set. I can fix this by
passing "-fno-strict-aliasing" to gcc. However, I thought this was fixed in
Py3, where I also get the above problem.

Does anyone have an idea if there is anything we can do in Cython to fix
this? Or does anyone see anything in the code that might trigger this?

Thanks for any hints!

Stefan

From carl.witty at gmail.com  Fri Feb 27 18:11:43 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Fri, 27 Feb 2009 09:11:43 -0800
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <49A814C1.5050302@behnel.de>
References: <49A814C1.5050302@behnel.de>
Message-ID: <ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>

On Fri, Feb 27, 2009 at 8:28 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I have a problem with a method in a subclass that overrides the parent
> method. The setup is basically as follows:
>
> -------------------------------
...
> -------------------------------
>
> The problem with this is that gcc 4.3.2(-1ubuntu12) generates a call to the
> baseclass's _copy() method instead of the one in _XSLTContext, which
> prevents the "_extension_elements" field from being set. I can fix this by
> passing "-fno-strict-aliasing" to gcc. However, I thought this was fixed in
> Py3, where I also get the above problem.
>
> Does anyone have an idea if there is anything we can do in Cython to fix
> this? Or does anyone see anything in the code that might trigger this?

AFAIK, Python 3 only fixes one (pervasive) instance of the problem.
See http://www.python.org/dev/peps/pep-3123/ for more information.
But Cython already uses the essence of this fix, by including "parent"
object and vtab structs as __pyx_base, rather than just duplicating
the field member list from the parent.

Are you sure that the problem is with the call to _copy?  I'd think
that an aliasing problem would be at least as likely (probably more
likely) to show up in the initialization; so maybe your _XSLTContext
object doesn't actually have the right vtable, or maybe the "_copy"
entry in the vtable somehow points to the wrong function.

Carl

From stefan_ml at behnel.de  Fri Feb 27 20:23:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 20:23:00 +0100
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>
References: <49A814C1.5050302@behnel.de>
	<ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>
Message-ID: <49A83D94.8050004@behnel.de>


Carl Witty wrote:
> Are you sure that the problem is with the call to _copy?  I'd think
> that an aliasing problem would be at least as likely (probably more
> likely) to show up in the initialization; so maybe your _XSLTContext
> object doesn't actually have the right vtable, or maybe the "_copy"
> entry in the vtable somehow points to the wrong function.

You're right, the vtable is set up incorrectly:

-------------------------------
(gdb) print __pyx_vtable_4lxml_5etree__XSLTContext
$5 = {__pyx_base = {_copy = 0xb7c7cb50
<__pyx_f_4lxml_5etree_12_BaseContext__copy>, _to_utf = 0xb7cc8510
<__pyx_f_4lxml_5etree_12_BaseContext__to_utf>,
[...]
_hold = 0xb7c32500 <__pyx_f_4lxml_5etree_12_BaseContext__hold>},
  register_context = 0xb7c6af70
<__pyx_f_4lxml_5etree_12_XSLTContext_register_context>, free_context =
0xb7c360f0 <__pyx_f_4lxml_5etree_12_XSLTContext_free_context>}
-------------------------------

So, it looks like "__pyx_base" is somehow broken here. When I compile with
"-fno-strict-aliasing", I get this:

-------------------------------
(gdb) print __pyx_vtable_4lxml_5etree__XSLTContext
$1 = {__pyx_base = {_copy = 0xb7b163f0
<__pyx_f_4lxml_5etree_12_XSLTContext__copy>, _to_utf = 0xb7bbdc20
<__pyx_f_4lxml_5etree_12_BaseContext__to_utf>,
[...]
_hold = 0xb7b24c60 <__pyx_f_4lxml_5etree_12_BaseContext__hold>},
  register_context = 0xb7b5e590
<__pyx_f_4lxml_5etree_12_XSLTContext_register_context>, free_context =
0xb7b28910 <__pyx_f_4lxml_5etree_12_XSLTContext_free_context>}
-------------------------------

Also, when I compile with gcc 4.2 instead of 4.3, it just works. Stepping
through the module init function, I noticed that gcc generates the
assignment code in the wrong order. The second of the two statements

-------------------------------
  __pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base =
*__pyx_vtabptr_4lxml_5etree__BaseContext;

  *(void(**)(void))&__pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base._copy
= (void(*)(void))__pyx_f_4lxml_5etree_12_XSLTContext__copy;
-------------------------------

ends up being executed before the first one, so that the first assignment
overwrites the value of _copy(). Great. I also noticed that this only
happens when I optimise with "-march=core2", while the "prescott"
architecture works. Looks like a gcc bug to me.

Thanks for the help.

Stefan


From stefan_ml at behnel.de  Fri Feb 27 20:38:35 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 20:38:35 +0100
Subject: [Cython] Final fix for #143 before a release?
Message-ID: <49A8413B.4030204@behnel.de>

Hi,

the last remaining ticket for 0.11 is #143. I think the right fix for this
release is to disable the typing of locals in .pxd files, and to change
#143 into a feature request.

I never looked into .pxd inline functions, but if someone could do the
above for now, I think we'd be ready for release, right?

Stefan

From carl.witty at gmail.com  Fri Feb 27 20:39:24 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Fri, 27 Feb 2009 11:39:24 -0800
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <49A83D94.8050004@behnel.de>
References: <49A814C1.5050302@behnel.de>
	<ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>
	<49A83D94.8050004@behnel.de>
Message-ID: <ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>

On Fri, Feb 27, 2009 at 11:23 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Also, when I compile with gcc 4.2 instead of 4.3, it just works. Stepping
> through the module init function, I noticed that gcc generates the
> assignment code in the wrong order. The second of the two statements
>
> -------------------------------
> ?__pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base =
> *__pyx_vtabptr_4lxml_5etree__BaseContext;
>
> ?*(void(**)(void))&__pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base._copy
> = (void(*)(void))__pyx_f_4lxml_5etree_12_XSLTContext__copy;
> -------------------------------
>
> ends up being executed before the first one, so that the first assignment
> overwrites the value of _copy(). Great. I also noticed that this only
> happens when I optimise with "-march=core2", while the "prescott"
> architecture works. Looks like a gcc bug to me.

OK, I'm guessing that's actually not technically a bug.  Since
__pyx_base has no components of type (void(*)(void)), an assignment to
a storage location with that type "can't possibly" interfere with an
assignment to __pyx_base (according to the ANSI C aliasing rules), and
gcc is perfectly within its rights to reorder the two.

The fix is to get rid of the casts in the vtable initializations.

Carl

From robertwb at math.washington.edu  Fri Feb 27 20:42:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 11:42:19 -0800
Subject: [Cython] better handling of numeric types
In-Reply-To: <e7ba66e40902270513m4abe28ffk3a51a71b785185a3@mail.gmail.com>
References: <e7ba66e40902261541i4dd7b7fmc41905418a85bf68@mail.gmail.com>
	<12882211-F93E-4CAA-B6FF-7781A5E3BFDA@math.washington.edu>
	<e7ba66e40902261705t53ef27fev4c0dc43733d5644d@mail.gmail.com>
	<6433709C-3BF9-49F1-9579-A71EBCD0528B@math.washington.edu>
	<e7ba66e40902270359j671673efg278d954429b2c64f@mail.gmail.com>
	<91C70965-680F-46D8-BBFB-495496011E1D@math.washington.edu>
	<e7ba66e40902270513m4abe28ffk3a51a71b785185a3@mail.gmail.com>
Message-ID: <3927A9FA-BAF7-4990-A10F-8111C3D44C58@math.washington.edu>

On Feb 27, 2009, at 5:13 AM, Lisandro Dalcin wrote:

> On Fri, Feb 27, 2009 at 10:15 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> On Feb 27, 2009, at 3:59 AM, Lisandro Dalcin wrote:
>>
>>> Next we can discuss how to optimize this....
>>
>> As long as we don't slow down the (very common) Python int -> int/
>> long code, I'm OK with deferring optimization.
>>
>
> However, do you realize that for Py3 these optimizations are a bit
> pointless?

I'm still not sure where I stand on the union of the long and int  
types--certainly it makes some things easier, but it also seems to  
slow other things down. In any case, I foresee Cython being used with  
Python 2.x for quite some time into the future.

> Moreover, if you look at core CPython 2.X sources, you will
> find many places were were optimizations could be made, but they are
> not there...

True, but who says we can't do better :).

> Anyway, I'll try to do my best...

Don't worry, I have high confidence in you.

- Robert


From robertwb at math.washington.edu  Fri Feb 27 20:45:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 11:45:45 -0800
Subject: [Cython] Final fix for #143 before a release?
In-Reply-To: <49A8413B.4030204@behnel.de>
References: <49A8413B.4030204@behnel.de>
Message-ID: <8F808DC1-1E7A-42FA-9480-A0945E4FAC21@math.washington.edu>

On Feb 27, 2009, at 11:38 AM, Stefan Behnel wrote:

> Hi,
>
> the last remaining ticket for 0.11 is #143. I think the right fix  
> for this
> release is to disable the typing of locals in .pxd files, and to  
> change
> #143 into a feature request.

+1. I've just been too busy to look into this. Inline pxd functions  
are more important, and the syntax was a bit hackish anyways.

> I never looked into .pxd inline functions, but if someone could do the
> above for now, I think we'd be ready for release, right?

I think that's about it. There was also another post to this list  
about the beta that was never resolved. Sage passes all doctests with  
the refnanny (except for a couple that time out).

- Robert



From stefan_ml at behnel.de  Fri Feb 27 20:56:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 20:56:40 +0100
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>
References: <49A814C1.5050302@behnel.de>	<ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>	<49A83D94.8050004@behnel.de>
	<ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>
Message-ID: <49A84578.1040802@behnel.de>

Hi,

Carl Witty wrote:
> On Fri, Feb 27, 2009 at 11:23 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Also, when I compile with gcc 4.2 instead of 4.3, it just works. Stepping
>> through the module init function, I noticed that gcc generates the
>> assignment code in the wrong order. The second of the two statements
>>
>> -------------------------------
>>  __pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base =
>> *__pyx_vtabptr_4lxml_5etree__BaseContext;
>>
>>  *(void(**)(void))&__pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base._copy
>> = (void(*)(void))__pyx_f_4lxml_5etree_12_XSLTContext__copy;
>> -------------------------------
>>
>> ends up being executed before the first one, so that the first assignment
>> overwrites the value of _copy(). Great. I also noticed that this only
>> happens when I optimise with "-march=core2", while the "prescott"
>> architecture works. Looks like a gcc bug to me.
> 
> OK, I'm guessing that's actually not technically a bug.  Since
> __pyx_base has no components of type (void(*)(void)), an assignment to
> a storage location with that type "can't possibly" interfere with an
> assignment to __pyx_base (according to the ANSI C aliasing rules), and
> gcc is perfectly within its rights to reorder the two.

Thanks for the explanation.


> The fix is to get rid of the casts in the vtable initializations.

This is now ticket #226.

http://trac.cython.org/cython_trac/ticket/226

Stefan


From redalastor at gmail.com  Fri Feb 27 21:46:21 2009
From: redalastor at gmail.com (Dan)
Date: Fri, 27 Feb 2009 15:46:21 -0500
Subject: [Cython] Passing a pointer from Python
Message-ID: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>

I need to computer the sum of the absolute differences between two
arrays of bytes. The underlying C++ library I'm using (Qt) gives me
address of the arrays and the number of bytes they contain.

I can pass those values to my Cython function but how do I cdef in it
so that it uses those value to refer to the array that's already in
memory rather than creating a new one?

From robertwb at math.washington.edu  Fri Feb 27 22:08:29 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 13:08:29 -0800
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
Message-ID: <0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>

On Feb 27, 2009, at 12:46 PM, Dan wrote:

> I need to computer the sum of the absolute differences between two
> arrays of bytes. The underlying C++ library I'm using (Qt) gives me
> address of the arrays and the number of bytes they contain.
>
> I can pass those values to my Cython function but how do I cdef in it
> so that it uses those value to refer to the array that's already in
> memory rather than creating a new one?

I'm sure this can be done, but we'd need more details to answer your  
question. Are you using some Python wrapper of Qt, or are you calling  
Qt directly from Cython? How are you getting your data?

  -Robert


From redalastor at gmail.com  Fri Feb 27 22:13:55 2009
From: redalastor at gmail.com (Dan)
Date: Fri, 27 Feb 2009 16:13:55 -0500
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
Message-ID: <c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>

> I'm sure this can be done, but we'd need more details to answer your
> question. Are you using some Python wrapper of Qt, or are you calling
> Qt directly from Cython? How are you getting your data?
>
> ?-Robert

I'm using PyQt. PyQt gives me Python ints which are the address of the
byte arrays and their lengths (both arrays are the same length). I'm
just passing those ints to a Cython function, Cython doesn't need
access to Qt or PyQt at all.

From robertwb at math.washington.edu  Fri Feb 27 22:18:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 13:18:43 -0800
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
Message-ID: <722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>

On Feb 27, 2009, at 1:13 PM, Dan wrote:

>> I'm sure this can be done, but we'd need more details to answer your
>> question. Are you using some Python wrapper of Qt, or are you calling
>> Qt directly from Cython? How are you getting your data?
>>
>>  -Robert
>
> I'm using PyQt. PyQt gives me Python ints which are the address of the
> byte arrays and their lengths (both arrays are the same length). I'm
> just passing those ints to a Cython function, Cython doesn't need
> access to Qt or PyQt at all.

If x is the address to a byte array as a python int, you can do

cdef byte *my_bytes = <byte*><size_t>x

In Cython 0.11, size_t will be a native type, but until then you can  
use Py_ssize_t instead.

- Robert


From stefan_ml at behnel.de  Fri Feb 27 22:21:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 22:21:31 +0100
Subject: [Cython] PATCH: use function pointer arguments in import/export
 api	functions
In-Reply-To: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>
References: <e7ba66e40902241344v6b6ce1eelc56e3c1c9a54beaf@mail.gmail.com>
Message-ID: <49A8595B.8040505@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Just to prevent this:
> warning: ISO C forbids conversion of function pointer to object pointer type

with this patch applied, I now get a warning on this line:

    p = *(void **)&f;

src/lxml/lxml.etree.c: In function '__Pyx_ExportFunction':


src/lxml/lxml.etree.c:148416: warning: dereferencing type-punned pointer
will break strict-aliasing rules


This needs fixing, I guess. That said, I'm more in favour of enabling clean
strict-aliasing than to be strictly C89 compliant, unless we can have both. :)

Stefan


From stefan_ml at behnel.de  Fri Feb 27 22:22:55 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 27 Feb 2009 22:22:55 +0100
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <49A84578.1040802@behnel.de>
References: <49A814C1.5050302@behnel.de>	<ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>	<49A83D94.8050004@behnel.de>	<ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>
	<49A84578.1040802@behnel.de>
Message-ID: <49A859AF.8060500@behnel.de>


Stefan Behnel wrote:
>> On Fri, Feb 27, 2009 at 11:23 AM, Stefan Behnel wrote:
>>> -------------------------------
>>>  __pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base =
>>> *__pyx_vtabptr_4lxml_5etree__BaseContext;
>>>
>>>  *(void(**)(void))&__pyx_vtable_4lxml_5etree__XSLTContext.__pyx_base._copy
>>> = (void(*)(void))__pyx_f_4lxml_5etree_12_XSLTContext__copy;
>>> -------------------------------
>>>
>> OK, I'm guessing that's actually not technically a bug.  Since
>> __pyx_base has no components of type (void(*)(void)), an assignment to
>> a storage location with that type "can't possibly" interfere with an
>> assignment to __pyx_base (according to the ANSI C aliasing rules), and
>> gcc is perfectly within its rights to reorder the two.
>>
>> The fix is to get rid of the casts in the vtable initializations.
> 
> This is now ticket #226.
> http://trac.cython.org/cython_trac/ticket/226

The following patch works for me. At least, it makes lxml compile and run
with -fstrict-aliasing and without aliasing warnings. Note that it's a
Py3-only change, so you will need Py3.0 to give it a try. Any feedback is
greatly appreciated.

Stefan


diff -r cb0f315bb4f5 Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py     Fri Feb 27 00:22:20 2009 -0800
+++ b/Cython/Compiler/ModuleNode.py     Fri Feb 27 22:11:13 2009 +0100
@@ -1969,11 +1969,22 @@
                         type.base_type.vtabptr_cname))
             for meth_entry in type.scope.cfunc_entries:
                 if meth_entry.func_cname:
+                    code.putln('#if PY_MAJOR_VERSION >= 3')
+                    cast = '(%s)' % (
+                        meth_entry.type.declaration_code('(*)'))
+                    code.putln(
+                        "%s.%s = %s%s;" % (
+                            type.vtable_cname,
+                            meth_entry.cname,
+                            cast,
+                            meth_entry.func_cname))
+                    code.putln('#else')
                     code.putln(
                         "*(void(**)(void))&%s.%s = (void(*)(void))%s;" % (
                             type.vtable_cname,
                             meth_entry.cname,
                             meth_entry.func_cname))
+                    code.putln('#endif')

     def generate_typeptr_assignment_code(self, entry, code):
         # Generate code to initialise the typeptr of an extension


From redalastor at gmail.com  Fri Feb 27 22:29:26 2009
From: redalastor at gmail.com (Dan)
Date: Fri, 27 Feb 2009 16:29:26 -0500
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
Message-ID: <c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>

> If x is the address to a byte array as a python int, you can do
>
> cdef byte *my_bytes = <byte*><size_t>x
>
> In Cython 0.11, size_t will be a native type, but until then you can
> use Py_ssize_t instead.
>
> - Robert

When I try to build it, it tells me byte is not a type identifier.

My code is as follow, what am I doing wrong?

def fitness(int new_image, int original_image, int size):

    cdef byte *new_pixels = <byte*><Py_ssize_t>size
    cdef byte *original_pixels = <byte*><Py_ssize_t>size

    cdef long spam = 0
    cdef int i = 0
    for i in range(len(new_pixels)):
        spam += abs(original_pixels[i] - new_pixels[i])

    return 1020 * len((original_pixels)) - spam

From dagss at student.matnat.uio.no  Fri Feb 27 23:03:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 27 Feb 2009 23:03:18 +0100 (CET)
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
Message-ID: <5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>

Dan wrote:
>> If x is the address to a byte array as a python int, you can do
>>
>> cdef byte *my_bytes = <byte*><size_t>x
>>
>> In Cython 0.11, size_t will be a native type, but until then you can
>> use Py_ssize_t instead.
>>
>> - Robert
>
> When I try to build it, it tells me byte is not a type identifier.

Use "char", not "byte".

Dag Sverre


From redalastor at gmail.com  Fri Feb 27 23:15:50 2009
From: redalastor at gmail.com (Dan)
Date: Fri, 27 Feb 2009 17:15:50 -0500
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
	<5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>
Message-ID: <c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>

> Use "char", not "byte".

Of course, you are right. However now it crashes the Python
interpreter when called.

Current code:

def fitness(int new_image, int original_image, int size):

    cdef unsigned char *new_pixels = <unsigned char*><Py_ssize_t>size
    cdef unsigned char *original_pixels = <unsigned char*><Py_ssize_t>size

    cdef long spam = 0
    cdef int i = 0
    for i in range(size):
        spam += abs(original_pixels[i] - new_pixels[i])

    return 255 * size - spam

From redalastor at gmail.com  Fri Feb 27 23:33:16 2009
From: redalastor at gmail.com (Dan)
Date: Fri, 27 Feb 2009 17:33:16 -0500
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
	<5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>
	<c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>
Message-ID: <c065d79a0902271433q4adba646le22870106db1ba4c@mail.gmail.com>

Testing and commenting out stuff revealed that the problem is
accessing the arrays. No other code will make the Python interpreter
crash.

So either there's something wrong with my declaration or the way I access it.

On Fri, Feb 27, 2009 at 5:15 PM, Dan <redalastor at gmail.com> wrote:
>> Use "char", not "byte".
>
> Of course, you are right. However now it crashes the Python
> interpreter when called.
>
> Current code:
>
> def fitness(int new_image, int original_image, int size):
>
> ? ?cdef unsigned char *new_pixels = <unsigned char*><Py_ssize_t>size
> ? ?cdef unsigned char *original_pixels = <unsigned char*><Py_ssize_t>size
>
> ? ?cdef long spam = 0
> ? ?cdef int i = 0
> ? ?for i in range(size):
> ? ? ? ?spam += abs(original_pixels[i] - new_pixels[i])
>
> ? ?return 255 * size - spam
>

From robertwb at math.washington.edu  Fri Feb 27 23:46:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Feb 2009 14:46:55 -0800
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <c065d79a0902271433q4adba646le22870106db1ba4c@mail.gmail.com>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
	<5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>
	<c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>
	<c065d79a0902271433q4adba646le22870106db1ba4c@mail.gmail.com>
Message-ID: <2D311273-A1D8-47EC-930E-E37C2D23B237@math.washington.edu>

On Feb 27, 2009, at 2:33 PM, Dan wrote:

> Testing and commenting out stuff revealed that the problem is
> accessing the arrays. No other code will make the Python interpreter
> crash.
>
> So either there's something wrong with my declaration or the way I  
> access it.
>
> On Fri, Feb 27, 2009 at 5:15 PM, Dan <redalastor at gmail.com> wrote:
>>> Use "char", not "byte".
>>
>> Of course, you are right. However now it crashes the Python
>> interpreter when called.
>>
>> Current code:
>>
>> def fitness(int new_image, int original_image, int size):
>>
>>    cdef unsigned char *new_pixels = <unsigned char*><Py_ssize_t>size
>>    cdef unsigned char *original_pixels = <unsigned  
>> char*><Py_ssize_t>size

I am almost sure you don't want to be casting size to an unsigned  
char*...

>>
>>    cdef long spam = 0
>>    cdef int i = 0
>>    for i in range(size):
>>        spam += abs(original_pixels[i] - new_pixels[i])
>>
>>    return 255 * size - spam
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From redalastor at gmail.com  Sat Feb 28 00:00:56 2009
From: redalastor at gmail.com (Dan)
Date: Fri, 27 Feb 2009 18:00:56 -0500
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <2D311273-A1D8-47EC-930E-E37C2D23B237@math.washington.edu>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
	<5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>
	<c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>
	<c065d79a0902271433q4adba646le22870106db1ba4c@mail.gmail.com>
	<2D311273-A1D8-47EC-930E-E37C2D23B237@math.washington.edu>
Message-ID: <c065d79a0902271500u56d3ca45l16db853b9c673058@mail.gmail.com>

> I am almost sure you don't want to be casting size to an unsigned
> char*...

So what *do* I want? How exactly should I write my declaration so I
can access the unsigned chars in the array?

From greg.ewing at canterbury.ac.nz  Sat Feb 28 01:54:01 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 28 Feb 2009 13:54:01 +1300
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>
References: <49A814C1.5050302@behnel.de>
	<ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>
	<49A83D94.8050004@behnel.de>
	<ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>
Message-ID: <49A88B29.3090506@canterbury.ac.nz>

Carl Witty wrote:
> Since
> __pyx_base has no components of type (void(*)(void)), an assignment to
> a storage location with that type "can't possibly" interfere with an
> assignment to __pyx_base (according to the ANSI C aliasing rules), and
> gcc is perfectly within its rights to reorder the two.

Blarg.

<rant>
I'm getting really annoyed by the way the C standards are
going. All these tricky rules concerning casting and aliasing
are making it very awkward to use the language as a high-level
assembler with any confidence that it will do what you ?&^%$
well *tell* it!
</rant>

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat Feb 28 02:08:35 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 28 Feb 2009 14:08:35 +1300
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
	<5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>
	<c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>
Message-ID: <49A88E93.3030508@canterbury.ac.nz>

Dan wrote:

> def fitness(int new_image, int original_image, int size):
> 
>     cdef unsigned char *new_pixels = <unsigned char*><Py_ssize_t>size
>     cdef unsigned char *original_pixels = <unsigned char*><Py_ssize_t>size

I think you want to cast 'new_image' and 'original_image' there,
not 'size'!

Also, it would be safer to declare the parameters directly as
being of type Py_ssize_t instead of going through int, just in
case int is not big enough.

   def fitness(Py_ssize_t new_image, Py_ssize_t original_image, Py_ssize_t size):

     cdef unsigned char *new_pixels = <unsigned char*>new_image
     cdef unsigned char *original_pixels = <unsigned char*>original_image

-- 
Greg

From redalastor at gmail.com  Sat Feb 28 02:33:06 2009
From: redalastor at gmail.com (Dan)
Date: Fri, 27 Feb 2009 20:33:06 -0500
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <49A88E93.3030508@canterbury.ac.nz>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
	<5c18709e0303876476def793becd4b39.squirrel@webmail.uio.no>
	<c065d79a0902271415yfb00d44s45c44d101a82a5fc@mail.gmail.com>
	<49A88E93.3030508@canterbury.ac.nz>
Message-ID: <c065d79a0902271733x41434940h5011a7c36facf17c@mail.gmail.com>

It works!

That simple method was a huge bottleneck in my code. It is now 80
times faster than it was using regular Python code and 16 times faster
than Python accessing directly the memory through a buffer.

Thanks a lot!

From stefan_ml at behnel.de  Sat Feb 28 07:30:33 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Feb 2009 07:30:33 +0100
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
Message-ID: <49A8DA09.8020404@behnel.de>

Hi,

Dan wrote:
>     cdef long spam = 0

Note that this uses a C long, which is size bound (as opposed to a Python
long). Depending on your data, you might need more than that, e.g. a "long
long" (although that's not 100% portable).


>     cdef int i = 0

Make sure you use Py_ssize_t also for i!


>     for i in range(len(new_pixels)):
>         spam += abs(original_pixels[i] - new_pixels[i])
> 
>     return 1020 * len((original_pixels)) - spam

Regarding this bit, try running Cython with the "--annotate" option. This
will generate an HTML file from your code that shows where things are
running in Python code (yellow) rather than C code (try a mouse click). 80
times faster sounds so slowish that there must still be type conversions
involved. For example, have you imported "abs" from stdlib.h instead of
using the Python builtin? And len() is a Python function, too, so you
better use the size that you already know, rather than converting types
between C and Python.

Stefan

From stefan_ml at behnel.de  Sat Feb 28 07:41:34 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Feb 2009 07:41:34 +0100
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <49A88B29.3090506@canterbury.ac.nz>
References: <49A814C1.5050302@behnel.de>	<ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>	<49A83D94.8050004@behnel.de>	<ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>
	<49A88B29.3090506@canterbury.ac.nz>
Message-ID: <49A8DC9E.4040806@behnel.de>


Greg Ewing wrote:
> I'm getting really annoyed by the way the C standards are
> going. All these tricky rules concerning casting and aliasing
> are making it very awkward to use the language as a high-level
> assembler with any confidence that it will do what you ?&^%$
> well *tell* it!

:) I'm the one who sumbled over this, so guess what I think. However, it's
not a big problem, as you have to build Py2 modules without strict aliasing
anyway. Only Py3 can build cleanly with that optimisation.

Stefan


From redalastor at gmail.com  Sat Feb 28 16:21:09 2009
From: redalastor at gmail.com (Dan)
Date: Sat, 28 Feb 2009 10:21:09 -0500
Subject: [Cython] Passing a pointer from Python
In-Reply-To: <49A8DA09.8020404@behnel.de>
References: <c065d79a0902271246p26a84e1ej34cf65282e0db646@mail.gmail.com>
	<0A23C9E4-66F7-4ABE-90BE-78BD7A67F39A@math.washington.edu>
	<c065d79a0902271313j6257a047g215b08bb8a7c2499@mail.gmail.com>
	<722686E8-472D-4A2A-8CA6-4A9967018716@math.washington.edu>
	<c065d79a0902271329n6c9254b6vde19646e3930d047@mail.gmail.com>
	<49A8DA09.8020404@behnel.de>
Message-ID: <c065d79a0902280721rdf34dc6gd068f557e1e4bd5d@mail.gmail.com>

> Note that this uses a C long, which is size bound (as opposed to a Python
> long). Depending on your data, you might need more than that, e.g. a "long
> long" (although that's not 100% portable).

In fact, an int is probably ok, I used a long to be on the safe side.

>> ? ? for i in range(len(new_pixels)):
>> ? ? ? ? spam += abs(original_pixels[i] - new_pixels[i])
>>
>> ? ? return 1020 * len((original_pixels)) - spam
>
> Regarding this bit, try running Cython with the "--annotate" option. This
> will generate an HTML file from your code that shows where things are
> running in Python code (yellow) rather than C code (try a mouse click). 80
> times faster sounds so slowish that there must still be type conversions
> involved. For example, have you imported "abs" from stdlib.h instead of
> using the Python builtin?

I already changed the len for the size. And yes, I was using the
python abs function. I imported the C one now. Now only the def line
and the return line are using Python (which is okay since the function
is called from Python).

Switching for the C abs is now 20 times faster than before. So roughly
a 1600 times improvement over pure Python.

Thanks!
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Sat Feb 28 16:41:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 28 Feb 2009 16:41:54 +0100 (CET)
Subject: [Cython] strict aliasing problem with subclass method
In-Reply-To: <49A88B29.3090506@canterbury.ac.nz>
References: <49A814C1.5050302@behnel.de>
	<ac1e4c420902270911t60d3501bqb696aeb21178cebb@mail.gmail.com>
	<49A83D94.8050004@behnel.de>
	<ac1e4c420902271139g2ff4ebf6hc02b7ccfeae95697@mail.gmail.com>
	<49A88B29.3090506@canterbury.ac.nz>
Message-ID: <fadfb500aec17fab30e3b85da1e1e76a.squirrel@webmail.uio.no>

Greg Ewing wrote:
> Carl Witty wrote:
>> Since
>> __pyx_base has no components of type (void(*)(void)), an assignment to
>> a storage location with that type "can't possibly" interfere with an
>> assignment to __pyx_base (according to the ANSI C aliasing rules), and
>> gcc is perfectly within its rights to reorder the two.
>
> Blarg.
>
> <rant>
> I'm getting really annoyed by the way the C standards are
> going. All these tricky rules concerning casting and aliasing
> are making it very awkward to use the language as a high-level
> assembler with any confidence that it will do what you ?&^%$
> well *tell* it!
> </rant>

There's another side of the coin though, so just for the sake of it: The
aliasing rules (and especially the "restrict" keyword) means that as long
as one is following the standards, C can finally start being able to
generate as efficient code as FORTRAN. There are certain things the
compiler simply cannot do automatically for you; FORTRAN "solves" this
problem by being a very restricted language, while C has taken a different
route and tries to keep the flexibility while still going in a direction
which makes it as efficient.

Whether this could have been done in a different way though I don't know
-- I guess it could, but I don't think it would be as natural, involving
more special magic syntax when you wanted more speed.

Disclaimer: I'm just learning these things and should probably know better
than to speak up :-) but if I'm wrong I'll learn something.

Dag Sverre