From dagss at student.matnat.uio.no  Sun Feb  1 00:28:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 01 Feb 2009 00:28:55 +0100
Subject: [Cython] char* and string object
In-Reply-To: <EAC9277B-CAE9-40FA-8EFF-DBEDE8521A46@me.com>
References: <EAC9277B-CAE9-40FA-8EFF-DBEDE8521A46@me.com>
Message-ID: <4984DEB7.5090101@student.matnat.uio.no>

Jean-Alexandre Peyroux wrote:
> Hello,
> 
> I think i did not understand the handling character string. This is an  
> example :
> 
> user.pyx :
> 
> cdef class User:
>     def __init__(self, char* name):
>         self.cname = name
> 
>     cpdef printName(self):
>         print "My name is %s" % self.cname
> 
> user.pxd :
> 
> cdef class User:
>     cdef readonly char* cname
> 
>     cpdef printName(self)
> 
> 
<snip>
> In [5]: u.cname
> Out[5]: '_3'
> 
> I do not understand. My string is completely random, as if it was  
> pointing in the wrong location of memory.
> I missed step ?

This is expected. When you do "self.cname = name", then you get a 
pointer to the character buffer inside the "name" string object. Then, 
"name" goes out of scope and is deallocated, along with its memory, and 
perhaps filled with something else.

Moral? Only use conversion to char* when you really need it because you 
call C code or similar, and also then you need to hold on to the 
corresponding Python string as well.

-- 
Dag Sverre

From peyroux at me.com  Sun Feb  1 00:23:36 2009
From: peyroux at me.com (Jean-Alexandre Peyroux)
Date: Sun, 01 Feb 2009 00:23:36 +0100
Subject: [Cython] char* and string object
In-Reply-To: <4984DEB7.5090101@student.matnat.uio.no>
References: <EAC9277B-CAE9-40FA-8EFF-DBEDE8521A46@me.com>
	<4984DEB7.5090101@student.matnat.uio.no>
Message-ID: <94C0DB9D-0AB5-413E-814E-DACD69DBB968@me.com>

Thank you for your advice.

Regards

Le 1 f?vr. 09 ? 00:28, Dag Sverre Seljebotn a ?crit :

> Jean-Alexandre Peyroux wrote:
>> Hello,
>>
>> I think i did not understand the handling character string. This is  
>> an
>> example :
>>
>> user.pyx :
>>
>> cdef class User:
>>    def __init__(self, char* name):
>>        self.cname = name
>>
>>    cpdef printName(self):
>>        print "My name is %s" % self.cname
>>
>> user.pxd :
>>
>> cdef class User:
>>    cdef readonly char* cname
>>
>>    cpdef printName(self)
>>
>>
> <snip>
>> In [5]: u.cname
>> Out[5]: '_3'
>>
>> I do not understand. My string is completely random, as if it was
>> pointing in the wrong location of memory.
>> I missed step ?
>
> This is expected. When you do "self.cname = name", then you get a
> pointer to the character buffer inside the "name" string object. Then,
> "name" goes out of scope and is deallocated, along with its memory,  
> and
> perhaps filled with something else.
>
> Moral? Only use conversion to char* when you really need it because  
> you
> call C code or similar, and also then you need to hold on to the
> corresponding Python string as well.
>
> -- 
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From wstein at gmail.com  Sun Feb  1 01:21:48 2009
From: wstein at gmail.com (William Stein)
Date: Sat, 31 Jan 2009 16:21:48 -0800
Subject: [Cython] http://shootout.alioth.debian.org/
Message-ID: <85e81ba30901311621h397e054cw9477f2d1b84b8f31@mail.gmail.com>

Hi,

Craig Citro just gave a very nice Cython talk (in lieu of Robert
Bradshaw), and there were about 30 questions from the audience, hence
a lot of interest:

   http://sage.math.washington.edu/talks/20090131-seapig/cython.pdf

One question that came up -- could somebody add cython to
http://shootout.alioth.debian.org/
?

It might be fun, since basically C wins every single one of the
language shootouts -- so Cython likely would do very well.

 -- William

-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From casevh at gmail.com  Mon Feb  2 08:38:26 2009
From: casevh at gmail.com (Case Vanhorsen)
Date: Sun, 1 Feb 2009 23:38:26 -0800
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
Message-ID: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>

Hello,

I just tried to install Cython 0.11 beta with Python 3.0. I get the
following error:

case:~/src/Cython-0.11.beta$ py30 setup.py install
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    from Cython.Compiler.Version import version
  File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
    from Shadow import *
ImportError: No module named Shadow

It does install successfully using Python 2.5 and 2.6. Please let me
know if there is any other information you need.

casevh

From dagss at student.matnat.uio.no  Mon Feb  2 10:20:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 10:20:27 +0100
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
In-Reply-To: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
References: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
Message-ID: <4986BADB.50202@student.matnat.uio.no>

Case Vanhorsen wrote:
> Hello,
>
> I just tried to install Cython 0.11 beta with Python 3.0. I get the
> following error:
>
> case:~/src/Cython-0.11.beta$ py30 setup.py install
> Traceback (most recent call last):
>   File "setup.py", line 5, in <module>
>     from Cython.Compiler.Version import version
>   File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
>     from Shadow import *
> ImportError: No module named Shadow
>
> It does install successfully using Python 2.5 and 2.6. Please let me
> know if there is any other information you need.
>   
I had the impression that Cython doesn't support being run under Python 
3 (though Stefan would have the definitive answer on that). I think you 
must install on Python 2.6, and then compile your .pyx to .c under 2.6, 
and THEN you can compile the c file with the Python 3 headers to get a 
Python 3 module.

Dag Sverre

From dalcinl at gmail.com  Mon Feb  2 18:12:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 14:12:19 -0300
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
Message-ID: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>

I'm now getting many warnings like below:

src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
src/petsc4py_PETSc.c:3793: warning: value computed is not used

A possible fix, written following the way core Python Py_XDECREF,
would be the following:


diff -r 83075bb3a319 Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py	Fri Jan 30 23:10:52 2009 +0100
+++ b/Cython/Compiler/ModuleNode.py	Mon Feb 02 14:08:33 2009 -0300
@@ -2334,7 +2334,7 @@
 #define __Pyx_GOTREF(r) __Pyx_Refnanny_GOTREF(__pyx_refchk, r, __LINE__)
 #define __Pyx_GIVEREF(r) __Pyx_Refnanny_GIVEREF(__pyx_refchk, r, __LINE__)
 #define __Pyx_DECREF(r) __Pyx_Refnanny_DECREF(__pyx_refchk, r, __LINE__)
-#define __Pyx_XDECREF(r) (r ? __Pyx_Refnanny_DECREF(__pyx_refchk, r,
__LINE__) : 0)
+#define __Pyx_XDECREF(r) if((r) == NULL) ; else __Pyx_DECREF(r)
 #define __Pyx_SetupRefcountContext(name) \
   void* __pyx_refchk = __Pyx_Refnanny_NewContext(name, __LINE__)
 #define __Pyx_FinishRefcountContext()
__Pyx_Refnanny_FinishContext(__pyx_refchk)
@@ -2347,6 +2347,6 @@
 #define __Pyx_SetupRefcountContext(name)
 #define __Pyx_FinishRefcountContext() 0
 #endif /* CYTHON_REFNANNY */
-#define __Pyx_XGIVEREF(r) (r ? __Pyx_GIVEREF(r) : 0)
-#define __Pyx_XGOTREF(r) (r ? __Pyx_GOTREF(r) : 0)
+#define __Pyx_XGIVEREF(r) if((r) == NULL) ; else __Pyx_GIVEREF(r)
+#define __Pyx_XGOTREF(r) if((r) == NULL) ; else __Pyx_GOTREF(r)
 """)


Can I push this?

Other possible way would be to use "do { ... } while(0)" blocks, I
prefer them much more, but that is not a common pattern in Python C
source code.




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb  2 18:31:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 18:31:54 +0100
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
In-Reply-To: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
References: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
Message-ID: <49872E0A.3090801@student.matnat.uio.no>

Lisandro Dalcin wrote:
> I'm now getting many warnings like below:
> 
> src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
> src/petsc4py_PETSc.c:3793: warning: value computed is not used
> 
> A possible fix, written following the way core Python Py_XDECREF,
> would be the following:
> 
> 
> diff -r 83075bb3a319 Cython/Compiler/ModuleNode.py
> --- a/Cython/Compiler/ModuleNode.py	Fri Jan 30 23:10:52 2009 +0100
> +++ b/Cython/Compiler/ModuleNode.py	Mon Feb 02 14:08:33 2009 -0300
> @@ -2334,7 +2334,7 @@
>  #define __Pyx_GOTREF(r) __Pyx_Refnanny_GOTREF(__pyx_refchk, r, __LINE__)
>  #define __Pyx_GIVEREF(r) __Pyx_Refnanny_GIVEREF(__pyx_refchk, r, __LINE__)
>  #define __Pyx_DECREF(r) __Pyx_Refnanny_DECREF(__pyx_refchk, r, __LINE__)
> -#define __Pyx_XDECREF(r) (r ? __Pyx_Refnanny_DECREF(__pyx_refchk, r,
> __LINE__) : 0)
> +#define __Pyx_XDECREF(r) if((r) == NULL) ; else __Pyx_DECREF(r)
...
> Can I push this?

Go ahead!

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Feb  2 18:34:16 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 18:34:16 +0100
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
In-Reply-To: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
References: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
Message-ID: <49872E98.4050800@student.matnat.uio.no>

Lisandro Dalcin wrote:
> I'm now getting many warnings like below:
> 
> src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
> src/petsc4py_PETSc.c:3793: warning: value computed is not used
> 

Does this mean that you compile with CYTHON_REFNANNY though?

-- 
Dag Sverre

From dalcinl at gmail.com  Mon Feb  2 19:07:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 15:07:21 -0300
Subject: [Cython] fixing refnanny macros to avoid GCC warnings
In-Reply-To: <49872E98.4050800@student.matnat.uio.no>
References: <e7ba66e40902020912t42606f56te51ad9714aa55eec@mail.gmail.com>
	<49872E98.4050800@student.matnat.uio.no>
Message-ID: <e7ba66e40902021007g6b6cd6a6vc23ebdf9f484606c@mail.gmail.com>

On Mon, Feb 2, 2009 at 2:34 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> I'm now getting many warnings like below:
>>
>> src/petsc4py_PETSc.c: In function '__pyx_f_8petsc4py_5PETSc_asarray':
>> src/petsc4py_PETSc.c:3793: warning: value computed is not used
>>
>
> Does this mean that you compile with CYTHON_REFNANNY though?
>
> --

No yet. I'm on the process of trying it. But right now I'm having
trouble surely related to temp allocation, and in the process to send
a example for your consideration.

> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb  2 21:28:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 17:28:06 -0300
Subject: [Cython] possible problems with temp allocation?
In-Reply-To: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
References: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
Message-ID: <e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>

See the attached pyx file.

A valgrind run points me to the line where you have
"__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;" basically, DECREF is being
called with __pyx_t_1 being NULL...


  /* "/u/dalcinl/tmp/foo.pyx":10
 *         out = numpy.asarray(out)
 *         out.flat[:] = array
 *     return out             # <<<<<<<<<<<<<<
 *
 *
 */
  __Pyx_INCREF(__pyx_v_out);
  __pyx_r = __pyx_v_out;
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  goto __pyx_L0;


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.pyx
Type: application/octet-stream
Size: 319 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/71442e89/attachment.obj 

From dalcinl at gmail.com  Mon Feb  2 21:41:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 17:41:19 -0300
Subject: [Cython] comments about refnanny implementation...
Message-ID: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>

Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
was mine, but you know... I'm so idiot sometimes... This is going to
be a REAL pain to make it work, specially if you want to make this
easily portable to the many platforms out there.... For example, Do
the current implementation work on Windows?

Would you agree to completelly change the way the refnany helper
routines are imported?. I'm thinking on a ad-hoc way of doing that,
similar but simpler to the way Cython extension modules can export API
functions.

The only requisite would be that the 'refnanny.so' extension module
can be imported. Then, when you build Cython extensions with
CYTHON_REFNANNY defined, some special code in the module init function
imports the 'refnanny' module and get the pointer to a struct having
pointers to the various routines... No more need to dlopen() or even
import refnanny in order to enable it....

What do you think?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb  2 22:01:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 22:01:30 +0100
Subject: [Cython] possible problems with temp allocation?
In-Reply-To: <e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>
References: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
	<e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>
Message-ID: <49875F2A.8040704@student.matnat.uio.no>

Lisandro Dalcin wrote:
> See the attached pyx file.
> 
> A valgrind run points me to the line where you have
> "__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;" basically, DECREF is being
> called with __pyx_t_1 being NULL...

Does this fix it?

http://hg.cython.org/cython-devel/rev/b9fc866182dc


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Feb  2 22:11:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 02 Feb 2009 22:11:26 +0100
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
Message-ID: <4987617E.6070707@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
> was mine, but you know... I'm so idiot sometimes... This is going to
> be a REAL pain to make it work, specially if you want to make this
> easily portable to the many platforms out there.... For example, Do
> the current implementation work on Windows?

I have no idea. I know next to nothing about Windows development (which 
goes beyond pre-2000-era Delphi GUI programming). And it would be a real 
pain for me to find or set up a Windows workstation with enough tools to 
find out.

> Would you agree to completelly change the way the refnany helper
> routines are imported?. I'm thinking on a ad-hoc way of doing that,
> similar but simpler to the way Cython extension modules can export API
> functions.
> 
> The only requisite would be that the 'refnanny.so' extension module
> can be imported. Then, when you build Cython extensions with
> CYTHON_REFNANNY defined, some special code in the module init function
> imports the 'refnanny' module and get the pointer to a struct having
> pointers to the various routines... No more need to dlopen() or even
> import refnanny in order to enable it....

This sounds like the way Cython implements "api", one could build on 
that. (But writing something dedicated in C might be just as well, 
because all the other things can potentially be subject to refnannying).

The reasons I've done what I've done is mostly to get something as quick 
as possible. If what I've done can't work in Windows, I'd be happy to 
accept a patch along the lines you suggest, but it is pretty far down on 
my own list. (It would mean an extra pointer lookup per refcount call I 
think, but speed is not an issue here...)

An alternative is to write runtests.py to disable the refnanny on 
Windows. As long as it is run on Unix prior to a Cython release it 
should work.


-- 
Dag Sverre

From dalcinl at gmail.com  Mon Feb  2 22:07:16 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 18:07:16 -0300
Subject: [Cython] possible problems with temp allocation?
In-Reply-To: <49875F2A.8040704@student.matnat.uio.no>
References: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>
	<e7ba66e40902021228q63c4df73k4c05d1e8370b9161@mail.gmail.com>
	<49875F2A.8040704@student.matnat.uio.no>
Message-ID: <e7ba66e40902021307h41703c72s1aca22323001c5ab@mail.gmail.com>

On Mon, Feb 2, 2009 at 6:01 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> See the attached pyx file.
>>
>> A valgrind run points me to the line where you have
>> "__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;" basically, DECREF is being
>> called with __pyx_t_1 being NULL...
>
> Does this fix it?
>
> http://hg.cython.org/cython-devel/rev/b9fc866182dc
>

Indeed. Many thanks!

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb  2 22:10:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 18:10:21 -0300
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <4987617E.6070707@student.matnat.uio.no>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
	<4987617E.6070707@student.matnat.uio.no>
Message-ID: <e7ba66e40902021310j56fd0797tc376a690971c9ba6@mail.gmail.com>

OK, let me write an test a patch. I'm starting right now...


On Mon, Feb 2, 2009 at 6:11 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
>> was mine, but you know... I'm so idiot sometimes... This is going to
>> be a REAL pain to make it work, specially if you want to make this
>> easily portable to the many platforms out there.... For example, Do
>> the current implementation work on Windows?
>
> I have no idea. I know next to nothing about Windows development (which
> goes beyond pre-2000-era Delphi GUI programming). And it would be a real
> pain for me to find or set up a Windows workstation with enough tools to
> find out.
>
>> Would you agree to completelly change the way the refnany helper
>> routines are imported?. I'm thinking on a ad-hoc way of doing that,
>> similar but simpler to the way Cython extension modules can export API
>> functions.
>>
>> The only requisite would be that the 'refnanny.so' extension module
>> can be imported. Then, when you build Cython extensions with
>> CYTHON_REFNANNY defined, some special code in the module init function
>> imports the 'refnanny' module and get the pointer to a struct having
>> pointers to the various routines... No more need to dlopen() or even
>> import refnanny in order to enable it....
>
> This sounds like the way Cython implements "api", one could build on
> that. (But writing something dedicated in C might be just as well,
> because all the other things can potentially be subject to refnannying).
>
> The reasons I've done what I've done is mostly to get something as quick
> as possible. If what I've done can't work in Windows, I'd be happy to
> accept a patch along the lines you suggest, but it is pretty far down on
> my own list. (It would mean an extra pointer lookup per refcount call I
> think, but speed is not an issue here...)
>
> An alternative is to write runtests.py to disable the refnanny on
> Windows. As long as it is run on Unix prior to a Cython release it
> should work.
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb  2 22:14:32 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 18:14:32 -0300
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <e7ba66e40902021310j56fd0797tc376a690971c9ba6@mail.gmail.com>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
	<4987617E.6070707@student.matnat.uio.no>
	<e7ba66e40902021310j56fd0797tc376a690971c9ba6@mail.gmail.com>
Message-ID: <e7ba66e40902021314x1396f55chbb4092987dc6e500@mail.gmail.com>

BTW, would it be hard for you to disable the generation of refnanny
stuff for functions that do not involve any Python object, just C
types (like those that can be called with the GIL released) ??

On Mon, Feb 2, 2009 at 6:10 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> OK, let me write an test a patch. I'm starting right now...
>
>
> On Mon, Feb 2, 2009 at 6:11 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Lisandro Dalcin wrote:
>>> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
>>> was mine, but you know... I'm so idiot sometimes... This is going to
>>> be a REAL pain to make it work, specially if you want to make this
>>> easily portable to the many platforms out there.... For example, Do
>>> the current implementation work on Windows?
>>
>> I have no idea. I know next to nothing about Windows development (which
>> goes beyond pre-2000-era Delphi GUI programming). And it would be a real
>> pain for me to find or set up a Windows workstation with enough tools to
>> find out.
>>
>>> Would you agree to completelly change the way the refnany helper
>>> routines are imported?. I'm thinking on a ad-hoc way of doing that,
>>> similar but simpler to the way Cython extension modules can export API
>>> functions.
>>>
>>> The only requisite would be that the 'refnanny.so' extension module
>>> can be imported. Then, when you build Cython extensions with
>>> CYTHON_REFNANNY defined, some special code in the module init function
>>> imports the 'refnanny' module and get the pointer to a struct having
>>> pointers to the various routines... No more need to dlopen() or even
>>> import refnanny in order to enable it....
>>
>> This sounds like the way Cython implements "api", one could build on
>> that. (But writing something dedicated in C might be just as well,
>> because all the other things can potentially be subject to refnannying).
>>
>> The reasons I've done what I've done is mostly to get something as quick
>> as possible. If what I've done can't work in Windows, I'd be happy to
>> accept a patch along the lines you suggest, but it is pretty far down on
>> my own list. (It would mean an extra pointer lookup per refcount call I
>> think, but speed is not an issue here...)
>>
>> An alternative is to write runtests.py to disable the refnanny on
>> Windows. As long as it is run on Unix prior to a Cython release it
>> should work.
>>
>>
>> --
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Mon Feb  2 22:25:41 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Mon, 02 Feb 2009 13:25:41 -0800
Subject: [Cython] comments about refnanny implementation...
In-Reply-To: <4987617E.6070707@student.matnat.uio.no>
References: <e7ba66e40902021241m19d32370p9d490230eab5d1e@mail.gmail.com>
	<4987617E.6070707@student.matnat.uio.no>
Message-ID: <498764D5.2040105@gmail.com>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:

Hi,

>> Dag, I know that the original idea of the dlopen() with RTLD_GLOBAL
>> was mine, but you know... I'm so idiot sometimes... This is going to
>> be a REAL pain to make it work, specially if you want to make this
>> easily portable to the many platforms out there.... For example, Do
>> the current implementation work on Windows?
> 
> I have no idea. I know next to nothing about Windows development (which 
> goes beyond pre-2000-era Delphi GUI programming). And it would be a real 
> pain for me to find or set up a Windows workstation with enough tools to 
> find out.

Hmm, judging from http://msdn.microsoft.com/en-us/library/ms684175.aspx 
and various manpages as well as my recollection of the way Python loads 
extension modules I believe that the problem does not exist on Windows, 
i.e. local vs. global symbols. But I obviously have not tested this.

>> Would you agree to completelly change the way the refnany helper
>> routines are imported?. I'm thinking on a ad-hoc way of doing that,
>> similar but simpler to the way Cython extension modules can export API
>> functions.
>>
>> The only requisite would be that the 'refnanny.so' extension module
>> can be imported. Then, when you build Cython extensions with
>> CYTHON_REFNANNY defined, some special code in the module init function
>> imports the 'refnanny' module and get the pointer to a struct having
>> pointers to the various routines... No more need to dlopen() or even
>> import refnanny in order to enable it....
> 
> This sounds like the way Cython implements "api", one could build on 
> that. (But writing something dedicated in C might be just as well, 
> because all the other things can potentially be subject to refnannying).
> 
> The reasons I've done what I've done is mostly to get something as quick 
> as possible. If what I've done can't work in Windows, I'd be happy to 
> accept a patch along the lines you suggest, but it is pretty far down on 
> my own list. (It would mean an extra pointer lookup per refcount call I 
> think, but speed is not an issue here...)
> 
> An alternative is to write runtests.py to disable the refnanny on 
> Windows. As long as it is run on Unix prior to a Cython release it 
> should work.


Well, it doesn't take much imagination to see someone coding Windows 
specific code that cannot be tested via Cygwin or Linux, so making this 
work on any supported platform would be a good thing. I have to port 
some code to MSVC that currently uses dlopen with RTLD_GLOBAL, but if 
Lisandro fixes the problem with Cython I would certainly be interested 
to see the result.

Cheers,

Michael

From dalcinl at gmail.com  Tue Feb  3 01:36:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Feb 2009 21:36:09 -0300
Subject: [Cython] PATCH: new refnanny API
Message-ID: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>

Here you have a preliminary patch for loading the refnanny API
skipping dlopen() hackery.

For this to work, just make Cython/Runtime/refnanny.pyx a extension
module, and put it somewhere in your PYTHONPATH. That's all!! Now
build other extensions with -DCYTHON_REFNANNY and enjoy...

Take into account that I'm not completelly happy with the current
patch, more work is required.

1) As namespaces are a honking great idea, I do not see the point of
using "Refnanny" prefixes on class and function names at refnanny.py

2) I'm a bit uncomfortable with the way I had to initialize stuff at
module initialization function.

3) Many unittests in mpi4py pass, but I'm also gettting some
segfaults. In particular, the refnanny stuff should no be emmited for
"pure" cdef functions (I mean, those with no Python objects at all in
args, retvals and local vars). For example, in mpi4py I have a
finalizer routine that is registered with Py_AtExit(), then it is
called after Py_Finalize(), thus no Python code should run. I'm also
getting some bad interaction between refnanny and exception handling,
but I need to dive deeper to catch the problem, no yet sure what is
going on.


We can continue discussion on cython-dev, but if any of you want to
make this a ticket, go ahead!



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: REFNANNY.diff
Type: application/octet-stream
Size: 6206 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/3d066110/attachment.obj 

From greg.ewing at canterbury.ac.nz  Tue Feb  3 04:44:32 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 03 Feb 2009 16:44:32 +1300
Subject: [Cython] Equivalent of __new__() in python
In-Reply-To: <f70460160808011248p1f5305c6kd57373315cc86f2f@mail.gmail.com>
References: <f70460160808011248p1f5305c6kd57373315cc86f2f@mail.gmail.com>
Message-ID: <4987BDA0.3010504@canterbury.ac.nz>

Arnaud Bergeron wrote:
> I want to have only one instance at a time with a single value of val.
> 
> I already know how to do this in python using the __new__() method and
> a WeakValueDictionary.  I am looking for a way to do the equivalent in
> cython.

This can't be done yet, because you don't get a chance
to directly override the tp_new slot. I intend to fix
this in some future version of Pyrex.

-- 
Greg

From dagss at student.matnat.uio.no  Tue Feb  3 10:16:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 10:16:24 +0100
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
Message-ID: <49880B68.2000102@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Here you have a preliminary patch for loading the refnanny API
> skipping dlopen() hackery.
>
> For this to work, just make Cython/Runtime/refnanny.pyx a extension
> module, and put it somewhere in your PYTHONPATH. That's all!! Now
> build other extensions with -DCYTHON_REFNANNY and enjoy...
>
> Take into account that I'm not completelly happy with the current
> patch, more work is required.
>
> 1) As namespaces are a honking great idea, I do not see the point of
> using "Refnanny" prefixes on class and function names at refnanny.py
>   
By all means.

> 2) I'm a bit uncomfortable with the way I had to initialize stuff at
> module initialization function.
>   
Looks OK with me.
> 3) Many unittests in mpi4py pass, but I'm also gettting some
> segfaults. In particular, the refnanny stuff should no be emmited for
> "pure" cdef functions (I mean, those with no Python objects at all in
> args, retvals and local vars). For example, in mpi4py I have a
> finalizer routine that is registered with Py_AtExit(), then it is
> called after Py_Finalize(), thus no Python code should run. I'm also
> getting some bad interaction between refnanny and exception handling,
> but I need to dive deeper to catch the problem, no yet sure what is
> going on.
>   
Can't you declare the function as "nogil"? I.e.

cdef nogil int myfunc(): ...

I already disable the refnanny if "nogil" is declared, and then Python 
operations are disallowed. Doing this automatically though is a bit 
hard, I'd have to make sure that no Python temporaries are ever 
constructed in the function, which is more than a five-minute job. If 
you give a good reason the nogil modifier doesn't do the trick I can 
look into it.

I've applied your patch, thanks! RTLD_GLOBAL-issues aside, it is a lot 
easier to set up the refnanny on other code than the test suite now.

Please vote: Should we install the refnanny into site-packages when 
doing a Cython install? Then we could just tell users "try compiling 
with the -DCYTHON_REFNANNY flag" to get diagnostics.

Dag Sverre

From muogoro at gmail.com  Tue Feb  3 11:30:26 2009
From: muogoro at gmail.com (Daniele Pianu)
Date: Tue, 3 Feb 2009 11:30:26 +0100
Subject: [Cython] Propagating exceptions with functions which returns void*
Message-ID: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>

Probably I've not completely understood the C-function
exception-propagating mechanism. I've a function with a prototype like
this

void* getItem( void* self, int i )

The function returns the i-th item in self. Some index and bounds
checking are implemented, so if, for example, I specify a negative
index or an index out of self bounds I raise and exception. In cython
the function is declared in this way:

cdef void* getItem( void* self, int i ) except NULL

But when an error occurs and the exception is raised I always receive
a message which says that the exception is ignored. Obviously, the
function continues its execution and terminates whit a segmentation
fault.

The question is: how can I propagate to caller functions the raised exception?
Thanks for your help (and for the great work that you've done with cython :D ),
Daniele

From dagss at student.matnat.uio.no  Tue Feb  3 11:35:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 11:35:06 +0100
Subject: [Cython] Propagating exceptions with functions which returns
 void*
In-Reply-To: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
Message-ID: <49881DDA.5040207@student.matnat.uio.no>

Daniele Pianu wrote:
> Probably I've not completely understood the C-function
> exception-propagating mechanism. I've a function with a prototype like
> this
>
> void* getItem( void* self, int i )
>   
Where is this function declared? What kind of exception is raised? C has 
no exceptions, and C++ exception needs a different form of declaration: 
*http://docs.cython.org/docs/pyrex_differences.html#c-exception-handling*

The "except NULL" is for propagating within-Cython exceptions, see 
http://docs.cython.org/docs/language_basics.html#error-return-values

Is this the reason for your problem? I might be missing the mark here.

Dag Sverre

From muogoro at gmail.com  Tue Feb  3 11:46:52 2009
From: muogoro at gmail.com (Daniele Pianu)
Date: Tue, 3 Feb 2009 11:46:52 +0100
Subject: [Cython] Propagating exceptions with functions which returns
	void*
In-Reply-To: <49881DDA.5040207@student.matnat.uio.no>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
	<49881DDA.5040207@student.matnat.uio.no>
Message-ID: <66a8c45d0902030246q7471da71u25f2f2abc8d909ac@mail.gmail.com>

The function is a cdef extension type's method. Here the part of the
implementation which raises the exceptions:

  cdef void *getItem( self, int i ) except NULL:
    cdef void* tmpPtr

    # Check NULL pointer
    if self.rawPtr == NULL:
      raise ValueError( "Null pointer" )
    # Check specified index
    if i < 0:
      raise IndexError( "Unvalid index" )
    # Bound checking
    if self._layout[0] > 0 and i >= self._layout[0]:
      raise IndexError( "Index out of bounds" )

    # Code used for get the i-th value
    .......

self.rawPtr is the void pointer stored by the extension type.
self._layout is a simple tuple which defines the "shape" of
self.rawPtr, used for bounds checking. The function is called by other
extension types which extend the extension type which defines the
getItem method. What do you main with "The "except NULL" is for
propagating within-Cython exceptions"? Do I need to put the getItem
call in a try-except block in the cython caller?

Daniele

2009/2/3 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
> Daniele Pianu wrote:
>> Probably I've not completely understood the C-function
>> exception-propagating mechanism. I've a function with a prototype like
>> this
>>
>> void* getItem( void* self, int i )
>>
> Where is this function declared? What kind of exception is raised? C has
> no exceptions, and C++ exception needs a different form of declaration:
> *http://docs.cython.org/docs/pyrex_differences.html#c-exception-handling*
>
> The "except NULL" is for propagating within-Cython exceptions, see
> http://docs.cython.org/docs/language_basics.html#error-return-values
>
> Is this the reason for your problem? I might be missing the mark here.
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Tue Feb  3 12:33:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 12:33:29 +0100
Subject: [Cython] Propagating exceptions with functions which returns
 void*
In-Reply-To: <66a8c45d0902030246q7471da71u25f2f2abc8d909ac@mail.gmail.com>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>	<49881DDA.5040207@student.matnat.uio.no>
	<66a8c45d0902030246q7471da71u25f2f2abc8d909ac@mail.gmail.com>
Message-ID: <49882B89.6020209@student.matnat.uio.no>

Daniele Pianu wrote:
> The function is a cdef extension type's method. Here the part of the
> implementation which raises the exceptions:
>
>   cdef void *getItem( self, int i ) except NULL:
>     cdef void* tmpPtr
>
>     # Check NULL pointer
>     if self.rawPtr == NULL:
>       raise ValueError( "Null pointer" )
>     # Check specified index
>     if i < 0:
>       raise IndexError( "Unvalid index" )
>     # Bound checking
>     if self._layout[0] > 0 and i >= self._layout[0]:
>       raise IndexError( "Index out of bounds" )
>
>     # Code used for get the i-th value
>     .......
>
> self.rawPtr is the void pointer stored by the extension type.
> self._layout is a simple tuple which defines the "shape" of
> self.rawPtr, used for bounds checking. The function is called by other
> extension types which extend the extension type which defines the
> getItem method. What do you main with "The "except NULL" is for
> propagating within-Cython exceptions"? Do I need to put the getItem
> call in a try-except block in the cython caller?
>   
I don't think what I said applies in your case. I'm still a bit 
confused, as "self" was typed as "void*" in your original example, while 
here getitem seems to be part of a class.

Here's a minimal testcase, which seems to work OK for me. Can you please 
modify it until it explains your situation?

Dag Sverre

"""
 >>> f()
Traceback (most recent call last):
...
ValueError: foo
"""
cdef class A:
    cdef void* ptr
    def __init__(self):
        self.ptr = NULL
    cdef void* getitem(self) except NULL:
        if self.ptr is NULL:
            raise ValueError("foo")

def f():
    cdef A a = A()
    print a.getitem() == NULL

From dalcinl at gmail.com  Tue Feb  3 14:03:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 11:03:39 -0200
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <49880B68.2000102@student.matnat.uio.no>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
	<49880B68.2000102@student.matnat.uio.no>
Message-ID: <e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>

On Tue, Feb 3, 2009 at 7:16 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>>
> Can't you declare the function as "nogil"? I.e.
>
> cdef nogil int myfunc(): ...
>

I'll give a try. That would be fine with me...

>
> I've applied your patch, thanks! RTLD_GLOBAL-issues aside, it is a lot
> easier to set up the refnanny on other code than the test suite now.
>

Indeed.

> Please vote: Should we install the refnanny into site-packages when
> doing a Cython install? Then we could just tell users "try compiling
> with the -DCYTHON_REFNANNY flag" to get diagnostics.
>

Perhaps betther would be to put iside Cython.Runtime, in such a way
that the full dotted name would be 'Cython.Runtime.refnanny' ?? I'm
not a big fan of populating site-packages directory. Perhaps if the
module is renamed to 'pyxrefnanny'?




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb  3 14:33:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 14:33:44 +0100
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>	<49880B68.2000102@student.matnat.uio.no>
	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
Message-ID: <498847B8.2030600@student.matnat.uio.no>

Lisandro Dalcin wrote:
>
> Perhaps betther would be to put iside Cython.Runtime, in such a way
> that the full dotted name would be 'Cython.Runtime.refnanny' ?? I'm
> not a big fan of populating site-packages directory. Perhaps if the
> module is renamed to 'pyxrefnanny'?
>   
If your code could be modified to first check for just "refnanny", and 
if that is not found, then attempt Cython.Runtime.refnanny?

This would allow runtests.py to compile refnanny outside of the package 
(which may be changed relative to an installed version), while falling 
back to the one in the package otherwise.

An alternative is to have a new toplevel package cythonruntime or 
cython_runtime. The point is that runtests.py must be able to load it 
into sys.path together with the code in the current directory 
(preferably without having to copy the whole Cython tree).

I tried doing this but it didn't work immediately so I'm giving up. Feel 
free :-)

Dag Sverre

From dalcinl at gmail.com  Tue Feb  3 14:45:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 11:45:02 -0200
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <498847B8.2030600@student.matnat.uio.no>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
	<49880B68.2000102@student.matnat.uio.no>
	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
	<498847B8.2030600@student.matnat.uio.no>
Message-ID: <e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>

On Tue, Feb 3, 2009 at 11:33 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>>
> If your code could be modified to first check for just "refnanny", and
> if that is not found, then attempt Cython.Runtime.refnanny?

OK, I'll implement this approach, I like it...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb  3 14:50:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 11:50:22 -0200
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>
	<49880B68.2000102@student.matnat.uio.no>
	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>
	<498847B8.2030600@student.matnat.uio.no>
	<e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>
Message-ID: <e7ba66e40902030550o5afb7d0dn4c94532192819571@mail.gmail.com>

Dag, I see you pushed my patch, but the refnanny functions are still
'public'... Any good reason for this?

2009/2/3 Lisandro Dalcin <dalcinl at gmail.com>:
> On Tue, Feb 3, 2009 at 11:33 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Lisandro Dalcin wrote:
>>>
>> If your code could be modified to first check for just "refnanny", and
>> if that is not found, then attempt Cython.Runtime.refnanny?
>
> OK, I'll implement this approach, I like it...
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb  3 14:55:20 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Feb 2009 14:55:20 +0100
Subject: [Cython] PATCH: new refnanny API
In-Reply-To: <e7ba66e40902030550o5afb7d0dn4c94532192819571@mail.gmail.com>
References: <e7ba66e40902021636u2e5fccdbwd89d2a85c6cc5a3c@mail.gmail.com>	<49880B68.2000102@student.matnat.uio.no>	<e7ba66e40902030503i3d6f1bf1i4fe772b35bc7cbfc@mail.gmail.com>	<498847B8.2030600@student.matnat.uio.no>	<e7ba66e40902030545v1e2fcb5eg3fda937244a95362@mail.gmail.com>
	<e7ba66e40902030550o5afb7d0dn4c94532192819571@mail.gmail.com>
Message-ID: <49884CC8.2090607@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Dag, I see you pushed my patch, but the refnanny functions are still
> 'public'... Any good reason for this?
>   
No, that was a slip (I had to merge and didn't notice).

You've obviously got more attention to detail here than me, so feel free 
to push this and other changes on the refnanny yourself if you want to.

Dag Sverre

From dalcinl at gmail.com  Tue Feb  3 18:06:10 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 15:06:10 -0200
Subject: [Cython] review this fix for bad interaction of refnanny + 'with
	gil'
Message-ID: <e7ba66e40902030906l70a55fd4j8bfc6c875098428@mail.gmail.com>

After the patch below (plus some changes in refnanny.pyx, that's for
other mail), I get the full mpi4py testsuite pass....

Basically, the refnanny context needs to be finished BEFORE the GIL
state is released...

Could any of you tell me if this fix would be required at some other
place? I got a bit lost. ;-) ...

diff -r e35f20896675 Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py	Tue Feb 03 14:00:31 2009 -0200
+++ b/Cython/Compiler/Nodes.py	Tue Feb 03 14:49:10 2009 -0200
@@ -1181,8 +1181,7 @@
         for entry in lenv.arg_entries:
             if entry.type.is_pyobject and
lenv.control_flow.get_state((entry.name, 'source')) != 'arg':
                 code.put_var_decref(entry)
-        if acquire_gil:
-            code.putln("PyGILState_Release(_save);")
+
         # code.putln("/* TODO: decref scope object */")
         # ----- Return
         # This code is duplicated in ModuleNode.generate_module_init_func
@@ -1199,6 +1198,9 @@
                                              Naming.retval_cname,
                                              err_val)

+        if acquire_gil:
+            code.putln("PyGILState_Release(_save);")
+
         if not self.return_type.is_void:
             code.putln("return %s;" % Naming.retval_cname)




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb  3 18:31:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 15:31:46 -0200
Subject: [Cython] more enhancement to refnanny API
Message-ID: <e7ba66e40902030931w65722a00n8d299a95c02219df@mail.gmail.com>

Dag, some ideas for your consideration ...

1) What about changing those bare void* context pointer to PyObject*

2) What about making FinishContext(PyObject**), and make it set the
arg to NULL on return?.

3) refnanny.Error (formely refnanny.RefnannyExeception) is raised in
Context.end() in case of errors. Is this fine? What about raising at
the end of FinishContext()?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb  3 20:40:33 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 17:40:33 -0200
Subject: [Cython] a problem with sizeof() return type
Message-ID: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>

Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.

Implementing this the right way would require to make Cython recognize
'size_t' as a native type, more or less in the way it already
recognizes 'Py_ssize_t'. What do you think about this?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Tue Feb  3 21:01:43 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 03 Feb 2009 12:01:43 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
Message-ID: <4988A2A7.8060204@gmail.com>

Lisandro Dalcin wrote:
> Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.

Well, you might be technically correct since that is what the C standard 
asks for, but according to various literature, i.e. for example

http://publications.gbdirect.co.uk/c_book/chapter5/sizeof_and_malloc.html

"The sizeof operator returns the size in bytes of its operand. Whether 
the result of sizeof is unsigned int or unsigned long is implementation 
defined?which is why the declaration of malloc above ducked the issue by 
omitting any parameter information; "


> Implementing this the right way would require to make Cython recognize
> 'size_t' as a native type, more or less in the way it already
> recognizes 'Py_ssize_t'. What do you think about this?
> 

Mmmh, do you want to define size_t in Cython or pull it in from 
somewhere? At least for C code that ought to be rather tricky for VS6 
for example which needs to work since we are still supporting Python 2.3 
on Windows. But you could work around the issue by defining size_t in 
Cython as either unsigned int, unsigned long or unsigned long long 
depending on the platform. But that could potentially clash with various 
C++ runtime libs.

Do you think that people will see overflow issues? I think the tradeoff 
of following the C standard here is rather high since in reality size_t 
in C is rather badly handled. And if you guys decide to follow the C 
standard here someone ought to break out Python 2.6 and VS 6 and verify 
that things are still working :)

Cheers,

Michael

From jasone at canonware.com  Tue Feb  3 21:29:01 2009
From: jasone at canonware.com (Jason Evans)
Date: Tue, 03 Feb 2009 12:29:01 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988A2A7.8060204@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
Message-ID: <4988A90D.30107@canonware.com>

Michael Abshoff wrote:
> Lisandro Dalcin wrote:
>> Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.
> 
> Well, you might be technically correct since that is what the C standard 
> asks for, but according to various literature, i.e. for example
> 
> http://publications.gbdirect.co.uk/c_book/chapter5/sizeof_and_malloc.html
> 
> "The sizeof operator returns the size in bytes of its operand. Whether 
> the result of sizeof is unsigned int or unsigned long is implementation 
> defined?which is why the declaration of malloc above ducked the issue by 
> omitting any parameter information; "

That book appears to have been published in 1991...

Personally, I've found the lack of native size_t knowledge in Cython to 
be a major pain.  In practice though, the following is sufficient for 
every modern system I know of, with the likely exception of Windows:

     cdef extern from "sys/types.h":
         ctypedef unsigned long size_t

Jason

From michael.abshoff at googlemail.com  Tue Feb  3 21:43:50 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 03 Feb 2009 12:43:50 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988A90D.30107@canonware.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>
Message-ID: <4988AC86.5070306@gmail.com>

Jason Evans wrote:
> Michael Abshoff wrote:
>> Lisandro Dalcin wrote:
>>> Cython treats sizeof() as if it returns 'int', but IMHO it should be 'size_t'.
>> Well, you might be technically correct since that is what the C standard 
>> asks for, but according to various literature, i.e. for example
>>
>> http://publications.gbdirect.co.uk/c_book/chapter5/sizeof_and_malloc.html
>>
>> "The sizeof operator returns the size in bytes of its operand. Whether 
>> the result of sizeof is unsigned int or unsigned long is implementation 
>> defined?which is why the declaration of malloc above ducked the issue by 
>> omitting any parameter information; "
> 
> That book appears to have been published in 1991...

Yeah, it has been a while, but then Python requires a C89 compiler :)

> Personally, I've found the lack of native size_t knowledge in Cython to 
> be a major pain.  In practice though, the following is sufficient for 
> every modern system I know of, with the likely exception of Windows:
> 
>      cdef extern from "sys/types.h":
>          ctypedef unsigned long size_t


Well, make it unsigned long long on 64 bit Windows and it ought to work 
nearly everywhere. I am not aware of any LLP64 bit OS that Cython 
supports besides 64 bit Windows. Obviously there is no sys/types.h on 
Windows, but that is a different story :)

> Jason

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dalcinl at gmail.com  Tue Feb  3 21:57:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 3 Feb 2009 18:57:09 -0200
Subject: [Cython] range optimization and noqil
Message-ID: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>

Trying to cythonizing this:

cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
        j += 1
    return j


I get this failure... Is this possible and easy to fix ?

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                 ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:18: Accessing Python global or builtin not
allowed without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                   ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:20: Converting to Python object not allowed without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                 ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:18: Constructing Python tuple not allowed without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
                 ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:18: Calling gil-requiring function without gil

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef int f() nogil:
    cdef int i,j=0, n=3
    for i in range(n):
            ^
------------------------------------------------------------

/u/dalcinl/tmp/gil.pyx:3:13: Iterating over Python object not allowed
without gil

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Feb  4 00:30:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Feb 2009 00:30:15 +0100
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988AC86.5070306@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com>
Message-ID: <4988D387.7070304@student.matnat.uio.no>

Michael Abshoff wrote:
> Jason Evans wrote:
>>      cdef extern from "sys/types.h":
>>          ctypedef unsigned long size_t
> 
> 
> Well, make it unsigned long long on 64 bit Windows and it ought to work 
> nearly everywhere. I am not aware of any LLP64 bit OS that Cython 
> supports besides 64 bit Windows. Obviously there is no sys/types.h on 
> Windows, but that is a different story :)

Just a slightly related note:

Note that with this kind of declaration, all Cython uses in 90% of the 
situations is the fact that it is an unsigned integral type. "unsigned 
char size_t" it will do the same in almost all situations -- in the end, 
"size_t" is simply emitted to the .c file.

The 10% is overflow exceptions from coercion from Python integer -- and 
pushing that decision too to the C compiler is on my personal todo-list 
(though far down). In general Cython prefers to leave knowledge of the 
size/capacity of types to the C compiler.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Feb  4 10:49:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Feb 2009 10:49:48 +0100
Subject: [Cython] range optimization and noqil
In-Reply-To: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>
References: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>
Message-ID: <498964BC.8000706@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Trying to cythonizing this:
>
> cdef int f() nogil:
>     cdef int i,j=0, n=3
>     for i in range(n):
>         j += 1
>     return j
>
>
> I get this failure... Is this possible and easy to fix ?
>   

Thanks, good point. This is now

http://trac.cython.org/cython_trac/ticket/205

I think it is definitely 0.12 stuff, so use a for-from construct if you 
need this for 0.11.

If you just want to run the refnanny, perhaps consider just disabling 
the line in the source raising the error on your local copy?

Dag Sverre

From dalcinl at gmail.com  Wed Feb  4 15:06:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 12:06:28 -0200
Subject: [Cython] range optimization and noqil
In-Reply-To: <498964BC.8000706@student.matnat.uio.no>
References: <e7ba66e40902031257s68ec0f9epb6fcd3ef5eda5bbc@mail.gmail.com>
	<498964BC.8000706@student.matnat.uio.no>
Message-ID: <e7ba66e40902040606o27ebc504tbb44ad6bb9106434@mail.gmail.com>

Yes, I changed my code to use for-loops.

But I still think that the refnanny should not be emmited for 'pure'
functions. Declaring them as nogil does the trick, but this can cause
other sorce code changed. For example, in Cython/Include, you have a
pxd with C stdlib stuff. These declarations do not have the 'nogil',
then you cannot use them inside other 'nogil' functions.

I'll try to work on this in the near future...


On Wed, Feb 4, 2009 at 7:49 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> Trying to cythonizing this:
>>
>> cdef int f() nogil:
>>     cdef int i,j=0, n=3
>>     for i in range(n):
>>         j += 1
>>     return j
>>
>>
>> I get this failure... Is this possible and easy to fix ?
>>
>
> Thanks, good point. This is now
>
> http://trac.cython.org/cython_trac/ticket/205
>
> I think it is definitely 0.12 stuff, so use a for-from construct if you
> need this for 0.11.
>
> If you just want to run the refnanny, perhaps consider just disabling
> the line in the source raising the error on your local copy?
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb  4 15:17:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 12:17:11 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988D387.7070304@student.matnat.uio.no>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com> <4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com> <4988D387.7070304@student.matnat.uio.no>
Message-ID: <e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>

Well, then how should we fix the return type in the code below ? (at
Cython/Compiler/ExprNodes.py)

class SizeofNode(ExprNode):
    #  Abstract base class for sizeof(x) expression nodes.

    type = PyrexTypes.c_int_type


I still think that the only sane way to fix that is to make Cython
understand 'size_t' as a already defined C type. That would let us to
properly define the return type of sizeof. Furthermore, that would let
users to forget about having to typedef ssize_t to something.

There are strong objections for my approach? What could be the
potential problems?


On Tue, Feb 3, 2009 at 9:30 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Michael Abshoff wrote:
>> Jason Evans wrote:
>>>      cdef extern from "sys/types.h":
>>>          ctypedef unsigned long size_t
>>
>>
>> Well, make it unsigned long long on 64 bit Windows and it ought to work
>> nearly everywhere. I am not aware of any LLP64 bit OS that Cython
>> supports besides 64 bit Windows. Obviously there is no sys/types.h on
>> Windows, but that is a different story :)
>
> Just a slightly related note:
>
> Note that with this kind of declaration, all Cython uses in 90% of the
> situations is the fact that it is an unsigned integral type. "unsigned
> char size_t" it will do the same in almost all situations -- in the end,
> "size_t" is simply emitted to the .c file.
>
> The 10% is overflow exceptions from coercion from Python integer -- and
> pushing that decision too to the C compiler is on my personal todo-list
> (though far down). In general Cython prefers to leave knowledge of the
> size/capacity of types to the C compiler.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Feb  4 15:37:13 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Feb 2009 15:37:13 +0100
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>
	<4988D387.7070304@student.matnat.uio.no>
	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
Message-ID: <4989A819.9040202@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Well, then how should we fix the return type in the code below ? (at
> Cython/Compiler/ExprNodes.py)
> 
> class SizeofNode(ExprNode):
>     #  Abstract base class for sizeof(x) expression nodes.
> 
>     type = PyrexTypes.c_int_type
> 
> 
> I still think that the only sane way to fix that is to make Cython
> understand 'size_t' as a already defined C type. That would let us to
> properly define the return type of sizeof. Furthermore, that would let
> users to forget about having to typedef ssize_t to something.
> 
> There are strong objections for my approach? What could be the
> potential problems?

The objections as I understand them is that not all C compilers 
understand the "size_t" type. Not all projects (amongst them CPython) 
adhere to the latest C standards.

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Wed Feb  4 15:40:10 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 04 Feb 2009 06:40:10 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4989A819.9040202@student.matnat.uio.no>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>	<4988D387.7070304@student.matnat.uio.no>	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
	<4989A819.9040202@student.matnat.uio.no>
Message-ID: <4989A8CA.1020809@gmail.com>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:
>> Well, then how should we fix the return type in the code below ? (at
>> Cython/Compiler/ExprNodes.py)
>>
>> class SizeofNode(ExprNode):
>>     #  Abstract base class for sizeof(x) expression nodes.
>>
>>     type = PyrexTypes.c_int_type
>>
>>
>> I still think that the only sane way to fix that is to make Cython
>> understand 'size_t' as a already defined C type. That would let us to
>> properly define the return type of sizeof. Furthermore, that would let
>> users to forget about having to typedef ssize_t to something.
>>
>> There are strong objections for my approach? What could be the
>> potential problems?
> 
> The objections as I understand them is that not all C compilers 
> understand the "size_t" type. Not all projects (amongst them CPython) 
> adhere to the latest C standards.
> 

Well, an arguing that those damn C compilers ought to be fixed to follow 
the C standard also is unrealistic. Note that one such broken compiler 
which needs help to understand what a size_t is even in C++ mode is VC6 
which is Python 2.3's official compiler on Windows, so not supporting it 
would be a bad move as long as Cython claims to support Python 2.3. And 
some people have reported bugs in Cython with Python 2.3 and VC6 in the 
last couple months, so it is still being used. That is sad since VC6 
leaves much to be desired, but we live in a world of legacy code :)

Cheers,

Michael

From dalcinl at gmail.com  Wed Feb  4 15:40:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 12:40:25 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4989A819.9040202@student.matnat.uio.no>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com> <4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com> <4988D387.7070304@student.matnat.uio.no>
	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
	<4989A819.9040202@student.matnat.uio.no>
Message-ID: <e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>

On Wed, Feb 4, 2009 at 12:37 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>>
>> There are strong objections for my approach? What could be the
>> potential problems?
>
> The objections as I understand them is that not all C compilers
> understand the "size_t" type. Not all projects (amongst them CPython)
> adhere to the latest C standards.
>

And then could you tell me how core CPython could be build with such a
C compiler...

Please look, at core CPython sources. To be sure, inspect Python-2.3
sources. They use 'size_t' freely after including "Python.h"


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Wed Feb  4 15:58:54 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 04 Feb 2009 06:58:54 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>
	<4988D387.7070304@student.matnat.uio.no>	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>	<4989A819.9040202@student.matnat.uio.no>
	<e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>
Message-ID: <4989AD2E.3080900@gmail.com>

Lisandro Dalcin wrote:
> On Wed, Feb 4, 2009 at 12:37 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>>> There are strong objections for my approach? What could be the
>>> potential problems?
>> The objections as I understand them is that not all C compilers
>> understand the "size_t" type. Not all projects (amongst them CPython)
>> adhere to the latest C standards.
>>
> 
> And then could you tell me how core CPython could be build with such a
> C compiler...
> 
> Please look, at core CPython sources. To be sure, inspect Python-2.3
> sources. They use 'size_t' freely after including "Python.h"
> 
> 

Well, is Python.h guaranteed to be included in every Cython extension? I 
guess it would be. But what about pure C files linked against an 
extension for example? People would be required to either include 
Python.h for size_t or find it somewhere else in the system. Not that 
this is hard, but it will confuse some people :)

I am sure Python 2.3 (and also later releases obviously) defines it 
appropriately if it isn't provided by the system. Overall I think this 
is a worthwhile change, the new code just ought to be verified on some 
of the older known bad compilers.

While poking around I found the following nugget in Sage:

# Memory management
cdef extern from "stdlib.h":
     ctypedef unsigned long size_t


So I would expect some buggy code to be exposed by the suggested change 
above.

Cheers,

Michael

From dalcinl at gmail.com  Wed Feb  4 16:31:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Feb 2009 13:31:01 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4989AD2E.3080900@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com> <4988A90D.30107@canonware.com>
	<4988AC86.5070306@gmail.com> <4988D387.7070304@student.matnat.uio.no>
	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>
	<4989A819.9040202@student.matnat.uio.no>
	<e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>
	<4989AD2E.3080900@gmail.com>
Message-ID: <e7ba66e40902040731x17bf511cj6273c38a292d43e4@mail.gmail.com>

On Wed, Feb 4, 2009 at 12:58 PM, Michael Abshoff
<michael.abshoff at googlemail.com> wrote:
>
> Well, is Python.h guaranteed to be included in every Cython extension?
> I guess it would be.
>

AFAIK, yes. See yourself:

class ModuleScope(Scope):
   ....
    def __init__(self, name, parent_module, context):
        ....
        self.python_include_files = ["Python.h", "structmember.h"]

> But what about pure C files linked against an
> extension for example? People would be required to either include
> Python.h for size_t or find it somewhere else in the system. Not that
> this is hard, but it will confuse some people :)
>

Not sure if I understand you. If pure C file want to use 'size_t',
they are in charge of it.

> I am sure Python 2.3 (and also later releases obviously) defines it
> appropriately if it isn't provided by the system. Overall I think this
> is a worthwhile change, the new code just ought to be verified on some
> of the older known bad compilers.
>

Well, then I'll try to write a patch for all you to review and test.

> While poking around I found the following nugget in Sage:
>
> # Memory management
> cdef extern from "stdlib.h":
>     ctypedef unsigned long size_t
>
>
> So I would expect some buggy code to be exposed by the suggested change
> above.
>

Well, such a change would expose buggy code and perhaps be backward
incompatible, right?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Feb  4 20:31:54 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Feb 2009 20:31:54 +0100
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
In-Reply-To: <4986BADB.50202@student.matnat.uio.no>
References: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
	<4986BADB.50202@student.matnat.uio.no>
Message-ID: <4989ED2A.2080303@behnel.de>



Dag Sverre Seljebotn wrote:
> Case Vanhorsen wrote:
>> I just tried to install Cython 0.11 beta with Python 3.0. I get the
>> following error:
>>
>> case:~/src/Cython-0.11.beta$ py30 setup.py install
>> Traceback (most recent call last):
>>   File "setup.py", line 5, in <module>
>>     from Cython.Compiler.Version import version
>>   File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
>>     from Shadow import *
>> ImportError: No module named Shadow
>>
>> It does install successfully using Python 2.5 and 2.6. Please let me
>> know if there is any other information you need.
>>   
> I had the impression that Cython doesn't support being run under Python 
> 3 (though Stefan would have the definitive answer on that). I think you 
> must install on Python 2.6, and then compile your .pyx to .c under 2.6, 
> and THEN you can compile the c file with the Python 3 headers to get a 
> Python 3 module.

That is true. While the generated C source code supports Python 2.3 through
3.0 without changes, the Cython compiler itself requires Python 2.3 through
2.6 to run.

As this does not limit Cython's functionality or applicability in any way,
it's not a major priority for us to migrate Cython's own source code to Py3.

Stefan

From michael.abshoff at googlemail.com  Wed Feb  4 21:15:57 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 04 Feb 2009 12:15:57 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <e7ba66e40902040731x17bf511cj6273c38a292d43e4@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>
	<4988A90D.30107@canonware.com>	<4988AC86.5070306@gmail.com>
	<4988D387.7070304@student.matnat.uio.no>	<e7ba66e40902040617u478a967ata96afef361bea1d4@mail.gmail.com>	<4989A819.9040202@student.matnat.uio.no>	<e7ba66e40902040640k235a5ec1kdec637f1e37442d3@mail.gmail.com>	<4989AD2E.3080900@gmail.com>
	<e7ba66e40902040731x17bf511cj6273c38a292d43e4@mail.gmail.com>
Message-ID: <4989F77D.2020903@gmail.com>

Lisandro Dalcin wrote:
> On Wed, Feb 4, 2009 at 12:58 PM, Michael Abshoff
> <michael.abshoff at googlemail.com> wrote:
>> Well, is Python.h guaranteed to be included in every Cython extension?
>> I guess it would be.
>>
> 
> AFAIK, yes. See yourself:
> 
> class ModuleScope(Scope):
>    ....
>     def __init__(self, name, parent_module, context):
>         ....
>         self.python_include_files = ["Python.h", "structmember.h"]

Good, I don't venture too deeply into the Cython compiler.

>> But what about pure C files linked against an
>> extension for example? People would be required to either include
>> Python.h for size_t or find it somewhere else in the system. Not that
>> this is hard, but it will confuse some people :)
>>
> 
> Not sure if I understand you. If pure C file want to use 'size_t',
> they are in charge of it.

Yes, it was completely dumb to raise the issue here. I needed some sleep 
and I hope I got caught up on sleep in the last couple hours :)

>> I am sure Python 2.3 (and also later releases obviously) defines it
>> appropriately if it isn't provided by the system. Overall I think this
>> is a worthwhile change, the new code just ought to be verified on some
>> of the older known bad compilers.
>>
> 
> Well, then I'll try to write a patch for all you to review and test.
> 
>> While poking around I found the following nugget in Sage:
>>
>> # Memory management
>> cdef extern from "stdlib.h":
>>     ctypedef unsigned long size_t
>>
>>
>> So I would expect some buggy code to be exposed by the suggested change
>> above.
>>
> 
> Well, such a change would expose buggy code and perhaps be backward
> incompatible, right?

Yes, but code that was buggy and is being exposed by fixing a bug in an 
upstream tool is acceptable since the developer needs to fix that bug 
anyway. It might be painful, but the end result is better code. Hadn't I 
looked around for size_t definitions in Sage we would have hit that bug 
the first time someone does a 4 GB+ allocation on Windows 64 bit mode 
once Sage has been ported to MSVC. That is not in the near future, but 
since I do porting of code all day I can tell you that it is one of the 
best tools to find bugs in code since all those wrong assumptions in 
your code come back to bite you in the ass :)

Cheers,

Michael

From casevh at gmail.com  Wed Feb  4 21:52:52 2009
From: casevh at gmail.com (Case Vanhorsen)
Date: Wed, 4 Feb 2009 12:52:52 -0800
Subject: [Cython] Cython 0.11 beta fails to install with Python 3.0
In-Reply-To: <4989ED2A.2080303@behnel.de>
References: <99e0b9530902012338x24b63692g34166dcdac217baf@mail.gmail.com>
	<4986BADB.50202@student.matnat.uio.no> <4989ED2A.2080303@behnel.de>
Message-ID: <99e0b9530902041252l5d8ceee6g5a9c4118e55c92@mail.gmail.com>

On Wed, Feb 4, 2009 at 11:31 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
>
> Dag Sverre Seljebotn wrote:
>> Case Vanhorsen wrote:
>>> I just tried to install Cython 0.11 beta with Python 3.0. I get the
>>> following error:
>>>
>>> case:~/src/Cython-0.11.beta$ py30 setup.py install
>>> Traceback (most recent call last):
>>>   File "setup.py", line 5, in <module>
>>>     from Cython.Compiler.Version import version
>>>   File "/home/case/src/Cython-0.11.beta/Cython/__init__.py", line 2, in <module>
>>>     from Shadow import *
>>> ImportError: No module named Shadow
>>>
>>> It does install successfully using Python 2.5 and 2.6. Please let me
>>> know if there is any other information you need.
>>>
>> I had the impression that Cython doesn't support being run under Python
>> 3 (though Stefan would have the definitive answer on that). I think you
>> must install on Python 2.6, and then compile your .pyx to .c under 2.6,
>> and THEN you can compile the c file with the Python 3 headers to get a
>> Python 3 module.
>
> That is true. While the generated C source code supports Python 2.3 through
> 3.0 without changes, the Cython compiler itself requires Python 2.3 through
> 2.6 to run.
>
> As this does not limit Cython's functionality or applicability in any way,
> it's not a major priority for us to migrate Cython's own source code to Py3.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Thanks for the information. That's perfectly fine; I just wasn't aware of that.

Case

From jasone at canonware.com  Thu Feb  5 20:00:01 2009
From: jasone at canonware.com (Jason Evans)
Date: Thu, 05 Feb 2009 11:00:01 -0800
Subject: [Cython] Docstrings for 'cdef public' attributes?
Message-ID: <498B3731.6090001@canonware.com>

Is there any reasonable way to attach docstrings to 'cdef public' 
attributes?  The following does not work:

cdef class Spam:
     cdef public int spam:
         """
             Docstring for spam.
         """

spam.pyx:2:9: Suite attached to non-function declaration

Thanks,
Jason

From stefan_ml at behnel.de  Thu Feb  5 20:59:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Feb 2009 20:59:27 +0100
Subject: [Cython] Docstrings for 'cdef public' attributes?
In-Reply-To: <498B3731.6090001@canonware.com>
References: <498B3731.6090001@canonware.com>
Message-ID: <498B451F.7060700@behnel.de>

Hi,

Jason Evans wrote:
> Is there any reasonable way to attach docstrings to 'cdef public' 
> attributes?

I'm not aware of any, but I miss that feature, too.


>  The following does not work:
> 
> cdef class Spam:
>      cdef public int spam:
>          """
>              Docstring for spam.
>          """

This is not a suitable syntax, as we might allow default values for
attributes one day, as in

	cdef public int spam = 1

Putting the docstring before the declaration doesn't work with Python's
syntax either.

I'm more in favour of a decorator as a potential future syntax:

	@cython.docstring("The infamous spam number")
	cdef public int spam = 1

Could you file a wishlist bug for this?

Thanks,

Stefan

From dagss at student.matnat.uio.no  Thu Feb  5 22:15:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 05 Feb 2009 22:15:04 +0100
Subject: [Cython] Docstrings for 'cdef public' attributes?
In-Reply-To: <498B451F.7060700@behnel.de>
References: <498B3731.6090001@canonware.com> <498B451F.7060700@behnel.de>
Message-ID: <498B56D8.2020609@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Jason Evans wrote:
>> Is there any reasonable way to attach docstrings to 'cdef public' 
>> attributes?
> 
> I'm not aware of any, but I miss that feature, too.
> 
> 
>>  The following does not work:
>>
>> cdef class Spam:
>>      cdef public int spam:
>>          """
>>              Docstring for spam.
>>          """
> 
> This is not a suitable syntax, as we might allow default values for
> attributes one day, as in
> 
> 	cdef public int spam = 1
> 
> Putting the docstring before the declaration doesn't work with Python's
> syntax either.
> 
> I'm more in favour of a decorator as a potential future syntax:
> 
> 	@cython.docstring("The infamous spam number")
> 	cdef public int spam = 1


How about introducing a Pythonic syntax for the whole construct, a la 
"property"?

spam = cython.cproperty(type=cython.int, init=1, doc="The infamous spam 
number")


-- 
Dag Sverre

From dalcinl at gmail.com  Thu Feb  5 22:29:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 5 Feb 2009 19:29:25 -0200
Subject: [Cython] Need comments about Python integer -> C size_t
Message-ID: <e7ba66e40902051329h64ed699ak884424aaeeec5048@mail.gmail.com>

Would the chain below be enough?:

PyInt/PyLong -> unsigned PY_LONG_LONG -> size_t


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From magnus at hetland.org  Fri Feb  6 12:17:37 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 12:17:37 +0100
Subject: [Cython] Measuring "C-ness"?
Message-ID: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>

I'm writing a system where I build data structures for searching using  
a blend of Cython and Python code. Ideally, though, the search  
operations (performed after the structures have been built) should be  
initiated from Python, but then stay in as pure C as possible (to keep  
the Python/Cython overhead to an absolute minimum).

Now, I could (and probably will) inspect the code to determine this  
"manually", but I was thinking -- perhaps I can automate it somehow?  
At least partially? And I thought I'd ask if you might have any ideas  
in that respect.

In other words, I'd like to profile the code, to see where it goes  
through Cython/the Python C API (or even Python code), so I can  
eliminate those calls. Could I just use a standard profiler here,  
perhaps? (I'm guessing all C stuff will then just be black-box  
references, but that's OK, I guess.) Any other ideas? Some global  
location where I could insert a counter into the generated code? Some  
existing Cython magic?

- M

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Fri Feb  6 12:41:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 6 Feb 2009 12:41:50 +0100 (CET)
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
Message-ID: <46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Magnus Lie Hetland wrote:
> I'd like to profile the code, to see where it goes
> through Cython/the Python C API (or even Python code), so I can
> eliminate those calls.

Have you looked at Cython's annotation (-a) feature? It will generate an
HTML page of your source code that highlights Python operations.

Stefan


From dagss at student.matnat.uio.no  Fri Feb  6 12:58:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 06 Feb 2009 12:58:15 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <498C25D7.1000803@student.matnat.uio.no>

Stefan Behnel wrote:
> Magnus Lie Hetland wrote:
>> I'd like to profile the code, to see where it goes
>> through Cython/the Python C API (or even Python code), so I can
>> eliminate those calls.
> 
> Have you looked at Cython's annotation (-a) feature? It will generate an
> HTML page of your source code that highlights Python operations.

Also, if you declare functions as "nogil" then Python operations will be 
disallowed.

-- 
Dag Sverre

From magnus at hetland.org  Fri Feb  6 12:51:09 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 12:51:09 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <498C25D7.1000803@student.matnat.uio.no>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<498C25D7.1000803@student.matnat.uio.no>
Message-ID: <2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>

On Feb 6, 2009, at 12:58, Dag Sverre Seljebotn wrote:

> Also, if you declare functions as "nogil" then Python operations  
> will be
> disallowed.

I see. To what degree does that involve Python C API stuff for  
converting objects and the like? (E.g., if I -- by accident -- convert  
between char* and str when calling one of my functions/methods from  
another, etc.)

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Fri Feb  6 13:20:45 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 6 Feb 2009 13:20:45 +0100 (CET)
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<498C25D7.1000803@student.matnat.uio.no>
	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>
Message-ID: <64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Magnus Lie Hetland wrote:
> On Feb 6, 2009, at 12:58, Dag Sverre Seljebotn wrote:
>> Also, if you declare functions as "nogil" then Python operations
>> will be disallowed.
>
> I see. To what degree does that involve Python C API stuff for
> converting objects and the like? (E.g., if I -- by accident -- convert
> between char* and str when calling one of my functions/methods from
> another, etc.)

Any Python operation that involves ref-counting is denied.

Stefan


From magnus at hetland.org  Fri Feb  6 13:37:42 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 13:37:42 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<498C25D7.1000803@student.matnat.uio.no>
	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>
	<64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>

On Feb 6, 2009, at 13:20, Stefan Behnel wrote:

> Any Python operation that involves ref-counting is denied.

Great! :) That, along with the -a switch, ought to take me far, I think.

Thanks,

- M

-- 
Magnus Lie Hetland
http://hetland.org



From dagss at student.matnat.uio.no  Fri Feb  6 14:40:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 06 Feb 2009 14:40:09 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<498C25D7.1000803@student.matnat.uio.no>	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>	<64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>
Message-ID: <498C3DB9.3040408@student.matnat.uio.no>

Magnus Lie Hetland wrote:
> On Feb 6, 2009, at 13:20, Stefan Behnel wrote:
> 
>> Any Python operation that involves ref-counting is denied.
> 
> Great! :) That, along with the -a switch, ought to take me far, I think.

If you think of something better while you do this then please tell us. 
A compiler directive for needing explicit "object" declarations, or a 
directive to disable automatic coercions between C types and objects, or 
perhaps a very smart IDE plugin which colors statements of different 
sorts in different colors? Probably easier to think about while you do 
this in practice.

-- 
Dag Sverre

From dalcinl at gmail.com  Fri Feb  6 15:35:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 12:35:19 -0200
Subject: [Cython] please, review this commit
Message-ID: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>

I had to push this patch.

changeset:   1694:5d63d5481ed5
tag:         tip
user:        Lisandro Dalcin <dalcinl at gmail.com>
date:        Fri Feb 06 12:29:32 2009 -0200
summary:     fix type conversion functions involving PY_LONG_LONG for Python 3

If any of you has a better idea, feel free to rollback it...

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From magnus at hetland.org  Fri Feb  6 16:13:16 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Fri, 6 Feb 2009 16:13:16 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <498C3DB9.3040408@student.matnat.uio.no>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>	<46668.213.61.181.86.1233920510.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<498C25D7.1000803@student.matnat.uio.no>	<2CEB197E-BBAF-4F03-AECC-B21532218BE9@hetland.org>	<64989.213.61.181.86.1233922845.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0CFF789F-AA5E-43B2-9981-44E29706346B@hetland.org>
	<498C3DB9.3040408@student.matnat.uio.no>
Message-ID: <C903B26B-BF00-42A0-B70C-B34C81B0766D@hetland.org>

On Feb 6, 2009, at 14:40, Dag Sverre Seljebotn wrote:

> Magnus Lie Hetland wrote:
>> On Feb 6, 2009, at 13:20, Stefan Behnel wrote:
>>
>>> Any Python operation that involves ref-counting is denied.
>>
>> Great! :) That, along with the -a switch, ought to take me far, I  
>> think.
>
> If you think of something better while you do this then please tell  
> us.

[snip]

Can't make any guarantees (of course), but I'll think about it :)

-- 
Magnus Lie Hetland
http://hetland.org



From dalcinl at gmail.com  Fri Feb  6 16:46:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 13:46:37 -0200
Subject: [Cython] negative integer -> unsigned integer,
	TypeError or OverflowError?
Message-ID: <e7ba66e40902060746g205d6114l1312abf691912d17@mail.gmail.com>

Core CPython is not consistent about this. What should we do in
Cython? Should we report this to Python-Dev?

Long story:

In core CPython (svn trunk), at Objects/longobjects.c, you should see
that in almost all cases OverflowError is raise. But see this one:

int
_PyLong_AsByteArray(PyLongObject* v,
		    unsigned char* bytes, size_t n,
		    int little_endian, int is_signed)
{
<...>
		if (!is_signed) {
			PyErr_SetString(PyExc_TypeError,
				"can't convert negative long to unsigned");
			return -1;
		}
<...>
}


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jasone at canonware.com  Fri Feb  6 17:12:48 2009
From: jasone at canonware.com (Jason Evans)
Date: Fri, 06 Feb 2009 08:12:48 -0800
Subject: [Cython] Docstrings for 'cdef public' attributes?
In-Reply-To: <498B451F.7060700@behnel.de>
References: <498B3731.6090001@canonware.com> <498B451F.7060700@behnel.de>
Message-ID: <498C6180.2040600@canonware.com>

Stefan Behnel wrote:
> Jason Evans wrote:
>> Is there any reasonable way to attach docstrings to 'cdef public' 
>> attributes?
> 
> I'm not aware of any, but I miss that feature, too.
> 
> Could you file a wishlist bug for this?

Done: http://trac.cython.org/cython_trac/ticket/206

Thanks,
Jason

From dalcinl at gmail.com  Fri Feb  6 19:21:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 16:21:21 -0200
Subject: [Cython] Bad code generation involving cimport and cdef public
	typed member in cdef classes
Message-ID: <e7ba66e40902061021q61bb9b06q80348f6ef4fbcb09@mail.gmail.com>

Perhaps you noticed warnings like this when building Cython

/u/dalcinl/Devel/cython-devel/Cython/Compiler/Scanning.c:8878:
warning: '__pyx_pf_6Cython_4Plex_8Scanners_7Scanner_5queue___get__'
defined but not used
/u/dalcinl/Devel/cython-devel/Cython/Compiler/Scanning.c:8897:
warning: '__pyx_pf_6Cython_4Plex_8Scanners_7Scanner_5queue___set__'
defined but not used

What happens? Cython.Compiler.Scanning does a 'cimport' or
Cython.Plex.Scanner. The later has a typed 'cdef public list queue'
member. As it is typed (a 'list'), Cython is smart and understand that
assignements needs type-checking, and builds a 'property' with get/set
routines for this. But the get/set routines SHOULD NOT be generated on
Cython/Compiler/Scanning.c.

It seem that the fix should go to Cython/Compiler/Nodes.py, at the
begining of CVarDefNode.analyse_declarations():

        if (dest_scope.is_c_class_scope
                and self.visibility == 'public'
                and base_type.is_pyobject
                and (base_type.is_builtin_type or base_type.is_extension_type)):
            self.need_properties = []
            need_property = True
            visibility = 'private'

The line 'need_property = True' should be 'False' instead  if the
dest_scope is a cimport'ed class.

Any idea about how can I fix this?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb  6 20:00:09 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Feb 2009 20:00:09 +0100
Subject: [Cython] negative integer -> unsigned integer,
	TypeError or OverflowError?
In-Reply-To: <e7ba66e40902060746g205d6114l1312abf691912d17@mail.gmail.com>
References: <e7ba66e40902060746g205d6114l1312abf691912d17@mail.gmail.com>
Message-ID: <498C88B9.4080007@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Core CPython is not consistent about this. What should we do in
> Cython? Should we report this to Python-Dev?
> 
> Long story:
> 
> In core CPython (svn trunk), at Objects/longobjects.c, you should see
> that in almost all cases OverflowError is raise. But see this one:
> 
> int
> _PyLong_AsByteArray(PyLongObject* v,
> 		    unsigned char* bytes, size_t n,
> 		    int little_endian, int is_signed)
> {
> <...>
> 		if (!is_signed) {
> 			PyErr_SetString(PyExc_TypeError,
> 				"can't convert negative long to unsigned");
> 			return -1;
> 		}
> <...>
> }

Raising an OverflowError sounds like the right thing. If CPython raises it
in almost all cases, it's ok if Cython does that, too.

Still, asking on python-dev would allow them to take a decision and maybe
even fix the above discrepancy.

Stefan

From dalcinl at gmail.com  Fri Feb  6 20:48:31 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 17:48:31 -0200
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
Message-ID: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>

What do you think about this?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb  6 20:57:14 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Feb 2009 20:57:14 +0100
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
In-Reply-To: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
References: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
Message-ID: <498C961A.3070403@behnel.de>


Lisandro Dalcin wrote:
> What do you think about this?

Fine with me. None is a very common value and often used in boolean checks like

    if reference_that_may_be_None:
        ...

I bet the C compiler will even be able to figure out that such a variable
is None in some simple cases.

Stefan


From dalcinl at gmail.com  Fri Feb  6 21:51:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 18:51:43 -0200
Subject: [Cython] ready to push changes to make Cython understand 'size_t'
	as a native C type
Message-ID: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>

The only thing that I'm not completely sure about is the
implementation of this, though it works fine in my 32bit Linux box

static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);

Appart, from that, the testsuite seems to run fine for me...


Of course, this change is not backward compatible... Cython will not
compile the code:

cdef extern from "stdlib.h":
    ctypedef unsigned long size_t

However, I've noticed that if I try to ctypedef Py_ssize_t, then
Cython does not even emit a warning !!!

What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
or 'size_t', provided that Cython treats them as pre-defined C types?


Finally, can I push the changes?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb  6 22:11:16 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Feb 2009 22:11:16 +0100
Subject: [Cython] ready to push changes to make Cython understand
 'size_t' as a native C type
In-Reply-To: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
Message-ID: <498CA774.1030506@behnel.de>

Hi Lisandro,

Lisandro Dalcin wrote:
> The only thing that I'm not completely sure about is the
> implementation of this, though it works fine in my 32bit Linux box
> 
> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);

I'd need to see the implementation to comment on this.


> Of course, this change is not backward compatible... Cython will not
> compile the code:
> 
> cdef extern from "stdlib.h":
>     ctypedef unsigned long size_t

Is this because of an explicit error, or just a side effect of the
implementation?


> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
> or 'size_t', provided that Cython treats them as pre-defined C types?

I didn't care at the time when I implemented the Py_ssize_t stuff, but when
you ask now, I prefer getting at least a warning. I doubt that Cython
handles a "ctypedef ... int" gracefully, for example.


> Finally, can I push the changes?

Could you file a bug report for now (if only for documentation purposes)
and post the patch there? We should be a bit careful with major changes by
now, as we are currently trying to stabilise Cython for the 0.11 release.

Thanks,

Stefan

From robertwb at math.washington.edu  Fri Feb  6 22:47:18 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 13:47:18 -0800
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
In-Reply-To: <498C961A.3070403@behnel.de>
References: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
	<498C961A.3070403@behnel.de>
Message-ID: <605AE54D-F7DE-4E47-8B8C-33C88823B504@math.washington.edu>

On Feb 6, 2009, at 11:57 AM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> What do you think about this?
>
> Fine with me. None is a very common value and often used in boolean  
> checks like
>
>     if reference_that_may_be_None:
>         ...
>
> I bet the C compiler will even be able to figure out that such a  
> variable
> is None in some simple cases.

+1 from me too--so it will check for True, False, and None before  
deferring to the generic code then.

- Robert


From robertwb at math.washington.edu  Fri Feb  6 22:49:54 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 13:49:54 -0800
Subject: [Cython] please, review this commit
In-Reply-To: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>
References: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>
Message-ID: <7636BBAB-0C85-4954-AA42-2AF8BB1A5DD1@math.washington.edu>

Looks good to me--do you have any tests to go with it?

On Feb 6, 2009, at 6:35 AM, Lisandro Dalcin wrote:

> I had to push this patch.
>
> changeset:   1694:5d63d5481ed5
> tag:         tip
> user:        Lisandro Dalcin <dalcinl at gmail.com>
> date:        Fri Feb 06 12:29:32 2009 -0200
> summary:     fix type conversion functions involving PY_LONG_LONG  
> for Python 3
>
> If any of you has a better idea, feel free to rollback it...
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Fri Feb  6 22:50:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 19:50:43 -0200
Subject: [Cython] special case Py_None at __Pyx_PyObject_IsTrue()
In-Reply-To: <605AE54D-F7DE-4E47-8B8C-33C88823B504@math.washington.edu>
References: <e7ba66e40902061148i7057b45fg7372d19a559c5e4d@mail.gmail.com>
	<498C961A.3070403@behnel.de>
	<605AE54D-F7DE-4E47-8B8C-33C88823B504@math.washington.edu>
Message-ID: <e7ba66e40902061350m6e665148l11f9f75bd5dfcc80@mail.gmail.com>

Pushed...

On Fri, Feb 6, 2009 at 7:47 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 6, 2009, at 11:57 AM, Stefan Behnel wrote:
>
>> Lisandro Dalcin wrote:
>>> What do you think about this?
>>
>> Fine with me. None is a very common value and often used in boolean
>> checks like
>>
>>     if reference_that_may_be_None:
>>         ...
>>
>> I bet the C compiler will even be able to figure out that such a
>> variable
>> is None in some simple cases.
>
> +1 from me too--so it will check for True, False, and None before
> deferring to the generic code then.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Feb  6 22:52:30 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 19:52:30 -0200
Subject: [Cython] please, review this commit
In-Reply-To: <7636BBAB-0C85-4954-AA42-2AF8BB1A5DD1@math.washington.edu>
References: <e7ba66e40902060635s6c8303dfrfb8e0700e85fbeed@mail.gmail.com>
	<7636BBAB-0C85-4954-AA42-2AF8BB1A5DD1@math.washington.edu>
Message-ID: <e7ba66e40902061352jd90693brd073cb6f467ab209@mail.gmail.com>

No yet.... Once I get the 'size_t' patch in, I'll try to write a good
testcase to check Py <-> C integral conversions...

On Fri, Feb 6, 2009 at 7:49 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Looks good to me--do you have any tests to go with it?
>
> On Feb 6, 2009, at 6:35 AM, Lisandro Dalcin wrote:
>
>> I had to push this patch.
>>
>> changeset:   1694:5d63d5481ed5
>> tag:         tip
>> user:        Lisandro Dalcin <dalcinl at gmail.com>
>> date:        Fri Feb 06 12:29:32 2009 -0200
>> summary:     fix type conversion functions involving PY_LONG_LONG
>> for Python 3
>>
>> If any of you has a better idea, feel free to rollback it...
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Feb  6 22:59:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 13:59:03 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <4988A2A7.8060204@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
Message-ID: <20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>

On Feb 3, 2009, at 12:01 PM, Michael Abshoff wrote:

> Lisandro Dalcin wrote:
>> Cython treats sizeof() as if it returns 'int', but IMHO it should  
>> be 'size_t'.
>
> Well, you might be technically correct since that is what the C  
> standard
> asks for, but according to various literature, i.e. for example
>
> http://publications.gbdirect.co.uk/c_book/chapter5/ 
> sizeof_and_malloc.html
>
> "The sizeof operator returns the size in bytes of its operand. Whether
> the result of sizeof is unsigned int or unsigned long is  
> implementation
> defined?which is why the declaration of malloc above ducked the  
> issue by
> omitting any parameter information; "

I think "int" is the best thing to return--it can always get coerced  
up to a size_t if needed, but the other way may prove problematic.  
Also, "int" is a more abstract (conceptually at least) type (better  
fitting with the ambiguity in the C standard), and we shouldn't have  
to worry about overflow in this case.

>> Implementing this the right way would require to make Cython  
>> recognize
>> 'size_t' as a native type, more or less in the way it already
>> recognizes 'Py_ssize_t'. What do you think about this?
>
> Mmmh, do you want to define size_t in Cython or pull it in from
> somewhere? At least for C code that ought to be rather tricky for VS6
> for example which needs to work since we are still supporting  
> Python 2.3
> on Windows. But you could work around the issue by defining size_t in
> Cython as either unsigned int, unsigned long or unsigned long long
> depending on the platform. But that could potentially clash with  
> various
> C++ runtime libs.

I am -1 on defining size_t ourselves, but would like to pull it from  
somewhere if we could.

> Do you think that people will see overflow issues? I think the  
> tradeoff
> of following the C standard here is rather high since in reality  
> size_t
> in C is rather badly handled. And if you guys decide to follow the C
> standard here someone ought to break out Python 2.6 and VS 6 and  
> verify
> that things are still working :)

For sure.

- Robert


From dalcinl at gmail.com  Fri Feb  6 23:06:31 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 20:06:31 -0200
Subject: [Cython] ready to push changes to make Cython understand
	'size_t' as a native C type
In-Reply-To: <498CA774.1030506@behnel.de>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
	<498CA774.1030506@behnel.de>
Message-ID: <e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>

Opened http://trac.cython.org/cython_trac/ticket/207

On Fri, Feb 6, 2009 at 7:11 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi Lisandro,
>
> Lisandro Dalcin wrote:
>> The only thing that I'm not completely sure about is the
>> implementation of this, though it works fine in my 32bit Linux box
>>
>> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);
>
> I'd need to see the implementation to comment on this.
>

http://trac.cython.org/cython_trac/attachment/ticket/207/SIZE_T.diff#preview

>
>> Of course, this change is not backward compatible... Cython will not
>> compile the code:
>>
>> cdef extern from "stdlib.h":
>>     ctypedef unsigned long size_t
>
> Is this because of an explicit error, or just a side effect of the
> implementation?
>

Cython generates the following error (IMHO, that's a good thing)

Error converting Pyrex file to C:
------------------------------------------------------------
...
#cython: embedsignature=True
ctypedef unsigned long size_t
                            ^
------------------------------------------------------------

/u/dalcinl/Devel/mpi4py-dev/src/mpi4py.MPI.pyx:2:29: Empty declarator


>
>> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
>> or 'size_t', provided that Cython treats them as pre-defined C types?
>
> I didn't care at the time when I implemented the Py_ssize_t stuff, but when
> you ask now, I prefer getting at least a warning. I doubt that Cython
> handles a "ctypedef ... int" gracefully, for example.
>

Sorry for the noise, I got confused.. Cython DO FAIL if you 'ctypedef
... Py_ssize_t'.

>
>> Finally, can I push the changes?
>
> Could you file a bug report for now (if only for documentation purposes)
> and post the patch there? We should be a bit careful with major changes by
> now, as we are currently trying to stabilise Cython for the 0.11 release.
>

OK. Just hope this can get in for the next release.

> Thanks,
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lists at cheimes.de  Fri Feb  6 23:38:19 2009
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 06 Feb 2009 23:38:19 +0100
Subject: [Cython] How access and modify errno?
Message-ID: <gmie4r$vnr$1@ger.gmane.org>

Hi fellow Cython users!

I need to set errno = 0 and later check it for != 0. How can I stop
Cython from mangling the name of my errno variable?

Example code:

cdef extern from "errno.h":
    int errno

def method():
    errno = 0

The Cython compiler insists on creating a new local variable "int
__pyx_v_errno" for errno. Is there a way to stop Cython from doing that
or do I have to use an extra header file with some macro magic?

Christian


From carl.witty at gmail.com  Fri Feb  6 23:42:51 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Fri, 6 Feb 2009 14:42:51 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
Message-ID: <ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>

On Fri, Feb 6, 2009 at 1:59 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 3, 2009, at 12:01 PM, Michael Abshoff wrote:
>
>> Lisandro Dalcin wrote:
>>> Cython treats sizeof() as if it returns 'int', but IMHO it should
>>> be 'size_t'.
>>
>> Well, you might be technically correct since that is what the C
>> standard
>> asks for, but according to various literature, i.e. for example
>>
>> http://publications.gbdirect.co.uk/c_book/chapter5/
>> sizeof_and_malloc.html
>>
>> "The sizeof operator returns the size in bytes of its operand. Whether
>> the result of sizeof is unsigned int or unsigned long is
>> implementation
>> defined?which is why the declaration of malloc above ducked the
>> issue by
>> omitting any parameter information; "
>
> I think "int" is the best thing to return--it can always get coerced
> up to a size_t if needed, but the other way may prove problematic.
> Also, "int" is a more abstract (conceptually at least) type (better
> fitting with the ambiguity in the C standard), and we shouldn't have
> to worry about overflow in this case.

Certainly the result of sizeof can overflow an int.  (Presumably on
64-bit Windows, it can overflow unsigned long, as well.)

Or are you saying that we don't have to worry about it because people
won't define such large types?  That may be true in practice, but it
makes me uncomfortable.

Carl

From robertwb at math.washington.edu  Fri Feb  6 23:50:44 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 14:50:44 -0800
Subject: [Cython] How access and modify errno?
In-Reply-To: <gmie4r$vnr$1@ger.gmane.org>
References: <gmie4r$vnr$1@ger.gmane.org>
Message-ID: <776B5273-748A-46B1-B49A-6140FEA4F635@math.washington.edu>

On Feb 6, 2009, at 2:38 PM, Christian Heimes wrote:

> Hi fellow Cython users!
>
> I need to set errno = 0 and later check it for != 0. How can I stop
> Cython from mangling the name of my errno variable?
>
> Example code:
>
> cdef extern from "errno.h":
>     int errno
>
> def method():

global errno

>     errno = 0
>
> The Cython compiler insists on creating a new local variable "int
> __pyx_v_errno" for errno. Is there a way to stop Cython from doing  
> that
> or do I have to use an extra header file with some macro magic?

No. Cython scoping is just like Python scoping--an assignment creates  
a local in the given scope.

- Robert


From michael.abshoff at googlemail.com  Fri Feb  6 23:52:48 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 06 Feb 2009 14:52:48 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
	<ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>
Message-ID: <498CBF40.8080204@gmail.com>

Carl Witty wrote:
> On Fri, Feb 6, 2009 at 1:59 PM, Robert Bradshaw

<SNIP>

>> I think "int" is the best thing to return--it can always get coerced
>> up to a size_t if needed, but the other way may prove problematic.
>> Also, "int" is a more abstract (conceptually at least) type (better
>> fitting with the ambiguity in the C standard), and we shouldn't have
>> to worry about overflow in this case.
> 
> Certainly the result of sizeof can overflow an int.  (Presumably on
> 64-bit Windows, it can overflow unsigned long, as well.)
> 
> Or are you saying that we don't have to worry about it because people
> won't define such large types?  That may be true in practice, but it
> makes me uncomfortable.

+1

There *is* actually code in the Sage library that has overflow potential 
on 64 bit Windows and it needs to be fixed. Debugging some strange 
segfault caused by the above is hard enough at it is, adding platform 
specific issues doesn't make it any better.

The C and C++ people did define size_t for a reason since all those 
assumptions about int or long being "big enough" does not hold true in 
the real world, so let's do this by the book.

> Carl

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb  6 23:56:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Feb 2009 14:56:07 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <498CBF40.8080204@gmail.com>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
	<ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>
	<498CBF40.8080204@gmail.com>
Message-ID: <C329E07F-C839-495D-AEB6-B89173D7A4EE@math.washington.edu>

On Feb 6, 2009, at 2:52 PM, Michael Abshoff wrote:

> Carl Witty wrote:
>> On Fri, Feb 6, 2009 at 1:59 PM, Robert Bradshaw
>
> <SNIP>
>
>>> I think "int" is the best thing to return--it can always get coerced
>>> up to a size_t if needed, but the other way may prove problematic.
>>> Also, "int" is a more abstract (conceptually at least) type (better
>>> fitting with the ambiguity in the C standard), and we shouldn't have
>>> to worry about overflow in this case.
>>
>> Certainly the result of sizeof can overflow an int.  (Presumably on
>> 64-bit Windows, it can overflow unsigned long, as well.)
>>
>> Or are you saying that we don't have to worry about it because people
>> won't define such large types?

That is what I was implying. But it isn't valid in retrospect because  
one might do, say, malloc(sizeof(T) * 4000000000)

>> That may be true in practice, but it
>> makes me uncomfortable.
>
> +1
>
> There *is* actually code in the Sage library that has overflow  
> potential
> on 64 bit Windows and it needs to be fixed. Debugging some strange
> segfault caused by the above is hard enough at it is, adding platform
> specific issues doesn't make it any better.
>
> The C and C++ people did define size_t for a reason since all those
> assumptions about int or long being "big enough" does not hold true in
> the real world, so let's do this by the book.

I hadn't yet read the rest of this thread where it looks like  
Python.h assumes (or defines?) size_t. Given that's the case, I'm  
very much in favor of making it native to Cython and the sizeof  
return type.

- Robert



From dalcinl at gmail.com  Fri Feb  6 23:57:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Feb 2009 20:57:27 -0200
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>
	<4988A2A7.8060204@gmail.com>
	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>
Message-ID: <e7ba66e40902061457v1abd84eegfa6debb1a7683497@mail.gmail.com>

On Fri, Feb 6, 2009 at 7:59 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 3, 2009, at 12:01 PM, Michael Abshoff wrote:
>
> I think "int" is the best thing to return--it can always get coerced
> up to a size_t if needed, but the other way may prove problematic.
> Also, "int" is a more abstract (conceptually at least) type (better
> fitting with the ambiguity in the C standard), and we shouldn't have
> to worry about overflow in this case.
>

I have to strongly disagree with you.

Try the output of this Cython code

print sizeof(char[1<<30]) * 3

versus the output of this C code:

#include "stdio.h"
int main() {
  printf("%u\n", sizeof(char[1<<30])*3);
}


For the first you get -1073741824 (clearly wrong), and for the second
3221225472. (Used a 32 bit box)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Sat Feb  7 00:03:27 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 06 Feb 2009 15:03:27 -0800
Subject: [Cython] a problem with sizeof() return type
In-Reply-To: <C329E07F-C839-495D-AEB6-B89173D7A4EE@math.washington.edu>
References: <e7ba66e40902031140h65e1f6b9g13599159c1ab4c11@mail.gmail.com>	<4988A2A7.8060204@gmail.com>	<20521CE9-B741-42E8-B950-B74EA71BAE5B@math.washington.edu>	<ac1e4c420902061442v193487b6x7c427e296d40e624@mail.gmail.com>	<498CBF40.8080204@gmail.com>
	<C329E07F-C839-495D-AEB6-B89173D7A4EE@math.washington.edu>
Message-ID: <498CC1BF.1010406@gmail.com>

Robert Bradshaw wrote:
> On Feb 6, 2009, at 2:52 PM, Michael Abshoff wrote:

<SNIP>

> I hadn't yet read the rest of this thread where it looks like  
> Python.h assumes (or defines?) size_t. Given that's the case, I'm  
> very much in favor of making it native to Cython and the sizeof  
> return type.

No problem, you must be dealing with a rather large amount of email :)

Anyway, welcome back.

> - Robert

Cheers,

Michael


> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From lists at cheimes.de  Sat Feb  7 00:34:32 2009
From: lists at cheimes.de (Christian Heimes)
Date: Sat, 07 Feb 2009 00:34:32 +0100
Subject: [Cython] How access and modify errno?
In-Reply-To: <776B5273-748A-46B1-B49A-6140FEA4F635@math.washington.edu>
References: <gmie4r$vnr$1@ger.gmane.org>
	<776B5273-748A-46B1-B49A-6140FEA4F635@math.washington.edu>
Message-ID: <gmihe8$8uf$1@ger.gmane.org>

Robert Bradshaw wrote:
> global errno
> 
>>     errno = 0
>>
>> The Cython compiler insists on creating a new local variable "int
>> __pyx_v_errno" for errno. Is there a way to stop Cython from doing  
>> that
>> or do I have to use an extra header file with some macro magic?
> 
> No. Cython scoping is just like Python scoping--an assignment creates  
> a local in the given scope.

Doh! That was too easy ... :(
Why didn't I think of global in the first place?

Thanks Robert!

Christian


From greg.ewing at canterbury.ac.nz  Sat Feb  7 09:24:39 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 07 Feb 2009 21:24:39 +1300
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
Message-ID: <498D4547.8060308@canterbury.ac.nz>

Magnus Lie Hetland wrote:
> Ideally, though, the search  
> operations (performed after the structures have been built) should be  
> initiated from Python, but then stay in as pure C as possible

One way to guarantee that something uses purely C
operations is to actually write it in C, as an
external piece of code, and call it from Cython.

-- 
Greg

From magnus at hetland.org  Sat Feb  7 12:44:55 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sat, 7 Feb 2009 12:44:55 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <498D4547.8060308@canterbury.ac.nz>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
Message-ID: <5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>

On Feb 7, 2009, at 09:24 , Greg Ewing wrote:

> Magnus Lie Hetland wrote:
>> Ideally, though, the search
>> operations (performed after the structures have been built) should be
>> initiated from Python, but then stay in as pure C as possible
>
> One way to guarantee that something uses purely C
> operations is to actually write it in C, as an
> external piece of code, and call it from Cython.


Absolutely an option. It may even be the most sensible thing to do,  
rather than trying to program in C using Cython :-)

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Sat Feb  7 14:08:05 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 07 Feb 2009 14:08:05 +0100
Subject: [Cython] ready to push changes to make Cython understand
 'size_t' as a native C type
In-Reply-To: <e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>	<498CA774.1030506@behnel.de>
	<e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>
Message-ID: <498D87B5.3040208@behnel.de>


Lisandro Dalcin wrote:
>>> The only thing that I'm not completely sure about is the
>>> implementation of this, though it works fine in my 32bit Linux box
>>>
>>> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);
> 
> http://trac.cython.org/cython_trac/attachment/ticket/207/SIZE_T.diff#preview

	static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b) {
 		 unsigned PY_LONG_LONG val =
				 __pyx_PyInt_AsUnsignedLongLong(b);
 		 if (unlikely(val == (unsigned PY_LONG_LONG)-1 &&
				 PyErr_Occurred())) {
 		       return (size_t)-1;

The "(size_t)-1" definitely looks funny. I noticed that "(unsigned
PY_LONG_LONG)-1" is also used in __pyx_PyInt_AsUnsignedLongLong(), so I
just hope it does the right thing.


 		  } else if (unlikely((unsigned PY_LONG_LONG)(size_t)val
							 != val)) {
 		       PyErr_SetString(PyExc_OverflowError, "value too
					 large to convert to size_t");

This should return an error value, so it lacks a "return (size_t)-1".

 		  }
 		  return val;
	}

It would be great if we had tests for the error cases. It's not obvious
that they work as expected. The same applies to the "unsigned long long"
conversion, IMHO.


>>> Of course, this change is not backward compatible... Cython will not
>>> compile the code:
>>>
>>> cdef extern from "stdlib.h":
>>>     ctypedef unsigned long size_t
>> Is this because of an explicit error, or just a side effect of the
>> implementation?
>>
> 
> Cython generates the following error (IMHO, that's a good thing)
> 
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> #cython: embedsignature=True
> ctypedef unsigned long size_t
>                             ^
> ------------------------------------------------------------
> 
> /u/dalcinl/Devel/mpi4py-dev/src/mpi4py.MPI.pyx:2:29: Empty declarator

Not really the most telling error message, but not cryptic either. Fine
with me.


>>> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
>>> or 'size_t', provided that Cython treats them as pre-defined C types?
>> I didn't care at the time when I implemented the Py_ssize_t stuff, but when
>> you ask now, I prefer getting at least a warning. I doubt that Cython
>> handles a "ctypedef ... int" gracefully, for example.
> 
> Sorry for the noise, I got confused.. Cython DO FAIL if you 'ctypedef
> ... Py_ssize_t'.

Same as above, I assume: it thinks that you are still collecting types and
modifiers, so it's missing the new type name at the end.


>>> Finally, can I push the changes?
>> Could you file a bug report for now (if only for documentation purposes)
>> and post the patch there? We should be a bit careful with major changes by
>> now, as we are currently trying to stabilise Cython for the 0.11 release.
>>
> 
> OK. Just hope this can get in for the next release.

Apart from what I said above, it looks ok, so I'd like to see it in as well.

Stefan


From dalcinl at gmail.com  Sat Feb  7 18:39:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 14:39:44 -0300
Subject: [Cython] ready to push changes to make Cython understand
	'size_t' as a native C type
In-Reply-To: <498D87B5.3040208@behnel.de>
References: <e7ba66e40902061251l390f0ec4g667293b94effe036@mail.gmail.com>
	<498CA774.1030506@behnel.de>
	<e7ba66e40902061406o6ae1a6c8l889eb9cf21c65624@mail.gmail.com>
	<498D87B5.3040208@behnel.de>
Message-ID: <e7ba66e40902070939k48194864q9eb5a4719f303304@mail.gmail.com>

On Sat, Feb 7, 2009 at 10:08 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>>>> The only thing that I'm not completely sure about is the
>>>> implementation of this, though it works fine in my 32bit Linux box
>>>>
>>>> static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b);
>>
>> http://trac.cython.org/cython_trac/attachment/ticket/207/SIZE_T.diff#preview
>
>        static INLINE size_t __pyx_PyInt_AsSize_t(PyObject* b) {
>                 unsigned PY_LONG_LONG val =
>                                 __pyx_PyInt_AsUnsignedLongLong(b);
>                 if (unlikely(val == (unsigned PY_LONG_LONG)-1 &&
>                                 PyErr_Occurred())) {
>                       return (size_t)-1;
>
> The "(size_t)-1" definitely looks funny. I noticed that "(unsigned
> PY_LONG_LONG)-1" is also used in __pyx_PyInt_AsUnsignedLongLong(), so I
> just hope it does the right thing.
>

That's the way things are implemented in core CPython, and AFAIK it
should works, at least if Cython-generated C code is compiled with the
same C compiler as core CPython library. I do not know I the C
standard is explicit about what you get when you cast a negative
integer to an unsigned integer.

>
>                  } else if (unlikely((unsigned PY_LONG_LONG)(size_t)val
>                                                         != val)) {
>                       PyErr_SetString(PyExc_OverflowError, "value too
>                                         large to convert to size_t");
>
> This should return an error value, so it lacks a "return (size_t)-1".
>

OK, I'll fix that ...

>
> It would be great if we had tests for the error cases. It's not obvious
> that they work as expected. The same applies to the "unsigned long long"
> conversion, IMHO.
>

I've added a testcase for overflowing 'size_t' conversions. But yes,
such testcases should be implemented for all integral types... I'll
give a try next week in some spare time.


>
>>>> Of course, this change is not backward compatible... Cython will not
>>>> compile the code:
>>>>
>>>> cdef extern from "stdlib.h":
>>>>     ctypedef unsigned long size_t
>>> Is this because of an explicit error, or just a side effect of the
>>> implementation?
>>>
>>
>> Cython generates the following error (IMHO, that's a good thing)
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>> #cython: embedsignature=True
>> ctypedef unsigned long size_t
>>                             ^
>> ------------------------------------------------------------
>>
>> /u/dalcinl/Devel/mpi4py-dev/src/mpi4py.MPI.pyx:2:29: Empty declarator
>
> Not really the most telling error message, but not cryptic either. Fine
> with me.
>
>
>>>> What should be the behavior if the user tries to ctypedef 'Py_ssize_t'
>>>> or 'size_t', provided that Cython treats them as pre-defined C types?
>>> I didn't care at the time when I implemented the Py_ssize_t stuff, but when
>>> you ask now, I prefer getting at least a warning. I doubt that Cython
>>> handles a "ctypedef ... int" gracefully, for example.
>>
>> Sorry for the noise, I got confused.. Cython DO FAIL if you 'ctypedef
>> ... Py_ssize_t'.
>
> Same as above, I assume: it thinks that you are still collecting types and
> modifiers, so it's missing the new type name at the end.
>
>
>>>> Finally, can I push the changes?
>>> Could you file a bug report for now (if only for documentation purposes)
>>> and post the patch there? We should be a bit careful with major changes by
>>> now, as we are currently trying to stabilise Cython for the 0.11 release.
>>>
>>
>> OK. Just hope this can get in for the next release.
>
> Apart from what I said above, it looks ok, so I'd like to see it in as well.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sat Feb  7 20:10:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 16:10:01 -0300
Subject: [Cython] problems with C numeric types
Message-ID: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>

After pushing the changes for the 'size_t' stuff, now I believe that
things were a bit broken before the patch, and they still are.

+ Go Cython.Compiler.PyrexTypes

+ Look at the module-level tuple variable 'rank_to_type_name'. You can
see that 'Py_ssize_t' and 'size_t' are AFTER the type 'PY_LONG_LONG'.

+ Now look at the implementation of the function 'widest_numeric_type()'.

Then, this bit of Cython::

cdef f():
    cdef unsigned long long a = 5
    cdef size_t b = 7
    return a + b

generates C code like this::

  /* "/u/dalcinl/Devel/cython-devel/tmp.pyx":5
 *     cdef unsigned long long a = 5
 *     cdef size_t b = 7
 *     return a + b             # <<<<<<<<<<<<<<
 */
  /* __pyx_t_1 allocated */
  __pyx_t_1 = __pyx_PyInt_FromSize_t((__pyx_v_a + __pyx_v_b)); if
(unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 5;
__pyx_clineno = __LINE__; goto __pyx_L1_error;}



This is IMHO totally wrong !!


Could we assume that PY_LONG_LONG is the widest integral?? Should we
assume that 'Py_ssize_t' and 'size_t' are between 'long' and
'PY_LONG_LONG'? In such case, I'll have to fix the 'rank_to_type_name'
tuple, and perhaps the 'sign_and_rank_to_type' dictionary (used as a
fallback in function 'widest_numeric_type()')

However, I'm not sure how to we will handle code like the one below
with the current implementation,

cdef f():
    cdef Py_ssize_t a = 5
    cdef size_t b = 7
    return a + b

I believe the value of 'a+b' should be promoted to 'signed long long',
but it cannot see the way to implent such behavior in the current
Cython codebase.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Feb  7 22:13:26 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 07 Feb 2009 22:13:26 +0100
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
Message-ID: <498DF976.2050403@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Could we assume that PY_LONG_LONG is the widest integral?? Should we
> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
> 'PY_LONG_LONG'?

For starters, Py_ssize_t is defined as a signed size_t according to PEP
353. However, it does not mention a relation to PY_LONG_LONG.

I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the latter being
the largest available type), and my (limited) knowledge about the "long"
type makes me assume that it should be <= Py_ssize_t. So, yes, I would expect

	long <= Py_ssize_t <= PY_LONG_LONG

and

	unsigned long <= size_t <= unsigned PY_LONG_LONG


> However, I'm not sure how to we will handle code like the one below
> with the current implementation,
> 
> cdef f():
>     cdef Py_ssize_t a = 5
>     cdef size_t b = 7
>     return a + b
> 
> I believe the value of 'a+b' should be promoted to 'signed long long',
> but it cannot see the way to implent such behavior in the current
> Cython codebase.

Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's << size_t
for non-negative types, but the missing bit shouldn't matter in almost all
cases. And if users really need to make use of it, I'm fine with requiring
them to write

	return <PY_LONG_LONG>a + <PY_LONG_LONG>b

Stefan

From robertwb at math.washington.edu  Sun Feb  8 00:02:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:02:36 -0800
Subject: [Cython] problems with C numeric types
In-Reply-To: <498DF976.2050403@behnel.de>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
Message-ID: <957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>

On Feb 7, 2009, at 1:13 PM, Stefan Behnel wrote:

> Hi,
>
> Lisandro Dalcin wrote:
>> Could we assume that PY_LONG_LONG is the widest integral?? Should we
>> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
>> 'PY_LONG_LONG'?
>
> For starters, Py_ssize_t is defined as a signed size_t according to  
> PEP
> 353. However, it does not mention a relation to PY_LONG_LONG.
>
> I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the  
> latter being
> the largest available type), and my (limited) knowledge about the  
> "long"
> type makes me assume that it should be <= Py_ssize_t. So, yes, I  
> would expect
>
> 	long <= Py_ssize_t <= PY_LONG_LONG
>
> and
>
> 	unsigned long <= size_t <= unsigned PY_LONG_LONG

I think this is the right ordering for us to make. As we want to  
produce C files often destined to be compiled on other architectures,  
we can't actually test to see the order.

>> However, I'm not sure how to we will handle code like the one below
>> with the current implementation,
>>
>> cdef f():
>>     cdef Py_ssize_t a = 5
>>     cdef size_t b = 7
>>     return a + b
>>
>> I believe the value of 'a+b' should be promoted to 'signed long  
>> long',
>> but it cannot see the way to implent such behavior in the current
>> Cython codebase.
>
> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<  
> size_t
> for non-negative types, but the missing bit shouldn't matter in  
> almost all
> cases. And if users really need to make use of it, I'm fine with  
> requiring
> them to write
>
> 	return <PY_LONG_LONG>a + <PY_LONG_LONG>b

Actually, the target should be size_t, following the C standard that  
operations between signed and unsigned have an unsigned result. But  
we certainly shouldn't be promoting it to a new type.

- Robert


From robertwb at math.washington.edu  Sun Feb  8 00:09:09 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:09:09 -0800
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
Message-ID: <0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>

On Feb 7, 2009, at 3:44 AM, Magnus Lie Hetland wrote:

> On Feb 7, 2009, at 09:24 , Greg Ewing wrote:
>
>> Magnus Lie Hetland wrote:
>>> Ideally, though, the search
>>> operations (performed after the structures have been built)  
>>> should be
>>> initiated from Python, but then stay in as pure C as possible
>>
>> One way to guarantee that something uses purely C
>> operations is to actually write it in C, as an
>> external piece of code, and call it from Cython.
>
> Absolutely an option. It may even be the most sensible thing to do,
> rather than trying to program in C using Cython :-)

Of course this requires one to keep two files in sync, make "cdef  
extern" declarations, and have a more complicated setup.py (i.e. no  
using pyximport), and risk gcc as well as Cython errors. I "program C  
using Cython" all the time--using the annotate option it is easy to  
see when a block of code gets compiled to pure C. But it's a matter  
of personal preference.

- Robert


From dalcinl at gmail.com  Sun Feb  8 00:09:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:09:01 -0300
Subject: [Cython] problems with C numeric types
In-Reply-To: <498DF976.2050403@behnel.de>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
Message-ID: <e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>

On Sat, Feb 7, 2009 at 6:13 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Lisandro Dalcin wrote:
>> Could we assume that PY_LONG_LONG is the widest integral?? Should we
>> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
>> 'PY_LONG_LONG'?
>
> For starters, Py_ssize_t is defined as a signed size_t according to PEP
> 353. However, it does not mention a relation to PY_LONG_LONG.
>
> I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the latter being
> the largest available type), and my (limited) knowledge about the "long"
> type makes me assume that it should be <= Py_ssize_t. So, yes, I would expect
>
>        long <= Py_ssize_t <= PY_LONG_LONG
>
> and
>
>        unsigned long <= size_t <= unsigned PY_LONG_LONG
>

I think the same... Then I'll review this and push fixes.

>
>> However, I'm not sure how to we will handle code like the one below
>> with the current implementation,
>>
>> cdef f():
>>     cdef Py_ssize_t a = 5
>>     cdef size_t b = 7
>>     return a + b
>>
>> I believe the value of 'a+b' should be promoted to 'signed long long',
>> but it cannot see the way to implent such behavior in the current
>> Cython codebase.
>

Well, I was partially wrong in the above comments...

> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's << size_t
> for non-negative types, but the missing bit shouldn't matter in almost all
> cases.

It will not matter until you get a segfault or strange results... ;-)


> And if users really need to make use of it, I'm fine with requiring
> them to write
>
>        return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>

Yes, you are right, but then better for users to not forget the casts...


In short, If we can do it better, go for it and push testcases+fixes...




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sun Feb  8 00:16:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:16:27 -0800
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>
Message-ID: <142903B3-CEDA-4916-BF9E-842CFF6C57E1@math.washington.edu>

On Feb 7, 2009, at 3:09 PM, Lisandro Dalcin wrote:

> On Sat, Feb 7, 2009 at 6:13 PM, Stefan Behnel <stefan_ml at behnel.de>  
> wrote:
>> Hi,
>>
>> Lisandro Dalcin wrote:
>>> Could we assume that PY_LONG_LONG is the widest integral?? Should we
>>> assume that 'Py_ssize_t' and 'size_t' are between 'long' and
>>> 'PY_LONG_LONG'?
>>
>> For starters, Py_ssize_t is defined as a signed size_t according  
>> to PEP
>> 353. However, it does not mention a relation to PY_LONG_LONG.
>>
>> I would at least expect Py_ssize_t to be <= PY_LONG_LONG (the  
>> latter being
>> the largest available type), and my (limited) knowledge about the  
>> "long"
>> type makes me assume that it should be <= Py_ssize_t. So, yes, I  
>> would expect
>>
>>        long <= Py_ssize_t <= PY_LONG_LONG
>>
>> and
>>
>>        unsigned long <= size_t <= unsigned PY_LONG_LONG
>>
>
> I think the same... Then I'll review this and push fixes.

Excellent.

>
>>
>>> However, I'm not sure how to we will handle code like the one below
>>> with the current implementation,
>>>
>>> cdef f():
>>>     cdef Py_ssize_t a = 5
>>>     cdef size_t b = 7
>>>     return a + b
>>>
>>> I believe the value of 'a+b' should be promoted to 'signed long  
>>> long',
>>> but it cannot see the way to implent such behavior in the current
>>> Cython codebase.
>>
>
> Well, I was partially wrong in the above comments...
>
>> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<  
>> size_t
>> for non-negative types, but the missing bit shouldn't matter in  
>> almost all
>> cases.
>
> It will not matter until you get a segfault or strange results... ;-)
>
>
>> And if users really need to make use of it, I'm fine with requiring
>> them to write
>>
>>        return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>>
>
> Yes, you are right, but then better for users to not forget the  
> casts...
>
> In short, If we can do it better, go for it and push testcases 
> +fixes...


This is like asking if someone does

cdef unsigned int a
cdef int b
return a + b

I don't think a + b should be "upgraded" to a long (which may not  
even be bigger), just because it might overflow.

- Robert


From dalcinl at gmail.com  Sun Feb  8 00:17:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:17:25 -0300
Subject: [Cython] problems with C numeric types
In-Reply-To: <957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>
Message-ID: <e7ba66e40902071517i6b7b88er97ebc79e473781aa@mail.gmail.com>

On Sat, Feb 7, 2009 at 8:02 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 7, 2009, at 1:13 PM, Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>>
>>> cdef f():
>>>     cdef Py_ssize_t a = 5
>>>     cdef size_t b = 7
>>>     return a + b
>>>
>>> I believe the value of 'a+b' should be promoted to 'signed long
>>> long',
>>
>> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<
>> size_t
>> for non-negative types, but the missing bit shouldn't matter in
>> almost all
>> cases. And if users really need to make use of it, I'm fine with
>> requiring
>> them to write
>>
>>       return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>
> Actually, the target should be size_t, following the C standard that
> operations between signed and unsigned have an unsigned result. But
> we certainly shouldn't be promoting it to a new type.
>

Yes, you are right, but then this testcase is expected to fail...
Should we do better?

__doc__ = """
>>> f()
-1
"""
def f():
    cdef Py_ssize_t a = -2
    cdef size_t b = 1
    return a + b


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sun Feb  8 00:18:42 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:18:42 -0300
Subject: [Cython] problems with C numeric types
In-Reply-To: <142903B3-CEDA-4916-BF9E-842CFF6C57E1@math.washington.edu>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<e7ba66e40902071509m5a5e457ex9a03e7aa441b28bc@mail.gmail.com>
	<142903B3-CEDA-4916-BF9E-842CFF6C57E1@math.washington.edu>
Message-ID: <e7ba66e40902071518w64c1879od634eb105a41c6e8@mail.gmail.com>

On Sat, Feb 7, 2009 at 8:16 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> This is like asking if someone does
>
> cdef unsigned int a
> cdef int b
> return a + b
>
> I don't think a + b should be "upgraded" to a long (which may not
> even be bigger), just because it might overflow.
>

OK.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sun Feb  8 00:21:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 20:21:17 -0300
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
Message-ID: <e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>

BTW,  a feature I really miss from the times I was a heavy SWIG user
is the ability to include verbatim C code snipets...


On Sat, Feb 7, 2009 at 8:09 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 7, 2009, at 3:44 AM, Magnus Lie Hetland wrote:
>
>> On Feb 7, 2009, at 09:24 , Greg Ewing wrote:
>>
>>> Magnus Lie Hetland wrote:
>>>> Ideally, though, the search
>>>> operations (performed after the structures have been built)
>>>> should be
>>>> initiated from Python, but then stay in as pure C as possible
>>>
>>> One way to guarantee that something uses purely C
>>> operations is to actually write it in C, as an
>>> external piece of code, and call it from Cython.
>>
>> Absolutely an option. It may even be the most sensible thing to do,
>> rather than trying to program in C using Cython :-)
>
> Of course this requires one to keep two files in sync, make "cdef
> extern" declarations, and have a more complicated setup.py (i.e. no
> using pyximport), and risk gcc as well as Cython errors. I "program C
> using Cython" all the time--using the annotate option it is easy to
> see when a block of code gets compiled to pure C. But it's a matter
> of personal preference.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sun Feb  8 00:23:09 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 15:23:09 -0800
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071517i6b7b88er97ebc79e473781aa@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
	<498DF976.2050403@behnel.de>
	<957C545F-05CF-43FF-9C0F-618EADE87346@math.washington.edu>
	<e7ba66e40902071517i6b7b88er97ebc79e473781aa@mail.gmail.com>
Message-ID: <7C4EE0C4-897E-417D-B02C-9E001C3FED4E@math.washington.edu>

On Feb 7, 2009, at 3:17 PM, Lisandro Dalcin wrote:

> On Sat, Feb 7, 2009 at 8:02 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Feb 7, 2009, at 1:13 PM, Stefan Behnel wrote:
>>> Lisandro Dalcin wrote:
>>>>
>>>> cdef f():
>>>>     cdef Py_ssize_t a = 5
>>>>     cdef size_t b = 7
>>>>     return a + b
>>>>
>>>> I believe the value of 'a+b' should be promoted to 'signed long
>>>> long',
>>>
>>> Wouldn't Py_ssize_t be a suitable target type? Admittedly, it's <<
>>> size_t
>>> for non-negative types, but the missing bit shouldn't matter in
>>> almost all
>>> cases. And if users really need to make use of it, I'm fine with
>>> requiring
>>> them to write
>>>
>>>       return <PY_LONG_LONG>a + <PY_LONG_LONG>b
>>
>> Actually, the target should be size_t, following the C standard that
>> operations between signed and unsigned have an unsigned result. But
>> we certainly shouldn't be promoting it to a new type.
>>
>
> Yes, you are right, but then this testcase is expected to fail...
> Should we do better?
>
> __doc__ = """
>>>> f()
> -1
> """
> def f():
>     cdef Py_ssize_t a = -2
>     cdef size_t b = 1
>     return a + b

That is because the doctest is wrong. Signed + unsigned = unsigned.  
Consider

def f():
     cdef Py_ssize_t a = -2
     cdef size_t b = 1
     print a + b, a + b < 0

Now the whole "a + b < 0" statement will be evaluated in C land, and  
will always be False, which would make

-1, False

a really inconsistent thing to print out.

- Robert


From wstein at gmail.com  Sun Feb  8 00:46:53 2009
From: wstein at gmail.com (William Stein)
Date: Sat, 7 Feb 2009 15:46:53 -0800
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
	<e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>
Message-ID: <85e81ba30902071546w13fe74ectfc3066301870e0fc@mail.gmail.com>

On Sat, Feb 7, 2009 at 3:21 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> BTW,  a feature I really miss from the times I was a heavy SWIG user
> is the ability to include verbatim C code snipets...
>

Warning: blatant hack below; just for fun.

You can put verbatim C code snippets in Cython by abusing cdef extern.
 Here is an example that illustrates doing this to wrap some C++ code.
 I wanted to put the verbatim C++ code "a[b]=c" in, so I did it like
so.  You
could put any C/C++ code inside that macros though, not just a[b]=c.

cdef extern from "vector.h":
    ctypedef struct intvec "std::vector<unsigned int>":
        int get_entry "operator[]"(int n)
    intvec intvec_factory "std::vector<unsigned int>"(int len)

cdef extern from "":
    void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"()
    void SET(intvec, int, int)

DEFINE_SET()

cdef intvec v = intvec_factory(10)
SET(v,2,399283)
print v.get_entry(2)

Obviously you should never do this :-)

 -- William

From robertwb at math.washington.edu  Sun Feb  8 01:16:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 16:16:39 -0800
Subject: [Cython] Propagating exceptions with functions which returns
	void*
In-Reply-To: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
References: <66a8c45d0902030230g27dcca0bja411bd2f705de6@mail.gmail.com>
Message-ID: <48264AC7-4E37-45A4-A87F-F1BEADBB1CB1@math.washington.edu>

On Feb 3, 2009, at 2:30 AM, Daniele Pianu wrote:

> Probably I've not completely understood the C-function
> exception-propagating mechanism. I've a function with a prototype like
> this
>
> void* getItem( void* self, int i )
>
> The function returns the i-th item in self. Some index and bounds
> checking are implemented, so if, for example, I specify a negative
> index or an index out of self bounds I raise and exception. In cython
> the function is declared in this way:
>
> cdef void* getItem( void* self, int i ) except NULL
>
> But when an error occurs and the exception is raised I always receive
> a message which says that the exception is ignored. Obviously, the
> function continues its execution and terminates whit a segmentation
> fault.
>
> The question is: how can I propagate to caller functions the raised  
> exception?
> Thanks for your help (and for the great work that you've done with  
> cython :D ),

Perhaps the function you're calling this from is a cdef function  
without a way to propagate the error? It's really hard to debug  
without some more context though, perhaps you could put a self- 
contained example up at http://pastebin.com/

- Robert

From greg.ewing at canterbury.ac.nz  Sun Feb  8 02:16:13 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 08 Feb 2009 14:16:13 +1300
Subject: [Cython] problems with C numeric types
In-Reply-To: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
References: <e7ba66e40902071110o78e172a5o6697fce227fb8829@mail.gmail.com>
Message-ID: <498E325D.30609@canterbury.ac.nz>

Lisandro Dalcin wrote:

> + Look at the module-level tuple variable 'rank_to_type_name'. You can
> see that 'Py_ssize_t' and 'size_t' are AFTER the type 'PY_LONG_LONG'.
> 
> + Now look at the implementation of the function 'widest_numeric_type()'.

The basic problem here is that the Cython type system has
been stretched a bit too far beyond what was originally
cocceived in Pyrex. In the beginning, Pyrex only knew
about built-in C types, and although it didn't know
their actual sizes, it did know their relative sizes, so
it could calculate a widest type.

But now that we have things like Py_ssize_t that can be
defined differently on different platforms, that's no
longer possible.

I think the only completely correct thing to do would be
to introduce the notion of ints of unknow size and/or
signedness, and to refuse to implicitly convert anything
of such a type. The programmer would have to be explicit
about the conversion wanted in that case, e.g. by casting
it to a standard C type.

-- 
Greg

From dalcinl at gmail.com  Sun Feb  8 02:39:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 7 Feb 2009 22:39:46 -0300
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <85e81ba30902071546w13fe74ectfc3066301870e0fc@mail.gmail.com>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
	<e7ba66e40902071521n309e9f72k559f6cf8cd4d325a@mail.gmail.com>
	<85e81ba30902071546w13fe74ectfc3066301870e0fc@mail.gmail.com>
Message-ID: <e7ba66e40902071739i2ef851as36f7f980ca138d03@mail.gmail.com>

Just hope you never need to build your Cython-generated code with a C
compiler that does not accept C++-style comments  ;-)

Seriously, such a feature would be really useful. I've never made a
serious proposition just because I cannot figure out a "cythonic"
syntax for this... Perhaps somethig like

include verbatim """
<C code>
"""



On Sat, Feb 7, 2009 at 8:46 PM, William Stein <wstein at gmail.com> wrote:
> On Sat, Feb 7, 2009 at 3:21 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> BTW,  a feature I really miss from the times I was a heavy SWIG user
>> is the ability to include verbatim C code snipets...
>>
>
> Warning: blatant hack below; just for fun.
>
> You can put verbatim C code snippets in Cython by abusing cdef extern.
>  Here is an example that illustrates doing this to wrap some C++ code.
>  I wanted to put the verbatim C++ code "a[b]=c" in, so I did it like
> so.  You
> could put any C/C++ code inside that macros though, not just a[b]=c.
>
> cdef extern from "vector.h":
>    ctypedef struct intvec "std::vector<unsigned int>":
>        int get_entry "operator[]"(int n)
>    intvec intvec_factory "std::vector<unsigned int>"(int len)
>
> cdef extern from "":
>    void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"()
>    void SET(intvec, int, int)
>
> DEFINE_SET()
>
> cdef intvec v = intvec_factory(10)
> SET(v,2,399283)
> print v.get_entry(2)
>
> Obviously you should never do this :-)
>
>  -- William
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sun Feb  8 03:56:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 18:56:08 -0800
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <498358B9.20005@behnel.de>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>
	<49830111.7030201@student.matnat.uio.no> <498358B9.20005@behnel.de>
Message-ID: <E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>

On Jan 30, 2009, at 11:44 AM, Stefan Behnel wrote:

> Hi,
>
> Dag Sverre Seljebotn wrote:
>> Magnus Lie Hetland wrote:
>>> For the for-from loop, this may be intended -- as you say, it mimics
>>> the C behavior (although it seems to fly in the face of "Python
>>> intuition"): Even with the existing Cython semantics, getstep() will
>>> be called after *every iteration*.
>>>
>>> It seems to me that in
>>>
>>>    for 0 <= i < n by getstep(): pass
>>>
>>> one would expect (in Python) that the step expression be evaluated
>>> only once. If that is not the case, fine. Then the for-from loop can
>>> safely be left alone as it is.
>>>
>> I think so -- the for-from loop is a C idiom from which one can  
>> expect C
>> behaviour from my perspective, and if this has been the behaviour  
>> so far
>> then backward compatability alone speaks against altering it. The  
>> range
>> optimization has been around for much shorter and clearly should  
>> behave
>> as Python range, so that story is different.
>
> I vote for a) diverging behaviour between for-range and for-from  
> for the
> loop-variable after loop termination, and

+1

> b) the obvious evaluate-once
> semantics for the range() optimisation and the obvious (?)
> evaluate-on-each-step semantics for the for-from loop.

+1

> And while I'm fine with removing the emphases from the for-from  
> loop in the
> docs, I really don't think we should remove it from the language.  
> However,
> it needs to be stated in the docs that it follows the semantics of  
> the C
> for loop as far as possible, and that the usual Python for-range  
> pattern is
> preferred for readability and semantic clarity.

My thoughts exactly.

- Robert



From robertwb at math.washington.edu  Sun Feb  8 04:00:52 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Feb 2009 19:00:52 -0800
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>
	<49830111.7030201@student.matnat.uio.no> <498358B9.20005@behnel.de>
	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
Message-ID: <0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>

On Feb 7, 2009, at 6:56 PM, Robert Bradshaw wrote:

> On Jan 30, 2009, at 11:44 AM, Stefan Behnel wrote:
>
>> Hi,
>>
>> Dag Sverre Seljebotn wrote:
>>> Magnus Lie Hetland wrote:
>>>> For the for-from loop, this may be intended -- as you say, it  
>>>> mimics
>>>> the C behavior (although it seems to fly in the face of "Python
>>>> intuition"): Even with the existing Cython semantics, getstep()  
>>>> will
>>>> be called after *every iteration*.
>>>>
>>>> It seems to me that in
>>>>
>>>>    for 0 <= i < n by getstep(): pass
>>>>
>>>> one would expect (in Python) that the step expression be evaluated
>>>> only once. If that is not the case, fine. Then the for-from loop  
>>>> can
>>>> safely be left alone as it is.
>>>>
>>> I think so -- the for-from loop is a C idiom from which one can
>>> expect C
>>> behaviour from my perspective, and if this has been the behaviour
>>> so far
>>> then backward compatability alone speaks against altering it. The
>>> range
>>> optimization has been around for much shorter and clearly should
>>> behave
>>> as Python range, so that story is different.

I just realized, however, that we are not following Python semantics  
for the empty loop:

sage: i = "foo"
sage: for i in range(10, 0): pass
sage: print i
foo

- Robert

From magnus at hetland.org  Sun Feb  8 08:36:24 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 08:36:24 +0100
Subject: [Cython] Measuring "C-ness"?
In-Reply-To: <0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
References: <5F5F48AB-133D-4A1B-A236-1D09F8BABF5D@hetland.org>
	<498D4547.8060308@canterbury.ac.nz>
	<5F9EBB25-3879-4A31-9DED-94C4F74C79A8@hetland.org>
	<0E24B0D9-E34B-45C8-8CDA-6D6598752033@math.washington.edu>
Message-ID: <A91B8970-42DF-4B10-8605-3957A4B38124@hetland.org>

On Feb 8, 2009, at 00:09 , Robert Bradshaw wrote:

> Of course this requires one to keep two files in sync, make "cdef
> extern" declarations, and have a more complicated setup.py (i.e. no
> using pyximport), and risk gcc as well as Cython errors. I "program C
> using Cython" all the time--using the annotate option it is easy to
> see when a block of code gets compiled to pure C. But it's a matter
> of personal preference.

Right. This was how I planned to do it originally (for the reasons you  
state, and because of syntax preferences etc.).

Hm. I guess I'll see how good an overview I get with the annotation  
(and, possibly, the nogil declarations).

-- 
Magnus Lie Hetland
http://hetland.org


From stefan_ml at behnel.de  Sun Feb  8 09:26:18 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Feb 2009 09:26:18 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>
	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
Message-ID: <498E972A.9090205@behnel.de>


Robert Bradshaw wrote:
> I just realized, however, that we are not following Python semantics  
> for the empty loop:
> 
> sage: i = "foo"
> sage: for i in range(10, 0): pass
> sage: print i
> foo

http://trac.cython.org/cython_trac/ticket/208

Stefan


From magnus at hetland.org  Sun Feb  8 10:30:54 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 10:30:54 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <498E972A.9090205@behnel.de>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>
	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
	<498E972A.9090205@behnel.de>
Message-ID: <4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>

On Feb 8, 2009, at 09:26, Stefan Behnel wrote:

>
> Robert Bradshaw wrote:
>> I just realized, however, that we are not following Python semantics
>> for the empty loop:
>>
>> sage: i = "foo"
>> sage: for i in range(10, 0): pass
>> sage: print i
>> foo
>
> http://trac.cython.org/cython_trac/ticket/208

I could have a look.

-- 
Magnus Lie Hetland
http://hetland.org



From magnus at hetland.org  Sun Feb  8 11:52:05 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 11:52:05 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>
	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>
	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>
	<498E972A.9090205@behnel.de>
	<4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
Message-ID: <F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>

On Feb 8, 2009, at 10:30, Magnus Lie Hetland wrote:

>> http://trac.cython.org/cython_trac/ticket/208
>
> I could have a look.


I tried what seemed to me to be the obvious solution here -- surround  
the loop with an if statement with the same condition that is used for  
ending the loop.

Early in "generate_execution_code":

> from_range = getattr(self, "from_range", False)


Then, before the loop:

> if from_range:
>     code.putln("if (%s%s %s %s) {" % (
>     self.bound1.result(), offset, self.relation2, self.bound2.result()
>     ))

An end brace is output after the "undo" operator (which is also  
conditional on from_range).

I realize we again risk double-evaluating bound1 and bound2 here, but  
I'm guessing that the expressions will have been evaluated before  
they're used here anyway (as with step).

It seems correct, and the tests work without the refnanny (except for  
the NumPy stuff, which all fails because I haven't properly installed  
NumPy at the moment; there also seemed to be a problem when running  
with the refnanny). Should I upload a patch to the tracker?

By the way, while thinking of this, it occurred to me that it might be  
cleaner to just ditch the for loop than to undo its semantics. It  
won't help on the extra check needed to see if we assign to the loop  
variable at all (i.e., what I'm doing here), but by using a while-loop  
with the proper test, we should be able to avoid the spurious  
increment and the "undo" operation. Worth a shot? (Or do C compilers  
somehow rely on the use of for loops for some kind of optimization  
inference or something? Then again, as was pointed out, the extra  
assignment is probably optimized away when not needed -- perhaps  
that's the best solution, after all.)

-- 
Magnus Lie Hetland
http://hetland.org



From stefan_ml at behnel.de  Sun Feb  8 12:45:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Feb 2009 12:45:40 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>	<498E972A.9090205@behnel.de>	<4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
	<F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>
Message-ID: <498EC5E4.6010203@behnel.de>

Hi,

Magnus Lie Hetland wrote:
> Should I upload a patch to the tracker?

Sure, that will allow us to take a look at it and compare it to the
while-loop alternative. Just use "hg export" to keep the meta data.


> it occurred to me that it might be
> cleaner to just ditch the for loop than to undo its semantics. It  
> won't help on the extra check needed to see if we assign to the loop  
> variable at all (i.e., what I'm doing here), but by using a while-loop  
> with the proper test, we should be able to avoid the spurious  
> increment and the "undo" operation. Worth a shot?

That's my impression, too. I attached an old (incomplete) patch I had lying
around, don't know if it's still any good. Generating a straight
WhileStatNode instead might already do the right thing (and be less
invasive, given that we are nearing a release). See the methods of the
IterationTransform class in Optimize.py for that.


> Or do C compilers  
> somehow rely on the use of for loops for some kind of optimization  
> inference or something?

No idea. But I doubt that a while loop would be slower than the equivalent
for loop. Plus, we need to fix the semantics here, which is more important
than the last bit of performance.


> there also seemed to be a problem when running  
> with the refnanny.

Could you be a bit more specific here? :)


> the tests work without the refnanny (except for
> the NumPy stuff, which all fails because I haven't properly installed
> NumPy at the moment;

It should actually check for numpy being installed and just skip the test
if not. At least, that's what it does here.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: general_loop_node.patch
Type: text/x-patch
Size: 3149 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090208/f71279d7/attachment.bin 

From magnus at hetland.org  Sun Feb  8 13:03:59 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Sun, 8 Feb 2009 13:03:59 +0100
Subject: [Cython] Fix for #196 (for loop bug)
In-Reply-To: <498EC5E4.6010203@behnel.de>
References: <C871579B-826F-4DB7-9915-1BF3B18FD2BE@hetland.org>	<4982EAD0.2060101@student.matnat.uio.no>	<3888E3C8-3CC3-42EA-81F2-41CC382F4A0D@hetland.org>	<49830111.7030201@student.matnat.uio.no>	<498358B9.20005@behnel.de>	<E79963EF-25A7-496F-9A71-6A056F8AAC19@math.washington.edu>	<0C25F994-0990-4357-9464-B042A0FE82A9@math.washington.edu>	<498E972A.9090205@behnel.de>	<4B86DBBB-0CBE-470D-A035-8E183F6A9C48@hetland.org>
	<F892E69F-74B3-4E6E-A886-F6AAEC205D3B@hetland.org>
	<498EC5E4.6010203@behnel.de>
Message-ID: <DDE58962-4864-4C05-8249-C7455B1596E1@hetland.org>

On Feb 8, 2009, at 12:45, Stefan Behnel wrote:

> Hi,
>
> Magnus Lie Hetland wrote:
>> Should I upload a patch to the tracker?
>
> Sure, that will allow us to take a look at it and compare it to the
> while-loop alternative. Just use "hg export" to keep the meta data.

Done:

http://trac.cython.org/cython_trac/attachment/ticket/208/if_patch.txt

> That's my impression, too. I attached an old (incomplete) patch I  
> had lying
> around, don't know if it's still any good.

Cool. Don't have time to look at it right now, myself, though.

>> there also seemed to be a problem when running
>> with the refnanny.
>
> Could you be a bit more specific here? :)

My mistake. It seems it was rather some cpp stuff that gave me some  
lost refcount messages. Ran it with --no-cpp and it went away.

>> the tests work without the refnanny (except for
>> the NumPy stuff, which all fails because I haven't properly installed
>> NumPy at the moment;
>
> It should actually check for numpy being installed and just skip the  
> test
> if not. At least, that's what it does here.

Hm. It seems I actually do have numpy installed for the same Python  
that I used for this. But the error I get is:

numpy_test.c:129:31: error: numpy/arrayobject.h: No such file or  
directorynumpy_test.c:129:31:
  error: numpy/arrayobject.h: No such file or directory

Followed by lots of "warning: data definition has no type or storage  
class" and syntax errors and undeclared objects in numpy_test.c.

The errors I end up with when running with no switches:

ERROR: compiling (c) and running numpy_test
ERROR: compiling (cpp) and running numpy_test
ERROR: compiling (cpp) and running withstat
FAIL: Doctest: tryfinally
     Error: References leaked:
       Acquired on lines: 396

(The last one appears twice.)

-- 
Magnus Lie Hetland
http://hetland.org



From ondrej at certik.cz  Mon Feb  9 05:06:52 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Sun, 8 Feb 2009 20:06:52 -0800
Subject: [Cython] the LICENSE.txt is unreadable
Message-ID: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>

Hi,

the LICENSE.txt looks like this on my system (Debian) if I open it in vim:

"""
Apache License^M                           Version 2.0, January 2004^M
                       http://www.apache.org/licenses/^M^M   TERMS AND
CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
Definitions.^M^M      "License" shall mean the terms and conditions
for use, reproduction,^M      and distribution as defined by Sections
1 through 9 of this document.^M^M      "Licensor" shall mean the
copyright owner or entity authorized by^M      the copyright owner
that is granting the License.^M^M      "Legal Entity" shall mean the
union of the acting entity and all^M      other entities that cont
[...]
"""

Is it just for me or for everyone?

You can pull a fix from my repository:

http://bitbucket.org/certik/cython-devel/

Ondrej

From sjmachin at lexicon.net  Mon Feb  9 08:54:23 2009
From: sjmachin at lexicon.net (John Machin)
Date: Mon, 09 Feb 2009 18:54:23 +1100
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
Message-ID: <498FE12F.5090903@lexicon.net>

On 9/02/2009 3:06 PM, Ondrej Certik wrote:
> Hi,
> 
> the LICENSE.txt looks like this on my system (Debian) if I open it in vim:
> 
> """
> Apache License^M                           Version 2.0, January 2004^M
>                        http://www.apache.org/licenses/^M^M   TERMS AND
> CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
> Definitions.^M^M      "License" shall mean the terms and conditions
> for use, reproduction,^M      and distribution as defined by Sections
> 1 through 9 of this document.^M^M      "Licensor" shall mean the
> copyright owner or entity authorized by^M      the copyright owner
> that is granting the License.^M^M      "Legal Entity" shall mean the
> union of the acting entity and all^M      other entities that cont
> [...]
> """
> 
> Is it just for me or for everyone?

The file appears to be using the Mac convention i.e. '\r' for line 
terminators. It's a while since I've used a text editor that didn't read 
files silently in whichever of the 3 conventions it found, and write 
them out in the same fashion unless told otherwise. Perhaps you could 
check if vim has an option to do that.

> You can pull a fix from my repository:

A fix for what?

From ondrej at certik.cz  Mon Feb  9 09:19:30 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 9 Feb 2009 00:19:30 -0800
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <498FE12F.5090903@lexicon.net>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
	<498FE12F.5090903@lexicon.net>
Message-ID: <85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>

On Sun, Feb 8, 2009 at 11:54 PM, John Machin <sjmachin at lexicon.net> wrote:
> On 9/02/2009 3:06 PM, Ondrej Certik wrote:
>> Hi,
>>
>> the LICENSE.txt looks like this on my system (Debian) if I open it in vim:
>>
>> """
>> Apache License^M                           Version 2.0, January 2004^M
>>                        http://www.apache.org/licenses/^M^M   TERMS AND
>> CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
>> Definitions.^M^M      "License" shall mean the terms and conditions
>> for use, reproduction,^M      and distribution as defined by Sections
>> 1 through 9 of this document.^M^M      "Licensor" shall mean the
>> copyright owner or entity authorized by^M      the copyright owner
>> that is granting the License.^M^M      "Legal Entity" shall mean the
>> union of the acting entity and all^M      other entities that cont
>> [...]
>> """
>>
>> Is it just for me or for everyone?
>
> The file appears to be using the Mac convention i.e. '\r' for line
> terminators. It's a while since I've used a text editor that didn't read
> files silently in whichever of the 3 conventions it found, and write
> them out in the same fashion unless told otherwise. Perhaps you could
> check if vim has an option to do that.
>
>> You can pull a fix from my repository:
>
> A fix for what?

To use linux convention. It may not be a fix though, depending on the above.

Ondrej

From dalcinl at gmail.com  Mon Feb  9 14:58:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 9 Feb 2009 11:58:21 -0200
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
	<498FE12F.5090903@lexicon.net>
	<85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>
Message-ID: <e7ba66e40902090558g30bd1814h8715ff7b9a10ac4b@mail.gmail.com>

Anyway, IMHO all files should use the same convention.

If the Linux one is used, Windows users would need this:

http://www.selenic.com/mercurial/wiki/index.cgi/Win32TextExtension


On Mon, Feb 9, 2009 at 6:19 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Sun, Feb 8, 2009 at 11:54 PM, John Machin <sjmachin at lexicon.net> wrote:
>> On 9/02/2009 3:06 PM, Ondrej Certik wrote:
>>> Hi,
>>>
>>> the LICENSE.txt looks like this on my system (Debian) if I open it in vim:
>>>
>>> """
>>> Apache License^M                           Version 2.0, January 2004^M
>>>                        http://www.apache.org/licenses/^M^M   TERMS AND
>>> CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION^M^M   1.
>>> Definitions.^M^M      "License" shall mean the terms and conditions
>>> for use, reproduction,^M      and distribution as defined by Sections
>>> 1 through 9 of this document.^M^M      "Licensor" shall mean the
>>> copyright owner or entity authorized by^M      the copyright owner
>>> that is granting the License.^M^M      "Legal Entity" shall mean the
>>> union of the acting entity and all^M      other entities that cont
>>> [...]
>>> """
>>>
>>> Is it just for me or for everyone?
>>
>> The file appears to be using the Mac convention i.e. '\r' for line
>> terminators. It's a while since I've used a text editor that didn't read
>> files silently in whichever of the 3 conventions it found, and write
>> them out in the same fashion unless told otherwise. Perhaps you could
>> check if vim has an option to do that.
>>
>>> You can pull a fix from my repository:
>>
>> A fix for what?
>
> To use linux convention. It may not be a fix though, depending on the above.
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From broughtj at gmail.com  Mon Feb  9 18:45:28 2009
From: broughtj at gmail.com (Tyler Brough)
Date: Mon, 9 Feb 2009 10:45:28 -0700
Subject: [Cython] Speeding up file io
Message-ID: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>

Hello All,

I am new to Cython, though I heard about it long ago.  I have been
programming in python using scipy and pytables a lot. I am building an HDF5
database using pytables from extremely large files containing tick by tick
stock data (one single day is several GBs). As you can imagine this is very
slow going, even trying to optimize the file io in python. What I am
wondering is will trying to do the file io in cython functions speed up the
process enough to make it worth while?  Can I simply access c FILE pointers
directly in cython and pass them back as python objects?  Your advice is
very much appreciated!

Best,

--TJB

P.S. I am working in Ubuntu 8.10.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090209/6496b9de/attachment.htm 

From stefan_ml at behnel.de  Mon Feb  9 19:14:33 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 9 Feb 2009 19:14:33 +0100 (CET)
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
Message-ID: <45800.213.61.181.86.1234203273.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Tyler Brough wrote:
> I am new to Cython, though I heard about it long ago.  I have been
> programming in python using scipy and pytables a lot. I am building an
> HDF5
> database using pytables from extremely large files containing tick by tick
> stock data (one single day is several GBs). As you can imagine this is
> very
> slow going, even trying to optimize the file io in python. What I am
> wondering is will trying to do the file io in cython functions speed up
> the process enough to make it worth while?

Cython will not magically speed up your I/O operations for you. It does,
however, allow you to work at the C level if you need to, thus avoiding
I/O overhead yourself, at the cost of writing low-level-ish code. Also, if
you keep your code free from Python-level operations, you can free the GIL
and thus leverage full thread parallelism on I/O operations.


> Can I simply access c FILE pointers
> directly in cython and pass them back as python objects?

Cython doesn't convert between FILE and file() automatically. However, the
Python 2.x C-API allows you to access the low-level FILE* from a file()
instance, so you can open a file at the Python level (also from Cython),
and then work with the FILE* in your I/O operations.

I never worked with pytables, so I can't say how that fits together, and
what Cython can do for you here. What makes you think the I/O performance
itself is your problem? What kind of operations are you performing on the
database?

Stefan


From dagss at student.matnat.uio.no  Mon Feb  9 19:28:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 9 Feb 2009 19:28:02 +0100 (CET)
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
Message-ID: <251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>

>Tyler Brough wrote:
 Hello All,
>
> I am new to Cython, though I heard about it long ago.  I have been
> programming in python using scipy and pytables a lot. I am building an
> HDF5
> database using pytables from extremely large files containing tick by tick
> stock data (one single day is several GBs). As you can imagine this is
> very
> slow going, even trying to optimize the file io in python. What I am
> wondering is will trying to do the file io in cython functions speed up
> the
> process enough to make it worth while?  Can I simply access c FILE
> pointers
> directly in cython and pass them back as python objects?  Your advice is
> very much appreciated!

A quick look on the pytables source reveal that they have plenty of C
files there, and also some Pyrex and/or Cython files. The Pytable authors
have likely thought of all the obvious ways to speed things up already...

But to answer your question; from Cython you can call all the C IO
functions directly, but a C FILE pointer cannot be passed into Python. You
might be able to access the underlying handle of Python file objects if
that's what you mean, you should look in the Python headers and CPython
documentation for that.

Please ask again if this missed the mark.

Dag Sverre


From robertwb at math.washington.edu  Mon Feb  9 21:16:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 9 Feb 2009 12:16:36 -0800
Subject: [Cython] Speeding up file io
In-Reply-To: <251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
Message-ID: <46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>

On Feb 9, 2009, at 10:28 AM, Dag Sverre Seljebotn wrote:

>> Tyler Brough wrote:
>  Hello All,
>>
>> I am new to Cython, though I heard about it long ago.  I have been
>> programming in python using scipy and pytables a lot. I am  
>> building an
>> HDF5
>> database using pytables from extremely large files containing tick  
>> by tick
>> stock data (one single day is several GBs). As you can imagine  
>> this is
>> very
>> slow going, even trying to optimize the file io in python. What I am
>> wondering is will trying to do the file io in cython functions  
>> speed up
>> the
>> process enough to make it worth while?  Can I simply access c FILE
>> pointers
>> directly in cython and pass them back as python objects?  Your  
>> advice is
>> very much appreciated!
>
> A quick look on the pytables source reveal that they have plenty of C
> files there, and also some Pyrex and/or Cython files. The Pytable  
> authors
> have likely thought of all the obvious ways to speed things up  
> already...
>
> But to answer your question; from Cython you can call all the C IO
> functions directly, but a C FILE pointer cannot be passed into  
> Python. You
> might be able to access the underlying handle of Python file  
> objects if
> that's what you mean, you should look in the Python headers and  
> CPython
> documentation for that.

You can pass file objects around in Python, which are basically  
wrappers around c FILE pointers (which can easily be extracted and  
used directly from Cython code). As has already been mentioned, it is  
unclear if this will provide speed gains (especially if your  
application is io bound rather than cpu bound).

- Robert


From dalcinl at gmail.com  Mon Feb  9 21:39:41 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 9 Feb 2009 18:39:41 -0200
Subject: [Cython] PATCH: fix reference count problems in tryfinally testcase,
	please 	review & comment
Message-ID: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>

Since Dag's refnanny was integrated in cython-devel,
test/run/tryfinally.pyx was failing because of reference count errors.

The patch below is the only way I can imagine to fix such problem. I
expect the patch could be controversial...


diff -r 3b522098d133 Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py	Sun Feb 08 09:25:28 2009 +0100
+++ b/Cython/Compiler/Nodes.py	Mon Feb 09 18:22:21 2009 -0200
@@ -1054,6 +1054,8 @@
         code.put_var_declarations(lenv.var_entries)
         init = ""
         if not self.return_type.is_void:
+            if self.return_type.is_pyobject:
+                init = " = NULL"
             code.putln(
                 "%s%s;" %
                     (self.return_type.declaration_code(
@@ -3371,6 +3373,9 @@
         if self.value:
             self.value.generate_evaluation_code(code)
             self.value.make_owned_reference(code)
+            if self.return_type.is_pyobject:
+                code.put_xdecref(Naming.retval_cname,
+                                 self.return_type)
             code.putln(
                 "%s = %s;" % (
                     Naming.retval_cname,

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb  9 22:53:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 09 Feb 2009 22:53:01 +0100
Subject: [Cython] PATCH: fix reference count problems in tryfinally
 testcase, please 	review & comment
In-Reply-To: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
Message-ID: <4990A5BD.1030707@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Since Dag's refnanny was integrated in cython-devel,
> test/run/tryfinally.pyx was failing because of reference count errors.
> 
> The patch below is the only way I can imagine to fix such problem. I
> expect the patch could be controversial...

Me and Stefan should be OK with it at least: See #200 in trac. (Whether 
you initialize to Py_None or NULL is a finer point I suppose, I don't 
have a feeling either way).

To avoid the XDECREF in most situations one could make it depend on a 
flag in ReturnStatNode and write a transform in Optimize.py which turns 
on the flag on all return nodes in functions which contain return nodes 
within finally clauses. But that could be done later if this fix is 
pushed now.

Dag Sverre


> 
> 
> diff -r 3b522098d133 Cython/Compiler/Nodes.py
> --- a/Cython/Compiler/Nodes.py	Sun Feb 08 09:25:28 2009 +0100
> +++ b/Cython/Compiler/Nodes.py	Mon Feb 09 18:22:21 2009 -0200
> @@ -1054,6 +1054,8 @@
>          code.put_var_declarations(lenv.var_entries)
>          init = ""
>          if not self.return_type.is_void:
> +            if self.return_type.is_pyobject:
> +                init = " = NULL"
>              code.putln(
>                  "%s%s;" %
>                      (self.return_type.declaration_code(
> @@ -3371,6 +3373,9 @@
>          if self.value:
>              self.value.generate_evaluation_code(code)
>              self.value.make_owned_reference(code)
> +            if self.return_type.is_pyobject:
> +                code.put_xdecref(Naming.retval_cname,
> +                                 self.return_type)
>              code.putln(
>                  "%s = %s;" % (
>                      Naming.retval_cname,
> 


-- 
Dag Sverre

From glenn at tarbox.org  Tue Feb 10 00:29:08 2009
From: glenn at tarbox.org (Glenn Tarbox, PhD)
Date: Mon, 9 Feb 2009 15:29:08 -0800
Subject: [Cython] Speeding up file io
In-Reply-To: <46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
	<46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
Message-ID: <f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>

On Mon, Feb 9, 2009 at 12:16 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On Feb 9, 2009, at 10:28 AM, Dag Sverre Seljebotn wrote:
>
> >> Tyler Brough wrote:
> >  Hello All,
> >>
> >> I am new to Cython, though I heard about it long ago.  I have been
> >> programming in python using scipy and pytables a lot. I am
> >> building an
> >> HDF5
> >> database using pytables from extremely large files containing tick
> >> by tick
> >> stock data (one single day is several GBs). As you can imagine
> >> this is
> >> very
> >> slow going, even trying to optimize the file io in python. What I am
> >> wondering is will trying to do the file io in cython functions
> >> speed up
> >> the
> >> process enough to make it worth while?  Can I simply access c FILE
> >> pointers
> >> directly in cython and pass them back as python objects?  Your
> >> advice is
> >> very much appreciated!
> >
> > A quick look on the pytables source reveal that they have plenty of C
> > files there, and also some Pyrex and/or Cython files. The Pytable
> > authors
> > have likely thought of all the obvious ways to speed things up
> > already...
> >
> > But to answer your question; from Cython you can call all the C IO
> > functions directly, but a C FILE pointer cannot be passed into
> > Python. You
> > might be able to access the underlying handle of Python file
> > objects if
> > that's what you mean, you should look in the Python headers and
> > CPython
> > documentation for that.
>
> You can pass file objects around in Python, which are basically
> wrappers around c FILE pointers (which can easily be extracted and
> used directly from Cython code). As has already been mentioned, it is
> unclear if this will provide speed gains (especially if your
> application is io bound rather than cpu bound).


The theme from Love Story keeps playing in my head as I read...

First, Cython will provide substantial advantages throughout your project if
you're dealing with tick data and python.  You're gonna have places where
numpy / PyTables abstractions aren't sufficient so you'll need your own
code.  If you're focusing on python, you need a mechanism to prototype and
migrate into native code easily... Cython is the answer to that.

However, having worked (and continue to work) with tick data, its not clear
that  PyTables / HDF5 for the lowest levels of your application is the way
to go.  I also started down that path and decided that I needed to go
native...  But that really means "other stuff" that works with the usual
Python subjects.

The real issue with tick data is: "Everything is Huge".  Python and the
library abstractions provided by Numpy, HDF5, PyTables etc are generally
fine if you fit perfectly inside.  For most, when it isn't perfect, a few
lines of python and all is right with the world...

not so much for you...

You should take a quick look at the finance package in Sage.  Inside, you'll
find small set of timeseries classes with support for related operations
written by William Stein.  William and I had spent some time discussing the
issues with tick-data and he was able to produce much of the core capability
in a few days.  That alone should speak volumes about Cython (although
William is, um, lets say "more productive" than most... to put it mildly).

When William wrote those classes, he determined that the abstractions
afforded by Numpy were inappropriate for time-series due to Numpy's
generality, which manifests itself at run time (in compiled C code) with a
number of calculations / logic to make Numpy do Numpy things.  When you're
working with a huge block of tick data, this overhead is gonna hurt.

So, his approach was to facilitate conversion between regular python lists,
numpy arrays, and the tick data structure which is a 1d block of doubles.
(malloc...)

The key to Python (really C-Python) is that it was written to be glue over
C.  Cython is many things, but amongst them is making the language
transition simpler by hiding the details of integrating with the Python
interpreter (you certainly know this).

So, I think you're gonna find that you're better off "going native" at the
tick level because at the Hugeness issue.  You'll likely save development
and certainly execution time with low level code specific to your problem
(maybe mmap'ed blocks of doubles to address the file issues discussed above)

Cython makes this a tolerable experience.

-glenn


>
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
*** Note: New Number
Glenn H. Tarbox, PhD ||  206-274-6919
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090209/57d8fd90/attachment-0001.htm 

From broughtj at gmail.com  Tue Feb 10 01:06:02 2009
From: broughtj at gmail.com (Tyler Brough)
Date: Mon, 9 Feb 2009 17:06:02 -0700
Subject: [Cython] Speeding up file io
In-Reply-To: <f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
	<46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
	<f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>
Message-ID: <ace293fe0902091606k1d1e0aean56002acae6cf4a53@mail.gmail.com>

Thank you all for your input.  I have a lot to think about.

--Tyler

On Mon, Feb 9, 2009 at 4:29 PM, Glenn Tarbox, PhD <glenn at tarbox.org> wrote:

>
>
> On Mon, Feb 9, 2009 at 12:16 PM, Robert Bradshaw <
> robertwb at math.washington.edu> wrote:
>
>> On Feb 9, 2009, at 10:28 AM, Dag Sverre Seljebotn wrote:
>>
>> >> Tyler Brough wrote:
>> >  Hello All,
>> >>
>> >> I am new to Cython, though I heard about it long ago.  I have been
>> >> programming in python using scipy and pytables a lot. I am
>> >> building an
>> >> HDF5
>> >> database using pytables from extremely large files containing tick
>> >> by tick
>> >> stock data (one single day is several GBs). As you can imagine
>> >> this is
>> >> very
>> >> slow going, even trying to optimize the file io in python. What I am
>> >> wondering is will trying to do the file io in cython functions
>> >> speed up
>> >> the
>> >> process enough to make it worth while?  Can I simply access c FILE
>> >> pointers
>> >> directly in cython and pass them back as python objects?  Your
>> >> advice is
>> >> very much appreciated!
>> >
>> > A quick look on the pytables source reveal that they have plenty of C
>> > files there, and also some Pyrex and/or Cython files. The Pytable
>> > authors
>> > have likely thought of all the obvious ways to speed things up
>> > already...
>> >
>> > But to answer your question; from Cython you can call all the C IO
>> > functions directly, but a C FILE pointer cannot be passed into
>> > Python. You
>> > might be able to access the underlying handle of Python file
>> > objects if
>> > that's what you mean, you should look in the Python headers and
>> > CPython
>> > documentation for that.
>>
>> You can pass file objects around in Python, which are basically
>> wrappers around c FILE pointers (which can easily be extracted and
>> used directly from Cython code). As has already been mentioned, it is
>> unclear if this will provide speed gains (especially if your
>> application is io bound rather than cpu bound).
>
>
> The theme from Love Story keeps playing in my head as I read...
>
> First, Cython will provide substantial advantages throughout your project
> if you're dealing with tick data and python.  You're gonna have places where
> numpy / PyTables abstractions aren't sufficient so you'll need your own
> code.  If you're focusing on python, you need a mechanism to prototype and
> migrate into native code easily... Cython is the answer to that.
>
> However, having worked (and continue to work) with tick data, its not clear
> that  PyTables / HDF5 for the lowest levels of your application is the way
> to go.  I also started down that path and decided that I needed to go
> native...  But that really means "other stuff" that works with the usual
> Python subjects.
>
> The real issue with tick data is: "Everything is Huge".  Python and the
> library abstractions provided by Numpy, HDF5, PyTables etc are generally
> fine if you fit perfectly inside.  For most, when it isn't perfect, a few
> lines of python and all is right with the world...
>
> not so much for you...
>
> You should take a quick look at the finance package in Sage.  Inside,
> you'll find small set of timeseries classes with support for related
> operations written by William Stein.  William and I had spent some time
> discussing the issues with tick-data and he was able to produce much of the
> core capability in a few days.  That alone should speak volumes about Cython
> (although William is, um, lets say "more productive" than most... to put it
> mildly).
>
> When William wrote those classes, he determined that the abstractions
> afforded by Numpy were inappropriate for time-series due to Numpy's
> generality, which manifests itself at run time (in compiled C code) with a
> number of calculations / logic to make Numpy do Numpy things.  When you're
> working with a huge block of tick data, this overhead is gonna hurt.
>
> So, his approach was to facilitate conversion between regular python lists,
> numpy arrays, and the tick data structure which is a 1d block of doubles.
> (malloc...)
>
> The key to Python (really C-Python) is that it was written to be glue over
> C.  Cython is many things, but amongst them is making the language
> transition simpler by hiding the details of integrating with the Python
> interpreter (you certainly know this).
>
> So, I think you're gonna find that you're better off "going native" at the
> tick level because at the Hugeness issue.  You'll likely save development
> and certainly execution time with low level code specific to your problem
> (maybe mmap'ed blocks of doubles to address the file issues discussed above)
>
> Cython makes this a tolerable experience.
>
> -glenn
>
>
>>
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> *** Note: New Number
> Glenn H. Tarbox, PhD ||  206-274-6919
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090209/eb3d6c7a/attachment.htm 

From wstein at gmail.com  Tue Feb 10 03:00:53 2009
From: wstein at gmail.com (William Stein)
Date: Mon, 9 Feb 2009 18:00:53 -0800
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902091606k1d1e0aean56002acae6cf4a53@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
	<251a9e27a26b2dfbd0f7e0d33ded0d42.squirrel@webmail.uio.no>
	<46007D8E-A910-4391-9FC4-B8AE6CEAADB5@math.washington.edu>
	<f049b35c0902091529u7820c6d5y9870a92292edf32a@mail.gmail.com>
	<ace293fe0902091606k1d1e0aean56002acae6cf4a53@mail.gmail.com>
Message-ID: <85e81ba30902091800y5702b7cbx9e46aa43d171e683@mail.gmail.com>

On Mon, Feb 9, 2009 at 4:06 PM, Tyler Brough <broughtj at gmail.com> wrote:
> Thank you all for your input.  I have a lot to think about.
>
> --Tyler

>> You should take a quick look at the finance package in Sage.  Inside,
>> you'll find small set of timeseries classes with support for related
>> operations written by William Stein.  William and I had spent some time
>> discussing the issues with tick-data and he was able to produce much of the
>> core capability in a few days.  That alone should speak volumes about Cython
>> (although William is, um, lets say "more productive" than most... to put it
>> mildly).

The code he's talking about is here:

http://www.sagemath.org/hg/sage-main/file/b0aa7ef45b3c/sage/finance/

You can try any of that stuff out live online without having to
install anything by
going to http://sagenb.org and making a free account.

 -- William

From robertwb at math.washington.edu  Tue Feb 10 03:44:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 9 Feb 2009 18:44:28 -0800
Subject: [Cython] the LICENSE.txt is unreadable
In-Reply-To: <e7ba66e40902090558g30bd1814h8715ff7b9a10ac4b@mail.gmail.com>
References: <85b5c3130902082006u5d2e7daepa0a23cdde9508fe9@mail.gmail.com>
	<498FE12F.5090903@lexicon.net>
	<85b5c3130902090019y5002e540jb8c9d8e84ec60dc1@mail.gmail.com>
	<e7ba66e40902090558g30bd1814h8715ff7b9a10ac4b@mail.gmail.com>
Message-ID: <ECBB045C-8272-4BFC-AA00-AE364D73A105@math.washington.edu>

On Feb 9, 2009, at 5:58 AM, Lisandro Dalcin wrote:

> Anyway, IMHO all files should use the same convention.

I agree. I just fixed the remaining inconsistencies. OS X uses \n for  
line breaks just fine.

> If the Linux one is used, Windows users would need this:
>
> http://www.selenic.com/mercurial/wiki/index.cgi/Win32TextExtension

Almost all the files already using a simple \n for line breaks, so I  
guess anyone who needs this is already using it. Most editors should  
handle all line endings fine nowadays.

- Robert


From robertwb at math.washington.edu  Tue Feb 10 08:03:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 9 Feb 2009 23:03:26 -0800
Subject: [Cython] PATCH: fix reference count problems in tryfinally
	testcase, please 	review & comment
In-Reply-To: <4990A5BD.1030707@student.matnat.uio.no>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
	<4990A5BD.1030707@student.matnat.uio.no>
Message-ID: <F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>

On Feb 9, 2009, at 1:53 PM, Dag Sverre Seljebotn wrote:

> Lisandro Dalcin wrote:
>> Since Dag's refnanny was integrated in cython-devel,
>> test/run/tryfinally.pyx was failing because of reference count  
>> errors.
>>
>> The patch below is the only way I can imagine to fix such problem. I
>> expect the patch could be controversial...
>
> Me and Stefan should be OK with it at least: See #200 in trac.  
> (Whether
> you initialize to Py_None or NULL is a finer point I suppose, I don't
> have a feeling either way).
>
> To avoid the XDECREF in most situations one could make it depend on a
> flag in ReturnStatNode and write a transform in Optimize.py which  
> turns
> on the flag on all return nodes in functions which contain return  
> nodes
> within finally clauses. But that could be done later if this fix is
> pushed now.

I'm not sure this is the right fix, but it does appear to solve the  
(serious) issue so we should include it and can optimize later.

- Robert



From magnus at hetland.org  Tue Feb 10 12:00:57 2009
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Tue, 10 Feb 2009 12:00:57 +0100
Subject: [Cython] Speeding up file io
In-Reply-To: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
References: <ace293fe0902090945i585a8bferea334f4093d0ac32@mail.gmail.com>
Message-ID: <C6D36111-6056-47B3-A738-813AB369BF97@hetland.org>

On Feb 9, 2009, at 18:45, Tyler Brough wrote:

> Hello All,
>
> I am new to Cython, though I heard about it long ago.  I have been  
> programming in python using scipy and pytables a lot. I am building  
> an HDF5 database using pytables from extremely large files  
> containing tick by tick stock data (one single day is several GBs).  
> As you can imagine this is very slow going, even trying to optimize  
> the file io in python. What I am wondering is will trying to do the  
> file io in cython functions speed up the process enough to make it  
> worth while?  Can I simply access c FILE pointers directly in cython  
> and pass them back as python objects?  Your advice is very much  
> appreciated!

I'm doing this sort of thing. I use file descriptors in my  
interactions between C and Python, as they are simply ints, and both C  
and Python have functions for using them (e.g., I use os.close() in  
Python to close a file based on a file descriptor, etc.).

(But, as someone else mentioned, there are already useful wrappers for  
HDF5 in Python that might be worth checking out.)

-- 
Magnus Lie Hetland
http://hetland.org



From dalcinl at gmail.com  Tue Feb 10 14:27:47 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 11:27:47 -0200
Subject: [Cython] PATCH: fix reference count problems in tryfinally
	testcase, please review & comment
In-Reply-To: <F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
	<4990A5BD.1030707@student.matnat.uio.no>
	<F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
Message-ID: <e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>

On Tue, Feb 10, 2009 at 5:03 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> I'm not sure this is the right fix, but it does appear to solve the
> (serious) issue so we should include it and can optimize later.
>

Indeed, the patch did not handle well all cases...

My fix do not work well when cdef functions returning PyObject*, not
sure at this point what to do.

I've pushed two additions to tryfinally testcase, both fail :-(

+
+
+def try_return_none_1():
+    try:
+        return
+    finally:
+        return
+
+cdef extern from *:
+    ctypedef struct PyObject
+    void Py_INCREF(object)
+
+cdef PyObject* _none():
+    ret = None
+    Py_INCREF(ret)
+    return <PyObject*> ret
+
+def try_return_none_2():
+    try:
+        return <object> _none()
+    finally:
+        return <object> _none()



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb 10 14:36:08 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 11:36:08 -0200
Subject: [Cython] failure of withstat
Message-ID: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>

See this:

$ python runtests.py --no-cleanup --no-c withstat
...
OK

$ python runtests.py --no-cleanup --no-cpp withstat
...
OK

$ python runtests.py --no-cleanup --no-cpp withstat
...
withstat.cpp: In function 'PyObject*
__pyx_pf_8withstat_no_as(PyObject*, PyObject*)':
withstat.cpp:801: error: '__pyx_v___tmpvar_2' was not declared in this scope
withstat.cpp:808: error: '__pyx_v___tmpvar_4' was not declared in this scope
withstat.cpp:825: error: '__pyx_v___tmpvar_3' was not declared in this scope
...
CompileError: command 'gcc' failed with exit status 1
...

$ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
< ... lots of output, a bit below >
867,868c838,839
<         __Pyx_DECREF(__pyx_v___tmpvar_1);
<         __pyx_v___tmpvar_1 = Py_None;
---
>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>         __pyx_v___tmpvar_40 = Py_None;


A problem with runtests.py? Perhaps a bad interaction on temp
allocation code and global module/class level variables in the Cython
codebase?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Feb 10 14:49:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Feb 2009 14:49:46 +0100 (CET)
Subject: [Cython] failure of withstat
In-Reply-To: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
Message-ID: <32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> $ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
> < ... lots of output, a bit below >
> 867,868c838,839
> <         __Pyx_DECREF(__pyx_v___tmpvar_1);
> <         __pyx_v___tmpvar_1 = Py_None;
> ---
>>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>>         __pyx_v___tmpvar_40 = Py_None;

Dag already hijacked a bug report for this.

http://trac.cython.org/cython_trac/ticket/151

Stefan


From dagss at student.matnat.uio.no  Tue Feb 10 14:54:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 10 Feb 2009 14:54:28 +0100
Subject: [Cython] failure of withstat
In-Reply-To: <32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <49918714.7060809@student.matnat.uio.no>

Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>   
>> $ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
>> < ... lots of output, a bit below >
>> 867,868c838,839
>> <         __Pyx_DECREF(__pyx_v___tmpvar_1);
>> <         __pyx_v___tmpvar_1 = Py_None;
>> ---
>>     
>>>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>>>         __pyx_v___tmpvar_40 = Py_None;
>>>       
>
> Dag already hijacked a bug report for this.
>
> http://trac.cython.org/cython_trac/ticket/151
>   
Heh. Well, it was my fix for #151 which broke things, so that's why...
 
I'm mystified as to what is going on though, and won't have more time to 
look at it for some days. Suggestions welcome.

Dag Sverre

From dalcinl at gmail.com  Tue Feb 10 15:10:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 12:10:39 -0200
Subject: [Cython] failure of withstat
In-Reply-To: <49918714.7060809@student.matnat.uio.no>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49918714.7060809@student.matnat.uio.no>
Message-ID: <e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>

Just to be sure...

Did you realize that a C++ only run (and also a C only run passing
--no-cpp) of the testsuite

$ python runtests.py --no-c withstat

do indeed succeed?


On Tue, Feb 10, 2009 at 11:54 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>
>>> $ diff BUILD/run/c/withstat.c BUILD/run/cpp/withstat.cpp
>>> < ... lots of output, a bit below >
>>> 867,868c838,839
>>> <         __Pyx_DECREF(__pyx_v___tmpvar_1);
>>> <         __pyx_v___tmpvar_1 = Py_None;
>>> ---
>>>
>>>>         __Pyx_DECREF(__pyx_v___tmpvar_40);
>>>>         __pyx_v___tmpvar_40 = Py_None;
>>>>
>>
>> Dag already hijacked a bug report for this.
>>
>> http://trac.cython.org/cython_trac/ticket/151
>>
> Heh. Well, it was my fix for #151 which broke things, so that's why...
>
> I'm mystified as to what is going on though, and won't have more time to
> look at it for some days. Suggestions welcome.
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb 10 15:22:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 10 Feb 2009 15:22:12 +0100
Subject: [Cython] failure of withstat
In-Reply-To: <e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49918714.7060809@student.matnat.uio.no>
	<e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>
Message-ID: <49918D94.20301@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Just to be sure...
>
> Did you realize that a C++ only run (and also a C only run passing
> --no-cpp) of the testsuite
>
> $ python runtests.py --no-c withstat
>
> do indeed succeed?
>   
Ah. Yes, I noticed, but after commenting on the ticket. So apparently it 
is NOT a C++-only issue, but something much more strange going on ...

Dag Sverre

From dalcinl at gmail.com  Tue Feb 10 15:41:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 12:41:21 -0200
Subject: [Cython] failure of withstat
In-Reply-To: <49918D94.20301@student.matnat.uio.no>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>
	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<49918714.7060809@student.matnat.uio.no>
	<e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>
	<49918D94.20301@student.matnat.uio.no>
Message-ID: <e7ba66e40902100641j2220c856u2ddcbee9b82a8324@mail.gmail.com>

On Tue, Feb 10, 2009 at 12:22 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>>
> Ah. Yes, I noticed, but after commenting on the ticket. So apparently it
> is NOT a C++-only issue, but something much more strange going on ...
>

Indeed. It seems Cython 'memoizes' counters for a module context, then
if you compile twice, the counters are not reinitialized. Or somethig
like that....

You know, module or class level globals could be rally evil :-) ...

$ cd Cython/Compiler/
$ find . -type f -print0 | xargs -0 -e grep -nH -e
TemplateTransform.temp_name_counter
./ParseTreeTransforms.py:549:        TemplateTransform.temp_name_counter += 1
./ParseTreeTransforms.py:550:        handle = "__tmpvar_%d" %
TemplateTransform.temp_name_counter
./TreeFragment.py:122:            TemplateTransform.temp_name_counter += 1
./TreeFragment.py:123:            handle = "__tmpvar_%d" %
TemplateTransform.temp_name_counter

Is the .temp_name_counter counter ever decremented? Could be this the
root of the problem?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb 10 16:07:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 10 Feb 2009 16:07:41 +0100
Subject: [Cython] failure of withstat
In-Reply-To: <e7ba66e40902100641j2220c856u2ddcbee9b82a8324@mail.gmail.com>
References: <e7ba66e40902100536t41664872ia4fb576d728ee9fc@mail.gmail.com>	<32943.213.61.181.86.1234273786.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<49918714.7060809@student.matnat.uio.no>	<e7ba66e40902100610w71a7214aj56c2caef1e824fe7@mail.gmail.com>	<49918D94.20301@student.matnat.uio.no>
	<e7ba66e40902100641j2220c856u2ddcbee9b82a8324@mail.gmail.com>
Message-ID: <4991983D.90802@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Feb 10, 2009 at 12:22 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>   
>> Ah. Yes, I noticed, but after commenting on the ticket. So apparently it
>> is NOT a C++-only issue, but something much more strange going on ...
>>
>>     
>
> Indeed. It seems Cython 'memoizes' counters for a module context, then
> if you compile twice, the counters are not reinitialized. Or somethig
> like that....
>
> You know, module or class level globals could be rally evil :-) ...
>
> $ cd Cython/Compiler/
> $ find . -type f -print0 | xargs -0 -e grep -nH -e
> TemplateTransform.temp_name_counter
> ./ParseTreeTransforms.py:549:        TemplateTransform.temp_name_counter += 1
> ./ParseTreeTransforms.py:550:        handle = "__tmpvar_%d" %
> TemplateTransform.temp_name_counter
> ./TreeFragment.py:122:            TemplateTransform.temp_name_counter += 1
> ./TreeFragment.py:123:            handle = "__tmpvar_%d" %
> TemplateTransform.temp_name_counter
>
> Is the .temp_name_counter counter ever decremented? Could be this the
> root of the problem?
>   
I tried binding temp_name_counter to Main.Context (which is available as 
self.context in WithTransform), and initializing it to 0. It made no 
difference (and I haven't got around to commiting that).

Dag Sverre


From stefan_ml at behnel.de  Tue Feb 10 18:34:20 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Feb 2009 18:34:20 +0100
Subject: [Cython] PATCH: fix reference count problems in
 tryfinally	testcase, please review & comment
In-Reply-To: <e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>	<4990A5BD.1030707@student.matnat.uio.no>	<F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
	<e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>
Message-ID: <4991BA9C.8000303@behnel.de>

Hi,

Lisandro Dalcin wrote:
> I've pushed two additions to tryfinally testcase, both fail :-(

Please don't add failing tests to the normal test suite (unless you have
the fix ready for immediate commit). Instead, make sure there is a bug
report for it (#200 in this case) and add a test to tests/bugs/, named:
"the_test_name_Tnnn.pyx", "Tnnn" being the ticket number, i.e. "T200" in
your case. The runtests.py script then allows it to be run by passing the
"-T200" at the command line.

Stefan


From dalcinl at gmail.com  Tue Feb 10 18:49:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Feb 2009 15:49:37 -0200
Subject: [Cython] PATCH: fix reference count problems in tryfinally
	testcase, please review & comment
In-Reply-To: <4991BA9C.8000303@behnel.de>
References: <e7ba66e40902091239w4837cfa0g484111cbb2047357@mail.gmail.com>
	<4990A5BD.1030707@student.matnat.uio.no>
	<F30C1A8D-D7C6-4D81-A738-0256A7D65F59@math.washington.edu>
	<e7ba66e40902100527r5eb0f075vec9b160ba07efd60@mail.gmail.com>
	<4991BA9C.8000303@behnel.de>
Message-ID: <e7ba66e40902100949n6c58889en2a4402e80a8587cb@mail.gmail.com>

Stefan, the two additions I've made are trivial variations of the
stuff already failing in test/run/tryfinally.pyx . Once the former
failing test pass, my additions should pass for free...

What should I do then? Move all the failing tests (even the former
one) to bugs/test_name_Tnnn.pyx ??



On Tue, Feb 10, 2009 at 3:34 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Lisandro Dalcin wrote:
>> I've pushed two additions to tryfinally testcase, both fail :-(
>
> Please don't add failing tests to the normal test suite (unless you have
> the fix ready for immediate commit). Instead, make sure there is a bug
> report for it (#200 in this case) and add a test to tests/bugs/, named:
> "the_test_name_Tnnn.pyx", "Tnnn" being the ticket number, i.e. "T200" in
> your case. The runtests.py script then allows it to be run by passing the
> "-T200" at the command line.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From glenn at tarbox.org  Tue Feb 10 19:35:32 2009
From: glenn at tarbox.org (Glenn Tarbox, PhD)
Date: Tue, 10 Feb 2009 10:35:32 -0800
Subject: [Cython] IbPy Cross Post...
Message-ID: <f049b35c0902101035k31c8ed37tc72ec08d0dcf707c@mail.gmail.com>

IbPy is a Python interface to the Interactive Brokers API and something I've
been involved with for a while.

A piece of IbPy is a code generator which takes IB's Java client code and
generates Python.  I've had issues with performance... Hence the message
below..

http://groups.google.com/group/ibpy-discuss/msg/1bf318a75376c64f

and pasted here:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

So, I had this weird thought last night about IbPy... and the java to
python generator / compiler...

and something occurred to me.

Cython....

I've been playing around with Cython for a while.  Its pretty amazing
stuff really and the syntax is very much python-like with a few
exceptions... those exceptions are where the Cython compiler does
things to make python code fast (i.e. where you know things about the
data, you can take the load off the Python interpreter and generate
native code)

The speedup can be dramatic... like 200x.  There are a number of
important Cython characteristics but its better to just read about it
(links below)

An important link first:  This to a talk William Stein gave to/at
Microsoft which has a little Cython example which gets executed in
real-time (i.e. C/C++ code gets generated and linked into python
dynamically through the sage workbook which is browser based).  Of
course, just like generating .pyc files, once this happens in real
life, the .dll or .so files are there so just need to be dynamically
linked at runtime unless there is a source code change.

If you're not a math person, just scroll down to the Cython example

http://sagenb.org/home/pub/202/

Note that if you go ahead and get an account on the Sage machines,
you'll be able to run this all yourself... live. (for the more
adventurous, building Sage is amazingly easy and it is completely self
contained... I could expand greatly but those so inclined should check
out Sage (links below)

But, thats not my point.

My point is that since IbPy uses the J2py compiler (called something
like that) its probably "relatively" easy to generate a little extra
python code in key areas to make it Cythonic (couplea type
specifications)... which will make it operate at native code speeds
yet still be completely python at the user level... just that instead
of the library being interpreted, its native...

no diff to the user at all...

and I need to be clear clear cuz it might not be obvious at first
glance.  I'm NOT saying the user interface is Cython, just that the
underlying IB handling library is Cython.  The user still codes in
Python (unless you want to use Cython which is another story
altogether and might make a lot of sense for those of us who run into
compute problems... this is sort of like using cPickle instead of
Pickle.  Same interface, just runs way faster cuz its compiled C code
instead of Python)

This effort might be amost a freebie given the existence of the code
generator used for IbPy.  The java code parsed to generate IbPy has
the information necessary to spew the few key type definitions where
IbPy spends all its cycles (Java is strongly typed), in particular,
the inner loops which do the comms handling and object generation.

Cython homepage: http://www.cython.org/

Sage homepage: http://www.sagemath.org/

-glenn

On Jan 30, 2:22 am, glenn
<gl...<http://groups.google.com/groups/unlock?_done=/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100%3Fshow_docid%3D7b027b93b5472100&msg=1bf318a75376c64f>
@tarbox.org> wrote:

- Hide quoted text -
- Show quoted text -
<http://groups.google.com/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100?hide_quotes=no#msg_1bf318a75376c64f>
> On Jan 29, 4:55 pm, Troy Melhase <troy.melh...<http://groups.google.com/groups/unlock?_done=/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100%3Fshow_docid%3D7b027b93b5472100&msg=1bf318a75376c64f>
@gmail.com> wrote:

> > I once did a bit of looking.  There are python interfaces of various

> > > maturities although I think I remember more about interfaces in Java
> > > and possibly CORBA (not something with which one dabbles in lightly,
> > > particularly with Python)
> > ...
> > > Therefore, you build python remote access to an IbPy process which can

> > > be accessed both from your gnumeric python and your PyQt process over
> > > the wire (pickle is your friend here).

> > What I'm hearing from this thread is that the choice for RPC between
> > an arbitrary process (gnumeric, OO) and an IbPy process (your script,
> > gui, app) is either non-existent or is substandard.  And you really
> > shouldn't have to make a choice, there should be a solid, thought-thru
> > solution for you.

> Wouldn't it be wonderful if everyone thought like you. :-)

> > Every time I've had to construct my own RPC mechanism, I've used a
> > small web server.  The benefits typically outweigh the downsides
> > (stateless nature) of the protocol.  So I think I'll code up an "ibpy
> > server" and an "ibpy command-line client" for the 0.7.7 release.

> I'll write some below, but if you were going to do this, I think it
> would be useful to take a step back and think broadly about the
> architecture.

> If you're gonna expose the interface "over the wire", the server side
> language can be independent of the client if you're willing to avoid
> python specific mechanisms (pickle)

> I had to move away from IbPy on the server side for performance
> reasons... when you pull depth and a bunch of instruments, throw some
> database stuff and maybe an analysis op or two and you quickly hit
> performance limits.

> In my case, I went all the way: I implemented an interface to a Java
> server using Google Protocol Buffers.  The reasons are many and its
> arguable whether its the right choice for an IB only system, but Java,
> C++ and Python are supported natively.

> Unfortunately, another problem quickly became apparent:  Protocol
> buffers are slow in python due to their implementation (exploit a lot
> of reflection... makes it elegant and all, but brings the python up
> again).  So, its great for lots of python apps, but not data at any
> real volume.

> However, my system is quite a bit more performance intensive than most
> need... particularly since I got a live tick feed which requires Java
> or C++ anyways.

> > The server would expose URLs like this:

> >     /ib/cancelOrder/<orderid>
> >     /ib/reqMktData/<tickid>
> >     /ib/reqHistoricalData/<tickid>

> > Those above would be available via HTTP POST, and other required
> > params would have to be sent as part of the request (e.g., in
> > reqMktData we can't have
> > /<tickid>/<contractfield1>/<contractfield2>/<etc>, so we collapse
> > those and only require the key (ticker id) as part of the URL).

> > These calls would be available via HTTP GET:

> >     /ib/order/<orderid>
> >     /ib/marketdata/<tickerid>
> >     /ib/histdata/<tickid>

> > In the case of these, they are data-returning calls, meaning that
> > clients can make requests to fetch the values whenever they like.
> > These calls would return data encoded as CSV, RSS, or JSON.

> So, the proposal is some kind of REST web services interface.  I think
> this is a great idea...

> But, then we're into why is the server in Python?

> Believe me, I do every bit of coding in Python I can (which includes
> Numpy, Matplotlib, Sage) but Python has performance issues.  So, if
> you're gonna build a server using "web stuff", I'm guessing that a
> rethink might be in order.

> It would likely be possible to generate a Python client side interface
> which works similarly to your current "Pythonic" API but with
> enhancements to support synchronous clients (i.e. no callbacks).  Of
> course, the design issues here can get tricky as there are results and
> the client may choose to poll which means the server side needs to
> queue responses. Not nasty technically, but the use case explosion can
> sometimes bite you.

> > As for the command line client, it would operate in one of two modes:
> > as an IbPy HTTP client or as an TWS library client.  In both cases,
> > the interface would be the same, and it would certainly allow for
> > integration with any program that can start other processes.
> > Examples:

> >     $ runibpy --clientid=1 --host=myserver --command=reqMktData
> > --tickerid=832 --rows=50
> >     $ runibpy --http --host=myserver:8080 --command=reqMktData ...

> > The only question here is how to deal with compound data structures
> > like the contract and order objects.

> you betcha... this is where I spent a great deal of time hacking.  It
> was more typing really as I took the IB java code, commented it all
> out, and typed the Protocol Buffer interface definitions kinda in-
> line.  It took a few days but there are constructs in protobuf which
> help with the more complex structures.

> Of course, one could also define an XSD, the schema isn't hugely
> nasty, but now people who use IbPy because its simple and Python are
> wrestling with a different kind of headache.... wrestling with XML can
> be kinda nasty although there are certainly good tools / libraries
> etc.

> > > In previous posts, there are references to the Twisted IbPy
> > > implementation I wrote.  Also, I built / maintain the twisted Qt
> > > reactor.  If you're going to do asynchronous processing, I highly
> > > recommend twisted although its a non-trivial elephant to swallow...
> > > IMHO, its worth it as is Qt (thanks Troy)

> > Does twisted have some kind of WSGI interface or support?  I'd be
> > willing to try that over the basic HTTP servers in the standard
> > library.

> Twisted has a full HTTP server implementation and would also be
> helpful in that it can easily handle multiple clients and IB without
> requiring threads or all the nastiness which comes with it.  Its also
> pretty high performance as its architected well and all... but its a
> non-trivial beast to swallow... as is asynchronous programming in
> general.  I highly recommend it, but I'm not sure you should take my
> recommendation :-)

> > Thoughts?

> As an aside, one of the advantages of the approach I took with a java
> based server, is that I can connect to multiple IB clients all pumping
> a bunch of data without issue.  This is, again, not the kind of
> problem one often finds "in the wild", but it helped me out quite a
> bit.

> Of course, I eventually gave up, wrote a check, and am getting every
> tick for every exchange I trade from another vendor (NxCore).  Next
> step is to go FIX...

> -glenn
> > troy

-- 
*** Note: New Number
Glenn H. Tarbox, PhD ||  206-274-6919
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090210/a9ce77a8/attachment-0001.htm 

From jasone at canonware.com  Wed Feb 11 20:32:31 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 11:32:31 -0800
Subject: [Cython] size_t problem
Message-ID: <499327CF.9040705@canonware.com>

I have a class declaration that looks like:

cdef class DistMatrix:
     # ...
     cdef readonly size_t ntaxa

With the recent size_t-related changes, I now get the following 
compilation error:

	C attribute of type 'size_t' cannot be accessed from Python

I hope this isn't an intentional change... =)

Jason

From jasone at canonware.com  Wed Feb 11 20:41:16 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 11:41:16 -0800
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <499329DC.6040205@canonware.com>

Jason Evans wrote:
> I have a class declaration that looks like:
> 
> cdef class DistMatrix:
>      # ...
>      cdef readonly size_t ntaxa
> 
> With the recent size_t-related changes, I now get the following 
> compilation error:
> 
> 	C attribute of type 'size_t' cannot be accessed from Python

Here's another problem:

     cpdef float distanceGet(self, size_t x, size_t y):
                                            ^
------------------------------------------------------------

/home/jasone/crux/crux/build/run/pkg/Crux/DistMatrix/__init__.pyx:359:44: 
Cannot convert Python object argument to type 'size_t' (when parsing 
input arguments)

Basically, size_t appears to have become an unusable type that is 
relegated to some sort of type purgatory.

Jason

From dagss at student.matnat.uio.no  Wed Feb 11 20:58:47 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 11 Feb 2009 20:58:47 +0100 (CET)
Subject: [Cython] size_t problem
In-Reply-To: <499329DC.6040205@canonware.com>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
Message-ID: <ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>

Jason Evans wrote:
> Jason Evans wrote:
>> I have a class declaration that looks like:
>>
>> cdef class DistMatrix:
>>      # ...
>>      cdef readonly size_t ntaxa
>>
>> With the recent size_t-related changes, I now get the following
>> compilation error:
>>
>> 	C attribute of type 'size_t' cannot be accessed from Python
>
> Here's another problem:
>
>      cpdef float distanceGet(self, size_t x, size_t y):
>                                             ^
> ------------------------------------------------------------
>
> /home/jasone/crux/crux/build/run/pkg/Crux/DistMatrix/__init__.pyx:359:44:
> Cannot convert Python object argument to type 'size_t' (when parsing
> input arguments)
>
> Basically, size_t appears to have become an unusable type that is
> relegated to some sort of type purgatory.

At this stage my vote is for rolling back the size_t changes and file a
ticket for 0.11.1 instead.

Dag Sverre


From dalcinl at gmail.com  Wed Feb 11 21:35:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 18:35:39 -0200
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>

It was not intentional, but indeed this is spotting a portability
problem on your code in previous Cython versions :-).

I'll try to fix this...

On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
> I have a class declaration that looks like:
>
> cdef class DistMatrix:
>     # ...
>     cdef readonly size_t ntaxa
>
> With the recent size_t-related changes, I now get the following
> compilation error:
>
>        C attribute of type 'size_t' cannot be accessed from Python
>
> I hope this isn't an intentional change... =)
>
> Jason
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Feb 11 21:37:56 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 12:37:56 -0800
Subject: [Cython] size_t problem
In-Reply-To: <ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
	<ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
Message-ID: <919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>

On Feb 11, 2009, at 11:58 AM, Dag Sverre Seljebotn wrote:

> Jason Evans wrote:
>> Jason Evans wrote:
>>> I have a class declaration that looks like:
>>>
>>> cdef class DistMatrix:
>>>      # ...
>>>      cdef readonly size_t ntaxa
>>>
>>> With the recent size_t-related changes, I now get the following
>>> compilation error:
>>>
>>> 	C attribute of type 'size_t' cannot be accessed from Python
>>
>> Here's another problem:
>>
>>      cpdef float distanceGet(self, size_t x, size_t y):
>>                                             ^
>> ------------------------------------------------------------
>>
>> /home/jasone/crux/crux/build/run/pkg/Crux/DistMatrix/__init__.pyx: 
>> 359:44:
>> Cannot convert Python object argument to type 'size_t' (when parsing
>> input arguments)
>>
>> Basically, size_t appears to have become an unusable type that is
>> relegated to some sort of type purgatory.
>
> At this stage my vote is for rolling back the size_t changes and  
> file a
> ticket for 0.11.1 instead.

I think the fix should be simple (I'll look at it later today, I'm  
off to catch the bus) but if not, lets roll it back.

- Robert



From dalcinl at gmail.com  Wed Feb 11 21:47:16 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 18:47:16 -0200
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>

Please also note that the code below also fails to cythonize. I bet
this gotcha is not related to my latest patches for 'size_t'

cdef class A:
   cdef readonly Py_ssize_t n


On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
> I have a class declaration that looks like:
>
> cdef class DistMatrix:
>     # ...
>     cdef readonly size_t ntaxa
>
> With the recent size_t-related changes, I now get the following
> compilation error:
>
>        C attribute of type 'size_t' cannot be accessed from Python
>
> I hope this isn't an intentional change... =)
>
> Jason
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Feb 11 22:20:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 13:20:24 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
Message-ID: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>

On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:

> Please also note that the code below also fails to cythonize. I bet
> this gotcha is not related to my latest patches for 'size_t'
>
>
> cdef class A:
>    cdef readonly Py_ssize_t n

Well, yes and no. People used to be able to write "ctypedef unsigned  
long size_t" and have it work.

I pushed a fix and some tests. The public attribute stuff is slightly  
hacky, but I'm going to be revising that to handle complex types  
anyways.

BTW, I didn't see ulong <= size_t <= ulonglong ordering in PyrexTypes  
yet, did we get consensus on this?

- Robert


From michael.abshoff at googlemail.com  Wed Feb 11 22:32:13 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 11 Feb 2009 13:32:13 -0800
Subject: [Cython] size_t problem
In-Reply-To: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
Message-ID: <499343DD.4020609@gmail.com>

Robert Bradshaw wrote:
> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
> 
>> Please also note that the code below also fails to cythonize. I bet
>> this gotcha is not related to my latest patches for 'size_t'
>>
>>
>> cdef class A:
>>    cdef readonly Py_ssize_t n
> 
> Well, yes and no. People used to be able to write "ctypedef unsigned  
> long size_t" and have it work.

Well, not to be a pedantic nit picker, but as we all should know now the 
above is not correct. It tends to work on most systems, but enabling 
people to do dumb things should not have priority over correct code. 
Sage has the same issue that someone defined size_t to be unsigned long 
and I could look up who did that, but I have a good idea who wrote that 
code, so no need to point fingers :)

Maybe the 0.11 release notes should prominently feature this change and 
show how to fix the problem? This might also be something where Pyrex 
and Cython have diverged, i.e. I assume the above is "legal" in Pyrex 
while it won't work in Cython 0.11 or later, so in that case it should 
be added to the wiki page in the Cython wiki where differences are listed.

> I pushed a fix and some tests. The public attribute stuff is slightly  
> hacky, but I'm going to be revising that to handle complex types  
> anyways.
> 
> BTW, I didn't see ulong <= size_t <= ulonglong ordering in PyrexTypes  
> yet, did we get consensus on this?
> 
> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dalcinl at gmail.com  Wed Feb 11 22:44:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 19:44:22 -0200
Subject: [Cython] size_t problem
In-Reply-To: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
Message-ID: <e7ba66e40902111344g3e74054fubeb4166ed7c00562@mail.gmail.com>

On Wed, Feb 11, 2009 at 7:20 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
>
> BTW, I didn't see ulong <= size_t <= ulonglong ordering in PyrexTypes
> yet, did we get consensus on this?
>

I have the patch ready to go mainstream, plus some hackery to make
Cython fail compilation for nonsense 'unsigned Py_ssize_t' or 'signed
Py_ssize_t' (the same for 'size_t').

Can I push this?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 11 22:50:58 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 19:50:58 -0200
Subject: [Cython] size_t problem
In-Reply-To: <499327CF.9040705@canonware.com>
References: <499327CF.9040705@canonware.com>
Message-ID: <e7ba66e40902111350j704fb629k3c7d8ae45641d0e6@mail.gmail.com>

On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
> I have a class declaration that looks like:
>
> cdef class DistMatrix:
>     # ...
>     cdef readonly size_t ntaxa
>

I'm now working on a patch to transform that in a property. However,
this should be done with care, regarding the difference between data
and non-data descriptors,
http://users.rcn.com/python/download/Descriptor.htm#id3 . I believe
the __set__ must be generated and it should raise AtributeError,
'can't set attribute'  (like builtin 'property' does when no 'set'
function is passed)

Comments?





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 11 22:59:16 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Feb 2009 19:59:16 -0200
Subject: [Cython] size_t problem
In-Reply-To: <919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
	<ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
	<919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>
Message-ID: <e7ba66e40902111359o158326c5sde4aa524eb015ef1@mail.gmail.com>

On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> I think the fix should be simple (I'll look at it later today, I'm
> off to catch the bus) but if not, lets roll it back.
>

Robert, IMHO, the below code is wrong...

c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) ==
sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")

what would happend if size_t is unsigned int, and sizeof(unsigend int)
< sizeof(unsigend long). You will end-up returning garbage...

As I said in previous mail, I believe the only sane way to fix this is
to use a property...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Feb 12 00:51:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 12 Feb 2009 00:51:00 +0100
Subject: [Cython] size_t problem
Message-ID: <3317244679.737037@smtp.netcom.no>

What about two nested ?-operators?

Dag Sverre Seljebotn
-----Original Message-----
From: Lisandro Dalcin <dalcinl at gmail.com>
Date: Wednesday, Feb 11, 2009 10:59 pm
Subject: Re: [Cython] size_t problem
To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net

On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
><robertwb at math.washington.edu> wrote:
>>
>> I think the fix should be simple (I'll look at it later today, I'm
>> off to catch the bus) but if not, lets roll it back.
>>
>
>Robert, IMHO, the below code is wrong...
>
>c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) =sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")
>
>what would happend if size_t is unsigned int, and sizeof(unsigend int)
>< sizeof(unsigend long). You will end-up returning garbage...
>
>As I said in previous mail, I believe the only sane way to fix this is
>to use a property...
>
>
>
>-- 
>Lisandro Dalc?n
>---------------
>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>Tel/Fax: +54-(0)342-451.1594
>_______________________________________________
>Cython-dev mailing list
>Cython-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/cython-dev
>


From jasone at canonware.com  Thu Feb 12 01:13:38 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 16:13:38 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
Message-ID: <499369B2.1040004@canonware.com>

Lisandro Dalcin wrote:
> [...] this is spotting a portability
> problem on your code in previous Cython versions :-).
> 
> On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
>> I have a class declaration that looks like:
>>
>> cdef class DistMatrix:
>>     # ...
>>     cdef readonly size_t ntaxa

I don't understand what portability problem you're referring to. 
Assuming that size_t was defined to be unsigned long or unsigned long 
long (depending on the platform), why would there be a problem?

Thanks,
Jason

P.S.  The changes Robert pushed today allow my code to build and run 
again.  Thanks Robert!

From michael.abshoff at googlemail.com  Thu Feb 12 01:46:07 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 11 Feb 2009 16:46:07 -0800
Subject: [Cython] size_t problem
In-Reply-To: <499369B2.1040004@canonware.com>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
	<499369B2.1040004@canonware.com>
Message-ID: <4993714F.7050903@gmail.com>

Jason Evans wrote:
> Lisandro Dalcin wrote:
>> [...] this is spotting a portability
>> problem on your code in previous Cython versions :-).
>>
>> On Wed, Feb 11, 2009 at 5:32 PM, Jason Evans <jasone at canonware.com> wrote:
>>> I have a class declaration that looks like:
>>>
>>> cdef class DistMatrix:
>>>     # ...
>>>     cdef readonly size_t ntaxa
> 
> I don't understand what portability problem you're referring to. 
> Assuming that size_t was defined to be unsigned long or unsigned long 
> long (depending on the platform), why would there be a problem?

Code created by Cython on one platform ought to work on another 
platform. And since Python.h provides that info about the size of size_t 
for you why bother to implement that independently?

> Thanks,
> Jason

Cheers,

Michael

> P.S.  The changes Robert pushed today allow my code to build and run 
> again.  Thanks Robert!
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From jasone at canonware.com  Thu Feb 12 02:17:12 2009
From: jasone at canonware.com (Jason Evans)
Date: Wed, 11 Feb 2009 17:17:12 -0800
Subject: [Cython] size_t problem
In-Reply-To: <4993714F.7050903@gmail.com>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>	<499369B2.1040004@canonware.com>
	<4993714F.7050903@gmail.com>
Message-ID: <49937898.9000903@canonware.com>

Michael Abshoff wrote:
> Jason Evans wrote:
>> I don't understand what portability problem you're referring to. 
>> Assuming that size_t was defined to be unsigned long or unsigned long 
>> long (depending on the platform), why would there be a problem?
> 
> Code created by Cython on one platform ought to work on another 
> platform. And since Python.h provides that info about the size of size_t 
> for you why bother to implement that independently?

Ah, okay.  I had to give up on the portability of Cython-generated code 
in order to add the ability to conditionally support mpi4py, so I 
haven't really been thinking in terms of portable Cython output lately. 
  Thanks for the explanation.

Jason

From robertwb at math.washington.edu  Thu Feb 12 03:44:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 18:44:19 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902111359o158326c5sde4aa524eb015ef1@mail.gmail.com>
References: <499327CF.9040705@canonware.com> <499329DC.6040205@canonware.com>
	<ec16a6a819a737eb09702309dbc09555.squirrel@webmail.uio.no>
	<919CC156-1E46-4EB5-BBA4-181644B7B086@math.washington.edu>
	<e7ba66e40902111359o158326c5sde4aa524eb015ef1@mail.gmail.com>
Message-ID: <01280D2A-712B-4677-BC6B-07AE7704D166@math.washington.edu>

On Feb 11, 2009, at 1:59 PM, Lisandro Dalcin wrote:

> On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> I think the fix should be simple (I'll look at it later today, I'm
>> off to catch the bus) but if not, lets roll it back.
>>
>
> Robert, IMHO, the below code is wrong...
>
> c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) ==
> sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")
>
> what would happend if size_t is unsigned int, and sizeof(unsigend int)
> < sizeof(unsigend long). You will end-up returning garbage...

Yes, this is the part that I said was a bit hacky, but I am almost  
sure that sizeof(size_t) = sizeof(int) < sizeof(long) is *really*  
rare. It would also break if, say, sizeof(long) < sizeof(size_t) <  
sizeof(long long) on some weird system with, say, a 43-bit size_t.

I actually considered doing a nested if statement, but decided it  
wasn't worth it for now--if anyone on this list is using such a  
system speak up :).

> As I said in previous mail, I believe the only sane way to fix this is
> to use a property...

This was exactly what I was thinking of doing, as well as some other  
changes/simplifications in there, as part of adapting complex numbers  
into the types framework. That's why I figured it wasn't worth  
putting lots of effort into it now, especially as I'd really like to  
push out a release soon. On that note, I've put up a temporary http:// 
hg.cython.org/cython-0.11/ which is for bugfixes only.

- Robert


From lodatom at gmail.com  Thu Feb 12 03:45:30 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Wed, 11 Feb 2009 21:45:30 -0500
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>
References: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>
Message-ID: <ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>

I wrote a small library for creating Numpy ufuncs using Cython.  Just
copy the pxd and pxi linked at
http://wiki.cython.org/MarkLodato/CreatingUfuncs and include the pxi
at the top of your file.  Then you can create 1-input/1-output or
2-input/1-output ufuncs operating of floats, doubles, or long doubles
easily.

For those unfamiliar with ufuncs, these are functions that operate on
NumPy arrays - for example, all of the mathematical operators of NumPy
(add, subtract, multiply, exp, etc) are ufuncs.  They are nice for the
user since they handle broadcasting and all the other stuff you expect
of NumPy.  They are also nice for the writer of the function, since
one needs only write a simple 1-D loop (or even a function that takes
scalars as input and output) and NumPy takes care of the rest.  That
is, the author need not worry about multi-dimensional arrays, error
handling, broadcasting rules, output buffers, etc.  For more details,
see http://docs.scipy.org/doc/numpy/reference/ufuncs.html.

Although NumPy makes it easy to write a ufunc, there's still a fair
amount of boilerplate.  To make the process simpler, I created this
library to do most of the work for you.  Just define a function that
takes a scalar (or two) as input and returns a scalar, then register
the function using register_ufunc_d() (or similar).  The wiki page
shows how to create an "lgamma" ufunc.  Here's another simple example:

{{{
include "numpy_ufuncs.pxi"

cdef double foo_dd(double a, double b):
   return (a-b) / (a+b)

foo = register_ufunc_dd(foo_dd, "foo", "returns (x1-x2)/(x1+x2)", PyUFunc_Zero)
}}}

Now in Python you can do:

{{{
In [1]: from foo import foo

In [2]: import numpy as np

In [3]: a = np.arange(16).reshape(4,4)

In [4]: b = np.arange(30,34)

In [5]: foo(a,b)
Out[5]:
array([[-1.        , -0.9375    , -0.88235294, -0.83333333],
      [-0.76470588, -0.72222222, -0.68421053, -0.65      ],
      [-0.57894737, -0.55      , -0.52380952, -0.5       ],
      [-0.42857143, -0.40909091, -0.39130435, -0.375     ]])

In [6]: foo(32, 16)
Out[6]: 0.33333333333333331
}}}

The great thing is that it's easy to write, easy to use, and fast!

I would like to add more common cases (particularly integer) and more
generic cases.  The NumPy C API only has 1-D loops for floating point
(e.g. PyUFunc_d_d), so I would have to write this myself.  For now
this fits my needs.

I'd be happy to hear any comments or suggestions you have.  Perhaps
this can be distributed with Cython?  It would be really nice to be
able to easily create ufuncs in Cython right out of the box.

~Mark Lodato

From greg.ewing at canterbury.ac.nz  Thu Feb 12 04:27:58 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 12 Feb 2009 16:27:58 +1300
Subject: [Cython] size_t problem
In-Reply-To: <69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
Message-ID: <4993973E.2030205@canterbury.ac.nz>

Robert Bradshaw wrote:
> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
> 
>>cdef class A:
>>   cdef readonly Py_ssize_t n
> 
> People used to be able to write "ctypedef unsigned  
> long size_t" and have it work.

Only if size_t is actually the size of an unsigned
long on the platform concerned, though. Otherwise
the Python runtime's idea of the struct layout won't
match what is actually there.

-- 
Greg

From robertwb at math.washington.edu  Thu Feb 12 04:35:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Feb 2009 19:35:08 -0800
Subject: [Cython] size_t problem
In-Reply-To: <4993973E.2030205@canterbury.ac.nz>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111247o7ab66f86jc975b1fadc46d461@mail.gmail.com>
	<69C88649-2820-4178-A340-B96970845CD1@math.washington.edu>
	<4993973E.2030205@canterbury.ac.nz>
Message-ID: <7289B038-B4A4-4E13-8CF5-6E57AB3B631A@math.washington.edu>

On Feb 11, 2009, at 7:27 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>> On Feb 11, 2009, at 12:47 PM, Lisandro Dalcin wrote:
>>
>>> cdef class A:
>>>   cdef readonly Py_ssize_t n
>>
>> People used to be able to write "ctypedef unsigned
>> long size_t" and have it work.
>
> Only if size_t is actually the size of an unsigned
> long on the platform concerned, though. Otherwise
> the Python runtime's idea of the struct layout won't
> match what is actually there.

What I meant was to use size_t one would write

cdef extern from *:
     ctypedef unsigned long size_t

which would work perfect as long as sizeof(unsigned long) == sizeof 
(size_t), and would also usually work fine if the sizes were  
different (if one never needed to pass to and from Python-land with a  
value large enough to care (though of course this is a bug waiting to  
happen)). Now that it's a builtin type, it broke the user being able  
to declare it, and we had to make sure to continue to support all use- 
cases.

- Robert


From dagss at student.matnat.uio.no  Thu Feb 12 10:51:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 10:51:25 +0100
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>
References: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>
	<ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>
Message-ID: <4993F11D.2050108@student.matnat.uio.no>

Mark Lodato wrote:
> I wrote a small library for creating Numpy ufuncs using Cython.  Just
> copy the pxd and pxi linked at
> http://wiki.cython.org/MarkLodato/CreatingUfuncs and include the pxi
> at the top of your file.  Then you can create 1-input/1-output or
> 2-input/1-output ufuncs operating of floats, doubles, or long doubles
> easily.
>
>   
Really nice! Excuse my ignorance about ufuncs, but is there a function 
pointer call overhead for every element this way, or is it all macro based?
> I'd be happy to hear any comments or suggestions you have.  Perhaps
> this can be distributed with Cython?  It would be really nice to be
> able to easily create ufuncs in Cython right out of the box.
>   
Definitely. It could be part of numpy.pxd I think; either that, or make 
a pxd package and have numpy.ufunc as a sub-pxd.

I wouldn't like to ship it as "pxi" (because include is kind of 
discouraged these days and it is not good if the Cython standard lib 
requires it). I think all the functions can be put in a pxd if you mark 
them "inline". (An alternative is to create a numpy_cython_ufuncs module 
in Cython/Runtime, but not unless we have to).

It would be best if you would play around in Cython/Includes and submit 
a patch.

Dag Sverre


From dagss at student.matnat.uio.no  Thu Feb 12 10:53:34 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 12 Feb 2009 10:53:34 +0100
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <4993F11D.2050108@student.matnat.uio.no>
References: <ca433830902111841m2aae741bh8631163594573b22@mail.gmail.com>	<ca433830902111845j13038727y35c8c223f6f0c5d8@mail.gmail.com>
	<4993F11D.2050108@student.matnat.uio.no>
Message-ID: <4993F19E.4050809@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Mark Lodato wrote:
>   
>> I wrote a small library for creating Numpy ufuncs using Cython.  Just
>> copy the pxd and pxi linked at
>> http://wiki.cython.org/MarkLodato/CreatingUfuncs and include the pxi
>> at the top of your file.  Then you can create 1-input/1-output or
>> 2-input/1-output ufuncs operating of floats, doubles, or long doubles
>> easily.
>>
>>   
>>     
> Really nice! Excuse my ignorance about ufuncs, but is there a function 
> pointer call overhead for every element this way, or is it all macro based?
>   
Also feel free to link to your wiki page from tutorials/numpy on the wiki.

Dag Sverre

From dalcinl at gmail.com  Thu Feb 12 14:55:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 11:55:44 -0200
Subject: [Cython] size_t problem
In-Reply-To: <49937898.9000903@canonware.com>
References: <499327CF.9040705@canonware.com>
	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>
	<499369B2.1040004@canonware.com> <4993714F.7050903@gmail.com>
	<49937898.9000903@canonware.com>
Message-ID: <e7ba66e40902120555t19e64155v3980bab1bb076e7e@mail.gmail.com>

On Wed, Feb 11, 2009 at 11:17 PM, Jason Evans <jasone at canonware.com> wrote:
> Michael Abshoff wrote:
>
> Ah, okay.  I had to give up on the portability of Cython-generated code
> in order to add the ability to conditionally support mpi4py, so I
> haven't really been thinking in terms of portable Cython output lately.
>  Thanks for the explanation.
>

I do not understand you here. Why is mpi4py a source of problems?
mpi4py should work the same on all platforms. Well, we have MPI_Aint
and MPI_Offset, I'm currently using typedefs, but this is an approach
I have to review and fix ASAP.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Feb 12 14:58:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 11:58:37 -0200
Subject: [Cython] size_t problem
In-Reply-To: <3317244679.737037@smtp.netcom.no>
References: <3317244679.737037@smtp.netcom.no>
Message-ID: <e7ba66e40902120558x348fe6fdy5cadaaf4df4c797@mail.gmail.com>

Yes, too nested ?: operators should work. We have to contemplate at
least on the three cases size_t in (uint, ulong, ulonglong).

 I'll work on it and push a patch.

On Wed, Feb 11, 2009 at 9:51 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> What about two nested ?-operators?
>
> Dag Sverre Seljebotn
> -----Original Message-----
> From: Lisandro Dalcin <dalcinl at gmail.com>
> Date: Wednesday, Feb 11, 2009 10:59 pm
> Subject: Re: [Cython] size_t problem
> To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net
>
> On Wed, Feb 11, 2009 at 6:37 PM, Robert Bradshaw
>><robertwb at math.washington.edu> wrote:
>>>
>>> I think the fix should be simple (I'll look at it later today, I'm
>>> off to catch the bus) but if not, lets roll it back.
>>>
>>
>>Robert, IMHO, the below code is wrong...
>>
>>c_size_t_type =      CSizeTType(6, 1, "(sizeof(size_t) =sizeof(unsigned long) ? T_ULONG : T_ULONGLONG)")
>>
>>what would happend if size_t is unsigned int, and sizeof(unsigend int)
>>< sizeof(unsigend long). You will end-up returning garbage...
>>
>>As I said in previous mail, I believe the only sane way to fix this is
>>to use a property...
>>
>>
>>
>>--
>>Lisandro Dalc?n
>>---------------
>>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>Tel/Fax: +54-(0)342-451.1594
>>_______________________________________________
>>Cython-dev mailing list
>>Cython-dev at codespeak.net
>>http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan at sun.ac.za  Thu Feb 12 17:40:30 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 12 Feb 2009 18:40:30 +0200
Subject: [Cython] Incomplete C++ example
In-Reply-To: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>
Message-ID: <9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>

Hi all,

I'm helping a student to wrap some C++ code using Cython, and noticed
that the example in the documentation is incomplete.  A header file,
Rectangle.h, is provided, but no implementation is given, so the
example cannot be compiled.

It would be great if you could add an implementation (see attached)
and modify the setup.py accordingly, i.e.

   Extension(
   "rectangle",
   ["rectangle.pyx", "Rectangle.cc"],
   language="c++")

Cheers
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Rectangle.cc
Type: text/x-c++src
Size: 503 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090212/a419c7ed/attachment.cc 

From jasone at canonware.com  Thu Feb 12 17:47:02 2009
From: jasone at canonware.com (Jason Evans)
Date: Thu, 12 Feb 2009 08:47:02 -0800
Subject: [Cython] size_t problem
In-Reply-To: <e7ba66e40902120555t19e64155v3980bab1bb076e7e@mail.gmail.com>
References: <499327CF.9040705@canonware.com>	<e7ba66e40902111235s37ca2405n76a5be1dbdf47df@mail.gmail.com>	<499369B2.1040004@canonware.com>
	<4993714F.7050903@gmail.com>	<49937898.9000903@canonware.com>
	<e7ba66e40902120555t19e64155v3980bab1bb076e7e@mail.gmail.com>
Message-ID: <49945286.40103@canonware.com>

Lisandro Dalcin wrote:
> On Wed, Feb 11, 2009 at 11:17 PM, Jason Evans <jasone at canonware.com> wrote:
>> Ah, okay.  I had to give up on the portability of Cython-generated code
>> in order to add the ability to conditionally support mpi4py, so I
>> haven't really been thinking in terms of portable Cython output lately.
>>  Thanks for the explanation.
> 
> I do not understand you here. Why is mpi4py a source of problems?
> mpi4py should work the same on all platforms. Well, we have MPI_Aint
> and MPI_Offset, I'm currently using typedefs, but this is an approach
> I have to review and fix ASAP.

The issue is that I need to be able to compile my software with the 
mpi4py code disabled (because it isn't a reasonable expectation that my 
users will always have MPI installed).  There is no way I've found to 
make a cimport of a non-existent module safe, except to wrap it in an IF 
block.

So, mpi4py is only indirectly the source of my problems, because it's 
the only third party Cython module I'm using at the moment.  mpi4py 
looks like a great piece of software, and it has worked well for the 
limited testing I've done with it.

Jason

From dalcinl at gmail.com  Thu Feb 12 21:26:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 18:26:05 -0200
Subject: [Cython] more 'size_t' stuff pushed
Message-ID: <e7ba66e40902121226q531eedcbya2d8480f4b5224b2@mail.gmail.com>

After a lot of googling, diving on CPython sources (2.3 to 3.0) and
help from Albert Strasheim and Matthew Knepley, I've pushed more stuff
for 'size_t' (and 'Py_ssize_t') support. Now Cython should be near to
do the right thing, but I cannot be sure until all this is not tested
on Window 32 and Windows 64.

A thing (related to recent fix by Robert) that I was not able to fix
is the unused variable warning below...

size_t.c: In function '__pyx_pf_6size_t_1A___init__':
size_t.c:390: warning: unused variable '__pyx_arg_a'
size_t.cpp: In function 'int __pyx_pf_6size_t_1A___init__(PyObject*,
PyObject*, PyObject*)':
size_t.cpp:390: warning: unused variable '__pyx_arg_a'
tryfinally.c: In function '__pyx_pf_10tryfinally_try_return_temp':

Other missing task is extend Robert's fix to 'long double' ...

BTW, Cython generates this C code... Is this PY_LONG_LONG fallback
definition supposed to be portable?

#define PY_SSIZE_T_CLEAN
#include "Python.h"
#include "structmember.h"
#ifndef PY_LONG_LONG
  #define PY_LONG_LONG LONG_LONG
#endif


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Feb 12 21:42:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Feb 2009 18:42:15 -0200
Subject: [Cython] Adding T_SIZET to structmember.h
Message-ID: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>

After a rather long hacking on Cython in order to support 'Py_ssize_t'
and 'size_t' the right way, I would like to propose the inclusion of a
new T_SIZET in structmember.h in order to suport 'size_t' struct
fields with PyMemberDef. Would such addition be accepted for 2.7 and
3.1?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lodatom at gmail.com  Fri Feb 13 01:48:24 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Thu, 12 Feb 2009 19:48:24 -0500
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
Message-ID: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>

> Really nice! Excuse my ignorance about ufuncs, but is there a function
> pointer call overhead for every element this way, or is it all macro based

Unfortunately, there is a function call for every single element
because I used the PyUFunc_*_* family of wrappers.  For something like
lgamma(), this is perfectly acceptable since you would have to call
the function once per element anyway.  But for your own functions, it
would be nice to eliminate this overhead.  One way around this would
be for me to re-implement the PyUFunc_*_* as static functions, then
have the user declare the element-wise function as inline so that the
compiler does the inlining for us.  Another way around the overhead
would be to build the ufunc wrapping into Cython, perhaps as a
decorator or something.  This would make the user's job really simple,
but I'm not sure if it's worth the effort.

There is also the overhead of using the ufunc machinery, but I think
it's probably safe to say this overhead is minimal since all the NumPy
operators and functions are implemented as ufuncs.

> I wouldn't like to ship it as "pxi" (because include is kind of
> discouraged these days and it is not good if the Cython standard lib
> requires it). I think all the functions can be put in a pxd if you mark
> them "inline". (An alternative is to create a numpy_cython_ufuncs module
> in Cython/Runtime, but not unless we have to).

How can I implement this as a pxd?  I used pxi for two reasons: (1)
import_ufunc() needs to be called before anything is used, so by
including the pxi I can force this to be done automatically, and (2)
the ufunc API functions expect arrays that will stay in existence
forever.  I used globals since this is the easiest route, but you
can't define globals in a pxd.  I guess I can malloc and just create a
memory leak, but this seems sloppy.

Another way to implement all this would be to do it as C macros in a
header file.  The downside to this approach is that the user would
have to add another -I to their CFLAGS.  Is there any way to include
pure C code (well, really CPP macros) into the pxd?  Perhaps a better
solution would be to offer these macros as a patch to NumPy, rather
than Cython?

> It would be best if you would play around in Cython/Includes and submit
> a patch.

I plan to :)

~Mark

From greg.ewing at canterbury.ac.nz  Fri Feb 13 02:18:52 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 13 Feb 2009 14:18:52 +1300
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
References: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
Message-ID: <4994CA7C.2060704@canterbury.ac.nz>

Mark Lodato wrote:
> Another way around the overhead
> would be to build the ufunc wrapping into Cython, perhaps as a
> decorator or something.

I've never looked at the implementation of ufuncs, so this may
be completely nonsensical, but would it be possible to pass
shape/strides info into the function, so that it could do its
own looping over the data?

-- 
Greg


From lodatom at gmail.com  Fri Feb 13 02:44:57 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Thu, 12 Feb 2009 20:44:57 -0500
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <4994CA7C.2060704@canterbury.ac.nz>
References: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
	<4994CA7C.2060704@canterbury.ac.nz>
Message-ID: <ca433830902121744s6823f5b9i2b7a39bdb60a6f45@mail.gmail.com>

On Thu, Feb 12, 2009 at 8:18 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I've never looked at the implementation of ufuncs, so this may
> be completely nonsensical, but would it be possible to pass
> shape/strides info into the function, so that it could do its
> own looping over the data?

Good question.  Not only is that possible, but that's what NumPy
expects.  You define a function that takes one dimensional arrays and
outputs one dimensional arrays (iterating over the single dimension
yourself), and the ufunc machinery takes care of handling the
multi-dimensional aspects, broadcasting, types, error handling, etc.

To make things even easier, NumPy includes a set of functions, such as
PyUFunc_d_d(), which perform the one dimensional loop, calling a
supplied function pointer for each element.  For example, if you
define a function, "double foo(double)", you can register your ufunc
with PyUFunc_d_d() as the "function" and foo() as the "data".  This is
easier, but the downside is the extra function call overhead.

The other thing to note is that you can register multiple functions,
each handling different data types, for a single ufunc.  This allows
NumPy to perform the function directly without having to create a
temporary array of the correct type.  In the NumPy source, there is an
"add" function for every possible data type, while my lgamma example
from the wiki page only defines float, double, and long double
functions.

~Mark

From dagss at student.matnat.uio.no  Fri Feb 13 10:16:11 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 13 Feb 2009 10:16:11 +0100
Subject: [Cython] Fwd: Using Cython to create NumPy Ufuncs
In-Reply-To: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
References: <ca433830902121648l1eeff3eas3befe9c679db3d91@mail.gmail.com>
Message-ID: <49953A5B.2080303@student.matnat.uio.no>

Mark Lodato wrote:
> How can I implement this as a pxd?  I used pxi for two reasons: (1)
> import_ufunc() needs to be called before anything is used, so by

I'm wondering whether this can be done automatically by giving the pxd 
the right name so that Cython imports the right module. If you declare 
external Python extension types in the pxd, at least, this happens. But 
may not work in this case.

> including the pxi I can force this to be done automatically, and (2)
> the ufunc API functions expect arrays that will stay in existence
> forever.  I used globals since this is the easiest route, but you
> can't define globals in a pxd.  I guess I can malloc and just create a
> memory leak, but this seems sloppy.

Yeah, sorry, I overlooked that part when glancing at it. I say just keep 
using a pxi; one can always change it later if we come up with something.

So, a) try to integrate the pxd in a nice way, since it exposes NumPy 
API (i.e. put it either inside the numpy.pxd file, or under it in the 
package hierarchy), and b) also supply a cython_ufunc_utils.pxi (or 
similar). If you put a pxi file in Cython/Includes it will always be in 
the include path as well.

-- 
Dag Sverre

From dalcinl at gmail.com  Fri Feb 13 16:52:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 13 Feb 2009 13:52:43 -0200
Subject: [Cython] import/cimport gotcha
Message-ID: <e7ba66e40902130752j453000b5yc68c9d74c11f1f4e@mail.gmail.com>

Some time ago I reported that the following:

from foo import Bar
from foo cimport Bar

do not put 'Bar' in the module dicti. Commenting the lines below seems
to solve the issue. Am I missing something (I bet the answer is yes)?
Any comments?

diff -r 1771db44498b Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py	Thu Feb 12 18:22:15 2009 -0200
+++ b/Cython/Compiler/Nodes.py	Fri Feb 13 13:42:23 2009 -0200
@@ -4600,8 +4600,9 @@
                         break
             else:
                 entry =  env.lookup(target.name)
-                if entry.is_type and entry.type.name == name and
entry.type.module_name == self.module.module_name.value:
-                    continue # already cimported
+                #if entry.is_type and entry.type.name == name and
entry.type.module_name == self.module.module_name.value:
+                #    continue # already cimported
+                #    pass
                 self.interned_items.append(
                     (env.intern_identifier(name), target))
                 target.analyse_target_expression(env, None)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Feb 13 20:11:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 13 Feb 2009 17:11:02 -0200
Subject: [Cython] possible fix for withstat failure
Message-ID: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>

Please review, if it works, then feel free to reimplement the methods,
as with the current patch is a bit ugly (two calls to
try_substitution()), and then commit! Good weekend.

diff -r 1771db44498b Cython/Compiler/TreeFragment.py
--- a/Cython/Compiler/TreeFragment.py	Thu Feb 12 18:22:15 2009 -0200
+++ b/Cython/Compiler/TreeFragment.py	Fri Feb 13 17:06:00 2009 -0200
@@ -160,6 +160,7 @@
     def visit_NameNode(self, node):
         temphandle = self.tempmap.get(node.name)
         if temphandle:
+            node = self.try_substitution(node, node.name)
             node.name = temphandle
             return node
             # Replace name with temporary


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Fri Feb 13 20:54:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 13 Feb 2009 20:54:30 +0100
Subject: [Cython] possible fix for withstat failure
In-Reply-To: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>
References: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>
Message-ID: <4995CFF6.3030802@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Please review, if it works, then feel free to reimplement the methods,
> as with the current patch is a bit ugly (two calls to
> try_substitution()), and then commit! Good weekend.

I did this (well, it amounts to the same) some hours ago, see

http://hg.cython.org/cython-devel/rev/3304bc6fd2c3


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 13 20:54:47 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 13 Feb 2009 20:54:47 +0100
Subject: [Cython] possible fix for withstat failure
In-Reply-To: <4995CFF6.3030802@student.matnat.uio.no>
References: <e7ba66e40902131111l218a115di4b95bff20e4d1627@mail.gmail.com>
	<4995CFF6.3030802@student.matnat.uio.no>
Message-ID: <4995D007.2040608@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:
>> Please review, if it works, then feel free to reimplement the methods,
>> as with the current patch is a bit ugly (two calls to
>> try_substitution()), and then commit! Good weekend.
> 
> I did this (well, it amounts to the same) some hours ago, see
> 
> http://hg.cython.org/cython-devel/rev/3304bc6fd2c3
> 
> 

Thanks though! :-)

-- 
Dag Sverre

From greg.ewing at canterbury.ac.nz  Sat Feb 14 02:37:31 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 14 Feb 2009 14:37:31 +1300
Subject: [Cython] [Python-Dev] Adding T_SIZET to structmember.h
In-Reply-To: <499611E6.1020104@v.loewis.de>
References: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>
	<5c6f2a5d0902130715x4d4f4beet7d936f9c549a1335@mail.gmail.com>
	<e7ba66e40902130744s4a14c9a0n5dfed4b5905775d5@mail.gmail.com>
	<499611E6.1020104@v.loewis.de>
Message-ID: <4996205B.8090503@canterbury.ac.nz>

Martin v. L?wis wrote:
>>Mark, the patch is not trivial, I cannot spend time on this until this
>>is accepted. Hope you understand.
> 
> I certainly do understand. So it's likely not going to happen.

Could some platform-dependent #defines be added to
the Cython-generated preamble in the meantime to get
the equivalent of this?

-- 
Greg

From jf.moulin at gmail.com  Sat Feb 14 17:14:46 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Sat, 14 Feb 2009 17:14:46 +0100
Subject: [Cython] function declaration problem
Message-ID: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>

Hi,

my first, probably very naive, question to the list.... I want to
create an extension class with a method whose definition changes as a
function of some test
at the time of creation of the instance.
Something like:

cdef class Test:

    cdef int x, foo

    def __init__(self,foo):

        if foo>4:
            self.f = self.fn1
        else:
            self.f = self.fn2

    cdef fn1(self,x):
        return x

    cdef fn2(self,x):
        return 2 * x


Is this possible, and if yes, how should I declare f?

Thanks in advance for your help.

JF

From greg.ewing at canterbury.ac.nz  Sat Feb 14 20:48:58 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 15 Feb 2009 08:48:58 +1300
Subject: [Cython] function declaration problem
In-Reply-To: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
References: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
Message-ID: <4997202A.7090402@canterbury.ac.nz>

Jean-Francois Moulin wrote:
> I want to
> create an extension class with a method whose definition changes as a
> function of some test

Store a function pointer as a C attribute and have
a fixed f method that calls it.

-- 
Greg

From stefan at sun.ac.za  Mon Feb 16 11:49:36 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 16 Feb 2009 12:49:36 +0200
Subject: [Cython] Optimised data structures
Message-ID: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>

Hi all,

Is the list of optimised data-structures documented somewhere?  For
example, I know that lists are fast to manipulate in Cython, but how
about dictionaries, etc.?

Re: the docs, there's a linking problem in

http://docs.cython.org/docs/tutorial.html#language-details

and the Index and Module files are missing.  I can send a patch, if needed.

Regards
St?fan

From greg.ewing at canterbury.ac.nz  Mon Feb 16 12:02:31 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 17 Feb 2009 00:02:31 +1300
Subject: [Cython] Optimised data structures
In-Reply-To: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
Message-ID: <499947C7.2040902@canterbury.ac.nz>

St?fan van der Walt wrote:

> Is the list of optimised data-structures documented somewhere?

In the Pyrex docs there's a list of all the types it
knows about and the API calls that it will generate
given the right circumstances.

Not sure if it's still up to date with respect to
Cython, though.

-- 
Greg

From stefan_ml at behnel.de  Mon Feb 16 12:00:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Feb 2009 12:00:31 +0100 (CET)
Subject: [Cython] Optimised data structures
In-Reply-To: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
Message-ID: <54297.213.61.181.86.1234782031.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

St?fan van der Walt wrote:
> Is the list of optimised data-structures documented somewhere?  For
> example, I know that lists are fast to manipulate in Cython, but how
> about dictionaries, etc.?

All builtin types should be fast to use. If not, that's a bug.

Stefan


From stefan_ml at behnel.de  Mon Feb 16 13:33:24 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Feb 2009 13:33:24 +0100 (CET)
Subject: [Cython] Optimised data structures
In-Reply-To: <499947C7.2040902@canterbury.ac.nz>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
	<499947C7.2040902@canterbury.ac.nz>
Message-ID: <64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Greg Ewing wrote:
> In the Pyrex docs there's a list of all the types it
> knows about and the API calls that it will generate
> given the right circumstances.
>
> Not sure if it's still up to date with respect to
> Cython, though.

It's definitely not. It's difficult to keep the docs in sync with all
places where optimisations are done in one way or another. Loops are a
good example.

I think it's actually best to just write your code the way you would in
Python, and then check the generated C code to see if it comes out as
expected. That way, you will not only spot places where Cython does not
optimise as expected (in which case a patch is appreciated, although a
wishlist bug report will also be helpful), but you will also find places
where Cython simply didn't know your intention, so that an additional
"cdef" in the right places will bring you a major speedup. That's pretty
orthogonal to type specific optimisations.

Stefan


From stefan at sun.ac.za  Mon Feb 16 13:44:56 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 16 Feb 2009 14:44:56 +0200
Subject: [Cython] Optimised data structures
In-Reply-To: <64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
	<499947C7.2040902@canterbury.ac.nz>
	<64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <9457e7c80902160444j76dce4f7tb4dbaf0ebd8c5cbc@mail.gmail.com>

2009/2/16 Stefan Behnel <stefan_ml at behnel.de>:
> I think it's actually best to just write your code the way you would in
> Python, and then check the generated C code to see if it comes out as
> expected.

Thanks for the advice (and thank you for annotated output)!

Regards
St?fan

From dalcinl at gmail.com  Mon Feb 16 14:01:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 16 Feb 2009 11:01:59 -0200
Subject: [Cython] [Python-Dev] Adding T_SIZET to structmember.h
In-Reply-To: <4996205B.8090503@canterbury.ac.nz>
References: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>
	<5c6f2a5d0902130715x4d4f4beet7d936f9c549a1335@mail.gmail.com>
	<e7ba66e40902130744s4a14c9a0n5dfed4b5905775d5@mail.gmail.com>
	<499611E6.1020104@v.loewis.de> <4996205B.8090503@canterbury.ac.nz>
Message-ID: <e7ba66e40902160501k55fc3a7fvc5154de79c2545ad@mail.gmail.com>

Greg, what you are suggesting is what I've implemented and pushed to
cython-devel. Of couse, the #define can only deal with size_t being
unsigned int/long/longlong. It should be enough, right?

On Fri, Feb 13, 2009 at 11:37 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Martin v. L?wis wrote:
>>>Mark, the patch is not trivial, I cannot spend time on this until this
>>>is accepted. Hope you understand.
>>
>> I certainly do understand. So it's likely not going to happen.
>
> Could some platform-dependent #defines be added to
> the Cython-generated preamble in the meantime to get
> the equivalent of this?
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb 16 16:55:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 16 Feb 2009 13:55:22 -0200
Subject: [Cython] [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
In-Reply-To: <E2F78134-ABBD-4822-A843-96CDB2127D77@mcs.anl.gov>
References: <760F2670-F665-45DC-8D34-CBDD3493EBDB@stanford.edu>
	<E2F78134-ABBD-4822-A843-96CDB2127D77@mcs.anl.gov>
Message-ID: <e7ba66e40902160755l3058678duc09b9cc7ccb997d2@mail.gmail.com>

Two separate issues here...

1) Regarding mpi4py, as it is based on the C API for MPI, it does not
have complex support natively. Of course, you could use

COMPLEX = MPI.DOUBLE.Create_contiguous(2)

and then the new 'COMPLEX' datatype will likely work for communicating
arrays with (double precision) complex entries.

1) Regarding petsc4py, the support for complex numbers is not ready
yet. The big issue is that Cython does not natively support them.
Rather than hacking petsc4py for complex support, I would prefer to
spend my time on hacking Cython to enable good complex support.
However, if there is a strong need for petsc4py+complex, I could work
on providing a patch to make it work.



Now a request to the Cython folks,

Could we create a branch 'cython-devel-complex' for start working on
good complex support?
This way I could push my preliminary work on this, all you can review
my progress, and other users can pull and build petsc4py with complex
support...



On Mon, Feb 16, 2009 at 1:17 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>
> Begin forwarded message:
>
> From: Wonseok Shin <wsshin at stanford.edu>
> Date: February 15, 2009 9:44:08 PM CST
> To: petsc-maint Maintenance <petsc-maint at mcs.anl.gov>
> Cc: petsc-maint at mcs.anl.gov
> Subject: Fwd: [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
> Dear PETSc developers,
> I have received the following quoted email from the PETSc installer of
> BIgben, PSC.
> Is it true that mpi4py and petsc4py cannot be used with PETSc built for
> complex numbers?  Is there any workaround?
>
> Regards,
> Wonseok Shin
> Ph.D candidate
> Department of Electrical Engineering, Stanford University
> wsshin at stanford.edu
>
> Begin forwarded message:
>
> From: "Rick Costa via RT" <remarks at psc.edu>
> Date: February 15, 2009 8:22:21 AM PST
> To: wsshin at stanford.edu
> Subject: Re: [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
> Reply-To: remarks at psc.edu
>
>
> ---------- Forwarded message ----------
> Date: Fri, 13 Feb 2009 01:50:27 -0500
> From: David O'Neal <oneal at psc.edu>
> To: R.F. Costa <costa at psc.edu>
> Subject: Re: PETsc additions
>
> Hi Rick.
>
> I've run into a problem with the build requested by the user. Support for
> Python
> features is incompatible with the complex data type...
>
> *********************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for
> details):
> -----------------------------------------------------------------------------------
> Cannot use mpi4py with complex numbers it is not coded for this capability
> *********************************************************************************
>
> At 08:21 AM 2/10/2009 -0500, you wrote:
>
> Dave
>
> Is it possible to make these additions to PETSc? Have you had a chance
>
> to install the bug fix to PETSc? Thanks.
>
> Rick
>
> ---------- Forwarded message ----------
>
> Date: Tue, 10 Feb 2009 07:58:55 -0500 (EST)
>
> From: Wonseok Shin via RT <remarks at psc.edu>
>
> Subject: [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
>
> Tue Feb 10 07:58:55 2009: Request 16673 was acted upon.
>
> Transaction: Ticket created by wsshin at stanford.edu
>
>      Queue: Remarks
>
>    Subject: Fwd: [PETSC #19028] PETSc and Python
>
>      Owner: Nobody
>
> Requestors: wsshin at stanford.edu
>
>     Status: new
>
> Ticket <URL: https://tickets.psc.edu/Ticket/Display.html?id=16673 >
>
>
>
> Dear Rick,
>
> I am Wonseok Shin using bigben.
>
> I would like to distribute input data to my PETSc program using
>
> Python.  I asked PETSc developers how to do that, and they sent me the
>
> email quoted below.
>
> Basically, the Python packages called mpi4py and petsc4py should be
>
> installed, and they said that this can be done by configuring PETSc
>
> with the options written in the quoted email.
>
> Is it possible to install mpi4py and petsc4py?  Could you forward this
>
> email to the PETSc or Python installer of bigben?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> wsshin at stanford.edu
>
>
>
> Begin forwarded message:
>
> From: Matthew Knepley <petsc-maint at mcs.anl.gov>
>
> Date: February 10, 2009 4:39:57 AM PST
>
> To: Wonseok Shin <wsshin at stanford.edu>
>
> Cc: petsc-maint Maintenance <petsc-maint at mcs.anl.gov>
>
> Subject: Re: [PETSC #19028] PETSc and Python
>
> Reply-To: petsc-maint at mcs.anl.gov
>
> You can use mpi4py and petsc4py. They both can be automatically
>
> installed by configure:
>
> --download-petsc4py --download-mpi4py
>
> They have a lot of documentation associated with them and examples.
>
>  Matt
>
> On Tue, Feb 10, 2009 at 5:42 AM, Wonseok Shin <wsshin at stanford.edu>
>
> wrote:
>
> Dear PETSc developers,
>
> I am interested in the interaction of PETSc and Python.  Is it
>
> possible to
>
> distribute data from Python to PETSc and vice versa?  I know how to
>
> do this
>
> for a single process, but don't know for multiple processes.
>
> If I need some Python MPI module, could you recommend a specific
>
> one that
>
> works well with PETSc?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> wsshin at stanford.edu
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
>
> experiments is infinitely more interesting than any results to which
>
> their experiments lead.
>
> -- Norbert Wiener
>
>
>
> Dear Rick,
>
> I am Wonseok Shin using bigben.
>
> I would like to distribute input data to my PETSc program using Python.  I
> asked PETSc developers how to do that, and they sent me the email quoted
> below.
>
> Basically, the Python packages called mpi4py and petsc4py should be
> installed, and they said that this can be done by configuring PETSc with the
> options written in the quoted email.
>
> Is it possible to install mpi4py and petsc4py?  Could you forward this email
> to the PETSc or Python installer of bigben?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> <mailto:wsshin at stanford.edu>wsshin at stanford.edu
>
>
>
> Begin forwarded message:
>
> From: Matthew Knepley
> <<mailto:petsc-maint at mcs.anl.gov>petsc-maint at mcs.anl.gov>
>
> Date: February 10, 2009 4:39:57 AM PST
>
> To: Wonseok Shin <<mailto:wsshin at stanford.edu>wsshin at stanford.edu>
>
> Cc: petsc-maint Maintenance
> <<mailto:petsc-maint at mcs.anl.gov>petsc-maint at mcs.anl.gov>
>
> Subject: Re: [PETSC #19028] PETSc and Python
>
> Reply-To: <mailto:petsc-maint at mcs.anl.gov>petsc-maint at mcs.anl.gov
>
> You can use mpi4py and petsc4py. They both can be automatically
>
> installed by configure:
>
> --download-petsc4py --download-mpi4py
>
> They have a lot of documentation associated with them and examples.
>
>  Matt
>
> On Tue, Feb 10, 2009 at 5:42 AM, Wonseok Shin
> <<mailto:wsshin at stanford.edu>wsshin at stanford.edu> wrote:
>
> Dear PETSc developers,
>
> I am interested in the interaction of PETSc and Python.  Is it possible to
>
> distribute data from Python to PETSc and vice versa?  I know how to do this
>
> for a single process, but don't know for multiple processes.
>
> If I need some Python MPI module, could you recommend a specific one that
>
> works well with PETSc?
>
>
> Best regards,
>
> Wonseok Shin
>
> Ph.D candidate
>
> Department of Electrical Engineering, Stanford University
>
> <mailto:wsshin at stanford.edu>wsshin at stanford.edu
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
>
> experiments is infinitely more interesting than any results to which
>
> their experiments lead.
>
> -- Norbert Wiener
>
>
>
> Our PETSc installer has created a new version of PETSc. You can run it
> by issuing the command
>
>   module load petsc/3.0.0-p3.complex
>
> I am not sure what it includes. I do not think it includes the Python
> additions you requested, because they are incompatible with the complex
> data type. I have asked for clarification on what this new version
> includes.
>
> Rick Costa
> PSC User Services
>
>
>
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Mon Feb 16 20:10:33 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Feb 2009 11:10:33 -0800
Subject: [Cython] [psc.edu #16673] Fwd: [PETSC #19028] PETSc and Python
In-Reply-To: <e7ba66e40902160755l3058678duc09b9cc7ccb997d2@mail.gmail.com>
References: <760F2670-F665-45DC-8D34-CBDD3493EBDB@stanford.edu>
	<E2F78134-ABBD-4822-A843-96CDB2127D77@mcs.anl.gov>
	<e7ba66e40902160755l3058678duc09b9cc7ccb997d2@mail.gmail.com>
Message-ID: <84066170-DE9C-4B0A-AFC1-AD32672EA4B0@math.washington.edu>

On Feb 16, 2009, at 7:55 AM, Lisandro Dalcin wrote:

> Two separate issues here...
>
> 1) Regarding mpi4py, as it is based on the C API for MPI, it does not
> have complex support natively. Of course, you could use
>
> COMPLEX = MPI.DOUBLE.Create_contiguous(2)
>
> and then the new 'COMPLEX' datatype will likely work for communicating
> arrays with (double precision) complex entries.
>
> 1) Regarding petsc4py, the support for complex numbers is not ready
> yet. The big issue is that Cython does not natively support them.
> Rather than hacking petsc4py for complex support, I would prefer to
> spend my time on hacking Cython to enable good complex support.
> However, if there is a strong need for petsc4py+complex, I could work
> on providing a patch to make it work.
>
>
>
> Now a request to the Cython folks,
>
> Could we create a branch 'cython-devel-complex' for start working on
> good complex support?
> This way I could push my preliminary work on this, all you can review
> my progress, and other users can pull and build petsc4py with complex
> support...

I have started doing a bit of work on this too, but first priority  
should be to get 0.11 released. Then we can focus on new features again.

- Robert

From robertwb at math.washington.edu  Mon Feb 16 20:18:41 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Feb 2009 11:18:41 -0800
Subject: [Cython] Optimised data structures
In-Reply-To: <64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <9457e7c80902160249g37a5fb77v56769e5b7dda32be@mail.gmail.com>
	<499947C7.2040902@canterbury.ac.nz>
	<64560.213.61.181.86.1234787604.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <9C4AB959-6070-4B83-9565-170928F1CE46@math.washington.edu>

On Feb 16, 2009, at 4:33 AM, Stefan Behnel wrote:

> Greg Ewing wrote:
>> In the Pyrex docs there's a list of all the types it
>> knows about and the API calls that it will generate
>> given the right circumstances.
>>
>> Not sure if it's still up to date with respect to
>> Cython, though.
>
> It's definitely not. It's difficult to keep the docs in sync with all
> places where optimisations are done in one way or another. Loops are a
> good example.

It would be nice to have this in the documentation, but there are so  
many little optimizations that it would be a lot to write them all  
down. And it's not very well defined to say "type X is optimized"  
because some operations may be optimized, and some not.

> I think it's actually best to just write your code the way you  
> would in
> Python, and then check the generated C code to see if it comes out as
> expected. That way, you will not only spot places where Cython does  
> not
> optimise as expected (in which case a patch is appreciated, although a
> wishlist bug report will also be helpful), but you will also find  
> places
> where Cython simply didn't know your intention, so that an additional
> "cdef" in the right places will bring you a major speedup. That's  
> pretty
> orthogonal to type specific optimisations.

I heartily second this.

- Robert


From greg.ewing at canterbury.ac.nz  Mon Feb 16 22:28:04 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 17 Feb 2009 10:28:04 +1300
Subject: [Cython] [Python-Dev] Adding T_SIZET to structmember.h
In-Reply-To: <e7ba66e40902160501k55fc3a7fvc5154de79c2545ad@mail.gmail.com>
References: <e7ba66e40902121242x2180b51drcb7de29b00a5548a@mail.gmail.com>
	<5c6f2a5d0902130715x4d4f4beet7d936f9c549a1335@mail.gmail.com>
	<e7ba66e40902130744s4a14c9a0n5dfed4b5905775d5@mail.gmail.com>
	<499611E6.1020104@v.loewis.de> <4996205B.8090503@canterbury.ac.nz>
	<e7ba66e40902160501k55fc3a7fvc5154de79c2545ad@mail.gmail.com>
Message-ID: <4999DA64.8020907@canterbury.ac.nz>

Lisandro Dalcin wrote:
> the #define can only deal with size_t being
> unsigned int/long/longlong. It should be enough, right?

Probably handles all the CPUs we're likely to
encounter at the moment.

-- 
Greg

From ondrej at certik.cz  Tue Feb 17 00:16:01 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 15:16:01 -0800
Subject: [Cython] where should .pxd files be installed
Message-ID: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>

Hi,

where should the .pxd files for a package be installed?
/usr/include/package_name/, /usr/lib/python/package_name/include, or
somewhere else?

Is there some place that cython looks at for those .pyx and .pxd
files? Is there some consensus/standard for this?

Ondrej

From robertwb at math.washington.edu  Tue Feb 17 01:37:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Feb 2009 16:37:38 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
Message-ID: <7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>

They should be installed in the tree alongside the .py files, and  
will be discovered just like normal Python modules.

On Feb 16, 2009, at 3:16 PM, Ondrej Certik wrote:

> Hi,
>
> where should the .pxd files for a package be installed?
> /usr/include/package_name/, /usr/lib/python/package_name/include, or
> somewhere else?
>
> Is there some place that cython looks at for those .pyx and .pxd
> files? Is there some consensus/standard for this?
>
> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ondrej at certik.cz  Tue Feb 17 01:52:09 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 16:52:09 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
Message-ID: <85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>

On Mon, Feb 16, 2009 at 4:37 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> They should be installed in the tree alongside the .py files, and
> will be discovered just like normal Python modules.

Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
some C++ library, then it has some .pyx and .pxd files and some .so
files as a result of cythonzing it, then:

C++ .so should go to /usr/lib/
C++ .h files should go to /usr/include
.pxd and other .py files and the .so as a result of cythonizing should
go along the .py files

Ondrej

From ondrej at certik.cz  Tue Feb 17 01:55:41 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 16:55:41 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
	<85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
Message-ID: <85b5c3130902161655i3745e8f9v2d4231bfb4360350@mail.gmail.com>

On Mon, Feb 16, 2009 at 4:52 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Mon, Feb 16, 2009 at 4:37 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> They should be installed in the tree alongside the .py files, and
>> will be discovered just like normal Python modules.
>
> Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
> some C++ library, then it has some .pyx and .pxd files and some .so
> files as a result of cythonzing it, then:
>
> C++ .so should go to /usr/lib/
> C++ .h files should go to /usr/include
> .pxd and other .py files and the .so as a result of cythonizing should
> go along the .py files

I am asking because I am using cmake, I made it work with Cython and
numpy nicely and I like it. When I get more experience with it, I post
here the files for other people to use.

My repository is here, see the master-ondrej branch:

http://github.com/certik/hermes2d/tree/master-ondrej


Ondrej

From ellisonbg.net at gmail.com  Tue Feb 17 02:04:42 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 16 Feb 2009 17:04:42 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
	<85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
Message-ID: <6ce0ac130902161704wb18549bs446e22337babd47f@mail.gmail.com>

> Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
> some C++ library, then it has some .pyx and .pxd files and some .so
> files as a result of cythonzing it, then:
>
> C++ .so should go to /usr/lib/
> C++ .h files should go to /usr/include

This is only if you want the library available outside of Python.
Otherwise, these things can go with the .py files in the package as
well.

> .pxd and other .py files and the .so as a result of cythonizing should
> go along the .py files

As I understand it, there are two things that are relevant about the
location of the .pxd files:

* When you are building your own project, the .pxd files need to be
"findable" by Cython.  In your case, this finding happens when cmake
is used to build call Cython and build extensions to begin with.

* When external projects want to use your .pxd files in a cimport
statement, they need to know where they are.  For that, it is really
helpful if your package can provide a get_pxd_include() function that
returns the include directory for your  pxd files.  Something like
(for your hermes2d project):

hermes2d.get_pxd_include()

Another issue for your project, is that people who want to link to it
will want to know where the .so and .h files are installed (the actual
C++ library).  Again, I would simply provide top-level functions for
this:

hermes2d.get_lib()
hermes2d.get_include()

With these 3 functions, anyone building Cython package based on your
code can quickly find out where everything is installed.

The only other thing that could complicate this is if there are any
other dependencies that end users need to build against your project.

Cheers,

Brian

ps - I know this is answering more than your initial question and is
slightly off topic for this list.

> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ondrej at certik.cz  Tue Feb 17 02:54:15 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 17:54:15 -0800
Subject: [Cython] where should .pxd files be installed
In-Reply-To: <6ce0ac130902161704wb18549bs446e22337babd47f@mail.gmail.com>
References: <85b5c3130902161516m5437e25ep7b3fff6cbcc3b6e7@mail.gmail.com>
	<7325E4BE-92B0-4E8B-AFF4-40984DF47EF6@math.washington.edu>
	<85b5c3130902161652qa39147dhe812630ba00355c9@mail.gmail.com>
	<6ce0ac130902161704wb18549bs446e22337babd47f@mail.gmail.com>
Message-ID: <85b5c3130902161754g31da5aecua049eec933c30ab4@mail.gmail.com>

On Mon, Feb 16, 2009 at 5:04 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Ok, thanks. So if my project uses C++ .cpp and .h files, then it has
>> some C++ library, then it has some .pyx and .pxd files and some .so
>> files as a result of cythonzing it, then:
>>
>> C++ .so should go to /usr/lib/
>> C++ .h files should go to /usr/include
>
> This is only if you want the library available outside of Python.
> Otherwise, these things can go with the .py files in the package as
> well.

Yes, I think I want my C++ library to behave like a regular C++ library.

>
>> .pxd and other .py files and the .so as a result of cythonizing should
>> go along the .py files
>
> As I understand it, there are two things that are relevant about the
> location of the .pxd files:
>
> * When you are building your own project, the .pxd files need to be
> "findable" by Cython.  In your case, this finding happens when cmake
> is used to build call Cython and build extensions to begin with.

My own project builds fine, there is no problem with that.

>
> * When external projects want to use your .pxd files in a cimport
> statement, they need to know where they are.  For that, it is really
> helpful if your package can provide a get_pxd_include() function that
> returns the include directory for your  pxd files.  Something like
> (for your hermes2d project):
>
> hermes2d.get_pxd_include()
>
> Another issue for your project, is that people who want to link to it
> will want to know where the .so and .h files are installed (the actual
> C++ library).  Again, I would simply provide top-level functions for
> this:
>
> hermes2d.get_lib()
> hermes2d.get_include()
>
> With these 3 functions, anyone building Cython package based on your
> code can quickly find out where everything is installed.

That's a good idea, I'll do that.

> The only other thing that could complicate this is if there are any
> other dependencies that end users need to build against your project.
>
> Cheers,
>
> Brian
>
> ps - I know this is answering more than your initial question and is
> slightly off topic for this list.

Thanks a lot for answering, in fact, this is exactly what I wanted to know. :)

I'll implement the functions above, that way it will be easy for
anyone to build against my project. There should be no other
dependencies, after I fix everything.

Ondrej

From ondrej at certik.cz  Tue Feb 17 05:26:24 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 16 Feb 2009 20:26:24 -0800
Subject: [Cython] how to initialize a class without calling it's constructor
Message-ID: <85b5c3130902162026g7a2bfaa7qe4af928f105b9e02@mail.gmail.com>

Hi,

I have the following class:

cdef class H1Space:

    def __init__(self, Mesh m, H1Shapeset s):
        self.thisptr = new_H1Space(m.thisptr, s.thisptr)

and it is initialized from 2 other classes. Sometimes however, I would
like to initialize it directly from a C++ instance of c_H1Space, e.g.
I would like to create the H1Space class *without* calling it's
constructor and just set the self.thisptr object to the C++ instance.

Currently I am using the following code and everything works for me:


cdef H1Space H1Space_from_c_H1Space(c_H1Space *h):
    cdef H1Space n
    n = <H1Space>PY_NEW(H1Space)
    n.thisptr = h
    return n


But it uses the PY_NEW macro:


#define PY_NEW(zzz_type) \
    (((PyTypeObject*)(zzz_type))->tp_new((PyTypeObject*)(zzz_type),
global_empty_tuple, NULL))

and global_empty_tuple() that I had to initialize, etc. I copied this
from Sage. Is there some better way to do the above, so that I don't
have to include the special macro above and the corresponding .h and
.c files in my project?

Thanks,
Ondrej

From wstein at gmail.com  Tue Feb 17 06:25:30 2009
From: wstein at gmail.com (William Stein)
Date: Mon, 16 Feb 2009 21:25:30 -0800
Subject: [Cython] project web servers
Message-ID: <85e81ba30902162125s6e0e33a7k3743896aa72019ce@mail.gmail.com>

Hi,

I had to migrate all the cython, Sage and mpir websites and related
servers to another machine, due to some hardware failures on the
server that was hosting them.    Please let me know if any of the
cython, Sage, or MPIR websites are suddenly broken.

 -- William

-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From wstein at gmail.com  Tue Feb 17 07:39:05 2009
From: wstein at gmail.com (William Stein)
Date: Mon, 16 Feb 2009 22:39:05 -0800
Subject: [Cython] [sage-devel] Re: project web servers
In-Reply-To: <ee247281-3b84-4c3b-82c0-e82dfcd330d1@d32g2000yqe.googlegroups.com>
References: <85e81ba30902162125s6e0e33a7k3743896aa72019ce@mail.gmail.com>
	<ee247281-3b84-4c3b-82c0-e82dfcd330d1@d32g2000yqe.googlegroups.com>
Message-ID: <85e81ba30902162239j14b1fc00o4b83fc6f85d308c1@mail.gmail.com>

On Mon, Feb 16, 2009 at 9:45 PM, mabshoff <mabshoff at googlemail.com> wrote:
>
>
>
> On Feb 16, 9:25 pm, William Stein <wst... at gmail.com> wrote:
>> Hi,
>>
>> I had to migrate all the cython, Sage and mpir websites and related
>> servers to another machine, due to some hardware failures on the
>> server that was hosting them.    Please let me know if any of the
>> cython, Sage, or MPIR websites are suddenly broken.
>
> Attaching sub-1kb patches is broken in Sage's trac with
>
>
> [9:39pm] mabs: wstein2: I cannot attach patches to trac all the
> sudden. It worked 10 minutes ago:
> [9:39pm] mabs: Traceback (most recent call last):
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/web/main.py", line 387, in
> dispatch_request
> [9:39pm] mabs:     dispatcher.dispatch(req)
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/web/main.py", line 237, in dispatch
> [9:39pm] mabs:     resp = chosen_handler.process_request(req)
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/attachment.py", line 361, in
> process_request
> [9:39pm] mabs:     self._do_save(req, attachment)
> [9:39pm] mabs:   File "/home/sage/sage_install/stable/local/lib/
> python2.5/site-packages/trac/attachment.py", line 450, in _do_save
> [9:39pm] mabs:     size = upload.file.len
>
> I tested and as soon as a patch is larger than 1023 bytes it works
> again.
>

Thanks for pointing this out.

I've upgraded the trac servers for sage, cython, and mpir all to
trac-0.11.3.  Again, let me know if you have any problems as a result.

Note that trac-0.11* has much better support for "workflows", but I
haven't done anything with that -- I just migrated the old installs.
Mike Hansen will likely incorporate better workflow support, at least
for the sage project.

William

From robertwb at math.washington.edu  Tue Feb 17 10:32:30 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 01:32:30 -0800
Subject: [Cython] how to initialize a class without calling it's
	constructor
In-Reply-To: <85b5c3130902162026g7a2bfaa7qe4af928f105b9e02@mail.gmail.com>
References: <85b5c3130902162026g7a2bfaa7qe4af928f105b9e02@mail.gmail.com>
Message-ID: <26B8DFAA-3CB3-4D73-9735-32709CA76C24@math.washington.edu>

On Feb 16, 2009, at 8:26 PM, Ondrej Certik wrote:

> Currently I am using the following code and everything works for me:

[...]

> But it uses the PY_NEW macro:
>
> #define PY_NEW(zzz_type) \
>     (((PyTypeObject*)(zzz_type))->tp_new((PyTypeObject*)(zzz_type),
> global_empty_tuple, NULL))
>
> and global_empty_tuple() that I had to initialize, etc. I copied this
> from Sage. Is there some better way to do the above, so that I don't
> have to include the special macro above and the corresponding .h and
> .c files in my project?

How about

cdef extern from *:
     ctypedef struct RichPyTypeObject "PyTypeObject":
         object tp_new(object, object, PyObject*)

cdef inline PY_NEW(T):
     return (<RichPyTypeObject*>x).tp_new(T, (), NULL)


- Robert


From robertwb at math.washington.edu  Tue Feb 17 10:47:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 01:47:25 -0800
Subject: [Cython] function declaration problem
In-Reply-To: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
References: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
Message-ID: <3D9D09CD-1214-46ED-8D24-B25F40C91F98@math.washington.edu>

On Feb 14, 2009, at 8:14 AM, Jean-Francois Moulin wrote:

> Hi,
>
> my first, probably very naive, question to the list.... I want to
> create an extension class with a method whose definition changes as a
> function of some test
> at the time of creation of the instance.
> Something like:
>
> cdef class Test:
>
>     cdef int x, foo

I don't think this does what you think it does--it makes Test have  
two attributes "x" and "foo"

>
>     def __init__(self,foo):
>
>         if foo>4:
>             self.f = self.fn1
>         else:
>             self.f = self.fn2
>
>     cdef fn1(self,x):
>         return x
>
>     cdef fn2(self,x):
>         return 2 * x
>
>
> Is this possible, and if yes, how should I declare f?

Declare it as a c function pointer


cdef class Test:

     cdef int (*f)(Test, int)

     def __init__(self, foo):

         if foo > 4:
             self.f = self.fn1
         else:
             self.f = self.fn2

     cdef int fn1(self, int x):
         return x

     cdef int fn2(self, int x):
         return 2 * x

     def test_f(self, x):
         return self.f(self, x)

 >>> a = Test(1)
 >>> a.test_f(10)
20
 >>> a = Test(100)
 >>> a.test_f(10)
10

- Robert

From robertwb at math.washington.edu  Tue Feb 17 10:56:48 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 01:56:48 -0800
Subject: [Cython] import/cimport gotcha
In-Reply-To: <e7ba66e40902130752j453000b5yc68c9d74c11f1f4e@mail.gmail.com>
References: <e7ba66e40902130752j453000b5yc68c9d74c11f1f4e@mail.gmail.com>
Message-ID: <1BBB9640-C0CE-41F5-82DE-25F4E031E03C@math.washington.edu>

On Feb 13, 2009, at 7:52 AM, Lisandro Dalcin wrote:

> Some time ago I reported that the following:
>
> from foo import Bar
> from foo cimport Bar
>
> do not put 'Bar' in the module dicti. Commenting the lines below seems
> to solve the issue. Am I missing something (I bet the answer is yes)?
> Any comments?

Hmm... those lines were added in changeset 1309 because of issues  
with compiling .py files. I think it may change the name to be an  
ordinary name, rather than a cimported one, but I'm not sure.

Does Cython compile itself after this patch? Also, rather than  
commenting stuff out, if it's not needed it should just be deleted.

> diff -r 1771db44498b Cython/Compiler/Nodes.py
> --- a/Cython/Compiler/Nodes.py	Thu Feb 12 18:22:15 2009 -0200
> +++ b/Cython/Compiler/Nodes.py	Fri Feb 13 13:42:23 2009 -0200
> @@ -4600,8 +4600,9 @@
>                          break
>              else:
>                  entry =  env.lookup(target.name)
> -                if entry.is_type and entry.type.name == name and
> entry.type.module_name == self.module.module_name.value:
> -                    continue # already cimported
> +                #if entry.is_type and entry.type.name == name and
> entry.type.module_name == self.module.module_name.value:
> +                #    continue # already cimported
> +                #    pass
>                  self.interned_items.append(
>                      (env.intern_identifier(name), target))
>                  target.analyse_target_expression(env, None)
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From jf.moulin at gmail.com  Tue Feb 17 11:01:40 2009
From: jf.moulin at gmail.com (Jean-Francois Moulin)
Date: Tue, 17 Feb 2009 11:01:40 +0100
Subject: [Cython] function declaration problem
In-Reply-To: <3D9D09CD-1214-46ED-8D24-B25F40C91F98@math.washington.edu>
References: <af6d35810902140814q6e2305b1sbe512a5faa9e0eb@mail.gmail.com>
	<3D9D09CD-1214-46ED-8D24-B25F40C91F98@math.washington.edu>
Message-ID: <af6d35810902170201x446d55f7u90b29f641a6aed66@mail.gmail.com>

Thanks a bunch for the informative answers!
One of these days I'll have to read a bit about C, for sure. I've been
spoiled by python ;0)
JF

From robertwb at math.washington.edu  Tue Feb 17 11:04:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 02:04:28 -0800
Subject: [Cython] Incomplete C++ example
In-Reply-To: <9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>
	<9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
Message-ID: <B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>

On Feb 12, 2009, at 8:40 AM, St?fan van der Walt wrote:

> Hi all,
>
> I'm helping a student to wrap some C++ code using Cython, and noticed
> that the example in the documentation is incomplete.  A header file,
> Rectangle.h, is provided, but no implementation is given, so the
> example cannot be compiled.
>
> It would be great if you could add an implementation (see attached)

There is already one attached (see the bottom of the page) but it  
should probably be made more prominent.

> and modify the setup.py accordingly, i.e.
>
>    Extension(
>    "rectangle",
>    ["rectangle.pyx", "Rectangle.cc"],
>    language="c++")


The example setup.py assumes Rectangle is part of an externally  
linked library (which is the most common use case). However, the  
example Setup.py is more explicit. It is a wiki, so it would be nice  
if someone (you?) improved the presentation.

- Robert


From dagss at student.matnat.uio.no  Tue Feb 17 11:23:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 17 Feb 2009 11:23:40 +0100
Subject: [Cython] Incomplete C++ example
In-Reply-To: <B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>	<9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
	<B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>
Message-ID: <499A902C.3060604@student.matnat.uio.no>

Robert Bradshaw wrote:
> The example setup.py assumes Rectangle is part of an externally  
> linked library (which is the most common use case). However, the  
> example Setup.py is more explicit. It is a wiki, so it would be nice  
> if someone (you?) improved the presentation.

I think Gabrial has also pulled it into the main documentation though, 
where the attachment is not present:

http://docs.cython.org/docs/wrapping_CPlusPlus.html


-- 
Dag Sverre

From robertwb at math.washington.edu  Tue Feb 17 11:15:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Feb 2009 02:15:04 -0800
Subject: [Cython] Incomplete C++ example
In-Reply-To: <499A902C.3060604@student.matnat.uio.no>
References: <mailman.86328.1234451867.4712.cython-dev@codespeak.net>	<9457e7c80902120840w81f1a98j730d1b6c34051754@mail.gmail.com>
	<B4B68EDF-E831-4EE2-BD79-5B140BD39F84@math.washington.edu>
	<499A902C.3060604@student.matnat.uio.no>
Message-ID: <E67CA6BB-8012-45A2-822D-2FF5E4C4A938@math.washington.edu>

On Feb 17, 2009, at 2:23 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> The example setup.py assumes Rectangle is part of an externally
>> linked library (which is the most common use case). However, the
>> example Setup.py is more explicit. It is a wiki, so it would be nice
>> if someone (you?) improved the presentation.
>
> I think Gabrial has also pulled it into the main documentation though,
> where the attachment is not present:
>
> http://docs.cython.org/docs/wrapping_CPlusPlus.html

Ah, good point. Yes, we need to have it there.

- Robert


From dalcinl at gmail.com  Wed Feb 18 21:39:50 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 18 Feb 2009 18:39:50 -0200
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
Message-ID: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>

Supose this fragment:

cdef extern from *:
    ctypedef long Py_ssize_t
    ctypedef unsigned long size_t

With this patch, these ctypedef's seems to be just ignored. This is
good for the sake of Cython-backward and Pyrex compatibility. However,
it would be nice to make Cython generate a warning. But the right
place to do that is not at the point of parsing basic C types, but on
parsing a ctypedef statement itself.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SPECIAL_C_TYPES.diff
Type: application/octet-stream
Size: 2943 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090218/fbdf4935/attachment.obj 

From robertwb at math.washington.edu  Thu Feb 19 09:22:10 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 00:22:10 -0800
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
Message-ID: <ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>

I think it makes sense to include these as basic_c_type_names, the  
same as long and char. However, I'm not to particular on the  
implementation. They should not be able to accept modifiers.

In terms of the behavior for code below, I agree an warning should be  
emitted and the ctypedef ignored.

- Robert


On Feb 18, 2009, at 12:39 PM, Lisandro Dalcin wrote:

> Supose this fragment:
>
> cdef extern from *:
>     ctypedef long Py_ssize_t
>     ctypedef unsigned long size_t
>
> With this patch, these ctypedef's seems to be just ignored. This is
> good for the sake of Cython-backward and Pyrex compatibility. However,
> it would be nice to make Cython generate a warning. But the right
> place to do that is not at the point of parsing basic C types, but on
> parsing a ctypedef statement itself.
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0) 
> 342-451.1594<SPECIAL_C_TYPES.diff>____________________________________ 
> ___________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Thu Feb 19 13:03:07 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 19 Feb 2009 10:03:07 -0200
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
Message-ID: <e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>

On Thu, Feb 19, 2009 at 6:22 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> I think it makes sense to include these as basic_c_type_names, the
> same as long and char. However, I'm not to particular on the
> implementation. They should not be able to accept modifiers.

Well, if Py_ssize_t and size_t are listed in basic_c_types_names, I
cannot see a simple way to make the parser reject a modifier. That's
the reason I want to treat them in a slightly special fashion.

BTW, did you mean 'int' instead of 'long'? AFAIK, 'long' is actually
'signed long int'.

> In terms of the behavior for code below, I agree an warning should be
> emitted and the ctypedef ignored.
>

Then I'll try to make this work, but this would still need to
special-case when parsing basic C types. If not, the mix of
<modifiers> <Py_ssize_t | size_t> get consumed and a the ctypedef
parsing will fail because of a empty typedef name (the same if you
wrote "typedef double" instead of "typedef double MyDouble").


Just a question: if the ctypedef is ignored, what should be the return
of p_ctypedef_statement() ??? None? a PassNode?



> - Robert
>
>
> On Feb 18, 2009, at 12:39 PM, Lisandro Dalcin wrote:
>
>> Supose this fragment:
>>
>> cdef extern from *:
>>     ctypedef long Py_ssize_t
>>     ctypedef unsigned long size_t
>>
>> With this patch, these ctypedef's seems to be just ignored. This is
>> good for the sake of Cython-backward and Pyrex compatibility. However,
>> it would be nice to make Cython generate a warning. But the right
>> place to do that is not at the point of parsing basic C types, but on
>> parsing a ctypedef statement itself.
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)
>> 342-451.1594<SPECIAL_C_TYPES.diff>____________________________________
>> ___________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Feb 19 13:29:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Feb 2009 13:29:40 +0100 (CET)
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
	<e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>
Message-ID: <43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> Then I'll try to make this work, but this would still need to
> special-case when parsing basic C types. If not, the mix of
> <modifiers> <Py_ssize_t | size_t> get consumed and a the ctypedef
> parsing will fail because of a empty typedef name (the same if you
> wrote "typedef double" instead of "typedef double MyDouble").

You can let the parser parse them as usual and once that succeeded, have
it check for the two names to drop the declaration with a warning. From my
side, it's ok to do that in the parser.


> Just a question: if the ctypedef is ignored, what should be the return
> of p_ctypedef_statement() ??? None? a PassNode?

A PassNode should be safe and allowed wherever we allow a ctypedef statement.

Stefan


From dalcinl at gmail.com  Thu Feb 19 14:44:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 19 Feb 2009 11:44:25 -0200
Subject: [Cython] I cannot push
Message-ID: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>

Yesterdey I was getting 404 errors and then changed the repo url in
.hg/hgrc. Is this the right url to push? Or perhaps my http password
was not added to the new server?

$ hg push
pushing to http://hg.cython.org/cython-devel
searching for changes
abort: authorization failed


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From wstein at gmail.com  Thu Feb 19 20:46:07 2009
From: wstein at gmail.com (William Stein)
Date: Thu, 19 Feb 2009 11:46:07 -0800
Subject: [Cython] I cannot push
In-Reply-To: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>
Message-ID: <85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>

On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Yesterdey I was getting 404 errors and then changed the repo url in
> .hg/hgrc. Is this the right url to push? Or perhaps my http password
> was not added to the new server?
>
> $ hg push
> pushing to http://hg.cython.org/cython-devel
> searching for changes
> abort: authorization failed

Can you try again now?

William

>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From robertwb at math.washington.edu  Thu Feb 19 21:03:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 12:03:04 -0800
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>
	<e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>
	<43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <D528A6E4-FD4F-4FA8-A5A6-AFC85F940679@math.washington.edu>

On Feb 19, 2009, at 4:29 AM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> Then I'll try to make this work, but this would still need to
>> special-case when parsing basic C types. If not, the mix of
>> <modifiers> <Py_ssize_t | size_t> get consumed and a the ctypedef
>> parsing will fail because of a empty typedef name (the same if you
>> wrote "typedef double" instead of "typedef double MyDouble").
>
> You can let the parser parse them as usual and once that succeeded,  
> have
> it check for the two names to drop the declaration with a warning.  
> From my
> side, it's ok to do that in the parser.
>
>> Just a question: if the ctypedef is ignored, what should be the  
>> return
>> of p_ctypedef_statement() ??? None? a PassNode?
>
> A PassNode should be safe and allowed wherever we allow a ctypedef  
> statement.

I am actually in favor of moving the logic out of the parser as much  
as possible. For example, I think the logic for rejecting "signed  
short size_t" belongs in PyrexTypes.simple_c_type, and raising an  
warning on "ctypedef int size_t" belongs in  
CTypeDefNode.analyse_declarations.

Does anyone else find it odd "ctypedef int short" is OK?

- Robert


From robertwb at math.washington.edu  Thu Feb 19 21:06:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 12:06:24 -0800
Subject: [Cython] I cannot push
In-Reply-To: <85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>
	<85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
Message-ID: <58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>

On Feb 19, 2009, at 11:46 AM, William Stein wrote:

> On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin  
> <dalcinl at gmail.com> wrote:
>> Yesterdey I was getting 404 errors and then changed the repo url in
>> .hg/hgrc. Is this the right url to push? Or perhaps my http password
>> was not added to the new server?
>>
>> $ hg push
>> pushing to http://hg.cython.org/cython-devel
>> searching for changes
>> abort: authorization failed
>
> Can you try again now?

The permissions on sage-devel/.hg didn't copy over. I fixed them, it  
should work now.

- Robert


From stefan_ml at behnel.de  Thu Feb 19 21:08:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Feb 2009 21:08:11 +0100
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <D528A6E4-FD4F-4FA8-A5A6-AFC85F940679@math.washington.edu>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>	<ACB4E70A-4586-463D-9AB4-80302F764B6B@math.washington.edu>	<e7ba66e40902190403h8735d60q62d8170fe7f08806@mail.gmail.com>	<43549.213.61.181.86.1235046580.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<D528A6E4-FD4F-4FA8-A5A6-AFC85F940679@math.washington.edu>
Message-ID: <499DBC2B.1080009@behnel.de>


Robert Bradshaw wrote:
> Does anyone else find it odd "ctypedef int short" is OK?

Ah, good point. Fine, let's make that a general feature.

Stefan


From dagss at student.matnat.uio.no  Fri Feb 20 01:56:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 01:56:06 +0100
Subject: [Cython] I cannot push
In-Reply-To: <58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>	<85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
	<58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>
Message-ID: <499DFFA6.2090202@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 19, 2009, at 11:46 AM, William Stein wrote:
> 
>> On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin  
>> <dalcinl at gmail.com> wrote:
>>> Yesterdey I was getting 404 errors and then changed the repo url in
>>> .hg/hgrc. Is this the right url to push? Or perhaps my http password
>>> was not added to the new server?
>>>
>>> $ hg push
>>> pushing to http://hg.cython.org/cython-devel
>>> searching for changes
>>> abort: authorization failed
>> Can you try again now?
> 
> The permissions on sage-devel/.hg didn't copy over. I fixed them, it  
> should work now.

I still cannot push to either cython-devel or cython-0.11.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 01:59:03 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 01:59:03 +0100
Subject: [Cython] Status for 0.11
Message-ID: <499E0057.9040504@student.matnat.uio.no>

I have a fix for #200 ready, waiting to be pushed when I have access. 
(I'll attach it to the ticket shortly as well.)

#143: I could take upon me to disable the current method without 
introducing a new one within a week. Robert, do you want me to do that, 
or do you have plans to do it properly?

#106 can be moved to 0.11.1 as Robert says.

That should leave running the refnanny on lxml and Sage before one can 
do a release(?). Has there been any attempts on that? I may also be able 
to spend some time (not too much) on refnannying Sage if Robert hasn't 
started on that (if so, I'd be grateful to know how I can add 
CYTHON_REFNANNY to CFLAGS in the Sage build easily)

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Feb 20 02:51:35 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 17:51:35 -0800
Subject: [Cython] I cannot push
In-Reply-To: <499DFFA6.2090202@student.matnat.uio.no>
References: <e7ba66e40902190544g19dae7bcge2c49a17e4264752@mail.gmail.com>	<85e81ba30902191146q21311549u41b2e1ab127f21ae@mail.gmail.com>
	<58F38826-8D43-4728-AE88-6715FDAFFB28@math.washington.edu>
	<499DFFA6.2090202@student.matnat.uio.no>
Message-ID: <5089743F-4665-4EA3-985C-53D9E7B90F56@math.washington.edu>

On Feb 19, 2009, at 4:56 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 19, 2009, at 11:46 AM, William Stein wrote:
>>
>>> On Thu, Feb 19, 2009 at 5:44 AM, Lisandro Dalcin
>>> <dalcinl at gmail.com> wrote:
>>>> Yesterdey I was getting 404 errors and then changed the repo url in
>>>> .hg/hgrc. Is this the right url to push? Or perhaps my http  
>>>> password
>>>> was not added to the new server?
>>>>
>>>> $ hg push
>>>> pushing to http://hg.cython.org/cython-devel
>>>> searching for changes
>>>> abort: authorization failed
>>> Can you try again now?
>>
>> The permissions on sage-devel/.hg didn't copy over. I fixed them, it
>> should work now.
>
> I still cannot push to either cython-devel or cython-0.11.

Me either ;).

OK, it should be good now.

- Robert



From robertwb at math.washington.edu  Fri Feb 20 02:58:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 17:58:03 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E0057.9040504@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>
Message-ID: <98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>

On Feb 19, 2009, at 4:59 PM, Dag Sverre Seljebotn wrote:

> I have a fix for #200 ready, waiting to be pushed when I have access.
> (I'll attach it to the ticket shortly as well.)

OK, I pushed this. I'm going to do more testing before I close it.

> #143: I could take upon me to disable the current method without
> introducing a new one within a week. Robert, do you want me to do  
> that,
> or do you have plans to do it properly?



> #106 can be moved to 0.11.1 as Robert says.

Moved.

> That should leave running the refnanny on lxml and Sage before one can
> do a release(?). Has there been any attempts on that? I may also be  
> able
> to spend some time (not too much) on refnannying Sage if Robert hasn't
> started on that (if so, I'd be grateful to know how I can add
> CYTHON_REFNANNY to CFLAGS in the Sage build easily)

I haven't done that yet. I'll give it a go and let you know if  
anything comes up (I can do the Sage side of things easily...).

- Robert


From michael.abshoff at googlemail.com  Fri Feb 20 05:17:57 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 19 Feb 2009 20:17:57 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>
	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>
Message-ID: <499E2EF5.2090907@gmail.com>

Robert Bradshaw wrote:
> On Feb 19, 2009, at 4:59 PM, Dag Sverre Seljebotn wrote:

<SNIP>

Dad, Robert,

>> That should leave running the refnanny on lxml and Sage before one can
>> do a release(?). Has there been any attempts on that? I may also be  
>> able
>> to spend some time (not too much) on refnannying Sage if Robert hasn't
>> started on that (if so, I'd be grateful to know how I can add
>> CYTHON_REFNANNY to CFLAGS in the Sage build easily)
> 
> I haven't done that yet. I'll give it a go and let you know if  
> anything comes up (I can do the Sage side of things easily...).

I got busy with other things, so refresh my mind: All I need to do is to 
add

  -DCYTHON_REFNANNY

to CFLAGS and after rebuilding the library run the test suite?


> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb 20 06:33:39 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 21:33:39 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E2EF5.2090907@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>
	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>
	<499E2EF5.2090907@gmail.com>
Message-ID: <4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>

On Feb 19, 2009, at 8:17 PM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 19, 2009, at 4:59 PM, Dag Sverre Seljebotn wrote:
>
> <SNIP>
>
> Dad, Robert,
>
>>> That should leave running the refnanny on lxml and Sage before  
>>> one can
>>> do a release(?). Has there been any attempts on that? I may also be
>>> able
>>> to spend some time (not too much) on refnannying Sage if Robert  
>>> hasn't
>>> started on that (if so, I'd be grateful to know how I can add
>>> CYTHON_REFNANNY to CFLAGS in the Sage build easily)
>>
>> I haven't done that yet. I'll give it a go and let you know if
>> anything comes up (I can do the Sage side of things easily...).
>
> I got busy with other things, so refresh my mind: All I need to do  
> is to
> add
>
>   -DCYTHON_REFNANNY
>
> to CFLAGS and after rebuilding the library run the test suite?

Yes, I think that would do the trick. You can also use Extension(...,  
define_macros=[('CYTHON_REFNANNY', None)]) in setup.py, but that'd  
probably be less simple.

- Robert


From michael.abshoff at googlemail.com  Fri Feb 20 06:47:07 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Thu, 19 Feb 2009 21:47:07 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>	<499E2EF5.2090907@gmail.com>
	<4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>
Message-ID: <499E43DB.4070702@gmail.com>

Robert Bradshaw wrote:
> On Feb 19, 2009, at 8:17 PM, Michael Abshoff wrote:

<SNIP>

Hi Robert,

>> I got busy with other things, so refresh my mind: All I need to do  
>> is to
>> add
>>
>>   -DCYTHON_REFNANNY
>>
>> to CFLAGS and after rebuilding the library run the test suite?
> 
> Yes, I think that would do the trick. You can also use Extension(...,  
> define_macros=[('CYTHON_REFNANNY', None)]) in setup.py, but that'd  
> probably be less simple.

Well, I will just patch it into the Python CFLAGS :). But a clean, 
on-demand solution in setup.py would be the way to go. I am not known 
for my elegant coding, so I will someone else do that.

Do you have a recent cython.spkg to test or should I just pull from devel?

> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb 20 06:49:20 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Feb 2009 21:49:20 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E43DB.4070702@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<98D3579B-4AF6-4E2A-BEDF-DC5B132A5FB3@math.washington.edu>	<499E2EF5.2090907@gmail.com>
	<4421665B-26CC-40EF-BB31-C6616F6861DC@math.washington.edu>
	<499E43DB.4070702@gmail.com>
Message-ID: <498CB5D1-3809-45F6-9351-185ED90CFCF2@math.washington.edu>

On Feb 19, 2009, at 9:47 PM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 19, 2009, at 8:17 PM, Michael Abshoff wrote:
>
> <SNIP>
>
> Hi Robert,
>
>>> I got busy with other things, so refresh my mind: All I need to do
>>> is to
>>> add
>>>
>>>   -DCYTHON_REFNANNY
>>>
>>> to CFLAGS and after rebuilding the library run the test suite?
>>
>> Yes, I think that would do the trick. You can also use Extension(...,
>> define_macros=[('CYTHON_REFNANNY', None)]) in setup.py, but that'd
>> probably be less simple.
>
> Well, I will just patch it into the Python CFLAGS :). But a clean,
> on-demand solution in setup.py would be the way to go. I am not known
> for my elegant coding, so I will someone else do that.

Yeah, I think we iterate over extensions to add gmp, etc. to the  
libraries anyways.

> Do you have a recent cython.spkg to test or should I just pull from  
> devel?

Just pull from devel.

- Robert

From stefan_ml at behnel.de  Fri Feb 20 08:33:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 08:33:50 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E0057.9040504@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>
Message-ID: <499E5CDE.6000804@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> #106 can be moved to 0.11.1 as Robert says.

+1. Could anyone still give some feedback if this still applies to current
cython-devel?


> That should leave running the refnanny on lxml and Sage before one can 
> do a release(?). Has there been any attempts on that?

I ran lxml's test suite with the refnanny and it looks mostly good, except
for a number of dropped AssertErrors and dropped unnamed(?) exceptions. It
doesn't say where they come from and I was unable to add any traceable
information to the output. There are two plain assert statements in the
refnanny code, but adding text to them doesn't change anything.

I'll post the output here as soon as I get to it.

Stefan

From stefan_ml at behnel.de  Fri Feb 20 08:42:21 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 08:42:21 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E5CDE.6000804@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de>
Message-ID: <499E5EDD.2070800@behnel.de>


Stefan Behnel wrote:
> I ran lxml's test suite with the refnanny and it looks mostly good, except
> for a number of dropped AssertErrors and dropped unnamed(?) exceptions. It
> doesn't say where they come from and I was unable to add any traceable
> information to the output. There are two plain assert statements in the
> refnanny code, but adding text to them doesn't change anything.
> 
> I'll post the output here as soon as I get to it.

Quicker than I though. Looks like only two separate problems. There are two
'void' methods in _ParserContext that do not propagate exceptions, but
enabling that doesn't change anything. Any idea what this might mean?

Stefan



Exception AssertionError in <lxml.etree._ParserContext object at 0x96010d4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960109c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9601064>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960102c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621fb4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621f7c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621f44>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621e9c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621e64>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621f0c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621ed4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621e2c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621df4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621dbc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621d84>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621d4c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621d14>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621bc4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621bfc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9621c34>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x9623684>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960efb4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ef7c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ef44>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ef0c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eed4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ee9c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ee64>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ee2c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960edf4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960edbc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ed84>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ed4c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ed14>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ecdc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eca4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ec6c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ec34>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ebfc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ebc4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eb8c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eb54>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eb1c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eae4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960eaac>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ea74>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ea3c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960ea04>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e9cc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e994>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e95c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e924>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e8ec>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e8b4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e87c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e844>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e684>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e41c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e144>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e0d4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x960e09c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f6f4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f80c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f4c4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f684>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987fcdc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f6bc>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f56c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f79c>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987fdf4>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f614>
ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x987f5dc>
ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in  ignored
Exception AssertionError in <lxml.etree._ParserContext object at 0x94426bc>
ignored


From dagss at student.matnat.uio.no  Fri Feb 20 09:45:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 09:45:30 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E5EDD.2070800@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>
Message-ID: <499E6DAA.7020209@student.matnat.uio.no>

Stefan Behnel wrote:
> Stefan Behnel wrote:
>> I ran lxml's test suite with the refnanny and it looks mostly good, except
>> for a number of dropped AssertErrors and dropped unnamed(?) exceptions. It
>> doesn't say where they come from and I was unable to add any traceable
>> information to the output. There are two plain assert statements in the
>> refnanny code, but adding text to them doesn't change anything.
>>
>> I'll post the output here as soon as I get to it.
> 
> Quicker than I though. Looks like only two separate problems. There are two
> 'void' methods in _ParserContext that do not propagate exceptions, but
> enabling that doesn't change anything. Any idea what this might mean?

Robert, Michael, this applies to you as well.

I think I have an idea. The refnanny raises normal exceptions when 
something is violated; this is great for the smaller tests in the Cython 
test suite but might become a problem in programs.

There might be more situations where what you see happens, but this is 
one I can think of:

def f():
     # this function raises AssertError, and then
     # during exit of the function, a this exception is replaced
     # (overwritten) with a refnanny exception

try:
     f()
except:
     pass

The solution may be to change the way Cython/Runtime/refnanny.pyx 
reports errors, for instance change Context.delref to do a 
print/sys.exit(1) thing instead of raising an exception, and then 
reinstall Cython.

(Note that in f() above, the refcount error could happen before the 
AssertError is raised -- refnanny.pyx keeps track of all errors that 
happen in a function, and then raises/replaces an exception on function 
return.)

-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Feb 20 10:25:36 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 10:25:36 +0100 (CET)
Subject: [Cython] Status for 0.11
In-Reply-To: <499E6DAA.7020209@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
Message-ID: <38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> The solution may be to change the way Cython/Runtime/refnanny.pyx
> reports errors, for instance change Context.delref to do a
> print/sys.exit(1) thing instead of raising an exception

I would prefer a simple sys.__stderr__.write() with some helpful
information in a dedicated code line that you could point gdb to. Killing
the process sounds everything but helpful. Note that sys.exit() also just
raises an exception, as opposed to os._exit().

Stefan


From robertwb at math.washington.edu  Fri Feb 20 10:31:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 01:31:43 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>

On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:

> Dag Sverre Seljebotn wrote:
>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>> reports errors, for instance change Context.delref to do a
>> print/sys.exit(1) thing instead of raising an exception
>
> I would prefer a simple sys.__stderr__.write() with some helpful
> information in a dedicated code line that you could point gdb to.  
> Killing
> the process sounds everything but helpful. Note that sys.exit()  
> also just
> raises an exception, as opposed to os._exit().

+1. If memory is leaking, I'd rather have it tell me and move on. Of  
course if there are reference errors later on it may segfault, but at  
least I'd have useful information already.

- Robert



From dagss at student.matnat.uio.no  Fri Feb 20 10:50:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 10:50:27 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <499E7CE3.7010302@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>> reports errors, for instance change Context.delref to do a
>> print/sys.exit(1) thing instead of raising an exception
> 
> I would prefer a simple sys.__stderr__.write() with some helpful
> information in a dedicated code line that you could point gdb to. Killing
> the process sounds everything but helpful. Note that sys.exit() also just
> raises an exception, as opposed to os._exit().

I was referring to doing this for lxml and Sage testing only -- in which 
case you get to do whatever you like :-) So just go ahead and don't use 
sys.exit(), it was just an example, so that you could figure out what is 
going on, I didn't mean to change it like this permanently.

I would like to continue with the current behaviour for the Cython test 
suite, in order to get the nice test failures (though if you think it is 
a problem, one could have runtests.py inspect global variable in the 
refnanny module instead).

One could always have refnanny.pyx check sys.environ for some variable 
to see whether it should print to stderr or raise an exception? Or it 
could be controlled by a global variable in refnanny which runtests.py 
could change...


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 10:53:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 10:53:15 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E7CE3.7010302@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<499E7CE3.7010302@student.matnat.uio.no>
Message-ID: <499E7D8B.3060800@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>> reports errors, for instance change Context.delref to do a
>>> print/sys.exit(1) thing instead of raising an exception
>> I would prefer a simple sys.__stderr__.write() with some helpful
>> information in a dedicated code line that you could point gdb to. Killing
>> the process sounds everything but helpful. Note that sys.exit() also just
>> raises an exception, as opposed to os._exit().
> 
> I was referring to doing this for lxml and Sage testing only -- in which 
> case you get to do whatever you like :-) So just go ahead and don't use 

Huhhmm. Actually, you get to do whatever you like with anything in 
Cython. But you know what I meant... :-)

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Fri Feb 20 10:59:11 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 01:59:11 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
Message-ID: <499E7EEF.2020304@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>> reports errors, for instance change Context.delref to do a
>>> print/sys.exit(1) thing instead of raising an exception
>> I would prefer a simple sys.__stderr__.write() with some helpful
>> information in a dedicated code line that you could point gdb to.  
>> Killing
>> the process sounds everything but helpful. Note that sys.exit()  
>> also just
>> raises an exception, as opposed to os._exit().
> 
> +1. If memory is leaking, I'd rather have it tell me and move on. Of  
> course if there are reference errors later on it may segfault, but at  
> least I'd have useful information already.

Not surprisingly we get bitten by the widely discussed ctypedef sizt_t 
thingy:

# Memory management
cdef extern from "stdlib.h":
     ctypedef unsigned long size_t


> - Robert

Before I go off and make a fool of myself fixing various potential 
Cython issues in 3.3.rc2: Robert - do you have a patch or a list of 
known issues? If the above is the only problem I can probably handle it :)

Cheers,

Michael

> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From michael.abshoff at googlemail.com  Fri Feb 20 11:11:35 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 02:11:35 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
Message-ID: <499E81D7.4000906@gmail.com>

Ok, after commenting out all the various size_t declarations (in *5* 
places) I am stuck with this:

python2.5 `which cython` --embed-positions --incref-local-binop 
-I/scratch/mabshoff/cython-0.11-nanny/sage-3.3.rc2-sage.math-only-x86_64-Linux/devel/sage-main 
-o sage/matrix/matrix_integer_dense.c sage/matrix/matrix_integer_dense.pyx

Error converting Pyrex file to C:
------------------------------------------------------------
...
                 raise MemoryError, "out of memory allocating 
multi-modular coefficent list"
             for j from 0 <= j < self._ncols:
                 matrix[i][j] = mpz_fdiv_ui(self._matrix[i][j], n)

         cdef Linbox_modn_dense L = Linbox_modn_dense()
         L.set(n, matrix, self._nrows, self._ncols)
                       ^
------------------------------------------------------------

/scratch/mabshoff/cython-0.11-nanny/sage-3.3.rc2-sage.math-only-x86_64-Linux/devel/sage-main/sage/matrix/matrix_integer_dense.pyx:1161:23: 
Cannot assign type 'sage.matrix.matrix_integer_dense.mod_int **' to 
'sage.libs.linbox.linbox.mod_int **'


But I guess at this point this should move off the Cython list :)

Cheers,

Michael

PS: my patch so far, but I might have done something very stupid to 
cause the above issue :)


diff -r 43f267532b7a sage/combinat/matrices/dancing_links.pyx
--- a/sage/combinat/matrices/dancing_links.pyx	Tue Feb 17 17:00:05 2009 
-0800
+++ b/sage/combinat/matrices/dancing_links.pyx	Fri Feb 20 02:09:18 2009 
-0800
@@ -25,8 +25,8 @@
  include "../../ext/python_int.pxi"
  include "../../ext/python_ref.pxi"

-cdef extern from "stdlib.h":
-     ctypedef unsigned long size_t
+#cdef extern from "stdlib.h":
+#     ctypedef unsigned long size_t

  cdef extern from "dancing_links_c.h":
      ctypedef struct vector_int "std::vector<int>":
diff -r 43f267532b7a sage/ext/cdefs.pxi
--- a/sage/ext/cdefs.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/ext/cdefs.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -2,7 +2,7 @@
  include "python.pxi"

  cdef extern from "stdlib.h":
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t
      void free(void *ptr)
      void *malloc(size_t size)
      void *realloc(void *ptr, size_t size)
diff -r 43f267532b7a sage/ext/python_mem.pxi
--- a/sage/ext/python_mem.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/ext/python_mem.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -1,5 +1,5 @@
  cdef extern from "Python.h":
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t

      #####################################################################
      # 9.2 Memory Interface
diff -r 43f267532b7a sage/ext/stdsage.pxi
--- a/sage/ext/stdsage.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/ext/stdsage.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -39,8 +39,8 @@


  # Memory management
-cdef extern from "stdlib.h":
-    ctypedef unsigned long size_t
+#cdef extern from "stdlib.h":
+#    ctypedef unsigned long size_t

  cdef extern from "stdsage.h":
      void  sage_free(void *p)
diff -r 43f267532b7a sage/libs/gmp/types.pxd
--- a/sage/libs/gmp/types.pxd	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/libs/gmp/types.pxd	Fri Feb 20 02:09:18 2009 -0800
@@ -3,7 +3,7 @@
      ### Type Declarations ###

      # Underlying typedefs
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t
      ctypedef unsigned long mp_limb_t
      ctypedef long mp_size_t
      ctypedef long mp_exp_t
diff -r 43f267532b7a sage/libs/pari/decl.pxi
--- a/sage/libs/pari/decl.pxi	Tue Feb 17 17:00:05 2009 -0800
+++ b/sage/libs/pari/decl.pxi	Fri Feb 20 02:09:18 2009 -0800
@@ -1,7 +1,7 @@
  include '../../ext/cdefs.pxi'

  cdef extern from "stdlib.h":
-    ctypedef unsigned long size_t
+    #ctypedef unsigned long size_t
      void free(void *ptr)
      void *malloc(size_t size)
      void *realloc(void *ptr, size_t size)


From robertwb at math.washington.edu  Fri Feb 20 11:45:02 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 02:45:02 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E7EEF.2020304@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
Message-ID: <E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>

On Feb 20, 2009, at 1:59 AM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>>> reports errors, for instance change Context.delref to do a
>>>> print/sys.exit(1) thing instead of raising an exception
>>> I would prefer a simple sys.__stderr__.write() with some helpful
>>> information in a dedicated code line that you could point gdb to.
>>> Killing
>>> the process sounds everything but helpful. Note that sys.exit()
>>> also just
>>> raises an exception, as opposed to os._exit().
>>
>> +1. If memory is leaking, I'd rather have it tell me and move on. Of
>> course if there are reference errors later on it may segfault, but at
>> least I'd have useful information already.
>
> Not surprisingly we get bitten by the widely discussed ctypedef sizt_t
> thingy:
>
> # Memory management
> cdef extern from "stdlib.h":
>      ctypedef unsigned long size_t
>
>
>> - Robert
>
> Before I go off and make a fool of myself fixing various potential
> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
> known issues? If the above is the only problem I can probably  
> handle it :)

I'll make a 0.11 ticket and post a patch. I got sage to compile, but  
it fails to start up.

- Robert


From robertwb at math.washington.edu  Fri Feb 20 11:48:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 02:48:38 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>
	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
Message-ID: <703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>

On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:

> On Feb 20, 2009, at 1:59 AM, Michael Abshoff wrote:
>
>> Robert Bradshaw wrote:
>>> On Feb 20, 2009, at 1:25 AM, Stefan Behnel wrote:
>>>
>>>> Dag Sverre Seljebotn wrote:
>>>>> The solution may be to change the way Cython/Runtime/refnanny.pyx
>>>>> reports errors, for instance change Context.delref to do a
>>>>> print/sys.exit(1) thing instead of raising an exception
>>>> I would prefer a simple sys.__stderr__.write() with some helpful
>>>> information in a dedicated code line that you could point gdb to.
>>>> Killing
>>>> the process sounds everything but helpful. Note that sys.exit()
>>>> also just
>>>> raises an exception, as opposed to os._exit().
>>>
>>> +1. If memory is leaking, I'd rather have it tell me and move on. Of
>>> course if there are reference errors later on it may segfault,  
>>> but at
>>> least I'd have useful information already.
>>
>> Not surprisingly we get bitten by the widely discussed ctypedef  
>> sizt_t
>> thingy:
>>
>> # Memory management
>> cdef extern from "stdlib.h":
>>      ctypedef unsigned long size_t
>>
>>
>>> - Robert
>>
>> Before I go off and make a fool of myself fixing various potential
>> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
>> known issues? If the above is the only problem I can probably
>> handle it :)
>
> I'll make a 0.11 ticket and post a patch. I got sage to compile, but
> it fails to start up.

http://trac.sagemath.org/sage_trac/ticket/4987



From michael.abshoff at googlemail.com  Fri Feb 20 12:00:21 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 03:00:21 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
Message-ID: <499E8D45.6020605@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:

<SNIP>

Hi Robert,

>>> Before I go off and make a fool of myself fixing various potential
>>> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
>>> known issues? If the above is the only problem I can probably
>>> handle it :)
>> I'll make a 0.11 ticket and post a patch. I got sage to compile, but
>> it fails to start up.
> 
> http://trac.sagemath.org/sage_trac/ticket/4987

Ahh, I see there was another size_t hiding in LinBox, I guess I should 
have at least looked into that file before giving up.

I have to get 3.3.rc3 out the door and will take your patch later today 
and see of I can figure out what the problem is.

Cheers,

Michael

> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Fri Feb 20 12:06:21 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 03:06:21 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E8D45.6020605@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
Message-ID: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>

On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:
>
> <SNIP>
>
> Hi Robert,
>
>>>> Before I go off and make a fool of myself fixing various potential
>>>> Cython issues in 3.3.rc2: Robert - do you have a patch or a list of
>>>> known issues? If the above is the only problem I can probably
>>>> handle it :)
>>> I'll make a 0.11 ticket and post a patch. I got sage to compile, but
>>> it fails to start up.
>>
>> http://trac.sagemath.org/sage_trac/ticket/4987
>
> Ahh, I see there was another size_t hiding in LinBox, I guess I should
> have at least looked into that file before giving up.

Yeah. I'm really happy to have native size_t now.

> I have to get 3.3.rc3 out the door and will take your patch later  
> today
> and see of I can figure out what the problem is.

Here's the log. (I made it print rather than raise an error). I'm  
going to try and make it print the source file as well as line.

----------------------------------------------------------------------
| Sage Version 3.2.3, Release Date: 2009-01-05                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
References leaked:
   Acquired on lines: 4278, 4287, 4345, 4354, 4412, 4421
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 7564
Too many decrefs on line 7577, reference acquired on lines []
NULL argument on line 42791
Too many decrefs on line 42813, reference acquired on lines []
NULL argument on line 1596
Too many decrefs on line 1617, reference acquired on lines []
NULL argument on line 1924
Too many decrefs on line 1948, reference acquired on lines []
NULL argument on line 819
Too many decrefs on line 846, reference acquired on lines []
NULL argument on line 1596
Too many decrefs on line 1617, reference acquired on lines []
NULL argument on line 819
Too many decrefs on line 846, reference acquired on lines []
NULL argument on line 1596
Too many decrefs on line 1617, reference acquired on lines []
NULL argument on line 1924
Too many decrefs on line 1940, reference acquired on lines []
NULL argument on line 819
Too many decrefs on line 846, reference acquired on lines []
References leaked:
   Acquired on lines: 1048, 1114
   Acquired on lines: 1049, 1115
   Acquired on lines: 1050, 1116
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8400, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 11296
Too many decrefs on line 11317, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8540
Too many decrefs on line 8561, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8562
Too many decrefs on line 8585, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10437
Too many decrefs on line 10458, reference acquired on lines []
NULL argument on line 7421
Too many decrefs on line 7434, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8406, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10869
Too many decrefs on line 10882, reference acquired on lines []
NULL argument on line 5211
Too many decrefs on line 5233, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10869
Too many decrefs on line 10882, reference acquired on lines []
NULL argument on line 5211
Too many decrefs on line 5233, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 10869
Too many decrefs on line 10882, reference acquired on lines []
NULL argument on line 5211
Too many decrefs on line 5233, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 7103
Too many decrefs on line 7116, reference acquired on lines []
NULL argument on line 42088
Too many decrefs on line 42110, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8562
Too many decrefs on line 8585, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8406, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 7421
Too many decrefs on line 7434, reference acquired on lines []
NULL argument on line 8393
Too many decrefs on line 8406, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8045, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 13813
Too many decrefs on line 13826, reference acquired on lines []
NULL argument on line 5157
Too many decrefs on line 5178, reference acquired on lines []
NULL argument on line 6200
Too many decrefs on line 6221, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4775, reference acquired on lines []
NULL argument on line 4760
Too many decrefs on line 4766, reference acquired on lines []
NULL argument on line 1361
Too many decrefs on line 1382, reference acquired on lines []
NULL argument on line 8032
Too many decrefs on line 8053, reference acquired on lines []
NULL argument on line 8916
Too many decrefs on line 8937, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
NULL argument on line 2800
Too many decrefs on line 2831, reference acquired on lines []
NULL argument on line 8182
Too many decrefs on line 8213, reference acquired on lines []
NULL argument on line 11428
Too many decrefs on line 11459, reference acquired on lines []
sage:
Exiting SAGE (CPU time 0m0.06s, Wall time 0m1.19s).
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/ 
local/bin/sage-sage: line 210:  6932 Bus error               sage- 
ipython "$@" -i


From michael.abshoff at googlemail.com  Fri Feb 20 12:21:29 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 03:21:29 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
Message-ID: <499E9239.5030205@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:

<SNIP>

Hi Robert,

>> Ahh, I see there was another size_t hiding in LinBox, I guess I should
>> have at least looked into that file before giving up.
> 
> Yeah. I'm really happy to have native size_t now.

Yep, I never expected that there were six different definitions of 
size_t in Sage.

>> I have to get 3.3.rc3 out the door and will take your patch later  
>> today
>> and see of I can figure out what the problem is.
> 
> Here's the log. (I made it print rather than raise an error). I'm  
> going to try and make it print the source file as well as line.

Ok, that would be *extremely* nice.

> ----------------------------------------------------------------------
> | Sage Version 3.2.3, Release Date: 2009-01-05                       |
> | Type notebook() for the GUI, and license() for information.        |
> ----------------------------------------------------------------------

<SNIP>

> NULL argument on line 819
> Too many decrefs on line 846, reference acquired on lines []
> References leaked:
>    Acquired on lines: 1048, 1114
>    Acquired on lines: 1049, 1115
>    Acquired on lines: 1050, 1116

Excellent, this is catching real bugs.

Dag: Next time I meet you in person dinner is on me for the refanny :)

<SNIP>

> sage:
> Exiting SAGE (CPU time 0m0.06s, Wall time 0m1.19s).
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/ 
> local/bin/sage-sage: line 210:  6932 Bus error               sage- 
> ipython "$@" -i

I am afraid what we will find when we run this code on the whole doctest 
suite, but hopefully this will just kill a whole class of bugs in Sage :)

I can even see this happening per default as long as I merge patches.

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dagss at student.matnat.uio.no  Fri Feb 20 12:51:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 12:51:09 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
Message-ID: <499E992D.6040600@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>> I have to get 3.3.rc3 out the door and will take your patch later  
>> today
>> and see of I can figure out what the problem is.
> 
> Here's the log. (I made it print rather than raise an error). I'm  
> going to try and make it print the source file as well as line.

Hehe, not much use without :-)

> Too many decrefs on line 4775, reference acquired on lines []

Be especially careful that errors like this may easily come from missing 
GOTREFs rather than a refcount error. I don't know whether I have 
GOTREF/GIVEREF everywhere they are needed, just that they are everywhere 
were needed within what we have coverage for.

> NULL argument on line 4760

Furthermore, if this is a GOTREF or GIVEREF, it may be that the refnanny 
part should be altered to XGOTREF/XGIVEREF for that situation.

> References leaked:
>    Acquired on lines: 1048, 1114
>    Acquired on lines: 1049, 1115
>    Acquired on lines: 1050, 1116

This kind of output could be clearer, but it means that three different 
objects were leaked in the function, each of which were incref-ed or 
gotref-ed two times.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 12:57:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 12:57:01 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499E9239.5030205@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E9239.5030205@gmail.com>
Message-ID: <499E9A8D.6030003@student.matnat.uio.no>

Michael Abshoff wrote:
> Robert Bradshaw wrote:
>> References leaked:
>>    Acquired on lines: 1048, 1114
>>    Acquired on lines: 1049, 1115
>>    Acquired on lines: 1050, 1116
> 
> Excellent, this is catching real bugs.

Well, the error message is not that clear -- it may also be that the 
function legitimately handed off a reference to somewhere else, but 
didn't declare that it did with a GIVEREF.

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Fri Feb 20 12:56:51 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 03:56:51 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E9A8D.6030003@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E9239.5030205@gmail.com>
	<499E9A8D.6030003@student.matnat.uio.no>
Message-ID: <499E9A83.2030106@gmail.com>

Dag Sverre Seljebotn wrote:
> Michael Abshoff wrote:

Hi Dag,

>> Robert Bradshaw wrote:
>>> References leaked:
>>>    Acquired on lines: 1048, 1114
>>>    Acquired on lines: 1049, 1115
>>>    Acquired on lines: 1050, 1116
>> Excellent, this is catching real bugs.
> 
> Well, the error message is not that clear -- it may also be that the 
> function legitimately handed off a reference to somewhere else, but 
> didn't declare that it did with a GIVEREF.
> 

Sure. Note that technically I didn't claim those were bugs in Sage :)

I just think that the refnanny and Sage will be good tests for each other.

Cheers,

Michael

From robertwb at math.washington.edu  Fri Feb 20 13:16:30 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 04:16:30 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499E992D.6040600@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E992D.6040600@student.matnat.uio.no>
Message-ID: <F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>

On Feb 20, 2009, at 3:51 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>>> I have to get 3.3.rc3 out the door and will take your patch later
>>> today
>>> and see of I can figure out what the problem is.
>>
>> Here's the log. (I made it print rather than raise an error). I'm
>> going to try and make it print the source file as well as line.
>
> Hehe, not much use without :-)

Full log at http://trac.sagemath.org/sage_trac/ticket/4987  
Fortunately, lots are the same error over and over. BTW, I don't know  
if printing them out like this could simplify a lot of your manual  
exception handling code.

>> Too many decrefs on line 4775, reference acquired on lines []
>
> Be especially careful that errors like this may easily come from  
> missing
> GOTREFs rather than a refcount error. I don't know whether I have
> GOTREF/GIVEREF everywhere they are needed, just that they are  
> everywhere
> were needed within what we have coverage for.

Yep. Either way, we should track them down.

>> NULL argument on line 4760
>
> Furthermore, if this is a GOTREF or GIVEREF, it may be that the  
> refnanny
> part should be altered to XGOTREF/XGIVEREF for that situation.

Yeah, it could be the case.

>> References leaked:
>>    Acquired on lines: 1048, 1114
>>    Acquired on lines: 1049, 1115
>>    Acquired on lines: 1050, 1116
>
> This kind of output could be clearer, but it means that three  
> different
> objects were leaked in the function, each of which were incref-ed or
> gotref-ed two times.

Thanks for the explanation.

- Robert

From dagss at student.matnat.uio.no  Fri Feb 20 14:01:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 14:01:53 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>
	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
Message-ID: <499EA9C1.9000308@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 3:51 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>>>> I have to get 3.3.rc3 out the door and will take your patch later
>>>> today
>>>> and see of I can figure out what the problem is.
>>> Here's the log. (I made it print rather than raise an error). I'm
>>> going to try and make it print the source file as well as line.
>> Hehe, not much use without :-)
> 
> Full log at http://trac.sagemath.org/sage_trac/ticket/4987  
> Fortunately, lots are the same error over and over. BTW, I don't know  
> if printing them out like this could simplify a lot of your manual  
> exception handling code.

Not really. One may enter refnanny.pyx during exception cleanup while an 
exception is set, and then refnanny.pyx goes on to doing lots of Python 
operations -- the first error flag check within refnanny.pyx will cause 
the execution stack to pop even if there wasn't a problem within 
refnanny.pyx itself.

It could however simplify the four lines at the end of every function in 
Cython-generated C code. Personally I prefer to have the testcases 
listed as "failed" when run from runtests.py (and even be able to flag 
refcount errors as expected in doctests, if we ever need that), but I 
don't have a strong opinion either way.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 14:14:19 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 14:14:19 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <499EA9C1.9000308@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
Message-ID: <499EACAB.4010502@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> It could however simplify the four lines at the end of every function in 
> Cython-generated C code. Personally I prefer to have the testcases 
> listed as "failed" when run from runtests.py (and even be able to flag 
> refcount errors as expected in doctests, if we ever need that), but I 
> don't have a strong opinion either way.

OK I see where you are coming from now and have come to agree with you 
instead.

http://trac.cython.org/cython_trac/ticket/212

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb 20 14:31:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 14:31:12 +0100
Subject: [Cython] size_t/#207
Message-ID: <499EB0A0.60602@student.matnat.uio.no>

Is this now fixed?

http://trac.cython.org/cython_trac/ticket/207

If not, I guess it should get 0.11 milestone as size_t is now introduced?


-- 
Dag Sverre

From dalcinl at gmail.com  Fri Feb 20 14:57:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 20 Feb 2009 11:57:28 -0200
Subject: [Cython] size_t/#207
In-Reply-To: <499EB0A0.60602@student.matnat.uio.no>
References: <499EB0A0.60602@student.matnat.uio.no>
Message-ID: <e7ba66e40902200557w1cbc36d2ie54ac80646229e85@mail.gmail.com>

Wait me a bit more of time, I'll push a fix in a few minutes to enable
backward compatibility. Next we can discuss how to emit warnings about
'ctypeded unsigned long size_t'

On Fri, Feb 20, 2009 at 11:31 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Is this now fixed?
>
> http://trac.cython.org/cython_trac/ticket/207
>
> If not, I guess it should get 0.11 milestone as size_t is now introduced?
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Feb 20 15:42:56 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 20 Feb 2009 12:42:56 -0200
Subject: [Cython] please close ticket #207,
	'size_t' support finished (and backward 	compatible)
Message-ID: <e7ba66e40902200642m520b9db5lf1f8459d150ecd68@mail.gmail.com>

After my last push, ticket #207 can be closed. Now Cython treats
Py_ssize_t/size_t as basic (but special) C types.

Moreover, the former practice of doing:

cdef extern from *:
    ctypedef unsigned long size_t

would be simply ignored. This is IMHO good for backward compatibility
with previous Cython released and also Pyrex. Of course, in the near
future we issue warnings at CTypedefNode.analize_declarations() (an
not only for size_t !!!).


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Feb 20 20:22:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 20:22:52 +0100
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>
Message-ID: <499F030C.4060400@behnel.de>

Hi Lisandro,

I get a couple of these with your latest change when compiling Cython
itself. Could you try to come up with a fix?

Thanks,

Stefan


Compiling module Cython.Plex.Scanners ...

Compiling module Cython.Compiler.Scanning ...

Compiling module Cython.Compiler.Parsing ...


Error converting Pyrex file to C:
------------------------------------------------------------
...

    bint PyObject_TypeCheck(object o, PyTypeObject *type)
    # Return true if the object o is of type type or a subtype of
    # type. Both parameters must be non-NULL.

    Py_ssize_t PyObject_Length(object o)
   ^
------------------------------------------------------------

/home/sbehnel/source/Python/cython/cython-work/Cython/Includes/python_object.pxd:240:4:
Unrecognised type modifier combination

Error converting Pyrex file to C:
------------------------------------------------------------
...
    bint PyObject_TypeCheck(object o, PyTypeObject *type)
    # Return true if the object o is of type type or a subtype of
    # type. Both parameters must be non-NULL.

    Py_ssize_t PyObject_Length(object o)
    Py_ssize_t PyObject_Size(object o)
   ^
------------------------------------------------------------

/home/sbehnel/source/Python/cython/cython-work/Cython/Includes/python_object.pxd:241:4:
Unrecognised type modifier combination
Compilation failed



From robertwb at math.washington.edu  Fri Feb 20 21:12:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 12:12:46 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499EACAB.4010502@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
	<499EACAB.4010502@student.matnat.uio.no>
Message-ID: <C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>


On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> It could however simplify the four lines at the end of every  
>> function in
>> Cython-generated C code. Personally I prefer to have the testcases
>> listed as "failed" when run from runtests.py (and even be able to  
>> flag
>> refcount errors as expected in doctests, if we ever need that), but I
>> don't have a strong opinion either way.

If it prints stuff out, it will still be listed as failed, right?

> OK I see where you are coming from now and have come to agree with you
> instead.
>
> http://trac.cython.org/cython_trac/ticket/212

I did push a bit of a patch in this direction

http://hg.cython.org/cython-devel/rev/dbb1a4421f5d

- Robert


From stefan_ml at behnel.de  Fri Feb 20 21:40:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 21:40:50 +0100
Subject: [Cython] PATH: special-case Py_ssize_t and size_t in the parser
In-Reply-To: <e7ba66e40902201230k494b2d1fn9266fdc999d7f012@mail.gmail.com>
References: <e7ba66e40902181239x40131f40g499fcd0c46398372@mail.gmail.com>	
	<499F030C.4060400@behnel.de>
	<e7ba66e40902201230k494b2d1fn9266fdc999d7f012@mail.gmail.com>
Message-ID: <499F1552.8040504@behnel.de>


Lisandro Dalcin wrote:
> On Fri, Feb 20, 2009 at 5:22 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> I get a couple of these with your latest change when compiling Cython
>> itself. Could you try to come up with a fix?
>>
> Stefan, I sincerely do not know what's going on here... I DO NOT get
> these errors!! Cython builds itself just fine in my box ...

Argh, sorry, my fault. I forgot to say "make clean" before building, so it
tried to build using outdated binary modules.

Well done, just keep going, nothing to see here ...

Stefan

From stefan_ml at behnel.de  Fri Feb 20 21:52:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Feb 2009 21:52:11 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<499E992D.6040600@student.matnat.uio.no>	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>	<499EA9C1.9000308@student.matnat.uio.no>	<499EACAB.4010502@student.matnat.uio.no>
	<C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
Message-ID: <499F17FB.10908@behnel.de>

Hi,

Robert Bradshaw wrote:
> On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> It could however simplify the four lines at the end of every  
>>> function in
>>> Cython-generated C code. Personally I prefer to have the testcases
>>> listed as "failed" when run from runtests.py (and even be able to  
>>> flag
>>> refcount errors as expected in doctests, if we ever need that), but I
>>> don't have a strong opinion either way.
> 
> If it prints stuff out, it will still be listed as failed, right?

Sure, she just needs to keep her mouth shut when all is ok, and burst stuff
into Python's stdout when things go wrong. That will make the doctests fail
for free. That way it's trivial to test for output in the doctests as well,
e.g. you could enforce a leak on purpose and expect the nanny's cry as output.

Stefan


From michael.f.ellis at gmail.com  Fri Feb 20 22:02:26 2009
From: michael.f.ellis at gmail.com (Mike Ellis)
Date: Fri, 20 Feb 2009 21:02:26 +0000 (UTC)
Subject: [Cython] C API declarations
References: <4974E631.10309@simula.no>
Message-ID: <loom.20090220T205230-146@post.gmane.org>


Ilmar Wilbers <ilmarw at ...> writes:

> 
> Hi,
> 
> I am having some serious problems with the C API declarations, chapter 
> 7.2.2 i n the manual.
> 
> Using the exact two example files does not work for me, I keep getting:
> marty.c:3: error: expected ?=?, ?,?, ?;?, ?asm? or ?__attribute__? 
> before ?car?
> marty.c: In function ?main?:
> marty.c:7: error: ?car? undeclared (first use in this function)
> marty.c:7: error: (Each undeclared identifier is reported only once
> marty.c:7: error: for each function it appears in.)
> 
<snip>
> Is there a problem with the header files that I don't understand? I run 
> Cython on delorean.pyx before running
> gcc -c -I/usr/include/python2.5 -lpython2.5 marty.c
> 
> Has anyone tried these examples lately?
> 
> Sincerly, Ilmar
> 

I ran into the same problem with the delorean example.  I can get marty.c to
compile by changing

Vehicle car;

to 

struct Vehicle car;

Getting it to run is another story. It dies deep inside import_delorean(). 
Heres a trace from gdb:

Program received signal SIGSEGV, Segmentation fault.
0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0xb7ea7ac7 in PyEval_GetGlobals () from /usr/lib/libpython2.5.so.1.0
#2  0xb7ec9187 in PyImport_Import () from /usr/lib/libpython2.5.so.1.0
#3  0x08048797 in __Pyx_ImportModule (name=0x8048bc7 "delorean")
    at delorean_api.h:24
#4  0x08048984 in import_delorean () at delorean_api.h:73
#5  0x08048a3e in main (argc=3, argv=0xbfdbfa34) at marty.c:6

I'm building with gcc (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu3) using the same command
line as Ilmar.

Anyone know what might be going wrong?

Thanks,
Mike




From dagss at student.matnat.uio.no  Fri Feb 20 22:21:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 22:21:05 +0100 (CET)
Subject: [Cython] Status for 0.11
In-Reply-To: <499F17FB.10908@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E992D.6040600@student.matnat.uio.no>
	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
	<499EACAB.4010502@student.matnat.uio.no>
	<C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
	<499F17FB.10908@behnel.de>
Message-ID: <690acc13b787caf0bdb159655a5863e3.squirrel@webmail.uio.no>

Stefan wrote:
> Hi,
>
> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> It could however simplify the four lines at the end of every
>>>> function in
>>>> Cython-generated C code. Personally I prefer to have the testcases
>>>> listed as "failed" when run from runtests.py (and even be able to
>>>> flag
>>>> refcount errors as expected in doctests, if we ever need that), but I
>>>> don't have a strong opinion either way.
>>
>> If it prints stuff out, it will still be listed as failed, right?
>
> Sure, she just needs to keep her mouth shut when all is ok, and burst
> stuff
> into Python's stdout when things go wrong. That will make the doctests
> fail
> for free. That way it's trivial to test for output in the doctests as

Well, not if you expect exceptions in the doctests... (I hate that aspect
of doctest myself; I'

Dag Sverre


From dagss at student.matnat.uio.no  Fri Feb 20 22:24:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Feb 2009 22:24:10 +0100 (CET)
Subject: [Cython] Status for 0.11
In-Reply-To: <499F17FB.10908@behnel.de>
References: <499E0057.9040504@student.matnat.uio.no>
	<499E5CDE.6000804@behnel.de> <499E5EDD.2070800@behnel.de>
	<499E6DAA.7020209@student.matnat.uio.no>
	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>
	<499E7EEF.2020304@gmail.com>
	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<499E992D.6040600@student.matnat.uio.no>
	<F495BD43-4264-441A-96D3-7A5F5F2FF59B@math.washington.edu>
	<499EA9C1.9000308@student.matnat.uio.no>
	<499EACAB.4010502@student.matnat.uio.no>
	<C8A8B658-1E58-4191-87AF-D66C1EF36F19@math.washington.edu>
	<499F17FB.10908@behnel.de>
Message-ID: <50f1fe2a55831242182c36b6e0fc6408.squirrel@webmail.uio.no>

Stefan wrote:
> Hi,
>
> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 5:14 AM, Dag Sverre Seljebotn wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> It could however simplify the four lines at the end of every
>>>> function in
>>>> Cython-generated C code. Personally I prefer to have the testcases
>>>> listed as "failed" when run from runtests.py (and even be able to
>>>> flag
>>>> refcount errors as expected in doctests, if we ever need that), but I
>>>> don't have a strong opinion either way.
>>
>> If it prints stuff out, it will still be listed as failed, right?
>
> Sure, she just needs to keep her mouth shut when all is ok, and burst
> stuff
> into Python's stdout when things go wrong. That will make the doctests

If an exception is expected from the function, doctest will not compare
the output printed before the exception happens (and I really dislike that
aspect of doctest). So in those cases, one could have the refnanny
complain without a failure.

Of course, you'll usually pay enough attention to see the output anyway,
so it's not a big deal.

Dag Sverre


From robertwb at math.washington.edu  Sat Feb 21 00:08:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 15:08:40 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
Message-ID: <A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>

On Feb 20, 2009, at 3:06 AM, Robert Bradshaw wrote:

> On Feb 20, 2009, at 3:00 AM, Michael Abshoff wrote:
>
>> Robert Bradshaw wrote:
>>> On Feb 20, 2009, at 2:45 AM, Robert Bradshaw wrote:
>>
>> <SNIP>
>>
>> Hi Robert,
>>
>>>>> Before I go off and make a fool of myself fixing various potential
>>>>> Cython issues in 3.3.rc2: Robert - do you have a patch or a  
>>>>> list of
>>>>> known issues? If the above is the only problem I can probably
>>>>> handle it :)
>>>> I'll make a 0.11 ticket and post a patch. I got sage to compile,  
>>>> but
>>>> it fails to start up.
>>>
>>> http://trac.sagemath.org/sage_trac/ticket/4987
>>
>> Ahh, I see there was another size_t hiding in LinBox, I guess I  
>> should
>> have at least looked into that file before giving up.
>
> Yeah. I'm really happy to have native size_t now.
>
>> I have to get 3.3.rc3 out the door and will take your patch later
>> today
>> and see of I can figure out what the problem is.
>
> Here's the log. (I made it print rather than raise an error). I'm
> going to try and make it print the source file as well as line.

One little fix in cpdef functions fixed almost all the issues. We're  
down to

----------------------------------------------------------------------
| Sage Version 3.2.3, Release Date: 2009-01-05                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage/rings/padics/padic_printing.cpp: __init__()
References leaked:
   Acquired on lines: 4278, 4287, 4345, 4354, 4412, 4421
sage/structure/factory.c: get_object()
References leaked:
   Acquired on lines: 1048, 1114
   Acquired on lines: 1049, 1115
   Acquired on lines: 1050, 1116
sage:
Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in  ignored
Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/ 
local/bin/sage-sage: line 587: 16443 Bus error               sage- 
ipython "$@" -i

- Robert


From robertwb at math.washington.edu  Sat Feb 21 07:44:14 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 22:44:14 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>
	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>
	<499E8D45.6020605@gmail.com>
	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>
	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
Message-ID: <6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>

I found and fixed two memory leaks. This refcounting nanny sure is  
nice. Sage now starts up without any warnings, but upon exiting I  
still get

> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in  ignored
> Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/
> local/bin/sage-sage: line 587: 16443 Bus error               sage-
> ipython "$@" -i

Next up is running the test suite.

- Robert

From michael.abshoff at googlemail.com  Sat Feb 21 07:56:14 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 22:56:14 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>
Message-ID: <499FA58E.9010303@gmail.com>

Robert Bradshaw wrote:

Hi Robert,

> I found and fixed two memory leaks. 

Got a patch for me? I could still get that one into 3.3 :)

> This refcounting nanny sure is  
> nice. Sage now starts up without any warnings, but upon exiting I  
> still get
> 
>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>> Exception exceptions.AssertionError in  ignored
>> Exception exceptions.AssertionError in  ignored
>> Exception exceptions.AssertionError in /Users/robert/sage/sage-cython/
>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>> ipython "$@" -i
> 
> Next up is running the test suite.

Cool, looking forward to all the fixes coming out of that.

> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From robertwb at math.washington.edu  Sat Feb 21 07:57:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 20 Feb 2009 22:57:45 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <499FA58E.9010303@gmail.com>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>
	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>
	<499FA58E.9010303@gmail.com>
Message-ID: <431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>

On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>
> Hi Robert,
>
>> I found and fixed two memory leaks.
>
> Got a patch for me? I could still get that one into 3.3 :)

Memory leaks in Cython, not in Sage. (Actually, probably not  
noticeable ones, one was leaking an exception, and the other was  
"leaking" the values True and False in most cases.

>> This refcounting nanny sure is
>> nice. Sage now starts up without any warnings, but upon exiting I
>> still get
>>
>>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>>> Exception exceptions.AssertionError in  ignored
>>> Exception exceptions.AssertionError in  ignored
>>> Exception exceptions.AssertionError in /Users/robert/sage/sage- 
>>> cython/
>>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>>> ipython "$@" -i
>>
>> Next up is running the test suite.
>
> Cool, looking forward to all the fixes coming out of that.

I'm going to run it overnight. Given that Sage doesn't exit cleanly  
(not sure what's up with that), I expect everything to fail, but  
hopefully it'll still be useful.

- Robert


From michael.abshoff at googlemail.com  Sat Feb 21 08:17:36 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Fri, 20 Feb 2009 23:17:36 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>
	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
Message-ID: <499FAA90.5020505@gmail.com>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:

Hi Robert,

>>> I found and fixed two memory leaks.
>> Got a patch for me? I could still get that one into 3.3 :)
> 
> Memory leaks in Cython, not in Sage. (Actually, probably not  
> noticeable ones, one was leaking an exception, and the other was  
> "leaking" the values True and False in most cases.

Well, we had 8 byte leaks in Sage that ate up all available RAM on the 
old sage.math in about two or three days, i.e. leaking about 1 to 2 
GB/hour, so if you hit them often enough even tiny leaks have a giant 
impact. And I have some cases where I do hope that the refnanny will 
point out ref count issues and explain leaks where I have been totally 
unable to pin point a cause in Sage code itself. Getting Sage to start 
up and exit clean will be great, but the test suite will likely expose 
many more issues :)

<SNIP>

> I'm going to run it overnight. Given that Sage doesn't exit cleanly  
> (not sure what's up with that), I expect everything to fail, but  
> hopefully it'll still be useful.

Sure. I took today's cython spkg and I am valgrinding Sage with it, so 
maybe I can give you some pointer what is going wrong.

> - Robert

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 


From dalcinl at gmail.com  Mon Feb  2 19:20:30 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 02 Feb 2009 18:20:30 -0000
Subject: [Cython] possible problems with temp allocation?
Message-ID: <e7ba66e40902021018j50faa276gccd1fbb325edb9c3@mail.gmail.com>

See the attached files. The line 452 in generated foo.c (attached just
for your convenience) seems to be wrong. Furthermore, valgrind pointed
me to that line.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.pyx
Type: application/octet-stream
Size: 319 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/ade53f48/attachment-0001.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.c
Type: text/x-csrc
Size: 47242 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090202/ade53f48/attachment-0001.c 

From dagss at student.matnat.uio.no  Sat Feb 21 16:26:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 16:26:26 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>
	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
Message-ID: <49A01D22.3070504@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:
> 
>> Robert Bradshaw wrote:
>>
>> Hi Robert,
>>
>>> I found and fixed two memory leaks.
>> Got a patch for me? I could still get that one into 3.3 :)
> 
> Memory leaks in Cython, not in Sage. (Actually, probably not  
> noticeable ones, one was leaking an exception, and the other was  
> "leaking" the values True and False in most cases.
> 
>>> This refcounting nanny sure is
>>> nice. Sage now starts up without any warnings, but upon exiting I
>>> still get
>>>
>>>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>>>> Exception exceptions.AssertionError in  ignored
>>>> Exception exceptions.AssertionError in  ignored
>>>> Exception exceptions.AssertionError in /Users/robert/sage/sage- 
>>>> cython/
>>>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>>>> ipython "$@" -i
>>> Next up is running the test suite.
>> Cool, looking forward to all the fixes coming out of that.
> 
> I'm going to run it overnight. Given that Sage doesn't exit cleanly  
> (not sure what's up with that), I expect everything to fail, but  
> hopefully it'll still be useful.

Don't work too hard on it yet -- there's some bugs in the refnanny now 
which I should fix, and which may produce these kinds of errors.

One suspect is here:

http://hg.cython.org/cython-devel/rev/4e19b8a61272

There are some more as well which could lead to similar symptoms which 
I'll have to wait with pushing -- ironically, the fix itself seems to 
produce a temp handling refcount bug on the refnanny.c code itself, so I 
must fix that simultaneously.

-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Feb 21 16:30:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 21 Feb 2009 07:30:53 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <49A01D22.3070504@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>
	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>
	<49A01D22.3070504@student.matnat.uio.no>
Message-ID: <AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>

On Feb 21, 2009, at 7:26 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 20, 2009, at 10:56 PM, Michael Abshoff wrote:
>>
>>> Robert Bradshaw wrote:
>>>
>>> Hi Robert,
>>>
>>>> I found and fixed two memory leaks.
>>> Got a patch for me? I could still get that one into 3.3 :)
>>
>> Memory leaks in Cython, not in Sage. (Actually, probably not
>> noticeable ones, one was leaking an exception, and the other was
>> "leaking" the values True and False in most cases.
>>
>>>> This refcounting nanny sure is
>>>> nice. Sage now starts up without any warnings, but upon exiting I
>>>> still get
>>>>
>>>>> Exiting SAGE (CPU time 0m0.07s, Wall time 2m13.31s).
>>>>> Exception exceptions.AssertionError in  ignored
>>>>> Exception exceptions.AssertionError in  ignored
>>>>> Exception exceptions.AssertionError in /Users/robert/sage/sage-
>>>>> cython/
>>>>> local/bin/sage-sage: line 587: 16443 Bus error               sage-
>>>>> ipython "$@" -i
>>>> Next up is running the test suite.
>>> Cool, looking forward to all the fixes coming out of that.
>>
>> I'm going to run it overnight. Given that Sage doesn't exit cleanly
>> (not sure what's up with that), I expect everything to fail, but
>> hopefully it'll still be useful.
>
> Don't work too hard on it yet -- there's some bugs in the refnanny now
> which I should fix, and which may produce these kinds of errors.

I'm running the tests, but probably won't be able to work on it much  
today anyways. So far I have a 7MB log (and it's less than half way  
done).

> One suspect is here:
>
> http://hg.cython.org/cython-devel/rev/4e19b8a61272
>
> There are some more as well which could lead to similar symptoms which
> I'll have to wait with pushing -- ironically, the fix itself seems to
> produce a temp handling refcount bug on the refnanny.c code itself,  
> so I
> must fix that simultaneously.

Cool, let us know when you smash this one.

- Robert



From dagss at student.matnat.uio.no  Sat Feb 21 17:00:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 17:00:25 +0100
Subject: [Cython] Status for 0.11
In-Reply-To: <AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>	<49A01D22.3070504@student.matnat.uio.no>
	<AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>
Message-ID: <49A02519.8030409@student.matnat.uio.no>

Robert Bradshaw wrote:
> I'm running the tests, but probably won't be able to work on it much  
> today anyways. So far I have a 7MB log (and it's less than half way  
> done).

Well, as you showed us with your last patch, one bug in this area tends 
to be repeated a lot of time. So IMHO anything but the top 20 lines in 
that log is worthless, because you'd like to rerun after fixing the 
first issue anyway to remove the duplicates...

(Of course, statistics is fun.)

>> One suspect is here:
>>
>> http://hg.cython.org/cython-devel/rev/4e19b8a61272
>>
>> There are some more as well which could lead to similar symptoms which
>> I'll have to wait with pushing -- ironically, the fix itself seems to
>> produce a temp handling refcount bug on the refnanny.c code itself,  
>> so I
>> must fix that simultaneously.
> 
> Cool, let us know when you smash this one.

http://hg.cython.org/cython-devel/rev/6692a0f0e87a

I won't get into a discussion on whether a fault like this is due to me 
over-relying on the test suite or the pathetic low coverage it has. It's 
fixed now; but I wonder how many more there are...

Anyway, having a refcount fault on every single "del a.b[c]" is 
something I can easily see accounting for some of the megabytes in that log.

-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Feb 21 16:57:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 21 Feb 2009 07:57:36 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <49A02519.8030409@student.matnat.uio.no>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>	<49A01D22.3070504@student.matnat.uio.no>
	<AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>
	<49A02519.8030409@student.matnat.uio.no>
Message-ID: <4EFE7985-C038-4B28-A2E8-8F4A3A1C5D6E@math.washington.edu>

On Feb 21, 2009, at 8:00 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> I'm running the tests, but probably won't be able to work on it much
>> today anyways. So far I have a 7MB log (and it's less than half way
>> done).
>
> Well, as you showed us with your last patch, one bug in this area  
> tends
> to be repeated a lot of time. So IMHO anything but the top 20 lines in
> that log is worthless, because you'd like to rerun after fixing the
> first issue anyway to remove the duplicates...
>
> (Of course, statistics is fun.)

Yes, there's a lot of duplicates

Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
test.log | grep "Acquired on lines" | sort | uniq | wc
       25     367    2431

Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
test.log | grep "sage/.*: " | sort | uniq
     sage/libs/pari/gen.c: _normalize_slice()
     sage/libs/singular/singular.cpp: sa2si_ZZ()
     sage/matrix/misc.c: matrix_rational_echelon_form_multimodular()
     sage/rings/polynomial/multi_polynomial_libsingular.cpp: subs()

Which, not surprisingly, are the "usual suspects" for memory leaks. I  
haven't hit sage/rings yet though, which may turn up a lot more.

>>> One suspect is here:
>>>
>>> http://hg.cython.org/cython-devel/rev/4e19b8a61272
>>>
>>> There are some more as well which could lead to similar symptoms  
>>> which
>>> I'll have to wait with pushing -- ironically, the fix itself  
>>> seems to
>>> produce a temp handling refcount bug on the refnanny.c code itself,
>>> so I
>>> must fix that simultaneously.
>>
>> Cool, let us know when you smash this one.
>
> http://hg.cython.org/cython-devel/rev/6692a0f0e87a
>
> I won't get into a discussion on whether a fault like this is due  
> to me
> over-relying on the test suite or the pathetic low coverage it has.  
> It's
> fixed now; but I wonder how many more there are...

Yeah. At least Sage provides pretty good coverage :).

> Anyway, having a refcount fault on every single "del a.b[c]" is
> something I can easily see accounting for some of the megabytes in  
> that log.

I don't think this is common, but happening just once in a lot of  
places could result in a big log.

- Robert

From michael.abshoff at googlemail.com  Sat Feb 21 17:13:09 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sat, 21 Feb 2009 08:13:09 -0800
Subject: [Cython] Status for 0.11
In-Reply-To: <4EFE7985-C038-4B28-A2E8-8F4A3A1C5D6E@math.washington.edu>
References: <499E0057.9040504@student.matnat.uio.no>	<499E5CDE.6000804@behnel.de>	<499E5EDD.2070800@behnel.de>	<499E6DAA.7020209@student.matnat.uio.no>	<38194.213.61.181.86.1235121936.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<0E0915BB-A4DE-4830-84C3-9D82C2E61CD3@math.washington.edu>	<499E7EEF.2020304@gmail.com>	<E67332F7-E173-4B38-954A-CA252088632B@math.washington.edu>	<703E41FD-A2AB-4E9F-97D2-411D86F4A0BD@math.washington.edu>	<499E8D45.6020605@gmail.com>	<DC9B6453-00F4-42C6-8364-957AF92775E0@math.washington.edu>	<A3A76EC4-A687-4FF9-BD67-62DEFBA7CB7C@math.washington.edu>	<6EABFCBD-32BB-4CF2-B520-98B23ABF3A86@math.washington.edu>	<499FA58E.9010303@gmail.com>	<431B4B6B-7EAF-439A-BB16-5C784CB1078F@math.washington.edu>	<49A01D22.3070504@student.matnat.uio.no>	<AC258079-5AF8-4612-8F38-95BF8E4B1974@math.washington.edu>	<49A02519.8030409@student.matnat.uio.no>
	<4EFE7985-C038-4B28-A2E8-8F4A3A1C5D6E@math.washington.edu>
Message-ID: <49A02815.3000403@gmail.com>

Robert Bradshaw wrote:
> On Feb 21, 2009, at 8:00 AM, Dag Sverre Seljebotn wrote:

Hi Robert,

>>
>> (Of course, statistics is fun.)
> 
> Yes, there's a lot of duplicates
> 
> Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
> test.log | grep "Acquired on lines" | sort | uniq | wc
>        25     367    2431
> 
> Robert-Bradshaws-Laptop:~/sage/sage-cython/devel/sage robert$ cat  
> test.log | grep "sage/.*: " | sort | uniq
>      sage/libs/pari/gen.c: _normalize_slice()
>      sage/libs/singular/singular.cpp: sa2si_ZZ()
>      sage/matrix/misc.c: matrix_rational_echelon_form_multimodular()
>      sage/rings/polynomial/multi_polynomial_libsingular.cpp: subs()
> 
> Which, not surprisingly, are the "usual suspects" for memory leaks. I  
> haven't hit sage/rings yet though, which may turn up a lot more.


Yep, the best strategy here after having run the complete test suite 
once for the "horrible before-picure" is likely to start with coercion 
and structure and then work your way through rings before then looking 
at what is left over. There are some other leaks not in the test suite I 
would like to examine with this once the noise from the low level stuff 
has been quited. Please let me know what I can do on my end to make 
Cython 0.11 happen faster since I cannot wait for it to be in an 
official Sage release. I am expecting similar miracles as from the 
0.10.3 Cython release which already fixed on bad, bad reference count 
related leak issue :)

Note that Sage 3.3 even with the patch from Sage's Cython 0.11 trac 
ticket applied does not compile with yesterday's Cython devel head and I 
did put the failure into a comment on that ticket. It was basically that 
Cython did throw an error for

  typedef unsigned long ulong

in the FLINT header.

Cheers,

Michael

From stefan_ml at behnel.de  Sat Feb 21 17:57:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 17:57:00 +0100
Subject: [Cython] Problems in Py 3.0.1
Message-ID: <49A0325C.9040901@behnel.de>

Hi,

just a quick note that I'm currently trying to get the test suite running
under Python 3.0.1. This release features a couple of fixes for quirks that
were found in the 3.0 release. It is now a lot stricter regarding Py3
concepts. I'll file a couple of new tickets as I move along, and I'll
nominate (most of) them for 0.11, especially those that make Cython
unusable on that target. I expect them to be easy to fix, but I'll
concentrate on finding them for now. So don't be surprised when the bug
list for 0.11 starts getting longer again.

I also noticed tons of warnings in the C++ "compile" tests regarding
useless code. I'm not sure why they pop up in Py3 and not in Py2.

Stefan

From stefan_ml at behnel.de  Sat Feb 21 18:06:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 18:06:04 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0325C.9040901@behnel.de>
References: <49A0325C.9040901@behnel.de>
Message-ID: <49A0347C.8000208@behnel.de>



Stefan Behnel wrote:
> I also noticed tons of warnings in the C++ "compile" tests regarding
> useless code. I'm not sure why they pop up in Py3 and not in Py2.

Those are actually due to the refnanny code. I had to disable the refnanny
in Py3, as we cannot currently compile Cython on that platform anyway, and
the refnanny is built as part of the normal Cython build.

Stefan


From dagss at student.matnat.uio.no  Sat Feb 21 18:24:22 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:24:22 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0347C.8000208@behnel.de>
References: <49A0325C.9040901@behnel.de> <49A0347C.8000208@behnel.de>
Message-ID: <49A038C6.4000007@student.matnat.uio.no>

Stefan Behnel wrote:
> 
> Stefan Behnel wrote:
>> I also noticed tons of warnings in the C++ "compile" tests regarding
>> useless code. I'm not sure why they pop up in Py3 and not in Py2.
> 
> Those are actually due to the refnanny code. I had to disable the refnanny
> in Py3, as we cannot currently compile Cython on that platform anyway, and
> the refnanny is built as part of the normal Cython build.

Have you done a pull recently? I expect some certain useless code to be 
removed now.

The refnanny can be as useful in Python 3 as in Python 2, though if it 
doesn't work that's certainly not a priority, as the issues it checks 
should be the same under both.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Feb 21 18:26:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:26:53 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A038C6.4000007@student.matnat.uio.no>
References: <49A0325C.9040901@behnel.de> <49A0347C.8000208@behnel.de>
	<49A038C6.4000007@student.matnat.uio.no>
Message-ID: <49A0395D.4060207@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Stefan Behnel wrote:
>>> I also noticed tons of warnings in the C++ "compile" tests regarding
>>> useless code. I'm not sure why they pop up in Py3 and not in Py2.
>> Those are actually due to the refnanny code. I had to disable the refnanny
>> in Py3, as we cannot currently compile Cython on that platform anyway, and
>> the refnanny is built as part of the normal Cython build.

To be clearer, runtests.py works with the refnanny without the normal 
Cython build happening first (runtests.py uses pyximport to build it). 
To test e.g. lxml with the refnanny under Python 3 you would need to 
manually compile and install Cython/Runtime/refnanny.pyx

(You probably already know this; just to be sure)

-- 
Dag Sverre

From jasone at canonware.com  Sat Feb 21 18:13:52 2009
From: jasone at canonware.com (Jason Evans)
Date: Sat, 21 Feb 2009 09:13:52 -0800
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
Message-ID: <49A03650.7050803@canonware.com>

The following line in ModuleNode.py causes hundreds of compiler warnings 
about statements with no effect when I compile Crux:

	#define __Pyx_FinishRefcountContext() 0

IMO, the 0 should simply be removed, but this would also be okay if you 
want a non-empty definition:

	#define __Pyx_FinishRefcountContext() do {} while (0)

Jason

From dagss at student.matnat.uio.no  Sat Feb 21 18:29:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:29:53 +0100
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
In-Reply-To: <49A03650.7050803@canonware.com>
References: <49A03650.7050803@canonware.com>
Message-ID: <49A03A11.7000907@student.matnat.uio.no>

Jason Evans wrote:
> The following line in ModuleNode.py causes hundreds of compiler warnings 
> about statements with no effect when I compile Crux:
> 
> 	#define __Pyx_FinishRefcountContext() 0
> 
> IMO, the 0 should simply be removed, but this would also be okay if you 
> want a non-empty definition:
> 
> 	#define __Pyx_FinishRefcountContext() do {} while (0)
> 

Ahh, right. Thanks!

http://hg.cython.org/cython-devel/rev/b5723a7a044a


-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Feb 21 18:16:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 18:16:47 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0395D.4060207@student.matnat.uio.no>
References: <49A0325C.9040901@behnel.de>
	<49A0347C.8000208@behnel.de>	<49A038C6.4000007@student.matnat.uio.no>
	<49A0395D.4060207@student.matnat.uio.no>
Message-ID: <49A036FF.8060003@behnel.de>


Dag Sverre Seljebotn wrote:
> To be clearer, runtests.py works with the refnanny without the normal 
> Cython build happening first (runtests.py uses pyximport to build it). 
> To test e.g. lxml with the refnanny under Python 3 you would need to 
> manually compile and install Cython/Runtime/refnanny.pyx

Then the next step is to port pyximport to Py3, right? :)

Stefan


From stefan_ml at behnel.de  Sat Feb 21 18:29:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 18:29:56 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A036FF.8060003@behnel.de>
References: <49A0325C.9040901@behnel.de>	<49A0347C.8000208@behnel.de>	<49A038C6.4000007@student.matnat.uio.no>	<49A0395D.4060207@student.matnat.uio.no>
	<49A036FF.8060003@behnel.de>
Message-ID: <49A03A14.8050206@behnel.de>


Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> To be clearer, runtests.py works with the refnanny without the normal 
>> Cython build happening first (runtests.py uses pyximport to build it). 
>> To test e.g. lxml with the refnanny under Python 3 you would need to 
>> manually compile and install Cython/Runtime/refnanny.pyx
> 
> Then the next step is to port pyximport to Py3, right? :)

ah, and: make it work even if Cython cannot be imported due to SyntaxErrors.

http://trac.cython.org/cython_trac/ticket/214

Stefan

From dalcinl at gmail.com  Sat Feb 21 18:39:08 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 14:39:08 -0300
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
In-Reply-To: <49A03A11.7000907@student.matnat.uio.no>
References: <49A03650.7050803@canonware.com>
	<49A03A11.7000907@student.matnat.uio.no>
Message-ID: <e7ba66e40902210939u72a6c30ej4dcc281d81f4ad81@mail.gmail.com>

A "(void)0" shoud also do the trick, right?

On Sat, Feb 21, 2009 at 2:29 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Jason Evans wrote:
>> The following line in ModuleNode.py causes hundreds of compiler warnings
>> about statements with no effect when I compile Crux:
>>
>>       #define __Pyx_FinishRefcountContext() 0
>>
>> IMO, the 0 should simply be removed, but this would also be okay if you
>> want a non-empty definition:
>>
>>       #define __Pyx_FinishRefcountContext() do {} while (0)
>>
>
> Ahh, right. Thanks!
>
> http://hg.cython.org/cython-devel/rev/b5723a7a044a
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sat Feb 21 18:58:46 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Feb 2009 18:58:46 +0100
Subject: [Cython] __Pyx_FinishRefcountContext() compiler warnings
In-Reply-To: <e7ba66e40902210939u72a6c30ej4dcc281d81f4ad81@mail.gmail.com>
References: <49A03650.7050803@canonware.com>	<49A03A11.7000907@student.matnat.uio.no>
	<e7ba66e40902210939u72a6c30ej4dcc281d81f4ad81@mail.gmail.com>
Message-ID: <49A040D6.8040600@student.matnat.uio.no>

It was a remnant after when the refnanny raised exceptions which I 
forgot to remove; the correct thing is to be a noop now. (It needed to 
return 0 earlier on and was used in an expression then.)

Dag Sverre

Lisandro Dalcin wrote:
> A "(void)0" shoud also do the trick, right?
> 
> On Sat, Feb 21, 2009 at 2:29 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Jason Evans wrote:
>>> The following line in ModuleNode.py causes hundreds of compiler warnings
>>> about statements with no effect when I compile Crux:
>>>
>>>       #define __Pyx_FinishRefcountContext() 0
>>>
>>> IMO, the 0 should simply be removed, but this would also be okay if you
>>> want a non-empty definition:
>>>
>>>       #define __Pyx_FinishRefcountContext() do {} while (0)
>>>
>> Ahh, right. Thanks!
>>
>> http://hg.cython.org/cython-devel/rev/b5723a7a044a
>>
>>
>> --
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> 
> 
> 


-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Feb 21 19:40:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 19:40:00 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A0325C.9040901@behnel.de>
References: <49A0325C.9040901@behnel.de>
Message-ID: <49A04A80.5020705@behnel.de>


Stefan Behnel wrote:
> just a quick note that I'm currently trying to get the test suite running
> under Python 3.0.1. This release features a couple of fixes for quirks that
> were found in the 3.0 release. It is now a lot stricter regarding Py3
> concepts.

Interesting enough, the size_t tests fail for me with a TypeError:

---------------------------------------
Failed example:
    a.foo(-1) #doctest: +ELLIPSIS
Expected:
    Traceback (most recent call last):
        ...
    OverflowError: ...
Got:
    Traceback (most recent call last):
      File "/usr/local/python3.0.1-debug/lib/python3.0/doctest.py", line
1234, in __run
        compileflags, 1), test.globs)
      File "<doctest size_t[10]>", line 1, in <module>
        a.foo(-1) #doctest: +ELLIPSIS
      File "size_t.pyx", line 53, in size_t.A.foo (size_t.cpp:569)
    TypeError: can't convert negative int to unsigned
---------------------------------------

Wasn't there supposed to be an OverflowError instead?

Stefan

From stefan_ml at behnel.de  Sat Feb 21 21:26:55 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 21:26:55 +0100
Subject: [Cython] string target type of struct-to-dict conversion
Message-ID: <49A0638F.5060409@behnel.de>

Hi,

I found this code in the struct_conversion.pyx test:

----------------------------------
    >>> test_constructor_kwds(1.25, 2.5, 128)
    {'y': 2.5, 'x': 1.25, 'color': 128}

...


def test_dict_construction(x, y, color):
    cdef Point p = {'color': color, 'x': x, 'y': y}
    return p

def test_constructor_kwds(x, y, color):
    cdef Point p = Point(x=x, y=y, color=color)
    return p
----------------------------------

Should we really base this feature on byte strings instead of unicode strings?

Note that the doctest at the top fails in Py3 anyway, as the repr() of a
byte string has a leading 'b'.

Stefan

From dalcinl at gmail.com  Sat Feb 21 22:26:24 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 18:26:24 -0300
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <49A04A80.5020705@behnel.de>
References: <49A0325C.9040901@behnel.de> <49A04A80.5020705@behnel.de>
Message-ID: <e7ba66e40902211326p757acd6biedaafb7b7bdf2c7c@mail.gmail.com>

Stefan, in Py3, PyInt is gone and we only have PyLong. Then the code
follows other path.

Some days ago I filed an issue and provided a patch wich was accepted,
now core Python (2.7 and 3.1) will raise OverflowError...

If you want, I can implement some vile hackery, I just need to check
myself if the PyLong is negative (PyLong implementation abuses the
tp_size member of the object struct to carry the sign)



On Sat, Feb 21, 2009 at 3:40 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Stefan Behnel wrote:
>> just a quick note that I'm currently trying to get the test suite running
>> under Python 3.0.1. This release features a couple of fixes for quirks that
>> were found in the 3.0 release. It is now a lot stricter regarding Py3
>> concepts.
>
> Interesting enough, the size_t tests fail for me with a TypeError:
>
> ---------------------------------------
> Failed example:
>    a.foo(-1) #doctest: +ELLIPSIS
> Expected:
>    Traceback (most recent call last):
>        ...
>    OverflowError: ...
> Got:
>    Traceback (most recent call last):
>      File "/usr/local/python3.0.1-debug/lib/python3.0/doctest.py", line
> 1234, in __run
>        compileflags, 1), test.globs)
>      File "<doctest size_t[10]>", line 1, in <module>
>        a.foo(-1) #doctest: +ELLIPSIS
>      File "size_t.pyx", line 53, in size_t.A.foo (size_t.cpp:569)
>    TypeError: can't convert negative int to unsigned
> ---------------------------------------
>
> Wasn't there supposed to be an OverflowError instead?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Feb 21 22:35:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Feb 2009 22:35:17 +0100
Subject: [Cython] Problems in Py 3.0.1
In-Reply-To: <e7ba66e40902211326p757acd6biedaafb7b7bdf2c7c@mail.gmail.com>
References: <49A0325C.9040901@behnel.de> <49A04A80.5020705@behnel.de>
	<e7ba66e40902211326p757acd6biedaafb7b7bdf2c7c@mail.gmail.com>
Message-ID: <49A07395.4030806@behnel.de>


Lisandro Dalcin wrote:
> Stefan, in Py3, PyInt is gone and we only have PyLong. Then the code
> follows other path.
> 
> Some days ago I filed an issue and provided a patch wich was accepted,
> now core Python (2.7 and 3.1) will raise OverflowError...

Right, I found that in the back of my head after writing the mail.


> If you want, I can implement some vile hackery, I just need to check
> myself if the PyLong is negative (PyLong implementation abuses the
> tp_size member of the object struct to carry the sign)

No, for now, I'm fine with whatever Python does here. I pushed a fix for
the test case.

Stefan

From dalcinl at gmail.com  Sat Feb 21 23:04:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 19:04:28 -0300
Subject: [Cython] On refnanny commit
In-Reply-To: <49A079F3.8020108@student.matnat.uio.no>
References: <49A079F3.8020108@student.matnat.uio.no>
Message-ID: <e7ba66e40902211404m9fc50fbo9867891e2e93574@mail.gmail.com>

Dag, this change was required for the case you pass --cleanup to Cython.

This flag is surely not used in runtests.py, so any of us never
noticed that the generated code (when --cleanup is used) do not
compile...

Remember that cleanup funtions are registered with "atexit" to run at
Python shutdown... I do not think that is a good place to be tracking
references....

BTW, I'll try to enable refnanny for the module init function.
Basically, we should GIVEREF on module globals before finishing the
refcount context. I'll give a try.



On Sat, Feb 21, 2009 at 7:02 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Hi, I just noticed your commit. I'm fine with it, I was just wondering if
> you could write a line or two about the reasons behind it?
>
> Also, I'm worried that user code in module scope will still be refnannied in
> that function, so you cannot remove the refnanny initialization for that
> function altogether... Did you work around that? (Didn't look at it in close
> detail.)
>
> --
> Dag Sverre
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sun Feb 22 00:19:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 00:19:41 +0100
Subject: [Cython] Py3 exception handling and chaining
Message-ID: <49A08C0D.2020508@behnel.de>

Hi,

I fixed most of the Py3 problems in the test suite. The last one relates to
exception chaining.

http://www.python.org/dev/peps/pep-3134/

Currently, when you write this:

    try: ...
    except SomeException:
        raise NewException

SomeException will remain in sys.exc_info(), or rather in
PyThreadState->exc_type/value/tb. Subsequently raised exceptions will find
it there and make it their "context". A way to fix this is to put this into
Pyx_ErrRestore() to clear the context exception when the current exception
is set:

    tmp_type = tstate->exc_type;
    tmp_value = tstate->exc_value;
    tmp_tb = tstate->exc_traceback;
    tstate->exc_type = 0;
    tstate->exc_value = 0;
    tstate->exc_traceback = 0;
    Py_XDECREF(tmp_type);
    Py_XDECREF(tmp_value);
    Py_XDECREF(tmp_tb);

While doing that, I also noticed that ReRaise is currently implemented like
this:

  static void __Pyx_ReRaise(void) {
    PyThreadState *tstate = PyThreadState_Get();
    PyObject *type = tstate->exc_type;
    PyObject *value = tstate->exc_value;
    PyObject *tb = tstate->exc_traceback;
    Py_XINCREF(type);
    Py_XINCREF(value);
    Py_XINCREF(tb);
    __Pyx_ErrRestore(type, value, tb);
  }

I wonder why this doesn't set exc_* to NULL either.

Both changes make the test suite pass in Py 3.0.1, and it still works in
Py2.6. Maybe Greg can tell us why it was originally implemented differently
and if there are potential issues?

In any case, this is more of a quick fix rather than the final solution.
The real fix would be to handle exception chaining correctly in Py3, i.e.
to set the context of an exception depending on where it is raised, and to
support the explicit "raise ... from ..." syntax to allow users to set the
cause.

http://trac.cython.org/cython_trac/ticket/217

Stefan

From dalcinl at gmail.com  Sun Feb 22 00:40:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 21 Feb 2009 20:40:37 -0300
Subject: [Cython] enabling refnanny in module init function
Message-ID: <e7ba66e40902211540g2ca9c7cfyb0f35bb7fc1608cf@mail.gmail.com>

Try the trivial patch below, the problems with globals get fixed...
However, enabling the refnanny will spot other reference count
problems, at least I get them in mpi4py ....


diff -r 3628c9268dfe Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py	Sat Feb 21 22:34:25 2009 +0100
+++ b/Cython/Compiler/ModuleNode.py	Sat Feb 21 20:32:14 2009 -0300
@@ -1651,11 +1651,9 @@
         env.use_utility_code(Nodes.traceback_utility_code)
         code.putln("%s = NULL;" % Naming.retval_cname)
         code.put_label(code.return_label)
-        # Disabled because of confusion with refcount of global
variables -- run ass2cglobal testcase to see
-        #code.put_finish_refcount_context()
-        code.putln("#if CYTHON_REFNANNY")
-        code.putln("if (__pyx_refchk) {};")
-        code.putln("#endif")
+
+        code.put_finish_refcount_context()
+
         code.putln("#if PY_MAJOR_VERSION < 3")
         code.putln("return;")
         code.putln("#else")
@@ -1793,7 +1791,7 @@
         for entry in env.var_entries:
             if entry.visibility != 'extern':
                 if entry.type.is_pyobject and entry.used:
-                    code.put_init_var_to_py_none(entry)
+                    code.put_init_var_to_py_none(entry, nanny=False)

     def generate_c_function_export_code(self, env, code):
         # Generate code to create PyCFunction wrappers for exported C
functions.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sun Feb 22 11:27:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 11:27:31 +0100
Subject: [Cython] enabling refnanny in module init function
In-Reply-To: <e7ba66e40902211540g2ca9c7cfyb0f35bb7fc1608cf@mail.gmail.com>
References: <e7ba66e40902211540g2ca9c7cfyb0f35bb7fc1608cf@mail.gmail.com>
Message-ID: <49A12893.5040401@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Try the trivial patch below, the problems with globals get fixed...
> However, enabling the refnanny will spot other reference count
> problems, at least I get them in mpi4py ....

Yes, I've looked into some of them and they really need fixing. I've pushed
your patch.

Stefan


From stefan_ml at behnel.de  Sun Feb 22 12:01:28 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 12:01:28 +0100
Subject: [Cython] module cleanup (was: On refnanny commit)
In-Reply-To: <e7ba66e40902211404m9fc50fbo9867891e2e93574@mail.gmail.com>
References: <49A079F3.8020108@student.matnat.uio.no>
	<e7ba66e40902211404m9fc50fbo9867891e2e93574@mail.gmail.com>
Message-ID: <49A13088.8030201@behnel.de>


Lisandro Dalcin wrote:
> Remember that cleanup funtions are registered with "atexit" to run at
> Python shutdown... I do not think that is a good place to be tracking
> references....

This is OK for Py2.x. However, for Python 3, the right solution would be to
put the cleanup code into the official module clear function m_clear(), as
described in PEP 3121.

http://www.python.org/dev/peps/pep-3121/

Then, the test runner could simply unload the module after running its
doctests, but still as part of the test itself.

There already is a ticket for general PEP 3121 support (#173), but I
extracted this as a new ticket #218, as I find it the most important part,
which can be done much sooner.

http://trac.cython.org/cython_trac/ticket/173
http://trac.cython.org/cython_trac/ticket/218

Stefan

From stefan_ml at behnel.de  Sun Feb 22 12:56:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 12:56:30 +0100
Subject: [Cython] embedded signatures and function annotations (PEP 3107)
Message-ID: <49A13D6E.1090003@behnel.de>

Hi,

I wonder why this function declaration

    cpdef str cf_sd(str s='spam'):
        return s

leads to this embedded signature:

    >>> cf_sd.__doc__
    "cf_sd(str s='spam') -> str"

The return value follows PEP 3107 style annotations, but wouldn't it make
sense to use PEP 3107 style annotations also for the parameter types like this:

    >>> cf_sd.__doc__
    "cf_sd(s : str = 'spam') -> str"

?

Stefan

From stefan_ml at behnel.de  Sun Feb 22 15:13:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Feb 2009 15:13:11 +0100
Subject: [Cython] Py3 exception handling and chaining
In-Reply-To: <49A08C0D.2020508@behnel.de>
References: <49A08C0D.2020508@behnel.de>
Message-ID: <49A15D77.4080905@behnel.de>

Hi again,

Stefan Behnel wrote:
> Currently, when you write this:
> 
>     try: ...
>     except SomeException:
>         raise NewException
> 
> SomeException will remain in sys.exc_info(), or rather in
> PyThreadState->exc_type/value/tb. Subsequently raised exceptions will find
> it there and make it their "context".
> [...]
> Both changes make the test suite pass in Py 3.0.1, and it still works in
> Py2.6.

They also seem to be the fix for an extremely long standing crash bug that
I experienced with lxml under Py3. So, for me, that means that the final
release of lxml 2.2 will finally support Py 3.0 completely and reliably.

Filed and closed as ticket 219.

http://trac.cython.org/cython_trac/ticket/219
http://hg.cython.org/cython-devel/rev/24bb0456d787

I actually found these things using a debug build of Py 3.0.1. It's a very
valuable source of information when it comes to tracking down crashes and a
number of other problems with extensions.

Stefan

From sturla at molden.no  Mon Feb 23 05:37:36 2009
From: sturla at molden.no (Sturla Molden)
Date: Mon, 23 Feb 2009 05:37:36 +0100 (CET)
Subject: [Cython] Fortran types in C
Message-ID: <70bfb5794dc767535a8679334bc64028.squirrel@webmail.uio.no>


>What is the best way of declaring Fortran types in C? I understand there
>is likely no perfect cross-platform solution here

ISO Fortran 2003 has standard bindings to ISO C. Most Fortran compilers
support them, including gfortran. These C bindings make C callable from
Fortran, but they also make Fortran callable from C in a standardized and
portable way.

The short receipe is: you don't declare any Fortran types in C. You
declare your Fortran functions with C types. This even includes
pass-by-value calling convention. Then the Fortran compiler will do the
correct conversion.

>Furthermore it would be very sweet to
>automatically be able to pass Python buffer objects as F90 assumed-shape
>arrays :-)

Take a look at the array-descriptor library of Chasm.

http://chasm-interop.sourceforge.net/


Regards,
Sturla Molden


From anatoly at gnu.org  Mon Feb 23 10:19:51 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Mon, 23 Feb 2009 15:19:51 +0600
Subject: [Cython] Wrapping callbacks
Message-ID: <20090223151951.5960dce1.anatoly@gnu.org>

Hello!

I'm trying to wrap function with callback parameter:

error_t fshelp_start_translator (fshelp_open_fn_t underlying_open_fn, void *cookie, char *name, char *argz, int argz_len, int timeout, fsys_t *control)

where fshelp_open_fn_t is

typedef error_t (*fshelp_open_fn_t) (int flags, file_t *node, mach_msg_type_name_t *node_type, task_t, void *cookie)

Is it possible?

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090223/300fab87/attachment.pgp 

From stefan_ml at behnel.de  Mon Feb 23 10:55:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 10:55:23 +0100 (CET)
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090223151951.5960dce1.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
Message-ID: <62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Anatoly A. Kazantsev wrote:
> I'm trying to wrap function with callback parameter:
>
> error_t fshelp_start_translator (fshelp_open_fn_t underlying_open_fn, void
> *cookie, char *name, char *argz, int argz_len, int timeout, fsys_t
> *control)
>
> where fshelp_open_fn_t is
>
> typedef error_t (*fshelp_open_fn_t) (int flags, file_t *node,
> mach_msg_type_name_t *node_type, task_t, void *cookie)

Have you looked at the callback example in the Cython distribution?

http://hg.cython.org/cython-devel/file/tip/Demos/callback

Stefan


From anatoly at gnu.org  Mon Feb 23 11:56:25 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Mon, 23 Feb 2009 16:56:25 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <20090223165625.5deaffe9.anatoly@gnu.org>

On Mon, 23 Feb 2009 10:55:23 +0100 (CET)
"Stefan Behnel" <stefan_ml at behnel.de> wrote:

> Anatoly A. Kazantsev wrote:
> > I'm trying to wrap function with callback parameter:
> >
> > error_t fshelp_start_translator (fshelp_open_fn_t underlying_open_fn, void
> > *cookie, char *name, char *argz, int argz_len, int timeout, fsys_t
> > *control)
> >
> > where fshelp_open_fn_t is
> >
> > typedef error_t (*fshelp_open_fn_t) (int flags, file_t *node,
> > mach_msg_type_name_t *node_type, task_t, void *cookie)
> 
> Have you looked at the callback example in the Cython distribution?

Yes, of course.

But I can not understand how to pass pointer of Python callback-function.
Additional parameter 'void *user_data' is used in the example for this purpose.

But I can't modify 'fshelp_open_fn_t' function type.

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090223/e13e8962/attachment-0001.pgp 

From greg.ewing at canterbury.ac.nz  Mon Feb 23 12:43:33 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Feb 2009 00:43:33 +1300
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090223165625.5deaffe9.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
Message-ID: <49A28BE5.1020709@canterbury.ac.nz>

Anatoly A. Kazantsev wrote:

> Additional parameter 'void *user_data' is used in the example for this purpose.
> 
> But I can't modify 'fshelp_open_fn_t' function type.

What's the 'void *cookie'? Is it an arbitrary pointer
that you can use for your own purposes? If so, use
it the same way as the user_data in the example.

-- 
Greg

From dalcinl at gmail.com  Mon Feb 23 20:09:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 17:09:27 -0200
Subject: [Cython] unicode can bite us ...
Message-ID: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>

Stefan, in this commit

changeset:   1743:58406a9db77b
user:        Stefan Behnel <scoder at users.berlios.de>
date:        Sat Feb 21 21:08:46 2009 +0100
summary:     unicode fixes for numpy test (untested, hopefully not
breaking everything)

your broke everything ;-)


BTW, I still want Cython to let me specify in an easy way that a
string literal should match the one Python normally uses for
identifier...

For example:

np.arange(10, dtype=s'i')


Then s'i' literal whould be PyString in Py2 and PyUnicode in Py3... We
could even make this syntax sugar really strict and reject
non-pure-ASCII literals.

We cannot assume all Python packages out there (specially those
targeted for scientific users)  are unicode-aware... My numpy
installation (numpy-1.2.0 as packaged in Fedora 10) is a good
example...

Stefan, we really need this... IMHO this is a clear case where
practicality beats purity....

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Mon Feb 23 21:24:20 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 21:24:20 +0100
Subject: [Cython] unicode can bite us ...
In-Reply-To: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
Message-ID: <49A305F4.3080107@behnel.de>


Lisandro Dalcin wrote:
> Stefan, in this commit
> 
> changeset:   1743:58406a9db77b
> user:        Stefan Behnel <scoder at users.berlios.de>
> date:        Sat Feb 21 21:08:46 2009 +0100
> summary:     unicode fixes for numpy test (untested, hopefully not
> breaking everything)
> 
> your broke everything ;-)

Sorry 'bout that! :)

I fixed that, changing the NumPy parameters from u'...' to b'...' should
make it work. I'll push it as soon as I get the conflict resolved that I
just got from you. :o)


> BTW, I still want Cython to let me specify in an easy way that a
> string literal should match the one Python normally uses for
> identifier...
> 
> For example:
> 
> np.arange(10, dtype=s'i')
> 
> Then s'i' literal whould be PyString in Py2 and PyUnicode in Py3...

However, that makes it sound like it's NumPy that's broken here. There's
absolutely no reason why the parameters should be byte strings.

I still doubt that such a feature is really needed. Any API that requires
byte strings where unicode strings would be appropriate is broken. The
above will become useless (and even semantically wrong) once NumPy is
fixed. Other libraries that currently require this will also be fixed in
the not so far future.


> Stefan, we really need this... IMHO this is a clear case where
> practicality beats purity....

Is NumPy actually Py3 compatible? If not, it's definitely not a case for a
decision either way.

Stefan

From stefan_ml at behnel.de  Mon Feb 23 21:35:15 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 21:35:15 +0100
Subject: [Cython] rev 5863fde464a0
Message-ID: <49A30883.3090603@behnel.de>

Hi Lisandro,

could you explain why this is needed? My fix would have been to add a
GIVEREF, as the default argument is being made global. I also liked the
code a lot better the old way. Is there a case where the original code
didn't work?

Stefan


@@ -1231,15 +1233,14 @@
             if default:
                 if not default.is_literal:
                     default.generate_evaluation_code(code)
-                    default.make_owned_reference(code)
-                    code.putln(
-                        "%s = %s;" % (
-                            arg.default_entry.cname,
-                            default.result_as(arg.default_entry.type)))
-                    if default.is_temp and default.type.is_pyobject:
-                        code.putln(
-                            "%s = 0;" %
-                                default.result())
+                    assign_code = "%s = %s;" % (
+                        arg.default_entry.cname,
+                        default.result_as(arg.default_entry.type))
+                    if default.type.is_pyobject:
+                        assign_code += " Py_INCREF(%s);" % \
+                            arg.type.as_pyobject(arg.default_entry.cname)
+                    code.putln(assign_code)
+                    default.generate_disposal_code(code)
                     default.free_temps(code)

From robert.kern at gmail.com  Mon Feb 23 22:03:38 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 23 Feb 2009 15:03:38 -0600
Subject: [Cython] unicode can bite us ...
In-Reply-To: <49A305F4.3080107@behnel.de>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
Message-ID: <gnv2va$tc8$1@ger.gmane.org>

On 2009-02-23 14:24, Stefan Behnel wrote:

> Is NumPy actually Py3 compatible?

Not even close.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From stefan_ml at behnel.de  Mon Feb 23 22:07:42 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Feb 2009 22:07:42 +0100
Subject: [Cython] rev 5863fde464a0
In-Reply-To: <49A30883.3090603@behnel.de>
References: <49A30883.3090603@behnel.de>
Message-ID: <49A3101E.4030501@behnel.de>

Hi again,

> @@ -1231,15 +1233,14 @@
>              if default:
>                  if not default.is_literal:
>                      default.generate_evaluation_code(code)
> -                    default.make_owned_reference(code)
> -                    code.putln(
> -                        "%s = %s;" % (
> -                            arg.default_entry.cname,
> -                            default.result_as(arg.default_entry.type)))
> -                    if default.is_temp and default.type.is_pyobject:
> -                        code.putln(
> -                            "%s = 0;" %
> -                                default.result())
> +                    assign_code = "%s = %s;" % (
> +                        arg.default_entry.cname,
> +                        default.result_as(arg.default_entry.type))
> +                    if default.type.is_pyobject:
> +                        assign_code += " Py_INCREF(%s);" % \
> +                            arg.type.as_pyobject(arg.default_entry.cname)
> +                    code.putln(assign_code)
> +                    default.generate_disposal_code(code)
>                      default.free_temps(code)

Here's what I pushed now (Nodes.py):

            if default:
                if not default.is_literal:
                    default.generate_evaluation_code(code)
                    default.make_owned_reference(code)
                    code.putln(
                        "%s = %s;" % (
                            arg.default_entry.cname,
                            default.result_as(arg.default_entry.type)))
                    if default.is_temp and default.type.is_pyobject:
                        code.putln("%s = 0;" % default.result())
                    default.free_temps(code)
                    code.put_giveref(arg.default_entry.cname)

Note that generate_evaluation_code() already cleans up the temps of the RHS
subexpressions, so there is no need to do anything but stealing the reference.

BTW, note that a doctest must be a unicode string. Otherwise, it will not
be executed in Py3.

Stefan


From robertwb at math.washington.edu  Mon Feb 23 22:27:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 23 Feb 2009 13:27:07 -0800
Subject: [Cython] unicode can bite us ...
In-Reply-To: <49A305F4.3080107@behnel.de>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
Message-ID: <72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>

On Feb 23, 2009, at 12:24 PM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> Stefan, in this commit
>>
>> changeset:   1743:58406a9db77b
>> user:        Stefan Behnel <scoder at users.berlios.de>
>> date:        Sat Feb 21 21:08:46 2009 +0100
>> summary:     unicode fixes for numpy test (untested, hopefully not
>> breaking everything)
>>
>> your broke everything ;-)
>
> Sorry 'bout that! :)
>
> I fixed that, changing the NumPy parameters from u'...' to b'...'  
> should
> make it work. I'll push it as soon as I get the conflict resolved  
> that I
> just got from you. :o)
>
>> BTW, I still want Cython to let me specify in an easy way that a
>> string literal should match the one Python normally uses for
>> identifier...
>>
>> For example:
>>
>> np.arange(10, dtype=s'i')
>>
>> Then s'i' literal whould be PyString in Py2 and PyUnicode in Py3...
>
> However, that makes it sound like it's NumPy that's broken here.  
> There's
> absolutely no reason why the parameters should be byte strings.

This is how unmarked string literals should work, right? Just like  
the Python interpreter.

> I still doubt that such a feature is really needed. Any API that  
> requires
> byte strings where unicode strings would be appropriate is broken. The
> above will become useless (and even semantically wrong) once NumPy is
> fixed. Other libraries that currently require this will also be  
> fixed in
> the not so far future.

I think you underestimate how long broken libraries will be out there.

>> Stefan, we really need this... IMHO this is a clear case where
>> practicality beats purity....
>
> Is NumPy actually Py3 compatible? If not, it's definitely not a  
> case for a
> decision either way.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Mon Feb 23 22:36:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 19:36:06 -0200
Subject: [Cython] Stefan, please fix this
Message-ID: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>

In commit df54195ce73a, you changed this bit of my original push.

------------------------ Cython/Compiler/ModuleNode.py ------------------------
@@ -1706,6 -1706,10 +1706,9 @@
                 code.put_decref_clear(entry.pystring_cname,
                                       PyrexTypes.py_object_type,
                                       nanny=False)
+        for entry in env.default_entries:
+            if entry.type.is_pyobject and entry.used:
-                code.putln("Py_DECREF(%s); %s = 0;" % (
-                        code.entry_as_pyobject(entry), entry.cname))
+                code.put_var_decref_clear(entry)
         code.putln("Py_INCREF(Py_None); return Py_None;")
         code.putln('}')


Please revert to my implementation (bare Py_DECREF and nullify), or
any other that does not break C compilation... IMHO, The module
cleanup function is not a place to play with refnanny. Moreover, the
module cleanup funcion is not refnanny-managed...

BTW, I believe that cleanup code should be generated when running the
testsuite...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Feb 23 22:57:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 23 Feb 2009 22:57:57 +0100
Subject: [Cython] rev 5863fde464a0
In-Reply-To: <49A3101E.4030501@behnel.de>
References: <49A30883.3090603@behnel.de> <49A3101E.4030501@behnel.de>
Message-ID: <49A31BE5.9070406@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi again,
> 
>> @@ -1231,15 +1233,14 @@
>>              if default:
>>                  if not default.is_literal:
>>                      default.generate_evaluation_code(code)
>> -                    default.make_owned_reference(code)
>> -                    code.putln(
>> -                        "%s = %s;" % (
>> -                            arg.default_entry.cname,
>> -                            default.result_as(arg.default_entry.type)))
>> -                    if default.is_temp and default.type.is_pyobject:
>> -                        code.putln(
>> -                            "%s = 0;" %
>> -                                default.result())
>> +                    assign_code = "%s = %s;" % (
>> +                        arg.default_entry.cname,
>> +                        default.result_as(arg.default_entry.type))
>> +                    if default.type.is_pyobject:
>> +                        assign_code += " Py_INCREF(%s);" % \
>> +                            arg.type.as_pyobject(arg.default_entry.cname)
>> +                    code.putln(assign_code)
>> +                    default.generate_disposal_code(code)
>>                      default.free_temps(code)
> 
> Here's what I pushed now (Nodes.py):
> 
>             if default:
>                 if not default.is_literal:
>                     default.generate_evaluation_code(code)
>                     default.make_owned_reference(code)
>                     code.putln(
>                         "%s = %s;" % (
>                             arg.default_entry.cname,
>                             default.result_as(arg.default_entry.type)))
>                     if default.is_temp and default.type.is_pyobject:
>                         code.putln("%s = 0;" % default.result())
>                     default.free_temps(code)
>                     code.put_giveref(arg.default_entry.cname)
> 
> Note that generate_evaluation_code() already cleans up the temps of the RHS
> subexpressions, so there is no need to do anything but stealing the reference.

+1; at least dropping that call to make_owned_reference was asking for 
trouble (as I suppose the object could or could not own the reference, 
and that could be a noop or not a noop, depending on circumstances)

-- 
Dag Sverre

From dalcinl at gmail.com  Mon Feb 23 22:48:38 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 19:48:38 -0200
Subject: [Cython] unicode can bite us ...
In-Reply-To: <72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
Message-ID: <e7ba66e40902231348y740d3d57gfd8fd91baabfc90d@mail.gmail.com>

On Mon, Feb 23, 2009 at 7:27 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
>
> This is how unmarked string literals should work, right? Just like
> the Python interpreter.
>

I'm on your side, Robert. At least, I would like to have a easy way to
specify a string literal matching whatever Python uses for
identifiers...

>
> I think you underestimate how long broken libraries will be out there.
>

Indeed... And I still thing we should provide facilities to alleviate
the str/unicode difference and the Py2/Py3 difference...




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Feb 23 23:53:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Feb 2009 19:53:09 -0300
Subject: [Cython] patch, func arg default vals, please review
Message-ID: <e7ba66e40902231453q1d1853bfn24db63882ad90629@mail.gmail.com>

Take a look, I believe the fix is valid, in such case please push it.
Today I lost my self-confidence, don't like to mess the codebase again
:-)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PATCH.diff
Type: application/octet-stream
Size: 1151 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090223/bd41b5cd/attachment.obj 

From anatoly at gnu.org  Tue Feb 24 07:11:16 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Tue, 24 Feb 2009 12:11:16 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <49A28BE5.1020709@canterbury.ac.nz>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
Message-ID: <20090224121116.4e732c2a.anatoly@gnu.org>

On Tue, 24 Feb 2009 00:43:33 +1300
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Anatoly A. Kazantsev wrote:
> 
> > Additional parameter 'void *user_data' is used in the example for this purpose.
> > 
> > But I can't modify 'fshelp_open_fn_t' function type.
> 
> What's the 'void *cookie'? Is it an arbitrary pointer
> that you can use for your own purposes? If so, use
> it the same way as the user_data in the example.

Yes, I can use this pointer :-)

But how can I wrap callbacks without such pointers. Or this pointers is a 'good'
practise in designing of callback functions?

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/0b4cd83b/attachment.pgp 

From robertwb at math.washington.edu  Tue Feb 24 08:24:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 23 Feb 2009 23:24:32 -0800
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090224121116.4e732c2a.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
Message-ID: <C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>

On Feb 23, 2009, at 10:11 PM, Anatoly A. Kazantsev wrote:

> On Tue, 24 Feb 2009 00:43:33 +1300
> Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
>> Anatoly A. Kazantsev wrote:
>>
>>> Additional parameter 'void *user_data' is used in the example for  
>>> this purpose.
>>>
>>> But I can't modify 'fshelp_open_fn_t' function type.
>>
>> What's the 'void *cookie'? Is it an arbitrary pointer
>> that you can use for your own purposes? If so, use
>> it the same way as the user_data in the example.
>
> Yes, I can use this pointer :-)
>
> But how can I wrap callbacks without such pointers. Or this  
> pointers is a 'good'
> practise in designing of callback functions?

Pointers are the only way to pass around functions in C. Pointers  
aren't bad, they're just powerful and potentially unforgiving.

- Robert


From anatoly at gnu.org  Tue Feb 24 08:42:18 2009
From: anatoly at gnu.org (Anatoly A. Kazantsev)
Date: Tue, 24 Feb 2009 13:42:18 +0600
Subject: [Cython] Wrapping callbacks
In-Reply-To: <C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
Message-ID: <20090224134218.6d65f7ce.anatoly@gnu.org>

On Mon, 23 Feb 2009 23:24:32 -0800
Robert Bradshaw <robertwb at math.washington.edu> wrote:

> Pointers are the only way to pass around functions in C. Pointers  
> aren't bad, they're just powerful and potentially unforgiving.

Yeah, I know. But what to do in a case when there's no such 'cookie' pointer in
callback-function definition.

-- 
Anatoly A. Kazantsev <anatoly at gnu.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090224/1a877e05/attachment.pgp 

From robertwb at math.washington.edu  Tue Feb 24 08:48:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 23 Feb 2009 23:48:57 -0800
Subject: [Cython] Wrapping callbacks
In-Reply-To: <20090224134218.6d65f7ce.anatoly@gnu.org>
References: <20090223151951.5960dce1.anatoly@gnu.org>
	<62814.213.61.181.86.1235382923.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<20090223165625.5deaffe9.anatoly@gnu.org>
	<49A28BE5.1020709@canterbury.ac.nz>
	<20090224121116.4e732c2a.anatoly@gnu.org>
	<C4048095-F3A9-4221-922D-6BB528B1B919@math.washington.edu>
	<20090224134218.6d65f7ce.anatoly@gnu.org>
Message-ID: <3B93E01F-8297-47F5-B814-A18D859F92D5@math.washington.edu>

On Feb 23, 2009, at 11:42 PM, Anatoly A. Kazantsev wrote:

> On Mon, 23 Feb 2009 23:24:32 -0800
> Robert Bradshaw <robertwb at math.washington.edu> wrote:
>
>> Pointers are the only way to pass around functions in C. Pointers
>> aren't bad, they're just powerful and potentially unforgiving.
>
> Yeah, I know. But what to do in a case when there's no such  
> 'cookie' pointer in
> callback-function definition.

It's standard practice to have such void* arguments for callback  
functions. Otherwise you'd have to store something globally and not  
use two concurrently (which is obviously not ideal).

- Robert


From stefan_ml at behnel.de  Tue Feb 24 09:29:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 09:29:07 +0100 (CET)
Subject: [Cython] unicode can bite us ...
In-Reply-To: <72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
Message-ID: <49698.213.61.181.86.1235464147.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Feb 23, 2009, at 12:24 PM, Stefan Behnel wrote:
>> Lisandro Dalcin wrote:
>>> np.arange(10, dtype=s'i')
>> There's
>> absolutely no reason why the parameters should be byte strings.
>
> This is how unmarked string literals should work, right? Just like
> the Python interpreter.

Well, at least, that's what's written in the code: a byte string. What I'm
saying is that /requiring/ a byte string at the interface level is wrong.


> I think you underestimate how long broken libraries will be out there.

Let's wait and see. It didn't take me very long to fix up the Py3 unicode
problems of lxml's API (those that were independent of Cython), so I would
expect that any library can be fixed in a couple of weeks - as long as the
tools play nicely. I have no idea how hard it is to fix these things in a
SWIG wrapper, for example, but SWIG will also be fixed one day (and Cython
code will likely not interfere with a SWIG wrapper anyway).

Anyway, the NumPy case is not a use case for a new syntax feature, as it
requires plain byte strings for now, not identifiers. Lisandro, if you can
come up with a compelling use case that cannot easily be fixed in user
code, I'll rethink my opposition. For now, I cannot see how this feature
can lead to anything but even more broken interfaces.

Stefan


From stefan_ml at behnel.de  Tue Feb 24 09:33:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 09:33:52 +0100 (CET)
Subject: [Cython] unicode can bite us ...
In-Reply-To: <e7ba66e40902231348y740d3d57gfd8fd91baabfc90d@mail.gmail.com>
References: <e7ba66e40902231109q4af27164p72185bf9f69b1ad7@mail.gmail.com>
	<49A305F4.3080107@behnel.de>
	<72577FB9-F088-4EDB-820E-A90AF6ADB4B0@math.washington.edu>
	<e7ba66e40902231348y740d3d57gfd8fd91baabfc90d@mail.gmail.com>
Message-ID: <54907.213.61.181.86.1235464432.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> I'm on your side, Robert. At least, I would like to have a easy way to
> specify a string literal matching whatever Python uses for
> identifiers...
>
> And I still thing we should provide facilities to alleviate
> the str/unicode difference and the Py2/Py3 difference...

Well, then show me a case where this is required or at least helpful. My
gut feeling is that this feature will just push the problem we have on one
side of wrapper code over to the other side, without solving anything.

Anyway, this is all just bike-shedding unless we have real code to discuss.

Stefan


From stefan_ml at behnel.de  Tue Feb 24 09:38:18 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Feb 2009 09:38:18 +0100 (CET)
Subject: [Cython] Stefan, please fix this
In-Reply-To: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
References: <e7ba66e40902231336y5fc3b31eg58a12245811cab3c@mail.gmail.com>
Message-ID: <59691.213.61.181.86.1235464698.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> +        for entry in env.default_entries:
> +            if entry.type.is_pyobject and entry.used:
> -                code.putln("Py_DECREF(%s); %s = 0;" % (
> -                        code.entry_as_pyobject(entry), entry.cname))
> +                code.put_var_decref_clear(entry)
>
> Please revert to my implementation (bare Py_DECREF and nullify), or
> any other that does not break C compilation... IMHO, The module
> cleanup function is not a place to play with refnanny. Moreover, the
> module cleanup funcion is not refnanny-managed...

I agree.


> BTW, I believe that cleanup code should be generated when running the
> testsuite...

I'd even go one step further.

http://trac.cython.org/cython_trac/ticket/