From greg.ewing at canterbury.ac.nz Fri Aug 1 01:38:28 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Aug 2008 11:38:28 +1200 Subject: [Cython] Temp framework proposal In-Reply-To: <33322.85.166.116.245.1216973764.squirrel@webmail.uio.no> References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> <488823F0.4040301@canterbury.ac.nz> <33086.85.166.116.245.1216906830.squirrel@webmail.uio.no> <48895C8E.4010309@canterbury.ac.nz> <33322.85.166.116.245.1216973764.squirrel@webmail.uio.no> Message-ID: <48924CF4.3050305@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > I think it is mostly the CloneNode that scares me, that doesn't look very > intuitive to use from a transform perspective. On further reflection, CloneNode isn't the right thing to use, since it assumes it's being inserted during the analysis phase and doesn't quite do what we want. I think the best thing would be to add a new node type, such as LocalRefNode (and maybe rename LocalNode to LocalDefNode to better distinguish them). I'm not sure I like the idea of re-using NameNode, since it's intended to represent a name appearing in the source code, and temp locals don't correspond to any name in the source. -- Greg From greg.ewing at canterbury.ac.nz Fri Aug 1 03:43:59 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Aug 2008 13:43:59 +1200 Subject: [Cython] Refactor or surgery on global constants? In-Reply-To: <48920CCC.4070301@student.matnat.uio.no> References: <48920CCC.4070301@student.matnat.uio.no> Message-ID: <48926A5F.4060009@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > 2) Move these things from the scopes to code.global (a global context of > the CCodeWriter). This fits nicely with utility_code going over as well. > This means that for now each scope simply pipe their things into code > where they are merged, while in time then e.g. StringNode could intern a > string during code generation rather than bothering with it during > analysis. This sounds good to me. -- Greg From stefan_ml at behnel.de Fri Aug 1 06:41:13 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 01 Aug 2008 06:41:13 +0200 Subject: [Cython] Cython and complex floats In-Reply-To: <488EE487.90102@student.matnat.uio.no> References: <488EE487.90102@student.matnat.uio.no> Message-ID: <489293E9.8020003@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Buffers and Python has the complex datatype, while I'm reasonably sure I > haven't found anything about it in Cython (?). You can use Python's complex types in Cython by simply declaring them as an extension type: cdef extern from "complexobject.h": struct Py_complex: double real double imag ctypedef class __builtin__.complex [object PyComplexObject]: cdef Py_complex cval # A function which uses the above type def spam(complex c): print "Real:", c.cval.real print "Imag:", c.cval.imag That's already done in Cython/Includes/python2.5.pxd. > 1) Complex numbers can be pulled out into a struct consisting of two > floats. Like this: > > cdef struct cdouble: > double real > double imag > > def f(object[cdouble] buf): > ... I'm not sure we need such syntactic sugar. All you gain is one ".cval" less. Or do I misunderstand your proposal here? > Operations on the fields must happen manually. > > 2) In addition, Cython gets support for the C99 "complex" float modifier. > > def complex double sum(object[complex double] buf): > cdef complex double result = 0 + 0j > ... > > Here, operations between complex datatypes are done natively by the > compiler. However the code generated is illegal on non-C99-compilers. I think it's fine for a specific feature to break compiler compatibility as long as it is clearly stated in the docs and it does not impact code that does not use it. Stefan From robertwb at math.washington.edu Fri Aug 1 06:58:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 31 Jul 2008 21:58:07 -0700 Subject: [Cython] Refactor or surgery on global constants? In-Reply-To: <48926A5F.4060009@canterbury.ac.nz> References: <48920CCC.4070301@student.matnat.uio.no> <48926A5F.4060009@canterbury.ac.nz> Message-ID: On Jul 31, 2008, at 6:43 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> 2) Move these things from the scopes to code.global (a global >> context of >> the CCodeWriter). This fits nicely with utility_code going over as >> well. >> This means that for now each scope simply pipe their things into code >> where they are merged, while in time then e.g. StringNode could >> intern a >> string during code generation rather than bothering with it during >> analysis. > > This sounds good to me. +1 from me too. - Robert From robertwb at math.washington.edu Fri Aug 1 07:03:12 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 31 Jul 2008 22:03:12 -0700 Subject: [Cython] Cython and complex floats In-Reply-To: <489293E9.8020003@behnel.de> References: <488EE487.90102@student.matnat.uio.no> <489293E9.8020003@behnel.de> Message-ID: On Jul 31, 2008, at 9:41 PM, Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> Buffers and Python has the complex datatype, while I'm reasonably >> sure I >> haven't found anything about it in Cython (?). > > You can use Python's complex types in Cython by simply declaring > them as an > extension type: > > cdef extern from "complexobject.h": > > struct Py_complex: > double real > double imag > > ctypedef class __builtin__.complex [object PyComplexObject]: > cdef Py_complex cval > > # A function which uses the above type > def spam(complex c): > print "Real:", c.cval.real > print "Imag:", c.cval.imag > > That's already done in Cython/Includes/python2.5.pxd. > > >> 1) Complex numbers can be pulled out into a struct consisting of two >> floats. Like this: >> >> cdef struct cdouble: >> double real >> double imag >> >> def f(object[cdouble] buf): >> ... > > I'm not sure we need such syntactic sugar. All you gain is one > ".cval" less. > > Or do I misunderstand your proposal here? I think the point is to operate on complex numbers without the object baggage (the same way one can operate on C doubles). I'm not sure what the best long term option is (out of the 3 you proposed) but 1 seems reasonable for now--it will be quite a bit of work to fully support complexes via macros (especially when evaluating mixed expressions) but should probably eventually be done. > > >> Operations on the fields must happen manually. >> >> 2) In addition, Cython gets support for the C99 "complex" float >> modifier. >> >> def complex double sum(object[complex double] buf): >> cdef complex double result = 0 + 0j >> ... >> >> Here, operations between complex datatypes are done natively by the >> compiler. However the code generated is illegal on non-C99-compilers. > > I think it's fine for a specific feature to break compiler > compatibility as > long as it is clearly stated in the docs and it does not impact > code that does > not use it. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Fri Aug 1 09:27:16 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 Aug 2008 09:27:16 +0200 Subject: [Cython] Cython and complex floats In-Reply-To: <489293E9.8020003@behnel.de> References: <488EE487.90102@student.matnat.uio.no> <489293E9.8020003@behnel.de> Message-ID: <4892BAD4.5070602@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> Buffers and Python has the complex datatype, while I'm reasonably sure I >> haven't found anything about it in Cython (?). > > You can use Python's complex types in Cython by simply declaring them as an > extension type: > > cdef extern from "complexobject.h": > > struct Py_complex: > double real > double imag > > ctypedef class __builtin__.complex [object PyComplexObject]: > cdef Py_complex cval > > # A function which uses the above type > def spam(complex c): > print "Real:", c.cval.real > print "Imag:", c.cval.imag > > That's already done in Cython/Includes/python2.5.pxd. Thanks. Buffers support more types though (Zf, Zd and Zg, ie complex float, complex double, complex long double), so I'm not sure how useful this would be. > > >> 1) Complex numbers can be pulled out into a struct consisting of two >> floats. Like this: >> >> cdef struct cdouble: >> double real >> double imag >> >> def f(object[cdouble] buf): >> ... > > I'm not sure we need such syntactic sugar. All you gain is one ".cval" less. > > Or do I misunderstand your proposal here? The idea is getting the data out of the buffer for numeric calculations. In the buffer it is stored as two floats (of whatever size), not as a Python object, so that's what the buffer "contains". I could add syntactic sugar to wrap it in Python objects but that would defeat the purpose of buffers as you would loose speed (just use a memoryview then). (Note that buffers support structs, I would just support structs first and then map "Zf" to "ff"). The idea was that I would initially just map "Zx" to "a struct of two x" and let the user deal with it. It is not a long-term solution, but perhaps we can get a user-base without it first. Of course, if I go with 2) or 3), any complex native type will coerce to the Python complex object (perhaps with a need for explicit cast to go down from long double). -- Dag Sverre From stefan_ml at behnel.de Fri Aug 1 10:22:29 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 01 Aug 2008 10:22:29 +0200 Subject: [Cython] Refactor or surgery on global constants? In-Reply-To: <48920CCC.4070301@student.matnat.uio.no> References: <48920CCC.4070301@student.matnat.uio.no> Message-ID: <4892C7C5.1040806@behnel.de> Hi, Dag Sverre Seljebotn wrote: > In making it possible to have runnable code in pxds, I need to make a > choice. The problem is that interned strings, objects etc. live in > seperate scopes and you get double copies, basically now I get the > following code twice in my c-file: > > static char __pyx_k___getbuffer__[] = "__getbuffer__"; > static PyObject *__pyx_kp___getbuffer__; > > which gcc isn't too thrilled about. > [...] > 2) Move these things from the scopes to code.global (a global context of > the CCodeWriter). This fits nicely with utility_code going over as well. > This means that for now each scope simply pipe their things into code > where they are merged, while in time then e.g. StringNode could intern a > string during code generation rather than bothering with it during > analysis. (I think this is a viable way forward, but may take an hour or > three longer for me to do.) Since I previously worked a bit on the string interning code, I think it would really benefit from such a change. Please go ahead. Stefan From dalcinl at gmail.com Fri Aug 1 18:10:12 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 1 Aug 2008 13:10:12 -0300 Subject: [Cython] vtables and the type cache Message-ID: In general, for Py >= 2.6, we sould invalidate the type cache every time we set something in the 'tp_dict' field. 'Pyx_SetVtable()' adds stuff there. However, this call is made immediately after 'PyType_Ready()', so at first it seems the call to 'Pyx_TypeModified()' is not needed. But I'm not completelly sure about this, and anyway perhaps the safest thing to do is to invalidate the cache, Comments? BTW, I've already have a patch for this. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From abergeron at gmail.com Fri Aug 1 21:48:01 2008 From: abergeron at gmail.com (Arnaud Bergeron) Date: Fri, 1 Aug 2008 15:48:01 -0400 Subject: [Cython] Equivalent of __new__() in python Message-ID: Sorry if this is the wrong list to ask questions like that, I couldn't find any other list for cython. I am looking for a way to implement the uniqueness of a class in Cython. I have a class say: cdef class Unique(object): cdef val def __init__(self, val): self.val = val And I want to have only one instance at a time with a single value of val. I already know how to do this in python using the __new__() method and a WeakValueDictionary. I am looking for a way to do the equivalent in cython. And if this requires monkey-patching the tp_new slot of the class, perhaps some pointers as to what should the replacement method should at least do in order not to screw the rest of the system. Arnaud From ondrej at certik.cz Sat Aug 2 19:40:23 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 2 Aug 2008 19:40:23 +0200 Subject: [Cython] couple gcc warnings problems Message-ID: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> Hi, $ cython -v Cython version 0.9.8 $ cython --convert-range step.pyx $ gcc -O3 -W -Wall -I/usr/include/python2.5/ -I/usr/include/numpy/ -c -o step.o step.c step.c: In function '__pyx_pf_4step_temperature': step.c:608: warning: unused parameter '__pyx_self' step.c: In function '__pyx_pf_4step_advance': step.c:739: warning: unused parameter '__pyx_self' step.c: In function '__pyx_pf_4step_assign_speeds': step.c:1068: warning: unused parameter '__pyx_self' step.c: In function '__pyx_pf_4step_fit_in_the_box': step.c:1336: warning: unused parameter '__pyx_self' step.c: In function '__pyx_pf_4step_get_f': step.c:1675: warning: unused parameter '__pyx_self' step.c: In function '__pyx_pf_4step_get_f2': step.c:1794: warning: unused parameter '__pyx_self' step.c: In function '__pyx_pf_4step_calculate_potential': step.c:1941: warning: unused parameter '__pyx_self' step.c: At top level: step.c:2052: warning: missing initializer step.c:2052: warning: (near initialization for '__pyx_string_tab[11].is_identifier') /usr/include/numpy/__multiarray_api.h:959: warning: '_import_array' defined but not used Now you can discover couple problems above. The last warning: step.c:2052: warning: missing initializer step.c:2052: warning: (near initialization for '__pyx_string_tab[11].is_identifier') could be fixed by this patch to the step.c file: $ quilt diff Index: MD-spheres/step.c =================================================================== --- MD-spheres.orig/step.c 2008-08-02 19:36:34.680765668 +0200 +++ MD-spheres/step.c 2008-08-02 19:36:50.085854557 +0200 @@ -2049,7 +2049,7 @@ {&__pyx_kp_append, __pyx_k_append, sizeof(__pyx_k_append), 1, 1, 1}, {&__pyx_kp_gauss, __pyx_k_gauss, sizeof(__pyx_k_gauss), 1, 1, 1}, {&__pyx_kp_get_f, __pyx_k_get_f, sizeof(__pyx_k_get_f), 0, 1, 1}, - {0, 0, 0, 0, 0} + {0, 0, 0, 0, 0, 0} }; static struct PyMethodDef __pyx_methods[] = { So this seems like a bug in Cython. As to the other warnings: step.c:608: warning: unused parameter '__pyx_self' If you look into the generated file: static PyObject *__pyx_pf_4step_temperature(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) { Why does Cython generate the "self" argument? This is just a regular function, not a method. And why isn't the first argument used? Is this a bug? Ondrej -------------- next part -------------- A non-text attachment was scrubbed... Name: step.pyx Type: application/octet-stream Size: 4221 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080802/c1dc4e03/attachment.obj From robertwb at math.washington.edu Sat Aug 2 20:41:53 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 2 Aug 2008 11:41:53 -0700 Subject: [Cython] couple gcc warnings problems In-Reply-To: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> References: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> Message-ID: On Aug 2, 2008, at 10:40 AM, Ondrej Certik wrote: > Hi, > > $ cython -v > Cython version 0.9.8 > $ cython --convert-range step.pyx > $ gcc -O3 -W -Wall -I/usr/include/python2.5/ -I/usr/include/numpy/ -c > -o step.o step.c > step.c: In function '__pyx_pf_4step_temperature': > step.c:608: warning: unused parameter '__pyx_self' > step.c: In function '__pyx_pf_4step_advance': > step.c:739: warning: unused parameter '__pyx_self' > step.c: In function '__pyx_pf_4step_assign_speeds': > step.c:1068: warning: unused parameter '__pyx_self' > step.c: In function '__pyx_pf_4step_fit_in_the_box': > step.c:1336: warning: unused parameter '__pyx_self' > step.c: In function '__pyx_pf_4step_get_f': > step.c:1675: warning: unused parameter '__pyx_self' > step.c: In function '__pyx_pf_4step_get_f2': > step.c:1794: warning: unused parameter '__pyx_self' > step.c: In function '__pyx_pf_4step_calculate_potential': > step.c:1941: warning: unused parameter '__pyx_self' > step.c: At top level: > step.c:2052: warning: missing initializer > step.c:2052: warning: (near initialization for > '__pyx_string_tab[11].is_identifier') > /usr/include/numpy/__multiarray_api.h:959: warning: '_import_array' > defined but not used > > > > > Now you can discover couple problems above. The last warning: > > step.c:2052: warning: missing initializer > step.c:2052: warning: (near initialization for > '__pyx_string_tab[11].is_identifier') > > could be fixed by this patch to the step.c file: > > $ quilt diff > Index: MD-spheres/step.c > =================================================================== > --- MD-spheres.orig/step.c 2008-08-02 19:36:34.680765668 +0200 > +++ MD-spheres/step.c 2008-08-02 19:36:50.085854557 +0200 > @@ -2049,7 +2049,7 @@ > {&__pyx_kp_append, __pyx_k_append, sizeof(__pyx_k_append), 1, 1, > 1}, > {&__pyx_kp_gauss, __pyx_k_gauss, sizeof(__pyx_k_gauss), 1, 1, 1}, > {&__pyx_kp_get_f, __pyx_k_get_f, sizeof(__pyx_k_get_f), 0, 1, 1}, > - {0, 0, 0, 0, 0} > + {0, 0, 0, 0, 0, 0} > }; > > static struct PyMethodDef __pyx_methods[] = { > > > > So this seems like a bug in Cython. As to the other warnings: > > step.c:608: warning: unused parameter '__pyx_self' > > If you look into the generated file: > > static PyObject *__pyx_pf_4step_temperature(PyObject *__pyx_self, > PyObject *__pyx_args, PyObject *__pyx_kwds) { > > > Why does Cython generate the "self" argument? This is just a regular > function, not a method. And why isn't the first argument used? Is this > a bug? This is not a bug, it is because Python functions and methods have exactly the same signatures to be able to treat them interchangeably as first class objects. I believe in this case the self argument is the actual module, though it is clearly unneeded. If you look at the built in modules they do the exact same thing. Is there anything we could do to suppress this warning (fool the compiler into thinking it's used, but still have it be optimized away)? - Robert From ondrej at certik.cz Sat Aug 2 21:10:24 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 2 Aug 2008 21:10:24 +0200 Subject: [Cython] couple gcc warnings problems In-Reply-To: References: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> Message-ID: <85b5c3130808021210s1b4c74c4ldb0cc150997f4fe7@mail.gmail.com> On Sat, Aug 2, 2008 at 8:41 PM, Robert Bradshaw wrote: > On Aug 2, 2008, at 10:40 AM, Ondrej Certik wrote: > >> Hi, >> >> $ cython -v >> Cython version 0.9.8 >> $ cython --convert-range step.pyx >> $ gcc -O3 -W -Wall -I/usr/include/python2.5/ -I/usr/include/numpy/ -c >> -o step.o step.c >> step.c: In function '__pyx_pf_4step_temperature': >> step.c:608: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_advance': >> step.c:739: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_assign_speeds': >> step.c:1068: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_fit_in_the_box': >> step.c:1336: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_get_f': >> step.c:1675: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_get_f2': >> step.c:1794: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_calculate_potential': >> step.c:1941: warning: unused parameter '__pyx_self' >> step.c: At top level: >> step.c:2052: warning: missing initializer >> step.c:2052: warning: (near initialization for >> '__pyx_string_tab[11].is_identifier') >> /usr/include/numpy/__multiarray_api.h:959: warning: '_import_array' >> defined but not used >> >> >> >> >> Now you can discover couple problems above. The last warning: >> >> step.c:2052: warning: missing initializer >> step.c:2052: warning: (near initialization for >> '__pyx_string_tab[11].is_identifier') >> >> could be fixed by this patch to the step.c file: >> >> $ quilt diff >> Index: MD-spheres/step.c >> =================================================================== >> --- MD-spheres.orig/step.c 2008-08-02 19:36:34.680765668 +0200 >> +++ MD-spheres/step.c 2008-08-02 19:36:50.085854557 +0200 >> @@ -2049,7 +2049,7 @@ >> {&__pyx_kp_append, __pyx_k_append, sizeof(__pyx_k_append), 1, 1, >> 1}, >> {&__pyx_kp_gauss, __pyx_k_gauss, sizeof(__pyx_k_gauss), 1, 1, 1}, >> {&__pyx_kp_get_f, __pyx_k_get_f, sizeof(__pyx_k_get_f), 0, 1, 1}, >> - {0, 0, 0, 0, 0} >> + {0, 0, 0, 0, 0, 0} >> }; >> >> static struct PyMethodDef __pyx_methods[] = { >> >> >> >> So this seems like a bug in Cython. As to the other warnings: >> >> step.c:608: warning: unused parameter '__pyx_self' >> >> If you look into the generated file: >> >> static PyObject *__pyx_pf_4step_temperature(PyObject *__pyx_self, >> PyObject *__pyx_args, PyObject *__pyx_kwds) { >> >> >> Why does Cython generate the "self" argument? This is just a regular >> function, not a method. And why isn't the first argument used? Is this >> a bug? > > This is not a bug, it is because Python functions and methods have > exactly the same signatures to be able to treat them interchangeably > as first class objects. I believe in this case the self argument is > the actual module, though it is clearly unneeded. If you look at the > built in modules they do the exact same thing. > > Is there anything we could do to suppress this warning (fool the > compiler into thinking it's used, but still have it be optimized away)? I don't know, it'd be nice though surpress the warnings, since this is a wanted behavior. So that gcc only warns when something goes wrong. How about the other thing with 0,0,0 above? That seems to me like a bug. Ondrej From cwitty at newtonlabs.com Sat Aug 2 21:56:27 2008 From: cwitty at newtonlabs.com (Carl Witty) Date: Sat, 2 Aug 2008 12:56:27 -0700 Subject: [Cython] couple gcc warnings problems In-Reply-To: References: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> Message-ID: On Sat, Aug 2, 2008 at 11:41 AM, Robert Bradshaw wrote: > Is there anything we could do to suppress this warning (fool the > compiler into thinking it's used, but still have it be optimized away)? Assigning the variable to itself worked in the following test: void foo(char *x) { x = x; } You can also suppress the warning with a gcc attribute, but that's compiler-specific, so you'd have to guard it with ifdefs etc. Carl From robertwb at math.washington.edu Sun Aug 3 01:41:18 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 2 Aug 2008 16:41:18 -0700 Subject: [Cython] Object not released when calling cpdef method from inherited class ? In-Reply-To: References: <1215488230.20373.14.camel@nikopol> Message-ID: <5F7B89E0-246E-4493-8CCA-EE22ECD7FD14@math.washington.edu> On Jul 7, 2008, at 8:42 PM, Robert Bradshaw wrote: > On Jul 7, 2008, at 8:37 PM, Guillaume Chereau wrote: > >> Hello all, >> >> I am currently using cython for a project related to openmoko cell >> phone : http://charlie137-2.blogspot.com/2008/07/introducing- >> tichy.html >> >> I noticed that the memory is constantly increasing when I use >> cython. I >> tracked down the problem to the case where I create a subclass of a >> cython class and then redefine a cpdef method, asking it to call the >> parent method, but ONLY if the cpdef method then call an other cpdef >> method ! >> >> Here is the smallest example I could come with that fails : >> >> >> == test.pyx == >> >> cdef class A: >> cpdef func(self): >> return >> >> cpdef test(self): >> self.func() >> >> >> == main.py == >> >> import test >> import gc >> >> class B(test.A): >> def test(self): >> test.A.test(self) >> >> b = B() >> >> for i in range(10): >> b.test() >> gc.collect() >> print len(gc.get_objects()) >> >> >> The output will show that some objects are not released. >> >> Is it a known bug ? Is there a way to avoid it ? > > This isn't a known bug, but does sound like something fishy is > going on. Thanks for the report. The easiest fix I have for now is > to make one or the other methods an ordinary def method > (sacrificing speed for better memory handling), but I'll be looking > into it. See http://hg.cython.org/cython-devel/rev/f612fb161cd0 -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080802/34cf2291/attachment.pgp From casevh at gmail.com Sun Aug 3 04:53:16 2008 From: casevh at gmail.com (Case Vanhorsen) Date: Sat, 2 Aug 2008 19:53:16 -0700 Subject: [Cython] Accessing the internals of a Python long Message-ID: <99e0b9530808021953j5af998a1m5e431ed1918de798@mail.gmail.com> I am trying to access the internal structure of a Python long. The code fragment below works when I use Pyrex 0.9.8.4 but generates a compile error when using Cython 0.9.8. Is there a better way to access the internals of a Python long? What is causing the cimport error and how can I fix it? Thanks, casevh ------------------------------------------------------ The code: cdef extern from "Python.h": ctypedef struct PyTypeObject: pass ctypedef struct PyObject: Py_ssize_t ob_refcnt PyTypeObject * ob_type cdef extern from "longintrepr.h": cdef struct _longobject: int ob_refcnt PyTypeObject * ob_type int ob_size unsigned int * ob_digit def test(temp = long(0)): cdef _longobject *l l = <_longobject *> temp print sizeof(l.ob_size) print sizeof(l.ob_digit[0]) The error: Error converting Pyrex file to C: ------------------------------------------------------------ ... int ob_size unsigned int * ob_digit def test(temp = long(0)): cdef _longobject *l l = <_longobject *> temp print sizeof(l.ob_size) ^ ------------------------------------------------------------ /home/case/code/cvh.pyx:16:18: 'cvh.test' is not a cimported module -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080802/0724b91a/attachment.htm From stefan_ml at behnel.de Sun Aug 3 09:12:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Aug 2008 09:12:57 +0200 Subject: [Cython] Equivalent of __new__() in python In-Reply-To: References: Message-ID: <48955A79.7000500@behnel.de> Hi, Arnaud Bergeron wrote: > Sorry if this is the wrong list to ask questions like that, I couldn't > find any other list for cython. cython-dev is our general purpose list, so it's fine to ask here. > I am looking for a way to implement the uniqueness of a class in > Cython. You mean a singleton. A class is always unique, but it's instances are not necessarily. There are many recipes for implementing singletons in Python, many of which also work in Cython. > I have a class say: > > cdef class Unique(object): > cdef val > > def __init__(self, val): > self.val = val > > And I want to have only one instance at a time with a single value of val. If you meant one instance for each distinct value of val, then a factory might work: cdef class _Unique(object): cdef object __weakref__ ... def Unique(val): # do your usual WeakValueDictionary handling here. return the_right_instance > And if this requires monkey-patching the tp_new slot of the class, > perhaps some pointers as to what should the replacement method should > at least do in order not to screw the rest of the system. I'd avoid that for now, unless you really require subclassing support etc. Stefan From stefan_ml at behnel.de Sun Aug 3 09:54:23 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Aug 2008 09:54:23 +0200 Subject: [Cython] couple gcc warnings problems In-Reply-To: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> References: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> Message-ID: <4895642F.1090400@behnel.de> Hi, Ondrej Certik wrote: > step.c:2052: warning: missing initializer > step.c:2052: warning: (near initialization for > '__pyx_string_tab[11].is_identifier') > > could be fixed by this patch to the step.c file: > > {&__pyx_kp_gauss, __pyx_k_gauss, sizeof(__pyx_k_gauss), 1, 1, 1}, > {&__pyx_kp_get_f, __pyx_k_get_f, sizeof(__pyx_k_get_f), 0, 1, 1}, > - {0, 0, 0, 0, 0} > + {0, 0, 0, 0, 0, 0} > }; Fixed. Stefan From dagss at student.matnat.uio.no Sun Aug 3 21:29:06 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 03 Aug 2008 21:29:06 +0200 Subject: [Cython] Refactor or surgery on global constants? In-Reply-To: <4892C7C5.1040806@behnel.de> References: <48920CCC.4070301@student.matnat.uio.no> <4892C7C5.1040806@behnel.de> Message-ID: <48960702.3060507@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> In making it possible to have runnable code in pxds, I need to make a >> choice. The problem is that interned strings, objects etc. live in >> seperate scopes and you get double copies, basically now I get the >> following code twice in my c-file: >> >> static char __pyx_k___getbuffer__[] = "__getbuffer__"; >> static PyObject *__pyx_kp___getbuffer__; >> >> which gcc isn't too thrilled about. >> [...] >> 2) Move these things from the scopes to code.global (a global context of >> the CCodeWriter). This fits nicely with utility_code going over as well. >> This means that for now each scope simply pipe their things into code >> where they are merged, while in time then e.g. StringNode could intern a >> string during code generation rather than bothering with it during >> analysis. (I think this is a viable way forward, but may take an hour or >> three longer for me to do.) > > Since I previously worked a bit on the string interning code, I think it would > really benefit from such a change. Please go ahead. 1) OK I've made some changes in this direction: - BlockNode now forwards the entries it deals with to Code.GlobalState, which tries to do the same thing but filters out duplicates (which comes from pxds and pyxes doing the same thing). In time, I expect all of BlockNode to vanish. - Also, the code dealing with interned strings/ints and cached builtins from ModuleNode is moved to Code.GlobalState; which uses a different strategy: Rather than iterating over the contents of a scope, it simply allocates many CodeWriter instances/buffers and outputs initialiazation code at the same time as the declaration code, though to a seperate buffer. (One can like or not like this I suppose -- only argument I have is it tends to keep code more "together" in one location, but the real reason is that it was easier this way.) - Because one cannot have insertion points within functions (would create h). Therefore I extracted some things from the module initialization function into two more functions, __Pyx_InitGlobals and __Pyx_InitCachedBuiltins. I could potentially get rid of this with a special "I promise not to use temps or modify labels"-insertion-point, but this was easier. (Also, one could argue for it from a design viewpoint: c-file globals are more tied to the c-file as such than necesarrily the module -- if a single C file were to implement both a module and some of its submodules [1], then cached builtins could be shared but the module init func would not be!) This all runs the test cases etc. and I consider it "done" in some sense, but I want to have some feedback before I consider it ready for merging. Of course, it represents a transitional solution. In the end, I expect one would add something like this: self.result_code = code.globalcode.py_string_literal("abc") and so on (details must be discussed when the day comes), but that, as my new mantra has become, "is pending a result_code refactoring". But the design now allows for creating such a thing, it would all happen within Code.py. (I'll get to that some day of course, especially as Greg has a lot of it done, but I have my GSoC priorities to worry about). [1] (Is this possible in Python? What I've done now opens up for Cython supporting this (think compiling a dir tree to a single .so) and get some degree of sharing constant literals etc., but that's hardly a priority.) 2) During writing this, I realized that some code could be written in a shorter fashion: (self.initwriter .putln("return 0;") .put_label(self.initwriter.error_label) .putln("return -1;") .putln("}") .exit_cfunc_scope() ) ...where I've changed some functions to return "self" (much like C++ iostreams). Do you like or dislike this? Consider both with and without the indendation-matching-the-C-code. It does create some dissymetry (see the put_label where you need a new reference to the writer), this could be fixed by adding put_error_label, or by going back The counterargument is that I could say "c = self.initwriter" first instead to get almost the same brevity. Of course, this is only an added option for new code. So inconsistency would be added. Still I'm +1/2 because of when doing all the C code I've written in Buffer.py I always thought that it tended to become quite long-winded to express what I wanted... (Some code is using this in my branch which I'll redo if the vote is not positive.) 3) For buffers, this means that I've manged to strip all NumPy specific code from Buffer.py, instead you currently can define __getbuffer__ and __releasebuffer__ in numpy.pxd I have not introduced a generic "final" mechanism or similar; rather an exception is introduced specifically for __getbuffer__ and __releasebuffer__ and specifically for the purpose of Py2.5-GetBuffer-faking. I'd like to leave it like this for now, but it will be a "hidden" feature and I will warn people not to rely on it. In time I see it like this: A "final" language feature is added, and anything declared "final" can reside in pxds. But that is not done now, hence the special cases for these two functions. -- Dag Sverre From dagss at student.matnat.uio.no Sun Aug 3 21:44:13 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 03 Aug 2008 21:44:13 +0200 Subject: [Cython] Type scope vs. runtime scope Message-ID: <48960A8D.1090607@student.matnat.uio.no> In numpy.pxd I want to have this: ctypedef npy_int64 int64 and be able to use it like this (this is what a NumPy user would expect to do): cimport numpy cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], numpy.int64) This however creates an error: 'int64' is not a constant, variable or function identifier since int64 is declared as a type in the scope. I've had rather complicated and grand schemes for this earlier, but what I'll propose this time is much simpler: When not in a "type context" (what you are writing cannot possibly be a C type), C types vanish from the scope/are not considered .. so that the ctypedef numpy.int64 does not Warning! In theory, this principle could allow code like x = 4 ctypedef int x cdef x v = x If you didn't like this but have time, you could have a look at http://wiki.cython.org/enhancements/runtimectypes which is another proposal I could use to achieve my ends, however I'm reluctant to think or discuss something going so much into language design right now. -- Dag Sverre From dagss at student.matnat.uio.no Sun Aug 3 22:04:37 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 03 Aug 2008 22:04:37 +0200 Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <48960A8D.1090607@student.matnat.uio.no> References: <48960A8D.1090607@student.matnat.uio.no> Message-ID: <48960F55.1050806@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > If you didn't like this but have time, you could have a look at > > http://wiki.cython.org/enhancements/runtimectypes > > which is another proposal I could use to achieve my ends, however I'm > reluctant to think or discuss something going so much into language > design right now. > Actually, I'll retract this last statement, as I rather like some of the consequences of that proposal (would mean that printing a "cdef numpy.float64 x" would be subject to NumPy's float64 formatting rules, for instance). (Even so, a straight approval of this now will just mean that I'll postpone the issue rather than implement the quicker scope change I proposed.) -- Dag Sverre From greg.ewing at canterbury.ac.nz Mon Aug 4 04:05:46 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 04 Aug 2008 14:05:46 +1200 Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <48960A8D.1090607@student.matnat.uio.no> References: <48960A8D.1090607@student.matnat.uio.no> Message-ID: <489663FA.7010307@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], numpy.int64) > > This however creates an error: > 'int64' is not a constant, variable or function identifier You might like to look into how extension types are made to behave as both types and runtime values -- this sounds like it might be something similar. -- Greg From dagss at student.matnat.uio.no Mon Aug 4 12:46:17 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 12:46:17 +0200 Subject: [Cython] Scope of Py_buffer Message-ID: <4896DDF9.80705@student.matnat.uio.no> I'm wondering if I could move Py_buffer out of the default scope, in order to reduce the namespace incompatability. (Py_buffer is present in the global scope for the purposes of the second argument to __getbuffer__ and __releasebuffer__ ). The alternatives I see are either "cython.Py_buffer" or "python_buffer.Py_buffer" (or both), where you need to cimport cython/python_buffer first. This means that one needs to cimport something in order to successfully implement __getbuffer__, however I consider this ok myself. But I certainly see that one could argue for it being a builtin for that reason. As the buffer support until now has only been available for Py3/Py2.6 which are still not released, this shouldn't need an deprecation warning or similar, while later it will be difficult to do this. -- Dag Sverre From dagss at student.matnat.uio.no Mon Aug 4 15:50:19 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 15:50:19 +0200 Subject: [Cython] Make positive int literals unsigned long? Message-ID: <4897091B.1000604@student.matnat.uio.no> If we make positive int literals unsigned long rather than plain long, constant-index buffer lookups are going to be optimized for free. Is there going to be any ill side-effects from doing this? (If not, I can always special-case int literals used for buffer lookups.) -- Dag Sverre From stefan_ml at behnel.de Mon Aug 4 16:14:00 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 4 Aug 2008 16:14:00 +0200 (CEST) Subject: [Cython] Scope of Py_buffer In-Reply-To: <4896DDF9.80705@student.matnat.uio.no> References: <4896DDF9.80705@student.matnat.uio.no> Message-ID: <33785.213.61.181.86.1217859240.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi Dag, Dag Sverre Seljebotn wrote: > I'm wondering if I could move Py_buffer out of the default scope, in > order to reduce the namespace incompatability. (Py_buffer is present in > the global scope for the purposes of the second argument to > __getbuffer__ and __releasebuffer__ ). The alternatives I see are either > "cython.Py_buffer" or "python_buffer.Py_buffer" (or both), where you > need to cimport cython/python_buffer first. I see Py_buffer in a similar light as Py_ssize_t. Would you also argue for that being moved into a separate namespace? Since Py_buffer is a requirement for implementing a standard Python C-API method, I think it makes sense to keep it in the global namespace as a standard data type of Cython. Stefan From stefan_ml at behnel.de Mon Aug 4 16:28:51 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 4 Aug 2008 16:28:51 +0200 (CEST) Subject: [Cython] Make positive int literals unsigned long? In-Reply-To: <4897091B.1000604@student.matnat.uio.no> References: <4897091B.1000604@student.matnat.uio.no> Message-ID: <48674.213.61.181.86.1217860131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > If we make positive int literals unsigned long rather than plain long, > constant-index buffer lookups are going to be optimized for free. Is > there going to be any ill side-effects from doing this? > > (If not, I can always special-case int literals used for buffer lookups.) What about enabling this, adding a test module dedicated to integer literals, casting and arithmetic, and running the (complete) test suite with -Wall -pedantic ? Stefan From dagss at student.matnat.uio.no Mon Aug 4 16:37:42 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 16:37:42 +0200 Subject: [Cython] Scope of Py_buffer In-Reply-To: <33785.213.61.181.86.1217859240.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <4896DDF9.80705@student.matnat.uio.no> <33785.213.61.181.86.1217859240.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48971436.3080102@student.matnat.uio.no> Stefan Behnel wrote: > Hi Dag, > > Dag Sverre Seljebotn wrote: >> I'm wondering if I could move Py_buffer out of the default scope, in >> order to reduce the namespace incompatability. (Py_buffer is present in >> the global scope for the purposes of the second argument to >> __getbuffer__ and __releasebuffer__ ). The alternatives I see are either >> "cython.Py_buffer" or "python_buffer.Py_buffer" (or both), where you >> need to cimport cython/python_buffer first. > > I see Py_buffer in a similar light as Py_ssize_t. Would you also argue for > that being moved into a separate namespace? I would, but that is water under the bridge now -- it would call for a round of deprecation warnings etc., so might as well leave it until Python compatability is otherwise much higher (though if Py_buffer is moved, one should definitly create cython.Py_ssize_t as well now as an alias). The point is that Py_buffer is probably not used in any code yet, or if it is, it is only for Py3 beta purposes, so a deprecation warning round is not needed IMO. > Since Py_buffer is a requirement for implementing a standard Python C-API > method, I think it makes sense to keep it in the global namespace as a > standard data type of Cython. Yes, and I mentioned this counter-argument too. For me it weighs less than a higher degree of Python compatability. I don't mind letting __builtin__ be strictly Python API, and require imports for using the C-API in any way (or anything else that we add above pure Python). Anyway, the only reason I'm bringing it up now is the timing, and seeing that you didn't like it I'm ready to drop it. (While on namespace issues, the Tao says "flat is better than nested" which is in favor of cython.shape rather than cython.buffer.shape?) -- Dag Sverre From dagss at student.matnat.uio.no Mon Aug 4 16:37:50 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 16:37:50 +0200 Subject: [Cython] Make positive int literals unsigned long? In-Reply-To: <48674.213.61.181.86.1217860131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <4897091B.1000604@student.matnat.uio.no> <48674.213.61.181.86.1217860131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4897143E.3020304@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> If we make positive int literals unsigned long rather than plain long, >> constant-index buffer lookups are going to be optimized for free. Is >> there going to be any ill side-effects from doing this? >> >> (If not, I can always special-case int literals used for buffer lookups.) > > What about enabling this, adding a test module dedicated to integer > literals, casting and arithmetic, and running the (complete) test suite > with > > -Wall -pedantic > > ? Sure, but that would have been a lot of work only to get a reply here afterwards that "this whole idea would obviously not work because X" from people who know the code base better :-) -- Dag Sverre From abergeron at gmail.com Mon Aug 4 16:55:06 2008 From: abergeron at gmail.com (Arnaud Bergeron) Date: Mon, 4 Aug 2008 10:55:06 -0400 Subject: [Cython] Equivalent of __new__() in python In-Reply-To: <48955A79.7000500@behnel.de> References: <48955A79.7000500@behnel.de> Message-ID: 2008/8/3 Stefan Behnel : > Hi, > > Arnaud Bergeron wrote: >> Sorry if this is the wrong list to ask questions like that, I couldn't >> find any other list for cython. > > cython-dev is our general purpose list, so it's fine to ask here. > > >> I am looking for a way to implement the uniqueness of a class in >> Cython. > > You mean a singleton. A class is always unique, but it's instances are not > necessarily. Yes, a singleton. > There are many recipes for implementing singletons in Python, many of which > also work in Cython. > > >> I have a class say: >> >> cdef class Unique(object): >> cdef val >> >> def __init__(self, val): >> self.val = val >> >> And I want to have only one instance at a time with a single value of val. > > If you meant one instance for each distinct value of val, then a factory might > work: Yes that's what I mean. > cdef class _Unique(object): > cdef object __weakref__ > ... > > def Unique(val): > # do your usual WeakValueDictionary handling here. > return the_right_instance > >> And if this requires monkey-patching the tp_new slot of the class, >> perhaps some pointers as to what should the replacement method should >> at least do in order not to screw the rest of the system. > > I'd avoid that for now, unless you really require subclassing support etc. I don't need subclassing right now so I'll probably go with the factory approach. If I have some free time, I'll try to do a patch against Cython implementing more or less what I am searching for. Thanks for your answers. Arnaud > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Mon Aug 4 19:46:11 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 Aug 2008 14:46:11 -0300 Subject: [Cython] cython-devel: array declaration problems with current trunk Message-ID: Current cython-devel trunk broke my code, for example, try to run cython on this snippet: cdef extern from *: cdef void foo(int[]) ctypedef int MyInt cdef void foo(MyInt[]) struct MyStruct: pass cdef void bar(MyStruct[]) ctypedef MyStruct* MyStructP cdef void baz(MyStructP[]) I get error messages like (just the first shown): Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef extern from *: cdef void foo(int[]) ctypedef int MyInt cdef void foo(MyInt[]) ^ ------------------------------------------------------------ /u/dalcinl/Devel/Cython/sandbox/arrdecl.pyx:6:23: Buffer types only allowed as function local variables -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Aug 4 19:53:09 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 Aug 2008 14:53:09 -0300 Subject: [Cython] couple gcc warnings problems In-Reply-To: References: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> Message-ID: Look what SWIG does: /* attribute recognised by some compilers to avoid 'unused' warnings */ #ifndef SWIGUNUSED # if defined(__GNUC__) # if !(defined(__cplusplus)) || (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 4)) # define SWIGUNUSED __attribute__ ((__unused__)) # else # define SWIGUNUSED # endif # elif defined(__ICC) # define SWIGUNUSED __attribute__ ((__unused__)) # else # define SWIGUNUSED # endif #endif #ifndef SWIGUNUSEDPARM # ifdef __cplusplus # define SWIGUNUSEDPARM(p) # else # define SWIGUNUSEDPARM(p) p SWIGUNUSED # endif #endif Then, this approach can be used for unused arguments as well as unused functions. Currently, I'm getting warnings about PyObject_GetBuffer() and PyObject_ReleaseBuffer() being defined and not used (though perhaps this should be handled using the 'utilitity_code' way) On Sat, Aug 2, 2008 at 3:41 PM, Robert Bradshaw wrote: > On Aug 2, 2008, at 10:40 AM, Ondrej Certik wrote: > >> Hi, >> >> $ cython -v >> Cython version 0.9.8 >> $ cython --convert-range step.pyx >> $ gcc -O3 -W -Wall -I/usr/include/python2.5/ -I/usr/include/numpy/ -c >> -o step.o step.c >> step.c: In function '__pyx_pf_4step_temperature': >> step.c:608: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_advance': >> step.c:739: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_assign_speeds': >> step.c:1068: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_fit_in_the_box': >> step.c:1336: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_get_f': >> step.c:1675: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_get_f2': >> step.c:1794: warning: unused parameter '__pyx_self' >> step.c: In function '__pyx_pf_4step_calculate_potential': >> step.c:1941: warning: unused parameter '__pyx_self' >> step.c: At top level: >> step.c:2052: warning: missing initializer >> step.c:2052: warning: (near initialization for >> '__pyx_string_tab[11].is_identifier') >> /usr/include/numpy/__multiarray_api.h:959: warning: '_import_array' >> defined but not used >> >> >> >> >> Now you can discover couple problems above. The last warning: >> >> step.c:2052: warning: missing initializer >> step.c:2052: warning: (near initialization for >> '__pyx_string_tab[11].is_identifier') >> >> could be fixed by this patch to the step.c file: >> >> $ quilt diff >> Index: MD-spheres/step.c >> =================================================================== >> --- MD-spheres.orig/step.c 2008-08-02 19:36:34.680765668 +0200 >> +++ MD-spheres/step.c 2008-08-02 19:36:50.085854557 +0200 >> @@ -2049,7 +2049,7 @@ >> {&__pyx_kp_append, __pyx_k_append, sizeof(__pyx_k_append), 1, 1, >> 1}, >> {&__pyx_kp_gauss, __pyx_k_gauss, sizeof(__pyx_k_gauss), 1, 1, 1}, >> {&__pyx_kp_get_f, __pyx_k_get_f, sizeof(__pyx_k_get_f), 0, 1, 1}, >> - {0, 0, 0, 0, 0} >> + {0, 0, 0, 0, 0, 0} >> }; >> >> static struct PyMethodDef __pyx_methods[] = { >> >> >> >> So this seems like a bug in Cython. As to the other warnings: >> >> step.c:608: warning: unused parameter '__pyx_self' >> >> If you look into the generated file: >> >> static PyObject *__pyx_pf_4step_temperature(PyObject *__pyx_self, >> PyObject *__pyx_args, PyObject *__pyx_kwds) { >> >> >> Why does Cython generate the "self" argument? This is just a regular >> function, not a method. And why isn't the first argument used? Is this >> a bug? > > This is not a bug, it is because Python functions and methods have > exactly the same signatures to be able to treat them interchangeably > as first class objects. I believe in this case the self argument is > the actual module, though it is clearly unneeded. If you look at the > built in modules they do the exact same thing. > > Is there anything we could do to suppress this warning (fool the > compiler into thinking it's used, but still have it be optimized away)? > > - Robert > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Aug 4 20:01:51 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 20:01:51 +0200 Subject: [Cython] couple gcc warnings problems In-Reply-To: References: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> Message-ID: <4897440F.30407@student.matnat.uio.no> Lisandro Dalcin wrote: > Look what SWIG does: > /* attribute recognised by some compilers to avoid 'unused' warnings */ > #ifndef SWIGUNUSED > # if defined(__GNUC__) > # if !(defined(__cplusplus)) || (__GNUC__ > 3 || (__GNUC__ == 3 && > __GNUC_MINOR__ >= 4)) > # define SWIGUNUSED __attribute__ ((__unused__)) > # else > # define SWIGUNUSED > # endif > # elif defined(__ICC) > # define SWIGUNUSED __attribute__ ((__unused__)) > # else > # define SWIGUNUSED > # endif > #endif > > #ifndef SWIGUNUSEDPARM > # ifdef __cplusplus > # define SWIGUNUSEDPARM(p) > # else > # define SWIGUNUSEDPARM(p) p SWIGUNUSED > # endif > #endif > > Then, this approach can be used for unused arguments as well as unused > functions. Currently, I'm getting warnings about PyObject_GetBuffer() > and PyObject_ReleaseBuffer() being defined and not used (though > perhaps this should be handled using the 'utilitity_code' way) Are you using the latest cython-devel? This should have been fixed now. (The other one is a real problem which I will fix, thanks for the test case and report.) -- Dag Sverre From robertwb at math.washington.edu Mon Aug 4 20:01:27 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 11:01:27 -0700 (PDT) Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: References: Message-ID: On Mon, 4 Aug 2008, Lisandro Dalcin wrote: > Current cython-devel trunk broke my code, for example, try to run > cython on this snippet: > > cdef extern from *: > > cdef void foo(int[]) > > ctypedef int MyInt > cdef void foo(MyInt[]) > > struct MyStruct: > pass > cdef void bar(MyStruct[]) > > ctypedef MyStruct* MyStructP > cdef void baz(MyStructP[]) > > > I get error messages like (just the first shown): > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef extern from *: > > cdef void foo(int[]) > > ctypedef int MyInt > cdef void foo(MyInt[]) > ^ > ------------------------------------------------------------ > > /u/dalcinl/Devel/Cython/sandbox/arrdecl.pyx:6:23: Buffer types only > allowed as function local variables > Sorry, I didn't test any code like that. I can guess where this got introduced. Dag, any comments? - Robert From dagss at student.matnat.uio.no Mon Aug 4 20:14:40 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 20:14:40 +0200 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: References: Message-ID: <48974710.4060906@student.matnat.uio.no> Robert Bradshaw wrote: > On Mon, 4 Aug 2008, Lisandro Dalcin wrote: > >> Current cython-devel trunk broke my code, for example, try to run >> cython on this snippet: >> >> cdef extern from *: >> >> cdef void foo(int[]) >> >> ctypedef int MyInt >> cdef void foo(MyInt[]) >> >> struct MyStruct: >> pass >> cdef void bar(MyStruct[]) >> >> ctypedef MyStruct* MyStructP >> cdef void baz(MyStructP[]) >> >> >> I get error messages like (just the first shown): >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> cdef extern from *: >> >> cdef void foo(int[]) >> >> ctypedef int MyInt >> cdef void foo(MyInt[]) >> ^ >> ------------------------------------------------------------ >> >> /u/dalcinl/Devel/Cython/sandbox/arrdecl.pyx:6:23: Buffer types only >> allowed as function local variables >> > > Sorry, I didn't test any code like that. I can guess where this got > introduced. Dag, any comments? You guessed right :-) I fixed this earlier, but only for native types (note that the int[] works ok); which obviously came from a wrong assumption. I'll find a way to fix it, check in the test case etc. etc. However, there is of course the deeper question of overloading syntax like this. I don't think the [] syntax buys anything above a regular pointer notation? If so, removing support for int[]-notation altogether would get rid of the issue and lead to a cleaner grammar. What do people think? (Three choices: Good, bad, good but backwards compatability overrides it.) (Anyway that can only happen after a round of deprecation warnings etc. so I'll definitely fix it.) -- Dag Sverre From dalcinl at gmail.com Mon Aug 4 20:17:09 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 Aug 2008 15:17:09 -0300 Subject: [Cython] couple gcc warnings problems In-Reply-To: <4897440F.30407@student.matnat.uio.no> References: <85b5c3130808021040g46e0c1cbn54431fb12aba9071@mail.gmail.com> <4897440F.30407@student.matnat.uio.no> Message-ID: On Mon, Aug 4, 2008 at 3:01 PM, Dag Sverre Seljebotn wrote: > > Are you using the latest cython-devel? This should have been fixed now. > No, Sorry! It seems this is fixed. You know, I had no chance to test my code agaist the current trunk, because of the other issue. > > (The other one is a real problem which I will fix, thanks for the test > case and report.) > OK. I'll wait for your fix. Keep me informed. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Aug 4 20:46:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 Aug 2008 20:46:04 +0200 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: References: Message-ID: <48974E6C.8010209@student.matnat.uio.no> Lisandro Dalcin wrote: > Current cython-devel trunk broke my code, for example, try to run > cython on this snippet: > > cdef extern from *: > > cdef void foo(int[]) > > ctypedef int MyInt > cdef void foo(MyInt[]) > > struct MyStruct: > pass > cdef void bar(MyStruct[]) > > ctypedef MyStruct* MyStructP > cdef void baz(MyStructP[]) > > > I get error messages like (just the first shown): > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef extern from *: > > cdef void foo(int[]) > > ctypedef int MyInt > cdef void foo(MyInt[]) > ^ > ------------------------------------------------------------ > > /u/dalcinl/Devel/Cython/sandbox/arrdecl.pyx:6:23: Buffer types only > allowed as function local variables > OK this is fixed now. Apologies. (The fix rules out empty [] for buffer access, but this fits in with the latest buffer discussion of "auto-acquire if enough defaults are provided".). -- Dag Sverre From robertwb at math.washington.edu Mon Aug 4 23:12:19 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 14:12:19 -0700 (PDT) Subject: [Cython] Make positive int literals unsigned long? In-Reply-To: <4897143E.3020304@student.matnat.uio.no> References: <4897091B.1000604@student.matnat.uio.no> <48674.213.61.181.86.1217860131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4897143E.3020304@student.matnat.uio.no> Message-ID: On Mon, 4 Aug 2008, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> If we make positive int literals unsigned long rather than plain long, >>> constant-index buffer lookups are going to be optimized for free. Is >>> there going to be any ill side-effects from doing this? >>> >>> (If not, I can always special-case int literals used for buffer lookups.) >> >> What about enabling this, adding a test module dedicated to integer >> literals, casting and arithmetic, and running the (complete) test suite >> with >> >> -Wall -pedantic >> >> ? > > Sure, but that would have been a lot of work only to get a reply here > afterwards that "this whole idea would obviously not work because X" > from people who know the code base better :-) I think in general that is a good idea, but the problem is that PyLong_FromUnsignedLong is slower than PyInt_FromLong and produces python longs rather than python ints. One would need a special type (or flag) to indicate that the top bit can be assumed to be 0. - Robert From dagss at student.matnat.uio.no Tue Aug 5 00:20:06 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 00:20:06 +0200 Subject: [Cython] Make positive int literals unsigned long? In-Reply-To: References: <4897091B.1000604@student.matnat.uio.no> <48674.213.61.181.86.1217860131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4897143E.3020304@student.matnat.uio.no> Message-ID: <48978096.6050208@student.matnat.uio.no> Robert Bradshaw wrote: > On Mon, 4 Aug 2008, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> If we make positive int literals unsigned long rather than plain long, >>>> constant-index buffer lookups are going to be optimized for free. Is >>>> there going to be any ill side-effects from doing this? >>>> >>>> (If not, I can always special-case int literals used for buffer lookups.) >>> What about enabling this, adding a test module dedicated to integer >>> literals, casting and arithmetic, and running the (complete) test suite >>> with >>> >>> -Wall -pedantic >>> >>> ? >> Sure, but that would have been a lot of work only to get a reply here >> afterwards that "this whole idea would obviously not work because X" >> from people who know the code base better :-) > > I think in general that is a good idea, but the problem is that > PyLong_FromUnsignedLong is slower than PyInt_FromLong and produces python > longs rather than python ints. One would need a special type (or flag) to > indicate that the top bit can be assumed to be 0. And there's my X. Do you see any weaknesses in this? (as a general, somewhat orthogonal runtime optimization ... the hit should be negligible compared to PyInt_FromUnsignedLong, or...?): static INLINE PyObject* __Pyx_ObjectFromUnsignedLong(unsigned long value) { if (top bit of value is 0) return PyInt_FromLong((long)value); else return PyLong_FromUnsignedLong(value); } -- Dag Sverre From robertwb at math.washington.edu Tue Aug 5 01:07:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 16:07:07 -0700 (PDT) Subject: [Cython] Make positive int literals unsigned long? In-Reply-To: <48978096.6050208@student.matnat.uio.no> References: <4897091B.1000604@student.matnat.uio.no> <48674.213.61.181.86.1217860131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4897143E.3020304@student.matnat.uio.no> <48978096.6050208@student.matnat.uio.no> Message-ID: On Tue, 5 Aug 2008, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Mon, 4 Aug 2008, Dag Sverre Seljebotn wrote: >> >>> Stefan Behnel wrote: >>>> Dag Sverre Seljebotn wrote: >>>>> If we make positive int literals unsigned long rather than plain long, >>>>> constant-index buffer lookups are going to be optimized for free. Is >>>>> there going to be any ill side-effects from doing this? >>>>> >>>>> (If not, I can always special-case int literals used for buffer lookups.) >>>> What about enabling this, adding a test module dedicated to integer >>>> literals, casting and arithmetic, and running the (complete) test suite >>>> with >>>> >>>> -Wall -pedantic >>>> >>>> ? >>> Sure, but that would have been a lot of work only to get a reply here >>> afterwards that "this whole idea would obviously not work because X" >>> from people who know the code base better :-) >> >> I think in general that is a good idea, but the problem is that >> PyLong_FromUnsignedLong is slower than PyInt_FromLong and produces python >> longs rather than python ints. One would need a special type (or flag) to >> indicate that the top bit can be assumed to be 0. > > And there's my X. > > Do you see any weaknesses in this? (as a general, somewhat orthogonal > runtime optimization ... the hit should be negligible compared to > PyInt_FromUnsignedLong, or...?): > > static INLINE PyObject* __Pyx_ObjectFromUnsignedLong(unsigned long value) { > if (top bit of value is 0) return PyInt_FromLong((long)value); > else return PyLong_FromUnsignedLong(value); > } > It would bloat the code a bit every time a literal integer is used as a Python object. I'm not sure if that's a big deal though. Another approach would be for the indexing code to be able to have some logic that detects the index is positive (which could be extended to apply to loop variables once control flow in in place for example), or just to special case this one (common) example. - Robert From greg.ewing at canterbury.ac.nz Tue Aug 5 03:18:02 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 05 Aug 2008 13:18:02 +1200 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: <48974710.4060906@student.matnat.uio.no> References: <48974710.4060906@student.matnat.uio.no> Message-ID: <4897AA4A.3090608@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > I don't think the [] syntax buys anything above a regular > pointer notation? If so, removing support for int[]-notation altogether > would get rid of the issue and lead to a cleaner grammar. The current declaration syntax has the advantage that it's the same as in C, so people familiar with C don't have to learn anything new. This would be a step away from that, although a fairly minor one. Another consideration is how much existing code it would break, which could be quite a lot. -- Greg From robertwb at math.washington.edu Tue Aug 5 03:32:43 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 18:32:43 -0700 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: <4897AA4A.3090608@canterbury.ac.nz> References: <48974710.4060906@student.matnat.uio.no> <4897AA4A.3090608@canterbury.ac.nz> Message-ID: <7AE03A9D-989B-4D97-8D81-0309BE4696EB@math.washington.edu> On Aug 4, 2008, at 6:18 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> I don't think the [] syntax buys anything above a regular >> pointer notation? If so, removing support for int[]-notation >> altogether >> would get rid of the issue and lead to a cleaner grammar. > > The current declaration syntax has the advantage that it's > the same as in C, so people familiar with C don't have to > learn anything new. This would be a step away from that, > although a fairly minor one. > > Another consideration is how much existing code it would > break, which could be quite a lot. +1 Also, c_type[] vs. object_type[] are easily discerned, and they both have similar (high level) meaning. - Robert From greg.ewing at canterbury.ac.nz Tue Aug 5 03:27:38 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 05 Aug 2008 13:27:38 +1200 Subject: [Cython] Make positive int literals unsigned long? In-Reply-To: References: <4897091B.1000604@student.matnat.uio.no> <48674.213.61.181.86.1217860131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4897143E.3020304@student.matnat.uio.no> Message-ID: <4897AC8A.7050704@canterbury.ac.nz> Robert Bradshaw wrote: > I think in general that is a good idea, but the problem is that > PyLong_FromUnsignedLong is slower than PyInt_FromLong and produces python > longs rather than python ints. One would need a special type (or flag) to > indicate that the top bit can be assumed to be 0. There could be a utility function that creates either an int or a long depending on the value passed to it. It might also be possible to choose between them at compile time in the case of literals, although it would be tricky because it would depend on sizeof(int) on the target platform. Some kind of macro might be able to do it. -- Greg From robertwb at math.washington.edu Tue Aug 5 04:25:25 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 19:25:25 -0700 Subject: [Cython] Refactor or surgery on global constants? In-Reply-To: <48960702.3060507@student.matnat.uio.no> References: <48920CCC.4070301@student.matnat.uio.no> <4892C7C5.1040806@behnel.de> <48960702.3060507@student.matnat.uio.no> Message-ID: On Aug 3, 2008, at 12:29 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> Dag Sverre Seljebotn wrote: >>> In making it possible to have runnable code in pxds, I need to >>> make a >>> choice. The problem is that interned strings, objects etc. live in >>> seperate scopes and you get double copies, basically now I get the >>> following code twice in my c-file: >>> >>> static char __pyx_k___getbuffer__[] = "__getbuffer__"; >>> static PyObject *__pyx_kp___getbuffer__; >>> >>> which gcc isn't too thrilled about. >>> [...] >>> 2) Move these things from the scopes to code.global (a global >>> context of >>> the CCodeWriter). This fits nicely with utility_code going over >>> as well. >>> This means that for now each scope simply pipe their things into >>> code >>> where they are merged, while in time then e.g. StringNode could >>> intern a >>> string during code generation rather than bothering with it during >>> analysis. (I think this is a viable way forward, but may take an >>> hour or >>> three longer for me to do.) >> >> Since I previously worked a bit on the string interning code, I >> think it would >> really benefit from such a change. Please go ahead. > > 1) OK I've made some changes in this direction: > > - BlockNode now forwards the entries it deals with to > Code.GlobalState, > which tries to do the same thing but filters out duplicates (which > comes > from pxds and pyxes doing the same thing). In time, I expect all of > BlockNode to vanish. > > - Also, the code dealing with interned strings/ints and cached > builtins > from ModuleNode is moved to Code.GlobalState; which uses a different > strategy: Rather than iterating over the contents of a scope, it > simply > allocates many CodeWriter instances/buffers and outputs > initialiazation > code at the same time as the declaration code, though to a seperate > buffer. (One can like or not like this I suppose -- only argument I > have > is it tends to keep code more "together" in one location, but the real > reason is that it was easier this way.) > > - Because one cannot have insertion points within functions (would > create h). Therefore I extracted some things from the module > initialization function into two more functions, __Pyx_InitGlobals and > __Pyx_InitCachedBuiltins. I could potentially get rid of this with a > special "I promise not to use temps or modify labels"-insertion-point, > but this was easier. (Also, one could argue for it from a design > viewpoint: c-file globals are more tied to the c-file as such than > necesarrily the module -- if a single C file were to implement both a > module and some of its submodules [1], then cached builtins could be > shared but the module init func would not be!) > > This all runs the test cases etc. and I consider it "done" in some > sense, but I want to have some feedback before I consider it ready for > merging. Thanks. Haven't actually looked at the code yet, but it all sounds good. > Of course, it represents a transitional solution. In the end, I expect > one would add something like this: > > self.result_code = code.globalcode.py_string_literal("abc") > > and so on (details must be discussed when the day comes), but that, as > my new mantra has become, "is pending a result_code refactoring". But > the design now allows for creating such a thing, it would all happen > within Code.py. (I'll get to that some day of course, especially as > Greg > has a lot of it done, but I have my GSoC priorities to worry about). Yep. Perfect is the enemy of the good. > [1] (Is this possible in Python? What I've done now opens up for > Cython > supporting this (think compiling a dir tree to a single .so) and get > some degree of sharing constant literals etc., but that's hardly a > priority.) > > 2) During writing this, I realized that some code could be written > in a > shorter fashion: > > (self.initwriter > .putln("return 0;") > .put_label(self.initwriter.error_label) > .putln("return -1;") > .putln("}") > .exit_cfunc_scope() > ) > > ...where I've changed some functions to return "self" (much like C++ > iostreams). Do you like or dislike this? Consider both with and > without > the indendation-matching-the-C-code. It does create some dissymetry > (see the put_label where you need a new reference to the writer), this > could be fixed by adding put_error_label, or by going back > > The counterargument is that I could say "c = self.initwriter" first > instead to get almost the same brevity. > > Of course, this is only an added option for new code. So inconsistency > would be added. Still I'm +1/2 because of when doing all the C code > I've > written in Buffer.py I always thought that it tended to become quite > long-winded to express what I wanted... > > (Some code is using this in my branch which I'll redo if the vote > is not > positive.) Honestly, I find that harder to read than the explicit way, and would rather stick with the "explicit is better than implicit" mantra on this one. > 3) For buffers, this means that I've manged to strip all NumPy > specific > code from Buffer.py, instead you currently can define __getbuffer__ > and > __releasebuffer__ in numpy.pxd Nice. > I have not introduced a generic "final" mechanism or similar; > rather an > exception is introduced specifically for __getbuffer__ and > __releasebuffer__ and specifically for the purpose of > Py2.5-GetBuffer-faking. I'd like to leave it like this for now, but it > will be a "hidden" feature and I will warn people not to rely on it. > > In time I see it like this: A "final" language feature is added, and > anything declared "final" can reside in pxds. But that is not done > now, > hence the special cases for these two functions. Yep. I would like to get a general sense of what people think of adding a "final" keyword to the language (for cdef functions only(?)). - Robert From robertwb at math.washington.edu Tue Aug 5 04:34:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 19:34:42 -0700 Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <48960A8D.1090607@student.matnat.uio.no> References: <48960A8D.1090607@student.matnat.uio.no> Message-ID: On Aug 3, 2008, at 12:44 PM, Dag Sverre Seljebotn wrote: > In numpy.pxd I want to have this: > > ctypedef npy_int64 int64 > > and be able to use it like this (this is what a NumPy user would > expect > to do): > > cimport numpy > cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], > numpy.int64) Yes. > This however creates an error: > 'int64' is not a constant, variable or function identifier > > since int64 is declared as a type in the scope. Where is this error being thrown from? Perhaps it could/should be checked after the buffer transforms? > I've had rather complicated and grand schemes for this earlier, but > what > I'll propose this time is much simpler: When not in a "type context" > (what you are writing cannot possibly be a C type), C types vanish > from > the scope/are not considered .. so that the ctypedef numpy.int64 > does not > > Warning! In theory, this principle could allow code like > > x = 4 > ctypedef int x > cdef x v = x I'm not completely following that, but it looks scary. > If you didn't like this but have time, you could have a look at > > http://wiki.cython.org/enhancements/runtimectypes > > which is another proposal I could use to achieve my ends, however I'm > reluctant to think or discuss something going so much into language > design right now. I'm not sure I like that proposal either. - Robert From robertwb at math.washington.edu Tue Aug 5 04:36:39 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 19:36:39 -0700 Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <489663FA.7010307@canterbury.ac.nz> References: <48960A8D.1090607@student.matnat.uio.no> <489663FA.7010307@canterbury.ac.nz> Message-ID: On Aug 3, 2008, at 7:05 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], >> numpy.int64) >> >> This however creates an error: >> 'int64' is not a constant, variable or function identifier > > You might like to look into how extension types are > made to behave as both types and runtime values -- > this sounds like it might be something similar. Yep, there could be a (dynamically created?) runtime object to represent those types if used in an object context, and would be much simper than the runtimectypes proposal. This may prevent catching some errors though. - Robert From robertwb at math.washington.edu Tue Aug 5 04:40:27 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 19:40:27 -0700 Subject: [Cython] Scope of Py_buffer In-Reply-To: <48971436.3080102@student.matnat.uio.no> References: <4896DDF9.80705@student.matnat.uio.no> <33785.213.61.181.86.1217859240.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48971436.3080102@student.matnat.uio.no> Message-ID: <6C5B7ED7-DDEE-450B-AED9-BDADCF70B4C5@math.washington.edu> On Aug 4, 2008, at 7:37 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi Dag, >> >> Dag Sverre Seljebotn wrote: >>> I'm wondering if I could move Py_buffer out of the default scope, in >>> order to reduce the namespace incompatability. (Py_buffer is >>> present in >>> the global scope for the purposes of the second argument to >>> __getbuffer__ and __releasebuffer__ ). The alternatives I see are >>> either >>> "cython.Py_buffer" or "python_buffer.Py_buffer" (or both), where you >>> need to cimport cython/python_buffer first. >> >> I see Py_buffer in a similar light as Py_ssize_t. Would you also >> argue for >> that being moved into a separate namespace? > > I would, but that is water under the bridge now -- it would call for a > round of deprecation warnings etc., so might as well leave it until > Python compatability is otherwise much higher (though if Py_buffer is > moved, one should definitly create cython.Py_ssize_t as well now as an > alias). > > The point is that Py_buffer is probably not used in any code yet, > or if > it is, it is only for Py3 beta purposes, so a deprecation warning > round > is not needed IMO. > >> Since Py_buffer is a requirement for implementing a standard >> Python C-API >> method, I think it makes sense to keep it in the global namespace >> as a >> standard data type of Cython. > > Yes, and I mentioned this counter-argument too. For me it weighs less > than a higher degree of Python compatability. I don't mind letting > __builtin__ be strictly Python API, and require imports for using the > C-API in any way (or anything else that we add above pure Python). > > Anyway, the only reason I'm bringing it up now is the timing, and > seeing > that you didn't like it I'm ready to drop it. I agree right now is the time to bring this up, but I'm for leaving it in the global namespace as well. > (While on namespace issues, the Tao says "flat is better than nested" > which is in favor of cython.shape rather than cython.buffer.shape?) Because "shape" is generic enough could very well have other meanings that are completely unrelated to buffers, and I'd rather have cython.buffer.shape than cython.buffer_shape. - Robert From stefan_ml at behnel.de Tue Aug 5 07:34:00 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Aug 2008 07:34:00 +0200 Subject: [Cython] Accessing the internals of a Python long In-Reply-To: <99e0b9530808021953j5af998a1m5e431ed1918de798@mail.gmail.com> References: <99e0b9530808021953j5af998a1m5e431ed1918de798@mail.gmail.com> Message-ID: <4897E648.9070803@behnel.de> Hi, Case Vanhorsen wrote: > I am trying to access the internal structure of a Python long. The code > fragment below works when I use Pyrex 0.9.8.4 but generates a compile error > when using Cython 0.9.8. > [...] > The error: > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > int ob_size > unsigned int * ob_digit > def test(temp = long(0)): > cdef _longobject *l > l = <_longobject *> temp > print sizeof(l.ob_size) > ^ > ------------------------------------------------------------ > > /home/case/code/cvh.pyx:16:18: 'cvh.test' is not a cimported module FWIW, I get a different error for the current head: Error converting Pyrex file to C: ------------------------------------------------------------ ... unsigned int * ob_digit def test(temp = long(0)): cdef _longobject *l l = <_longobject *> temp print sizeof(l.ob_size) print sizeof(l.ob_digit[0]) ^ ------------------------------------------------------------ /.../pylong.pyx:17:28: Expected: type Looks like a bug to me. Stefan From dagss at student.matnat.uio.no Tue Aug 5 08:36:37 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 Aug 2008 08:36:37 +0200 (CEST) Subject: [Cython] Type scope vs. runtime scope In-Reply-To: References: <48960A8D.1090607@student.matnat.uio.no> Message-ID: <63592.193.157.229.67.1217918197.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Aug 3, 2008, at 12:44 PM, Dag Sverre Seljebotn wrote: > >> In numpy.pxd I want to have this: >> >> ctypedef npy_int64 int64 >> >> and be able to use it like this (this is what a NumPy user would >> expect >> to do): >> >> cimport numpy >> cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], >> numpy.int64) > > Yes. > >> This however creates an error: >> 'int64' is not a constant, variable or function identifier >> >> since int64 is declared as a type in the scope. > > Where is this error being thrown from? Perhaps it could/should be > checked after the buffer transforms? The problem is that once numpy.int64 is declared as a ctypedef, it is not longer available to be called at runtime as a python object, so you cannot get the reference to pass to numpy.zeros. To be clear, in numpy.pxd it says cdef extern ... : ctypedef int npy_int64 ctypedef npy_int64 int64 while some consequence of numpy/__init__.py deep down somewhere is assigning something to the int64 attribute of the numpy module. (This has nothing to do with buffers.) One solution is requiring people to do "cimport c_numpy", but I'd like to avoid that as well. Unless this is resolved (on a language design level, technically it as too many solutions...), I need to figure out a way to mangle the names without loosing usability too much... >> Warning! In theory, this principle could allow code like >> >> x = 4 >> ctypedef int x >> cdef x v = x > > I'm not completely following that, but it looks scary. My proposed solution is to allow that ctypedefs and other type declarations in general do not alter the "runtime scope", so that the "type named x" and "module attribute named x" are two different things and can be referenced by the same name from the same scope. >> If you didn't like this but have time, you could have a look at >> >> http://wiki.cython.org/enhancements/runtimectypes >> >> which is another proposal I could use to achieve my ends, however I'm >> reluctant to think or discuss something going so much into language >> design right now. > > > I'm not sure I like that proposal either. I'm not too happy about the syntax I proposed, where to put the declarations etc. But I still like the general concept of having something say "use this C type, but when coercing to a Python object use this class". This would be a perfect fit for the numpy types as you can currently do P> a = numpy.float64(34) P> 34.0 P> type(a) (or something like that), so having cdef numpy.int64 x and have x coerce not to a Python int but a (runtime) numpy.int64 would be nice, even though the compile-time int64 is an entirely different beast it expresses the same information and exposes about the same API (same operators behave in the same way). Dag Sverre From robertwb at math.washington.edu Tue Aug 5 08:52:25 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 Aug 2008 23:52:25 -0700 Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <63592.193.157.229.67.1217918197.squirrel@webmail.uio.no> References: <48960A8D.1090607@student.matnat.uio.no> <63592.193.157.229.67.1217918197.squirrel@webmail.uio.no> Message-ID: <2F49098A-E3FF-42CD-B780-EDE4C3005B75@math.washington.edu> On Aug 4, 2008, at 11:36 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Aug 3, 2008, at 12:44 PM, Dag Sverre Seljebotn wrote: >> >>> In numpy.pxd I want to have this: >>> >>> ctypedef npy_int64 int64 >>> >>> and be able to use it like this (this is what a NumPy user would >>> expect >>> to do): >>> >>> cimport numpy >>> cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], >>> numpy.int64) >> >> Yes. >> >>> This however creates an error: >>> 'int64' is not a constant, variable or function identifier >>> >>> since int64 is declared as a type in the scope. >> >> Where is this error being thrown from? Perhaps it could/should be >> checked after the buffer transforms? > > The problem is that once numpy.int64 is declared as a ctypedef, it > is not > longer available to be called at runtime as a python object, so you > cannot > get the reference to pass to numpy.zeros. > > To be clear, in numpy.pxd it says > > cdef extern ... : ctypedef int npy_int64 > ctypedef npy_int64 int64 > > while some consequence of numpy/__init__.py deep down somewhere is > assigning something to the int64 attribute of the numpy module. > > (This has nothing to do with buffers.) > > One solution is requiring people to do "cimport c_numpy", but I'd > like to > avoid that as well. Unless this is resolved (on a language design > level, > technically it as too many solutions...), I need to figure out a > way to > mangle the names without loosing usability too much... OK, I completely missed the point of what you were trying to do last time (I blame bad linewrapping in the numpy.zeros function ;-). What if, when you wrote ctypedef npy_int64 int64 it would do a check to see if int64 was already defined (as a normal variable) and if so would attach that information to the type (same as with extension types that have a dual nature). When used in a variable context it would extract this information. - Robert From dagss at student.matnat.uio.no Tue Aug 5 08:56:57 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 Aug 2008 08:56:57 +0200 (CEST) Subject: [Cython] cython namespace In-Reply-To: <6C5B7ED7-DDEE-450B-AED9-BDADCF70B4C5@math.washington.edu> References: <4896DDF9.80705@student.matnat.uio.no> <33785.213.61.181.86.1217859240.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48971436.3080102@student.matnat.uio.no> <6C5B7ED7-DDEE-450B-AED9-BDADCF70B4C5@math.washington.edu> Message-ID: <63601.193.157.229.67.1217919417.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Aug 4, 2008, at 7:37 AM, Dag Sverre Seljebotn wrote: >> (While on namespace issues, the Tao says "flat is better than nested" >> which is in favor of cython.shape rather than cython.buffer.shape?) > > Because "shape" is generic enough could very well have other meanings > that are completely unrelated to buffers, and I'd rather have > cython.buffer.shape than cython.buffer_shape. (I hit the result_code wall with these functions, so this discussion lost its importance for some days at least. But for the record:) Python doesn't have "list_len", "tuple_len" etc. Even if shape should get more meanings (which I doubt), cython.shape is likely to mean "get the shape of my argument", so you differ the meanings by the type (just like "len" does). We can't add slots, but instead we have compile-time information. The point is getting a natural pattern people can use for looping and which they don't have to think too much about -- I'd like to "hide" buffers from the average NumPy user. Currently there's nothing requiring a NumPy user to know about the buffer PEP (it "just works"), this import would be the only place the word "buffer" shows up and it kind of makes it "less integrated" and less Pythonic (kind of like requiring you to call list_len). (Thought experiment: In line with this, one could have shape try to return the right thing for regularily-sized nested Python lists as well. Not that it is worth doing, but if one wanted to, it would be natural to have that functionality below the same shape name in the namespace.) - In the case of buffer_pointer, it would be confusing to have just "pointer" anyway, so you would need cython.buffer.buffer_pointer. - ndim seems to be like shape. - strides and suboffsets are quite buffer-specific though. I'm still for "flat better than nested", but failing that, I'd propose that everything is in cython.buffer but shape is also in cython. Dag Sverre From dagss at student.matnat.uio.no Tue Aug 5 09:06:46 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 Aug 2008 09:06:46 +0200 (CEST) Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <2F49098A-E3FF-42CD-B780-EDE4C3005B75@math.washington.edu> References: <48960A8D.1090607@student.matnat.uio.no> <63592.193.157.229.67.1217918197.squirrel@webmail.uio.no> <2F49098A-E3FF-42CD-B780-EDE4C3005B75@math.washington.edu> Message-ID: <63605.193.157.229.67.1217920006.squirrel@webmail.uio.no> Robert Bradshaw wrote: > > On Aug 4, 2008, at 11:36 PM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On Aug 3, 2008, at 12:44 PM, Dag Sverre Seljebotn wrote: >>> >>>> In numpy.pxd I want to have this: >>>> >>>> ctypedef npy_int64 int64 >>>> >>>> and be able to use it like this (this is what a NumPy user would >>>> expect >>>> to do): >>>> >>>> cimport numpy >>>> cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], >>>> numpy.int64) >>> >>> Yes. >>> >>>> This however creates an error: >>>> 'int64' is not a constant, variable or function identifier >>>> >>>> since int64 is declared as a type in the scope. >>> >>> Where is this error being thrown from? Perhaps it could/should be >>> checked after the buffer transforms? >> >> The problem is that once numpy.int64 is declared as a ctypedef, it >> is not >> longer available to be called at runtime as a python object, so you >> cannot >> get the reference to pass to numpy.zeros. >> >> To be clear, in numpy.pxd it says >> >> cdef extern ... : ctypedef int npy_int64 >> ctypedef npy_int64 int64 >> >> while some consequence of numpy/__init__.py deep down somewhere is >> assigning something to the int64 attribute of the numpy module. >> >> (This has nothing to do with buffers.) >> >> One solution is requiring people to do "cimport c_numpy", but I'd >> like to >> avoid that as well. Unless this is resolved (on a language design >> level, >> technically it as too many solutions...), I need to figure out a >> way to >> mangle the names without loosing usability too much... > > OK, I completely missed the point of what you were trying to do last > time (I blame bad linewrapping in the numpy.zeros function ;-). > > What if, when you wrote > > ctypedef npy_int64 int64 > > it would do a check to see if int64 was already defined (as a normal > variable) and if so would attach that information to the type (same > as with extension types that have a dual nature). When used in a > variable context it would extract this information 1) But you do not know if int64 is already defined until run-time? 2) How would this act differently from what I proposed? (Well, it only acts on typedefs...) What I am asking here is not how to implement it (I'll have a look at the source before asking that), but how the language should behave. Is this wanted behaviour, or is ctypedef int x x = 3 cdef x v = x too scary? (Which seems to be a consequence of your proposal as well, if it is not the same as mine.) Dag Sverre From dagss at student.matnat.uio.no Tue Aug 5 09:15:35 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 Aug 2008 09:15:35 +0200 (CEST) Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <63605.193.157.229.67.1217920006.squirrel@webmail.uio.no> References: <48960A8D.1090607@student.matnat.uio.no> <63592.193.157.229.67.1217918197.squirrel@webmail.uio.no> <2F49098A-E3FF-42CD-B780-EDE4C3005B75@math.washington.edu> <63605.193.157.229.67.1217920006.squirrel@webmail.uio.no> Message-ID: <63621.193.157.229.67.1217920535.squirrel@webmail.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> >> On Aug 4, 2008, at 11:36 PM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> On Aug 3, 2008, at 12:44 PM, Dag Sverre Seljebotn wrote: >>>> >>>>> In numpy.pxd I want to have this: >>>>> >>>>> ctypedef npy_int64 int64 >>>>> >>>>> and be able to use it like this (this is what a NumPy user would >>>>> expect >>>>> to do): >>>>> >>>>> cimport numpy >>>>> cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], >>>>> numpy.int64) >>>> >>>> Yes. >>>> >>>>> This however creates an error: >>>>> 'int64' is not a constant, variable or function identifier >>>>> >>>>> since int64 is declared as a type in the scope. >>>> >>>> Where is this error being thrown from? Perhaps it could/should be >>>> checked after the buffer transforms? >>> >>> The problem is that once numpy.int64 is declared as a ctypedef, it >>> is not >>> longer available to be called at runtime as a python object, so you >>> cannot >>> get the reference to pass to numpy.zeros. >>> >>> To be clear, in numpy.pxd it says >>> >>> cdef extern ... : ctypedef int npy_int64 >>> ctypedef npy_int64 int64 >>> >>> while some consequence of numpy/__init__.py deep down somewhere is >>> assigning something to the int64 attribute of the numpy module. >>> >>> (This has nothing to do with buffers.) >>> >>> One solution is requiring people to do "cimport c_numpy", but I'd >>> like to >>> avoid that as well. Unless this is resolved (on a language design >>> level, >>> technically it as too many solutions...), I need to figure out a >>> way to >>> mangle the names without loosing usability too much... >> >> OK, I completely missed the point of what you were trying to do last >> time (I blame bad linewrapping in the numpy.zeros function ;-). >> >> What if, when you wrote >> >> ctypedef npy_int64 int64 >> >> it would do a check to see if int64 was already defined (as a normal >> variable) and if so would attach that information to the type (same >> as with extension types that have a dual nature). When used in a >> variable context it would extract this information > > 1) But you do not know if int64 is already defined until run-time? > > 2) How would this act differently from what I proposed? (Well, it only > acts on typedefs...) > > What I am asking here is not how to implement it (I'll have a look at the > source before asking that), but how the language should behave. Is this > wanted behaviour, or is > > ctypedef int x > x = 3 > cdef x v = x > > too scary? (Which seems to be a consequence of your proposal as well, if > it is not the same as mine.) OK what about this: 1) The above is illegal, because it is all under Cython control. 2) If you have a cimport with the same name as an import, then ctypedefs of non-cdef-classes in the cimport do not affect the runtime namespace of the import. It has complexity against it though; simply allowing the example above might be more straightforward and much less magic. Users don't have to shoot themselves in the foot even if we let them. Dag Sverre From robertwb at math.washington.edu Tue Aug 5 09:16:45 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 5 Aug 2008 00:16:45 -0700 Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <63605.193.157.229.67.1217920006.squirrel@webmail.uio.no> References: <48960A8D.1090607@student.matnat.uio.no> <63592.193.157.229.67.1217918197.squirrel@webmail.uio.no> <2F49098A-E3FF-42CD-B780-EDE4C3005B75@math.washington.edu> <63605.193.157.229.67.1217920006.squirrel@webmail.uio.no> Message-ID: <64D35068-1F15-4454-ABF6-3311E67BC098@math.washington.edu> On Aug 5, 2008, at 12:06 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> >> On Aug 4, 2008, at 11:36 PM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> On Aug 3, 2008, at 12:44 PM, Dag Sverre Seljebotn wrote: >>>> >>>>> In numpy.pxd I want to have this: >>>>> >>>>> ctypedef npy_int64 int64 >>>>> >>>>> and be able to use it like this (this is what a NumPy user would >>>>> expect >>>>> to do): >>>>> >>>>> cimport numpy >>>>> cdef numpy.ndarray[numpy.int64, 2] = numpy.zeros([10, 10], >>>>> numpy.int64) >>>> >>>> Yes. >>>> >>>>> This however creates an error: >>>>> 'int64' is not a constant, variable or function identifier >>>>> >>>>> since int64 is declared as a type in the scope. >>>> >>>> Where is this error being thrown from? Perhaps it could/should be >>>> checked after the buffer transforms? >>> >>> The problem is that once numpy.int64 is declared as a ctypedef, it >>> is not >>> longer available to be called at runtime as a python object, so you >>> cannot >>> get the reference to pass to numpy.zeros. >>> >>> To be clear, in numpy.pxd it says >>> >>> cdef extern ... : ctypedef int npy_int64 >>> ctypedef npy_int64 int64 >>> >>> while some consequence of numpy/__init__.py deep down somewhere is >>> assigning something to the int64 attribute of the numpy module. >>> >>> (This has nothing to do with buffers.) >>> >>> One solution is requiring people to do "cimport c_numpy", but I'd >>> like to >>> avoid that as well. Unless this is resolved (on a language design >>> level, >>> technically it as too many solutions...), I need to figure out a >>> way to >>> mangle the names without loosing usability too much... >> >> OK, I completely missed the point of what you were trying to do last >> time (I blame bad linewrapping in the numpy.zeros function ;-). >> >> What if, when you wrote >> >> ctypedef npy_int64 int64 >> >> it would do a check to see if int64 was already defined (as a normal >> variable) and if so would attach that information to the type (same >> as with extension types that have a dual nature). When used in a >> variable context it would extract this information > > 1) But you do not know if int64 is already defined until run-time? Unless one is using the import *, one always knows. > 2) How would this act differently from what I proposed? (Well, it only > acts on typedefs...) typedefs are the only way to make new c types anyways. I guess it is essentially the same, I was just thinking of it from a different angle. > > What I am asking here is not how to implement it (I'll have a look > at the > source before asking that), but how the language should behave. Is > this > wanted behaviour, or is > > ctypedef int x > x = 3 > cdef x v = x > > too scary? (Which seems to be a consequence of your proposal as > well, if > it is not the same as mine.) Yes, I guess that is the behavior we want. We already handle int vs. int, list vs. list, CdefClass vs. CdefClass, so I guess we could do this. On that note, why doesn't it die when you put "int" in there instead? Hmm... - Robert From dagss at student.matnat.uio.no Tue Aug 5 13:51:11 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 13:51:11 +0200 Subject: [Cython] Multiple cimports to same name? Message-ID: <48983EAF.5010305@student.matnat.uio.no> While dealing with my pragma code I encountered this issue which I want to note down (I need to know how to deal with it for pragmas): cimport python_int as a a.PyInt_Check(3) cimport python_dict as a a.PyInt_Check(3) a.PyDict_New() Apparently, only the first cimport is considered, so that PyDict_New is turned into a module dict lookup and PyInt_Check is a native C call also the second time. If the order of the cimports are reversed, the opposite happen. It wouldn't take long to raise a compiler error on the second cimport as a fix for now ("name a already taken by a cimport"), but since it would break backwards compatability to some degree (of already broken code, I suppose) I wanted to check with you first. Long term I suppose having the second PyInt_Check above turn into a dictionary lookup above is the wanted behaviour, but given that we give a compiler error now then such behaviour will be backwards compatible. -- Dag Sverre From dagss at student.matnat.uio.no Tue Aug 5 14:49:57 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 14:49:57 +0200 Subject: [Cython] Done with compiler options/pragmas Message-ID: <48984C75.40101@student.matnat.uio.no> Feedback welcome on the below set of conventions -- I just picked some conventions to get going and implemented them as they are easily changeable. So feel free to comment on all choices of names, letters etc. Terminology first -- I don't really like the word pragma... "code generation options" are perhaps better? I've called them "(compiler) options" below and in the code base, while calling what is options today "arguments" (as it is command line arguments they really are). Anyway, the system is this: Options.py contains a list of compiler options to be controlled through this system. Currently this is only a boolean "boundscheck". ALL options we define can then consistently be toggled in the four ways listed below. During a transform step, ALL nodes in the parse tree has a node.options dictionary added giving all options (with defaults filled in) in effect for that exact node (this only adds a reference of course, so consider that dict immutable). IndexNode then simply checks self.options['boundscheck']. Ways to set options: a) #cython: boundscheck=True, other=False #cython: third = no before any code in the file (but whitespace is allowed, I'm using Plex/the parser). The boolean value is case-insensitive and true/false/yes/no is allowed (is this too much? The thing is, for command-line arguments "True" is rather unfriendly and I'd like it to be consistent). b) A command line argument "cython -O boundscheck=False,other=tRUe -O third=yes". This overrides the #cython comments, but NOT c) and d) listed below. The syntax of the string passed to -O is the same as the one passed to #cython, which is why I allowed yes/no in it. (For instance, ssh has options like this: ssh -o ForwardX11=no). c) A decorator works on function-scope: cimport cython @cython.boundscheck(True) def f(): ... The decorator is stripped away before the decorator transform. There is not really such symbol as cython.boundscheck in the scope, it is a magic option only. So of course you cannot do a = cython.boundscheck @a ... d) A with statement: cimport cython with cython.boundscheck(False): ... The entire with statement is then stripped out of the source prior to the with transformation. BTW, on "cimport cython": This is magic as well. I.e. it always is in front of the include path, and there will be no way to have a pxd file named "cython.pxd" imported. Is this OK? (I can change it but it would be less simple in more than one place, and people really should always have the cython scope available.) -- Dag Sverre From dalcinl at gmail.com Tue Aug 5 16:37:28 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 Aug 2008 11:37:28 -0300 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48984C75.40101@student.matnat.uio.no> References: <48984C75.40101@student.matnat.uio.no> Message-ID: On Tue, Aug 5, 2008 at 9:49 AM, Dag Sverre Seljebotn wrote: > > Terminology first -- I don't really like the word pragma... "code > generation options" are perhaps better? I've called them "(compiler) > options" below and in the code base, while calling what is options today > "arguments" (as it is command line arguments they really are). Well, I also do not like the word 'pragma'. but take into accout that is meaning should be clear for people with some C experience. > Ways to set options: > a) > #cython: boundscheck=True, other=False > #cython: third = no > > before any code in the file (but whitespace is allowed, I'm using > Plex/the parser). The boolean value is case-insensitive and > true/false/yes/no is allowed (is this too much? The thing is, for > command-line arguments "True" is rather unfriendly and I'd like it to be > consistent). It is not to much at all. I would even ask for '1' and '0' also being accepted for booleans. > b) A command line argument "cython -O boundscheck=False,other=tRUe -O > third=yes". This overrides the #cython comments, but NOT c) and d) > listed below. Well, '-O' smells to 'optimization' for me. > c) A decorator works on function-scope: > > cimport cython > @cython.boundscheck(True) > def f(): > ... > > d) A with statement: > cimport cython > with cython.boundscheck(False): > ... I really prefer the decorator form. The 'with' form could require a lot of identation if you want to set a lot for options. > BTW, on "cimport cython": This is magic as well. I.e. it always is in > front of the include path, and there will be no way to have a pxd file > named "cython.pxd" imported. Is this OK? (I can change it but it would > be less simple in more than one place, and people really should always > have the cython scope available.) If it is too magic, then perhaps 'cimport' should not be the way to access the options. What about 'cython' being a sort or 'reserved global'. Furthermore, perhaps 'cython' is not a good name. Perhaps in the near future Pyrex can take some of your ideas. So perhaps is better to find a 'neutral' name. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Aug 5 16:47:55 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 Aug 2008 11:47:55 -0300 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: <48974E6C.8010209@student.matnat.uio.no> References: <48974E6C.8010209@student.matnat.uio.no> Message-ID: Dag, I still have problems with this (the 'sizeof(int[3])' part): cdef inline object newarray_int3(int n, int (**p)[3]): if n < 0: n = 0 cdef int (*array)[3] # = NULL ## XXX cdef object ob = allocate(n*sizeof(int[3]), &array) ^ ------------------------------------------------------------ /u/dalcinl/Devel/Cython/mpi4py-dev/src/MPI/asarray.pxi:13:44: Expected: type Please, do not blame me for being too pedantic. I have to deal with this because I need to wrap some MPI calls like the following: int MPI_Group_range_incl(MPI_Group group, int n, int ranges[][3], MPI_Group *newgroup) It has been really hard for me to find an alternative way do build the 'ranges' arguments avoiding Cython or C compiler errors/warnings. However, perhaps I could change sizeof(int[3]) to 3*sizeof(int). It seems it should be equivalent, but I'm not completelly sure? Can someone comment on this? On Mon, Aug 4, 2008 at 3:46 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Current cython-devel trunk broke my code, for example, try to run >> cython on this snippet: >> >> cdef extern from *: >> >> cdef void foo(int[]) >> >> ctypedef int MyInt >> cdef void foo(MyInt[]) >> >> struct MyStruct: >> pass >> cdef void bar(MyStruct[]) >> >> ctypedef MyStruct* MyStructP >> cdef void baz(MyStructP[]) >> >> >> I get error messages like (just the first shown): >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> cdef extern from *: >> >> cdef void foo(int[]) >> >> ctypedef int MyInt >> cdef void foo(MyInt[]) >> ^ >> ------------------------------------------------------------ >> >> /u/dalcinl/Devel/Cython/sandbox/arrdecl.pyx:6:23: Buffer types only >> allowed as function local variables >> > > OK this is fixed now. Apologies. > > (The fix rules out empty [] for buffer access, but this fits in with the > latest buffer discussion of "auto-acquire if enough defaults are > provided".). > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Aug 5 17:34:35 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 5 Aug 2008 17:34:35 +0200 (CEST) Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48984C75.40101@student.matnat.uio.no> References: <48984C75.40101@student.matnat.uio.no> Message-ID: <57166.213.61.181.86.1217950475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Feedback welcome on the below set of conventions -- I just picked some > conventions to get going and implemented them as they are easily > changeable. Thanks for doing this. > Terminology first -- I don't really like the word pragma... "code > generation options" are perhaps better? I've called them "(compiler) > options" below and in the code base, while calling what is options today > "arguments" (as it is command line arguments they really are). I think a good term is "compiler directives". > Ways to set options: > > a) > > #cython: boundscheck=True, other=False > #cython: third = no > > before any code in the file (but whitespace is allowed, I'm using > Plex/the parser). The boolean value is case-insensitive and > true/false/yes/no is allowed (is this too much? The thing is, for > command-line arguments "True" is rather unfriendly and I'd like it to be > consistent). I would prefer having a strict Python syntax in the source, even if the cmd line handles it in a more relaxed way. > c) A decorator works on function-scope: > > cimport cython > @cython.boundscheck(True) > def f(): > ... > > The decorator is stripped away before the decorator transform. Nice. > There is > not really such symbol as cython.boundscheck in the scope, it is a magic > option only. So of course you cannot do > > a = cython.boundscheck > @a > ... Fine with me. > d) A with statement: > > cimport cython > with cython.boundscheck(False): > ... > > The entire with statement is then stripped out of the source prior to > the with transformation. Fine. We should take a little care that we document directives that cannot be used that way, e.g. because they only apply to functions and not to simple blocks (not sure we ever need that, just a side-note). > BTW, on "cimport cython": This is magic as well. I.e. it always is in > front of the include path, and there will be no way to have a pxd file > named "cython.pxd" imported. Is this OK? (I can change it but it would > be less simple in more than one place, and people really should always > have the cython scope available.) Having a cython.pxd would be nice for documentation purposes. However, I think it's perfectly ok to make it a virtual file as long as you have to cimport it before you can use it. Stefan From dagss at student.matnat.uio.no Tue Aug 5 17:44:49 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 Aug 2008 17:44:49 +0200 (CEST) Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: References: <48974E6C.8010209@student.matnat.uio.no> Message-ID: <64325.193.157.229.67.1217951089.squirrel@webmail.uio.no> Lisandro Dalcin wrote: > Dag, I still have problems with this (the 'sizeof(int[3])' part): > > cdef inline object newarray_int3(int n, int (**p)[3]): > if n < 0: n = 0 > cdef int (*array)[3] # = NULL ## XXX > cdef object ob = allocate(n*sizeof(int[3]), &array) > ^ > ------------------------------------------------------------ > > /u/dalcinl/Devel/Cython/mpi4py-dev/src/MPI/asarray.pxi:13:44: Expected: > type > > Please, do not blame me for being too pedantic. I have to deal with > this because I need to wrap some MPI calls like the following: No no, this is great help for me!; these things happen because of cases I didn't consider where the [] notation is used. I'll get on to it right away, these things shouldn't break. Dag Sverre From dagss at student.matnat.uio.no Tue Aug 5 18:11:49 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 Aug 2008 18:11:49 +0200 (CEST) Subject: [Cython] Done with compiler options/pragmas In-Reply-To: References: <48984C75.40101@student.matnat.uio.no> Message-ID: <64386.193.157.229.67.1217952709.squirrel@webmail.uio.no> Lisandro Dalcin wrote: > On Tue, Aug 5, 2008 at 9:49 AM, Dag Sverre Seljebotn > wrote: >> >> Ways to set options: >> a) >> #cython: boundscheck=True, other=False >> #cython: third = no >> >> before any code in the file (but whitespace is allowed, I'm using >> Plex/the parser). The boolean value is case-insensitive and >> true/false/yes/no is allowed (is this too much? The thing is, for >> command-line arguments "True" is rather unfriendly and I'd like it to be >> consistent). > > It is not to much at all. I would even ask for '1' and '0' also being > accepted for booleans. I'm going to side with Stefan here and tighten it up: "here should be one? and preferably only one ?obvious way to do it." So so far it stands +2 to True/False in source and allow yes/no on command line as well, and +1 for keeping it as it is but add 1 and 0 as well. I want to mention another alternative though: #cython: boundscheck(True) some_other(False) because this makes things consistent within the pyx file and leaves the way open for multiple arguments (but = on the command line is fine because () is such a problem to input using a shell...) >> b) A command line argument "cython -O boundscheck=False,other=tRUe -O >> third=yes". This overrides the #cython comments, but NOT c) and d) >> listed below. > > Well, '-O' smells to 'optimization' for me. True. Then again, most of the options will be optimization-related. Any other suggestions? -d or -D for directive perhaps, if they are not taken. -D means #define in C of course... >> c) A decorator works on function-scope: >> >> cimport cython >> @cython.boundscheck(True) >> def f(): >> ... >> >> d) A with statement: >> cimport cython >> with cython.boundscheck(False): >> ... > > I really prefer the decorator form. The 'with' form could require a > lot of identation if you want to set a lot for options. Well, then you use the decorator form in your code. The with form is useful if you really need to optimize a single line within a big function, without requiring you to create a new function. In my buffer test case I am switching back and forth from bounds checking within a function and it works excellently... Hopefully we can have class decorators as well though. > If it is too magic, then perhaps 'cimport' should not be the way to > access the options. What about 'cython' being a sort or 'reserved > global'. Furthermore, perhaps 'cython' is not a good name. Perhaps in > the near future Pyrex can take some of your ideas. So perhaps is > better to find a 'neutral' name. As I am against tainting the global namespace with no-Python names that is not going to get my support, so this is +2 for requiring the cimport (me and Stefan) and -1 (from you) so far. As for Pyrex compatability -- in general, I'm not very up for going into discussions about keeping compatability with "what Pyrex might do". If you have a lot of Python code that you try to modify and compile in Cython, it is much more obvious what @cython.X is about that @options.X (which options? Something in Python?) If Greg asks for it I'll definitely reconsider my vote though. Perhaps "compile", but there's a lot of other stuff we want to have in the cython namespace that doesn't fit with compile. Dag Sverre From dagss at student.matnat.uio.no Tue Aug 5 18:56:09 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 18:56:09 +0200 Subject: [Cython] Reevaluating buffer syntax Message-ID: <48988629.506@student.matnat.uio.no> So, Lisandro has shown that the buffer [] syntax has some problems, and I'd like to write some on that. As it is, things are going to work "by accident"; because dtype is before ndim, you either have non-numerals first or you have a keyword argument (ndim=2), so any number or nothing following [ means array notation. It is technically possible to make the distinction on the base type instead (but some work to make it happen, it means that the parser cannot resolve the meaning of the [], I have to postpone it until during analysis phase and create a "TrailingBracketsTypeNode" -- and because of the ordering of dtype and ndim, I luckily don't need to). Still, it feels quite unsatisfying for a syntax -- one could easily be led to think that cdef ndarray[int, 2] means a 1D array of ints with two elements. So I'm reconsidering some of these options now: - Follow Dan Gindikin and Pex: Make a new syntax in the Lexicon for integer literals followed by a "D", as meaning "dimensionality". So you need to do cdef ndarray[int, 2D] As [] tends to mean "array" and the contents "length", this has a certain appeal, it is more unambigious what it all means. Another advantage is that this could be hard-linked to the ndim keyword, so that if you do cdef class DoubleArray: __cythonbufferdefaults__ = {'dtype': double} then one could do cdef DoubleArray[2D] arr (This would get really strange without the D suffix.) - Require that ndim is given by name. The verbosity of this is a big problem IMO. - Both? Either [0-9]+D, or ndim=[0-9]+ -- Dag Sverre From dagss at student.matnat.uio.no Tue Aug 5 18:56:24 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 18:56:24 +0200 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: References: <48974E6C.8010209@student.matnat.uio.no> Message-ID: <48988638.4070302@student.matnat.uio.no> Lisandro Dalcin wrote: > Dag, I still have problems with this (the 'sizeof(int[3])' part): > > cdef inline object newarray_int3(int n, int (**p)[3]): > if n < 0: n = 0 > cdef int (*array)[3] # = NULL ## XXX > cdef object ob = allocate(n*sizeof(int[3]), &array) > ^ > ------------------------------------------------------------ > > /u/dalcinl/Devel/Cython/mpi4py-dev/src/MPI/asarray.pxi:13:44: Expected: type > OK fix is in now. -- Dag Sverre From cwitty at newtonlabs.com Tue Aug 5 19:03:53 2008 From: cwitty at newtonlabs.com (Carl Witty) Date: Tue, 5 Aug 2008 10:03:53 -0700 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <57166.213.61.181.86.1217950475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <48984C75.40101@student.matnat.uio.no> <57166.213.61.181.86.1217950475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Tue, Aug 5, 2008 at 8:34 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> There is >> not really such symbol as cython.boundscheck in the scope, it is a magic >> option only. So of course you cannot do >> >> a = cython.boundscheck >> @a >> ... > > Fine with me. Hopefully this will fail at compile-time rather than runtime? >> d) A with statement: >> >> cimport cython >> with cython.boundscheck(False): >> ... >> >> The entire with statement is then stripped out of the source prior to >> the with transformation. > > Fine. We should take a little care that we document directives that cannot > be used that way, e.g. because they only apply to functions and not to > simple blocks (not sure we ever need that, just a side-note). Again, such directives should fail at compile-time. (The restriction should be documented, too, of course.) Carl From dalcinl at gmail.com Tue Aug 5 19:06:07 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 Aug 2008 14:06:07 -0300 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <64386.193.157.229.67.1217952709.squirrel@webmail.uio.no> References: <48984C75.40101@student.matnat.uio.no> <64386.193.157.229.67.1217952709.squirrel@webmail.uio.no> Message-ID: On Tue, Aug 5, 2008 at 1:11 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> It is not to much at all. I would even ask for '1' and '0' also being >> accepted for booleans. > > I'm going to side with Stefan here and tighten it up: "here should be one? > and preferably only one ?obvious way to do it." > > So so far it stands +2 to True/False in source and allow yes/no on command > line as well, and +1 for keeping it as it is but add 1 and 0 as well. I was not clear enough. For the command line, accept all of true,false | yes/no | 1,0 . For souce code, I can live with just True/False, but remember that using 1 and 0 for booleans is frequent in Python code, though I realize that such usages could be considered a bit dated and should be discouraged. >>> b) A command line argument "cython -O boundscheck=False,other=tRUe -O >>> third=yes". This overrides the #cython comments, but NOT c) and d) >>> listed below. >> >> Well, '-O' smells to 'optimization' for me. > > True. Then again, most of the options will be optimization-related. > > Any other suggestions? -d or -D for directive perhaps, if they are not > taken. -D means #define in C of course... Well, I'm not sure. Perhaps '-O' was a good idea after all. >> I really prefer the decorator form. The 'with' form could require a >> lot of identation if you want to set a lot for options. > > Well, then you use the decorator form in your code. > > Hopefully we can have class decorators as well though. > OK, That's great. I missunderstood you. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Aug 5 19:22:26 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 19:22:26 +0200 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: References: <48984C75.40101@student.matnat.uio.no> <57166.213.61.181.86.1217950475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48988C52.7080002@student.matnat.uio.no> Carl Witty wrote: > On Tue, Aug 5, 2008 at 8:34 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> There is >>> not really such symbol as cython.boundscheck in the scope, it is a magic >>> option only. So of course you cannot do >>> >>> a = cython.boundscheck >>> @a >>> ... >> Fine with me. > > Hopefully this will fail at compile-time rather than runtime? Thanks!, good point. No, I need to fix this (it disappears from scope and thus is deferred to runtime for lookup..) >>> d) A with statement: >>> >>> cimport cython >>> with cython.boundscheck(False): >>> ... >>> >>> The entire with statement is then stripped out of the source prior to >>> the with transformation. >> Fine. We should take a little care that we document directives that cannot >> be used that way, e.g. because they only apply to functions and not to >> simple blocks (not sure we ever need that, just a side-note). > > Again, such directives should fail at compile-time. (The restriction > should be documented, too, of course.) Yep, this one is a no-brainer and easy to do (once we have such an option to do it with). -- Dag Sverre From stefan_ml at behnel.de Tue Aug 5 20:02:00 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Aug 2008 20:02:00 +0200 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: <64325.193.157.229.67.1217951089.squirrel@webmail.uio.no> References: <48974E6C.8010209@student.matnat.uio.no> <64325.193.157.229.67.1217951089.squirrel@webmail.uio.no> Message-ID: <48989598.2080303@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Dag, I still have problems with this (the 'sizeof(int[3])' part): >> >> cdef inline object newarray_int3(int n, int (**p)[3]): >> if n < 0: n = 0 >> cdef int (*array)[3] # = NULL ## XXX >> cdef object ob = allocate(n*sizeof(int[3]), &array) >> ^ >> ------------------------------------------------------------ >> >> /u/dalcinl/Devel/Cython/mpi4py-dev/src/MPI/asarray.pxi:13:44: Expected: >> type >> >> Please, do not blame me for being too pedantic. I have to deal with >> this because I need to wrap some MPI calls like the following: > > No no, this is great help for me!; these things happen because of cases I > didn't consider where the [] notation is used. Plus, they give us valuable test cases that make sure this is the last time we break these things by accident. Stefan From stefan_ml at behnel.de Tue Aug 5 20:23:50 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Aug 2008 20:23:50 +0200 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: References: <48984C75.40101@student.matnat.uio.no> Message-ID: <48989AB6.6060100@behnel.de> Hi, Lisandro Dalcin wrote: > On Tue, Aug 5, 2008 at 9:49 AM, Dag Sverre Seljebotn >> Ways to set options: >> a) >> #cython: boundscheck=True, other=False >> #cython: third = no >> >> before any code in the file (but whitespace is allowed, I'm using >> Plex/the parser). The boolean value is case-insensitive and >> true/false/yes/no is allowed (is this too much? The thing is, for >> command-line arguments "True" is rather unfriendly and I'd like it to be >> consistent). > > It is not to much at all. I would even ask for '1' and '0' also being > accepted for booleans. I think they would be ok on the cmd line, yes. >> b) A command line argument "cython -O boundscheck=False,other=tRUe -O >> third=yes". This overrides the #cython comments, but NOT c) and d) >> listed below. > > Well, '-O' smells to 'optimization' for me. Yes, and Python uses it exactly for that. "-X" has been "reserved for implementation-specific arguments" in Python 3 due to request by Jython (and IronPython?), maybe we should just use that. Stefan From dagss at student.matnat.uio.no Tue Aug 5 20:33:23 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 20:33:23 +0200 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48989AB6.6060100@behnel.de> References: <48984C75.40101@student.matnat.uio.no> <48989AB6.6060100@behnel.de> Message-ID: <48989CF3.2010107@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Lisandro Dalcin wrote: >> On Tue, Aug 5, 2008 at 9:49 AM, Dag Sverre Seljebotn >>> Ways to set options: >>> a) >>> #cython: boundscheck=True, other=False >>> #cython: third = no >>> >>> before any code in the file (but whitespace is allowed, I'm using >>> Plex/the parser). The boolean value is case-insensitive and >>> true/false/yes/no is allowed (is this too much? The thing is, for >>> command-line arguments "True" is rather unfriendly and I'd like it to be >>> consistent). >> It is not to much at all. I would even ask for '1' and '0' also being >> accepted for booleans. > > I think they would be ok on the cmd line, yes. > > >>> b) A command line argument "cython -O boundscheck=False,other=tRUe -O >>> third=yes". This overrides the #cython comments, but NOT c) and d) >>> listed below. >> Well, '-O' smells to 'optimization' for me. > > Yes, and Python uses it exactly for that. > > "-X" has been "reserved for implementation-specific arguments" in Python 3 due > to request by Jython (and IronPython?), maybe we should just use that. Nice! +1 -- Dag Sverre From stefan_ml at behnel.de Tue Aug 5 20:32:52 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Aug 2008 20:32:52 +0200 Subject: [Cython] Reevaluating buffer syntax In-Reply-To: <48988629.506@student.matnat.uio.no> References: <48988629.506@student.matnat.uio.no> Message-ID: <48989CD4.9080900@behnel.de> Hi, Dag Sverre Seljebotn wrote: > cdef ndarray[int, 2D] > > - Require that ndim is given by name. The verbosity of this is a big > problem IMO. One of the two, not both. FWIW, I'm +0 on the keyword argument. There's also the question regarding the brackets: [] or (). Wouldn't ndarray(int, ndim=2) match the proposed assumption syntax? Stefan From dalcinl at gmail.com Tue Aug 5 20:44:21 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 Aug 2008 15:44:21 -0300 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48989AB6.6060100@behnel.de> References: <48984C75.40101@student.matnat.uio.no> <48989AB6.6060100@behnel.de> Message-ID: On Tue, Aug 5, 2008 at 3:23 PM, Stefan Behnel wrote: > "-X" has been "reserved for implementation-specific arguments" in Python 3 due > to request by Jython (and IronPython?), maybe we should just use that. Of course! +1 too from my side. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Aug 5 21:20:48 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 Aug 2008 21:20:48 +0200 Subject: [Cython] Reevaluating buffer syntax In-Reply-To: <48989CD4.9080900@behnel.de> References: <48988629.506@student.matnat.uio.no> <48989CD4.9080900@behnel.de> Message-ID: <4898A810.9060807@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> cdef ndarray[int, 2D] >> >> - Require that ndim is given by name. The verbosity of this is a big >> problem IMO. > > One of the two, not both. FWIW, I'm +0 on the keyword argument. > > There's also the question regarding the brackets: [] or (). Wouldn't > > ndarray(int, ndim=2) > > match the proposed assumption syntax? The proposed assumption syntax and the proposed buffer syntax has been the same thing pretty much, as both were motivated from the usecase first. Using () for buffers were considered and we ruled it out because it seems like a constructor (which has different arguments): cdef ndarray(int, 2) buf = ndarray([1, 2, 3, 4], dtype=int) Also () is used in a lot of places already. <> is another option, but it is already used for casts (and potentially C++ templates). Still I'm starting to lean towards <> as a solution for this as well. Pex uses {}. I might be wrong in this, but it just looks too ugly, and for some reason looks more like a dict than [] looks like a list (I suppose because [] is already overloaded for doing indexing). Actually I'm not sure how big the problem is -- there's never any real need for the int[] syntax, you usually do "int varname[43]" or "int*" or 34*sizeof(int) instead. You still have to "know" for other the other syntax proposals that the number is the ndim and not length... but perhaps [] contribute to that problem... More: buffer(ndarray, int, 2) ndarray.buffer(int, 2) Both less transparent and harder to use... -- Dag Sverre From greg.ewing at canterbury.ac.nz Wed Aug 6 01:46:16 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 06 Aug 2008 11:46:16 +1200 Subject: [Cython] Type scope vs. runtime scope In-Reply-To: <64D35068-1F15-4454-ABF6-3311E67BC098@math.washington.edu> References: <48960A8D.1090607@student.matnat.uio.no> <63592.193.157.229.67.1217918197.squirrel@webmail.uio.no> <2F49098A-E3FF-42CD-B780-EDE4C3005B75@math.washington.edu> <63605.193.157.229.67.1217920006.squirrel@webmail.uio.no> <64D35068-1F15-4454-ABF6-3311E67BC098@math.washington.edu> Message-ID: <4898E648.501@canterbury.ac.nz> Robert Bradshaw wrote: > On that note, why doesn't it die when you put "int" in there > instead? The built-in C type names are treated as a special case by the parser. -- Greg From greg.ewing at canterbury.ac.nz Wed Aug 6 02:07:10 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 06 Aug 2008 12:07:10 +1200 Subject: [Cython] Multiple cimports to same name? In-Reply-To: <48983EAF.5010305@student.matnat.uio.no> References: <48983EAF.5010305@student.matnat.uio.no> Message-ID: <4898EB2E.2050102@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > cimport python_int as a > a.PyInt_Check(3) > cimport python_dict as a > > Apparently, only the first cimport is considered, I would regard this as a bug -- it shouldn't be letting you redeclare a that way. I tried the following with Pyrex 0.9.8.4: cimport blarg as m cimport zax as m and got /Users/greg/foo/foo.pyx:2:8: 'm' does not match previous declaration /Users/greg/foo/foo.pyx:1:8: Previous declaration is here -- Greg From greg.ewing at canterbury.ac.nz Wed Aug 6 02:17:05 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 06 Aug 2008 12:17:05 +1200 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48984C75.40101@student.matnat.uio.no> References: <48984C75.40101@student.matnat.uio.no> Message-ID: <4898ED81.3080200@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > ALL options we define can then consistently be toggled in the four ways > listed below. That's rather a lot of ways. Are they all really necessary? Shouldn't there be just one obvious way of specifying these things in the source instead of three? -- Greg From greg.ewing at canterbury.ac.nz Wed Aug 6 02:21:20 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 06 Aug 2008 12:21:20 +1200 Subject: [Cython] cython-devel: array declaration problems with current trunk In-Reply-To: References: <48974E6C.8010209@student.matnat.uio.no> Message-ID: <4898EE80.3040004@canterbury.ac.nz> Lisandro Dalcin wrote: > However, perhaps I could change sizeof(int[3]) to 3*sizeof(int). It > seems it should be equivalent, but I'm not completelly sure? Yes, I think it will be equivalent. (It had better be, otherwise a lot of code that allocates memory for arrays is broken...) But the original syntax should be made to work if possible. -- Greg From greg.ewing at canterbury.ac.nz Wed Aug 6 02:32:07 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 06 Aug 2008 12:32:07 +1200 Subject: [Cython] Reevaluating buffer syntax In-Reply-To: <48988629.506@student.matnat.uio.no> References: <48988629.506@student.matnat.uio.no> Message-ID: <4898F107.2060404@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > - Follow Dan Gindikin and Pex: Make a new syntax in the Lexicon for > integer literals followed by a "D", as meaning "dimensionality". So you > need to do > > cdef ndarray[int, 2D] I would be slightly worried that this could conflict with some future syntax for decimal float literals. Also, there would still be some tricky parsing issues, since you have to get all the way up to the 2D (or at least the comma) before you realise what's going on. Another approach would be to use a different kind of brackets, such as cdef ndarray{int, 2D} -- Greg From robertwb at math.washington.edu Wed Aug 6 05:21:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 5 Aug 2008 20:21:42 -0700 Subject: [Cython] Multiple cimports to same name? In-Reply-To: <48983EAF.5010305@student.matnat.uio.no> References: <48983EAF.5010305@student.matnat.uio.no> Message-ID: <42E08C1E-993C-4F5A-A2E8-1B832C78D05F@math.washington.edu> On Aug 5, 2008, at 4:51 AM, Dag Sverre Seljebotn wrote: > While dealing with my pragma code I encountered this issue which I > want > to note down (I need to know how to deal with it for pragmas): > > cimport python_int as a > a.PyInt_Check(3) > cimport python_dict as a > a.PyInt_Check(3) > a.PyDict_New() > > Apparently, only the first cimport is considered, so that > PyDict_New is > turned into a module dict lookup and PyInt_Check is a native C call > also > the second time. If the order of the cimports are reversed, the > opposite > happen. > > It wouldn't take long to raise a compiler error on the second > cimport as > a fix for now ("name a already taken by a cimport"), but since it > would > break backwards compatability to some degree (of already broken > code, I > suppose) I wanted to check with you first. I think there should be a compile time error here. Code that violates this assumption is clearly wrong and easily fixed. Cython is much looser about allowing re-declarations of things then it should be--and this needs to be fixed. > Long term I suppose having the second PyInt_Check above turn into a > dictionary lookup above is the wanted behaviour, but given that we > give > a compiler error now then such behaviour will be backwards compatible. I'm not even sure how to sanely do that with cimports. What happens here? cimport x as a def foo(): return a.foo() cimport y as a - Robert From robertwb at math.washington.edu Wed Aug 6 05:37:35 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 5 Aug 2008 20:37:35 -0700 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48984C75.40101@student.matnat.uio.no> References: <48984C75.40101@student.matnat.uio.no> Message-ID: On Aug 5, 2008, at 5:49 AM, Dag Sverre Seljebotn wrote: > Feedback welcome on the below set of conventions -- I just picked some > conventions to get going and implemented them as they are easily > changeable. So feel free to comment on all choices of names, > letters etc. > > Terminology first -- I don't really like the word pragma... "code > generation options" are perhaps better? I've called them "(compiler) > options" below and in the code base, while calling what is options > today > "arguments" (as it is command line arguments they really are). > > Anyway, the system is this: Options.py contains a list of compiler > options to be controlled through this system. Currently this is only a > boolean "boundscheck". > > ALL options we define can then consistently be toggled in the four > ways > listed below. During a transform step, ALL nodes in the parse tree > has a > node.options dictionary added giving all options (with defaults filled > in) in effect for that exact node (this only adds a reference of > course, > so consider that dict immutable). > > IndexNode then simply checks self.options['boundscheck']. > > Ways to set options: > > a) > > #cython: boundscheck=True, other=False > #cython: third = no > > before any code in the file (but whitespace is allowed, I'm using > Plex/the parser). The boolean value is case-insensitive and > true/false/yes/no is allowed (is this too much? The thing is, for > command-line arguments "True" is rather unfriendly and I'd like it > to be > consistent). I think we should follow a similar format as in PEP 0263. Specifically, I'm not even sure the "cython" is necessary--the compiler directives are arbitrary key-value pairs, and they are looked up as needed. The ones that Cython uses/supports will be documented. For the command line, we should be flexible, but a constant literal for the values of the directives (not all of them may be booleans). > b) A command line argument "cython -O boundscheck=False,other=tRUe -O > third=yes". This overrides the #cython comments, but NOT c) and d) > listed below. +1, except for -X as mentioned. > The syntax of the string passed to -O is the same as the one passed to > #cython, which is why I allowed yes/no in it. (For instance, ssh has > options like this: ssh -o ForwardX11=no). > > c) A decorator works on function-scope: > > cimport cython > @cython.boundscheck(True) > def f(): > ... > > The decorator is stripped away before the decorator transform. > There is > not really such symbol as cython.boundscheck in the scope, it is a > magic > option only. So of course you cannot do > > a = cython.boundscheck > @a > ... > d) A with statement: > > cimport cython > with cython.boundscheck(False): > ... > > The entire with statement is then stripped out of the source prior to > the with transformation. These are fine, though it does start to sound like a lot of ways to do the same thing (they all have valid use cases though). I also like these in the sense that a fake "cython" object could be provided whose __getattr__ always returns an object which (1) returns the identity on __call__ and (2) implements a trivial __enter__/__exit__ so these would be valid in Python too. > > > BTW, on "cimport cython": This is magic as well. I.e. it always is in > front of the include path, and there will be no way to have a pxd file > named "cython.pxd" imported. Is this OK? (I can change it but it would > be less simple in more than one place, and people really should always > have the cython scope available.) I think that's sensible. - Robert From robertwb at math.washington.edu Wed Aug 6 06:04:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 5 Aug 2008 21:04:33 -0700 Subject: [Cython] Reevaluating buffer syntax In-Reply-To: <4898A810.9060807@student.matnat.uio.no> References: <48988629.506@student.matnat.uio.no> <48989CD4.9080900@behnel.de> <4898A810.9060807@student.matnat.uio.no> Message-ID: <0D10A4D2-D8EB-49E8-A126-199AE9B11D7B@math.washington.edu> On Aug 5, 2008, at 12:20 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> Dag Sverre Seljebotn wrote: >>> cdef ndarray[int, 2D] >>> >>> - Require that ndim is given by name. The verbosity of this is a big >>> problem IMO. >> >> One of the two, not both. FWIW, I'm +0 on the keyword argument. >> >> There's also the question regarding the brackets: [] or (). Wouldn't >> >> ndarray(int, ndim=2) >> >> match the proposed assumption syntax? > > The proposed assumption syntax and the proposed buffer syntax has been > the same thing pretty much, as both were motivated from the usecase > first. > > Using () for buffers were considered and we ruled it out because it > seems like a constructor (which has different arguments): > > cdef ndarray(int, 2) buf = ndarray([1, 2, 3, 4], dtype=int) > > Also () is used in a lot of places already. My thoughts as well. > <> is another option, but it is already used for casts (and > potentially > C++ templates). Still I'm starting to lean towards <> as a solution > for > this as well. I'm not a huge fan of this one, but it's certainly better than most options. > Pex uses {}. I might be wrong in this, but it just looks too ugly, and > for some reason looks more like a dict than [] looks like a list (I > suppose because [] is already overloaded for doing indexing). Same here. > Actually I'm not sure how big the problem is -- there's never any real > need for the int[] syntax, you usually do "int varname[43]" or > "int*" or > 34*sizeof(int) instead. You still have to "know" for other the other > syntax proposals that the number is the ndim and not length... but > perhaps [] contribute to that problem... I would like to see something like int[n] be useable to create automatically memory managed arrays of ints in the future. (Having to learn malloc, etc. seems to be one of the most un-pythonic thing of using Cython). > More: > buffer(ndarray, int, 2) > ndarray.buffer(int, 2) > > Both less transparent and harder to use... Yes, -1. I think the best option is to use the D suffix. I'm not too worried about future compatibility with a decimal literal (if something like that ever comes), 'cause it wouldn't impose backwards incompatibility anyways. - Robert From stefan_ml at behnel.de Wed Aug 6 07:52:48 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 Aug 2008 07:52:48 +0200 Subject: [Cython] cython namespace In-Reply-To: <63601.193.157.229.67.1217919417.squirrel@webmail.uio.no> References: <4896DDF9.80705@student.matnat.uio.no> <33785.213.61.181.86.1217859240.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48971436.3080102@student.matnat.uio.no> <6C5B7ED7-DDEE-450B-AED9-BDADCF70B4C5@math.washington.edu> <63601.193.157.229.67.1217919417.squirrel@webmail.uio.no> Message-ID: <48993C30.10803@behnel.de> Hi, Dag Sverre Seljebotn wrote: > I'm still for "flat better than nested", but failing that, I'd propose > that everything is in cython.buffer but shape is also in cython. Could we wait with the latter until we actually have something different than buffers that it's worth getting the shape of? Until then, I think everything that relates to buffers should be in cython.buffer. People can do their cimports as they like anyway, but it should be clear where things are coming from. Stefan From dagss at student.matnat.uio.no Wed Aug 6 10:49:52 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 10:49:52 +0200 Subject: [Cython] Warning about another bug Message-ID: <489965B0.3040001@student.matnat.uio.no> Obviously my past efforts weren't good enough: DEF MYLEN = 3 c = sizeof(MyStruct[MYLEN]) So it seems that the parser can never discern the current syntax (MYLEN could have been a typedef and MyStruct and extension type, and then it would have been buffer syntax). So if keeping the current syntax, I seem to have no choice but to create a TrailingBracketTypeNode and delay the decision until type analysis. This takes some work, and adds complexity, so I want to be sure that it is needed first. So, can I leave this bug in until the outcome of the syntax is more conclusive? (I'm currently gathering some response from the NumPy list and I want to see if they come up with anything...) -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 6 10:52:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 10:52:12 +0200 Subject: [Cython] Reevaluating buffer syntax In-Reply-To: <0D10A4D2-D8EB-49E8-A126-199AE9B11D7B@math.washington.edu> References: <48988629.506@student.matnat.uio.no> <48989CD4.9080900@behnel.de> <4898A810.9060807@student.matnat.uio.no> <0D10A4D2-D8EB-49E8-A126-199AE9B11D7B@math.washington.edu> Message-ID: <4899663C.40302@student.matnat.uio.no> Robert Bradshaw wrote: > I would like to see something like int[n] be useable to create > automatically memory managed arrays of ints in the future. (Having to > learn malloc, etc. seems to be one of the most un-pythonic thing of > using Cython). Can you reconfirm this, I always assumed it would look like cdef int myarr[n] i.e. the [] on the variable rather than the type (which is never a problem). -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 6 11:27:49 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 11:27:49 +0200 Subject: [Cython] More array/buffer syntax In-Reply-To: <7AE03A9D-989B-4D97-8D81-0309BE4696EB@math.washington.edu> References: <48974710.4060906@student.matnat.uio.no> <4897AA4A.3090608@canterbury.ac.nz> <7AE03A9D-989B-4D97-8D81-0309BE4696EB@math.washington.edu> Message-ID: <48996E95.9060402@student.matnat.uio.no> Robert Bradshaw wrote: > On Aug 4, 2008, at 6:18 PM, Greg Ewing wrote: > >> Dag Sverre Seljebotn wrote: >> >>> I don't think the [] syntax buys anything above a regular >>> pointer notation? If so, removing support for int[]-notation >>> altogether >>> would get rid of the issue and lead to a cleaner grammar. >> The current declaration syntax has the advantage that it's >> the same as in C, so people familiar with C don't have to >> learn anything new. This would be a step away from that, >> although a fairly minor one. >> >> Another consideration is how much existing code it would >> break, which could be quite a lot. > > +1 > > Also, c_type[] vs. object_type[] are easily discerned, and they both > have similar (high level) meaning. Well, there is an assymetry. If you did cdef int[handler=ndarray, ndim=2] or cdef int[ndarray, 2] you would be more correct. Hey...Perhaps that is not such a bad idea? Putting the dtype on the outside and the container type on the inside? This would mimic C in some way. One could have cdef int[ndim=2, mode='full'] would be an int buffer on an object. And cdef int[len=44, mode='native'] would mean the C int[44]... Just brainstorming though. -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 6 15:06:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 15:06:12 +0200 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <4898ED81.3080200@canterbury.ac.nz> References: <48984C75.40101@student.matnat.uio.no> <4898ED81.3080200@canterbury.ac.nz> Message-ID: <4899A1C4.3050504@student.matnat.uio.no> Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> ALL options we define can then consistently be toggled in the four ways >> listed below. > > That's rather a lot of ways. Are they all really > necessary? Shouldn't there be just one obvious way of > specifying these things in the source instead of three? The with statement fits in all situations, but it means another level of indentation in the source just for the directive, so you end up indenting and dedenting all your source all the time...having the two non-indentation-options just leads to much higher usability. (Introducing non-Python syntax would be another way, though I'd oppose that myself. In the end of the day it is a matter of taste I suppose; but there appears to be strong support for keeping all of them.) -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 6 17:18:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 17:18:44 +0200 Subject: [Cython] A brand new syntax idea I actually like! Message-ID: <4899C0D4.1020607@student.matnat.uio.no> Realistically, the old [] syntax is here for the next Cython release so I might move on to fix that bug... But here's an idea for a syntax I actually like (let's hear about you)...: Somehow specify that the variable in question has TWO types. I.e., it supports the ndarray interface, as well as the buffer interface. So, for buffers this would becom cdef ndarray,buffer(int, 2) buf = ... # ndarray supporting buffer cdef buffer(int, 2) buf = ... # plain object supporting buffer cdef object,buffer(int, 2) buf = ... # same as above If it is possible with objects that inherit from two extension types (didn't try it) this could even be used like this: cdef MyType1,MyType2 obj = ... This would make the syntax "explain" what happens today anyway, with "cdef MyType1 obj" being shorthand for "cdef object,MyType1 obj". I.e. "first try to resolve compile-time towards MyType1, and then towards object". *running away and ducking* :-) -- Dag Sverre From robertwb at math.washington.edu Wed Aug 6 18:24:11 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 Aug 2008 09:24:11 -0700 Subject: [Cython] Warning about another bug In-Reply-To: <489965B0.3040001@student.matnat.uio.no> References: <489965B0.3040001@student.matnat.uio.no> Message-ID: On Aug 6, 2008, at 1:49 AM, Dag Sverre Seljebotn wrote: > Obviously my past efforts weren't good enough: > > DEF MYLEN = 3 > c = sizeof(MyStruct[MYLEN]) > > So it seems that the parser can never discern the current syntax > (MYLEN > could have been a typedef and MyStruct and extension type, and then it > would have been buffer syntax). > > So if keeping the current syntax, I seem to have no choice but to > create > a TrailingBracketTypeNode and delay the decision until type analysis. > This takes some work, and adds complexity, so I want to be sure > that it > is needed first. I'm not understanding why we need to introduce a new node here, why not let the sizeof operator be OK with index nodes? > So, can I leave this bug in until the outcome of the syntax is more > conclusive? (I'm currently gathering some response from the NumPy list > and I want to see if they come up with anything...) Yes, as long as we don't take too long. - Robert From robertwb at math.washington.edu Wed Aug 6 18:24:59 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 Aug 2008 09:24:59 -0700 Subject: [Cython] Reevaluating buffer syntax In-Reply-To: <4899663C.40302@student.matnat.uio.no> References: <48988629.506@student.matnat.uio.no> <48989CD4.9080900@behnel.de> <4898A810.9060807@student.matnat.uio.no> <0D10A4D2-D8EB-49E8-A126-199AE9B11D7B@math.washington.edu> <4899663C.40302@student.matnat.uio.no> Message-ID: <146CB7B2-1F15-4BB6-9728-C918848AAEFE@math.washington.edu> On Aug 6, 2008, at 1:52 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> I would like to see something like int[n] be useable to create >> automatically memory managed arrays of ints in the future. (Having to >> learn malloc, etc. seems to be one of the most un-pythonic thing of >> using Cython). > > Can you reconfirm this, I always assumed it would look like > > cdef int myarr[n] > > i.e. the [] on the variable rather than the type (which is never a > problem). I'm talking about dynamically created ones. Not sure of the right syntax yet though. - Robert From robertwb at math.washington.edu Wed Aug 6 18:27:50 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 Aug 2008 09:27:50 -0700 Subject: [Cython] More array/buffer syntax In-Reply-To: <48996E95.9060402@student.matnat.uio.no> References: <48974710.4060906@student.matnat.uio.no> <4897AA4A.3090608@canterbury.ac.nz> <7AE03A9D-989B-4D97-8D81-0309BE4696EB@math.washington.edu> <48996E95.9060402@student.matnat.uio.no> Message-ID: On Aug 6, 2008, at 2:27 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Aug 4, 2008, at 6:18 PM, Greg Ewing wrote: >> >>> Dag Sverre Seljebotn wrote: >>> >>>> I don't think the [] syntax buys anything above a regular >>>> pointer notation? If so, removing support for int[]-notation >>>> altogether >>>> would get rid of the issue and lead to a cleaner grammar. >>> The current declaration syntax has the advantage that it's >>> the same as in C, so people familiar with C don't have to >>> learn anything new. This would be a step away from that, >>> although a fairly minor one. >>> >>> Another consideration is how much existing code it would >>> break, which could be quite a lot. >> >> +1 >> >> Also, c_type[] vs. object_type[] are easily discerned, and they both >> have similar (high level) meaning. > > Well, there is an assymetry. If you did > > cdef int[handler=ndarray, ndim=2] > > or > > cdef int[ndarray, 2] > > you would be more correct. Hey...Perhaps that is not such a bad idea? > Putting the dtype on the outside and the container type on the inside? > This would mimic C in some way. > > One could have > > cdef int[ndim=2, mode='full'] > > would be an int buffer on an object. And > > cdef int[len=44, mode='native'] > > would mean the C int[44]... > > Just brainstorming though. Hmm... I think that it's better to put the object first (as that's the "primary" type--it's still recounted and all, the rest is extra information about it). - Robert From dagss at student.matnat.uio.no Wed Aug 6 18:35:42 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Aug 2008 18:35:42 +0200 Subject: [Cython] Warning about another bug In-Reply-To: References: <489965B0.3040001@student.matnat.uio.no> Message-ID: <4899D2DE.1090104@student.matnat.uio.no> Robert Bradshaw wrote: > On Aug 6, 2008, at 1:49 AM, Dag Sverre Seljebotn wrote: > >> Obviously my past efforts weren't good enough: >> >> DEF MYLEN = 3 >> c = sizeof(MyStruct[MYLEN]) >> >> So it seems that the parser can never discern the current syntax >> (MYLEN >> could have been a typedef and MyStruct and extension type, and then it >> would have been buffer syntax). >> >> So if keeping the current syntax, I seem to have no choice but to >> create >> a TrailingBracketTypeNode and delay the decision until type analysis. >> This takes some work, and adds complexity, so I want to be sure >> that it >> is needed first. > > I'm not understanding why we need to introduce a new node here, why > not let the sizeof operator be OK with index nodes? Yes, if you can guarantee me that the sizeof operator is the only place this happens, and that the sizeof name cannot ever be overriden and so on, I could do that. The thing is, I've been bitten so many times by wrong assumptions by now that I'd rather use a new rule controlling the thing: If [] is used on a pyobject, then it is buffer, on a native type, then array. But then I need to defer the resolution of what the [] means until type analysis time. (Basically I need to merge the buffer and array type nodes and set some flags, so the number of node types are reduced.) > >> So, can I leave this bug in until the outcome of the syntax is more >> conclusive? (I'm currently gathering some response from the NumPy list >> and I want to see if they come up with anything...) > > Yes, as long as we don't take too long. I'll likely fix it tomorrow, if the syntax is kept (which I think it will be). -- Dag Sverre From robertwb at math.washington.edu Wed Aug 6 18:32:43 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 Aug 2008 09:32:43 -0700 Subject: [Cython] A brand new syntax idea I actually like! In-Reply-To: <4899C0D4.1020607@student.matnat.uio.no> References: <4899C0D4.1020607@student.matnat.uio.no> Message-ID: On Aug 6, 2008, at 8:18 AM, Dag Sverre Seljebotn wrote: > Realistically, the old [] syntax is here for the next Cython > release so > I might move on to fix that bug... > > But here's an idea for a syntax I actually like (let's hear about > you)...: > > Somehow specify that the variable in question has TWO types. I.e., it > supports the ndarray interface, as well as the buffer interface. > > So, for buffers this would becom > > cdef ndarray,buffer(int, 2) buf = ... # ndarray supporting buffer > > cdef buffer(int, 2) buf = ... # plain object supporting buffer > cdef object,buffer(int, 2) buf = ... # same as above > > If it is possible with objects that inherit from two extension types > (didn't try it) this could even be used like this: > > cdef MyType1,MyType2 obj = ... > > This would make the syntax "explain" what happens today anyway, with > "cdef MyType1 obj" being shorthand for "cdef object,MyType1 obj". I.e. > "first try to resolve compile-time towards MyType1, and then towards > object". > > *running away and ducking* :-) -1. 99% of the time such declarations would be either impossible or implied (e.g. with inheritance), and the Python way to do things like this in multiple inheritance (which we can't do from Cython without an additional virtual lookup penalty). Also, this syntax is much more verbose, and hard to parse (think about argument declarations, or "cdef a, b" declares two variables, but "cdef a, b c" declares one? - Robert From robertwb at math.washington.edu Wed Aug 6 18:35:50 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 Aug 2008 09:35:50 -0700 Subject: [Cython] Warning about another bug In-Reply-To: <4899D2DE.1090104@student.matnat.uio.no> References: <489965B0.3040001@student.matnat.uio.no> <4899D2DE.1090104@student.matnat.uio.no> Message-ID: <22A45948-FD2D-4060-B731-1AC9FF6F8DD7@math.washington.edu> On Aug 6, 2008, at 9:35 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Aug 6, 2008, at 1:49 AM, Dag Sverre Seljebotn wrote: >> >>> Obviously my past efforts weren't good enough: >>> >>> DEF MYLEN = 3 >>> c = sizeof(MyStruct[MYLEN]) >>> >>> So it seems that the parser can never discern the current syntax >>> (MYLEN >>> could have been a typedef and MyStruct and extension type, and >>> then it >>> would have been buffer syntax). >>> >>> So if keeping the current syntax, I seem to have no choice but to >>> create >>> a TrailingBracketTypeNode and delay the decision until type >>> analysis. >>> This takes some work, and adds complexity, so I want to be sure >>> that it >>> is needed first. >> >> I'm not understanding why we need to introduce a new node here, why >> not let the sizeof operator be OK with index nodes? > > Yes, if you can guarantee me that the sizeof operator is the only > place > this happens, and that the sizeof name cannot ever be overriden and so > on, I could do that. > > The thing is, I've been bitten so many times by wrong assumptions > by now > that I'd rather use a new rule controlling the thing: If [] is used > on a > pyobject, then it is buffer, on a native type, then array. But then I > need to defer the resolution of what the [] means until type analysis > time. I think this makes sense. > (Basically I need to merge the buffer and array type nodes and set > some flags, so the number of node types are reduced.) > >> >>> So, can I leave this bug in until the outcome of the syntax is more >>> conclusive? (I'm currently gathering some response from the NumPy >>> list >>> and I want to see if they come up with anything...) >> >> Yes, as long as we don't take too long. > > I'll likely fix it tomorrow, if the syntax is kept (which I think it > will be). > > -- > Dag Sverre > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Thu Aug 7 17:49:41 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 7 Aug 2008 12:49:41 -0300 Subject: [Cython] about to buffer syntax Message-ID: I'm still convinced that Fortran 90 array declarators could provide a more powerfull syntax for buffers, For this, the dimensions should be specified with a tuple, examples below: cdef ndarray[int, ()] a # a scalar cdef ndarray[int, (3,)] # a1D array with 3 items cdef ndarray[int, (:,)] # a 1D array of any size cdef ndarray[int (2,3)] # 2D array with 2 rows and 3 columns cdef ndarray[int (:,3)] # 2D array with any number of rows and 3 columns cdef ndarray[int (:,:)] # 2D array with any number of rows and columns cdef ndarray[float, (3,3,:)] # 3D array, a 'sequence' of 3x3 array We could even use Ellipsis '...' in a similar sense numpy uses them cdef ndarray[int, (...)] # any number of dimension cdef ndarray[int, (:, ...)] # at least 1D cdef ndarray[int, (:, : ,...)] # at least 3D cdef ndarray[int, (3, 5, ...)] # at least 2D, but required shape in first two dims. What do you think? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Aug 8 06:23:06 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Aug 2008 21:23:06 -0700 Subject: [Cython] about to buffer syntax In-Reply-To: References: Message-ID: <13927C8B-8F4D-41F3-99A6-0A02D777F73F@math.washington.edu> On Aug 7, 2008, at 8:49 AM, Lisandro Dalcin wrote: > I'm still convinced that Fortran 90 array declarators could provide a > more powerfull syntax for buffers, For this, the dimensions should be > specified with a tuple, examples below: > > cdef ndarray[int, ()] a # a scalar > > cdef ndarray[int, (3,)] # a1D array with 3 items > > cdef ndarray[int, (:,)] # a 1D array of any size > > cdef ndarray[int (2,3)] # 2D array with 2 rows and 3 columns > > cdef ndarray[int (:,3)] # 2D array with any number of rows and 3 > columns > > cdef ndarray[int (:,:)] # 2D array with any number of rows and > columns > > cdef ndarray[float, (3,3,:)] # 3D array, a 'sequence' of 3x3 array > > > We could even use Ellipsis '...' in a similar sense numpy uses them > > cdef ndarray[int, (...)] # any number of dimension > > cdef ndarray[int, (:, ...)] # at least 1D > > cdef ndarray[int, (:, : ,...)] # at least 3D > > cdef ndarray[int, (3, 5, ...)] # at least 2D, but required shape in > first two dims. > > > What do you think? Right now all we're specifying is the dimension and type (certainly the most important), for which this syntax seems overkill and obscure. Potentially, if we want to support more in the future, we could look at using this notation which would be backwards compatible. - Robert From dagss at student.matnat.uio.no Sat Aug 9 21:03:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 09 Aug 2008 21:03:00 +0200 Subject: [Cython] about to buffer syntax Message-ID: <3301160609.1914700@smtp.netcom.no> My initial reaction is that I like it! However it might not be needed... It would have to be (:n, :) rather than (n, :) for consistency. The thing is, you don't really need the shape in the syntax currently, only ndim. (One can use an assert statement if one wants the check, there doesn't appear to be any advantages to knowing the shape compile-time, unless perhaps if it is very small.) however with this perhaps support for specifying a starting point (index start point, subtracted on all lookups) could be added, that would be potentially very useful but also tend to "add cruft"...add something that is not there in either Python or C... Time and users will have to tell! But I enjoyed such a fresh idea in this discussion. I think one could make the ndim keyword mandatory at first, to delay the decision about which positional argument should be 2nd. Dag Sverre Seljebotn -----Original Message----- From: "Lisandro Dalcin" Date: Thursday, Aug 7, 2008 5:49 pm Subject: [Cython] about to buffer syntax To: Cython-dev Reply-To: cython-dev at codespeak.net I'm still convinced that Fortran 90 array declarators could provide a >more powerfull syntax for buffers, For this, the dimensions should be >specified with a tuple, examples below: > >cdef ndarray[int, ()] a # a scalar > >cdef ndarray[int, (3,)] # a1D array with 3 items > >cdef ndarray[int, (:,)] # a 1D array of any size > >cdef ndarray[int (2,3)] # 2D array with 2 rows and 3 columns > >cdef ndarray[int (:,3)] # 2D array with any number of rows and 3 columns > >cdef ndarray[int (:,:)] # 2D array with any number of rows and columns > >cdef ndarray[float, (3,3,:)] # 3D array, a 'sequence' of 3x3 array > > >We could even use Ellipsis '...' in a similar sense numpy uses them > >cdef ndarray[int, (...)] # any number of dimension > >cdef ndarray[int, (:, ...)] # at least 1D > >cdef ndarray[int, (:, : ,...)] # at least 3D > >cdef ndarray[int, (3, 5, ...)] # at least 2D, but required shape in >first two dims. > > >What do you think? > > >-- >Lisandro Dalc?n >--------------- >Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >PTLC - G?emes 3450, (3000) Santa Fe, Argentina >Tel/Fax: +54-(0)342-451.1594 >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Mon Aug 11 17:42:55 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 Aug 2008 12:42:55 -0300 Subject: [Cython] about to buffer syntax In-Reply-To: <3301160609.1914700@smtp.netcom.no> References: <3301160609.1914700@smtp.netcom.no> Message-ID: 2008/8/9 Dag Sverre Seljebotn : > My initial reaction is that I like it! However it might not be needed... > > It would have to be (:n, :) rather than (n, :) for consistency. > > The thing is, you don't really need the shape in the syntax currently, only ndim. (One can use an assert statement if one wants the check, there doesn't appear to be any advantages to knowing the shape compile-time, unless perhaps if it is very small.) > > however with this perhaps support for specifying a starting point (index start point, subtracted on all lookups) could be added, that would be potentially very useful but also tend to "add cruft"...add something that is not there in either Python or C... > Time and users will have to tell! But I enjoyed such a fresh idea in this discussion. Well, I believe that people with some background in Fortran 90 will definitely like and support my proposal. > I think one could make the ndim keyword mandatory at first, to delay the decision about which positional argument should be 2nd. Well, I believe that is a good approach. IMHO, declaring cdef ndarray[int, 3] tmp is not really clear that '3' means a 3D array. The intention of my proposal is just to provide a syntax that is less ambiguous and have room for more facilities (shape checking, negative starts, require shape at least xD, etc.) > -----Original Message----- > From: "Lisandro Dalcin" > Date: Thursday, Aug 7, 2008 5:49 pm > Subject: [Cython] about to buffer syntax > To: Cython-dev Reply-To: cython-dev at codespeak.net > > I'm still convinced that Fortran 90 array declarators could provide a >>more powerfull syntax for buffers, For this, the dimensions should be >>specified with a tuple, examples below: >> >>cdef ndarray[int, ()] a # a scalar >> >>cdef ndarray[int, (3,)] # a1D array with 3 items >> >>cdef ndarray[int, (:,)] # a 1D array of any size >> >>cdef ndarray[int (2,3)] # 2D array with 2 rows and 3 columns >> >>cdef ndarray[int (:,3)] # 2D array with any number of rows and 3 columns >> >>cdef ndarray[int (:,:)] # 2D array with any number of rows and columns >> >>cdef ndarray[float, (3,3,:)] # 3D array, a 'sequence' of 3x3 array >> >> >>We could even use Ellipsis '...' in a similar sense numpy uses them >> >>cdef ndarray[int, (...)] # any number of dimension >> >>cdef ndarray[int, (:, ...)] # at least 1D >> >>cdef ndarray[int, (:, : ,...)] # at least 3D >> >>cdef ndarray[int, (3, 5, ...)] # at least 2D, but required shape in >>first two dims. >> >> >>What do you think? >> >> >>-- >>Lisandro Dalc?n >>--------------- >>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >>PTLC - G?emes 3450, (3000) Santa Fe, Argentina >>Tel/Fax: +54-(0)342-451.1594 >>_______________________________________________ >>Cython-dev mailing list >>Cython-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/cython-dev >> > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From simon at arrowtheory.com Tue Aug 12 17:04:09 2008 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 12 Aug 2008 11:04:09 -0400 Subject: [Cython] setup.py example Message-ID: <20080812110409.1570c23b.simon@arrowtheory.com> Does someone a simple cython setup.py example ? I've been hunting around for one without any luck. thanks, Simon. From jim-crow at rambler.ru Tue Aug 12 17:18:07 2008 From: jim-crow at rambler.ru (Anatoly A. Kazantsev) Date: Tue, 12 Aug 2008 22:18:07 +0700 Subject: [Cython] setup.py example In-Reply-To: <20080812110409.1570c23b.simon@arrowtheory.com> References: <20080812110409.1570c23b.simon@arrowtheory.com> Message-ID: <20080812221807.80596efd.jim-crow@rambler.ru> On Tue, 12 Aug 2008 11:04:09 -0400 Simon Burton wrote: > > Does someone a simple cython setup.py example ? I've been hunting around for one without any luck. > > thanks, > > Simon. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev See my project at http://pypi.python.org/pypi/PyHurd/ -- Anatoly A. Kazantsev Protect your digital freedom and privacy, eliminate DRM, learn more at http://www.defectivebydesign.org/what_is_drm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080812/b16f32e0/attachment.pgp From simon at arrowtheory.com Tue Aug 12 17:53:43 2008 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 12 Aug 2008 11:53:43 -0400 Subject: [Cython] cython regression ? Message-ID: <20080812115343.ad19b567.simon@arrowtheory.com> $ cython foo.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef class Foo: public unsigned int zap ^ ------------------------------------------------------------ foo.pyx:5:11: Syntax error in simple statement list $ cython --version Cython version 0.9.8 Or did I break my cython install ? Simon. From robertwb at math.washington.edu Tue Aug 12 18:45:43 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 12 Aug 2008 09:45:43 -0700 Subject: [Cython] setup.py example In-Reply-To: <20080812221807.80596efd.jim-crow@rambler.ru> References: <20080812110409.1570c23b.simon@arrowtheory.com> <20080812221807.80596efd.jim-crow@rambler.ru> Message-ID: There's also one in the /Demos directory of the Cython install. On Aug 12, 2008, at 8:18 AM, Anatoly A. Kazantsev wrote: > On Tue, 12 Aug 2008 11:04:09 -0400 > Simon Burton wrote: > >> >> Does someone a simple cython setup.py example ? I've been hunting >> around for one without any luck. >> >> thanks, >> >> Simon. >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > See my project at http://pypi.python.org/pypi/PyHurd/ > > -- > Anatoly A. Kazantsev > > Protect your digital freedom and privacy, eliminate DRM, learn more at > http://www.defectivebydesign.org/what_is_drm > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080812/22d50c3e/attachment.pgp From dalcinl at gmail.com Tue Aug 12 19:44:47 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 12 Aug 2008 14:44:47 -0300 Subject: [Cython] py3k patch, changes in PyUnicode API Message-ID: I've just updated py3k following the trunk: PyUnicode_AsString[AndSize] are no longer available (well, we can still access it prefixing with _PyXXX, but this is marked as deprecated). So we are going to need the attached patch. Note however that I've not tested if the %U format specifier is available in latest oficial beta release. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: pyx_chk_kw_str.diff Type: text/x-patch Size: 734 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080812/e2f8b7d0/attachment.bin From stefan_ml at behnel.de Tue Aug 12 21:25:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 12 Aug 2008 21:25:36 +0200 Subject: [Cython] py3k patch, changes in PyUnicode API In-Reply-To: References: Message-ID: <48A1E3B0.4090406@behnel.de> Hi, thanks for the patch. Lisandro Dalcin wrote: > PyUnicode_AsString[AndSize] are no longer available (well, we can > still access it prefixing with _PyXXX, but this is marked as > deprecated). So we are going to need the attached patch. Note however > that I've not tested if the %U format specifier is available in latest > oficial beta release. Works for me in Py3b2. Applied to cython-devel. Stefan From robertwb at math.washington.edu Tue Aug 12 21:28:40 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 12 Aug 2008 12:28:40 -0700 Subject: [Cython] cython regression ? In-Reply-To: <20080812115343.ad19b567.simon@arrowtheory.com> References: <20080812115343.ad19b567.simon@arrowtheory.com> Message-ID: On Aug 12, 2008, at 8:53 AM, Simon Burton wrote: > $ cython foo.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > > > cdef class Foo: > > public unsigned int zap > ^ > ------------------------------------------------------------ > > foo.pyx:5:11: Syntax error in simple statement list > > $ cython --version > Cython version 0.9.8 > > > Or did I break my cython install ? Don't you have to write cdef class Foo: cdef public unsigned int zap ? - Robert From simon at arrowtheory.com Tue Aug 12 21:35:42 2008 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 12 Aug 2008 15:35:42 -0400 Subject: [Cython] cython regression ? In-Reply-To: References: <20080812115343.ad19b567.simon@arrowtheory.com> Message-ID: <20080812153542.458ad432.simon@arrowtheory.com> On Tue, 12 Aug 2008 12:28:40 -0700 Robert Bradshaw wrote: > On Aug 12, 2008, at 8:53 AM, Simon Burton wrote: > > > $ cython foo.pyx > > > > Error converting Pyrex file to C: > > ------------------------------------------------------------ > > ... > > > > > > cdef class Foo: > > > > public unsigned int zap > > ^ > > ------------------------------------------------------------ > > > > foo.pyx:5:11: Syntax error in simple statement list > > > > $ cython --version > > Cython version 0.9.8 > > > > > > Or did I break my cython install ? > > Don't you have to write > > cdef class Foo: > cdef public unsigned int zap > > ? > > - Robert Yes apparently so. However, this was working in Cython version 0.9.6.12 Simon. From stefan_ml at behnel.de Tue Aug 12 21:43:49 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 12 Aug 2008 21:43:49 +0200 Subject: [Cython] cython regression ? In-Reply-To: <20080812153542.458ad432.simon@arrowtheory.com> References: <20080812115343.ad19b567.simon@arrowtheory.com> <20080812153542.458ad432.simon@arrowtheory.com> Message-ID: <48A1E7F5.7030601@behnel.de> Hi, Simon Burton wrote: > On Tue, 12 Aug 2008 12:28:40 -0700 > Robert Bradshaw wrote: > >> On Aug 12, 2008, at 8:53 AM, Simon Burton wrote: >>> $ cython foo.pyx >>> Error converting Pyrex file to C: >>> cdef class Foo: >>> public unsigned int zap >>> foo.pyx:5:11: Syntax error in simple statement list >>> >> Don't you have to write >> >> cdef class Foo: >> cdef public unsigned int zap > > Yes apparently so. > However, this was working in Cython version 0.9.6.12 Then I'm happy we fixed that bug already. :) Stefan From robertwb at math.washington.edu Wed Aug 13 06:20:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 12 Aug 2008 21:20:31 -0700 Subject: [Cython] SciPy 2008, release? Message-ID: <6A21822E-9A39-4A37-B846-97C65604CB2C@math.washington.edu> I am giving a tutorial at the upcoming SciPy conference http:// conference.scipy.org/ and would like to have a new version of Cython out by then (specifically, the audience will be very interested in fast NumPy array access). On the other hand, due to the extensive amount of (good) changes in the development branch, we should be extra careful with this release. I am currently merging Dag's branch (declared stable) into the repository and will try and fix anything that comes up. If all goes well, I propose we call a feature freeze and release an official beta for people to try out. - Robert From alexmdac at gmail.com Wed Aug 13 11:25:39 2008 From: alexmdac at gmail.com (Alex Mendes da Costa) Date: Wed, 13 Aug 2008 02:25:39 -0700 Subject: [Cython] cython-generated class doesn't work when used as an iterator In-Reply-To: <2eeb88480808130217i3f358ab1p94ed42b8dc156427@mail.gmail.com> References: <2eeb88480808130217i3f358ab1p94ed42b8dc156427@mail.gmail.com> Message-ID: <2eeb88480808130225m6902f9acu5055cec7275c335c@mail.gmail.com> Hi, I'm using cython to wrap DJB's primegen library ( http://cr.yp.to/primegen.html). Here's my pyx file: cdef extern from "primegen.h": ctypedef unsigned long uint32 ctypedef unsigned long long uint64 ctypedef struct primegen: uint32 buf[16][2048] uint64 p[512] int num int pos uint64 base uint64 L void primegen_init(primegen* pg) uint64 primegen_next(primegen* pg) uint64 primegen_peek(primegen* pg) uint64 primegen_count(primegen* pg, uint64 to) void primegen_skipto(primegen* pg, uint64 to) cdef class PrimeGen: cdef primegen pg def __new__(self): primegen_init(&self.pg) def next(self): return primegen_next(&self.pg) def peek(self): return primegen_peek(&self.pg) def count(self, uint64 to): return primegen_count(&self.pg, to) def skipto(self, uint64 to): primegen_skipto(&self.pg, to) Notice that the class has a method called "next". This means that it should be usable as an iterator. Here's what happens if I try to do that: Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from primegen import PrimeGen >>> class C: ... def __iter__(self): ... return PrimeGen() ... >>> for i in C(): ... print i ... Traceback (most recent call last): File "", line 1, in TypeError: __iter__ returned non-iterator of type 'primegen.PrimeGen' How come this doesn't work? The generated class definitely has the "next" method required by the iterator protocol. >>> p = PrimeGen() >>> dir(p) ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'count', 'next', 'peek', 'skipto'] Thanks, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080813/87f816de/attachment-0001.htm From robertwb at math.washington.edu Wed Aug 13 11:31:24 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 02:31:24 -0700 Subject: [Cython] cython-generated class doesn't work when used as an iterator In-Reply-To: <2eeb88480808130225m6902f9acu5055cec7275c335c@mail.gmail.com> References: <2eeb88480808130217i3f358ab1p94ed42b8dc156427@mail.gmail.com> <2eeb88480808130225m6902f9acu5055cec7275c335c@mail.gmail.com> Message-ID: <97AF29BF-F7EA-4527-A017-5AD6429D0403@math.washington.edu> You actually need to provide a __next__ method. See http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/ Manual/special_methods.html - Robert On Aug 13, 2008, at 2:25 AM, Alex Mendes da Costa wrote: > Hi, > > I'm using cython to wrap DJB's primegen library (http://cr.yp.to/ > primegen.html). > > Here's my pyx file: > > cdef extern from "primegen.h": > ctypedef unsigned long uint32 > ctypedef unsigned long long uint64 > > ctypedef struct primegen: > uint32 buf[16][2048] > uint64 p[512] > int num > int pos > uint64 base > uint64 L > > void primegen_init(primegen* pg) > uint64 primegen_next(primegen* pg) > uint64 primegen_peek(primegen* pg) > uint64 primegen_count(primegen* pg, uint64 to) > void primegen_skipto(primegen* pg, uint64 to) > > cdef class PrimeGen: > cdef primegen pg > > def __new__(self): > primegen_init(&self.pg) > > def next(self): > return primegen_next(&self.pg) > > def peek(self): > return primegen_peek(&self.pg) > > def count(self, uint64 to): > return primegen_count(&self.pg, to) > > def skipto(self, uint64 to): > primegen_skipto(&self.pg, to) > > Notice that the class has a method called "next". This means that > it should be usable as an iterator. Here's what happens if I try > to do that: > > Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> from primegen import PrimeGen > >>> class C: > ... def __iter__(self): > ... return PrimeGen() > ... > >>> for i in C(): > ... print i > ... > Traceback (most recent call last): > File "", line 1, in > TypeError: __iter__ returned non-iterator of type 'primegen.PrimeGen' > > How come this doesn't work? The generated class definitely has the > "next" method required by the iterator protocol. > > >>> p = PrimeGen() > >>> dir(p) > ['__class__', '__delattr__', '__doc__', '__getattribute__', > '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', > '__repr__', '__setattr__', '__str__', 'count', 'next', 'peek', > 'skipto'] > > Thanks, > Alex > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From wstein at gmail.com Wed Aug 13 11:44:36 2008 From: wstein at gmail.com (William Stein) Date: Wed, 13 Aug 2008 02:44:36 -0700 Subject: [Cython] cython-generated class doesn't work when used as an iterator In-Reply-To: <2eeb88480808130225m6902f9acu5055cec7275c335c@mail.gmail.com> References: <2eeb88480808130217i3f358ab1p94ed42b8dc156427@mail.gmail.com> <2eeb88480808130225m6902f9acu5055cec7275c335c@mail.gmail.com> Message-ID: <85e81ba30808130244v77bae3afx7bfab8e8b9dc484b@mail.gmail.com> On Wed, Aug 13, 2008 at 2:25 AM, Alex Mendes da Costa wrote: > Hi, > > I'm using cython to wrap DJB's primegen library > (http://cr.yp.to/primegen.html). Are you going to contribute this code and the wrapper to Sage (http://sagemath.org)? I've wanted to add primegen to Sage for a long time -- in fact, I asked DJB about its license on a panel discussion and in his response he released primegen under a public domain license. -- William > > Here's my pyx file: > > cdef extern from "primegen.h": > ctypedef unsigned long uint32 > ctypedef unsigned long long uint64 > > ctypedef struct primegen: > uint32 buf[16][2048] > uint64 p[512] > int num > int pos > uint64 base > uint64 L > > void primegen_init(primegen* pg) > uint64 primegen_next(primegen* pg) > uint64 primegen_peek(primegen* pg) > uint64 primegen_count(primegen* pg, uint64 to) > void primegen_skipto(primegen* pg, uint64 to) > > cdef class PrimeGen: > cdef primegen pg > > def __new__(self): > primegen_init(&self.pg) > > def next(self): > return primegen_next(&self.pg) > > def peek(self): > return primegen_peek(&self.pg) > > def count(self, uint64 to): > return primegen_count(&self.pg, to) > > def skipto(self, uint64 to): > primegen_skipto(&self.pg, to) > > Notice that the class has a method called "next". This means that it should > be usable as an iterator. Here's what happens if I try to do that: > > Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> from primegen import PrimeGen >>>> class C: > ... def __iter__(self): > ... return PrimeGen() > ... >>>> for i in C(): > ... print i > ... > Traceback (most recent call last): > File "", line 1, in > TypeError: __iter__ returned non-iterator of type 'primegen.PrimeGen' > > How come this doesn't work? The generated class definitely has the "next" > method required by the iterator protocol. > >>>> p = PrimeGen() >>>> dir(p) > ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', > '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', > '__setattr__', '__str__', 'count', 'next', 'peek', 'skipto'] > > Thanks, > Alex > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From alexmdac at gmail.com Wed Aug 13 12:03:06 2008 From: alexmdac at gmail.com (Alex Mendes da Costa) Date: Wed, 13 Aug 2008 03:03:06 -0700 Subject: [Cython] cython-generated class doesn't work when used as an iterator In-Reply-To: <85e81ba30808130244v77bae3afx7bfab8e8b9dc484b@mail.gmail.com> References: <2eeb88480808130217i3f358ab1p94ed42b8dc156427@mail.gmail.com> <2eeb88480808130225m6902f9acu5055cec7275c335c@mail.gmail.com> <85e81ba30808130244v77bae3afx7bfab8e8b9dc484b@mail.gmail.com> Message-ID: <2eeb88480808130303q7aaf4098rab55630647d34c37@mail.gmail.com> Yes, you can incorporate this code into Sage if you like. Alex On Wed, Aug 13, 2008 at 2:44 AM, William Stein wrote: > On Wed, Aug 13, 2008 at 2:25 AM, Alex Mendes da Costa > wrote: > > Hi, > > > > I'm using cython to wrap DJB's primegen library > > (http://cr.yp.to/primegen.html). > > Are you going to contribute this code and the wrapper > to Sage (http://sagemath.org)? I've wanted to add > primegen to Sage for a long time -- in fact, I asked > DJB about its license on a panel discussion and > in his response he released primegen under a public > domain license. > > -- William > > > > > Here's my pyx file: > > > > cdef extern from "primegen.h": > > ctypedef unsigned long uint32 > > ctypedef unsigned long long uint64 > > > > ctypedef struct primegen: > > uint32 buf[16][2048] > > uint64 p[512] > > int num > > int pos > > uint64 base > > uint64 L > > > > void primegen_init(primegen* pg) > > uint64 primegen_next(primegen* pg) > > uint64 primegen_peek(primegen* pg) > > uint64 primegen_count(primegen* pg, uint64 to) > > void primegen_skipto(primegen* pg, uint64 to) > > > > cdef class PrimeGen: > > cdef primegen pg > > > > def __new__(self): > > primegen_init(&self.pg) > > > > def next(self): > > return primegen_next(&self.pg) > > > > def peek(self): > > return primegen_peek(&self.pg) > > > > def count(self, uint64 to): > > return primegen_count(&self.pg, to) > > > > def skipto(self, uint64 to): > > primegen_skipto(&self.pg, to) > > > > Notice that the class has a method called "next". This means that it > should > > be usable as an iterator. Here's what happens if I try to do that: > > > > Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) > > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > > Type "help", "copyright", "credits" or "license" for more information. > >>>> from primegen import PrimeGen > >>>> class C: > > ... def __iter__(self): > > ... return PrimeGen() > > ... > >>>> for i in C(): > > ... print i > > ... > > Traceback (most recent call last): > > File "", line 1, in > > TypeError: __iter__ returned non-iterator of type 'primegen.PrimeGen' > > > > How come this doesn't work? The generated class definitely has the > "next" > > method required by the iterator protocol. > > > >>>> p = PrimeGen() > >>>> dir(p) > > ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', > > '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', > > '__setattr__', '__str__', 'count', 'next', 'peek', 'skipto'] > > > > Thanks, > > Alex > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > > > > > > > -- > William Stein > Associate Professor of Mathematics > University of Washington > http://wstein.org > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080813/cd743100/attachment.htm From robertwb at math.washington.edu Wed Aug 13 12:11:27 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 03:11:27 -0700 Subject: [Cython] Cython 0.9.8.1 beta Message-ID: Beta up at http://cython.org/Cython-0.9.8.1-beta1.tar.gz - Robert From ndbecker2 at gmail.com Wed Aug 13 17:35:25 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 13 Aug 2008 11:35:25 -0400 Subject: [Cython] Head fails tests Message-ID: I did hg pull -u python setup.py build sudo python setup.py install python runtests.py running doctests in wundram1 ... ====================================================================== ERROR: compiling and running ct_DEF ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 261, in run self.runCompileTest() File "runtests.py", line 159, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 248, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '38:19: Invalid compile-time expression' Bunch more errors, including: AttributeError: 'NoneType' object has no attribute 'groups' ValueError: invalid literal for int() with base 10: '0F' AssertionError: From dagss at student.matnat.uio.no Wed Aug 13 18:39:08 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 13 Aug 2008 18:39:08 +0200 Subject: [Cython] Head fails tests In-Reply-To: References: Message-ID: <48A30E2C.2080002@student.matnat.uio.no> Neal Becker wrote: > I did > hg pull -u > python setup.py build > sudo python setup.py install > python runtests.py > > running doctests in wundram1 ... > ====================================================================== > ERROR: compiling and running ct_DEF > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "runtests.py", line 261, in run > self.runCompileTest() > File "runtests.py", line 159, in runCompileTest > self.directory, self.expect_errors, self.annotate) > File "runtests.py", line 248, in compile > self.assertEquals(None, unexpected_error) > AssertionError: None != '38:19: Invalid compile-time expression' > > Bunch more errors, including: > AttributeError: 'NoneType' object has no attribute 'groups' > ValueError: invalid literal for int() with base 10: '0F' > AssertionError: Strange, I cannot reproduce...so this could be a scary one. Are you referring to the cython-devel repo? Can you do a fresh clone and try running only the wundram1 test? hg clone http://hg.cython.org/cython-devel cd cython-devel python runtests.py wundram1 Does the same thing happen then? What Python version (python -V) are you using? -- Dag Sverre From robertwb at math.washington.edu Wed Aug 13 18:39:19 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 09:39:19 -0700 Subject: [Cython] Head fails tests In-Reply-To: <48A30E2C.2080002@student.matnat.uio.no> References: <48A30E2C.2080002@student.matnat.uio.no> Message-ID: <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> FYI, the beta is based on http://hg.cython.org/cython-dagss/ (which should have everything from cython-devel). Try pulling from there. - Robert On Aug 13, 2008, at 9:39 AM, Dag Sverre Seljebotn wrote: > Neal Becker wrote: >> I did >> hg pull -u >> python setup.py build >> sudo python setup.py install >> python runtests.py >> >> running doctests in wundram1 ... >> ===================================================================== >> = >> ERROR: compiling and running ct_DEF >> --------------------------------------------------------------------- >> - >> Traceback (most recent call last): >> File "runtests.py", line 261, in run >> self.runCompileTest() >> File "runtests.py", line 159, in runCompileTest >> self.directory, self.expect_errors, self.annotate) >> File "runtests.py", line 248, in compile >> self.assertEquals(None, unexpected_error) >> AssertionError: None != '38:19: Invalid compile-time expression' >> >> Bunch more errors, including: >> AttributeError: 'NoneType' object has no attribute 'groups' >> ValueError: invalid literal for int() with base 10: '0F' >> AssertionError: > > Strange, I cannot reproduce...so this could be a scary one. > > Are you referring to the cython-devel repo? Can you do a fresh > clone and > try running only the wundram1 test? > > hg clone http://hg.cython.org/cython-devel > cd cython-devel > python runtests.py wundram1 > > Does the same thing happen then? What Python version (python -V) > are you > using? > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Wed Aug 13 18:49:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 13 Aug 2008 18:49:44 +0200 Subject: [Cython] Head fails tests In-Reply-To: <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> Message-ID: <48A310A8.4040907@student.matnat.uio.no> Robert Bradshaw wrote: > FYI, the beta is based on http://hg.cython.org/cython-dagss/ (which > should have everything from cython-devel). Try pulling from there. > On that note..: As cython-dagss has everything in devel and is the basis of the beta, can't we just push it back to -devel and have -devel be the branch that will mature to the release? I could then commit anything "stabilizing" into -devel and anything "destabilizing" to -dagss... -- Dag Sverre From robertwb at math.washington.edu Wed Aug 13 18:49:24 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 09:49:24 -0700 Subject: [Cython] Head fails tests In-Reply-To: <48A310A8.4040907@student.matnat.uio.no> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> Message-ID: <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> On Aug 13, 2008, at 9:49 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> FYI, the beta is based on http://hg.cython.org/cython-dagss/ (which >> should have everything from cython-devel). Try pulling from there. >> > > On that note..: As cython-dagss has everything in devel and is the > basis > of the beta, can't we just push it back to -devel and have -devel > be the > branch that will mature to the release? > > I could then commit anything "stabilizing" into -devel and anything > "destabilizing" to -dagss... Yes. My only concern is that Stefan's lxml was having trouble compiling with the dagss branch (not sure if this is still the case) and I didn't want that to be a show-stopper for Stefan. If this is not the case anymore, then I would like to push -dagss into -devel. - Robert From robertwb at math.washington.edu Wed Aug 13 18:49:55 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 09:49:55 -0700 Subject: [Cython] Head fails tests In-Reply-To: References: Message-ID: <15403B89-DF71-49BE-9C18-296B18B4C4C2@math.washington.edu> On Aug 13, 2008, at 8:35 AM, Neal Becker wrote: > I did > hg pull -u > python setup.py build > sudo python setup.py install > python runtests.py > > running doctests in wundram1 ... > ====================================================================== > ERROR: compiling and running ct_DEF > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "runtests.py", line 261, in run > self.runCompileTest() > File "runtests.py", line 159, in runCompileTest > self.directory, self.expect_errors, self.annotate) > File "runtests.py", line 248, in compile > self.assertEquals(None, unexpected_error) > AssertionError: None != '38:19: Invalid compile-time expression' > > Bunch more errors, including: > AttributeError: 'NoneType' object has no attribute 'groups' > ValueError: invalid literal for int() with base 10: '0F' > AssertionError: Thanks for the bug reports, I think I fixed these. - Robert From simon at arrowtheory.com Wed Aug 13 19:01:23 2008 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 13 Aug 2008 13:01:23 -0400 Subject: [Cython] cython regression ? In-Reply-To: <48A1E7F5.7030601@behnel.de> References: <20080812115343.ad19b567.simon@arrowtheory.com> <20080812153542.458ad432.simon@arrowtheory.com> <48A1E7F5.7030601@behnel.de> Message-ID: <20080813130123.24810185.simon@arrowtheory.com> On Tue, 12 Aug 2008 21:43:49 +0200 Stefan Behnel wrote: > > Then I'm happy we fixed that bug already. :) > > Stefan OK, but i'm finding some more regressions in my code. Here is a buffer class that compiles fine: cdef class Buffer: cdef public char * data def __cinit__(self): self.data = NULL But when i separate the definition into a pxd file i get this: Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef class Buffer: def __cinit__(self): self.data = NULL ^ ------------------------------------------------------------ 8:20: Cannot convert 'void *' to Python object Here is the pxd file: cdef class Buffer: cdef public char *data If I do an strace, it seems cython does not find the pxd: stat64(".../zap/zap.foo.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) stat64(".../zap/zap/foo.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) stat64(".../zap/zap/foo/__init__.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) The current directory is zap, and this is the package in which foo lives. So, the file is at .../zap/foo.pxd I tried running cython from parent directory, and also from setup.py, and also build_ext --inplace. lost, Simon. From simon at arrowtheory.com Wed Aug 13 19:53:13 2008 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 13 Aug 2008 13:53:13 -0400 Subject: [Cython] cython regression ? In-Reply-To: <20080813130123.24810185.simon@arrowtheory.com> References: <20080812115343.ad19b567.simon@arrowtheory.com> <20080812153542.458ad432.simon@arrowtheory.com> <48A1E7F5.7030601@behnel.de> <20080813130123.24810185.simon@arrowtheory.com> Message-ID: <20080813135313.0fbd40ab.simon@arrowtheory.com> On Wed, 13 Aug 2008 13:01:23 -0400 Simon Burton wrote: > > If I do an strace, it seems cython does not find the pxd: > > stat64(".../zap/zap.foo.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) > stat64(".../zap/zap/foo.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) > stat64(".../zap/zap/foo/__init__.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) It seems to be the presence of .../zap/__init__.py that triggers this behaviour. Simon. From dagss at student.matnat.uio.no Wed Aug 13 21:18:38 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 13 Aug 2008 21:18:38 +0200 Subject: [Cython] Warning about another bug In-Reply-To: <22A45948-FD2D-4060-B731-1AC9FF6F8DD7@math.washington.edu> References: <489965B0.3040001@student.matnat.uio.no> <4899D2DE.1090104@student.matnat.uio.no> <22A45948-FD2D-4060-B731-1AC9FF6F8DD7@math.washington.edu> Message-ID: <48A3338E.8030800@student.matnat.uio.no> Robert Bradshaw wrote: > On Aug 6, 2008, at 9:35 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On Aug 6, 2008, at 1:49 AM, Dag Sverre Seljebotn wrote: >>> >>>> Obviously my past efforts weren't good enough: >>>> >>>> DEF MYLEN = 3 >>>> c = sizeof(MyStruct[MYLEN]) >>>> >>>> So it seems that the parser can never discern the current syntax >>>> (MYLEN >>>> could have been a typedef and MyStruct and extension type, and >>>> then it >>>> would have been buffer syntax). >>>> >>>> So if keeping the current syntax, I seem to have no choice but to >>>> create >>>> a TrailingBracketTypeNode and delay the decision until type >>>> analysis. >>>> This takes some work, and adds complexity, so I want to be sure >>>> that it >>>> is needed first. >>> I'm not understanding why we need to introduce a new node here, why >>> not let the sizeof operator be OK with index nodes? >> Yes, if you can guarantee me that the sizeof operator is the only >> place >> this happens, and that the sizeof name cannot ever be overriden and so >> on, I could do that. >> >> The thing is, I've been bitten so many times by wrong assumptions >> by now >> that I'd rather use a new rule controlling the thing: If [] is used >> on a >> pyobject, then it is buffer, on a native type, then array. But then I >> need to defer the resolution of what the [] means until type analysis >> time. > > I think this makes sense. To backtrack to this discussion: - Apparently, sizeof will always be sizeof -- it is resolved in the parser by string comparison and p_sizeof is called (it is considered an operator, like in C). - Also, a flag nonempty ("context requires that some variable name is present") is passed to the type parsing code, which is set to True in the kind of situations Lisandro Dalcin reported. So the fix was really easy, in the parser only, and I didn't attempt the much more complicated solution outlined earlier. (BTW, an assumption made by this is that we don't allow cdef external_func(object[int]) I.e. if exporting a function using a buffer, you must do pxd file: cdef myfunc(object) pyx file: cdef myfunc(object[int] param): ... This makes kind of sense as the caller knows nothing about the buffer access anyway, it's an implementation shorthand. If it is too inconvenient we could work around it later but the changes are quite extensive and I'd rather do more useful stuff. BTW this was already explictly disallowed in existing buffer code, so only the error message changes by this.) -- Dag Sverre From robertwb at math.washington.edu Wed Aug 13 21:14:00 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 12:14:00 -0700 (PDT) Subject: [Cython] cython regression ? In-Reply-To: <20080813135313.0fbd40ab.simon@arrowtheory.com> References: <20080812115343.ad19b567.simon@arrowtheory.com> <20080812153542.458ad432.simon@arrowtheory.com> <48A1E7F5.7030601@behnel.de> <20080813130123.24810185.simon@arrowtheory.com> <20080813135313.0fbd40ab.simon@arrowtheory.com> Message-ID: On Wed, 13 Aug 2008, Simon Burton wrote: > On Wed, 13 Aug 2008 13:01:23 -0400 > Simon Burton wrote: > >> >> If I do an strace, it seems cython does not find the pxd: >> >> stat64(".../zap/zap.foo.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) >> stat64(".../zap/zap/foo.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) >> stat64(".../zap/zap/foo/__init__.pxd", 0xbfc81ee8) = -1 ENOENT (No such file or directory) > > It seems to be the presence of .../zap/__init__.py that triggers this behaviour. > > Simon. Just to confirm, your directory structure is zap/ __init__.py foo.pyx foo.pxd Or are you doing something else? - Robert From robertwb at math.washington.edu Wed Aug 13 21:17:35 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 12:17:35 -0700 (PDT) Subject: [Cython] Warning about another bug In-Reply-To: <48A3338E.8030800@student.matnat.uio.no> References: <489965B0.3040001@student.matnat.uio.no> <4899D2DE.1090104@student.matnat.uio.no> <22A45948-FD2D-4060-B731-1AC9FF6F8DD7@math.washington.edu> <48A3338E.8030800@student.matnat.uio.no> Message-ID: On Wed, 13 Aug 2008, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Aug 6, 2008, at 9:35 AM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> On Aug 6, 2008, at 1:49 AM, Dag Sverre Seljebotn wrote: >>>> >>>>> Obviously my past efforts weren't good enough: >>>>> >>>>> DEF MYLEN = 3 >>>>> c = sizeof(MyStruct[MYLEN]) >>>>> >>>>> So it seems that the parser can never discern the current syntax >>>>> (MYLEN >>>>> could have been a typedef and MyStruct and extension type, and >>>>> then it >>>>> would have been buffer syntax). >>>>> >>>>> So if keeping the current syntax, I seem to have no choice but to >>>>> create >>>>> a TrailingBracketTypeNode and delay the decision until type >>>>> analysis. >>>>> This takes some work, and adds complexity, so I want to be sure >>>>> that it >>>>> is needed first. >>>> I'm not understanding why we need to introduce a new node here, why >>>> not let the sizeof operator be OK with index nodes? >>> Yes, if you can guarantee me that the sizeof operator is the only >>> place >>> this happens, and that the sizeof name cannot ever be overriden and so >>> on, I could do that. >>> >>> The thing is, I've been bitten so many times by wrong assumptions >>> by now >>> that I'd rather use a new rule controlling the thing: If [] is used >>> on a >>> pyobject, then it is buffer, on a native type, then array. But then I >>> need to defer the resolution of what the [] means until type analysis >>> time. >> >> I think this makes sense. > > To backtrack to this discussion: > > - Apparently, sizeof will always be sizeof -- it is resolved in the > parser by string comparison and p_sizeof is called (it is considered an > operator, like in C). > > - Also, a flag nonempty ("context requires that some variable name is > present") is passed to the type parsing code, which is set to True in > the kind of situations Lisandro Dalcin reported. > > So the fix was really easy, in the parser only, and I didn't attempt the > much more complicated solution outlined earlier. Excelent. > (BTW, an assumption made by this is that we don't allow > > cdef external_func(object[int]) > > I.e. if exporting a function using a buffer, you must do > > pxd file: > cdef myfunc(object) > > pyx file: > cdef myfunc(object[int] param): ... > > This makes kind of sense as the caller knows nothing about the buffer > access anyway, it's an implementation shorthand. If it is too > inconvenient we could work around it later but the changes are quite > extensive and I'd rather do more useful stuff. BTW this was already > explictly disallowed in existing buffer code, so only the error message > changes by this.) I think this fine. - Robert From simon at arrowtheory.com Wed Aug 13 22:12:09 2008 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 13 Aug 2008 16:12:09 -0400 Subject: [Cython] cython regression ? In-Reply-To: References: <20080812115343.ad19b567.simon@arrowtheory.com> <20080812153542.458ad432.simon@arrowtheory.com> <48A1E7F5.7030601@behnel.de> <20080813130123.24810185.simon@arrowtheory.com> <20080813135313.0fbd40ab.simon@arrowtheory.com> Message-ID: <20080813161209.231938cb.simon@arrowtheory.com> On Wed, 13 Aug 2008 12:14:00 -0700 (PDT) Robert Bradshaw wrote: > > Just to confirm, your directory structure is > > zap/ > __init__.py > foo.pyx > foo.pxd > > Or are you doing something else? > > - Robert That's correct. Simon. From phaustin at gmail.com Wed Aug 13 23:25:45 2008 From: phaustin at gmail.com (Phil Austin) Date: Wed, 13 Aug 2008 14:25:45 -0700 Subject: [Cython] setup.py example In-Reply-To: References: <20080812110409.1570c23b.simon@arrowtheory.com> <20080812221807.80596efd.jim-crow@rambler.ru> Message-ID: <48A35159.1000905@gmail.com> Robert Bradshaw wrote: > There's also one in the /Demos directory of the Cython install. Does anyone have a setup.py that shows how to pass flags to cython? I'd like to specify "cython --convert-range" thanks, Phil From robertwb at math.washington.edu Thu Aug 14 06:22:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 21:22:42 -0700 Subject: [Cython] setup.py example In-Reply-To: <48A35159.1000905@gmail.com> References: <20080812110409.1570c23b.simon@arrowtheory.com> <20080812221807.80596efd.jim-crow@rambler.ru> <48A35159.1000905@gmail.com> Message-ID: On Aug 13, 2008, at 2:25 PM, Phil Austin wrote: > Robert Bradshaw wrote: >> There's also one in the /Demos directory of the Cython install. > > Does anyone have a setup.py that shows how to pass flags to cython? I'm not sure of a way to do that. > I'd like to specify "cython --convert-range" Fortunately for you, this is now enabled by default: http:// hg.cython.org/cython/file/4e4808271bdf/Cython/Compiler/Options.py - Robert From stefan_ml at behnel.de Thu Aug 14 07:38:12 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 14 Aug 2008 07:38:12 +0200 Subject: [Cython] Head fails tests In-Reply-To: <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> Message-ID: <48A3C4C4.1060104@behnel.de> Hi Robert, Robert Bradshaw wrote: > My only concern is that Stefan's lxml was having trouble > compiling with the dagss branch (not sure if this is still the case) > and I didn't want that to be a show-stopper for Stefan. If this is > not the case anymore, then I would like to push -dagss into -devel. Still having issues with it, but those were introduced by the new UnicodeType. It seems that it doesn't behave well enough yet. It's currently a subtype of BuiltinObjectType, but that induces problems with attribute access. Letting it inherit from PyObjectType didn't provide a quick fix either. I don't know when I will have time to look into this any further, sorry for breaking it. May take a couple of days, unless someone else gets down to it. If fixing it becomes too complicated, feel free to revert any of the unicode changes for now that you consider offensive. Stefan From robertwb at math.washington.edu Thu Aug 14 08:02:02 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 13 Aug 2008 23:02:02 -0700 Subject: [Cython] Head fails tests In-Reply-To: <48A3C4C4.1060104@behnel.de> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> Message-ID: <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> On Aug 13, 2008, at 10:38 PM, Stefan Behnel wrote: > Hi Robert, > > Robert Bradshaw wrote: >> My only concern is that Stefan's lxml was having trouble >> compiling with the dagss branch (not sure if this is still the case) >> and I didn't want that to be a show-stopper for Stefan. If this is >> not the case anymore, then I would like to push -dagss into -devel. > > Still having issues with it, but those were introduced by the new > UnicodeType. > It seems that it doesn't behave well enough yet. It's currently a > subtype of > BuiltinObjectType, but that induces problems with attribute access. > Letting it > inherit from PyObjectType didn't provide a quick fix either. > > I don't know when I will have time to look into this any further, > sorry for > breaking it. May take a couple of days, unless someone else gets > down to it. > If fixing it becomes too complicated, feel free to revert any of > the unicode > changes for now that you consider offensive. I haven't been able to get any failures on my end, and did fix some things related to this, so maybe I should try compiling lxml. Any tips, or do I just download and compile? - Robert From stefan_ml at behnel.de Thu Aug 14 08:34:47 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 14 Aug 2008 08:34:47 +0200 (CEST) Subject: [Cython] Head fails tests In-Reply-To: <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> Message-ID: <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Aug 13, 2008, at 10:38 PM, Stefan Behnel wrote: >> Still having issues with it, but those were introduced by the new >> UnicodeType. >> It seems that it doesn't behave well enough yet. It's currently a >> subtype of >> BuiltinObjectType, but that induces problems with attribute access. > > I haven't been able to get any failures on my end, and did fix some > things related to this I tried with the most recent -dagss branch and got an exception in ExprNodes.py, line 2215, saying that the object_type doesn't have a scope. That's because UnicodeType inherits from BuiltinPythonObject but does not set up a local class scope (as the normal builtins do). I think what we might try is to replace the UnicodeType class with the actual BuiltinPythonObject instance that represents the unicode type, i.e. the one that is created in Builtin.py. That sounds like a clean solution to me (well, if it works, that is...) > so maybe I should try compiling lxml. Any > tips, or do I just download and compile? If you can arrange it, that'd be nice. You'll need the -dev packages of libxml2 and libxslt installed, the rest should work out of the box. Stefan From robertwb at math.washington.edu Thu Aug 14 09:50:43 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 14 Aug 2008 00:50:43 -0700 Subject: [Cython] Head fails tests In-Reply-To: <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> On Aug 13, 2008, at 11:34 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On Aug 13, 2008, at 10:38 PM, Stefan Behnel wrote: >>> Still having issues with it, but those were introduced by the new >>> UnicodeType. >>> It seems that it doesn't behave well enough yet. It's currently a >>> subtype of >>> BuiltinObjectType, but that induces problems with attribute access. >> >> I haven't been able to get any failures on my end, and did fix some >> things related to this > > I tried with the most recent -dagss branch and got an exception in > ExprNodes.py, line 2215, saying that the object_type doesn't have a > scope. > That's because UnicodeType inherits from BuiltinPythonObject but > does not > set up a local class scope (as the normal builtins do). > > I think what we might try is to replace the UnicodeType class with the > actual BuiltinPythonObject instance that represents the unicode > type, i.e. > the one that is created in Builtin.py. That sounds like a clean > solution > to me (well, if it works, that is...) I'm really confused as to what the UnicodeType is suppose to represent. Is it a char* or a PyObject*? Currently it's both, which means that conversion doesn't happen either direction. >> so maybe I should try compiling lxml. Any >> tips, or do I just download and compile? > > If you can arrange it, that'd be nice. You'll need the -dev > packages of > libxml2 and libxslt installed, the rest should work out of the box. I'll try that then. - Robert From stefan_ml at behnel.de Thu Aug 14 11:08:09 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 14 Aug 2008 11:08:09 +0200 (CEST) Subject: [Cython] Head fails tests In-Reply-To: <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> Message-ID: <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Aug 13, 2008, at 11:34 PM, Stefan Behnel wrote: >> I think what we might try is to replace the UnicodeType class with the >> actual BuiltinPythonObject instance that represents the unicode >> type, i.e. >> the one that is created in Builtin.py. That sounds like a clean >> solution to me (well, if it works, that is...) > > I'm really confused as to what the UnicodeType is suppose to > represent. Is it a char* or a PyObject*? Currently it's both, which > means that conversion doesn't happen either direction. It was supposed to be more or less what I describe above. The idea is to let unicode literals have their actual Python type (i.e. PyUnicodeObject), which is not compatible with a char*. That will assure that we always use the right Python C-API operations on them instead of trying to handle them as a special char*. It's just that my current implementation is totally flawed and should have used the 'real' unicode builtin type instead, that we create in Builtin.py anyway. From a quick glance, this doesn't seem to be a quick change, though, as we currently only declare builtin types and keep no further (easy-to-find) reference to them, that we could retrieve in Symtab.py's add_string_const(). Stefan From greg.ewing at canterbury.ac.nz Thu Aug 14 11:12:01 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 14 Aug 2008 21:12:01 +1200 Subject: [Cython] Head fails tests In-Reply-To: <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48A3F6E1.8040803@canterbury.ac.nz> Stefan Behnel wrote: > From a quick glance, this doesn't seem to be a quick change, > though, as we currently only declare builtin types and keep no further > (easy-to-find) reference to them, that we could retrieve in Symtab.py's > add_string_const(). You could put some init code in Builtin.py that looks it up in the builtin scope and stashes it in a global somewhere. -- Greg From robertwb at math.washington.edu Thu Aug 14 11:16:04 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 14 Aug 2008 02:16:04 -0700 Subject: [Cython] Head fails tests In-Reply-To: <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Aug 14, 2008, at 2:08 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On Aug 13, 2008, at 11:34 PM, Stefan Behnel wrote: >>> I think what we might try is to replace the UnicodeType class >>> with the >>> actual BuiltinPythonObject instance that represents the unicode >>> type, i.e. >>> the one that is created in Builtin.py. That sounds like a clean >>> solution to me (well, if it works, that is...) >> >> I'm really confused as to what the UnicodeType is suppose to >> represent. Is it a char* or a PyObject*? Currently it's both, which >> means that conversion doesn't happen either direction. > > It was supposed to be more or less what I describe above. The idea > is to > let unicode literals have their actual Python type (i.e. > PyUnicodeObject), > which is not compatible with a char*. That will assure that we > always use > the right Python C-API operations on them instead of trying to > handle them > as a special char*. > > It's just that my current implementation is totally flawed and > should have > used the 'real' unicode builtin type instead, that we create in > Builtin.py > anyway. From a quick glance, this doesn't seem to be a quick change, > though, as we currently only declare builtin types and keep no further > (easy-to-find) reference to them, that we could retrieve in > Symtab.py's > add_string_const(). I changed it to use the buitin type, and am reverting back a bit to the old way of doing things. Unicode literals start out as python objects, so we don't need to worry about users doing C operations on them. Maybe we could sort this out a better solution later... - Robert From robertwb at math.washington.edu Thu Aug 14 11:16:57 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 14 Aug 2008 02:16:57 -0700 Subject: [Cython] Head fails tests In-Reply-To: <48A3F6E1.8040803@canterbury.ac.nz> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48A3F6E1.8040803@canterbury.ac.nz> Message-ID: <273B827D-3466-4CFE-8326-6DD64E41AEE2@math.washington.edu> On Aug 14, 2008, at 2:12 AM, Greg Ewing wrote: > Stefan Behnel wrote: >> From a quick glance, this doesn't seem to be a quick change, >> though, as we currently only declare builtin types and keep no >> further >> (easy-to-find) reference to them, that we could retrieve in >> Symtab.py's >> add_string_const(). > > You could put some init code in Builtin.py that looks > it up in the builtin scope and stashes it in a global > somewhere. The commonly used ones do get stashed as global variables in the Builtin module, the rest one would have to do a lookup to get. - Robert From robertwb at math.washington.edu Thu Aug 14 11:23:29 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 14 Aug 2008 02:23:29 -0700 Subject: [Cython] Head fails tests In-Reply-To: <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <9C31290D-C25C-42DB-91E6-CAB21910757E@math.washington.edu> On Aug 14, 2008, at 2:08 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On Aug 13, 2008, at 11:34 PM, Stefan Behnel wrote: >>> I think what we might try is to replace the UnicodeType class >>> with the >>> actual BuiltinPythonObject instance that represents the unicode >>> type, i.e. >>> the one that is created in Builtin.py. That sounds like a clean >>> solution to me (well, if it works, that is...) >> >> I'm really confused as to what the UnicodeType is suppose to >> represent. Is it a char* or a PyObject*? Currently it's both, which >> means that conversion doesn't happen either direction. > > It was supposed to be more or less what I describe above. The idea > is to > let unicode literals have their actual Python type (i.e. > PyUnicodeObject), > which is not compatible with a char*. That will assure that we > always use > the right Python C-API operations on them instead of trying to > handle them > as a special char*. > > It's just that my current implementation is totally flawed and > should have > used the 'real' unicode builtin type instead, that we create in > Builtin.py > anyway. From a quick glance, this doesn't seem to be a quick change, > though, as we currently only declare builtin types and keep no further > (easy-to-find) reference to them, that we could retrieve in > Symtab.py's > add_string_const(). In any case, this issue has been resolved for now. u"..." strings have type unicode, and lxml cythonizes fine. I've pushed to cython- devel. - Robert From robertwb at math.washington.edu Thu Aug 14 11:39:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 14 Aug 2008 02:39:07 -0700 Subject: [Cython] Cython 0.9.8.1 beta 2 In-Reply-To: References: Message-ID: <66DF5956-7160-4FC8-8742-F1E05035105F@math.washington.edu> See http://cython.org/Cython-0.9.8.1-beta2.tar.gz We are now working off of http://hg.cython.org/cython-devel - Robert From stefan_ml at behnel.de Thu Aug 14 13:24:17 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 14 Aug 2008 13:24:17 +0200 (CEST) Subject: [Cython] Head fails tests In-Reply-To: References: <48A30E2C.2080002@student.matnat.uio.no> <3430B348-22D5-4270-A97A-BE01E05673FD@math.washington.edu> <48A310A8.4040907@student.matnat.uio.no> <7F8651BC-6831-4B92-A32C-EB2CDA984610@math.washington.edu> <48A3C4C4.1060104@behnel.de> <6477D633-A271-4803-A939-6DEE8029E54E@math.washington.edu> <37311.213.61.181.86.1218695687.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <41F23709-9A4D-4F58-B273-1E5F7D9AB64E@math.washington.edu> <38686.213.61.181.86.1218704889.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <47501.213.61.181.86.1218713057.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw schrieb: > On Aug 14, 2008, at 2:08 AM, Stefan Behnel wrote: >> should have >> used the 'real' unicode builtin type instead, that we create in >> Builtin.py anyway. > > I changed it to use the buitin type, and am reverting back a bit to > the old way of doing things. Unicode literals start out as python > objects, so we don't need to worry about users doing C operations on > them. Ah, yes, I had forgotten about the global types in Builtin.py (and missed them when skipping over it). I think that fixes it. Stefan From dagss at student.matnat.uio.no Thu Aug 14 18:41:27 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 14 Aug 2008 18:41:27 +0200 (CEST) Subject: [Cython] Making sure annotations work Message-ID: <51277.193.157.229.67.1218732087.squirrel@webmail.uio.no> While reading Robert's latest changes it occured to me that as CCodeWriter has changed quite a bit, this may affect the AnnotationCCodeWriter and introduce bugs. Specifically, will annotation work correctly when insertion points are created? Could somebody have a look over my changes to the annotation writer (do a diff since the prev release) and see if they make sense, and if you think they are enough, and do some basic testing? I'm very removed from annotation writing (never used it and don't know exactly how it should work) so it would be better if someone else could step in. I am mentioning this specifically since (to my knowledge) there are no tests for annotation writing, so regression testing and typical beta testing would tend not to catch bugs there. Dag Sverre From robertwb at math.washington.edu Thu Aug 14 19:02:45 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 14 Aug 2008 10:02:45 -0700 Subject: [Cython] Making sure annotations work In-Reply-To: <51277.193.157.229.67.1218732087.squirrel@webmail.uio.no> References: <51277.193.157.229.67.1218732087.squirrel@webmail.uio.no> Message-ID: <2540E050-1173-42C5-8301-5F001908B306@math.washington.edu> On Aug 14, 2008, at 9:41 AM, Dag Sverre Seljebotn wrote: > While reading Robert's latest changes it occured to me that as > CCodeWriter > has changed quite a bit, this may affect the AnnotationCCodeWriter and > introduce bugs. Specifically, will annotation work correctly when > insertion points are created? > > Could somebody have a look over my changes to the annotation writer > (do a > diff since the prev release) and see if they make sense, and if you > think > they are enough, and do some basic testing? I'm very removed from > annotation writing (never used it and don't know exactly how it should > work) so it would be better if someone else could step in. > > I am mentioning this specifically since (to my knowledge) there are no > tests for annotation writing, so regression testing and typical beta > testing would tend not to catch bugs there. Yes, I did have to fix some stuff to get AnnotationCCodeWriter working. I'm still looking at it, but it seems to be doing fine now, insertion points and all. (It works by queueing up all metadata in a dict based on positions, and then inserting then at a later point, defering to the default codewriter routines to actually output the C file.) This could be because all the inserted stuff is not directly tied to a line of code in the pyx file however. - Robert From dagss at student.matnat.uio.no Thu Aug 14 21:00:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 14 Aug 2008 21:00:04 +0200 Subject: [Cython] Errors and pipelines (and small bug) Message-ID: <48A480B4.8010005@student.matnat.uio.no> I stumbled over this case which crashes Cython: http://hg.cython.org/cython-dagss/rev/50ec882441d0 Now, I could just go on and fix it in the switch transform; however I believe this is suboptimal... What happens is that an error is reported during expressino analysis, and then life goes on, to code which expects everything to be in order. So I propose this change: There is a cutoff in the pipeline at which stage errors are reported (and if any, processing stops). Any transforms after this stage (like those in Optimize.py) can happily assume there are no errors in the code. I can do it in 10 minutes but I wanted some feedback first. (In particular, a simpler solution would be to always report only 1 error -- this is what Python does, but Cython (and gcc) usually tries to gather up more than one error.) (I'm simply assuming that this is present somewhere in the Pyrex code so that one aborts right prior to code generation, so this is really a problem with the new pipeline stuff). -- Dag Sverre From greg.ewing at canterbury.ac.nz Fri Aug 15 02:41:29 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 15 Aug 2008 12:41:29 +1200 Subject: [Cython] Errors and pipelines (and small bug) In-Reply-To: <48A480B4.8010005@student.matnat.uio.no> References: <48A480B4.8010005@student.matnat.uio.no> Message-ID: <48A4D0B9.4040306@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > So I propose this change: There is a cutoff in the pipeline at which > stage errors are reported (and if any, processing stops). That sounds reasonable. You could even go further and have a cutoff after every stage of the pipeline, so that any stage can assume it has valid data from the previous stage. A refinement would be to have "fatal" and "non-fatal" errors, where fatal errors terminate compilation at the end of the current pipeline stage. -- Greg From robertwb at math.washington.edu Fri Aug 15 03:15:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 14 Aug 2008 18:15:31 -0700 (PDT) Subject: [Cython] Errors and pipelines (and small bug) In-Reply-To: <48A4D0B9.4040306@canterbury.ac.nz> References: <48A480B4.8010005@student.matnat.uio.no> <48A4D0B9.4040306@canterbury.ac.nz> Message-ID: On Fri, 15 Aug 2008, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> So I propose this change: There is a cutoff in the pipeline at which >> stage errors are reported (and if any, processing stops). > > That sounds reasonable. You could even go further and have > a cutoff after every stage of the pipeline, so that any > stage can assume it has valid data from the previous stage. Often it is useful to maximimize the number of errors caught in a single invocation of the compiler. One thing I hate doing is (run the compiler, fix the only reported error) x 10. > A refinement would be to have "fatal" and "non-fatal" > errors, where fatal errors terminate compilation at > the end of the current pipeline stage. This sounds like a very good idea. Non-fatal errors would result in a valid tree that could be carried on to the next stage, but invalid compilation. It might be to hackish, but we could also run each stage in a try-catch, and if it catches any Exceptions after a "non-fatal" error it silently ignores them. - Robert From dagss at student.matnat.uio.no Fri Aug 15 10:59:49 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 15 Aug 2008 10:59:49 +0200 (CEST) Subject: [Cython] Errors and pipelines (and small bug) In-Reply-To: References: <48A480B4.8010005@student.matnat.uio.no> <48A4D0B9.4040306@canterbury.ac.nz> Message-ID: <51617.193.157.229.67.1218790789.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Fri, 15 Aug 2008, Greg Ewing wrote: > >> Dag Sverre Seljebotn wrote: >> >>> So I propose this change: There is a cutoff in the pipeline at which >>> stage errors are reported (and if any, processing stops). >> >> That sounds reasonable. You could even go further and have >> a cutoff after every stage of the pipeline, so that any >> stage can assume it has valid data from the previous stage. > > Often it is useful to maximimize the number of errors caught in a single > invocation of the compiler. One thing I hate doing is (run the compiler, > fix the only reported error) x 10. > >> A refinement would be to have "fatal" and "non-fatal" >> errors, where fatal errors terminate compilation at >> the end of the current pipeline stage. > > This sounds like a very good idea. Non-fatal errors would result in a > valid tree that could be carried on to the next stage, but invalid > compilation. > > It might be to hackish, but we could also run each stage in a try-catch, > and if it catches any Exceptions after a "non-fatal" error it silently > ignores them. I already did this :-) sort of, but non-fatal error might very well require that processing continues within the transform. The current system is that transforms can raise a CompileError if processing cannot continue, or call self.context.nonfatal_error(CompileError(..)) if they just want to report an error and let the execution thread move on. It seems too much to add yet another mechanism to abort just a single transform (you just use a try-except clause in __call__ to drop out of the transformation stack and report the error as nonfatal before returning). So I'll just add a pipeline step that checks the nonfatal error list and stops compilation if there were nonfatal errors (prior to code generation). One can drop out before this by raising a CompileError and let it go all the way up to the pipeline runner. I'm mostly asking here for what we do for the beta -- the kind of errors we report could also be considered a regression; so too large a change doesn't seem good, but obviously Cython shouldn't crash either. But it seems like we agree. Dag Sverre From dagss at student.matnat.uio.no Fri Aug 15 14:18:45 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 15 Aug 2008 14:18:45 +0200 Subject: [Cython] Errors and pipelines (and small bug) In-Reply-To: <51617.193.157.229.67.1218790789.squirrel@webmail.uio.no> References: <48A480B4.8010005@student.matnat.uio.no> <48A4D0B9.4040306@canterbury.ac.nz> <51617.193.157.229.67.1218790789.squirrel@webmail.uio.no> Message-ID: <48A57425.4020902@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Fri, 15 Aug 2008, Greg Ewing wrote: >> >>> Dag Sverre Seljebotn wrote: >>> >>>> So I propose this change: There is a cutoff in the pipeline at which >>>> stage errors are reported (and if any, processing stops). >>> That sounds reasonable. You could even go further and have >>> a cutoff after every stage of the pipeline, so that any >>> stage can assume it has valid data from the previous stage. >> Often it is useful to maximimize the number of errors caught in a single >> invocation of the compiler. One thing I hate doing is (run the compiler, >> fix the only reported error) x 10. >> >>> A refinement would be to have "fatal" and "non-fatal" >>> errors, where fatal errors terminate compilation at >>> the end of the current pipeline stage. >> This sounds like a very good idea. Non-fatal errors would result in a >> valid tree that could be carried on to the next stage, but invalid >> compilation. >> >> It might be to hackish, but we could also run each stage in a try-catch, >> and if it catches any Exceptions after a "non-fatal" error it silently >> ignores them. > > I already did this :-) sort of, but non-fatal error might very well > require that processing continues within the transform. The current system > is that transforms can raise a CompileError if processing cannot continue, > or call self.context.nonfatal_error(CompileError(..)) if they just want to > report an error and let the execution thread move on. > > It seems too much to add yet another mechanism to abort just a single > transform (you just use a try-except clause in __call__ to drop out of the > transformation stack and report the error as nonfatal before returning). > > So I'll just add a pipeline step that checks the nonfatal error list and > stops compilation if there were nonfatal errors (prior to code > generation). One can drop out before this by raising a CompileError and > let it go all the way up to the pipeline runner. OK I now have an implementation of this. The problem is that there are a lot (around 40) instances of error being called in the code generation phase for various illegal condition. I'd argue that these should be checked for before code generation, so that code generation can assume everything is OK. That should make for cleaner code. But I'm not sure whether to move forward with this now. It's in -dagss anyway, but may not hit -devel. (It breaks some testcases which tested for more than one error, if some of those errors are reported during code generation.) So, the quick options for making the beta stop crashing seems to be: - Make SwitchTransform handle the error case gracefully. - Push my change and don't care about the regressions in the error tests (alternatively split them up or move just the error reporting that breaks from code generation to analysis or a visitor) I might pick one of these eventually but need to go on with "my own stuff" as well. (There will be some warning/a vote before a release, right?) -- Dag Sverre From dagss at student.matnat.uio.no Fri Aug 15 20:19:36 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 15 Aug 2008 20:19:36 +0200 Subject: [Cython] [Fwd: [Numpy-discussion] VS 2003 problems with cython-generated code] Message-ID: <48A5C8B8.50001@student.matnat.uio.no> I remember this being discussed here: http://codespeak.net/pipermail/cython-dev/2008-May/000844.html Was this resolved back then? -------- Original Message -------- Subject: [Numpy-discussion] VS 2003 problems with cython-generated code Date: Fri, 15 Aug 2008 11:49:28 -0500 From: David Cournapeau Reply-To: Discussion of Numerical Python To: Discussion of Numerical Python Hi, I noticed this morning that numpy 1.2 is not buildable with VS 2003 (the one you have to use for official python releases for at least python 2.5, and maybe 2.4). When we generate C code, both with internal code (numpy/core/code_generator) and with external tools (cython/pyrex for mtrand), the string literals generated for docstrings are too long for visual studio. We have to break them (e.g. "foo bar" becomes "foo""bar"), but doing so with cython-generated code is only doable by changing cython itself. So I did patch cython to break those, and regenerate the mtrand.c. This is done in the vs_longstring branch. Is it ok to put this for 1.2 ? Without it, I don't see a way to have numpy 1.2 buildable with VS. cheers, David P.S: I attached the necessary patches to cython bug tracker too, so that hopefully, the problem can be solved for a future version of cython. _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Dag Sverre From dagss at student.matnat.uio.no Sat Aug 16 00:04:38 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 16 Aug 2008 00:04:38 +0200 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: References: <48984C75.40101@student.matnat.uio.no> Message-ID: <48A5FD76.20407@student.matnat.uio.no> Robert Bradshaw wrote: >> Ways to set options: >> >> a) >> >> #cython: boundscheck=True, other=False >> #cython: third = no >> >> before any code in the file (but whitespace is allowed, I'm using >> Plex/the parser). The boolean value is case-insensitive and >> true/false/yes/no is allowed (is this too much? The thing is, for >> command-line arguments "True" is rather unfriendly and I'd like it >> to be >> consistent). > > I think we should follow a similar format as in PEP 0263. > Specifically, I'm not even sure the "cython" is necessary--the > compiler directives are arbitrary key-value pairs, and they are > looked up as needed. The ones that Cython uses/supports will be > documented. > > For the command line, we should be flexible, but a constant literal > for the values of the directives (not all of them may be booleans). I've changed this now in -devel. I realize this is not entirely according to process, but you can always back it out (my problem is that I want to share it and -dagss is blocked by that error reporting bugfix proposal...) The arguments seemed compelling: - It is reasonably self-contained and trivial, and is a new feature so beta-testers wouldn't regression test it anyway - There was +3 for this change (including myself) - It's too late to change this syntax in a few days You can now do # boundscheck = True # boundscheck = False at the beginning of the file, with whitespace variations only. Everything else is ignored (except using something else than True or False on rhs, which raises an error). We can loosen up this in a later release if we want to. The regex is: ^#\s*([a-z]+)\s*=(.*)$ and then the first group is matched against a list of possible options and the second group stripped for leading and trailing whitespace and parsed. (I've also changed the compiler directive switch to -X now) -- Dag Sverre From hoytak at cs.ubc.ca Sat Aug 16 16:05:02 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Sat, 16 Aug 2008 17:05:02 +0300 Subject: [Cython] Fwd: some errors annotating code In-Reply-To: <4db580fd0808160657m7f87231eqe0162052b4a6e715@mail.gmail.com> References: <4db580fd0808160657m7f87231eqe0162052b4a6e715@mail.gmail.com> Message-ID: <4db580fd0808160705p7929b9ar8432aba3699d9938@mail.gmail.com> Hello, I've been playing around with the cython code annotation feature, and I think I've found a couple easily fixable bugs in the html code annotation feature. I'm using the latest version from the mercurial repository (1053:e5976a417edf). First bug. I get the following backtrace: hoytak at ubuntu804desktop:~/workspace/clsynth-cython/clsynth$ cython -a transforms.pyx Traceback (most recent call last): File "/home/hoytak/sysroot/bin/cython", line 8, in main(command_line = 1) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 694, in main result = compile(sources, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 671, in compile return compile_multiple(source, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 641, in compile_multiple result = run_pipeline(source, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 503, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 169, in run_pipeline data = phase(data) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 114, in generate_pyx_code module_node.process_implementation(options, result) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 67, in process_implementation self.generate_c_code(env, options, result) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 283, in generate_c_code self.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 157, in annotate self.body.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 287, in annotate stat.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 287, in annotate stat.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 157, in annotate self.body.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 287, in annotate stat.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 3067, in annotate case.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 3044, in annotate body.annotate(code) NameError: global name 'body' is not defined It seems this error can be fixed by adding a "self." in 3044 in Nodes.py (see attached patch). When this is fixed, I get another error, namely hoytak at ubuntu804desktop:~/workspace/clsynth-cython/clsynth$ cython -a transforms.pyx Traceback (most recent call last): File "/home/hoytak/sysroot/bin/cython", line 8, in main(command_line = 1) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 694, in main result = compile(sources, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 671, in compile return compile_multiple(source, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 641, in compile_multiple result = run_pipeline(source, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 503, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 169, in run_pipeline data = phase(data) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 114, in generate_pyx_code module_node.process_implementation(options, result) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 67, in process_implementation self.generate_c_code(env, options, result) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 283, in generate_c_code self.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 157, in annotate self.body.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 287, in annotate stat.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 287, in annotate stat.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 157, in annotate self.body.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 287, in annotate stat.annotate(code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 3068, in annotate self.else_clause.annotate(code) AttributeError: 'NoneType' object has no attribute 'annotate' The documentation for this method says that self.else_clause can be None, so I assume adding a test for that works as a patch. After doing that, everything works fine for me. I attached the patch; let me know if I'm missing something If you want me to file a bug report, I can do that also. --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ -------------- next part -------------- A non-text attachment was scrubbed... Name: annotate_patch.diff Type: text/x-diff Size: 740 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080816/8a513bd0/attachment.bin From stefan_ml at behnel.de Sat Aug 16 21:54:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 16 Aug 2008 21:54:27 +0200 Subject: [Cython] [Pyrex] Pyrex exception reraise behavior In-Reply-To: <48882739.80006@canterbury.ac.nz> References: <48882739.80006@canterbury.ac.nz> Message-ID: <48A73073.9030302@behnel.de> Hi, Greg Ewing wrote: > There are some differences, yes. Python makes the > sys.exc_info() vars dynamically scoped by saving and > restoring them in the Python stack frame every time > a call occurs. > > That would be too expensive for Pyrex, so it uses > a different approach -- it stores information about > the caught exception in the C stack frame at the > point where it's caught (i.e. in the except clause). > The raise statement is required to be lexically > enclosed in the except clause so that it can access > this information. > > There may be some other differences as well. What > you appear to be seeing is a leaking of the > sys.exc_info() outside the scope where it would > normally be available in Python. Especially in Py3, where sys.exc_info() is reset when leaving the except block that caught the exception. I have a patch ready that fixes this with relatively little performance impact, although it requires remembering the currently raised exception whenever we enter a try statement (that's just three pointer copy operations). I think the OP's re-raising problem fits into this, too, although it would require some more thoughts to fix. There are some more issues with exception handling in Py3. Due to Python keeps exceptions in frames, I currently get crashes with Cython's exception handling (AFAICT unchanged from Pyrex) in Py3.0 beta2 (beta 1 works fine). I've been trying to dig into this problem, but it's pretty far from obvious to me how this can be fixed, or even what exactly goes wrong here. I attached the (somewhat hackish) patch I'm currently working with. It also contains a couple of test cases that show the behaviour I'm targeting, which is superficially compatible with Python 3. One of them crashes reliably for me. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: exception-handling.patch Type: text/x-patch Size: 9051 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080816/5a09ab6b/attachment-0001.bin From robertwb at math.washington.edu Sat Aug 16 22:27:47 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 16 Aug 2008 13:27:47 -0700 Subject: [Cython] Fwd: some errors annotating code In-Reply-To: <4db580fd0808160705p7929b9ar8432aba3699d9938@mail.gmail.com> References: <4db580fd0808160657m7f87231eqe0162052b4a6e715@mail.gmail.com> <4db580fd0808160705p7929b9ar8432aba3699d9938@mail.gmail.com> Message-ID: Thanks! On Aug 16, 2008, at 7:05 AM, Hoyt Koepke wrote: > Hello, > > I've been playing around with the cython code annotation feature, and > I think I've found a couple easily fixable bugs in the html code > annotation feature. I'm using the latest version from the mercurial > repository (1053:e5976a417edf). > > First bug. I get the following backtrace: > > hoytak at ubuntu804desktop:~/workspace/clsynth-cython/clsynth$ cython -a > transforms.pyx > Traceback (most recent call last): > File "/home/hoytak/sysroot/bin/cython", line 8, in > main(command_line = 1) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 694, in main > result = compile(sources, options) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 671, in compile > return compile_multiple(source, options) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 641, in compile_multiple > result = run_pipeline(source, options) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 503, in run_pipeline > err, enddata = context.run_pipeline(pipeline, source) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 169, in run_pipeline > data = phase(data) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 114, in generate_pyx_code > module_node.process_implementation(options, result) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/ModuleNode.py", > line 67, in process_implementation > self.generate_c_code(env, options, result) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/ModuleNode.py", > line 283, in generate_c_code > self.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 157, in annotate > self.body.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 287, in annotate > stat.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 287, in annotate > stat.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 157, in annotate > self.body.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 287, in annotate > stat.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 3067, in annotate > case.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 3044, in annotate > body.annotate(code) > NameError: global name 'body' is not defined > > > > It seems this error can be fixed by adding a "self." in 3044 in > Nodes.py (see attached patch). When this is fixed, I get another > error, namely > > > hoytak at ubuntu804desktop:~/workspace/clsynth-cython/clsynth$ cython -a > transforms.pyx > Traceback (most recent call last): > File "/home/hoytak/sysroot/bin/cython", line 8, in > main(command_line = 1) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 694, in main > result = compile(sources, options) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 671, in compile > return compile_multiple(source, options) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 641, in compile_multiple > result = run_pipeline(source, options) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 503, in run_pipeline > err, enddata = context.run_pipeline(pipeline, source) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 169, in run_pipeline > data = phase(data) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Main.py", > line 114, in generate_pyx_code > module_node.process_implementation(options, result) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/ModuleNode.py", > line 67, in process_implementation > self.generate_c_code(env, options, result) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/ModuleNode.py", > line 283, in generate_c_code > self.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 157, in annotate > self.body.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 287, in annotate > stat.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 287, in annotate > stat.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 157, in annotate > self.body.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 287, in annotate > stat.annotate(code) > File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/ > Compiler/Nodes.py", > line 3068, in annotate > self.else_clause.annotate(code) > AttributeError: 'NoneType' object has no attribute 'annotate' > > > > The documentation for this method says that self.else_clause can be > None, so I assume adding a test for that works as a patch. After > doing that, > everything works fine for me. I attached the patch; let me know if > I'm missing > something If you want me to file a bug report, I can do that also. > > --Hoyt > > > > -- > +++++++++++++++++++++++++++++++++++ > Hoyt Koepke > UBC Department of Computer Science > http://www.cs.ubc.ca/~hoytak/ > hoytak at gmail.com > +++++++++++++++++++++++++++++++++++ > > > > -- > +++++++++++++++++++++++++++++++++++ > Hoyt Koepke > UBC Department of Computer Science > http://www.cs.ubc.ca/~hoytak/ > hoytak at gmail.com > ++++++++++++++++++++++++++++++++++ > +_______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From greg.ewing at canterbury.ac.nz Sun Aug 17 02:35:50 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 17 Aug 2008 12:35:50 +1200 Subject: [Cython] [Pyrex] Pyrex exception reraise behavior In-Reply-To: <48A73073.9030302@behnel.de> References: <48882739.80006@canterbury.ac.nz> <48A73073.9030302@behnel.de> Message-ID: <48A77266.8060204@canterbury.ac.nz> Stefan Behnel wrote: > Greg Ewing wrote: > > > What > > you appear to be seeing is a leaking of the > > sys.exc_info() outside the scope where it would > > normally be available in Python. > > Especially in Py3, where sys.exc_info() is reset when leaving the except block > that caught the exception. But again, does this actually cause any problem? Is anyone likely to be looking at the result of sys.exc_info() and relying on it being empty when not handling an exception? As far as Pyrex is concerned, I'm not all that worried if it doesn't behave exactly the same as Python in all the corner cases, as long as it does something reasonable. I can understand, however, that the goals for Cython may be different. -- Greg From robertwb at math.washington.edu Sun Aug 17 02:43:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 16 Aug 2008 17:43:31 -0700 Subject: [Cython] Errors and pipelines (and small bug) In-Reply-To: <48A57425.4020902@student.matnat.uio.no> References: <48A480B4.8010005@student.matnat.uio.no> <48A4D0B9.4040306@canterbury.ac.nz> <51617.193.157.229.67.1218790789.squirrel@webmail.uio.no> <48A57425.4020902@student.matnat.uio.no> Message-ID: <5AB6F191-B170-49DA-A340-CE38DAC6DE9F@math.washington.edu> On Aug 15, 2008, at 5:18 AM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Robert Bradshaw wrote: >>> On Fri, 15 Aug 2008, Greg Ewing wrote: >>> >>>> Dag Sverre Seljebotn wrote: >>>> >>>>> So I propose this change: There is a cutoff in the pipeline at >>>>> which >>>>> stage errors are reported (and if any, processing stops). >>>> That sounds reasonable. You could even go further and have >>>> a cutoff after every stage of the pipeline, so that any >>>> stage can assume it has valid data from the previous stage. >>> Often it is useful to maximimize the number of errors caught in a >>> single >>> invocation of the compiler. One thing I hate doing is (run the >>> compiler, >>> fix the only reported error) x 10. >>> >>>> A refinement would be to have "fatal" and "non-fatal" >>>> errors, where fatal errors terminate compilation at >>>> the end of the current pipeline stage. >>> This sounds like a very good idea. Non-fatal errors would result >>> in a >>> valid tree that could be carried on to the next stage, but invalid >>> compilation. >>> >>> It might be to hackish, but we could also run each stage in a try- >>> catch, >>> and if it catches any Exceptions after a "non-fatal" error it >>> silently >>> ignores them. >> >> I already did this :-) sort of, but non-fatal error might very well >> require that processing continues within the transform. The >> current system >> is that transforms can raise a CompileError if processing cannot >> continue, >> or call self.context.nonfatal_error(CompileError(..)) if they just >> want to >> report an error and let the execution thread move on. >> >> It seems too much to add yet another mechanism to abort just a single >> transform (you just use a try-except clause in __call__ to drop >> out of the >> transformation stack and report the error as nonfatal before >> returning). >> >> So I'll just add a pipeline step that checks the nonfatal error >> list and >> stops compilation if there were nonfatal errors (prior to code >> generation). One can drop out before this by raising a >> CompileError and >> let it go all the way up to the pipeline runner. > > OK I now have an implementation of this. > > The problem is that there are a lot (around 40) instances of error > being > called in the code generation phase for various illegal condition. > > I'd argue that these should be checked for before code generation, so > that code generation can assume everything is OK. That should make for > cleaner code. > > But I'm not sure whether to move forward with this now. It's in -dagss > anyway, but may not hit -devel. (It breaks some testcases which tested > for more than one error, if some of those errors are reported during > code generation.) > > So, the quick options for making the beta stop crashing seems to be: > - Make SwitchTransform handle the error case gracefully. > - Push my change and don't care about the regressions in the error > tests > (alternatively split them up or move just the error reporting that > breaks from code generation to analysis or a visitor) > > I might pick one of these eventually but need to go on with "my own > stuff" as well. (There will be some warning/a vote before a > release, right?) I just wrote a simple one at http://hg.cython.org/cython-devel/rev/ b1516f6bf662 . It's much less invasive, and doesn't break any of the current error reporting. - Robert From robertwb at math.washington.edu Sun Aug 17 02:48:17 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 16 Aug 2008 17:48:17 -0700 Subject: [Cython] Cython 0.9.8.1 release candidate In-Reply-To: <66DF5956-7160-4FC8-8742-F1E05035105F@math.washington.edu> References: <66DF5956-7160-4FC8-8742-F1E05035105F@math.washington.edu> Message-ID: See http://cython.org/Cython-0.9.8.1-rc1.tar.gz Baring any catastrophic failures, we will release on Monday. There's a lot of new stuff in this version, so please test it out. - Robert On Aug 14, 2008, at 2:39 AM, Robert Bradshaw wrote: > See http://cython.org/Cython-0.9.8.1-beta2.tar.gz > > We are now working off of http://hg.cython.org/cython-devel > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sun Aug 17 06:54:48 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 16 Aug 2008 21:54:48 -0700 Subject: [Cython] Boundchecking question Message-ID: I've been playing with the (very cool!) buffer stuff Dag did getting ready for the SciPy conference. I've seen a 20% increase in speed not adjusting for negative indices, but making sure the indices are unsigned is a pain (especially if there is arithmetic involved). One proposal that has been brought up is to make positive integer constants unsigned, which could lead to complications elsewhere and would still require care. Another proposal that I'd like to throw out there is to have negative index realigning part of bounds checking, i.e. negative indices are not checked for if bounds checking is off. Thoughts? - Robert From dagss at student.matnat.uio.no Sun Aug 17 09:23:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 17 Aug 2008 09:23:00 +0200 Subject: [Cython] Boundchecking question Message-ID: <3301809832.108787@smtp.netcom.no> I'm not sure what you mean -- a negative bound is only checked for once per dimension. I.e. Positive ints are only checked for negativity once per dim. (Remember that one must also check for the negative bound, i.e. -n, but this is only done for negative numbers). There is potential for optimizing the exceptional case when bounds are off (but the code would be a little less clean and exceptional cases are just that). Could you perhaps post some C code modified as you suggest? Finally, if what you propose is a change in semantics, then this already met resistance on the numpy list. Perhaps a buffer-unsigned-indices-only compiler directive. (However I think that it is possible to write your code using unsigneds so that you use unsigned everywhere instead of signed, and then this comes automatically...) I don't care too much about constants, they're not likely to be used in performance critical code I think. Dag Sverre Seljebotn -----Original Message----- From: Robert Bradshaw Date: Sunday, Aug 17, 2008 6:55 am Subject: [Cython] Boundchecking question To: Cython-dev Reply-To: cython-dev at codespeak.net I've been playing with the (very cool!) buffer stuff Dag did getting >ready for the SciPy conference. I've seen a 20% increase in speed not >adjusting for negative indices, but making sure the indices are >unsigned is a pain (especially if there is arithmetic involved). One >proposal that has been brought up is to make positive integer >constants unsigned, which could lead to complications elsewhere and >would still require care. Another proposal that I'd like to throw out >there is to have negative index realigning part of bounds checking, >i.e. negative indices are not checked for if bounds checking is off. > >Thoughts? > >- Robert > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Sun Aug 17 09:58:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 17 Aug 2008 09:58:00 +0200 Subject: [Cython] Boundchecking question Message-ID: <3301811889.177242@smtp.netcom.no> ah, arithmetic on the indices. Yes, that is a bit inconvenient. Another thing you could try in your tests is tuning branch prediction (unlikely(...)).. Dag Sverre Seljebotn -----Original Message----- From: "Dag Sverre Seljebotn" Date: Sunday, Aug 17, 2008 9:26 am Subject: Re: [Cython] Boundchecking question To: Reply-To: cython-dev at codespeak.net I'm not sure what you mean -- a negative bound is only checked for once per dimension. I.e. Positive ints are only checked for negativity once per dim. > >(Remember that one must also check for the negative bound, i.e. -n, but this is only done for negative numbers). > >There is potential for optimizing the exceptional case when bounds are off (but the code would be a little less clean and exceptional cases are just that). > >Could you perhaps post some C code modified as you suggest? > >Finally, if what you propose is a change in semantics, then this already met resistance on the numpy list. Perhaps a buffer-unsigned-indices-only compiler directive. > >(However I think that it is possible to write your code using unsigneds so that you use unsigned everywhere instead of signed, and then this comes automatically...) > >I don't care too much about constants, they're not likely to be used in performance critical code I think. > >Dag Sverre Seljebotn >-----Original Message----- >From: Robert Bradshaw >Date: Sunday, Aug 17, 2008 6:55 am >Subject: [Cython] Boundchecking question >To: Cython-dev Reply-To: cython-dev at codespeak.net > >I've been playing with the (very cool!) buffer stuff Dag did getting >ready for the SciPy conference. I've seen a 20% increase in speed not >adjusting for negative indices, but making sure the indices are >unsigned is a pain (especially if there is arithmetic involved). One >proposal that has been brought up is to make positive integer >constants unsigned, which could lead to complications elsewhere and >would still require care. Another proposal that I'd like to throw out >there is to have negative index realigning part of bounds checking, >i.e. negative indices are not checked for if bounds checking is off. > >>Thoughts? > >>- Robert > >>_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Sun Aug 17 10:08:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 17 Aug 2008 10:08:00 +0200 Subject: [Cython] Boundchecking question Message-ID: <3301812499.238260@smtp.netcom.no> Sorry about all the posts, but I just now realized what you are saying.... well: - I write an algorithm where I don't care about speed and use negative indices - someone else takes it and turns off bounds checking as it doesn't raise any indexerrors, so "why not"? - poof! I think this would be very obscure. The recipe people would tend to follow seems to be "run with bounds checking on, if all the tests pass then you can turn it off". This would break that way of working. I vote for a new compiler directive to do this, and then a new higher-level directive including both this and bounds checking. Alternatively, a buffer option allow_negative which is set to True by default. Dag Sverre Seljebotn -----Original Message----- From: Robert Bradshaw Date: Sunday, Aug 17, 2008 6:55 am Subject: [Cython] Boundchecking question To: Cython-dev Reply-To: cython-dev at codespeak.net I've been playing with the (very cool!) buffer stuff Dag did getting >ready for the SciPy conference. I've seen a 20% increase in speed not >adjusting for negative indices, but making sure the indices are >unsigned is a pain (especially if there is arithmetic involved). One >proptosal that has been brought up is to make positive integer >constants unsigned, which could lead to complications elsewhere and >would still require care. Another proposal that I'd like to throw out >there is to have negative index realigning part of bounds checking, >i.e. negative indices are not checked for if bounds checking is off. > >Thoughts? > >- Robert > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Sun Aug 17 10:11:36 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 17 Aug 2008 01:11:36 -0700 Subject: [Cython] Boundchecking question In-Reply-To: <3301809832.108787@smtp.netcom.no> References: <3301809832.108787@smtp.netcom.no> Message-ID: <4608C8A2-E073-4532-A42A-063398E7BFD6@math.washington.edu> On Aug 17, 2008, at 12:23 AM, Dag Sverre Seljebotn wrote: > I'm not sure what you mean -- a negative bound is only checked for > once per dimension. I.e. Positive ints are only checked for > negativity once per dim. > > (Remember that one must also check for the negative bound, i.e. -n, > but this is only done for negative numbers). I am talking about the case where bounds checking is turned off, and "once per dimension" is relatively expensive. Even when it predicts correctly it still does the test. I've got some code using 2D buffers in the inner loop. Here's the timings: CPU time: 1.50 s, Wall time: 1.52 s -- standard buffers CPU time: 1.00 s, Wall time: 1.01 s -- boundscheck=False CPU time: 0.77 s, Wall time: 0.78 s -- unsigned indices (Note, I'm not even going to time the non-buffer version, but its certainly on the order of minutes--buffers are awsome!). > There is potential for optimizing the exceptional case when bounds > are off (but the code would be a little less clean and exceptional > cases are just that). > > Could you perhaps post some C code modified as you suggest? Essentially it's the C code one gets if one manually casts the indices to unsigned ints. It's just that that makes the code significantly uglier. > Finally, if what you propose is a change in semantics, then this > already met resistance on the numpy list. Perhaps a buffer-unsigned- > indices-only compiler directive. I was thinking of the case that bounds checking is turned off, in which case we already have a change in semantics, right? I could see this being a distinct compiler directive though, that's probably the way to go. > (However I think that it is possible to write your code using > unsigneds so that you use unsigned everywhere instead of signed, > and then this comes automatically...) > > I don't care too much about constants, they're not likely to be > used in performance critical code I think. Constants will be optimized away. But if I write "i+1" where i is an unsigned int, the result is still signed. > > Dag Sverre Seljebotn > > -----Original Message----- > From: Robert Bradshaw > Date: Sunday, Aug 17, 2008 6:55 am > Subject: [Cython] Boundchecking question > To: Cython-dev Reply-To: cython- > dev at codespeak.net > > I've been playing with the (very cool!) buffer stuff Dag did getting >> ready for the SciPy conference. I've seen a 20% increase in speed not >> adjusting for negative indices, but making sure the indices are >> unsigned is a pain (especially if there is arithmetic involved). One >> proposal that has been brought up is to make positive integer >> constants unsigned, which could lead to complications elsewhere and >> would still require care. Another proposal that I'd like to throw out >> there is to have negative index realigning part of bounds checking, >> i.e. negative indices are not checked for if bounds checking is off. >> >> Thoughts? >> >> - Robert >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sun Aug 17 10:15:36 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 17 Aug 2008 01:15:36 -0700 Subject: [Cython] Boundchecking question In-Reply-To: <3301812499.238260@smtp.netcom.no> References: <3301812499.238260@smtp.netcom.no> Message-ID: On Aug 17, 2008, at 1:08 AM, Dag Sverre Seljebotn wrote: > Sorry about all the posts, but I just now realized what you are > saying.... > > well: > - I write an algorithm where I don't care about speed and use > negative indices > - someone else takes it and turns off bounds checking as it doesn't > raise any indexerrors, so "why not"? > - poof! > > I think this would be very obscure. The recipe people would tend to > follow seems to be "run with bounds checking on, if all the tests > pass then you can turn it off". This would break that way of working. Yep, certainly. > I vote for a new compiler directive to do this, and then a new > higher-level directive including both this and bounds checking. > Alternatively, a buffer option allow_negative which is set to True > by default. My conclusion too. I still think we should put the option in a buffers namespace. "Flat is better than nested." vs "Namespaces are one honking great idea -- let's do more of those!" But the pragma doesn't turn of bounds checking for, other objects, e.g. tuples and lists (which could be nice to do). - Robert > > Dag Sverre Seljebotn > -----Original Message----- > From: Robert Bradshaw > Date: Sunday, Aug 17, 2008 6:55 am > Subject: [Cython] Boundchecking question > To: Cython-dev Reply-To: cython- > dev at codespeak.net > > I've been playing with the (very cool!) buffer stuff Dag did getting >> ready for the SciPy conference. I've seen a 20% increase in speed not >> adjusting for negative indices, but making sure the indices are >> unsigned is a pain (especially if there is arithmetic involved). One >> proptosal that has been brought up is to make positive integer >> constants unsigned, which could lead to complications elsewhere and >> would still require care. Another proposal that I'd like to throw out >> there is to have negative index realigning part of bounds checking, >> i.e. negative indices are not checked for if bounds checking is off. >> >> Thoughts? >> >> - Robert >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Sun Aug 17 10:18:10 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 17 Aug 2008 10:18:10 +0200 Subject: [Cython] [Pyrex] Pyrex exception reraise behavior In-Reply-To: <48A77266.8060204@canterbury.ac.nz> References: <48882739.80006@canterbury.ac.nz> <48A73073.9030302@behnel.de> <48A77266.8060204@canterbury.ac.nz> Message-ID: <48A7DEC2.9010900@behnel.de> Hi Greg, Greg Ewing wrote: > Stefan Behnel wrote: > >> Greg Ewing wrote: >> >>> What you appear to be seeing is a leaking of the >>> sys.exc_info() outside the scope where it would >>> normally be available in Python. >> Especially in Py3, where sys.exc_info() is reset when leaving the except block >> that caught the exception. > > But again, does this actually cause any problem? Well, yes, the same problems for which the exception catching semantics were redesigned in Py3. Leaving the exception in sys.exc_info will keep it alive longer than necessary, even for an undefined period of time, as it will only be deleted when the next exception is raised. In the case of an exception raised by Python, this will keep alive all related frames in the traceback, including their local variables etc. This can be expensive. http://www.python.org/dev/peps/pep-0344/ http://www.python.org/dev/peps/pep-3110/ Stefan From dagss at student.matnat.uio.no Sun Aug 17 10:18:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 17 Aug 2008 10:18:00 +0200 Subject: [Cython] Boundchecking question Message-ID: <3301813133.283027@smtp.netcom.no> On boundscheck, I'm thinking that it means "make the assumption that this code will not raise an IndexError". That's it. (And I think it should apply to lists and tuples as well if possible. (One could add a more fine-grained buffer_bounds_check too then, of course. Or "boundscheck = all", "boundscheck=None", "boundscheck = list,buffer"). But let's leave that for now, we can be backwards compatible later. Dag Sverre Seljebotn -----Original Message----- From: Robert Bradshaw Date: Sunday, Aug 17, 2008 10:11 am Subject: Re: [Cython] Boundchecking question To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Aug 17, 2008, at 12:23 AM, Dag Sverre Seljebotn wrote: > >> I'm not sure what you mean -- a negative bound is only checked for > once per dimension. I.e. Positive ints are only checked for > negativity once per dim. > >> (Remember that one must also check for the negative bound, i.e. -n, > but this is only done for negative numbers). > >I am talking about the case where bounds checking is turned off, and >"once per dimension" is relatively expensive. Even when it predicts >correctly it still does the test. I've got some code using 2D buffers >in the inner loop. Here's the timings: > >CPU time: 1.50 s, Wall time: 1.52 s -- standard buffers >CPU time: 1.00 s, Wall time: 1.01 s -- boundscheck=False >CPU time: 0.77 s, Wall time: 0.78 s -- unsigned indices > >(Note, I'm not even going to time the non-buffer version, but its >certainly on the order of minutes--buffers are awsome!). > >> There is potential for optimizing the exceptional case when bounds > are off (but the code would be a little less clean and exceptional > cases are just that). > >> Could you perhaps post some C code modified as you suggest? > >Essentially it's the C code one gets if one manually casts the >indices to unsigned ints. It's just that that makes the code >significantly uglier. > >> Finally, if what you propose is a change in semantics, then this > already met resistance on the numpy list. Perhaps a buffer-unsigned- > indices-only compiler directive. > >I was thinking of the case that bounds checking is turned off, in >which case we already have a change in semantics, right? I could see >this being a distinct compiler directive though, that's probably the >way to go. > >> (However I think that it is possible to write your code using > unsigneds so that you use unsigned everywhere instead of signed, > and then this comes automatically...) > >> I don't care too much about constants, they're not likely to be > used in performance critical code I think. > >Constants will be optimized away. But if I write "i+1" where i is an >unsigned int, the result is still signed. > >> > Dag Sverre Seljebotn > >> -----Original Message----- > From: Robert Bradshaw > Date: Sunday, Aug 17, 2008 6:55 am > Subject: [Cython] Boundchecking question > To: Cython-dev Reply-To: cython- > dev at codespeak.net > >> I've been playing with the (very cool!) buffer stuff Dag did getting >> ready for From cwitty at newtonlabs.com Sun Aug 17 18:36:15 2008 From: cwitty at newtonlabs.com (Carl Witty) Date: Sun, 17 Aug 2008 09:36:15 -0700 Subject: [Cython] Boundchecking question In-Reply-To: <4608C8A2-E073-4532-A42A-063398E7BFD6@math.washington.edu> References: <3301809832.108787@smtp.netcom.no> <4608C8A2-E073-4532-A42A-063398E7BFD6@math.washington.edu> Message-ID: On Sun, Aug 17, 2008 at 1:11 AM, Robert Bradshaw wrote: > Constants will be optimized away. But if I write "i+1" where i is an > unsigned int, the result is still signed. Are you talking about Cython or C here? In C, if i is an unsigned int, then i+1 is also an unsigned int. If that's not the case in Cython, I think it should be changed to match C. Carl From stefan_ml at behnel.de Sun Aug 17 23:23:12 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 17 Aug 2008 23:23:12 +0200 Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48A5FD76.20407@student.matnat.uio.no> References: <48984C75.40101@student.matnat.uio.no> <48A5FD76.20407@student.matnat.uio.no> Message-ID: <48A896C0.5050606@behnel.de> Hi, I missed the end of this discussion, but here is a comment anyway. Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> I think we should follow a similar format as in PEP 0263. I do not think that PEP 263 provides a suitable example here, as the syntax was specifically chosen as the lowest common divisor of a number of existing encoding hints for different editors, each of which has an explicit syntax that isn't quite compatible with any other syntax. >> Specifically, I'm not even sure the "cython" is necessary--the >> compiler directives are arbitrary key-value pairs, and they are >> looked up as needed. The ones that Cython uses/supports will be >> documented. > > You can now do > > # boundscheck = True > # boundscheck = False The problem with this is that users who are not aware of directives can end up breaking their code by commenting out an assignment. Even if that's unlikely, it is not immediately clear from the syntax that it is /not/ just an assignment that was removed from the code by commenting it out, but something that actually changes the way things work. A compiler directive should be clearly recognisable as such, especially when it invades the privacy of the user's comments. > at the beginning of the file, with whitespace variations only. > Everything else is ignored (except using something else than True or > False on rhs, which raises an error). I like the fact that this only works at the very beginning of the source file (preferably after the encoding hint), and I also like that it only works with True/False and looks like a keyword argument. Stefan From dagss at student.matnat.uio.no Sun Aug 17 23:37:10 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 17 Aug 2008 23:37:10 +0200 (CEST) Subject: [Cython] Done with compiler options/pragmas In-Reply-To: <48A896C0.5050606@behnel.de> References: <48984C75.40101@student.matnat.uio.no> <48A5FD76.20407@student.matnat.uio.no> <48A896C0.5050606@behnel.de> Message-ID: <52838.193.157.229.67.1219009030.squirrel@webmail.uio.no> Stefan wrote: > Hi, > > I missed the end of this discussion, but here is a comment anyway. > > Dag Sverre Seljebotn wrote: >> Robert Bradshaw wrote: >>> I think we should follow a similar format as in PEP 0263. > > I do not think that PEP 263 provides a suitable example here, as the > syntax > was specifically chosen as the lowest common divisor of a number of > existing > encoding hints for different editors, each of which has an explicit syntax > that isn't quite compatible with any other syntax. > > >>> Specifically, I'm not even sure the "cython" is necessary--the >>> compiler directives are arbitrary key-value pairs, and they are >>> looked up as needed. The ones that Cython uses/supports will be >>> documented. >> >> You can now do >> >> # boundscheck = True >> # boundscheck = False > > The problem with this is that users who are not aware of directives can > end up > breaking their code by commenting out an assignment. Even if that's > unlikely, > it is not immediately clear from the syntax that it is /not/ just an > assignment that was removed from the code by commenting it out, but > something > that actually changes the way things work. A compiler directive should be > clearly recognisable as such, especially when it invades the privacy of > the > user's comments. Yes, and this is now reverted in -devel; you have to use a cython: prefix. I just remembered (I had forgot) that my stance was to do "boundscheck(True)" instead in order to become consistent with the decorators and with statements, so that there is only one syntax within pyx files. But it is too late to do that now I think (at least for this release; and supporting both is not a problem). Dag Sverre From hoytak at cs.ubc.ca Mon Aug 18 10:13:57 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 18 Aug 2008 11:13:57 +0300 Subject: [Cython] Another annotator bug Message-ID: <4db580fd0808180113r1f594857j81f5973641ca1e3c@mail.gmail.com> Hello, In continuing to play around with the annotation function, I've found another bug (and I think I have it patched as well). For me, it shows up when there is an error in my code. It doesn't seem to matter what that error is, as I've fixed several and still get the same backtrace. This is in r1071 After running cython -a transforms.pyx, it prints out an error message and then raises Traceback (most recent call last): File "/home/hoytak/sysroot/bin/cython", line 8, in main(command_line = 1) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 698, in main result = compile(sources, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 675, in compile return compile_multiple(source, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 645, in compile_multiple result = run_pipeline(source, options) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 507, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 169, in run_pipeline data = phase(data) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Main.py", line 114, in generate_pyx_code module_node.process_implementation(options, result) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 67, in process_implementation self.generate_c_code(env, options, result) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 273, in generate_c_code self.generate_declarations_for_modules(env, modules, h_code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 387, in generate_declarations_for_modules env, modules, vtab_list, vtabslot_list, code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 376, in generate_type_definitions self.generate_obj_struct_definition(entry.type, code) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/ModuleNode.py", line 704, in generate_obj_struct_definition code.mark_pos(type.pos) File "/home/hoytak/sysroot/lib/python2.5/site-packages/Cython/Compiler/Annotate.py", line 44, in mark_pos if self.last_pos: AttributeError: 'AnnotationCCodeWriter' object has no attribute 'last_pos' The problem is that self.last_pos is not always assigned in the constructor of AnnotationCCodeWriter, which seems to be the case on my code. I've attached a patch which fixes it for me (patch-last_pos.diff). In looking at the code, it seems that "self.annotator = StringIO()" can be moved inside the "if create_from is None:" block. If that's the case, then the second patch (patch-full.diff) does that as well. --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-full.diff Type: text/x-diff Size: 714 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080818/ff3d2095/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-last_node.diff Type: text/x-diff Size: 662 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080818/ff3d2095/attachment-0001.bin From dagss at student.matnat.uio.no Mon Aug 18 14:48:01 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 18 Aug 2008 14:48:01 +0200 Subject: [Cython] Request for review: runtests.py and NumPy issues Message-ID: <48A96F81.1040306@student.matnat.uio.no> As we're so close to a release I'm going to post a patch here instead of commiting it straight away. The patch fixes an issue with runtests.py with a lot of releases of NumPy. The latest svn version doesn't have the problem but the one shipped with the latest Ubuntu does, so I expect a lot of users may encounter it. The problem is that NumPy doesn't like distutils.unixccompiler being imported already. One could think that the test environment should be further "sandboxed" and that all modules should be removed, but I don't know anything about sys.modules really so I'll leave that to somebody else. Something else: The "cimport" triggers a module import so the numpy testcase fails on systems where numpy is not installed. This isn't all that serious I feel (a failing numpy testcase will communicate the right thing to users not having numpy installed) so I think we should wait until after the release (and then add some kind of "_USES" special testcase variable). -- Dag Sverre -------------- next part -------------- A non-text attachment was scrubbed... Name: runtests_hide_modules.diff Type: text/x-diff Size: 833 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080818/b2d3fc5c/attachment.bin From dagss at student.matnat.uio.no Mon Aug 18 15:47:25 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 18 Aug 2008 15:47:25 +0200 Subject: [Cython] Another annotator bug In-Reply-To: <4db580fd0808180113r1f594857j81f5973641ca1e3c@mail.gmail.com> References: <4db580fd0808180113r1f594857j81f5973641ca1e3c@mail.gmail.com> Message-ID: <48A97D6D.5000109@student.matnat.uio.no> Hoyt Koepke wrote: > Hello, > > In continuing to play around with the annotation function, I've found > another bug (and I think I have it patched as well). For me, it shows > up when there is an error in my code. It doesn't seem to matter what > that error is, as I've fixed several and still get the same backtrace. > This is in r1071 > > After running cython -a transforms.pyx, it prints out an error message > and then raises > > The problem is that self.last_pos is not always assigned in the > constructor of AnnotationCCodeWriter, which seems to be the case on my > code. I've attached a patch which fixes it for me > (patch-last_pos.diff). In looking at the code, it seems that I believe the right thing to do here is to copy last_pos in create_from (see below), but I'm not commiting as I'd like Robert to confirm (as there's going to be a release today). The cleanup of self.annotator is correct. Thanks! --- a/Cython/Compiler/Annotate.py Mon Aug 18 07:49:35 2008 +0200 +++ b/Cython/Compiler/Annotate.py Mon Aug 18 15:45:02 2008 +0200 @@ -29,6 +29,7 @@ class AnnotationCCodeWriter(CCodeWriter) self.annotation_buffer = create_from.annotation_buffer self.annotations = create_from.annotations self.code = create_from.code + self.last_pos = create_from.last_pos def create_new(self, create_from, buffer, copy_formatting): return AnnotationCCodeWriter(create_from, buffer, copy_formatting) -- Dag Sverre From dagss at student.matnat.uio.no Mon Aug 18 15:50:58 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 18 Aug 2008 15:50:58 +0200 Subject: [Cython] Another annotator bug In-Reply-To: <48A97D6D.5000109@student.matnat.uio.no> References: <4db580fd0808180113r1f594857j81f5973641ca1e3c@mail.gmail.com> <48A97D6D.5000109@student.matnat.uio.no> Message-ID: <48A97E42.4040500@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Hoyt Koepke wrote: >> Hello, >> >> In continuing to play around with the annotation function, I've found >> another bug (and I think I have it patched as well). For me, it shows >> up when there is an error in my code. It doesn't seem to matter what >> that error is, as I've fixed several and still get the same backtrace. >> This is in r1071 >> >> After running cython -a transforms.pyx, it prints out an error message >> and then raises >> >> The problem is that self.last_pos is not always assigned in the >> constructor of AnnotationCCodeWriter, which seems to be the case on my >> code. I've attached a patch which fixes it for me >> (patch-last_pos.diff). In looking at the code, it seems that > > I believe the right thing to do here is to copy last_pos in create_from > (see below), but I'm not commiting as I'd like Robert to confirm (as > there's going to be a release today). The cleanup of self.annotator is > correct. > > Thanks! Attaching a patch properly doing the above. -- Dag Sverre -------------- next part -------------- A non-text attachment was scrubbed... Name: annotation_fix.diff Type: text/x-diff Size: 1049 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080818/dd0150dd/attachment.bin From stefan_ml at behnel.de Mon Aug 18 15:56:20 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 18 Aug 2008 15:56:20 +0200 (CEST) Subject: [Cython] Request for review: runtests.py and NumPy issues In-Reply-To: <48A96F81.1040306@student.matnat.uio.no> References: <48A96F81.1040306@student.matnat.uio.no> Message-ID: <51677.213.61.181.86.1219067780.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Something else: The "cimport" triggers a module import so the numpy > testcase fails on systems where numpy is not installed. This isn't all > that serious I feel (a failing numpy testcase will communicate the right > thing to users not having numpy installed) so I think we should wait > until after the release (and then add some kind of "_USES" special > testcase variable). Can't we keep a mapping of required imports (i.e. package names) to test case name patterns in runtests.py, and then try-except on the import before considering a matching test(s) for inclusion? That way, we could skip the tests that have unsatisfied external requirements entirely. If we additionally used a standard naming pattern like "numpy_xyz.pyx" for numpy related test cases, that would become trivial to handle. Stefan From stefan_ml at behnel.de Mon Aug 18 17:10:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 18 Aug 2008 17:10:07 +0200 (CEST) Subject: [Cython] Request for review: runtests.py and NumPy issues In-Reply-To: <48A96F81.1040306@student.matnat.uio.no> References: <48A96F81.1040306@student.matnat.uio.no> Message-ID: <48331.213.61.181.86.1219072207.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > The problem is that NumPy doesn't like > distutils.unixccompiler being imported already. Could you elaborate on this a little? Which part of NumPy doesn't "like" the module being imported? Why isn't it enough to import NumPy first (and maybe do some stuff with it) and /then/ run the tests? To check if NumPy is installed (i.e. to see if you can run the test), you'll have to import it anyway. Stefan From dagss at student.matnat.uio.no Mon Aug 18 18:22:32 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 18 Aug 2008 18:22:32 +0200 Subject: [Cython] Request for review: runtests.py and NumPy issues In-Reply-To: <48331.213.61.181.86.1219072207.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <48A96F81.1040306@student.matnat.uio.no> <48331.213.61.181.86.1219072207.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48A9A1C8.5070000@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> The problem is that NumPy doesn't like >> distutils.unixccompiler being imported already. > > Could you elaborate on this a little? Which part of NumPy doesn't "like" > the module being imported? Why isn't it enough to import NumPy first (and > maybe do some stuff with it) and /then/ run the tests? To check if NumPy > is installed (i.e. to see if you can run the test), you'll have to import > it anyway. > Actually, you are right -- this problem now disappear because numpy is imported before distutils (after implementing your previous proposal). The point is the ordering of the imports: Py> import numpy is ok, however: Py> import distutils.unixccompiler Py> import numpy Traceback (most recent call last): ... Again, this is a bug in NumPy, but I like working around it and not requiring bleeding edge for our testcase to run (in fact it should be able to run in order to check how we do for older releases as well). I have now commited the addition of your proposal to runtests.py, i.e. at the top it now says: # Lists external modules, and a matcher matching tests # which should be excluded if the module is not present. EXT_DEP_MODULES = { 'numpy' : re.compile('.*\.numpy_.*').match } It commited it right away to -devel as it is an isolated change (please remove it if it causes problems anywhere, I've tested it with Py2.3, Py2.5, Py2.6b1 and Py3.0b2 (latter two some beta version)) -- Dag Sverre From hoytak at cs.ubc.ca Mon Aug 18 18:43:05 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 18 Aug 2008 19:43:05 +0300 Subject: [Cython] Another annotator bug In-Reply-To: <48A97D6D.5000109@student.matnat.uio.no> References: <4db580fd0808180113r1f594857j81f5973641ca1e3c@mail.gmail.com> <48A97D6D.5000109@student.matnat.uio.no> Message-ID: <4db580fd0808180943g6e888d02wf5274d193f009349@mail.gmail.com> > I believe the right thing to do here is to copy last_pos in create_from > (see below), but I'm not commiting as I'd like Robert to confirm (as there's > going to be a release today). The cleanup of self.annotator is correct. Yes, after looking at things a bit more, I agree. It also fixes my problem. --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From robertwb at math.washington.edu Mon Aug 18 19:28:44 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Aug 2008 10:28:44 -0700 (PDT) Subject: [Cython] Boundchecking question In-Reply-To: References: <3301809832.108787@smtp.netcom.no> <4608C8A2-E073-4532-A42A-063398E7BFD6@math.washington.edu> Message-ID: On Sun, 17 Aug 2008, Carl Witty wrote: > On Sun, Aug 17, 2008 at 1:11 AM, Robert Bradshaw > wrote: >> Constants will be optimized away. But if I write "i+1" where i is an >> unsigned int, the result is still signed. > > Are you talking about Cython or C here? In C, if i is an unsigned > int, then i+1 is also an unsigned int. If that's not the case in > Cython, I think it should be changed to match C. Ah, so in C, signed ints get promoted to unsigned ints. Interesting. Cython should be fixed to reflect this, and this will make declaring indices non-negative much easier (though I'm not going to do that until next release as there may be unintended side effects). - Robert From stefan_ml at behnel.de Mon Aug 18 21:06:00 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 18 Aug 2008 21:06:00 +0200 Subject: [Cython] Request for review: runtests.py and NumPy issues In-Reply-To: <48A9A1C8.5070000@student.matnat.uio.no> References: <48A96F81.1040306@student.matnat.uio.no> <48331.213.61.181.86.1219072207.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48A9A1C8.5070000@student.matnat.uio.no> Message-ID: <48A9C818.1090204@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Again, this is a bug in NumPy, but I like working around it and not > requiring bleeding edge for our testcase to run +1 > It commited it right away to -devel as it is an isolated change Thanks. Stefan From stefan_ml at behnel.de Mon Aug 18 22:07:43 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 18 Aug 2008 22:07:43 +0200 Subject: [Cython] print statement under Py3k In-Reply-To: <48A9CC0D.2010200@student.matnat.uio.no> References: <48A7293D.20407@behnel.de> <48A8A83E.7080206@student.matnat.uio.no> <48A9106D.3010108@behnel.de> <48A9CC0D.2010200@student.matnat.uio.no> Message-ID: <48A9D68F.1050500@behnel.de> Hi, Dag Sverre Seljebotn wrote: > There's a problem if you do an empty print (newline > print) right after a non-newline print. The attached test case works in > Py2 but fails in Py3 (the line with "2 2 2" is output as "2 2 2 " instead). > for i in range(3): > print 2, > print Right, that makes sense. The Py3 print() function (that we map the print statement to when running under Py3) has an explicit 'end' keyword that keeps it from having to do any "softspace" special casing. So this is a difference that is hard to emulate in Py3, which (as opposed to Py2) does not expose the softspace hint at all. http://www.python.org/dev/peps/pep-3105/ Open poll: a) would it be acceptable if this difference persisted, i.e. if Cython code that uses something like the example above would behave different in Py3 b) should Cython try to special case this in Py3 to emulate the Py2 behaviour in common cases c) should Cython stick to the Py3 behaviour also in Py2 (thus breaking the complete semantics of the print statement) ? (I am not asking here if we should support the print() function in Cython source code, which we may do anyway some day. But that's orthogonal.) Stefan From dagss at student.matnat.uio.no Mon Aug 18 22:24:22 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 18 Aug 2008 22:24:22 +0200 Subject: [Cython] print statement under Py3k In-Reply-To: <48A9D68F.1050500@behnel.de> References: <48A7293D.20407@behnel.de> <48A8A83E.7080206@student.matnat.uio.no> <48A9106D.3010108@behnel.de> <48A9CC0D.2010200@student.matnat.uio.no> <48A9D68F.1050500@behnel.de> Message-ID: <48A9DA76.60902@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> There's a problem if you do an empty print (newline >> print) right after a non-newline print. The attached test case works in >> Py2 but fails in Py3 (the line with "2 2 2" is output as "2 2 2 " instead). > >> for i in range(3): >> print 2, >> print > > Right, that makes sense. The Py3 print() function (that we map the print > statement to when running under Py3) has an explicit 'end' keyword that keeps > it from having to do any "softspace" special casing. So this is a difference > that is hard to emulate in Py3, which (as opposed to Py2) does not expose the > softspace hint at all. > > http://www.python.org/dev/peps/pep-3105/ > > Open poll: > > a) would it be acceptable if this difference persisted, i.e. if Cython code > that uses something like the example above would behave different in Py3 > b) should Cython try to special case this in Py3 to emulate the Py2 behaviour > in common cases +1; though it ends up on the list of priorities we all keep in the back of our head and then it is sure to end up pretty low :-) But I think this is the optimal option. -- Dag Sverre From robertwb at math.washington.edu Tue Aug 19 00:43:05 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Aug 2008 15:43:05 -0700 (PDT) Subject: [Cython] print statement under Py3k In-Reply-To: <48A9D68F.1050500@behnel.de> References: <48A7293D.20407@behnel.de> <48A8A83E.7080206@student.matnat.uio.no> <48A9106D.3010108@behnel.de> <48A9CC0D.2010200@student.matnat.uio.no> <48A9D68F.1050500@behnel.de> Message-ID: On Mon, 18 Aug 2008, Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> There's a problem if you do an empty print (newline >> print) right after a non-newline print. The attached test case works in >> Py2 but fails in Py3 (the line with "2 2 2" is output as "2 2 2 " instead). > >> for i in range(3): >> print 2, >> print > > Right, that makes sense. The Py3 print() function (that we map the print > statement to when running under Py3) has an explicit 'end' keyword that keeps > it from having to do any "softspace" special casing. So this is a difference > that is hard to emulate in Py3, which (as opposed to Py2) does not expose the > softspace hint at all. > > http://www.python.org/dev/peps/pep-3105/ > > Open poll: > > a) would it be acceptable if this difference persisted, i.e. if Cython code > that uses something like the example above would behave different in Py3 > b) should Cython try to special case this in Py3 to emulate the Py2 behaviour > in common cases > c) should Cython stick to the Py3 behaviour also in Py2 (thus breaking the > complete semantics of the print statement) > > ? I certainly think (b) is the rigth option, depending on how hard and/or hackish it is. After that is should fall back to (a). I really don't like option (c), not only does it break existing code but the print statement doesn't even exist in Py3 so trying to "match" it is probably a bad thing. > > (I am not asking here if we should support the print() function in Cython > source code, which we may do anyway some day. But that's orthogonal.) > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Tue Aug 19 03:41:27 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Aug 2008 18:41:27 -0700 Subject: [Cython] Another annotator bug In-Reply-To: <48A97E42.4040500@student.matnat.uio.no> References: <4db580fd0808180113r1f594857j81f5973641ca1e3c@mail.gmail.com> <48A97D6D.5000109@student.matnat.uio.no> <48A97E42.4040500@student.matnat.uio.no> Message-ID: Yes, this looks correct. Thanks. On Aug 18, 2008, at 6:50 AM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Hoyt Koepke wrote: >>> Hello, >>> >>> In continuing to play around with the annotation function, I've >>> found >>> another bug (and I think I have it patched as well). For me, it >>> shows >>> up when there is an error in my code. It doesn't seem to matter >>> what >>> that error is, as I've fixed several and still get the same >>> backtrace. >>> This is in r1071 >>> >>> After running cython -a transforms.pyx, it prints out an error >>> message >>> and then raises >>> >>> The problem is that self.last_pos is not always assigned in the >>> constructor of AnnotationCCodeWriter, which seems to be the case >>> on my >>> code. I've attached a patch which fixes it for me >>> (patch-last_pos.diff). In looking at the code, it seems that >> I believe the right thing to do here is to copy last_pos in >> create_from (see below), but I'm not commiting as I'd like Robert >> to confirm (as there's going to be a release today). The cleanup >> of self.annotator is correct. >> Thanks! > > Attaching a patch properly doing the above. > > -- > Dag Sverre > diff -r 265a5de27a1b Cython/Compiler/Annotate.py > --- a/Cython/Compiler/Annotate.py Mon Aug 18 15:49:36 2008 +0200 > +++ b/Cython/Compiler/Annotate.py Mon Aug 18 15:50:03 2008 +0200 > @@ -19,8 +19,8 @@ class AnnotationCCodeWriter(CCodeWriter) > > def __init__(self, create_from=None, buffer=None, > copy_formatting=True): > CCodeWriter.__init__(self, create_from, buffer, > copy_formatting=True) > - self.annotation_buffer = StringIO() > if create_from is None: > + self.annotation_buffer = StringIO() > self.annotations = [] > self.last_pos = None > self.code = {} > @@ -28,6 +28,7 @@ class AnnotationCCodeWriter(CCodeWriter) > # When creating an insertion point, keep references to > the same database > self.annotation_buffer = create_from.annotation_buffer > self.annotations = create_from.annotations > + self.last_pos = create_from.last_pos > self.code = create_from.code > > def create_new(self, create_from, buffer, copy_formatting): > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Tue Aug 19 06:12:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Aug 2008 21:12:14 -0700 Subject: [Cython] Any thoughts on a DEFINED keyword? Message-ID: Does anyone else have any thoughts on this: http://trac.cython.org/ cython_trac/ticket/27 - Robert From robertwb at math.washington.edu Tue Aug 19 07:09:13 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Aug 2008 22:09:13 -0700 Subject: [Cython] Cython 0.9.8.1 Released Message-ID: <773CCC04-40F1-4EAF-A12F-12D4B14B0AA2@math.washington.edu> Sources are up in their usual places. The largest addition was Dag Seljebotn's buffer support which is amazing. Kudos to him for all his hard work (and to Enthought/Google for funding him this summer). Part of this effort involved moving to a pipeline/transformation based compilation system, which has allowed for many other nice features (e.g. long if statements get translated into c switch statements if possible). Another big inclusion is Paul Prescod's pyximport. Now one can simply start up Python and type import pyximport; pyximport.install() import foo which will compile foo.pyx for you. We would like to be able to (optionally) specify all options via directives in the files themselves, which would make this even more useful. There are numerous fixes and improvements by Stefan Behnel, Dag Seljebotn, and Robert Bradshaw that are not listed here. Among them are better support for Py3, unicode, C++, literals, testing framework, etc. We also merged patches by Jim Kleckner, Hoyt Koepke, Marcus Bitzl, Kirill Smelkov, and Carl Witty. Also, a (far from exhaustive) summary of the tickets closed, see: http://trac.cython.org/cython_trac/query?status=closed&milestone=0.9.8.1 - Robert From dagss at student.matnat.uio.no Tue Aug 19 07:27:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 19 Aug 2008 07:27:03 +0200 (CEST) Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: References: Message-ID: <53749.193.157.229.67.1219123623.squirrel@webmail.uio.no> > Does anyone else have any thoughts on this: http://trac.cython.org/ > cython_trac/ticket/27 I'm not sure if I get the motivating usecase ... it looks like someone is using include but should be using cimport. Whatever someones needs is to require such protections it would be better to find a more high-level way of going about it IMO (e.g. if using cimport is absolutely impossible, perhaps that is an argument for include_once or similar). The feature doesn't really hurt though, so I'm -0.1 (and may be +1 if I could see a better/more detailed motivating usecase). Dag Sverre From stefan_ml at behnel.de Tue Aug 19 11:25:42 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 19 Aug 2008 11:25:42 +0200 (CEST) Subject: [Cython] print statement under Py3k In-Reply-To: References: <48A7293D.20407@behnel.de> <48A8A83E.7080206@student.matnat.uio.no> <48A9106D.3010108@behnel.de> <48A9CC0D.2010200@student.matnat.uio.no> <48A9D68F.1050500@behnel.de> Message-ID: <42835.213.61.181.86.1219137942.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Mon, 18 Aug 2008, Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >> >>> for i in range(3): >>> print 2, >>> print >> >> Right, that makes sense. The Py3 print() function (that we map the print >> statement to when running under Py3) has an explicit 'end' keyword that >> keeps >> it from having to do any "softspace" special casing. So this is a >> difference >> that is hard to emulate in Py3, which (as opposed to Py2) does not >> expose the softspace hint at all. >> >> http://www.python.org/dev/peps/pep-3105/ >> >> Open poll: >> >> a) would it be acceptable if this difference persisted, i.e. if Cython >> code >> that uses something like the example above would behave different in >> Py3 >> b) should Cython try to special case this in Py3 to emulate the Py2 >> behaviour >> in common cases >> c) should Cython stick to the Py3 behaviour also in Py2 (thus breaking >> the >> complete semantics of the print statement) > > I certainly think (b) is the rigth option, depending on how hard and/or > hackish it is. After that is should fall back to (a). I really don't like > option (c), not only does it break existing code but the print statement > doesn't even exist in Py3 so trying to "match" it is probably a bad > thing. When I say "emulate", this means that we have to keep track of the print call and insert the "," induced space /before/ the next print instead of appending it to the current print. This means that code that lets Python output things in the meantime would still break, but output that comes only from Cython could be made to work. Since this is definitely hackish, I would then want to see the "print_statement" future import implemented also (i.e. the print function implemented in Cython on Py2), so that we can use a well-defined function instead of something that may or may not break in Py3. That should actually be easier than trying to get the above semantics of the print statement right in as many cases as possible. Stefan From dagss at student.matnat.uio.no Tue Aug 19 11:45:37 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 19 Aug 2008 11:45:37 +0200 (CEST) Subject: [Cython] Repository usage Message-ID: <53849.193.157.229.67.1219139137.squirrel@webmail.uio.no> There's been some talk already but now seems the time to make votes etc. I propose that bugfixes that a) are relatively simple, and b) doesn't rely on or interfere features implemented since the last release, are coded against and pushed to "cython" and then merged into "cython-devel". "cython" should still be considered relatively stable, so every commit there is like a micro-patch-release of Cython. This should take some pressure off cython-devel (although keeping it as stable as possible is still good). Cython 0.9.8.2 may then come entirely from "cython" if it is primarily a bugfix release, or from "cython-devel" if it stabilizes. One should think about some of the things that may be done to cython-devel soon, especially import Greg's result_code-calculation-in-generation-phase changes, and the other refactorings related to/made possible by that. These are mildly dangerous (at least they lead to changes "everywhere") but also needs to be in the main stream of development. An alternative would perhaps be renaming "dagss" to "experimental" (but IMO all new features as well as new framework changes so go into it then). Thoughts? Dag Sverre From robertwb at math.washington.edu Tue Aug 19 13:41:03 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 19 Aug 2008 04:41:03 -0700 Subject: [Cython] Cython 0.9.8.1.1 Released In-Reply-To: <773CCC04-40F1-4EAF-A12F-12D4B14B0AA2@math.washington.edu> References: <773CCC04-40F1-4EAF-A12F-12D4B14B0AA2@math.washington.edu> Message-ID: <1E01FAB3-5241-4757-95D8-F9D067485E1D@math.washington.edu> Despite a week long beta period, the .pxd files weren't in the final bundle. A renumber is required for the pypi repository. On Aug 18, 2008, at 10:09 PM, Robert Bradshaw wrote: > Sources are up in their usual places. The largest addition was Dag > Seljebotn's buffer support which is amazing. Kudos to him for all his > hard work (and to Enthought/Google for funding him this summer). Part > of this effort involved moving to a pipeline/transformation based > compilation system, which has allowed for many other nice features > (e.g. long if statements get translated into c switch statements if > possible). > > Another big inclusion is Paul Prescod's pyximport. Now one can simply > start up Python and type > > import pyximport; pyximport.install() > import foo > > which will compile foo.pyx for you. We would like to be able to > (optionally) specify all options via directives in the files > themselves, which would make this even more useful. > > There are numerous fixes and improvements by Stefan Behnel, Dag > Seljebotn, and Robert Bradshaw that are not listed here. Among them > are better support for Py3, unicode, C++, literals, testing > framework, etc. We also merged patches by Jim Kleckner, Hoyt Koepke, > Marcus Bitzl, Kirill Smelkov, and Carl Witty. > > Also, a (far from exhaustive) summary of the tickets closed, see: > > http://trac.cython.org/cython_trac/query? > status=closed&milestone=0.9.8.1 > > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From strawman at astraw.com Tue Aug 19 15:37:43 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 19 Aug 2008 15:37:43 +0200 Subject: [Cython] Cython 0.9.8.1.1 Released In-Reply-To: <1E01FAB3-5241-4757-95D8-F9D067485E1D@math.washington.edu> References: <773CCC04-40F1-4EAF-A12F-12D4B14B0AA2@math.washington.edu> <1E01FAB3-5241-4757-95D8-F9D067485E1D@math.washington.edu> Message-ID: <48AACCA7.1010008@astraw.com> Apologies if this has already been noted (I literally just now joined the mailing list), but the page at http://cython.org/ says "The latest release of Cython is 0.9.8.1.1 (released 2008-09-19)", which should probably say "2008-08-19" instead. -Andrew Robert Bradshaw wrote: > Despite a week long beta period, the .pxd files weren't in the final > bundle. A renumber is required for the pypi repository. > > On Aug 18, 2008, at 10:09 PM, Robert Bradshaw wrote: > >> Sources are up in their usual places. The largest addition was Dag >> Seljebotn's buffer support which is amazing. Kudos to him for all his >> hard work (and to Enthought/Google for funding him this summer). Part >> of this effort involved moving to a pipeline/transformation based >> compilation system, which has allowed for many other nice features >> (e.g. long if statements get translated into c switch statements if >> possible). >> >> Another big inclusion is Paul Prescod's pyximport. Now one can simply >> start up Python and type >> >> import pyximport; pyximport.install() >> import foo >> >> which will compile foo.pyx for you. We would like to be able to >> (optionally) specify all options via directives in the files >> themselves, which would make this even more useful. >> >> There are numerous fixes and improvements by Stefan Behnel, Dag >> Seljebotn, and Robert Bradshaw that are not listed here. Among them >> are better support for Py3, unicode, C++, literals, testing >> framework, etc. We also merged patches by Jim Kleckner, Hoyt Koepke, >> Marcus Bitzl, Kirill Smelkov, and Carl Witty. >> >> Also, a (far from exhaustive) summary of the tickets closed, see: >> >> http://trac.cython.org/cython_trac/query? >> status=closed&milestone=0.9.8.1 >> >> >> - Robert >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From strawman at astraw.com Tue Aug 19 16:26:39 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 19 Aug 2008 16:26:39 +0200 Subject: [Cython] python version for conditional compilation? Message-ID: <48AAD81F.1070509@astraw.com> Hi, looking at the Cython Sphinx documentation, I don't see the Python version as a variable I can use for conditional compilation. This would be handy in the case of newly added features in the Python-C API, for example. Would this be feasible to add (alongside UNAME_SYSNAME and friends)? (The location of the docs I was looking at: http://www.mudskipper.ca/cython-doc/docs/language_basics.html#compile-time-definitions ) From ggellner at uoguelph.ca Tue Aug 19 16:37:36 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Tue, 19 Aug 2008 10:37:36 -0400 (EDT) Subject: [Cython] python version for conditional compilation? In-Reply-To: <48AAD81F.1070509@astraw.com> Message-ID: <1229728208.814831219156656914.JavaMail.root@huron.cs.uoguelph.ca> >>----- Original Message ----- >>From: "Andrew Straw" >>To: cython-dev at codespeak.net >>Sent: Tuesday, August 19, 2008 7:26:39 AM GMT -08:00 US/Canada Pacific >>Subject: [Cython] python version for conditional compilation? >> >>Hi, looking at the Cython Sphinx documentation, I don't see the Python >>version as a variable I can use for conditional compilation. This would >>be handy in the case of newly added features in the Python-C API, for >>example. Would this be feasible to add (alongside UNAME_SYSNAME and >>friends)? >> >>(The location of the docs I was looking at: >>http://www.mudskipper.ca/cython-doc/docs/language_basics.html#compile-time-definitions >>) I maintain the Sphinx documentation, I don't fully know these features, but I will try to hunt down some people at Scipy08 that do, and I will write some of this in. Thanks for the heads up. Gabriel From stefan_ml at behnel.de Tue Aug 19 16:42:17 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 19 Aug 2008 16:42:17 +0200 (CEST) Subject: [Cython] Cython 0.9.8.1.1 Released In-Reply-To: <48AACCA7.1010008@astraw.com> References: <773CCC04-40F1-4EAF-A12F-12D4B14B0AA2@math.washington.edu> <1E01FAB3-5241-4757-95D8-F9D067485E1D@math.washington.edu> <48AACCA7.1010008@astraw.com> Message-ID: <54190.213.61.181.86.1219156937.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Andrew Straw wrote: > Apologies if this has already been noted (I literally just now joined > the mailing list), but the page at http://cython.org/ says "The latest > release of Cython is 0.9.8.1.1 (released 2008-09-19)", which should > probably say "2008-08-19" instead. That's ok, we're just emphasizing the fact that Cython is actually faster than light, so you can already see it (and use it) a month before it's released. Stefan PS: thanks for noting that. :) From stefan_ml at behnel.de Tue Aug 19 16:53:15 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 19 Aug 2008 16:53:15 +0200 (CEST) Subject: [Cython] python version for conditional compilation? In-Reply-To: <48AAD81F.1070509@astraw.com> References: <48AAD81F.1070509@astraw.com> Message-ID: <63574.213.61.181.86.1219157595.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Andrew Straw wrote: > Hi, looking at the Cython Sphinx documentation, I don't see the Python > version as a variable I can use for conditional compilation. This would > be handy in the case of newly added features in the Python-C API, for > example. Would this be feasible to add (alongside UNAME_SYSNAME and > friends)? Feasible, yes, but it doesn't make much sense to check the Python version at Cython compile time. The C code that gets generated by Cython will run on all Python versions from 2.3 to 3.0, and on all major platforms. You will loose that if you start generating platform-specific C code for the system that happens to run Cython on your code. Stefan From robertwb at math.washington.edu Tue Aug 19 19:13:37 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 19 Aug 2008 10:13:37 -0700 Subject: [Cython] Cython 0.9.8.1.1 Released In-Reply-To: <48AACCA7.1010008@astraw.com> References: <773CCC04-40F1-4EAF-A12F-12D4B14B0AA2@math.washington.edu> <1E01FAB3-5241-4757-95D8-F9D067485E1D@math.washington.edu> <48AACCA7.1010008@astraw.com> Message-ID: <5CE2A92E-836D-49C1-A747-58CB3B04AF80@math.washington.edu> On Aug 19, 2008, at 6:37 AM, Andrew Straw wrote: > Apologies if this has already been noted (I literally just now joined > the mailing list), but the page at http://cython.org/ says "The latest > release of Cython is 0.9.8.1.1 (released 2008-09-19)", which should > probably say "2008-08-19" instead. > > -Andrew Nice catch. Thanks. - Robert From robertwb at math.washington.edu Tue Aug 19 19:17:51 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 19 Aug 2008 10:17:51 -0700 Subject: [Cython] python version for conditional compilation? In-Reply-To: <63574.213.61.181.86.1219157595.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <48AAD81F.1070509@astraw.com> <63574.213.61.181.86.1219157595.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <8F420D03-ED79-4C2B-BCF3-CD5550469190@math.washington.edu> On Aug 19, 2008, at 7:53 AM, Stefan Behnel wrote: > Andrew Straw wrote: >> Hi, looking at the Cython Sphinx documentation, I don't see the >> Python >> version as a variable I can use for conditional compilation. This >> would >> be handy in the case of newly added features in the Python-C API, for >> example. Would this be feasible to add (alongside UNAME_SYSNAME and >> friends)? > > Feasible, yes, but it doesn't make much sense to check the Python > version > at Cython compile time. The C code that gets generated by Cython > will run > on all Python versions from 2.3 to 3.0, and on all major platforms. > You > will loose that if you start generating platform-specific C code > for the > system that happens to run Cython on your code. If you want to do this I would recommend putting them in an actual .h file (with a cdef extern import block) so that the resulting .c files could be used on multiple Python versions. Out of curiosity, what of the Python/C API do you need? (I'd rather to fix Cython to not have to directly call it.) - Robert From cb at pdos.csail.mit.edu Tue Aug 19 19:59:53 2008 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Tue, 19 Aug 2008 13:59:53 -0400 Subject: [Cython] Any thoughts on a DEFINED keyword? Message-ID: <20080819175953.GA16796@pdos.lcs.mit.edu> Hey, guys. Thanks for considering this. I can think of at least 3 use cases, though there may be more. As with most language mechanisms (beyond a tiny core), one can convert each use case into special higher level substitute that can do a more complete job. There is also charm to just one easy to implement/understand facility. 1) Idempotent inclusion (already described) cimport does not do everything. "cimport *" not working is just one example unlikely to go away. Another would be cross-file inline-ability of tiny little functions. On those occasions when a user needs to 'include', it's helpful to have protections against multiple inclusion types of errors. As per my original post, that the same "do once" behavior can be achieved already through even uglier, lower level compile-time state comparison: DEF FOO_1 = 1 DEF FOO_2 = 1 IF FOO_1 == FOO_2: DEF FOO_2 = 2 ... Yes, this could be replaced with an "include_once" as per Dag's reply, which is better since include-ees need no indent. This is about as easy to implement as DEFINED. 2) Compile-time constant defaulting/overriding In client-module: # client knows objects in the B-tree DEF M = 128 # are small and wants large branches include "Btree.pxi" # to minimize TLB misses/latencies. In btree.pxi: # but the impl module can provide a if not DEFINED M: # reasonable default definition DEF M = 10 int someArray[M] ( Please don't pick on my B-tree paramter example. If one accepts that there is any utility to compile-time parameters, one can surely make one's own e.g. where it is nice to override them. ) Yes, this could also be done with a "WEAKDEF" syntax that would not overwrite an already defined DEF. That's probably about as easy to implement as DEFINED. This general style of thing could also be done even better with a smarter include: defining_include "Btree.pxi" M=2, X=Y, ... which could even scope DEFs of the parameters to the range of the included text. This "parametrized file" form would involve new semantics for any DEF'd name in include-ees as being "weak" which is kind of subtle and is clearly quite a bit more work. 3) Compile-time function availability testing "enumerate" is currently available at compile-time, but "sorted" and "reversed" are not. "oct" and "hex", but no "bin" (not in python either...). Who can anticipate what Python and you guys may want to add (or subtract!) from the builtin namespace? It's nice to let developers phase in the dependence of their code on new Cython versions. It's both simple and Pythonic to just query the namespace to see if what you need is there, only here the queyr is at compile-time rather than run-time. In that light, "has_key" or "HAS_KEY" may be better than "DEFINED". I chose "DEFINED" to match the pattern of "IF"/"DEF". "HAS_KEY" suggests HAS_KEY(COMPILE_ENV, "mystring") which needs two compile-time objects -- the function and dict. Finally, yes, a full compile-time meta-programming facility allowing type inspection and such would be better. However, that requires working out and implementing many details. It seems reasonable in lieu of all that being done to just complete the IF/DEF system with something simple like "DEFINED". cb From stefan_ml at behnel.de Tue Aug 19 20:21:52 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 19 Aug 2008 20:21:52 +0200 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <20080819175953.GA16796@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> Message-ID: <48AB0F40.907@behnel.de> Hi, Chuck Blake wrote: > 1) Idempotent inclusion (already described) > 2) Compile-time constant defaulting/overriding > > In client-module: # client knows objects in the B-tree > DEF M = 128 # are small and wants large branches > include "Btree.pxi" # to minimize TLB misses/latencies. > > In btree.pxi: # but the impl module can provide a > if not DEFINED M: # reasonable default definition > DEF M = 10 > int someArray[M] > > This general style of thing could also be done even better with > a smarter include: > defining_include "Btree.pxi" M=2, X=Y, ... That's a funny idea. We could extend the "include" syntax into include("theinc.pxi", **kwargs) with include "theinc.pxi" being a special case allowed for backward compatibility. The keyword arguments to the include would override any compile time DEFs done in the include. Although a function might imply that include is not a keyword. Maybe include "theinc.pxi" DEF A=1, ... would work better. I'm not totally clear about the semantic implications, BTW, and I'm not sure that the feature /itself/ is a good idea. I just think it doesn't look too bad. > 3) Compile-time function availability testing > > "enumerate" is currently available at compile-time, but "sorted" > and "reversed" are not. "oct" and "hex", but no "bin" (not in > python either...). Who can anticipate what Python and you guys > may want to add (or subtract!) from the builtin namespace? > It's nice to let developers phase in the dependence of their > code on new Cython versions. I think this is a non-issue. New Cython versions are trivial to install. That said, most people who use extensions that were written in Cython will not install Cython at all, and instead rely on the distributors to package the readily generated (and tested) C code with the source releases. So, if you want to use bleading-edge Cython features in your code, you're pretty free to do so without caring too much about older Cython versions. Stefan From dagss at student.matnat.uio.no Tue Aug 19 22:01:59 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 19 Aug 2008 22:01:59 +0200 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <20080819175953.GA16796@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> Message-ID: <48AB26B7.701@student.matnat.uio.no> Chuck Blake wrote: > Hey, guys. Thanks for considering this. > > I can think of at least 3 use cases, though there may be more. > As with most language mechanisms (beyond a tiny core), one can > convert each use case into special higher level substitute that > can do a more complete job. There is also charm to just one > easy to implement/understand facility. This is going to be interesting :-) Yes, most of the usecases I can see better higher-level constructs for (which I'll list briefly below for completeness, without making those the core message of this post I hope). However, you are very right in saying that this provides a solution *now*, not later. It can always be deprecated in some years. However, note that another option is using other macro languages (like m4; or something more modern) to preprocess your Cython files for the same result. This will give you all the power you wish for and more. On the one hand, your patch is already there. On the other hand, C++ meta-programming-through-templates stands as a permanent warning about the dangers of introducing too powerful low-level concepts rather than the appropriate high-level one :-) (I'm still about +0 for this). > 1) Idempotent inclusion (already described) > > cimport does not do everything. "cimport *" not working > is just one example unlikely to go away. Another would > be cross-file inline-ability of tiny little functions. > On those occasions when a user needs to 'include', it's > helpful to have protections against multiple inclusion > types of errors. > > As per my original post, that the same "do once" behavior > can be achieved already through even uglier, lower level > compile-time state comparison: > > DEF FOO_1 = 1 > DEF FOO_2 = 1 > IF FOO_1 == FOO_2: > DEF FOO_2 = 2 > ... > Yes, this could be replaced with an "include_once" as per > Dag's reply, which is better since include-ees need no > indent. This is about as easy to implement as DEFINED. Personally I'd like to get rid of any need for include -- it's the kind of thing that lower-level languages keep around and higher-level languages avoid. I would very much like to implement inlineable code in pxd files (and I am about halfway there, but it's not a priority and I'll have much less time for Cython now after summer). > 2) Compile-time constant defaulting/overriding > > In client-module: # client knows objects in the B-tree > DEF M = 128 # are small and wants large branches > include "Btree.pxi" # to minimize TLB misses/latencies. > > In btree.pxi: # but the impl module can provide a > if not DEFINED M: # reasonable default definition > DEF M = 10 > int someArray[M] > > ( Please don't pick on my B-tree paramter example. If one accepts > that there is any utility to compile-time parameters, one can > surely make one's own e.g. where it is nice to override them. ) No, I think it was an excellent example. We've touched upon both C++-style templates (not likely to come) and a "assumptions" language feature that would give you this thing but as a higher-level concept. This stuff is kind of needed for SciPy people to be able to use Cython for writing library utilities working for multiple data types. So both a quick fix now and a higher level concept later is likely to be welcome for them. (OTOH, they are currently using a preprocessor technique with C to do this, so they could likely just use the same preprocessor for their Cython code until something proper is in place.) -- Dag Sverre From cb at pdos.csail.mit.edu Tue Aug 19 22:10:21 2008 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Tue, 19 Aug 2008 16:10:21 -0400 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <48AB0F40.907@behnel.de> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48AB0F40.907@behnel.de> Message-ID: <20080819201021.GA19222@pdos.lcs.mit.edu> >Although a function might imply that include is not a keyword. Maybe > include "theinc.pxi" DEF A=1, ... Yeah. That's not a bad syntax, either. It would be nice to have this kind of local scoping of compile-time constants/parameterized file. >I'm not totally clear about the semantic implications, BTW, and I'm not sure >that the feature /itself/ is a good idea. I just think it doesn't look too bad. If you find this idea interesting, you might also be interested in the idea of adding just a little more power to compile-time defs to templating power over included files with very little mechanism. E.g., define max_tmpl.pxi: TYPE def NAMES(x)(TYPE a, TYPE b): return a > b if a else b Then a client can simply include "max_tmpl.pxi" TYPE="int", NAMES(x)=("int_" + x) and just use the defined function int_max() as they like... If something like NAMES() can refer to its outer definitions in its own, you can do a nested hierarchy of namespaces: include "max_tmpl.pxi" TYPE="int", NAMES(x)=NAMES("int_" + x) >if you want to use bleading-edge Cython features in your code, you're pretty >free to do so without caring too much about older Cython versions. Eh. I don't think we're in any disagreement. To the extent trivial distutils installs and packaging systems make life easy when you need the latest Cython, great! To the extent Cython allows one to write Cython-version-agnostic or Cython-feature-agnostic code, that's nice, too, but not a major issue. I only called "nice", and I totally grant that this is not the strongest use case for DEFINED (if there were include_once & WEAKDEF or a scoping-defining include). Even the idempotent inclusion bit is only "nice compared to the messier state checking version". Weak compile-time definitions/default parameters is likely the most natural and direct use of something like DEFINED. Better than using DEFINED to implement weak definitions would be a TYPE(name) yielding something like one of {"bool", "int", "float", "str", ""}. The last value, "", could mean "undefined" and obviate DEFINED(name). This would allow more care in using compile-time values. E.g., rounding up if a client DEF's a float where an int belongs or maybe doing something different based on the type of the value. A glance at the compiler source suggests this is a bit harder to do, involving ExprNode values at a later stage in Parsing. Compile-time parameter issues become more interesting if you conceive of things like 'pyximport' being more than a compilation convenience in an edit-debug cycle, but rather a hook to tuning compile-time parameters at run-time based on how they are about to be used by the end user. Under an assumption that you are willing to wait for the C compiler to do its thing, you may be able to give the C compiler more specific info and thus get faster generated code. cb From dagss at student.matnat.uio.no Tue Aug 19 23:32:50 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 19 Aug 2008 23:32:50 +0200 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <20080819201021.GA19222@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48AB0F40.907@behnel.de> <20080819201021.GA19222@pdos.lcs.mit.edu> Message-ID: <48AB3C02.9030706@student.matnat.uio.no> Chuck Blake wrote: > E.g., define max_tmpl.pxi: > > TYPE def NAMES(x)(TYPE a, TYPE b): > return a > b if a else b > > Then a client can simply > > include "max_tmpl.pxi" TYPE="int", NAMES(x)=("int_" + x) > > and just use the defined function int_max() as they like... > > If something like NAMES() can refer to its outer definitions in its own, > you can do a nested hierarchy of namespaces: > > include "max_tmpl.pxi" TYPE="int", NAMES(x)=NAMES("int_" + x) > This made my mind up: I do not think we should reinvent the wheel here. Rather than focusing on improving the DEF, IF and include statements, we should look for an existing macro language for our uses (and rather deprecate DEF, IF and include than make them more powerful). We could even find a preprocessing language that is made "official", distributed with Cython and automatically invoked and all (and even modify it to take care of IF, DEF and include as they are today and refactor this out of core Cython) -- I'm just saying that I'd like these kind of low-level string manipulations to stay out the Cython compiler core and use an existing package (any existing package) for this. The Cython compiler can then be devoted to the higher-level constructs. C has a clear seperation between the language and the preprocessor, with great success. (Still, anything which is already implemented is, well, already implemented and "free" in some sense.). -- Dag Sverre From cb at pdos.csail.mit.edu Wed Aug 20 00:39:16 2008 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Tue, 19 Aug 2008 18:39:16 -0400 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <48AB3C02.9030706@student.matnat.uio.no> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48AB0F40.907@behnel.de> <20080819201021.GA19222@pdos.lcs.mit.edu> <48AB3C02.9030706@student.matnat.uio.no> Message-ID: <20080819223916.GA22105@pdos.lcs.mit.edu> > This made my mind up: I do not think we should reinvent the wheel here. This was just following up what Stefan seemed interested in. It was just an alternative to using the far simpler use case of DEFINED which was merely if not DEFINED(foo) and bears little relation to C++ templates. You do seem to understand that from below, but I wanted to mention it more explicitly. > Rather than focusing on improving the DEF, IF and include statements, we > should look for an existing macro language for our uses (and rather > deprecate DEF, IF and include than make them more powerful). > [..] > C has a clear seperation between the language and the preprocessor, with > great success. That's fine if someone actually does the integration work of running it from the tool chain. Keeping the Cython compiler about more typed files rather than include-like generic file combination and about typed values/names rather than token pasting is probably wise separation of concerns. That being said, until there is some drop-in replacement for DEF/IF/include, why not round out the triplet with DEFINED as any replacement will likely have that sort of feature? Seems like a practical answer until you have some more fully functional textual manipulation system integrated. And Pyrex has DEF/IF/include as well, so you may be stuck with at least only deprecating them/providing them via a different mechanism unless Greg shares your separation biases. He seemed to have trouble getting rid of for "from" and I suspect "include" would share this trait. > (Still, anything which is already implemented is, well, already implemented > and "free" in some sense.). It's pretty simple and less work than this e-mail chain and decidedly less work than any toolchain integration working on Unix, Windows, etc. Best, cb From dagss at student.matnat.uio.no Wed Aug 20 00:54:08 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 00:54:08 +0200 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <20080819223916.GA22105@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48AB0F40.907@behnel.de> <20080819201021.GA19222@pdos.lcs.mit.edu> <48AB3C02.9030706@student.matnat.uio.no> <20080819223916.GA22105@pdos.lcs.mit.edu> Message-ID: <48AB4F10.4000703@student.matnat.uio.no> Chuck Blake wrote: >> This made my mind up: I do not think we should reinvent the wheel here. > > This was just following up what Stefan seemed interested in. It was just > an alternative to using the far simpler use case of DEFINED which was > merely if not DEFINED(foo) and bears little relation to C++ templates. > You do seem to understand that from below, but I wanted to mention it > more explicitly. Yes, I was following up myself, I could have been more clear. For the record, I'm still +0 on Chuck's proposal. -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 20 01:15:38 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 01:15:38 +0200 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <48AB4F10.4000703@student.matnat.uio.no> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48AB0F40.907@behnel.de> <20080819201021.GA19222@pdos.lcs.mit.edu> <48AB3C02.9030706@student.matnat.uio.no> <20080819223916.GA22105@pdos.lcs.mit.edu> <48AB4F10.4000703@student.matnat.uio.no> Message-ID: <48AB541A.3060808@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Chuck Blake wrote: >>> This made my mind up: I do not think we should reinvent the wheel here. >> This was just following up what Stefan seemed interested in. It was just >> an alternative to using the far simpler use case of DEFINED which was >> merely if not DEFINED(foo) and bears little relation to C++ templates. >> You do seem to understand that from below, but I wanted to mention it >> more explicitly. > > Yes, I was following up myself, I could have been more clear. For the > record, I'm still +0 on Chuck's proposal. > Also, I think it is important at this stage for Cython that patches are accepted unless there are good reasons against it, i.e that without any -1 it gets applied. But that is just a personal opinion. -- Dag Sverre From greg.ewing at canterbury.ac.nz Wed Aug 20 03:29:03 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 20 Aug 2008 13:29:03 +1200 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <20080819175953.GA16796@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> Message-ID: <48AB735F.1090003@canterbury.ac.nz> Chuck Blake wrote: > 3) Compile-time function availability testing A separate compile-time directive would be better for this. Doing this with DEFINED would require the runtime built-in functions to be part of the compile- time namespace, which is not currently the case, and IMO it would be conceptually wrong to make it so. -- Greg From robertwb at math.washington.edu Wed Aug 20 03:47:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 19 Aug 2008 18:47:22 -0700 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <20080819175953.GA16796@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> Message-ID: On Aug 19, 2008, at 10:59 AM, Chuck Blake wrote: > Hey, guys. Thanks for considering this. > > I can think of at least 3 use cases, though there may be more. > As with most language mechanisms (beyond a tiny core), one can > convert each use case into special higher level substitute that > can do a more complete job. There is also charm to just one > easy to implement/understand facility. > > 1) Idempotent inclusion (already described) > > cimport does not do everything. "cimport *" not working > is just one example unlikely to go away. "cimport *" works right now. > Another would > be cross-file inline-ability of tiny little functions. > On those occasions when a user needs to 'include', it's > helpful to have protections against multiple inclusion > types of errors. I think we should rather support inline functions in .pxd files. Non- inline functions can be cimported. Includes should be much less used then the are now, but they have their place (when you actually want to textually include code in multiple places.) > > As per my original post, that the same "do once" behavior > can be achieved already through even uglier, lower level > compile-time state comparison: > > DEF FOO_1 = 1 > DEF FOO_2 = 1 > IF FOO_1 == FOO_2: > DEF FOO_2 = 2 > ... > Yes, this could be replaced with an "include_once" as per > Dag's reply, which is better since include-ees need no > indent. This is about as easy to implement as DEFINED. > > 2) Compile-time constant defaulting/overriding > > In client-module: # client knows objects in the B-tree > DEF M = 128 # are small and wants large branches > include "Btree.pxi" # to minimize TLB misses/latencies. > > In btree.pxi: # but the impl module can provide a > if not DEFINED M: # reasonable default definition > DEF M = 10 > int someArray[M] > > ( Please don't pick on my B-tree paramter example. If one accepts > that there is any utility to compile-time parameters, one can > surely make one's own e.g. where it is nice to override them. ) > > Yes, this could also be done with a "WEAKDEF" syntax that would > not overwrite an already defined DEF. That's probably about as > easy to implement as DEFINED. > > This general style of thing could also be done even better with > a smarter include: > defining_include "Btree.pxi" M=2, X=Y, ... > which could even scope DEFs of the parameters to the range of > the included text. This "parametrized file" form would involve > new semantics for any DEF'd name in include-ees as being "weak" > which is kind of subtle and is clearly quite a bit more work. I much prefer DEFINED to WEAKDEF. It's easier to understand and requires less language changes. > > 3) Compile-time function availability testing > > "enumerate" is currently available at compile-time, but "sorted" > and "reversed" are not. "oct" and "hex", but no "bin" (not in > python either...). Who can anticipate what Python and you guys > may want to add (or subtract!) from the builtin namespace? > It's nice to let developers phase in the dependence of their > code on new Cython versions. > > It's both simple and Pythonic to just query the namespace to see > if what you need is there, only here the queyr is at compile-time > rather than run-time. In that light, "has_key" or "HAS_KEY" may > be better than "DEFINED". I chose "DEFINED" to match the pattern > of "IF"/"DEF". "HAS_KEY" suggests HAS_KEY(COMPILE_ENV, > "mystring") > which needs two compile-time objects -- the function and dict. > > Finally, yes, a full compile-time meta-programming facility allowing > type inspection and such would be better. However, that requires > working out and implementing many details. It seems reasonable in > lieu of all that being done to just complete the IF/DEF system with > something simple like "DEFINED". Personally, I find the IF/DEF currently in the compiler to be ugly. However, given that they exist, throwing a DEFINED in there rounds out the options. I would *much* rather see a patch that lets distutils run a preprocesor (your choice of the C preprocessor or something more modern) rather than building more preprocessing into the language itself. - Robert From fperez.net at gmail.com Wed Aug 20 08:03:19 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 19 Aug 2008 23:03:19 -0700 Subject: [Cython] python version for conditional compilation? In-Reply-To: <1229728208.814831219156656914.JavaMail.root@huron.cs.uoguelph.ca> References: <48AAD81F.1070509@astraw.com> <1229728208.814831219156656914.JavaMail.root@huron.cs.uoguelph.ca> Message-ID: On Tue, Aug 19, 2008 at 7:37 AM, Gabriel Gellner wrote: > I maintain the Sphinx documentation, I don't fully know these features, but I will try to hunt down some people at Scipy08 that do, > and I will write some of this in. Thanks for the heads up. Tomorrow, Michael Droetboom will be in the intro tutorial lab teaching ( Moore Laboratory, Room 070 ). He's very well versed in all things Sphinx. Since both Perry and I will do some of the teaching, Michael should have some down time (esp. in the morning where you could ask him). Cheers, f From stefan_ml at behnel.de Wed Aug 20 08:09:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 20 Aug 2008 08:09:02 +0200 Subject: [Cython] preprocessor integration In-Reply-To: References: <20080819175953.GA16796@pdos.lcs.mit.edu> Message-ID: <48ABB4FE.4000405@behnel.de> Hi, Robert Bradshaw wrote: > I think we should rather support inline functions in .pxd files. Non- > inline functions can be cimported. Includes should be much less used > then the are now, but they have their place (when you actually want > to textually include code in multiple places.) > [...] > I would *much* rather see a patch that lets distutils run a > preprocesor (your choice of the C preprocessor or something more > modern) rather than building more preprocessing into the language > itself. Does it have to be distutils running the preprocessor? The use-cases I've seen so far appeared to be based on an import or include, so I think it might be enough to enable the preprocessor only for .pxd files (and maybe .pxi files, depends on the targeted expressiveness of .pxd files). That way, any templates would automatically end up in a separate file (where reuse is possible) and could get configured from within Cython source code. IMHO, that sounds better than inventing a new calling convention from within distutils. And in the worst case, you can always write an almost empty .pyx file with a single cimport, and /maybe/ an additional function call. :) We might also consider making the preprocessor interface pluggable, so that people could register their own preprocessor for a DSL. That way, we'd keep Cython itself free from any futuristic niche language extensions and would just require that whatever people output from their preprocessors is valid .pxd code. You could then ship your preprocessor in a separate package directory of your project sources, or distribute it through PyPI and require it in your setup.py. A plugin interface could work like this: - say this in Cython code: cimport "abc_file.pxdx" a=1 b=... where the generic .pxdx extension enables the preprocessing and the keyword arguments are passed to the preprocessor plugin. I'm not sure about the exact eval semantics here, being able to pass expressions instead of evaluating them at compile time might be nice. Maybe the plugin could get an eval callback into Cython and always do the evaluation itself if it needs to. - have the source file start with a #pre: a.b.xyz_plugin a=2 x="abc" ... line that links it to the a.b.xyz_plugin and defines default arguments for the plugin which get overridden by the arguments in the source code - let Cython import and invoke the a.b.xyz_plugin with the name of the file and all keyword arguments expanded - let the a.b.xyz_plugin return a string that contains a parsable .pxd file, which Cython will use to resolve the cimport in the .pyx file. - maybe add an additional error callback into the plugin that Cython could invoke with a source position to let the plugin provide the 'real' source lines that contained the error. Having other people experiment with their own preprocessors would make it much easier to learn some best practices and cool features, and to fold them back into some built-in preprocessor in the long term, while leaving all niche features out of the core. Stefan From strawman at astraw.com Wed Aug 20 09:16:08 2008 From: strawman at astraw.com (Andrew Straw) Date: Wed, 20 Aug 2008 09:16:08 +0200 Subject: [Cython] python version for conditional compilation? In-Reply-To: <8F420D03-ED79-4C2B-BCF3-CD5550469190@math.washington.edu> References: <48AAD81F.1070509@astraw.com> <63574.213.61.181.86.1219157595.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <8F420D03-ED79-4C2B-BCF3-CD5550469190@math.washington.edu> Message-ID: <48ABC4B8.1080407@astraw.com> Robert Bradshaw wrote: > On Aug 19, 2008, at 7:53 AM, Stefan Behnel wrote: > > >> Andrew Straw wrote: >> >>> Hi, looking at the Cython Sphinx documentation, I don't see the >>> Python >>> version as a variable I can use for conditional compilation. This >>> would >>> be handy in the case of newly added features in the Python-C API, for >>> example. Would this be feasible to add (alongside UNAME_SYSNAME and >>> friends)? >>> >> Feasible, yes, but it doesn't make much sense to check the Python >> version >> at Cython compile time. The C code that gets generated by Cython >> will run >> on all Python versions from 2.3 to 3.0, and on all major platforms. >> You >> will loose that if you start generating platform-specific C code >> for the >> system that happens to run Cython on your code. >> > > Fair enough. But doesn't the UNAME_SYSNAME also make the C code platform-specific? (Not that one step down a slippery slope is an argument to take another. In fact, as noted below, my desire for this feature has dropped to near zero.) > If you want to do this I would recommend putting them in an actual .h > file (with a cdef extern import block) so that the resulting .c files > could be used on multiple Python versions. > Yes, that was my thought as well. But, I'm spoiled by not having to write C code with Cython/Pyrex, so I get lazy. Particularly when I noticed UNAME_SYSNAME, which should allow me to drop the .h file I had already written. > Out of curiosity, what of the Python/C API do you need? (I'd rather > to fix Cython to not have to directly call it.) > I'm looking into PEP 3118 at the moment. I now see the Dag has sort of backported a Cython-ified version of it, though, so in fact I'm dropping my pursuit of the Python/C API in favor of testing the cython-devel branch. This looks very exciting. :) -Andrew From strawman at astraw.com Wed Aug 20 11:07:31 2008 From: strawman at astraw.com (Andrew Straw) Date: Wed, 20 Aug 2008 11:07:31 +0200 Subject: [Cython] patch for bad buffer creation Message-ID: <48ABDED3.60500@astraw.com> Hi, I tracked down an issue that was giving me segfaults with a poor __getbuffer__ implementation and modified the compiler to catch this case. I couldn't figure out how to create an account on Trac, so instead of submitting a patch there, I'm just sending this email. Included is a testcase added to tests/run/bufaccess.pyx. This is a patch against cython-devel and I release this under the PSF license (like the rest of Cython). I had to use "hg diff -b" to create this patch while ignoring whitespace because I have emacs configured to strip trailing whitespace, which caused lots of false positives on the code. So, hopefully, this patch will still apply cleanly. (The patch also includes a tiny fix to the ErrorBuffer implementation in tests/run/bufaccess.pyx.) -Andrew -------------- next part -------------- A non-text attachment was scrubbed... Name: bad_buffer_implementation_check.patch Type: text/x-diff Size: 3035 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080820/d3989a12/attachment-0001.bin From dagss at student.matnat.uio.no Wed Aug 20 12:17:51 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 12:17:51 +0200 Subject: [Cython] patch for bad buffer creation In-Reply-To: <48ABDED3.60500@astraw.com> References: <48ABDED3.60500@astraw.com> Message-ID: <48ABEF4F.1070104@student.matnat.uio.no> Andrew Straw wrote: > Hi, > > I tracked down an issue that was giving me segfaults with a poor > __getbuffer__ implementation and modified the compiler to catch this case. Background first: The situation is that this patch guards against returning erronous data from a GetBuffer call. According to the PEP (3118), I think it is pretty clear that data should either be set or an exception raised, so this patch only helps detect when you create an invalid __getbuffer__ implementation. (If I am wrong and the PEP says that the buf field can be set to NULL on return then I'll of course apply the patch!) I'd like input from others on this before making up my mind permanently, but I'm -1 so far. The reason is that this belongs in a unit test that you would write for your new buffer implementation, without adding another redundtant if-test to all Cython buffer acquisitions that works against correct getbuffer implementations. With a unit test like that, you avoid runtime penalty everywhere, and you also get to test strides, shape etc. for which you cannot insert validation in a generic way in Cython (and so your patch check for correct "buf" but not correct "strides" or "shape"). So what I would propose instead (though I'll never get time for it :-) ) is to have a reusable "buffer tester" library, i.e. use Cython to create an application which checks that a given buffer can be acquired and read, and then feed your custom buffer to it. We could thus create a "standard PEP 3118 test" or similar. For now this would involve manual __Pyx_GetBuffer calls like this: cdef extern from *: cdef int __Pyx_GetBuffer(obj, Py_buffer* info, int flags) except -1 ... releasebuffer .. ... use __Pyx_GetBuffer and unit test the result for your buffer ... (I have planned a function "cython.buffer.bufptr" though which would make this easier; then you'd simply do ... def testAcquisition(self): cdef object[int] mybuf = ... if cython.buffer.bufptr(mybuf) == NULL: self.fail("__getbuffer__ didn't fill out the buf field") ) -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 20 12:20:19 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 12:20:19 +0200 Subject: [Cython] patch for bad buffer creation In-Reply-To: <48ABEF4F.1070104@student.matnat.uio.no> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> Message-ID: <48ABEFE3.8000308@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Andrew Straw wrote: >> Hi, >> >> I tracked down an issue that was giving me segfaults with a poor >> __getbuffer__ implementation and modified the compiler to catch this case. > > I'd like input from others on this before making up my mind permanently, > but I'm -1 so far. The reason is that this belongs in a unit test that *However*, thanks a lot for creating the patch and bringing up the issue! (And I don't make the final call here either, it may go in.) -- Dag Sverre From strawman at astraw.com Wed Aug 20 12:52:11 2008 From: strawman at astraw.com (Andrew Straw) Date: Wed, 20 Aug 2008 12:52:11 +0200 Subject: [Cython] patch for bad buffer creation In-Reply-To: <48ABEF4F.1070104@student.matnat.uio.no> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> Message-ID: <48ABF75B.7000607@astraw.com> Dag Sverre Seljebotn wrote: > Andrew Straw wrote: > >> Hi, >> >> I tracked down an issue that was giving me segfaults with a poor >> __getbuffer__ implementation and modified the compiler to catch this case. >> > > Background first: > > The situation is that this patch guards against returning erronous data > from a GetBuffer call. According to the PEP (3118), I think it is pretty > clear that data should either be set or an exception raised, so this > patch only helps detect when you create an invalid __getbuffer__ > implementation. > I understand your reasoning, but Cython's __getbuffer__ works outside the scope of PEP 3118. The PEP doesn't specify Cython behavior, and a great aspect of Cython, IMO, is that it makes Python/C bridges easier, not to adhere to the minimum standards specified by a PEP. Thus a small amount of additional error checking seems OK to me. Also, as another significant difference, Cython's implementation is working in Python earlier than 2.6, whereas the Python development process will only bring this out with 2.6. (I'm certainly not complaining -- it's just to point out that the PEP doesn't directly apply.) > (If I am wrong and the PEP says that the buf field can be set to NULL on > return then I'll of course apply the patch!) > > I'd like input from others on this before making up my mind permanently, > but I'm -1 so far. The reason is that this belongs in a unit test that > you would write for your new buffer implementation, without adding > another redundtant if-test to all Cython buffer acquisitions that works > against correct getbuffer implementations. > Yes, that unit test would be good to have, regardless. > With a unit test like that, you avoid runtime penalty everywhere, and > you also get to test strides, shape etc. for which you cannot insert > validation in a generic way in Cython (and so your patch check for > correct "buf" but not correct "strides" or "shape"). > Yes, there is the runtime penalty of checking in C whether a pointer is NULL. Given that using buffers often saves copying large amounts of memory, I think that's an acceptable cost. Furthermore, there is already a runtime check in the code (checking the number of dimesions in the buffer using ndim, line 534 of Cython/Compiler/Buffer.py). According to your reasoning above, that check should be removed, too. I would be against that -- my position is that a couple sanity checks costing a single C comparison each is worth fighting segfaults and other undefined behavior. > So what I would propose instead (though I'll never get time for it :-) ) > is to have a reusable "buffer tester" library, i.e. use Cython to create > an application which checks that a given buffer can be acquired and > read, and then feed your custom buffer to it. We could thus create a > "standard PEP 3118 test" or similar. > Yes, as mentioned above, I agree this is a good idea, but I think orthogonal to the issue here. > > *However*, thanks a lot for creating the patch and bringing up the > issue! (And I don't make the final call here either, it may go in.) My pleasure. Thank you for writing this stuff in the first place -- it is very exciting. I especially like the possibility that I can basically start using PEP 3118 now without waiting for Python 2.6 to become standard (or for it to be implemented by the Numpy C API, either). Also, don't forget about the tiny issue of the ErrorBuffer implementation in tests/run/bufaccess.pyx addressed in my patch. -Andrew From dagss at student.matnat.uio.no Wed Aug 20 14:58:02 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 14:58:02 +0200 Subject: [Cython] patch for bad buffer creation In-Reply-To: <48ABF75B.7000607@astraw.com> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> Message-ID: <48AC14DA.1070303@student.matnat.uio.no> Hi, this discussion was interesting. While I may seem to have en edge below I mean it all for the sake of the discussion and hope we can continue to have this "in a good spirit". (Inserting this note was easier than redoing my mail. I'm excited, not angry :-) ) Andrew Straw wrote: > Dag Sverre Seljebotn wrote: > >> Andrew Straw wrote: >> >> >>> Hi, >>> >>> I tracked down an issue that was giving me segfaults with a poor >>> __getbuffer__ implementation and modified the compiler to catch this case. >>> >>> >> Background first: >> >> The situation is that this patch guards against returning erronous data >> from a GetBuffer call. According to the PEP (3118), I think it is pretty >> clear that data should either be set or an exception raised, so this >> patch only helps detect when you create an invalid __getbuffer__ >> implementation. >> >> > I understand your reasoning, but Cython's __getbuffer__ works outside > the scope of PEP 3118. The PEP doesn't specify Cython behavior, and a > Only when the C file is compiled under Python 2.x. When compiling the C file under Python 3, __getbuffer__ fills in the slot specified by the PEP. > great aspect of Cython, IMO, is that it makes Python/C bridges easier, > not to adhere to the minimum standards specified by a PEP. Thus a small > amount of additional error checking seems OK to me. Also, as another > Well, in this case, the error check should be done by inserting some code automatically in the definition of __getbuffer__, so that Python 3 consumers also get advantage of it, and so that we don't second guess how good non-Cython implementations are. So the check is still inserted in the wrong place. (I don't think it is worth it though because you can only check some things but not the most important/non-trivial ones like strides. The author of __getbuffer__ has to know some dirty details anyway. Also __getbuffer__ is supposed to be as close to the metal as possible I think, though I'm likely to follow whatever Stefan's opinion is here) > significant difference, Cython's implementation is working in Python > earlier than 2.6, whereas the Python development process will only bring > this out with 2.6. (I'm certainly not complaining -- it's just to point > out that the PEP doesn't directly apply.) > As it does apply when compiling the Cython code with newer implementations, the PEP does apply because the same Cython code should have the same semantics everywhere. > Yes, there is the runtime penalty of checking in C whether a pointer is > NULL. Given that using buffers often saves copying large amounts of > My concerns aren't runtime performance as such (which is negligible in this case) but keeping the code and design clean. If I read that C code (and didn't know about your reasons behind it), my first thought would be "why is this check here, this cannot ever be NULL by specification". If people start second-guessing implementations of hard specifications "just in case" we will have all sorts of inconsistency and problems (how much second-guessing do you do?) Sprinkling tests more or less at random to trap specific corner cases never makes for good code --- it is a symptom that the whole approach is wrong, and one should test things properly at the right location. > memory, I think that's an acceptable cost. Furthermore, there is already > a runtime check in the code (checking the number of dimesions in the > buffer using ndim, line 534 of Cython/Compiler/Buffer.py). According to > your reasoning above, that check should be removed, too. I would be > I disagree. "ndim" is explicitly provided by the PEP so that you can do this checking, and is used by end-users in a routinely fashion: cdef numpy.ndarray[numpy.int_t, ndim=2] arr = numpy.zeros((2,3,4,5,6), dtype=numpy.int) Here an exception is raised, yet there's nothing wrong with the numpy implementation of __getbuffer__! Adding try/except to deal with this situation dynamically and so on is perfectly OK (and this is intended and documented behaviour), while if __getbuffer__ is broken, it is plain broken. > against that -- my position is that a couple sanity checks costing a > single C comparison each is worth fighting segfaults and other undefined > behavior. Well, the above being said, there is some interest in a compiler directive to insert null-pointer-checks "everywhere". Note that you can currently do cdef ndarray arr = None print arr.shape[0] and this will likely segfault too! There's so many ways for a Cython program to segfault and this must be dealt with in a more systematic manner. >> So what I would propose instead (though I'll never get time for it :-) ) >> is to have a reusable "buffer tester" library, i.e. use Cython to create >> an application which checks that a given buffer can be acquired and >> read, and then feed your custom buffer to it. We could thus create a >> "standard PEP 3118 test" or similar. >> >> > Yes, as mentioned above, I agree this is a good idea, but I think > orthogonal to the issue here. > I mentioned it because a proper unit test for __getbuffer__ in the first place makes the test your patch does redundant, while it traps potentially a lot more and less special cases. And as mentioned above, the problem with redundant code is not performance but program complexity and clarity. > Also, don't forget about the tiny issue of the ErrorBuffer > implementation in tests/run/bufaccess.pyx addressed in my patch. > Thanks (I did forget!), I'll have a look at it. About using PEP 3118, make sure to read the docs for this at http://wiki.cython.org/enhancements/buffer , there's some limits to how far the emulation can get you (i.e. it only works within Cython code and you need to do all the proper cimports of the potential buffer classes even if you are not using anything in the pxd directly. Oh, and inheritance isn't implemented and is likely to break.). Anyway, enough procrastination for me now :-) I believe we understand each other's position, so it is up to other people here to give positive votes to counter my -1. Dag Sverre From dagss at student.matnat.uio.no Wed Aug 20 15:11:22 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 15:11:22 +0200 Subject: [Cython] Cython project for uni students? Message-ID: <48AC17FA.7060507@student.matnat.uio.no> There's a course in my university (University of Oslo) that teaches scripting "technique" in general and Python especially. It's an intermediate-level course and assumes that the students already know programming. The focus is more the way of problem solving with scripting rather than languages as such. Students have the opportunity to select a project (the equivalent of about 7-10 days) to work on and have evaluated, and the course lecturers are positive with my providing them some nicely defined Cython projects. So, any suggestions? Are people be ok with this? As it is only a few days, we're more thinking general "clean up stuff" type of projects than exciting new features. Here's mine: 1) Test system for compiler invocation and usage on file trees. This is not tested today. I.e. test all the behaviour with "whether a pyx file is found in directories with __init__.py" and so on. One should have a script to dynamically create test directories of various structures and then invoke the compiler in various ways (single/multiple) on it. 2) Slightly related: Unit test for invoking the compiler from the command-line, and then rewrite CmdLine.py to use optparse (with backwards-compatible behaviour in all situations), and reverse the relationship between Main.py and CmdLine.py 3) Implement a new compiler directive in order to raise exceptions rather than segfault when attributes are accessed on typed variables set to None. (More hand-holding needed but potentially more exciting and useful, and holding hands may still be less work than doing it ourself + recruitment.) The timeframe for the project is the whole autumn I believe but most students do them around october/november I think. -- Dag Sverre From malkarouri at gmail.com Wed Aug 20 15:32:19 2008 From: malkarouri at gmail.com (Muhammad Alkarouri) Date: Wed, 20 Aug 2008 14:32:19 +0100 Subject: [Cython] Unable to import pyximport In-Reply-To: References: Message-ID: Hi everyone, Looks like pyximport is a cool addition to cython. Unfortunately, after I installed Cython through easy_install) I wasn't able to import pyximport. In [2]: Cython? Type: module Base Class: String Form: Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Cython-0.9.8.1.1-py2.5-macosx-10.3-i386.egg/Cython/__init__.py Docstring: In [3]: import pyximport --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /Users/malkarouri/ in () ImportError: No module named pyximport When I looked in the package, there is a folder pyximport beside Cython (sys.path contains /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Cython-0.9.8.1.1-py2.5-macosx-10.3-i386.egg/, the parent directory of both). But pyximport doesn't contain an __init__.py file. Is that the problem? Many thanks, Muhammad Alkarouri From dagss at student.matnat.uio.no Wed Aug 20 15:52:40 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 20 Aug 2008 15:52:40 +0200 Subject: [Cython] Beta tutorial for NumPy users Message-ID: <48AC21A8.2000608@student.matnat.uio.no> I'm nearly done with my "Cython for NumPy users" tutorial: http://wiki.cython.org/tutorials/numpy I'll post it to numpy-discuss in some hours (and draw attention the new release as well) but if there's anything you'd like mentioned before people start reading please respond. (Of course, it's a wiki, so any improvements just go ahead and do it if you like.) Dag Sverre From dalcinl at gmail.com Wed Aug 20 16:02:48 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 20 Aug 2008 11:02:48 -0300 Subject: [Cython] Cython project for uni students? In-Reply-To: <48AC17FA.7060507@student.matnat.uio.no> References: <48AC17FA.7060507@student.matnat.uio.no> Message-ID: On Wed, Aug 20, 2008 at 10:11 AM, Dag Sverre Seljebotn wrote: > 1) Test system for compiler invocation and usage on file trees. This is > not tested today. I.e. test all the behaviour with "whether a pyx file > is found in directories with __init__.py" and so on. One should have a > script to dynamically create test directories of various structures and > then invoke the compiler in various ways (single/multiple) on it. +1 > 2) Slightly related: Unit test for invoking the compiler from the > command-line, and then rewrite CmdLine.py to use optparse (with > backwards-compatible behaviour in all situations), and reverse the > relationship between Main.py and CmdLine.py +0 > 3) Implement a new compiler directive in order to raise exceptions > rather than segfault when attributes are accessed on typed variables set > to None. (More hand-holding needed but potentially more exciting and > useful, and holding hands may still be less work than doing it ourself + > recruitment.) Definitely +1. I would say that this issue is the only one that really annoys me in Cython. I know, I can (and I do) use 'Type arg is not None' in argument lists, but I never liked it. Furthermore, in other situations, you get segfaulting code if None is not explicitely checked. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Wed Aug 20 20:43:57 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Aug 2008 11:43:57 -0700 (PDT) Subject: [Cython] preprocessor integration In-Reply-To: <48ABB4FE.4000405@behnel.de> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> Message-ID: On Wed, 20 Aug 2008, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> I think we should rather support inline functions in .pxd files. Non- >> inline functions can be cimported. Includes should be much less used >> then the are now, but they have their place (when you actually want >> to textually include code in multiple places.) >> [...] >> I would *much* rather see a patch that lets distutils run a >> preprocesor (your choice of the C preprocessor or something more >> modern) rather than building more preprocessing into the language >> itself. > > Does it have to be distutils running the preprocessor? The use-cases I've seen > so far appeared to be based on an import or include, so I think it might be > enough to enable the preprocessor only for .pxd files (and maybe .pxi files, > depends on the targeted expressiveness of .pxd files). That way, any templates > would automatically end up in a separate file (where reuse is possible) and > could get configured from within Cython source code. IMHO, that sounds better > than inventing a new calling convention from within distutils. And in the > worst case, you can always write an almost empty .pyx file with a single > cimport, and /maybe/ an additional function call. :) > > We might also consider making the preprocessor interface pluggable, so that > people could register their own preprocessor for a DSL. That way, we'd keep > Cython itself free from any futuristic niche language extensions and would > just require that whatever people output from their preprocessors is valid > .pxd code. You could then ship your preprocessor in a separate package > directory of your project sources, or distribute it through PyPI and require > it in your setup.py. > > A plugin interface could work like this: > > - say this in Cython code: > cimport "abc_file.pxdx" a=1 b=... > where the generic .pxdx extension enables the preprocessing and the > keyword arguments are passed to the preprocessor plugin. I'm not sure > about the exact eval semantics here, being able to pass expressions > instead of evaluating them at compile time might be nice. Maybe the plugin > could get an eval callback into Cython and always do the evaluation > itself if it needs to. > > - have the source file start with a > #pre: a.b.xyz_plugin a=2 x="abc" ... > line that links it to the a.b.xyz_plugin and defines default arguments for > the plugin which get overridden by the arguments in the source code > > - let Cython import and invoke the a.b.xyz_plugin with the name of the file > and all keyword arguments expanded > > - let the a.b.xyz_plugin return a string that contains a parsable .pxd file, > which Cython will use to resolve the cimport in the .pyx file. > > - maybe add an additional error callback into the plugin that Cython could > invoke with a source position to let the plugin provide the 'real' source > lines that contained the error. > > Having other people experiment with their own preprocessors would make it much > easier to learn some best practices and cool features, and to fold them back > into some built-in preprocessor in the long term, while leaving all niche > features out of the core. > > Stefan This is an interesting proposal, but I think it is much more heavyweight than I would like. It requires defining a preprocessor API, writing plugins in Python, etc. What about a directive like # cython: preprocess = [gcc -E | m4 | ...] This would pipe the file (be it a .pyx, .pxd, .pxi, .py, whatever) through the preprocessor before parsing, and only the output need be valid Cython code. - Robert From robertwb at math.washington.edu Wed Aug 20 20:47:02 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Aug 2008 11:47:02 -0700 (PDT) Subject: [Cython] patch for bad buffer creation In-Reply-To: <48ABDED3.60500@astraw.com> References: <48ABDED3.60500@astraw.com> Message-ID: On Wed, 20 Aug 2008, Andrew Straw wrote: > Hi, > > I tracked down an issue that was giving me segfaults with a poor > __getbuffer__ implementation and modified the compiler to catch this case. > > I couldn't figure out how to create an account on Trac, Anonymous account creation is disable only because it's the most effective method (so far) of blocking spam. Feel free to email me oflist with an .htpasswd file (or desired username/password) and I'll create an account for you. > so instead of > submitting a patch there, I'm just sending this email. Included is a testcase > added to tests/run/bufaccess.pyx. This is a patch against cython-devel and I > release this under the PSF license (like the rest of Cython). > > I had to use "hg diff -b" to create this patch while ignoring whitespace > because I have emacs configured to strip trailing whitespace, which caused > lots of false positives on the code. So, hopefully, this patch will still > apply cleanly. Thanks. I see Dag has replied, I'm going to go read his comments. > (The patch also includes a tiny fix to the ErrorBuffer implementation in > tests/run/bufaccess.pyx.) > > -Andrew > From robertwb at math.washington.edu Wed Aug 20 20:58:00 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Aug 2008 11:58:00 -0700 (PDT) Subject: [Cython] patch for bad buffer creation In-Reply-To: <48AC14DA.1070303@student.matnat.uio.no> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> <48AC14DA.1070303@student.matnat.uio.no> Message-ID: On Wed, 20 Aug 2008, Dag Sverre Seljebotn wrote: > Hi, > > this discussion was interesting. While I may seem to have en edge below > I mean it all for the sake of the discussion and hope we can continue to > have this "in a good spirit". Yes, me too. Buffer support is a huge and wonderful feature, but given its scope and newness I am sure there is room for improvement. Thanks for bringing this up. > (Inserting this note was easier than redoing my mail. I'm excited, not > angry :-) ) > > Andrew Straw wrote: >> Dag Sverre Seljebotn wrote: >> >>> Andrew Straw wrote: >>> >>> >>>> Hi, >>>> >>>> I tracked down an issue that was giving me segfaults with a poor >>>> __getbuffer__ implementation and modified the compiler to catch this case. >>>> >>>> >>> Background first: >>> >>> The situation is that this patch guards against returning erronous data >>> from a GetBuffer call. According to the PEP (3118), I think it is pretty >>> clear that data should either be set or an exception raised, so this >>> patch only helps detect when you create an invalid __getbuffer__ >>> implementation. >>> >>> >> I understand your reasoning, but Cython's __getbuffer__ works outside >> the scope of PEP 3118. The PEP doesn't specify Cython behavior, and a >> > Only when the C file is compiled under Python 2.x. When compiling the C > file under Python 3, __getbuffer__ fills in the slot specified by the PEP. > >> great aspect of Cython, IMO, is that it makes Python/C bridges easier, >> not to adhere to the minimum standards specified by a PEP. Thus a small >> amount of additional error checking seems OK to me. Also, as another >> > Well, in this case, the error check should be done by inserting some > code automatically in the definition of __getbuffer__, so that Python 3 > consumers also get advantage of it, and so that we don't second guess > how good non-Cython implementations are. So the check is still inserted > in the wrong place. I agree, this is the right place to put the check. And I think it is very worth the (relatively small) overhead to insert a check here. - Robert From robertwb at math.washington.edu Wed Aug 20 21:03:18 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Aug 2008 12:03:18 -0700 (PDT) Subject: [Cython] cython regression ? In-Reply-To: <20080820140530.9a520fe2.simon@arrowtheory.com> References: <20080812115343.ad19b567.simon@arrowtheory.com> <20080812153542.458ad432.simon@arrowtheory.com> <48A1E7F5.7030601@behnel.de> <20080813130123.24810185.simon@arrowtheory.com> <20080813135313.0fbd40ab.simon@arrowtheory.com> <20080813161209.231938cb.simon@arrowtheory.com> <20080814115356.a963adc6.simon@arrowtheory.com> <20080814131014.5f10aa97.simon@arrowtheory.com> <20080815131741.bbbccb40.simon@arrowtheory.com> <20080818130020.81bdcff4.simon@arrowtheory.com> <20080820140530.9a520fe2.simon@arrowtheory.com> Message-ID: On Wed, 20 Aug 2008, Simon Burton wrote: > On Tue, 19 Aug 2008 04:45:19 -0700 > Robert Bradshaw wrote: > >> >> Have you looked at http://wiki.cython.org/PackageHierarchy? >> >> - Robert >> > > That fixed it.. ( include_dirs = ["."] ) Excellent. Hopefully this will work for other people who stumble accross this thread too. > By the way, i did a presentation on cython last night at nyc python user group. > It was (very) well received. Cool. Cython was well recieved at the SciPy 2008 conference as well. > One thing I would suggest is putting a prominent > link to the pyrex docs on the cython front page... yes ? It has been confusing It's the second line under Documentation at http://cython.org. Or were you refering to something else? - Robert From cb at pdos.csail.mit.edu Thu Aug 21 00:08:09 2008 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Wed, 20 Aug 2008 18:08:09 -0400 Subject: [Cython] toolchain integration (was Re: preprocessor integration) In-Reply-To: References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> Message-ID: <20080820220809.GA31485@pdos.lcs.mit.edu> Something worth mentioning at this point is that *proper* preprocessor integration has unelaborated and perhaps unconsidered subtleties. You should be wary of things like cpp caring about lexical structure (not expanding macros inside text that looks like a C strings, deleting text you don't want because it looks like a C comment, as in glob('''foo/*'''), and so on. There are also consequences for the main language parser since one wants to propagate source coordinates to be a good toolchain citizen. Pyrexc/Cython are already not such good citizens. They could but do not put file and line number directives in the generated C. Yes, perhaps in Pyrex's/Cython's case every other output C line would need to be a #line and one may want to turn it off to look at generated code. Yet, it seems to me the default should be to generate these for naive or "regular" users who upon seeing an uninitialized variable warning or what have you at line "foo.pyx:55" not at some line in the generated code. Those who want #file/#line off (to debug the code generator or declutter the generated code) are likely more empowered to easily disable #file/#line generation than those confused by source coordinates and wanting "unused variable" and "unitialized variable" type warnings. :) Even more significantly than things like data flow gcc warnings, this would let me set a breakpoint in GDB like "breakpoint foo.pyx:55". Yes, gdb doesn't yet understand Cython, and I'd have to use "inf loc" or something to look at the local variables, but it would be nice to even just make it easier to stop at the right Cython function (rather than mangling or mucking about in source code). Source coordinate propagation is a good thing. I actually have a little profiling toolkit that would be able to tell me which line numbers of pyx were my performance problems, if only the coordinates were propagated. If you really want to ditch include/IF/DEF/DEFINED/etc. and use some more general tool, that tool could/should generate #file or #line directives. Then on the input side of the equation, Cython should parse those in and propagate them (this INPUT side of things is on the original Pyrex TODO list, but strangely the OUTPUT which is needed to make 'gdb' or 'addr2line' and such easy to use wasn't on the list -- there are some e-mail refs, though, so this may have all come up before). { Cython actually has discretion to require the input directives be spelled some way besides "#file" if you don't want to support cpp. The # spelling seems pretty reasonable to me. Cython just has to actually parse that kind of comment rather than ignore it. } Just some issues to keep in mind for anyone that wants to work on this... cb From robertwb at math.washington.edu Thu Aug 21 01:37:38 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Aug 2008 16:37:38 -0700 (PDT) Subject: [Cython] toolchain integration (was Re: preprocessor integration) In-Reply-To: <20080820220809.GA31485@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> <20080820220809.GA31485@pdos.lcs.mit.edu> Message-ID: On Wed, 20 Aug 2008, Chuck Blake wrote: > Something worth mentioning at this point is that *proper* preprocessor > integration has unelaborated and perhaps unconsidered subtleties. > > You should be wary of things like cpp caring about lexical structure > (not expanding macros inside text that looks like a C strings, > deleting text you don't want because it looks like a C comment, > as in glob('''foo/*'''), and so on. Yes, these are good points. I don't want to go down the path of trying to add our own special pre-processing language into Cython... > There are also consequences for the main language parser since one > wants to propagate source coordinates to be a good toolchain citizen. > Pyrexc/Cython are already not such good citizens. They could but > do not put file and line number directives in the generated C. > > Yes, perhaps in Pyrex's/Cython's case every other output C line would > need to be a #line and one may want to turn it off to look at generated > code. Yet, it seems to me the default should be to generate these > for naive or "regular" users who upon seeing an uninitialized variable > warning or what have you at line "foo.pyx:55" not at some line in the > generated code. Those who want #file/#line off (to debug the code > generator or declutter the generated code) are likely more empowered > to easily disable #file/#line generation than those confused by source > coordinates and wanting "unused variable" and "unitialized variable" > type warnings. :) As a bit of defense, this is because nearly all C-level compile warndings/errors are not usually tied to a specific line in a .pyx file (e.g. unsed variable warnings are nearly always automatically created ones). And when trying to diagnose segfaults pyx file line numbers are usually not granular enough to figure out what is going on (for me at least). The correspondance between C and pyx code is far from one-to-one, and likely to diverge even further in the future. > Even more significantly than things like data flow gcc warnings, this > would let me set a breakpoint in GDB like "breakpoint foo.pyx:55". > Yes, gdb doesn't yet understand Cython, and I'd have to use "inf loc" > or something to look at the local variables, but it would be nice to > even just make it easier to stop at the right Cython function (rather > than mangling or mucking about in source code). If you declare function names to be "public" then the name will not get mangled. > Source coordinate propagation is a good thing. I actually have a > little profiling toolkit that would be able to tell me which line > numbers of pyx were my performance problems, if only the coordinates > were propagated. That sounds very cool. I would love to have something like that. > If you really want to ditch include/IF/DEF/DEFINED/etc. and use some more > general tool, that tool could/should generate #file or #line directives. > Then on the input side of the equation, Cython should parse those in and > propagate them (this INPUT side of things is on the original Pyrex TODO list, > but strangely the OUTPUT which is needed to make 'gdb' or 'addr2line' and > such easy to use wasn't on the list -- there are some e-mail refs, though, > so this may have all come up before). { Cython actually has discretion to > require the input directives be spelled some way besides "#file" if you don't > want to support cpp. The # spelling seems pretty reasonable to me. Cython > just has to actually parse that kind of comment rather than ignore it. } > > Just some issues to keep in mind for anyone that wants to work on this... I would be in favor of an option that outputs #file and #line directives. Perhaps we could accept this (or something similar) as input too, though I'm not sure what the best syntax is. I think the better solution for this is to output a symbol table which relates Cython line numbers/variables/functions/... with their C counterparts (much like gcc does with -g) which debuggers and profilers could read. - Robert From greg.ewing at canterbury.ac.nz Thu Aug 21 02:28:25 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 21 Aug 2008 12:28:25 +1200 Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: References: <48AC17FA.7060507@student.matnat.uio.no> Message-ID: <48ACB6A9.2050905@canterbury.ac.nz> Lisandro Dalcin wrote: > I know, I can (and I do) use 'Type arg is not > None' in argument lists, but I never liked it. Speaking of that, I've been thinking for a while now about making "not None" the default, and requiring "or None" to specify that None is an acceptable parameter value. Having seen a number of pieces of Pyrex code that were susceptible to being crashed because "not None" wasn't used where it should have been, I think this would be a safer default. The change may break some existing code, though. What do people think? -- Greg From greg.ewing at canterbury.ac.nz Thu Aug 21 03:04:53 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 21 Aug 2008 13:04:53 +1200 Subject: [Cython] toolchain integration (was Re: preprocessor integration) In-Reply-To: <20080820220809.GA31485@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> <20080820220809.GA31485@pdos.lcs.mit.edu> Message-ID: <48ACBF35.2090105@canterbury.ac.nz> Chuck Blake wrote: > Pyrexc/Cython are already not such good citizens. They could but > do not put file and line number directives in the generated C. There are reasons for that. In theory, Pyrex should never generate invalid C code, so if there are any C compilation errors, I usually want to see where they came from in the C file so that I can figure out what went wrong. I know this doesn't apply to things like misspelled names referring to external C functions, but there's no way of selectively reporting .pyx file positions just for those, and they're usually easy to track down by searching for the relevant name. There's also the problem that some of the code Pyrex generates isn't clearly attributable to any particular place in a .pxd or .pyx file. -- Greg From stefan_ml at behnel.de Thu Aug 21 07:20:03 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Aug 2008 07:20:03 +0200 Subject: [Cython] python version for conditional compilation? In-Reply-To: <8F420D03-ED79-4C2B-BCF3-CD5550469190@math.washington.edu> References: <48AAD81F.1070509@astraw.com> <63574.213.61.181.86.1219157595.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <8F420D03-ED79-4C2B-BCF3-CD5550469190@math.washington.edu> Message-ID: <48ACFB03.8010904@behnel.de> Hi, Robert Bradshaw wrote: > On Aug 19, 2008, at 7:53 AM, Stefan Behnel wrote: > >> Andrew Straw wrote: >>> Hi, looking at the Cython Sphinx documentation, I don't see the >>> Python >>> version as a variable I can use for conditional compilation. This >>> would >>> be handy in the case of newly added features in the Python-C API, for >>> example. Would this be feasible to add (alongside UNAME_SYSNAME and >>> friends)? >> Feasible, yes, but it doesn't make much sense to check the Python >> version >> at Cython compile time. The C code that gets generated by Cython >> will run >> on all Python versions from 2.3 to 3.0, and on all major platforms. >> You >> will loose that if you start generating platform-specific C code >> for the >> system that happens to run Cython on your code. > > If you want to do this I would recommend putting them in an actual .h > file (with a cdef extern import block) so that the resulting .c files > could be used on multiple Python versions. No .h file required as the Python version is already in Python's patchlevel.h. A simple cdef extern from *: cdef int PY_VERSION_HEX ... if PY_VERSION_HEX >= 0x02050000: handle_py2_5() will do the trick. Note that I use "if", not "IF" here. The C compiler will do the right thing. I also added a Cython/Include/python_version.pxd that you can use. Stefan From stefan_ml at behnel.de Thu Aug 21 07:26:03 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Aug 2008 07:26:03 +0200 Subject: [Cython] Any thoughts on a DEFINED keyword? In-Reply-To: <53749.193.157.229.67.1219123623.squirrel@webmail.uio.no> References: <53749.193.157.229.67.1219123623.squirrel@webmail.uio.no> Message-ID: <48ACFC6B.40709@behnel.de> Hi, Dag Sverre Seljebotn wrote: > if using cimport is absolutely impossible, > perhaps that is an argument for include_once or similar What would the use case for an "include_once"? Stefan From cb at pdos.csail.mit.edu Thu Aug 21 09:13:27 2008 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Thu, 21 Aug 2008 03:13:27 -0400 Subject: [Cython] toolchain integration (was Re: preprocessor integration) In-Reply-To: <48ACBF35.2090105@canterbury.ac.nz> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> <20080820220809.GA31485@pdos.lcs.mit.edu> <48ACBF35.2090105@canterbury.ac.nz> Message-ID: <20080821071327.GA8080@pdos.lcs.mit.edu> I'm not sure you read what I said very carefully { about use case 3, too, but that use case is admittedly pretty lame :-/ }. Greg Ewing wrote: >There are reasons for that. In theory, Pyrex should never >generate invalid C code, so I never mentioned compiler errors, but only warnings. Obviously, errors are more important for the code generator writer, as are file & line numbers in the generated code. Yet, some warnings come from data flow analysis in gcc. Yes, you could reinvent such analysis. It sure seems easier to emit #line and let gcc (or whatever compiler) tell users and try to keep warnings produced by the generated code to a minimum. They're already very thin. I've mostly only gotten incompletely filled-in struct initializer warnings. >if there are any C compilation errors, I usually want to >see where they came from in the C file so that I can figure >out what went wrong. I noted that more powerful users [ system authors obviously included :) ] might prefer generated lines but are also more empowered to flip the switch to get whichever they want. Of course, I'd just happy to be able to turn it on any old way, (and don't really expect to be able to any time soon). Just trying to motivate the best default and motivate the behavior... >there's no way of selectively reporting .pyx file positions just for those Well, don't be selective. :) Just emit a #line for everything and let the whole feature be turned on or off. Easy or not to search through my pyx source, it still seems better to have a language system refer to as high-level source coordinates as possible. Yacc and other C-generating preprocessors or C-targeting compilers do this. File, line, and maybe function coordinates are a tenuous but useful thread connecting the highest level to the lowest in language toolchains. >There's also the problem that some of the code Pyrex >generates isn't clearly attributable to any particular >place in a .pxd or .pyx file. The same is true of assembly from C or C from Yacc as mentioned above. A more significant issue than missing values is that with certain kinds of optimization (e.g. common subexpression elimination) you can get a one->many mapping of what is generated to the source (since multiple source addresses are all related to one generated bit). I don't know if any sorts of transforms like that happen currently in going from Cython But, consider -- minor imperfections (loose ends and knots in the thread, if you will) don't seem to degrade the value much in practice (at least mine). If you have the thread 99.9% of the times you need it, and it often saves 5X or who knows how much human time in addressing some issue at the pyx source to be told where something is based on the source you edit vs. generated code..., well, it seems a pretty clear win. { To NOT be a win, it would have to be thousands of times harder to deal with things in the 0.1% of the time you don't have the thread. It's not clearly any harder at all. Maybe just strip it with sed -i -e'/^#file/d' -e'/^#line/d' foo.c or cython --no-coords or something. } I'm not saying it's not "work", and other judgements are of course in play as to why it's not present either historically or on anyone's task list. Cython's current ability to emit the source lines with <<<<< pointers makes me strongly suspect that it's not much work. Parsing upstream/pre-processed directives like #file and #line is likely more work, and more valuable as a pre-processor toolchain becomes done as per Stefan's ambitious description. I argue here more about "shoulds" and reasons for things, only because you put it in those terms. This kind of feature is the sort that language designers often see less value in because they can read what is generated easily enough. More external, casual and/or novice language users really do appreciate it. If the point of Cython is to NOT have to learn and use and deal with all the guts of the Python C API, then the better integrated the tools are, the better the user experience. E.g., I'm pretty sure from using it that Cython --annotate to generate HTML of what is most "Pythonic" is pretty heuristic and approximate. Yet, it's still useful and interesting. Yes, *I'm* fine with reading generated code to find out what I need, but I've been using Python 16 years, seen Great Renamings in the C API come & go and I'm the sort to disassemble his code to see what the C compiler is up to. There is a LOT of Python C API noise that can be fore- boding to less intrepid users. Higher level/propagated source coords are just one more thing in the vein of making less necessary to look at the generated code. Source-level debugging in gdb with a "backtrace" that understood PyObjectCall and the ability to "print " Cython expressions would be a dream I barely dare utter (and also likely a mountain of work).....HOWEVER... Robert Bradshaw wrote: >> Source coordinate propagation is a good thing. I actually have a >> little profiling toolkit that would be able to tell me which line >> numbers of pyx were my performance problems, if only the coordinates >> were propagated. >That sounds very cool. I would love to have something like that. .....if you give me source coordinate propagation, then I can at least give people I work with (or others) source-level profiling with almost no effort, at least on Linux. http://pdos.csail.mit.edu/cb/pct All you do is make sure -g is on in all your CFLAGS and then you can just go "PCT_FMT=line% profile python myEntryPoint". Code in modules that Python dlopen()s is tracked on Linux as it is used. { All you need to do is re-read /proc/PID/maps whenever an unknown address is found, and record where in virutal memory the .so is; Wrapping dlopen() with LD_PRELOAD would be a more direct & efficient, but invasive/tricky approach }. I believe that if you emit #file and #line correctly that the rest of the toolchain including things like "addr2line" will just work. I've profiled C, C++, Java/gcj, Fortran, and even gdc, the GNU D compiler, all with the gnu toolchain with no trouble and yielding pretty interpretable results. In the case of Python loading modules from Cython and SciPy/NumPy you can get pretty interestingly mixed multi-language profiling reports, actually, with C API things, my own Cython code, and even Fortran lib calls in there. At the moment, I just get by with function-level (with mangled names (or not with 'public')) profiling of Cython code, or line-level reports on the generated C. That is, again, kind of fine for me, but it would be nicer to be able to give a little more to others. A line-level report would be complementary to cython --annotate. The former would tell you which regions get run a lot while the latter which are "Pythonic". The combination of both "Pythonic" and used a lot is almost exactly what you should be paying attention to in terms of missing cdef's/easy speed-up opportunities while not wasting time optimizing code that doesn't get run a lot. Why, it wouldn't be too hard to make a source-level HTML annotater for my profiler and then have a browser where you looked at a sizeable Cython program in the two mark-ups side-by-side, one for Pythonic-ness and one for CPU-time-hogging. There's already an annotated disassembly mark-up, but without HTML. Another possibility would be to try to combine or weight the scores into an overall "pay attention to me" score. All the best! cb From greg.ewing at canterbury.ac.nz Thu Aug 21 09:35:36 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 21 Aug 2008 19:35:36 +1200 Subject: [Cython] toolchain integration (was Re: preprocessor integration) In-Reply-To: <20080821071327.GA8080@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> <20080820220809.GA31485@pdos.lcs.mit.edu> <48ACBF35.2090105@canterbury.ac.nz> <20080821071327.GA8080@pdos.lcs.mit.edu> Message-ID: <48AD1AC8.30800@canterbury.ac.nz> Chuck Blake wrote: > I'm not sure you read what I said very carefully { about use case 3, too, > but that use case is admittedly pretty lame :-/ }. I didn't mean to say you shouldn't change it. I was just explaining how it came to be the way it is now. My original intention was that Pyrex would always generate correct C code, so the user should never see an error (or even a warning) from the C compiler. Given that, there was no need to emit #line directives, and in fact some reasons not to. It's become apparent since that it's not feasible for Pyrex to guarantee correct C code in all cases, since there's always a possibility of a mismatch between Pyrex declarations and external .h files. So there may be some benefit in generating #line directives after all. -- Greg From robertwb at math.washington.edu Thu Aug 21 10:13:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Aug 2008 01:13:31 -0700 Subject: [Cython] toolchain integration (was Re: preprocessor integration) In-Reply-To: <20080821071327.GA8080@pdos.lcs.mit.edu> References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> <20080820220809.GA31485@pdos.lcs.mit.edu> <48ACBF35.2090105@canterbury.ac.nz> <20080821071327.GA8080@pdos.lcs.mit.edu> Message-ID: On Aug 21, 2008, at 12:13 AM, Chuck Blake wrote: > I'm not sure you read what I said very carefully { about use case > 3, too, > but that use case is admittedly pretty lame :-/ }. > > Greg Ewing wrote: >> There are reasons for that. In theory, Pyrex should never >> generate invalid C code, so > > I never mentioned compiler errors, but only warnings. Obviously, > errors are > more important for the code generator writer, as are file & line > numbers in > the generated code. Yet, some warnings come from data flow > analysis in gcc. > Yes, you could reinvent such analysis. It sure seems easier to > emit #line > and let gcc (or whatever compiler) tell users and try to keep warnings > produced by the generated code to a minimum. They're already very > thin. > I've mostly only gotten incompletely filled-in struct initializer > warnings. I second the fact that Pyrex should never generate invalid C code. Short of invalid extern declarations, have you ever had any C errors/ warnings that were due to bad Pyrex/Cython code? >> if there are any C compilation errors, I usually want to >> see where they came from in the C file so that I can figure >> out what went wrong. > > I noted that more powerful users [ system authors obviously > included :) ] > might prefer generated lines but are also more empowered to flip > the switch > to get whichever they want. Of course, I'd just happy to be able > to turn > it on any old way, (and don't really expect to be able to any time > soon). > Just trying to motivate the best default and motivate the behavior... If there are errors, it probably won't be fixed by changing the pyx file, rather the compiler itself needs fixing. > I'm not saying it's not "work", and other judgements are of course > in play > as to why it's not present either historically or on anyone's task > list. > Cython's current ability to emit the source lines with <<<<< > pointers makes > me strongly suspect that it's not much work. No, technically it should be relatively easy to do (though eliminating spurious associations in code that follows associated code). > Parsing upstream/pre-processed > directives like #file and #line is likely more work, and more > valuable as a > pre-processor toolchain becomes done as per Stefan's ambitious > description. > > I argue here more about "shoulds" and reasons for things, only > because you put > it in those terms. This kind of feature is the sort that language > designers > often see less value in because they can read what is generated > easily enough. > More external, casual and/or novice language users really do > appreciate it. > If the point of Cython is to NOT have to learn and use and deal > with all the > guts of the Python C API, then the better integrated the tools are, > the better > the user experience. Sure. Currently, exposing the C doesn't have much benifit for such casual users. > E.g., I'm pretty sure from using it that Cython --annotate to > generate HTML of > what is most "Pythonic" is pretty heuristic and approximate. Yet, > it's still > useful and interesting. Yes, *I'm* fine with reading generated > code to find > out what I need, but I've been using Python 16 years, seen Great > Renamings in > the C API come & go and I'm the sort to disassemble his code to see > what the > C compiler is up to. There is a LOT of Python C API noise that can > be fore- > boding to less intrepid users. > > Higher level/propagated source coords are just one more thing in > the vein of > making less necessary to look at the generated code. Source-level > debugging > in gdb with a "backtrace" that understood PyObjectCall and the > ability to > "print " Cython expressions would be a dream I barely dare utter > (and also > likely a mountain of work).....HOWEVER... If someone could write something that understands PyObjectCall, that would be awesome. I think such a thing would be a wrapper over, say, gdb. > > Robert Bradshaw wrote: >>> Source coordinate propagation is a good thing. I actually have a >>> little profiling toolkit that would be able to tell me which line >>> numbers of pyx were my performance problems, if only the coordinates >>> were propagated. >> That sounds very cool. I would love to have something like that. > > .....if you give me source coordinate propagation, then I can at > least give > people I work with (or others) source-level profiling with almost > no effort, > at least on Linux. > > http://pdos.csail.mit.edu/cb/pct > > All you do is make sure -g is on in all your CFLAGS and then you > can just > go "PCT_FMT=line% profile python myEntryPoint". Code in modules > that Python > dlopen()s is tracked on Linux as it is used. { All you need to do > is re-read > /proc/PID/maps whenever an unknown address is found, and record > where in > virutal memory the .so is; Wrapping dlopen() with LD_PRELOAD would > be a > more direct & efficient, but invasive/tricky approach }. > > I believe that if you emit #file and #line correctly that the rest > of the > toolchain including things like "addr2line" will just work. I've > profiled > C, C++, Java/gcj, Fortran, and even gdc, the GNU D compiler, all > with the > gnu toolchain with no trouble and yielding pretty interpretable > results. > > In the case of Python loading modules from Cython and SciPy/NumPy > you can > get pretty interestingly mixed multi-language profiling reports, > actually, > with C API things, my own Cython code, and even Fortran lib calls > in there. > At the moment, I just get by with function-level (with mangled > names (or > not with 'public')) profiling of Cython code, or line-level reports > on the > generated C. > > That is, again, kind of fine for me, but it would be nicer to be > able to > give a little more to others. A line-level report would be > complementary > to cython --annotate. The former would tell you which regions get > run a lot > while the latter which are "Pythonic". The combination of both > "Pythonic" > and used a lot is almost exactly what you should be paying > attention to in > terms of missing cdef's/easy speed-up opportunities while not > wasting time > optimizing code that doesn't get run a lot. > > Why, it wouldn't be too hard to make a source-level HTML annotater > for my > profiler and then have a browser where you looked at a sizeable Cython > program in the two mark-ups side-by-side, one for Pythonic-ness and > one > for CPU-time-hogging. There's already an annotated disassembly > mark-up, > but without HTML. Another possibility would be to try to combine or > weight the scores into an overall "pay attention to me" score. > > All the best! Thanks. I'll definitely plan to take a look, and I will certainly add #file and #line output to Cython (but I'm still not convinced that it should be on by default). - Robert. From robertwb at math.washington.edu Thu Aug 21 10:16:40 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Aug 2008 01:16:40 -0700 Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <48ACB6A9.2050905@canterbury.ac.nz> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> Message-ID: <1ECBED99-F639-41CF-AE4B-1FEAA066F5AC@math.washington.edu> On Aug 20, 2008, at 5:28 PM, Greg Ewing wrote: > Lisandro Dalcin wrote: > >> I know, I can (and I do) use 'Type arg is not >> None' in argument lists, but I never liked it. > > Speaking of that, I've been thinking for a while now > about making "not None" the default, and requiring > "or None" to specify that None is an acceptable > parameter value. > > Having seen a number of pieces of Pyrex code that > were susceptible to being crashed because "not None" > wasn't used where it should have been, I think > this would be a safer default. The change may > break some existing code, though. > > What do people think? I use the fact that extension types can be None all over the place. Is this exclusively for arguments? Perhaps that could be OK--at least it wouldn't break existing code silently. What would it be called? "or None"? I think a mode that emits tests before every cdef attribute access to verify validity could be very useful as well. - Robert From dagss at student.matnat.uio.no Thu Aug 21 13:14:35 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 13:14:35 +0200 Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: <48ACB6A9.2050905@canterbury.ac.nz> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> Message-ID: <48AD4E1B.3080506@student.matnat.uio.no> Greg Ewing wrote: > Lisandro Dalcin wrote: > >> I know, I can (and I do) use 'Type arg is not >> None' in argument lists, but I never liked it. > > Speaking of that, I've been thinking for a while now > about making "not None" the default, and requiring > "or None" to specify that None is an acceptable > parameter value. > > Having seen a number of pieces of Pyrex code that > were susceptible to being crashed because "not None" > wasn't used where it should have been, I think > this would be a safer default. The change may > break some existing code, though. > > What do people think? > For Cython at least, I'm -1 on this. The reason is that it breaks backwards compatability (although admittedly in a nice way) while there is another kind of behaviour I would prefer to move to: That None is allowed, but is checked for so that it is safe. The trivial solution might slow things down a lot, but going beyong the trivial isn't horribly difficult either by using a visitor [1]. What I'd like to see in addition then is some kind of type specifier saying that a variable cannot be None which could also be used for variables. cdef MyClass not None var is obviously problematic, but that's what I'd like just with a better syntax. Something like cdef cython.notnone(MyClass) var cdef notnone MyClass var cdef MyClass! var or something along these lines. The effect of this would be to have the check happen at variable assignment rather than attribute lookup. Finally, a compiler directive could be set to assume that no AttributeErrors (with respect to None) occurs to remove these checks, but it would not be the default. I know, I have lots of plans that I can never get to the bottom of, but at least I feel the existance of this solution means one should at least wait with changing the "not None" behaviour for Cython. [1] Like just checking if you're within an if-test where the local variable was checked to not be None + only the first attribute lookup on a local variable in the same block needs a check, because an exception drops you out of it. These two rules gets you a long way. -- Dag Sverre From dagss at student.matnat.uio.no Thu Aug 21 13:42:09 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 13:42:09 +0200 Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: <48AD4E1B.3080506@student.matnat.uio.no> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <48AD4E1B.3080506@student.matnat.uio.no> Message-ID: <48AD5491.6050100@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > [1] Like just checking if you're within an if-test where the local > variable was checked to not be None + only the first attribute lookup on > a local variable in the same block needs a check, because an exception > drops you out of it. These two rules gets you a long way. > (All this Cython only but keeping the thread intact...) Sorry: And any assignments to a local variable will reset the state to now known, that must be dealt with too. I'm not saying that this is a magic bullet, just that code like cdef MyClass x = ... if x is not None: block without assignment to x could be a convenient way to declare intent and generate efficient code (kind of like how if-statements are currently translated to switch statements under certain strict conditions) (Hmm, in that view, a special type like cython.nonnone(MyClass) is perhaps not needed, an "assert x is not None" after every assignment to x could fill the same role) -- Dag Sverre From stefan_ml at behnel.de Thu Aug 21 13:44:46 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Aug 2008 13:44:46 +0200 (CEST) Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <1ECBED99-F639-41CF-AE4B-1FEAA066F5AC@math.washington.edu> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <1ECBED99-F639-41CF-AE4B-1FEAA066F5AC@math.washington.edu> Message-ID: <54333.213.61.181.86.1219319086.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Aug 20, 2008, at 5:28 PM, Greg Ewing wrote: >> I've been thinking for a while now >> about making "not None" the default, and requiring >> "or None" to specify that None is an acceptable >> parameter value. > > I use the fact that extension types can be None all over the place. > Is this exclusively for arguments? Yes, I think that's what Greg meant. That's the only place where this makes sense: user input into functions that use the Python calling convention. This change would affect user code at the Python API level, which I find a pretty huge impact. > Perhaps that could be OK--at least > it wouldn't break existing code silently. Well, it would still break it, though. And silent or not depends on there being enough tests in place that check if the sematically changed function really accepts None as input. I'd certainly not expect 100% coverage for this in any test suite. >From my gut feeling, I would say that in most places in lxml where I allow None to be passed for an extension type, it's in optional arguments, i.e. arguments that default to None anyway. So this wouldn't break too much there. But there are definitely other places where I do the check explicitly in the code and not in the signature, as I want to check for certain other input characteristics at the same time (or correlations with other parameters). Those places might change behaviour without easily being noticed, e.g. raise a different exception or at least a different (misleading?) exception message. > What would it be called? "or None"? That would be a sensible notation IMHO, although "or None" in Python code has slightly different semantics. It would return None whenever the parameter /itself/ considers its boolean value False, which is not the same as "being None". > I think a mode that emits tests before every cdef attribute access to > verify validity could be very useful as well. Such an option would certainly be nice for testing. Stefan From strawman at astraw.com Thu Aug 21 14:50:13 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 21 Aug 2008 14:50:13 +0200 Subject: [Cython] patch for bad buffer creation In-Reply-To: References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> <48AC14DA.1070303@student.matnat.uio.no> Message-ID: <48AD6485.7060405@astraw.com> Robert Bradshaw wrote: >>> Dag Sverre Seljebotn wrote: >>> >>>> Andrew Straw wrote: >>> great aspect of Cython, IMO, is that it makes Python/C bridges easier, >>> not to adhere to the minimum standards specified by a PEP. Thus a small >>> amount of additional error checking seems OK to me. Also, as another >>> >> Well, in this case, the error check should be done by inserting some >> code automatically in the definition of __getbuffer__, so that Python 3 >> consumers also get advantage of it, and so that we don't second guess >> how good non-Cython implementations are. So the check is still inserted >> in the wrong place. > > I agree, this is the right place to put the check. And I think it is very > worth the (relatively small) overhead to insert a check here. Just to be clear (re-reading this bit after Robert's response makes me realize I'm confused about your meaning): what do you mean by "the definition of __getbuffer__" -- the C Python implementation of PyObject_GetBuffer()? I thought my patch did affect the (Cython) definition of __getbuffer__. Also, wouldn't Cython emit the same C code for Python 3? -Andrew From dagss at student.matnat.uio.no Thu Aug 21 15:34:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 15:34:12 +0200 Subject: [Cython] Prototype for None behaviour Message-ID: <48AD6ED4.7070306@student.matnat.uio.no> Rather than just talking I prototyped my last suggestion about None behaviour, it is up in the -dagss branch: http://hg.cython.org/cython-dagss/rev/4d543e06c926 Run the test "noneattributeacc" without cleanup and inspect the C file. The exceptional circumstances under which it will work are too numerous too mention, I basically made a testcase and only did the very minimum for that to work (so this is only marginally better than just talking), but it does work and I kind of consider it proof of concept. It is possible to improve it a lot, but it will likely never be able to optimize flag = (x is not None) if flag: ... However, we can make it so that if you help it like this, flag = (x is not None) if flag: assert x is not None # or cython.check_None(x) then things get optimized. So, in effect, I think this makes a case for steering Cython in a direction where _safe access for None is always enabled by default_. Here's the result, my own comments after # sign. __pyx_1 = (__pyx_v_var != Py_None); if (__pyx_1) { /* "/home/dagss/cython/d/tests/run/noneattributeacc.pyx":49 * def check_and_assign(MyClass var): * if var is not None: * print var.a # <<<<<<<<<<<<<< * var = None * print var.a */ # # No None check needed here # __pyx_2 = PyInt_FromLong(((struct __pyx_obj_16noneattributeacc_MyClass *)__pyx_v_var)->a); if (unlikely(!__pyx_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 49; __pyx_clineno = __LINE__; goto __pyx_L1_error;} /* "/home/dagss/cython/d/tests/run/noneattributeacc.pyx":50 * var = None # <<<<<<<<<<<<<< * */ Py_INCREF(Py_None); Py_DECREF(__pyx_v_var); __pyx_v_var = Py_None; /* "/home/dagss/cython/d/tests/run/noneattributeacc.pyx":51 * print var.a * var = None * print var.a # <<<<<<<<<<<<<< * */ # # Note that a None-check is inserted. # if (unlikely(__pyx_v_var == Py_None)) { __Pyx_RaiseNoneAttributeError("a"); {__pyx_filename = __pyx_f[0]; __pyx_lineno = 51; __pyx_clineno = __LINE__; goto __pyx_L1_error;} } __pyx_2 = PyInt_FromLong(((struct __pyx_obj_16noneattributeacc_MyClass *)__pyx_v_var)->a); if (unlikely(!__pyx_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 51; __pyx_clineno = __LINE__; goto __pyx_L1_error;} ... -- Dag Sverre From dalcinl at gmail.com Thu Aug 21 16:02:22 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 21 Aug 2008 11:02:22 -0300 Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: <48ACB6A9.2050905@canterbury.ac.nz> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> Message-ID: On Wed, Aug 20, 2008 at 9:28 PM, Greg Ewing wrote: > Speaking of that, I've been thinking for a while now > about making "not None" the default, and requiring > "or None" to specify that None is an acceptable > parameter value. Definitely +1 for me. > Having seen a number of pieces of Pyrex code that > were susceptible to being crashed because "not None" > wasn't used where it should have been, I think > this would be a safer default. Indeed. My own code suffered for this flaw, I realized about this Cython behavior reading the generated C code, and then I went to the docs to comfirm this. Up to now, I'm not yet completelly sure that all my code is free of "None" crashes. > The change may > break some existing code, though. Yes, but surely is will make others codes to become correct. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From cb at pdos.csail.mit.edu Thu Aug 21 17:15:36 2008 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Thu, 21 Aug 2008 11:15:36 -0400 Subject: [Cython] toolchain integration (was Re: preprocessor integration) In-Reply-To: References: <20080819175953.GA16796@pdos.lcs.mit.edu> <48ABB4FE.4000405@behnel.de> <20080820220809.GA31485@pdos.lcs.mit.edu> <48ACBF35.2090105@canterbury.ac.nz> <20080821071327.GA8080@pdos.lcs.mit.edu> Message-ID: <20080821151535.GA19849@pdos.lcs.mit.edu> Robert Bradshaw [robertwb at math.washington.edu] wrote: >I second the fact that Pyrex should never generate invalid C code. :) >If there are errors, it probably won't be fixed by changing the pyx >file, rather the compiler itself needs fixing. My concern was again more about warnings from things in user-level pyx code that do not necessarily constitute "invalid code", but *could* constitute it. You know, cases end users are better judges of, ultimately, which usually is close to tantamount to a "good warning use case". Examples I'm thinking of are things like expressions "always being true due to the range of an unsigned type" (or for other reasons), or "using variables before they're initialized", and that sort of thing. { I've actually found a few bugs in my own code this way, though only with mucking about in the generated code or doing a "cython-demangle" in my brain, but such things are less friendly to more casual users. } >(but I'm still not convinced that it should be on by default). Yes, for tracking down bugs in Cython you probably don't want #file/#line, but the typical user (hopefully!) isn't in that business. Like Linus defaulting drivers in Linux to what he likes, you surely have the discretionary power to pick what Cython debuggers particularly want/use most of the time. :) More seriously, it might make sense at this stage of Cython to default to not emitting #line and at a later stage to default to emitting it. The choice also interacts with meta-programs that could generate bogus C, too. It would probably be best to just import Cython compiler flags from a $CYTHON_FLAGS, $HOME/.cythonrc, etc., so that it could just be very easy to switch around. Anyway, I'm less interested in the default setting process or default value than the feature itself. >Short of invalid extern declarations, have you ever had any C errors/ >warnings that were due to bad Pyrex/Cython code? I have seen some innocuous warnings. Things that come to mind are unsigned being greater than 0 (in checks on index bounds I think), code that can never be reached due to the goto/label layout, and un-filled-in initializers...just off the top of my head. These were all innocuous. There are enough unused variable warnings that I had to put -Wno-unused in my CFLAGS. The un-filled-in init thing happens with classes (for a while I had a 1-line patch to add an EmptySlot("tp_del") to slot_table in Compiler/TypeSlots.py. The default args impl also triggers this, and possibly other impl things. More significantly Cython has no field initializer syntax (so correlating it to pyx code is not meaningful anyway). Hence -Wno-missing-field-initializers makes sense in CFLAGS, too. { Often it makes great sense to elide fields in C code anyway for portability across versions of the structs. } >Thanks. I'll definitely plan to take a look, and I will certainly add >#file and #line output to Cython Fabulous! :) Thanks. cb From robertwb at math.washington.edu Thu Aug 21 18:14:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Aug 2008 09:14:31 -0700 Subject: [Cython] patch for bad buffer creation In-Reply-To: <48AD6485.7060405@astraw.com> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> <48AC14DA.1070303@student.matnat.uio.no> <48AD6485.7060405@astraw.com> Message-ID: <0FC4688E-2C00-4A96-9C24-036F1C3F9611@math.washington.edu> On Aug 21, 2008, at 5:50 AM, Andrew Straw wrote: > Robert Bradshaw wrote: > >>>> Dag Sverre Seljebotn wrote: >>>> >>>>> Andrew Straw wrote: > >>>> great aspect of Cython, IMO, is that it makes Python/C bridges >>>> easier, >>>> not to adhere to the minimum standards specified by a PEP. Thus >>>> a small >>>> amount of additional error checking seems OK to me. Also, as >>>> another >>>> >>> Well, in this case, the error check should be done by inserting some >>> code automatically in the definition of __getbuffer__, so that >>> Python 3 >>> consumers also get advantage of it, and so that we don't second >>> guess >>> how good non-Cython implementations are. So the check is still >>> inserted >>> in the wrong place. >> >> I agree, this is the right place to put the check. And I think it >> is very >> worth the (relatively small) overhead to insert a check here. > > Just to be clear (re-reading this bit after Robert's response makes me > realize I'm confused about your meaning): what do you mean by "the > definition of __getbuffer__" -- the C Python implementation of > PyObject_GetBuffer()? I thought my patch did affect the (Cython) > definition of __getbuffer__. What I was thinking is that when a user writes a __getbuffer__ method, a check is inserted at the end of their method before it returns to make sure all is sane. > Also, wouldn't Cython emit the same C code > for Python 3? Python 3 actually has a buffer protocol, so we don't need to emulate it. - Robert From strawman at astraw.com Thu Aug 21 18:30:22 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 21 Aug 2008 18:30:22 +0200 Subject: [Cython] patch for bad buffer creation In-Reply-To: <0FC4688E-2C00-4A96-9C24-036F1C3F9611@math.washington.edu> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> <48AC14DA.1070303@student.matnat.uio.no> <48AD6485.7060405@astraw.com> <0FC4688E-2C00-4A96-9C24-036F1C3F9611@math.washington.edu> Message-ID: <48AD981E.6010408@astraw.com> Robert Bradshaw wrote: > On Aug 21, 2008, at 5:50 AM, Andrew Straw wrote: > > >> Robert Bradshaw wrote: >> >> >>>>> Dag Sverre Seljebotn wrote: >>>>> >>>>> >>>>>> Andrew Straw wrote: >>>>>> >>>>> great aspect of Cython, IMO, is that it makes Python/C bridges >>>>> easier, >>>>> not to adhere to the minimum standards specified by a PEP. Thus >>>>> a small >>>>> amount of additional error checking seems OK to me. Also, as >>>>> another >>>>> >>>>> >>>> Well, in this case, the error check should be done by inserting some >>>> code automatically in the definition of __getbuffer__, so that >>>> Python 3 >>>> consumers also get advantage of it, and so that we don't second >>>> guess >>>> how good non-Cython implementations are. So the check is still >>>> inserted >>>> in the wrong place. >>>> >>> I agree, this is the right place to put the check. And I think it >>> is very >>> worth the (relatively small) overhead to insert a check here. >>> >> Just to be clear (re-reading this bit after Robert's response makes me >> realize I'm confused about your meaning): what do you mean by "the >> definition of __getbuffer__" -- the C Python implementation of >> PyObject_GetBuffer()? I thought my patch did affect the (Cython) >> definition of __getbuffer__. >> > > What I was thinking is that when a user writes a __getbuffer__ > method, a check is inserted at the end of their method before it > returns to make sure all is sane. > > (I'm actually totally content to let this particular issue die now, but I'm still curious about the Cython internals, so if you'll humour me...) OK, so there'd be part of the Cython compiler that checked if the name of the current method was "__getbuffer__" and if the class was an extension ctypedef, upon which it would insert some check before the method returned? I don't know the internals of Cython enough to know whether that would be a gratuitous hack or reasonable behavior... But if you point me in the right direction, I'll have a look. It does seem like the idea of a "training wheels" mode for Cython could be pretty useful -- a few of the other threads are mentioning various ideas, and it would probably make development of Cython code easier if there was a safe(r) but slower mode and a as fast as possible mode. >> Also, wouldn't Cython emit the same C code >> for Python 3? >> > > Python 3 actually has a buffer protocol, so we don't need to emulate it. > No need to emulate the buffer protocol, but there is a need for extension types defined in Cython to have something that goes from a __getbuffer__ Cython method to a Python C API call. I can't see how that would disappear. That's what I mean. -Andrew From dagss at student.matnat.uio.no Thu Aug 21 18:33:08 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 18:33:08 +0200 (CEST) Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> Message-ID: <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> Lisandro Dalcin wrote: > On Wed, Aug 20, 2008 at 9:28 PM, Greg Ewing > wrote: >> Speaking of that, I've been thinking for a while now >> about making "not None" the default, and requiring >> "or None" to specify that None is an acceptable >> parameter value. > > Definitely +1 for me. > >> Having seen a number of pieces of Pyrex code that >> were susceptible to being crashed because "not None" >> wasn't used where it should have been, I think >> this would be a safer default. > > Indeed. My own code suffered for this flaw, I realized about this > Cython behavior reading the generated C code, and then I went to the > docs to comfirm this. Up to now, I'm not yet completelly sure that all > my code is free of "None" crashes. > >> The change may >> break some existing code, though. > > Yes, but surely is will make others codes to become correct. But it does it by making a hard restriction on what kind of code you can write -- I'd like to be able to pass None to functions (in a safe manner), not just disable the feature. If you make this change now you essentially "can't go back" since you will then get code written which may depend on None being checked for at the beginning of the function. So it's a one way change, and we do get results more quickler but OTOH we permanently say no to (in my opinion) nicer behaviour. We all agree that segfaults are not nice, I just think the exception should be raised when accessing the field/attribute rather than when entering the function... (i.e. go the Java way like in the prototype I recently posted, see the behaviour in http://hg.cython.org/cython-dagss/file/4d543e06c926/tests/run/noneattributeacc.pyx ). Dag Sverre From dagss at student.matnat.uio.no Thu Aug 21 18:45:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 18:45:03 +0200 (CEST) Subject: [Cython] patch for bad buffer creation In-Reply-To: <48AD981E.6010408@astraw.com> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> <48AC14DA.1070303@student.matnat.uio.no> <48AD6485.7060405@astraw.com> <0FC4688E-2C00-4A96-9C24-036F1C3F9611@math.washington.edu> <48AD981E.6010408@astraw.com> Message-ID: <56019.193.157.229.67.1219337103.squirrel@webmail.uio.no> Andrew Straw wrote: > Robert Bradshaw wrote: >> On Aug 21, 2008, at 5:50 AM, Andrew Straw wrote: >> >> >>> Robert Bradshaw wrote: >>> >>> >>>>>> Dag Sverre Seljebotn wrote: >>>>>> >>>>>> >>>>>>> Andrew Straw wrote: >>>>>>> >>>>>> great aspect of Cython, IMO, is that it makes Python/C bridges >>>>>> easier, >>>>>> not to adhere to the minimum standards specified by a PEP. Thus >>>>>> a small >>>>>> amount of additional error checking seems OK to me. Also, as >>>>>> another >>>>>> >>>>>> >>>>> Well, in this case, the error check should be done by inserting some >>>>> code automatically in the definition of __getbuffer__, so that >>>>> Python 3 >>>>> consumers also get advantage of it, and so that we don't second >>>>> guess >>>>> how good non-Cython implementations are. So the check is still >>>>> inserted >>>>> in the wrong place. >>>>> >>>> I agree, this is the right place to put the check. And I think it >>>> is very >>>> worth the (relatively small) overhead to insert a check here. >>>> >>> Just to be clear (re-reading this bit after Robert's response makes me >>> realize I'm confused about your meaning): what do you mean by "the >>> definition of __getbuffer__" -- the C Python implementation of >>> PyObject_GetBuffer()? I thought my patch did affect the (Cython) >>> definition of __getbuffer__. >>> >> >> What I was thinking is that when a user writes a __getbuffer__ >> method, a check is inserted at the end of their method before it >> returns to make sure all is sane. >> >> > (I'm actually totally content to let this particular issue die now, but > I'm still curious about the Cython internals, so if you'll humour me...) Definitely! > OK, so there'd be part of the Cython compiler that checked if the name > of the current method was "__getbuffer__" and if the class was an > extension ctypedef, upon which it would insert some check before the > method returned? I don't know the internals of Cython enough to know > whether that would be a gratuitous hack or reasonable behavior... But if > you point me in the right direction, I'll have a look. Pick one: 1. Stick a transform in PostParseTransforms.py which intercepts FuncDefNodes, and if the name matches (and they are a method of a cdef class) you replace the body with a TryFinallyNode where the try-part gets the old body and the finally part gets an IfStatNode which has a condition node which checks things and a RaiseStatNode to raise the exception. 2. Have a look at Nodes.FuncDefNode.generate_function_definition and insert a check for self.name == "__getbuffer__" (and somehow a check that you are in a cdef class, not sure, well, at least "self.scope.parent_scope" should be a cdef class scope, for scopes see Symtab.py), and if it matches, then simply output the C code at the right place (after the label that is jumped to for return statements). 2 is much quicker so that is likely what you'd like to do, unless you prefer the coding style of 1. >>> Also, wouldn't Cython emit the same C code >>> for Python 3? >>> >> >> Python 3 actually has a buffer protocol, so we don't need to emulate it. >> > No need to emulate the buffer protocol, but there is a need for > extension types defined in Cython to have something that goes from a > __getbuffer__ Cython method to a Python C API call. I can't see how that > would disappear. That's what I mean. Not sure what you are asking but here's the whole thing: - __getbuffer__ is output as a normal C function - if you are compiling in Python 3, the address of it is put in the getbuffer slot of the type - if you are compiling in Python 2, it is called in our fake __Pyx_GetBuffer implementation (which under Python 3 is simply #define-ed to PyObject_GetBuffer). Then, when we need a buffer, __Pyx_GetBuffer is called,. What you modified was __Pyx_GetBuffer_$DTYPE, which is code used by the buffer consumer, not the buffer exporter. The difference isn't that big but putting the test in __getbuffer__ means that non-Cython Python 3 code also gets benefit of the test, and it is where it logically belongs as then the PEP isn't "violated and then corrected", rather you get help in order to not violate it in the first place. If you do this I'm +1. Dag Sverre From dagss at student.matnat.uio.no Thu Aug 21 18:46:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 18:46:03 +0200 (CEST) Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> Message-ID: <56021.193.157.229.67.1219337163.squirrel@webmail.uio.no> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Wed, Aug 20, 2008 at 9:28 PM, Greg Ewing >> >> wrote: >>> Speaking of that, I've been thinking for a while now >>> about making "not None" the default, and requiring >>> "or None" to specify that None is an acceptable >>> parameter value. >> >> Definitely +1 for me. >> >>> Having seen a number of pieces of Pyrex code that >>> were susceptible to being crashed because "not None" >>> wasn't used where it should have been, I think >>> this would be a safer default. >> >> Indeed. My own code suffered for this flaw, I realized about this >> Cython behavior reading the generated C code, and then I went to the >> docs to comfirm this. Up to now, I'm not yet completelly sure that all >> my code is free of "None" crashes. >> >>> The change may >>> break some existing code, though. >> >> Yes, but surely is will make others codes to become correct. > > But it does it by making a hard restriction on what kind of code you can > write -- I'd like to be able to pass None to functions (in a safe manner), > not just disable the feature. I meant: Not just disable the feature by default so that I have to write "MyClass or None". Dag Sverre From dagss at student.matnat.uio.no Thu Aug 21 18:49:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 18:49:04 +0200 (CEST) Subject: [Cython] patch for bad buffer creation In-Reply-To: <56019.193.157.229.67.1219337103.squirrel@webmail.uio.no> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> <48AC14DA.1070303@student.matnat.uio.no> <48AD6485.7060405@astraw.com> <0FC4688E-2C00-4A96-9C24-036F1C3F9611@math.washington.edu> <48AD981E.6010408@astraw.com> <56019.193.157.229.67.1219337103.squirrel@webmail.uio.no> Message-ID: <56023.193.157.229.67.1219337344.squirrel@webmail.uio.no> Dag Sverre Seljebotn wrote: > 2. Have a look at Nodes.FuncDefNode.generate_function_definition and > insert a check for self.name == "__getbuffer__" (and somehow a check that > you are in a cdef class, not sure, well, at least > "self.scope.parent_scope" should be a cdef class scope, for scopes see > Symtab.py), and if it matches, then simply output the C code at the right > place (after the label that is jumped to for return statements). Sorry, that should be self.local_scope, and I'm not entirely sure about the name of the enclosing scope (but something like owner_scope or parent_scope, see in Symtab.py). Dag Sverre From stefan_ml at behnel.de Thu Aug 21 19:22:08 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Aug 2008 19:22:08 +0200 Subject: [Cython] patch for bad buffer creation In-Reply-To: <48AD981E.6010408@astraw.com> References: <48ABDED3.60500@astraw.com> <48ABEF4F.1070104@student.matnat.uio.no> <48ABF75B.7000607@astraw.com> <48AC14DA.1070303@student.matnat.uio.no> <48AD6485.7060405@astraw.com> <0FC4688E-2C00-4A96-9C24-036F1C3F9611@math.washington.edu> <48AD981E.6010408@astraw.com> Message-ID: <48ADA440.9080903@behnel.de> Hi, Andrew Straw wrote: > OK, so there'd be part of the Cython compiler that checked if the name > of the current method was "__getbuffer__" and if the class was an > extension ctypedef, upon which it would insert some check before the > method returned? I don't know the internals of Cython enough to know > whether that would be a gratuitous hack or reasonable behavior... :) well, Cython generates the C code, so it actually sees the pointer that this special method returns. It's trivial to put some additional sanity check code in the final cleanup code that gets generated for that method. > It does seem like the idea of a "training wheels" mode for Cython could > be pretty useful -- a few of the other threads are mentioning various > ideas, and it would probably make development of Cython code easier if > there was a safe(r) but slower mode and a as fast as possible mode. Maybe you could start a Wiki page on this topic? http://wiki.cython.org/enhancements/safetymode Stefan From dalcinl at gmail.com Thu Aug 21 22:19:01 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 21 Aug 2008 17:19:01 -0300 Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> Message-ID: Dag, you proposal is a strong one. Just pulled from cython-dagss. Then some comments below. Could you optimize the case below? def default(MyClass var): if var is None: var = MyClass(4, 5) print var.a Additionally, no checks should be done in this case: def strinct(MyClass var not None): print var.a On Thu, Aug 21, 2008 at 1:33 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Wed, Aug 20, 2008 at 9:28 PM, Greg Ewing >> wrote: >>> Speaking of that, I've been thinking for a while now >>> about making "not None" the default, and requiring >>> "or None" to specify that None is an acceptable >>> parameter value. >> >> Definitely +1 for me. >> >>> Having seen a number of pieces of Pyrex code that >>> were susceptible to being crashed because "not None" >>> wasn't used where it should have been, I think >>> this would be a safer default. >> >> Indeed. My own code suffered for this flaw, I realized about this >> Cython behavior reading the generated C code, and then I went to the >> docs to comfirm this. Up to now, I'm not yet completelly sure that all >> my code is free of "None" crashes. >> >>> The change may >>> break some existing code, though. >> >> Yes, but surely is will make others codes to become correct. > > But it does it by making a hard restriction on what kind of code you can > write -- I'd like to be able to pass None to functions (in a safe manner), > not just disable the feature. > > If you make this change now you essentially "can't go back" since you will > then get code written which may depend on None being checked for at the > beginning of the function. So it's a one way change, and we do get results > more quickler but OTOH we permanently say no to (in my opinion) nicer > behaviour. > > We all agree that segfaults are not nice, I just think the exception > should be raised when accessing the field/attribute rather than when > entering the function... (i.e. go the Java way like in the prototype I > recently posted, see the behaviour in > > http://hg.cython.org/cython-dagss/file/4d543e06c926/tests/run/noneattributeacc.pyx > > ). > > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Thu Aug 21 22:54:01 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 22:54:01 +0200 (CEST) Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> Message-ID: <56289.193.157.229.67.1219352041.squirrel@webmail.uio.no> Lisandro Dalcin wrote: > Dag, you proposal is a strong one. Just pulled from cython-dagss. Then > some comments below. Well, it is meant only as input to the discussion so that Greg's proposal isn't taken on right away (because there's no clean backwards-compatible way back then). I don't think we have the developer resources for it now -- I'd need up to a week fulltime to do the whole thing properly and I don't have that (and there are other priorities within Cython too). I even found a few critical bugs in my prototype since I wrote it. For now we could do a simple non-optimized thing (check on *every* access) which is controlled by a compiler directive (but it would default to off then). That should only take a few hours. > Could you optimize the case below? > > def default(MyClass var): > if var is None: var = MyClass(4, 5) > print var.a Are you sure you can optimize it? MyClass could be set to a function returning None at run-time, so you don't know that it isn't None. (Or __new__ could return None, if extension types support that kind of thing, I don't know). (I have a proposal in the pipeline that would solve this issue but there's enough on the mailing list for this week so it will have to wait, but given that mystic proposal which I won't talk about, the answer is "yes", it could be optimized, but it would likely take some work and not be in the initial version of this thing). > Additionally, no checks should be done in this case: > > def strinct(MyClass var not None): > print var.a Yes, this is trivial. What worries me most is this: for i in range(large): print var.a In order to optimize this, one would need to unroll the first iteration of the loop. So that is getting so complicated that it likely will never happen I think. As a fun aside, once you get into nested expressions like this: if (a is not None and b is not None and c == 2) or (a is None and c == 1 ... then we've got an instance of SAT (i.e. can't solve in polynomial time "unless P = NP"). But I think there's a rather natural cutoff point for the complexity of expressions we consider. Dag Sverre From dagss at student.matnat.uio.no Thu Aug 21 23:11:55 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 21 Aug 2008 23:11:55 +0200 Subject: [Cython] [Off-topic] Re: Proposed change to behaviour of "not None" In-Reply-To: <56289.193.157.229.67.1219352041.squirrel@webmail.uio.no> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> <56289.193.157.229.67.1219352041.squirrel@webmail.uio.no> Message-ID: <48ADDA1B.30108@student.matnat.uio.no> Dag Sverre Seljebotn wrote: >As a fun aside, once you get into nested expressions like this: > > if (a is not None and b is not None and c == 2) or (a is None and c == 1 ... > > then we've got an instance of SAT (i.e. can't solve in polynomial time > "unless P = NP"). But I think there's a rather natural cutoff point for > the complexity of expressions we consider. > Nope, I am now fairly sure that this is wrong. I now think it is polynomial time (as you only need to know whether a or b is forced to None or not-None in isolation without having to consider the other one). But I digress (I love those few precious moments where NP-completeness-theory is useful in a real program though). -- Dag Sverre From dalcinl at gmail.com Thu Aug 21 23:35:03 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 21 Aug 2008 18:35:03 -0300 Subject: [Cython] PATCH: talking about uninitialised vars warnings from GCC, please review this Message-ID: I'm getting warnings about uninitialised variables in try/finally blocks. As far as I can follow the flow in the generated C code, the warning is actually superfluous. Any chance for this being pushed? diff -r 6488f2d89e72 Cython/Compiler/Nodes.py --- a/Cython/Compiler/Nodes.py Tue Aug 19 04:13:56 2008 -0700 +++ b/Cython/Compiler/Nodes.py Thu Aug 21 18:24:40 2008 -0300 @@ -3667,6 +3667,9 @@ class TryFinallyStatNode(StatNode): code.putln( "int %s;" % Naming.exc_lineno_name) code.use_label(catch_label) + if error_label_used and self.preserve_exception: + code.put( + "%s = 0; " % Naming.exc_lineno_name) code.putln( "__pyx_why = 0; goto %s;" % catch_label) for i in cases_used: -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Aug 22 00:47:53 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Aug 2008 15:47:53 -0700 (PDT) Subject: [Cython] Proposed change to behaviour of "not None" In-Reply-To: <56289.193.157.229.67.1219352041.squirrel@webmail.uio.no> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <56014.193.157.229.67.1219336388.squirrel@webmail.uio.no> <56289.193.157.229.67.1219352041.squirrel@webmail.uio.no> Message-ID: On Thu, 21 Aug 2008, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Dag, you proposal is a strong one. Just pulled from cython-dagss. Then >> some comments below. > > Well, it is meant only as input to the discussion so that Greg's proposal > isn't taken on right away (because there's no clean backwards-compatible > way back then). I don't think we have the developer resources for it now > -- I'd need up to a week fulltime to do the whole thing properly and I > don't have that (and there are other priorities within Cython too). Yes. I think the point that one cannot ever safely go back to the old behavior is critical and one of the main reasons I wouldn't like to make this change (except as a compiler option). > I even found a few critical bugs in my prototype since I wrote it. > > For now we could do a simple non-optimized thing (check on *every* access) > which is controlled by a compiler directive (but it would default to off > then). That should only take a few hours. > >> Could you optimize the case below? >> >> def default(MyClass var): >> if var is None: var = MyClass(4, 5) >> print var.a > > Are you sure you can optimize it? MyClass could be set to a function > returning None at run-time, so you don't know that it isn't None. (Or > __new__ could return None, if extension types support that kind of thing, > I don't know). > > (I have a proposal in the pipeline that would solve this issue but there's > enough on the mailing list for this week so it will have to wait, but > given that mystic proposal which I won't talk about, the answer is "yes", > it could be optimized, but it would likely take some work and not be in > the initial version of this thing). > >> Additionally, no checks should be done in this case: >> >> def strinct(MyClass var not None): >> print var.a > > Yes, this is trivial. > > What worries me most is this: > > for i in range(large): > print var.a > > In order to optimize this, one would need to unroll the first iteration of > the loop. So that is getting so complicated that it likely will never > happen I think. > > As a fun aside, once you get into nested expressions like this: > > if (a is not None and b is not None and c == 2) or (a is None and c == 1 ... > > then we've got an instance of SAT (i.e. can't solve in polynomial time > "unless P = NP"). But I think there's a rather natural cutoff point for > the complexity of expressions we consider. Optimizations of this nature should be considered when we have some kind of control flow analysis implemented. Variable assignment will certainly be tracked. This check could be done on the first attribute access attempt, and then known to be true until an assignment is made. One could try and optimize things even further, like simple "if a is not None: ...", but beyond that I would guess the return on investment drops off quickly. - Robert From greg.ewing at canterbury.ac.nz Fri Aug 22 01:13:40 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 22 Aug 2008 11:13:40 +1200 Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <1ECBED99-F639-41CF-AE4B-1FEAA066F5AC@math.washington.edu> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <1ECBED99-F639-41CF-AE4B-1FEAA066F5AC@math.washington.edu> Message-ID: <48ADF6A4.1060709@canterbury.ac.nz> Robert Bradshaw wrote: > I use the fact that extension types can be None all over the place. Is > this exclusively for arguments? Yes, it's only for arguments, and only for Python functions (i.e. def, not cdef). The idea is to protect the boundary between Python and C land from getting passed invalid values. Checking for None on all accesses is a separate issue. I wouldn't object to an option for that, although the cost of turning it on is going to be very high, unless you can optimise some of the tests away with flow analysis somehow. -- Greg From robertwb at math.washington.edu Fri Aug 22 08:03:40 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Aug 2008 23:03:40 -0700 Subject: [Cython] PATCH: talking about uninitialised vars warnings from GCC, please review this In-Reply-To: References: Message-ID: On Aug 21, 2008, at 2:35 PM, Lisandro Dalcin wrote: > I'm getting warnings about uninitialised variables in try/finally > blocks. As far as I can follow the flow in the generated C code, the > warning is actually superfluous. Any chance for this being pushed? > > diff -r 6488f2d89e72 Cython/Compiler/Nodes.py > --- a/Cython/Compiler/Nodes.py Tue Aug 19 04:13:56 2008 -0700 > +++ b/Cython/Compiler/Nodes.py Thu Aug 21 18:24:40 2008 -0300 > @@ -3667,6 +3667,9 @@ class TryFinallyStatNode(StatNode): > code.putln( > "int %s;" % Naming.exc_lineno_name) > code.use_label(catch_label) > + if error_label_used and self.preserve_exception: > + code.put( > + "%s = 0; " % Naming.exc_lineno_name) > code.putln( > "__pyx_why = 0; goto %s;" % catch_label) > for i in cases_used: > > Yes, I would very much like to reduce warnings. Could you give me an example of some code that this fixes? - Robert From robertwb at math.washington.edu Fri Aug 22 08:45:17 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Aug 2008 23:45:17 -0700 Subject: [Cython] Beta tutorial for NumPy users In-Reply-To: <48AC21A8.2000608@student.matnat.uio.no> References: <48AC21A8.2000608@student.matnat.uio.no> Message-ID: <2BADB765-B590-4643-8679-F4F1A7982106@math.washington.edu> On Aug 20, 2008, at 6:52 AM, Dag Sverre Seljebotn wrote: > I'm nearly done with my "Cython for NumPy users" tutorial: > > http://wiki.cython.org/tutorials/numpy > > I'll post it to numpy-discuss in some hours (and draw attention the > new > release as well) but if there's anything you'd like mentioned before > people start reading please respond. (Of course, it's a wiki, so any > improvements just go ahead and do it if you like.) Thanks! Great introduction to getting started with Cython too. Do you think that should be split off into its own page? - Robert From fperez.net at gmail.com Fri Aug 22 08:54:59 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 21 Aug 2008 23:54:59 -0700 Subject: [Cython] Beta tutorial for NumPy users In-Reply-To: <2BADB765-B590-4643-8679-F4F1A7982106@math.washington.edu> References: <48AC21A8.2000608@student.matnat.uio.no> <2BADB765-B590-4643-8679-F4F1A7982106@math.washington.edu> Message-ID: On Thu, Aug 21, 2008 at 11:45 PM, Robert Bradshaw wrote: > > Thanks! Great introduction to getting started with Cython too. Do you > think that should be split off into its own page? If you split it, please leave a link to the generic intro in the numpy page. It makes for a great resource to point people to who come from numpy and want a one-stop shop for 'fast'. Cheers, f From robertwb at math.washington.edu Fri Aug 22 09:36:15 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Aug 2008 00:36:15 -0700 Subject: [Cython] Unable to import pyximport In-Reply-To: References: Message-ID: On Aug 20, 2008, at 6:32 AM, Muhammad Alkarouri wrote: > Hi everyone, > > Looks like pyximport is a cool addition to cython. Yes, very. Thanks again Paul Prescod! > Unfortunately, > after I installed Cython through easy_install) I wasn't able to import > pyximport. > > In [2]: Cython? > Type: module > Base Class: > String Form: '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ > site-packages/Cython-0.9.8.1.1-py2.5-macosx-10.3-i386.egg/Cython/ > __init__.pyc'> > Namespace: Interactive > File: > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ > site-packages/Cython-0.9.8.1.1-py2.5-macosx-10.3-i386.egg/Cython/ > __init__.py > Docstring: > > > > In [3]: import pyximport > ---------------------------------------------------------------------- > ----- > ImportError Traceback (most recent > call last) > > /Users/malkarouri/ in () > > ImportError: No module named pyximport > > > When I looked in the package, there is a folder pyximport beside > Cython (sys.path contains > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ > site-packages/Cython-0.9.8.1.1-py2.5-macosx-10.3-i386.egg/, > the parent directory of both). But pyximport doesn't contain an > __init__.py file. Is that the problem? That could be it. Could you add one and see if that works for you? - Robert From robertwb at math.washington.edu Fri Aug 22 09:51:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Aug 2008 00:51:14 -0700 Subject: [Cython] Repository usage In-Reply-To: <53849.193.157.229.67.1219139137.squirrel@webmail.uio.no> References: <53849.193.157.229.67.1219139137.squirrel@webmail.uio.no> Message-ID: <78AD29D5-CD28-4F17-860F-021B0153735D@math.washington.edu> On Aug 19, 2008, at 2:45 AM, Dag Sverre Seljebotn wrote: > There's been some talk already but now seems the time to make votes > etc. > > I propose that bugfixes that a) are relatively simple, and b) > doesn't rely > on or interfere features implemented since the last release, are coded > against and pushed to "cython" and then merged into "cython-devel". > > "cython" should still be considered relatively stable, so every commit > there is like a micro-patch-release of Cython. +1 > This should take some pressure off cython-devel (although keeping > it as > stable as possible is still good). Cython 0.9.8.2 may then come > entirely > from "cython" if it is primarily a bugfix release, or from "cython- > devel" > if it stabilizes. > > One should think about some of the things that may be done to > cython-devel > soon, especially import Greg's result_code-calculation-in- > generation-phase > changes, and the other refactorings related to/made possible by that. > These are mildly dangerous (at least they lead to changes > "everywhere") > but also needs to be in the main stream of development. > > An alternative would perhaps be renaming "dagss" to > "experimental" (but > IMO all new features as well as new framework changes so go into it > then). > > Thoughts? One thing to consider is that a single repository can have multiple branches. I propose that cython-devel be relatively stable, i.e. the entire test suite should pass before pushing there (I've been trying to do that, but we should formalize this requirement) but can still do some bolder things. I'm happy to set up a repository for anyone to wants to have one public (and, of course, you can host your own too). These can be more experimental. Also, not all experimental stuff should get merged in. On a related note, I had a lot of people ask me "so, what's new in Cython lately." It would like to start using http://trac.cython.org/ cython_trac more to keep track of everything of significance that goes in. - Robert From kirr at mns.spb.ru Fri Aug 22 09:47:36 2008 From: kirr at mns.spb.ru (Kirill Smelkov) Date: Fri, 22 Aug 2008 11:47:36 +0400 Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <48ACB6A9.2050905@canterbury.ac.nz> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> Message-ID: <200808221147.37360.kirr@mns.spb.ru> ? ????????? ?? ??????? 21 ??????? 2008 Greg Ewing ???????(a): > Lisandro Dalcin wrote: > > > I know, I can (and I do) use 'Type arg is not > > None' in argument lists, but I never liked it. > > Speaking of that, I've been thinking for a while now > about making "not None" the default, and requiring > "or None" to specify that None is an acceptable > parameter value. > > Having seen a number of pieces of Pyrex code that > were susceptible to being crashed because "not None" > wasn't used where it should have been, I think > this would be a safer default. The change may > break some existing code, though. > > What do people think? I'm +1 for having "not None" the default. I was always wondering why was it the vice versa when I was stydying Pyrex -- for internal functions this was creating additional overhead and additional inconvenience, and I'd better put "or None" explicitely where it makes sence -- for high-level functions who interface with Python level. Bearing backward compatibility in mind, maybe let's create a compile option to switch this behaviour? And we'll remove old behaviour step-by-step: o first declare there is new "not None" by default with explicit switch, saying it will be the default in the next release. o then make default "not None" default and allow users to switch back with cython/pyrex option, but say this old behaviour will be removed in the next release o remove old behaviour in the next release. Sounds reasonable? Thanks, Kirill. From robertwb at math.washington.edu Fri Aug 22 10:13:23 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Aug 2008 01:13:23 -0700 Subject: [Cython] about to buffer syntax In-Reply-To: References: <3301160609.1914700@smtp.netcom.no> Message-ID: <29F2E9C5-5B27-4FEE-890E-D287CF5470F1@math.washington.edu> On Aug 11, 2008, at 8:42 AM, Lisandro Dalcin wrote: > 2008/8/9 Dag Sverre Seljebotn : >> My initial reaction is that I like it! However it might not be >> needed... >> >> It would have to be (:n, :) rather than (n, :) for consistency. >> >> The thing is, you don't really need the shape in the syntax >> currently, only ndim. (One can use an assert statement if one >> wants the check, there doesn't appear to be any advantages to >> knowing the shape compile-time, unless perhaps if it is very small.) >> >> however with this perhaps support for specifying a starting point >> (index start point, subtracted on all lookups) could be added, >> that would be potentially very useful but also tend to "add >> cruft"...add something that is not there in either Python or C... > >> Time and users will have to tell! But I enjoyed such a fresh idea >> in this discussion. > > Well, I believe that people with some background in Fortran 90 will > definitely like and support my proposal. > >> I think one could make the ndim keyword mandatory at first, to >> delay the decision about which positional argument should be 2nd. > > Well, I believe that is a good approach. IMHO, declaring > > cdef ndarray[int, 3] tmp > > is not really clear that '3' means a 3D array. The intention of my > proposal is just to provide a syntax that is less ambiguous and have > room for more facilities (shape checking, negative starts, require > shape at least xD, etc.) Your syntax is certainly more flexible. Given what we support, ndarray [int, (:,:,:)] is a rather obscure way to declare a 3-dim array, but if we add more features in the future this seems backwards compatible with what we have now. - Robert From robertwb at math.washington.edu Fri Aug 22 10:19:37 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Aug 2008 01:19:37 -0700 Subject: [Cython] Accessing the internals of a Python long In-Reply-To: <99e0b9530808021953j5af998a1m5e431ed1918de798@mail.gmail.com> References: <99e0b9530808021953j5af998a1m5e431ed1918de798@mail.gmail.com> Message-ID: On Aug 2, 2008, at 7:53 PM, Case Vanhorsen wrote: > I am trying to access the internal structure of a Python long. The > code fragment below works when I use Pyrex 0.9.8.4 but generates a > compile error when using Cython 0.9.8. > > Is there a better way to access the internals of a Python long? > > What is causing the cimport error and how can I fix it? FYI, this has been resolved. The problem is whether to interpret a.b as a cimported type (from the module a) or an attribute. This decision is made at parsing, and Pyrex assumed one way and Cython the other. I've made it accept both. > > > Thanks, > > casevh > > ------------------------------ > ------------------------ > > The code: > > cdef extern from "Python.h": > ctypedef struct PyTypeObject: > pass > ctypedef struct PyObject: > Py_ssize_t ob_refcnt > PyTypeObject * ob_type > cdef extern from "longintrepr.h": > cdef struct _longobject: > int ob_refcnt > PyTypeObject * ob_type > int ob_size > unsigned int * ob_digit > def test(temp = long(0)): > cdef _longobject *l > l = <_longobject *> temp > print sizeof(l.ob_size) > print sizeof(l.ob_digit[0]) > > The error: > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > int ob_size > unsigned int * ob_digit > def test(temp = long(0)): > cdef _longobject *l > l = <_longobject *> temp > print sizeof(l.ob_size) > ^ > ------------------------------------------------------------ > > /home/case/code/cvh.pyx:16:18: 'cvh.test' is not a cimported module > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Fri Aug 22 10:21:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Aug 2008 01:21:31 -0700 Subject: [Cython] Make positive int literals unsigned long? In-Reply-To: <4897091B.1000604@student.matnat.uio.no> References: <4897091B.1000604@student.matnat.uio.no> Message-ID: <4120E224-37F4-4B2C-B317-800F33096B84@math.washington.edu> On Aug 4, 2008, at 6:50 AM, Dag Sverre Seljebotn wrote: > If we make positive int literals unsigned long rather than plain long, > constant-index buffer lookups are going to be optimized for free. Is > there going to be any ill side-effects from doing this? > > (If not, I can always special-case int literals used for buffer > lookups.) As Carl Witty pointed out, arithmetic between signed an unsigned types results in unsigned types, so (1) that means literals will definitely not be unsigned and (2) we don't need them to be. (To get an unsigned literal, do 1u). - Robert From robertwb at math.washington.edu Fri Aug 22 10:36:01 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Aug 2008 01:36:01 -0700 Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <200808221147.37360.kirr@mns.spb.ru> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <200808221147.37360.kirr@mns.spb.ru> Message-ID: <83227FC1-272D-421A-A7FE-435DB531C1F9@math.washington.edu> On Aug 22, 2008, at 12:47 AM, Kirill Smelkov wrote: > ? ????????? ?? ??????? 21 ??????? 2008 > Greg Ewing ???????(a): >> Lisandro Dalcin wrote: >> >>> I know, I can (and I do) use 'Type arg is not >>> None' in argument lists, but I never liked it. >> >> Speaking of that, I've been thinking for a while now >> about making "not None" the default, and requiring >> "or None" to specify that None is an acceptable >> parameter value. >> >> Having seen a number of pieces of Pyrex code that >> were susceptible to being crashed because "not None" >> wasn't used where it should have been, I think >> this would be a safer default. The change may >> break some existing code, though. >> >> What do people think? > > I'm +1 for having "not None" the default. > > I was always wondering why was it the vice versa when I was > stydying Pyrex -- > for internal functions this was creating additional overhead and > additional > inconvenience, and I'd better put "or None" explicitely where it > makes sence > -- for high-level functions who interface with Python level. > > > Bearing backward compatibility in mind, maybe let's create a > compile option to > switch this behaviour? And we'll remove old behaviour step-by-step: > > o first declare there is new "not None" by default with explicit > switch, saying > it will be the default in the next release. > o then make default "not None" default and allow users to switch > back with > cython/pyrex option, but say this old behaviour will be removed > in the next > release > o remove old behaviour in the next release. > > Sounds reasonable? I am still -1 to the idea, but keep trying to convince me :). Certainly I would like things to be more friendly then they are right now, but having two kinds of extension class types that are declared exactly the same way, one of which can take a None and the other can't, seems odd. Certainly enabling a switch to check for this is something I want to have, and not limited to python function arguments either. - Robert From dagss at student.matnat.uio.no Fri Aug 22 11:29:48 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Aug 2008 11:29:48 +0200 Subject: [Cython] Repository usage In-Reply-To: <78AD29D5-CD28-4F17-860F-021B0153735D@math.washington.edu> References: <53849.193.157.229.67.1219139137.squirrel@webmail.uio.no> <78AD29D5-CD28-4F17-860F-021B0153735D@math.washington.edu> Message-ID: <48AE870C.50106@student.matnat.uio.no> Robert Bradshaw wrote: > On Aug 19, 2008, at 2:45 AM, Dag Sverre Seljebotn wrote: > >> There's been some talk already but now seems the time to make votes >> etc. >> >> I propose that bugfixes that a) are relatively simple, and b) >> doesn't rely >> on or interfere features implemented since the last release, are coded >> against and pushed to "cython" and then merged into "cython-devel". >> >> "cython" should still be considered relatively stable, so every commit >> there is like a micro-patch-release of Cython. > > +1 > >> This should take some pressure off cython-devel (although keeping >> it as >> stable as possible is still good). Cython 0.9.8.2 may then come >> entirely >> from "cython" if it is primarily a bugfix release, or from "cython- >> devel" >> if it stabilizes. >> >> One should think about some of the things that may be done to >> cython-devel >> soon, especially import Greg's result_code-calculation-in- >> generation-phase >> changes, and the other refactorings related to/made possible by that. >> These are mildly dangerous (at least they lead to changes >> "everywhere") >> but also needs to be in the main stream of development. >> >> An alternative would perhaps be renaming "dagss" to >> "experimental" (but >> IMO all new features as well as new framework changes so go into it >> then). >> >> Thoughts? > > One thing to consider is that a single repository can have multiple > branches. Yes, but they seem to be more hidden, i.e. the web interface for instance won't make the fact that there are multiple branches obvious to a visitor? So they seem to be more useful for private use but since one would have to know about them they seem less useful for sharing stuff. If you know of a good way to use them please share... (svn branches are actually more useful here as they are obviously present as directories in the folder structure) > I propose that cython-devel be relatively stable, i.e. the entire > test suite should pass before pushing there (I've been trying to do > that, but we should formalize this requirement) but can still do some > bolder things. I'm happy to set up a repository for anyone to wants Fine. Though I think it should actually be possible to push failing testcases without a fix if a bug is discovered? If one doesn't have time for a fix it is better to push a testcase than not? (and stick a ticket in trac) > On a related note, I had a lot of people ask me "so, what's new in > Cython lately." It would like to start using http://trac.cython.org/ > cython_trac more to keep track of everything of significance that > goes in. Yes I had almost completely forgot about trac -- I'll start using it to track anything non-trivial I do (i.e. create tickets and assign to myself). Will I be warned if a ticket (i.e. typically towards buffers) is assigned to me? -- Dag Sverre From hoytak at cs.ubc.ca Fri Aug 22 17:46:00 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Fri, 22 Aug 2008 18:46:00 +0300 Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <83227FC1-272D-421A-A7FE-435DB531C1F9@math.washington.edu> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <200808221147.37360.kirr@mns.spb.ru> <83227FC1-272D-421A-A7FE-435DB531C1F9@math.washington.edu> Message-ID: <4db580fd0808220846v1b1b59ffv455dc979d5bc844f@mail.gmail.com> Hello, I'm a relative newbie to cython/pyrex, so I won't be offended if this suggestion gets thrown out quickly as long as I learn something in the process. From my googling, it seems that decorators aren't really supported by cython/pyrex yet (http://wiki.cython.org/enhancements/decorators) but are on cython's to-do list. It strikes me that decorators would be a natural way of handling this, i.e. having a @notNone decorator for functions that has the affect of adding "not None" to every argument. If part of the reason for switching the default is that writing "not None" for every argument is awkward, this would be a reasonable middle ground. -- Hoyt On Fri, Aug 22, 2008 at 11:36 AM, Robert Bradshaw wrote: > On Aug 22, 2008, at 12:47 AM, Kirill Smelkov wrote: > >> ? ????????? ?? ??????? 21 ??????? 2008 >> Greg Ewing ???????(a): >>> Lisandro Dalcin wrote: >>> >>>> I know, I can (and I do) use 'Type arg is not >>>> None' in argument lists, but I never liked it. >>> >>> Speaking of that, I've been thinking for a while now >>> about making "not None" the default, and requiring >>> "or None" to specify that None is an acceptable >>> parameter value. >>> >>> Having seen a number of pieces of Pyrex code that >>> were susceptible to being crashed because "not None" >>> wasn't used where it should have been, I think >>> this would be a safer default. The change may >>> break some existing code, though. >>> >>> What do people think? >> >> I'm +1 for having "not None" the default. >> >> I was always wondering why was it the vice versa when I was >> stydying Pyrex -- >> for internal functions this was creating additional overhead and >> additional >> inconvenience, and I'd better put "or None" explicitely where it >> makes sence >> -- for high-level functions who interface with Python level. >> >> >> Bearing backward compatibility in mind, maybe let's create a >> compile option to >> switch this behaviour? And we'll remove old behaviour step-by-step: >> >> o first declare there is new "not None" by default with explicit >> switch, saying >> it will be the default in the next release. >> o then make default "not None" default and allow users to switch >> back with >> cython/pyrex option, but say this old behaviour will be removed >> in the next >> release >> o remove old behaviour in the next release. >> >> Sounds reasonable? > > I am still -1 to the idea, but keep trying to convince me :). > Certainly I would like things to be more friendly then they are right > now, but having two kinds of extension class types that are declared > exactly the same way, one of which can take a None and the other > can't, seems odd. Certainly enabling a switch to check for this is > something I want to have, and not limited to python function > arguments either. > > - Robert > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Fri Aug 22 18:00:21 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Aug 2008 18:00:21 +0200 Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <4db580fd0808220846v1b1b59ffv455dc979d5bc844f@mail.gmail.com> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <200808221147.37360.kirr@mns.spb.ru> <83227FC1-272D-421A-A7FE-435DB531C1F9@math.washington.edu> <4db580fd0808220846v1b1b59ffv455dc979d5bc844f@mail.gmail.com> Message-ID: <48AEE295.1080604@student.matnat.uio.no> Hoyt Koepke wrote: > Hello, > > I'm a relative newbie to cython/pyrex, so I won't be offended if this > suggestion gets thrown out quickly as long as I learn something in the > process. From my googling, it seems that decorators aren't really > supported by cython/pyrex yet > (http://wiki.cython.org/enhancements/decorators) but are on cython's > to-do list. It strikes me that decorators would be a natural way of > handling this, i.e. having a @notNone decorator for functions that has > the affect of adding "not None" to every argument. If part of the > reason for switching the default is that writing "not None" for every > argument is awkward, this would be a reasonable middle ground. > http://wiki.cython.org/docs/compilerdirectives :-) So yes, this is definitely doable. -- Dag Sverre From robertwb at math.washington.edu Fri Aug 22 18:48:34 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Aug 2008 09:48:34 -0700 Subject: [Cython] [Pyrex] Proposed change to behaviour of "not None" In-Reply-To: <4db580fd0808220846v1b1b59ffv455dc979d5bc844f@mail.gmail.com> References: <48AC17FA.7060507@student.matnat.uio.no> <48ACB6A9.2050905@canterbury.ac.nz> <200808221147.37360.kirr@mns.spb.ru> <83227FC1-272D-421A-A7FE-435DB531C1F9@math.washington.edu> <4db580fd0808220846v1b1b59ffv455dc979d5bc844f@mail.gmail.com> Message-ID: <1D25B6AA-1489-406C-A02A-AFE9B6E95CD8@math.washington.edu> On Aug 22, 2008, at 8:46 AM, Hoyt Koepke wrote: > Hello, > > I'm a relative newbie to cython/pyrex, so I won't be offended if this > suggestion gets thrown out quickly as long as I learn something in the > process. From my googling, it seems that decorators aren't really > supported by cython/pyrex yet > (http://wiki.cython.org/enhancements/decorators) but are on cython's > to-do list. In the latest version, they're on Cython's done list (function decorators that is, and class decorators should be easy given that). > It strikes me that decorators would be a natural way of > handling this, i.e. having a @notNone decorator for functions that has > the affect of adding "not None" to every argument. If part of the > reason for switching the default is that writing "not None" for every > argument is awkward, this would be a reasonable middle ground. Yes, that's a great idea. When I say I want to add it as a compiler option, I'm thinking like http://wiki.cython.org/docs/ compilerdirectives . - Robert From dalcinl at gmail.com Fri Aug 22 19:20:42 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 22 Aug 2008 14:20:42 -0300 Subject: [Cython] PATCH: talking about uninitialised vars warnings from GCC, please review this In-Reply-To: References: Message-ID: Well, the following snippet generates the warning (GCC 4.1.1). I believe the warning is superfluous, but the generated C code is rather contrived, it seems my GCC cannot "unroll" such pieces of code with 'goto' and 'switch' all around. def dowork(): pass def cleanup(): pass def test(): try: dowork() finally: cleanup() On Fri, Aug 22, 2008 at 3:03 AM, Robert Bradshaw wrote: > On Aug 21, 2008, at 2:35 PM, Lisandro Dalcin wrote: > >> I'm getting warnings about uninitialised variables in try/finally >> blocks. As far as I can follow the flow in the generated C code, the >> warning is actually superfluous. Any chance for this being pushed? >> >> diff -r 6488f2d89e72 Cython/Compiler/Nodes.py >> --- a/Cython/Compiler/Nodes.py Tue Aug 19 04:13:56 2008 -0700 >> +++ b/Cython/Compiler/Nodes.py Thu Aug 21 18:24:40 2008 -0300 >> @@ -3667,6 +3667,9 @@ class TryFinallyStatNode(StatNode): >> code.putln( >> "int %s;" % Naming.exc_lineno_name) >> code.use_label(catch_label) >> + if error_label_used and self.preserve_exception: >> + code.put( >> + "%s = 0; " % Naming.exc_lineno_name) >> code.putln( >> "__pyx_why = 0; goto %s;" % catch_label) >> for i in cases_used: >> >> > > > Yes, I would very much like to reduce warnings. Could you give me an > example of some code that this fixes? > > - Robert > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todo