From robertwb at math.washington.edu Wed Jul 2 06:49:41 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 1 Jul 2008 21:49:41 -0700 Subject: [Cython] .pxd's that depend on other .pxd's In-Reply-To: <20080630151202.GA4534@tilt> References: <20080630151202.GA4534@tilt> Message-ID: Did this work in a former version of Cython? This may be an issue with the way the __init__ stuff was merge in. A short-term workaround would be to just use relative imports (i.e. "from translatable cimport Translatable" for now. - Robert On Jun 30, 2008, at 8:12 AM, Peter Todd wrote: > Using the attached tarball I get this error message with Cython 0.9.8: > > > pete at tilt:~/t/bug$ cython Tuke/id.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > from Tuke.translatable cimport Translatable > ^ > ------------------------------------------------------------ > > /home/pete/t/bug/Tuke/id.pyx:1:0: 'Tuke.translatable.pxd' not found > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > from Tuke.translatable cimport Translatable > ^ > ------------------------------------------------------------ > > /home/pete/t/bug/Tuke/id.pyx:1:31: Name 'Translatable' not declared > in module 'Tuke.translatable' > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > from Tuke.translatable cimport Translatable > > cdef class Id(Translatable): > ^ > ------------------------------------------------------------ > > /home/pete/t/bug/Tuke/id.pyx:3:5: 'Translatable' is not declared > pete at tilt:~/t/bug$ > > Is this correct behavior? The files are pretty simple, I just want > the class in > id.pyx to be a sub-class of something in translatable.pxd > > -- > http://petertodd.org 'peter'[:-1]@petertodd.org > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080701/b5c81024/attachment.pgp From stefan_ml at behnel.de Wed Jul 2 08:03:18 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 02 Jul 2008 08:03:18 +0200 Subject: [Cython] .pxd's that depend on other .pxd's In-Reply-To: <20080630151202.GA4534@tilt> References: <20080630151202.GA4534@tilt> Message-ID: <486B1A26.401@behnel.de> Hi, Peter Todd wrote: > Using the attached tarball I get this error message with Cython 0.9.8: > > > pete at tilt:~/t/bug$ cython Tuke/id.pyx > /home/pete/t/bug/Tuke/id.pyx:1:0: 'Tuke.translatable.pxd' not found Do you have an __init__.py in the Tuke package? Stefan From dagss at student.matnat.uio.no Wed Jul 2 11:00:14 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Jul 2008 11:00:14 +0200 Subject: [Cython] Best way to fake buffers in Py2? Message-ID: <486B439E.7060805@student.matnat.uio.no> Part of my buffer proposal is to provide our own implementation of it for Python 2 (I'm unsure about the status for 2.6, it seems like not the whole buffer API might be backported). For now I'll just do whatever will get us something to play with, but what do people think is a long-term solution for this? What is needed is that for Python 2 is that we bundle something like this (psuedo-code): #ifdef Py2 PyAPI_FUNC(int) PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) { if (numpy is available && obj instanceof numpy.ndarray) { view->buf = obj->data; view->strides = obj->strides; view->shape = obj->shape; ... } else if (PIL is available && obj instanceof Imaging.Image) { ... } ... other buffers ... } #endif Now the question is, how do I best do this? The "X is available" (and if so, #include its headers...) is the tricky part. I'm thinking along the lines of having a new special attribute on the classes in question, something like this: cdef extern from "cythonbufferbackwardscompat.h": cdef int NumPyGetBuffer(PyObject*, Py_buffer*, int) cdef int NumPyReleaseBuffer(PyObject*, Py_buffer*) cdef class numpy.ndarray ... : __cython_py2_getbuffer__ = "NumPyGetBuffer" __cython_py2_releasebuffer__ = "NumPyReleaseBuffer" And then generate a pyx-file-specific local PyObject_GetBuffer depending on the __cython_py2_buffer__ we can find: #ifdef Py2 static int PyObject_GetBuffer(PyObject *, Py_buffer *, int); #endif ... code using PyObject_GetBuffer ... #ifdef Py2 PyAPI_FUNC(int) PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) { if (obj is numpy.ndarray) { return NumPyGetBuffer(obj, view, flags); } else if ... } else { raise not supported-exception } } #endif Thoughts? A disadvantage is that "cythonbufferbackwardscompat.h": (and .c?) must be provided, but inlineable-code-in-pxds can help with that on a later date (Robert: This is something I'll push past midterm-eval if needed; if so I'll just make it work by hard-coding NumPy-specific stuff directly). -- Dag Sverre From dagss at student.matnat.uio.no Wed Jul 2 11:02:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Jul 2008 11:02:03 +0200 Subject: [Cython] Best way to fake buffers in Py2? In-Reply-To: <486B439E.7060805@student.matnat.uio.no> References: <486B439E.7060805@student.matnat.uio.no> Message-ID: <486B440B.6090501@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > And then generate a pyx-file-specific local PyObject_GetBuffer depending > on the __cython_py2_buffer__ we can find: > > #ifdef Py2 > static int PyObject_GetBuffer(PyObject *, Py_buffer *, int); > #endif > > ... code using PyObject_GetBuffer ... > > #ifdef Py2 > PyAPI_FUNC(int) PyObject_GetBuffer(PyObject *obj, Py_buffer *view, > int flags) { Sorry, this should have been declared static int as well. -- Dag Sverre From robertwb at math.washington.edu Wed Jul 2 11:18:45 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 2 Jul 2008 02:18:45 -0700 Subject: [Cython] Best way to fake buffers in Py2? In-Reply-To: <486B439E.7060805@student.matnat.uio.no> References: <486B439E.7060805@student.matnat.uio.no> Message-ID: <6AD216CF-BF97-42CF-AAFA-5E19CF77B2BD@math.washington.edu> On Jul 2, 2008, at 2:00 AM, Dag Sverre Seljebotn wrote: > Part of my buffer proposal is to provide our own implementation of it > for Python 2 (I'm unsure about the status for 2.6, it seems like > not the > whole buffer API might be backported). For now I'll just do whatever > will get us something to play with, but what do people think is a > long-term solution for this? > > What is needed is that for Python 2 is that we bundle something like > this (psuedo-code): > > #ifdef Py2 > PyAPI_FUNC(int) PyObject_GetBuffer(PyObject *obj, Py_buffer *view, > int flags) { > > if (numpy is available && obj instanceof numpy.ndarray) { > view->buf = obj->data; > view->strides = obj->strides; > view->shape = obj->shape; > ... > } else if (PIL is available && obj instanceof Imaging.Image) { > ... > } ... other buffers ... > > } > #endif That looks good. > Now the question is, how do I best do this? The "X is > available" (and if > so, #include its headers...) is the tricky part. I'm thinking along > the > lines of having a new special attribute on the classes in question, > something like this: > > cdef extern from "cythonbufferbackwardscompat.h": > cdef int NumPyGetBuffer(PyObject*, Py_buffer*, int) > cdef int NumPyReleaseBuffer(PyObject*, Py_buffer*) > > cdef class numpy.ndarray ... : > __cython_py2_getbuffer__ = "NumPyGetBuffer" > __cython_py2_releasebuffer__ = "NumPyReleaseBuffer" > > And then generate a pyx-file-specific local PyObject_GetBuffer > depending > on the __cython_py2_buffer__ we can find: > > #ifdef Py2 > static int PyObject_GetBuffer(PyObject *, Py_buffer *, int); > #endif > > ... code using PyObject_GetBuffer ... > > #ifdef Py2 > PyAPI_FUNC(int) PyObject_GetBuffer(PyObject *obj, Py_buffer *view, > int flags) { > > if (obj is numpy.ndarray) { > return NumPyGetBuffer(obj, view, flags); > } else if ... > > } else { raise not supported-exception } > > } > #endif > > > Thoughts? A disadvantage is that "cythonbufferbackwardscompat.h": (and > .c?) must be provided, but inlineable-code-in-pxds can help with > that on > a later date (Robert: This is something I'll push past midterm-eval if > needed; if so I'll just make it work by hard-coding NumPy-specific > stuff > directly). Yes, hard coding a NumPy-specific buffer function is fine at this point. Inlineable-code-in-pxds should be able to make this much cleaner: cdef class numpy.ndarray ... : cdef inline __cython_py2_getbuffer__(PyObject* self, Py_buffer* buf, int k): ... cdef inline __cython_py2_releasebuffer__(PyObject* self, Py_buffer* buf, int k): .... (though those particular names seem overly verbose...why not just __getbuffer__ and __releasebuffer__, maybe with a decorator to specify py2 only) - Robert From fperez.net at gmail.com Wed Jul 2 23:50:04 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 2 Jul 2008 14:50:04 -0700 Subject: [Cython] Is this a bug in ye'ole cython? Message-ID: Hi folks, I'm wondering if this is expected behavior. Consider some trivial little class like class Simpleton: def __str__(self): return "A simpleton" def incr(self,x): """Increment x by one. Examples: >>> s = Simpleton() >>> s.incr(1) 2 >>> s.incr(10) 12 """ return x+1 If I build this thing in cython, the Simpleton.incr.__module__ attribute is set to None, while if I make it a pure python module, it's correctly set to the module name: In [18]: p sprimes.primes.Simpleton.incr.__module__ None In [19]: p sprimes.pyprimes.Simpleton.incr.__module__ sprimes.pyprimes This difference causes the doctest to miss any doctest examples that might be included in the methods. Note that for *top-level* functions in the extension module, the __module__ attribute is correctly set: In [21]: p sprimes.primes.primes In [22]: p sprimes.primes.primes.__module__ sprimes.primes so the problem appears to be only for classes. This is running cython 0.9.8. Thanks for any feedback, f From robertwb at math.washington.edu Thu Jul 3 03:19:37 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 2 Jul 2008 18:19:37 -0700 Subject: [Cython] Is this a bug in ye'ole cython? In-Reply-To: References: Message-ID: No, this sounds like a bug. On Jul 2, 2008, at 2:50 PM, Fernando Perez wrote: > Hi folks, > > I'm wondering if this is expected behavior. Consider some trivial > little class like > > class Simpleton: > def __str__(self): > return "A simpleton" > > def incr(self,x): > """Increment x by one. > > Examples: > >>>> s = Simpleton() >>>> s.incr(1) > 2 >>>> s.incr(10) > 12 > """ > return x+1 > > If I build this thing in cython, the Simpleton.incr.__module__ > attribute is set to None, while if I make it a pure python module, > it's correctly set to the module name: > > > In [18]: p sprimes.primes.Simpleton.incr.__module__ > None > > In [19]: p sprimes.pyprimes.Simpleton.incr.__module__ > sprimes.pyprimes > > This difference causes the doctest to miss any doctest examples that > might be included in the methods. > > Note that for *top-level* functions in the extension module, the > __module__ attribute is correctly set: > > In [21]: p sprimes.primes.primes > > > In [22]: p sprimes.primes.primes.__module__ > sprimes.primes > > so the problem appears to be only for classes. > > This is running cython 0.9.8. > > Thanks for any feedback, > > f > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From fperez.net at gmail.com Thu Jul 3 07:40:37 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 2 Jul 2008 22:40:37 -0700 Subject: [Cython] Is this a bug in ye'ole cython? In-Reply-To: References: Message-ID: On Wed, Jul 2, 2008 at 6:19 PM, Robert Bradshaw wrote: > No, this sounds like a bug. OK, thanks. I was able to work around it for my purposes (doctesting of extension code using nose) so it's no biggie for me, but it's still probably worth fixing. Cheers, f From dagss at student.matnat.uio.no Fri Jul 4 23:46:25 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 04 Jul 2008 23:46:25 +0200 Subject: [Cython] Buffer must know exact C types Message-ID: <486E9A31.7040300@student.matnat.uio.no> Currently, wherever a C type is used in Cython, it is enough to specify sort-of what kind of class of type it is... the following will work all right: cdef extern ...: ctypedef int int8 ctypedef int int16 ctypedef int int32 ctypedef int int64 However, when types are used in buffers: cdef object[int16, 3] x one must translate the type (dtype) into a character representation (as listed in http://docs.python.org/lib/module-struct.html). For instance, signed short is "h" and signed int is "i"; and one can imagine int16 to be any of these (which one being decided by the included header-file; NumPy comes with a long list defining intXX for a lot of cases; you can choose whether to use a C-dependant type like "npy_longlong" or a fixed type like "npy_int64"). I see two options: a) Start requiring exact ctypedefs in Cython; at least to get correct buffer behaviour. However, for NumPy this would require a lot of "#ifdefs" Cython-side. (NumPy comes with two options for each type, b) "Somehow" get the C compiler to map the C-compiler-resolved type to the char. With C++ this would be easy: template struct TypeChar {}; template <> struct TypeChar { static const char value = 'h'; }; template <> struct TypeChar { static const char value = 'i'; }; int main() { int t; cout << TypeChar::value; } however I'm not sure if any tricks are possible in C to get something like this? -- Dag Sverre From dagss at student.matnat.uio.no Fri Jul 4 23:49:33 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 04 Jul 2008 23:49:33 +0200 Subject: [Cython] Buffer must know exact C types In-Reply-To: <486E9A31.7040300@student.matnat.uio.no> References: <486E9A31.7040300@student.matnat.uio.no> Message-ID: <486E9AED.2000302@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > int main() { > int t; > cout << TypeChar::value; > } > > however I'm not sure if any tricks are possible in C to get something > like this? > To be clear, C++ would then also allow this: typedef int foo; int main() { foo t; cout << TypeChar::value; } and so, one wouldn't need an "exact" foo ctypedef in Cython, one would just use the template to map the type to the character when needed... -- Dag Sverre From stefan_ml at behnel.de Sat Jul 5 07:11:34 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 05 Jul 2008 07:11:34 +0200 Subject: [Cython] Buffer must know exact C types In-Reply-To: <486E9A31.7040300@student.matnat.uio.no> References: <486E9A31.7040300@student.matnat.uio.no> Message-ID: <486F0286.7060105@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Currently, wherever a C type is used in Cython, it is enough to specify > sort-of what kind of class of type it is... the following will work all > right: > > cdef extern ...: > ctypedef int int8 > ctypedef int int16 > ctypedef int int32 > ctypedef int int64 > > > However, when types are used in buffers: > > cdef object[int16, 3] x > > one must translate the type (dtype) into a character representation (as > listed in http://docs.python.org/lib/module-struct.html). What about going the opposite way and either a) replacing the type definition in buffers by the type character entirely (not sure, but this might not work if you require a specific type name to appear in the C code), or b) by requiring users to provide both the type and the character in a suitable buffer declaration syntax? Stefan From dagss at student.matnat.uio.no Sat Jul 5 11:09:46 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 05 Jul 2008 11:09:46 +0200 Subject: [Cython] Buffer must know exact C types In-Reply-To: <486F0286.7060105@behnel.de> References: <486E9A31.7040300@student.matnat.uio.no> <486F0286.7060105@behnel.de> Message-ID: <486F3A5A.9050204@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> Currently, wherever a C type is used in Cython, it is enough to specify >> sort-of what kind of class of type it is... the following will work all >> right: >> >> cdef extern ...: >> ctypedef int int8 >> ctypedef int int16 >> ctypedef int int32 >> ctypedef int int64 >> >> >> However, when types are used in buffers: >> >> cdef object[int16, 3] x >> >> one must translate the type (dtype) into a character representation (as >> listed in http://docs.python.org/lib/module-struct.html). > > What about going the opposite way and either a) replacing the type definition > in buffers by the type character entirely (not sure, but this might not work > if you require a specific type name to appear in the C code), or b) by > requiring users to provide both the type and the character in a suitable > buffer declaration syntax? Thanks, new ideas like that I was looking for! After thinking about it it turns out that it won't really help for the NumPy "int8/int16/..." types though; it will make the user code potentially unportable: cdef ndarray["i", 2] = zeros((10, 10), dtype=int32) Making it part of the "ctypedef" syntax: ctypedef int int32 [bufchar "i"] would still mean #ifdefs in pxd files. In fact, I think it turns out that the mapping to characters is not the real problem here; the mapping to exact C types is (since the buffer interface will only carry across native C types (the fact that they are encoded as characters is not crucial here)). I think my solution for now will be simply to drop the bit-size typedefs from numpy.pxd and require exact ctypedefs. So one can still do cdef numpy.ndarray[numpy.uint, 2] = ... but not use "uint8" or any other bit-size-specific types. I can also insert a simple check for sizeof(numpy.uint) == sizeof(unsigned int) in the C code at module loading time for all types which are used in buffers (it should be optimized away if things are ok, and will break things early if not). It looks like /usr/include/tgmath.h pulls what I want off; but the code gets very ugly and only works on some compilers; a warning sign that I perhaps don't want to do this but should go for the least obscure route of requiring exact ctypedefs. Though perhaps I can use the floating_type macro to add to the type check proposed above. Anyway, excerpt (I don't understand a tenth of it yet though :-) ): # define __floating_type(type) (((type) 0.25) && ((type) 0.25 - 1)) # define __tgmath_real_type_sub(T, E) \ __typeof__ (*(0 ? (__typeof__ (0 ? (double *) 0 : (void *) (E))) 0 \ : (__typeof__ (0 ? (T *) 0 : (void *) (!(E)))) 0)) ... # define __TGMATH_BINARY_REAL_IMAG(Val1, Val2, Fct, Cfct) \ (__extension__ (((sizeof (__real__ (Val1)) > sizeof (double) \ || sizeof (__real__ (Val2)) > sizeof (double)) \ && __builtin_classify_type (__real__ (Val1) \ + __real__ (Val2)) == 8) \ ? ((sizeof (__real__ (Val1)) == sizeof (Val1) \ && sizeof (__real__ (Val2)) == sizeof (Val2)) \ ? (__typeof ((__tgmath_real_type (Val1)) 0 \ + (__tgmath_real_type (Val2)) 0)) \ __tgml(Fct) (Val1, Val2) \ : (__typeof ((__tgmath_real_type (Val1)) 0 \ + (__tgmath_real_type (Val2)) 0)) \ ... #define cos(Val) __TGMATH_UNARY_REAL_IMAG (Val, cos, ccos) -- Dag Sverre From robertwb at math.washington.edu Sat Jul 5 20:02:36 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 5 Jul 2008 11:02:36 -0700 Subject: [Cython] Buffer must know exact C types In-Reply-To: <486F3A5A.9050204@student.matnat.uio.no> References: <486E9A31.7040300@student.matnat.uio.no> <486F0286.7060105@behnel.de> <486F3A5A.9050204@student.matnat.uio.no> Message-ID: <5B222EAE-FC9E-4647-8BB3-C5E24E5CA40B@math.washington.edu> On Jul 5, 2008, at 2:09 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> Dag Sverre Seljebotn wrote: >>> Currently, wherever a C type is used in Cython, it is enough to >>> specify >>> sort-of what kind of class of type it is... the following will >>> work all >>> right: >>> >>> cdef extern ...: >>> ctypedef int int8 >>> ctypedef int int16 >>> ctypedef int int32 >>> ctypedef int int64 >>> >>> >>> However, when types are used in buffers: >>> >>> cdef object[int16, 3] x >>> >>> one must translate the type (dtype) into a character >>> representation (as >>> listed in http://docs.python.org/lib/module-struct.html). >> >> What about going the opposite way and either a) replacing the type >> definition >> in buffers by the type character entirely (not sure, but this >> might not work >> if you require a specific type name to appear in the C code), or >> b) by >> requiring users to provide both the type and the character in a >> suitable >> buffer declaration syntax? > > Thanks, new ideas like that I was looking for! > > After thinking about it it turns out that it won't really help for the > NumPy "int8/int16/..." types though; it will make the user code > potentially unportable: > > cdef ndarray["i", 2] = zeros((10, 10), dtype=int32) > > Making it part of the "ctypedef" syntax: > > ctypedef int int32 [bufchar "i"] > > would still mean #ifdefs in pxd files. In fact, I think it turns out > that the mapping to characters is not the real problem here; the > mapping > to exact C types is (since the buffer interface will only carry across > native C types (the fact that they are encoded as characters is not > crucial here)). > > I think my solution for now will be simply to drop the bit-size > typedefs > from numpy.pxd and require exact ctypedefs. So one can still do > > cdef numpy.ndarray[numpy.uint, 2] = ... > > but not use "uint8" or any other bit-size-specific types. I can also > insert a simple check for sizeof(numpy.uint) == sizeof(unsigned > int) in > the C code at module loading time for all types which are used in > buffers (it should be optimized away if things are ok, and will break > things early if not). I think we want to have the actual type of the array (e.g. what is returned on arr[0]) in the type spec rather than forcing users to manually translate between "i" and it. If I understand correctly, the issue here is how to map cython/c types to the "format" string of the buffer. There are three things to match: Numeric type (int/float)--this can be detected at compile time by (1)/(2) == 0 vs the buffer format string Signed--this can be detected at compile time by (-1) < 0 vs the buffer format string Size--this can be detected at compile time by sizeof() vs the "itemsize" field of the buffer, or calcsize from the struct module if you need. All other types are required to match exactly (or something like that, some thought would have to be put into how to handle objects). For hybrid types I think it would be enough to just look at the total size after comparing the structs for (int/float/other and signed/ unsigned). Then you could use numpy.uint and numpy.unit32, etc. - Robert From dagss at student.matnat.uio.no Sat Jul 5 20:50:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 05 Jul 2008 20:50:30 +0200 Subject: [Cython] Buffer must know exact C types In-Reply-To: <5B222EAE-FC9E-4647-8BB3-C5E24E5CA40B@math.washington.edu> References: <486E9A31.7040300@student.matnat.uio.no> <486F0286.7060105@behnel.de> <486F3A5A.9050204@student.matnat.uio.no> <5B222EAE-FC9E-4647-8BB3-C5E24E5CA40B@math.washington.edu> Message-ID: <486FC276.1010506@student.matnat.uio.no> Robert Bradshaw wrote: > If I understand correctly, the issue here is how to map cython/c > types to the "format" string of the buffer. There are three things to > match: > > Numeric type (int/float)--this can be detected at compile time by > (1)/(2) == 0 vs the buffer format string > Signed--this can be detected at compile time by (-1) < 0 vs the > buffer format string > Size--this can be detected at compile time by sizeof() vs the > "itemsize" field of the buffer, or calcsize from the struct module if > you need. Very nice! Or simply sizof(type) vs sizeof(int). > All other types are required to match exactly (or something like > that, some thought would have to be put into how to handle objects). > For hybrid types I think it would be enough to just look at the total > size after comparing the structs for (int/float/other and signed/ > unsigned). Then you could use numpy.uint and numpy.unit32, etc. OK, writeup for clarity in the discussion, this is what needs to happen: - A format string is recieved run-time when acquiring a buffer - We have a compile-time assumed type for the buffer - They need to be compared for sanity-checking And this is how I see it happen (which is quite different from what I thought a day ago): - For each type used in a buffer, Cython will output a string-validating inline function for that type. - For structs (and other complicated things) this essentially boils down to checking if a string corresponds to a known structure and is relatively straightforward if potentially messy/wordy. (These will not even be supported until working buffers for simple types hits cython-devel IMO, iterative programming and all that...) - When the exact C type is used directly, something like this may be used (if I bother to treat it specially): int Pyx_typestring_matches_unsigned_int(char* s) { /* skipping the one-char check for now*/ return *s == 'U'; } - When a typedef-ed type is output, something like this is used: int Pyx_typestring_matches_npy_uint8(char* s) { /* skipping the one-char check for now*/ switch (*s) { case 'i': return (((npy_uint8)1)/(npy_uint8)2) == 0) && ((npy_uint8)-1 < 0) && sizeof(npy_uint8) == sizeof(int); ... } } This should translate to a switch statement where every branch but one will return false, and I think it is reasonable to hope the compiler will make it as efficient as the former case. Thanks! :-) -- Dag Sverre From greg.ewing at canterbury.ac.nz Sun Jul 6 02:18:03 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 06 Jul 2008 12:18:03 +1200 Subject: [Cython] Buffer must know exact C types In-Reply-To: <486F3A5A.9050204@student.matnat.uio.no> References: <486E9A31.7040300@student.matnat.uio.no> <486F0286.7060105@behnel.de> <486F3A5A.9050204@student.matnat.uio.no> Message-ID: <48700F3B.7080109@canterbury.ac.nz> How does one write portable code in C to do this? If it's possible to write portably in C, then there must be a way to have Cython generate that portable C from some portable representation in Cython source. -- Greg From robertwb at math.washington.edu Sun Jul 6 02:55:24 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 5 Jul 2008 17:55:24 -0700 Subject: [Cython] Buffer must know exact C types In-Reply-To: <486FC276.1010506@student.matnat.uio.no> References: <486E9A31.7040300@student.matnat.uio.no> <486F0286.7060105@behnel.de> <486F3A5A.9050204@student.matnat.uio.no> <5B222EAE-FC9E-4647-8BB3-C5E24E5CA40B@math.washington.edu> <486FC276.1010506@student.matnat.uio.no> Message-ID: On Jul 5, 2008, at 11:50 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> If I understand correctly, the issue here is how to map cython/c >> types to the "format" string of the buffer. There are three things to >> match: >> >> Numeric type (int/float)--this can be detected at compile time by >> (1)/(2) == 0 vs the buffer format string >> Signed--this can be detected at compile time by (-1) < 0 vs the >> buffer format string >> Size--this can be detected at compile time by sizeof() vs the >> "itemsize" field of the buffer, or calcsize from the struct module if >> you need. > > Very nice! > > Or simply sizof(type) vs sizeof(int). > >> All other types are required to match exactly (or something like >> that, some thought would have to be put into how to handle objects). >> For hybrid types I think it would be enough to just look at the total >> size after comparing the structs for (int/float/other and signed/ >> unsigned). Then you could use numpy.uint and numpy.unit32, etc. > > > OK, writeup for clarity in the discussion, this is what needs to > happen: > > - A format string is recieved run-time when acquiring a buffer > - We have a compile-time assumed type for the buffer > - They need to be compared for sanity-checking Exactly. > And this is how I see it happen (which is quite different from what I > thought a day ago): > > - For each type used in a buffer, Cython will output a string- > validating > inline function for that type. > > - For structs (and other complicated things) this essentially boils > down > to checking if a string corresponds to a known structure and is > relatively straightforward if potentially messy/wordy. (These will not > even be supported until working buffers for simple types hits > cython-devel IMO, iterative programming and all that...) > > - When the exact C type is used directly, something like this may be > used (if I bother to treat it specially): > > int Pyx_typestring_matches_unsigned_int(char* s) { > /* skipping the one-char check for now*/ > return *s == 'U'; > } > > - When a typedef-ed type is output, something like this is used: > > int Pyx_typestring_matches_npy_uint8(char* s) { > > /* skipping the one-char check for now*/ > > switch (*s) { > case 'i': > return (((npy_uint8)1)/(npy_uint8)2) == 0) && > ((npy_uint8)-1 < 0) && sizeof(npy_uint8) == sizeof(int); > ... > } > } > > This should translate to a switch statement where every branch but one > will return false, and I think it is reasonable to hope the compiler > will make it as efficient as the former case. Yep, looks good to me. The compiler will have no trouble evaluating these constant expressions and pruning the tree. In retrospect, I think the ((type)1)/((type)2) == 0 trick is probably overkill, the person who declares the type should ctypedef it as an int or float correctly, we don't need to be double-guessing that. > Thanks! :-) Any time :-) - Robert From dagss at student.matnat.uio.no Sun Jul 6 11:00:47 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 6 Jul 2008 11:00:47 +0200 (CEST) Subject: [Cython] Buffer must know exact C types In-Reply-To: <48700F3B.7080109@canterbury.ac.nz> References: <486E9A31.7040300@student.matnat.uio.no> <486F0286.7060105@behnel.de> <486F3A5A.9050204@student.matnat.uio.no> <48700F3B.7080109@canterbury.ac.nz> Message-ID: <58465.193.157.229.67.1215334847.squirrel@webmail.uio.no> Greg wrote: > How does one write portable code in C to do this? > > If it's possible to write portably in C, then there > must be a way to have Cython generate that portable > C from some portable representation in Cython source. The problem (which is now solved) arises because Cython ctypedefs are not required to be exact, only approximate. When writing C code, your typedefs are exact and you can always know (at least in principle) the exact type you are dealing with. So if you have something like this: #ifdef A typedef short foo #else typedef int foo #endif ...then you can always throw in "#define FOO_LETTER 'h'" or "#define FOO_LETTER 'i'" inside those #ifdefs. However, if it says this in Cython: ctypedef int foo then you don't "really" know the exact type of foo (and this feature will be relied on for numpy.pxd to provide "int16", "int64" etc., and I wouldn't like to remove it and thus require duplication of all such #ifdefs in the pxd files). Dag Sverre From gael.varoquaux at normalesup.org Sun Jul 6 23:07:31 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 6 Jul 2008 23:07:31 +0200 Subject: [Cython] Schedule for the SciPy08 conferencez Message-ID: <20080706210731.GC25810@phare.normalesup.org> We have received a large number of excellent contributions for papers for the SciPy 2008 conference. The program committee has had to make a difficult selection and we are happy to bring to you a preliminary schedule: Thursday ========= **8:00** Registration/Breakfast **8:55** Welcome (Travis Vaught) **9:10** Keynote (Alex Martelli) **10:00** State of SciPy (Travis Vaught, Jarrod Millman) **10:40** -- Break -- **11:00** Sympy - Python library for symbolic mathematics: introduction and applications (Ond?ej ?ertik) **11:40** Interval arithmetic: Python implementation and applications (Stefano Taschini) **12:00** Experiences Using Scipy for Computer Vision Research (Damian Eads) **12:20** -- Lunch -- **1:40** The new NumPy documentation framework (St?fan Van der Walt) **2:00** Matplotlib solves the riddle of the sphinx (Michael Droettboom) **2:40** The SciPy documentation project (Joe Harrington) **3:00** -- Break -- **3:40** Sage: creating a viable free Python-based open source alternatice to Magma, Maple, Mathematica and Matlab (William Stein) **4:20** Open space for lightning talks Friday ======== **8:30** Breakfast **9:00** Pysynphot: A Python Re-Implementation of a Legacy App in Astronomy (Perry Greenfield) **9:40** How the Large Synoptic Survey Telescope (LSST) is using Python (Robert Lupton) **10:00** Real-time Astronomical Time-series Classification and Broadcast Pipeline (Dan Starr) **10:20** Analysis and Visualization of Multi-Scale Astrophysical Simulations using Python and NumPy (Matthew Turk) **10:40** -- Break -- **11:00** Exploring network structure, dynamics, and function using NetworkX (Aric Hagberg) **11:40** Mayavi: Making 3D data visualization reusable (Prabhu Ramachandran, Ga?l Varoquaux) **12:00** Finite Element Modeling of Contact and Impact Problems Using Python (Ryan Krauss) **12:20** -- Lunch -- **2:00** PyCircuitScape: A Tool for Landscape Ecology (Viral Shah) **2:20** Summarizing Complexity in High Dimensional Spaces (Karl Young) **2:40** UFuncs: A generic function mechanism in Python (Travis Oliphant) **3:20** -- Break -- **3:40** NumPy Optimization: Manual tuning and automated approaches (Evan Patterson) **4:00** Converting Python functions to dynamically-compiled C (Ilan Schnell) **4:20** unPython: Converting Python numerical programs into C (Rahul Garg) **4:40** Implementing the Grammar of Graphics for Python (Robert Kern) **5:00** Ask the experts session. A more detailled booklet including the abstract text will be available soon. We are looking forward to seeing you in Caltech, Ga?l Varoquaux, on behalf of the program committee. -- SciPy2008 conference. Program committee Anne Archibald, McGill University Matthew Brett Perry Greenfield, Space Telescope Science Institute Charles Harris Ryan Krauss, Southern Illinois University Ga?l Varoquaux St?fan van der Walt, University of Stellenbosch From charlie at openmoko.org Tue Jul 8 05:37:10 2008 From: charlie at openmoko.org (Guillaume Chereau) Date: Tue, 08 Jul 2008 11:37:10 +0800 Subject: [Cython] Object not released when calling cpdef method from inherited class ? Message-ID: <1215488230.20373.14.camel@nikopol> Hello all, I am currently using cython for a project related to openmoko cell phone : http://charlie137-2.blogspot.com/2008/07/introducing-tichy.html I noticed that the memory is constantly increasing when I use cython. I tracked down the problem to the case where I create a subclass of a cython class and then redefine a cpdef method, asking it to call the parent method, but ONLY if the cpdef method then call an other cpdef method ! Here is the smallest example I could come with that fails : == test.pyx == cdef class A: cpdef func(self): return cpdef test(self): self.func() == main.py == import test import gc class B(test.A): def test(self): test.A.test(self) b = B() for i in range(10): b.test() gc.collect() print len(gc.get_objects()) The output will show that some objects are not released. Is it a known bug ? Is there a way to avoid it ? I tried with both Cython 0.9.6 and Cython 0.9.8 cheers, -Charlie -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080708/0fb6b6e8/attachment.pgp From robertwb at math.washington.edu Tue Jul 8 05:42:38 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 7 Jul 2008 20:42:38 -0700 Subject: [Cython] Object not released when calling cpdef method from inherited class ? In-Reply-To: <1215488230.20373.14.camel@nikopol> References: <1215488230.20373.14.camel@nikopol> Message-ID: On Jul 7, 2008, at 8:37 PM, Guillaume Chereau wrote: > Hello all, > > I am currently using cython for a project related to openmoko cell > phone : http://charlie137-2.blogspot.com/2008/07/introducing- > tichy.html > > I noticed that the memory is constantly increasing when I use > cython. I > tracked down the problem to the case where I create a subclass of a > cython class and then redefine a cpdef method, asking it to call the > parent method, but ONLY if the cpdef method then call an other cpdef > method ! > > Here is the smallest example I could come with that fails : > > > == test.pyx == > > cdef class A: > cpdef func(self): > return > > cpdef test(self): > self.func() > > > == main.py == > > import test > import gc > > class B(test.A): > def test(self): > test.A.test(self) > > b = B() > > for i in range(10): > b.test() > gc.collect() > print len(gc.get_objects()) > > > The output will show that some objects are not released. > > Is it a known bug ? Is there a way to avoid it ? This isn't a known bug, but does sound like something fishy is going on. Thanks for the report. The easiest fix I have for now is to make one or the other methods an ordinary def method (sacrificing speed for better memory handling), but I'll be looking into it. > I tried with both Cython 0.9.6 and Cython 0.9.8 > > cheers, > -Charlie > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080707/043950cb/attachment.pgp From languitar at semipol.de Tue Jul 8 18:49:53 2008 From: languitar at semipol.de (Johannes Wienke) Date: Tue, 08 Jul 2008 18:49:53 +0200 Subject: [Cython] Function signature does not match previous declaration Message-ID: <48739AB1.8080900@semipol.de> Hi, I've got one header file in my project, that raises a lot of "Function signature does not match previous declaration" warnings but I don't know why, especially because everything works. One warning e. g. is this one: warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does not match previous declaration Line 87 in that file is: iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) iwImage and iwType are defined like this: ctypedef struct iwImage: guchar **data int planes iwType type int width, height int rowstride iwColtab ctab void *reserved ctypedef enum iwType: IW_8U IW_16U IW_32S IW_FLOAT IW_DOUBLE The original definitions in the header file look like this: typedef struct iwImage { guchar **data; int planes; iwType type; int width, height; int rowstride; iwColtab ctab; void *reserved; } iwImage; typedef enum { IW_8U, IW_16U, IW_32S, IW_FLOAT, IW_DOUBLE } iwType; iwImage* iw_img_new_alloc (int width, int height, int planes, iwType type); Has anyone got an idea why cython generates these warnings? Thanks for the help! Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080708/75cc49af/attachment.pgp From cwitty at newtonlabs.com Tue Jul 8 19:55:31 2008 From: cwitty at newtonlabs.com (Carl Witty) Date: Tue, 8 Jul 2008 10:55:31 -0700 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <48739AB1.8080900@semipol.de> References: <48739AB1.8080900@semipol.de> Message-ID: On Tue, Jul 8, 2008 at 9:49 AM, Johannes Wienke wrote: > Hi, > > I've got one header file in my project, that raises a lot of "Function > signature does not match previous declaration" warnings but I don't know > why, especially because everything works. One warning e. g. is this one: > > warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does > not match previous declaration > > Line 87 in that file is: > iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) ... This warning does not mean that the Cython declaration doesn't match the C declaration (Cython knows nothing about the C declaration). Instead, it means that there are two Cython declarations that don't match. Search all your .pxd and .pxi files for another declaration of iw_img_new_alloc; also, make sure that Gimage.pxi is not included multiple times. Carl From languitar at semipol.de Tue Jul 8 20:18:33 2008 From: languitar at semipol.de (Johannes Wienke) Date: Tue, 08 Jul 2008 20:18:33 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: References: <48739AB1.8080900@semipol.de> Message-ID: <4873AF79.8000804@semipol.de> Am 07/08/2008 07:55 PM schrieb Carl Witty: > On Tue, Jul 8, 2008 at 9:49 AM, Johannes Wienke wrote: >> warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does >> not match previous declaration >> >> Line 87 in that file is: >> iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) > ... > > This warning does not mean that the Cython declaration doesn't match > the C declaration (Cython knows nothing about the C declaration). > Instead, it means that there are two Cython declarations that don't > match. Search all your .pxd and .pxi files for another declaration of > iw_img_new_alloc; also, make sure that Gimage.pxi is not included > multiple times. I can't find any of those things in my project: This is what I can grab about thos things: languitar at bird ~/BA-workspace/shIP $ grep -in "Gimage.pxi" `find . -name '*.px*'` | grep -v svn ./src/ship/icewing/preview.pxd:2:include "icewing/gui/Gimage.pxi" ./src/ship/icewing/image_utils.pxd:1:include "icewing/gui/Gimage.pxi" ./include/icewing/gui/Grender.pxi:3:include "icewing/gui/Gimage.pxi" ./include/icewing/gui/Gpreview.pxi:4:include "icewing/gui/Gimage.pxi" languitar at bird ~/BA-workspace/shIP $ grep -in "Gimage.pxi" `find . -name '*.py*'` | grep -v svn ./src/ship/icewing/converters.pyx:9:include "icewing/gui/Gimage.pxi" languitar at bird ~/BA-workspace/shIP $ grep -in "iw_img_new_alloc" `find . -name '*.px*'` | grep -v svn ./include/icewing/gui/Gimage.pxi:87: iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) languitar at bird ~/BA-workspace/shIP $ grep -in "iw_img_new_alloc" `find . -name '*.py*'` | grep -v svn ./src/ship/icewing/converters.pyx:241: cdef iwImage *image = iw_img_new_alloc(data.size[0], - Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080708/9d785303/attachment.pgp From dagss at student.matnat.uio.no Tue Jul 8 20:24:46 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 08 Jul 2008 20:24:46 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <4873AF79.8000804@semipol.de> References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> Message-ID: <4873B0EE.7030508@student.matnat.uio.no> Johannes Wienke wrote: > Am 07/08/2008 07:55 PM schrieb Carl Witty: >> On Tue, Jul 8, 2008 at 9:49 AM, Johannes Wienke wrote: >>> warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does >>> not match previous declaration >>> >>> Line 87 in that file is: >>> iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) >> ... >> >> This warning does not mean that the Cython declaration doesn't match >> the C declaration (Cython knows nothing about the C declaration). >> Instead, it means that there are two Cython declarations that don't >> match. Search all your .pxd and .pxi files for another declaration of >> iw_img_new_alloc; also, make sure that Gimage.pxi is not included >> multiple times. > > I can't find any of those things in my project: This is what I can grab > about thos things: > > languitar at bird ~/BA-workspace/shIP $ grep -in "Gimage.pxi" `find . -name > '*.px*'` | grep -v svn > ./src/ship/icewing/preview.pxd:2:include "icewing/gui/Gimage.pxi" > ./src/ship/icewing/image_utils.pxd:1:include "icewing/gui/Gimage.pxi" > ./include/icewing/gui/Grender.pxi:3:include "icewing/gui/Gimage.pxi" > ./include/icewing/gui/Gpreview.pxi:4:include "icewing/gui/Gimage.pxi" Here it is included multiple times, each pxd file gets a copy of the function definitions in the pxi files. You're probably using pxi in the wrong place here, switch it to a pxd file and use cimport. -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 8 20:26:28 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 08 Jul 2008 20:26:28 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <4873B0EE.7030508@student.matnat.uio.no> References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> <4873B0EE.7030508@student.matnat.uio.no> Message-ID: <4873B154.2090303@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Johannes Wienke wrote: >> Am 07/08/2008 07:55 PM schrieb Carl Witty: >>> On Tue, Jul 8, 2008 at 9:49 AM, Johannes Wienke wrote: >>>> warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does >>>> not match previous declaration >>>> >>>> Line 87 in that file is: >>>> iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) >>> ... >>> >>> This warning does not mean that the Cython declaration doesn't match >>> the C declaration (Cython knows nothing about the C declaration). >>> Instead, it means that there are two Cython declarations that don't >>> match. Search all your .pxd and .pxi files for another declaration of >>> iw_img_new_alloc; also, make sure that Gimage.pxi is not included >>> multiple times. >> I can't find any of those things in my project: This is what I can grab >> about thos things: >> >> languitar at bird ~/BA-workspace/shIP $ grep -in "Gimage.pxi" `find . -name >> '*.px*'` | grep -v svn >> ./src/ship/icewing/preview.pxd:2:include "icewing/gui/Gimage.pxi" >> ./src/ship/icewing/image_utils.pxd:1:include "icewing/gui/Gimage.pxi" >> ./include/icewing/gui/Grender.pxi:3:include "icewing/gui/Gimage.pxi" >> ./include/icewing/gui/Gpreview.pxi:4:include "icewing/gui/Gimage.pxi" > > Here it is included multiple times, each pxd file gets a copy of the > function definitions in the pxi files. You're probably using pxi in the > wrong place here, switch it to a pxd file and use cimport. Thinking about it this is likely not the cause of your problem though... -- Dag Sverre From languitar at semipol.de Tue Jul 8 20:26:10 2008 From: languitar at semipol.de (Johannes Wienke) Date: Tue, 08 Jul 2008 20:26:10 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <4873B0EE.7030508@student.matnat.uio.no> References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> <4873B0EE.7030508@student.matnat.uio.no> Message-ID: <4873B142.2010802@semipol.de> Am 07/08/2008 08:24 PM schrieb Dag Sverre Seljebotn: > Johannes Wienke wrote: >> Am 07/08/2008 07:55 PM schrieb Carl Witty: >>> On Tue, Jul 8, 2008 at 9:49 AM, Johannes Wienke wrote: >>>> warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does >>>> not match previous declaration >>>> >>>> Line 87 in that file is: >>>> iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) >>> ... [...] >> languitar at bird ~/BA-workspace/shIP $ grep -in "Gimage.pxi" `find . -name >> '*.px*'` | grep -v svn >> ./src/ship/icewing/preview.pxd:2:include "icewing/gui/Gimage.pxi" >> ./src/ship/icewing/image_utils.pxd:1:include "icewing/gui/Gimage.pxi" >> ./include/icewing/gui/Grender.pxi:3:include "icewing/gui/Gimage.pxi" >> ./include/icewing/gui/Gpreview.pxi:4:include "icewing/gui/Gimage.pxi" > > Here it is included multiple times, each pxd file gets a copy of the > function definitions in the pxi files. You're probably using pxi in the > wrong place here, switch it to a pxd file and use cimport. > Well, I thought pxi files are used for definitions from other header files and pxd files for definitions of your own cython code. Or is that wrong? Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080708/b930ab3f/attachment.pgp From dagss at student.matnat.uio.no Tue Jul 8 20:35:21 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 08 Jul 2008 20:35:21 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <4873B142.2010802@semipol.de> References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> <4873B0EE.7030508@student.matnat.uio.no> <4873B142.2010802@semipol.de> Message-ID: <4873B369.8090703@student.matnat.uio.no> Johannes Wienke wrote: > Am 07/08/2008 08:24 PM schrieb Dag Sverre Seljebotn: >> Johannes Wienke wrote: >>> Am 07/08/2008 07:55 PM schrieb Carl Witty: >>>> On Tue, Jul 8, 2008 at 9:49 AM, Johannes Wienke wrote: >>>>> warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does >>>>> not match previous declaration >>>>> >>>>> Line 87 in that file is: >>>>> iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) >>>> ... > [...] >>> languitar at bird ~/BA-workspace/shIP $ grep -in "Gimage.pxi" `find . -name >>> '*.px*'` | grep -v svn >>> ./src/ship/icewing/preview.pxd:2:include "icewing/gui/Gimage.pxi" >>> ./src/ship/icewing/image_utils.pxd:1:include "icewing/gui/Gimage.pxi" >>> ./include/icewing/gui/Grender.pxi:3:include "icewing/gui/Gimage.pxi" >>> ./include/icewing/gui/Gpreview.pxi:4:include "icewing/gui/Gimage.pxi" >> Here it is included multiple times, each pxd file gets a copy of the >> function definitions in the pxi files. You're probably using pxi in the >> wrong place here, switch it to a pxd file and use cimport. >> > > Well, I thought pxi files are used for definitions from other header > files and pxd files for definitions of your own cython code. Or is that > wrong? pxi files are not for any specific purpose; they simply insert the file verbatim into the spot of the include (and thus often can lead to defining things too many times). (There are ups and downs to this; I'd never use them myself for any purpose but that's only a subjective feeling). It is perfectly ok to cimport a pxd file into another pxd file. -- Dag Sverre From languitar at semipol.de Tue Jul 8 20:45:10 2008 From: languitar at semipol.de (Johannes Wienke) Date: Tue, 08 Jul 2008 20:45:10 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <4873B369.8090703@student.matnat.uio.no> References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> <4873B0EE.7030508@student.matnat.uio.no> <4873B142.2010802@semipol.de> <4873B369.8090703@student.matnat.uio.no> Message-ID: <4873B5B6.1080905@semipol.de> Am 07/08/2008 08:35 PM schrieb Dag Sverre Seljebotn: > Johannes Wienke wrote: >> Am 07/08/2008 08:24 PM schrieb Dag Sverre Seljebotn: >>> Johannes Wienke wrote: >>>> Am 07/08/2008 07:55 PM schrieb Carl Witty: >>>>> On Tue, Jul 8, 2008 at 9:49 AM, Johannes Wienke wrote: >>>>>> warning: include/icewing/gui/Gimage.pxi:87:11: Function signature does >>>>>> not match previous declaration >>>>>> >>>>>> Line 87 in that file is: >>>>>> iwImage* iw_img_new_alloc(int width, int height, int planes, iwType type) >>>>> ... >> [...] >>>> languitar at bird ~/BA-workspace/shIP $ grep -in "Gimage.pxi" `find . -name >>>> '*.px*'` | grep -v svn >>>> ./src/ship/icewing/preview.pxd:2:include "icewing/gui/Gimage.pxi" >>>> ./src/ship/icewing/image_utils.pxd:1:include "icewing/gui/Gimage.pxi" >>>> ./include/icewing/gui/Grender.pxi:3:include "icewing/gui/Gimage.pxi" >>>> ./include/icewing/gui/Gpreview.pxi:4:include "icewing/gui/Gimage.pxi" >>> Here it is included multiple times, each pxd file gets a copy of the >>> function definitions in the pxi files. You're probably using pxi in the >>> wrong place here, switch it to a pxd file and use cimport. >>> >> Well, I thought pxi files are used for definitions from other header >> files and pxd files for definitions of your own cython code. Or is that >> wrong? > > pxi files are not for any specific purpose; they simply insert the file > verbatim into the spot of the include (and thus often can lead to > defining things too many times). (There are ups and downs to this; I'd > never use them myself for any purpose but that's only a subjective > feeling). It is perfectly ok to cimport a pxd file into another pxd file. Ok, nevertheless I use the approach with the pxi files for all other modules and definitions in my program and only Gimage.pxi generates these warnings. Furthermore only some of the functions have those warnings, not all in the file. - Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080708/c11eca0b/attachment.pgp From cwitty at newtonlabs.com Tue Jul 8 21:18:38 2008 From: cwitty at newtonlabs.com (Carl Witty) Date: Tue, 8 Jul 2008 12:18:38 -0700 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <4873B5B6.1080905@semipol.de> References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> <4873B0EE.7030508@student.matnat.uio.no> <4873B142.2010802@semipol.de> <4873B369.8090703@student.matnat.uio.no> <4873B5B6.1080905@semipol.de> Message-ID: On Tue, Jul 8, 2008 at 11:45 AM, Johannes Wienke wrote: > Ok, nevertheless I use the approach with the pxi files for all other > modules and definitions in my program and only Gimage.pxi generates > these warnings. Furthermore only some of the functions have those > warnings, not all in the file. I suspect that with multiple inclusions, either iwType or iwImage (or both) defines a "new" type each time it's loaded. So the first declaration doesn't match the second declaration, because it's got "different" argument types. If you look at which functions have warnings: is it, perhaps, only the functions that take an argument of type iwType? If not, is there some other pattern of that sort in which functions have warnings? Carl From languitar at semipol.de Tue Jul 8 21:41:08 2008 From: languitar at semipol.de (Johannes Wienke) Date: Tue, 08 Jul 2008 21:41:08 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> <4873B0EE.7030508@student.matnat.uio.no> <4873B142.2010802@semipol.de> <4873B369.8090703@student.matnat.uio.no> <4873B5B6.1080905@semipol.de> Message-ID: <4873C2D4.3060706@semipol.de> Am 07/08/2008 09:18 PM schrieb Carl Witty: > On Tue, Jul 8, 2008 at 11:45 AM, Johannes Wienke wrote: >> Ok, nevertheless I use the approach with the pxi files for all other >> modules and definitions in my program and only Gimage.pxi generates >> these warnings. Furthermore only some of the functions have those >> warnings, not all in the file. > > I suspect that with multiple inclusions, either iwType or iwImage (or > both) defines a "new" type each time it's loaded. So the first > declaration doesn't match the second declaration, because it's got > "different" argument types. > > If you look at which functions have warnings: is it, perhaps, only the > functions that take an argument of type iwType? If not, is there some > other pattern of that sort in which functions have warnings? It seems that all functions, that use mapped enums are the problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080708/b76c5dbb/attachment-0001.pgp From n.tautenhahn at gmx.de Wed Jul 9 19:30:50 2008 From: n.tautenhahn at gmx.de (Nikolas Tautenhahn) Date: Wed, 09 Jul 2008 19:30:50 +0200 Subject: [Cython] Pex / 64bit issues (?) Message-ID: <4874F5CA.2020100@gmx.de> Hi there, I have a small snippet of pex code (actually it is only one function, no cdef classes or stuff, but uses decorated arrays) which runs fine on my laptop (opensuse 10.3, 32bit linux) but won't run at my computer at university (64bit, ubuntu). Running pex on my .px file gets me a Traceback (most recent call last): File "/home/btmwtaut/bin/pex", line 221, in module=pimport(name,force_build=force_build) File "/home/btmwtaut/src/pex-0.8/pex/pimport.py", line 368, in pimport py_module_obj = do_python_import(pex_module_name,abspath,pkg_context) File "/home/btmwtaut/src/pex-0.8/pex/pimport.py", line 236, in do_python_import module_obj=my_import(pex_module_name,global_dict) File "/home/btmwtaut/src/pex-0.8/pex/pimport.py", line 224, in my_import mod = __import__(name,global_dict) File "c_heur_solve.pyx", line 46, in c_heur_solve assert sizeof(long)==4 # use these for our checksum AssertionError I already use %whencompiling: scope.pragma_gen_fastio=False %whencompiling: scope.pragma_gen_hashmeth=False I have numpy and python sources installed, I used a standalone cython on this account without problems. (Pex, however, uses its own cython for this (I think - at least it complained when it couldn't find cython.py in ~/src/pex-0.8/.../)) Any thoughts what the problem might be? Best wishes, Nikolas Tautenhahn uname -a: Linux btmwxi 2.6.22-15-generic #1 SMP Tue Jun 10 08:52:15 UTC 2008 x86_64 GNU/Linux Python version: 2.5.1 c_heur_solve.px: %whencompiling: scope.pragma_gen_fastio=False %whencompiling: scope.pragma_gen_hashmeth=False def rowcheck(g, evil_list): cdef lenn = len(g.n), lenM = len(g.M), lenl = len(g.losers), lene = len(evil_list), res rowset = set(g.rows) loserset = set(g.loserlist) for evil in evil_list: if evil[0].issubset(rowset) and evil[1].issubset(loserset): return False cdef ndarray W cdef ndarray L cdef ndarray R cdef bint done_something = 0 for i from 0 <= i < lenM: for j from 0 <= j < lenn: W{i,j} = g.M[i][j] for i from 0 <= i < lenl: for j from 0 <= j < lenn: L{i,j} = g.losers[i][j] for i from 0 <= i < lenM: for j from 0 <= j < lenl: for k from 0 <= k < lenn: R{i*lenl + j, k} = W{i,k} - L{j,k} for i from 0 <= i < lenM*lenl: for j from i <= j < lenM*lenl: found_m1 = -2 for k from 0 <= k < lenn: res = R{i,k} + R{j,k} if res != 0: if res == -1 and found_m1 < 0: found_m1 = k elif res == 1 and found_m1 == k-1: continue else: break else: wset = set() lset = set() wset.add(g.rows[i / lenl]) wset.add(g.rows[j / lenl]) lset.add(g.loserlist[i % lenl]) lset.add(g.loserlist[j % lenl]) evil_list.add((frozenset(wset), frozenset(lset))) return False return True From parlar at gmail.com Thu Jul 10 05:55:42 2008 From: parlar at gmail.com (Jay Parlar) Date: Wed, 9 Jul 2008 23:55:42 -0400 Subject: [Cython] cimport across packages Message-ID: I'm having trouble with .pxd files across packages. My main program directory has a few packages in it, one called 'sm' and one called 'filters'. In 'filters', I have a dc.pyx and a dc.pxd, implementing an extension class called DC. In 'sm', I have a sensor_manager.pyx that does the following: from filters.dc cimport DC I get the following error when running setup.py: /Users/parlarjb/src/gui/sm/sensor_manager.pyx:31:0: 'filters.dc.pxd' not found The important part of my setup.py is: setup( name = "Vibration Analysis", ext_modules=[ Extension("filters.dc", ["filters/dc.pyx"]), Extension("sm.sensor_manager", ["sm/sensor_manager.pyx"]), ], cmdclass = {'build_ext': build_ext} ) Any thoughts? Jay P. From jim-crow at rambler.ru Thu Jul 10 06:39:43 2008 From: jim-crow at rambler.ru (Anatoly A. Kazantsev) Date: Thu, 10 Jul 2008 11:39:43 +0700 Subject: [Cython] cimport across packages In-Reply-To: References: Message-ID: <20080710113943.2abd5584.jim-crow@rambler.ru> On Wed, 9 Jul 2008 23:55:42 -0400 "Jay Parlar" wrote: > I'm having trouble with .pxd files across packages. > > My main program directory has a few packages in it, one called 'sm' > and one called 'filters'. > > In 'filters', I have a dc.pyx and a dc.pxd, implementing an extension > class called DC. > > In 'sm', I have a sensor_manager.pyx that does the following: > > from filters.dc cimport DC > > I get the following error when running setup.py: > > /Users/parlarjb/src/gui/sm/sensor_manager.pyx:31:0: 'filters.dc.pxd' not found > > The important part of my setup.py is: > > setup( > name = "Vibration Analysis", > ext_modules=[ > Extension("filters.dc", ["filters/dc.pyx"]), > Extension("sm.sensor_manager", ["sm/sensor_manager.pyx"]), > ], > cmdclass = {'build_ext': build_ext} > ) > > > Any thoughts? > > Jay P. Rename 'dc.pxd' to 'filters.dc.pxd'. That's works for me. -- Anatoly A. Kazantsev Protect your digital freedom and privacy, eliminate DRM, learn more at http://www.defectivebydesign.org/what_is_drm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080710/b4c6a0e6/attachment.pgp From parlar at gmail.com Thu Jul 10 07:02:48 2008 From: parlar at gmail.com (Jay Parlar) Date: Thu, 10 Jul 2008 01:02:48 -0400 Subject: [Cython] cimport across packages In-Reply-To: <20080710113943.2abd5584.jim-crow@rambler.ru> References: <20080710113943.2abd5584.jim-crow@rambler.ru> Message-ID: > Rename 'dc.pxd' to 'filters.dc.pxd'. > That's works for me. > > That doesn't seem to make a difference for me... I tried renaming it to filters.dc.pxd, and putting it in both the main directory, and the filters directory. Jay P. From stefan_ml at behnel.de Thu Jul 10 07:39:03 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 10 Jul 2008 07:39:03 +0200 Subject: [Cython] cimport across packages In-Reply-To: References: Message-ID: <4875A077.1030404@behnel.de> Hi, I assume you are using Cython 0.9.8? Jay Parlar wrote: > I'm having trouble with .pxd files across packages. > > My main program directory has a few packages in it, one called 'sm' > and one called 'filters'. Are these really packages? I.e. do they have an __init__.py file? Stefan From stefan_ml at behnel.de Thu Jul 10 08:40:08 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 10 Jul 2008 08:40:08 +0200 Subject: [Cython] generators Message-ID: <4875AEC8.9070006@behnel.de> Hi, since there isn't currently a dedicated CEP page on generator support, I'm wondering how they should be implemented. I kind of imagine that once closures are in place, it might be enough to put all local variables of a generator function into a closure to keep their state, and then put a label behind each yield statement and store the current label in the closure whenever we return from the function. That would allow us to just jump to the label on re-entry and continue the execution. I do see that there might be a problem with deallocation of the closure, though, as it's hard to tell when the generator is at an end. So an extended way of doing it would be a transformation of the function into an extension class, where local variables become class attributes. In order to avoid having to split up the function body, it could be transformed into a single __next__ method, and the execution could follow the goto-label scheme as described above. That would even be independent of the closure support, although we might want to avoid code duplication here. Would that make sense? Stefan From robertwb at math.washington.edu Thu Jul 10 09:17:34 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 10 Jul 2008 00:17:34 -0700 (PDT) Subject: [Cython] generators In-Reply-To: <4875AEC8.9070006@behnel.de> References: <4875AEC8.9070006@behnel.de> Message-ID: On Thu, 10 Jul 2008, Stefan Behnel wrote: > Hi, > > since there isn't currently a dedicated CEP page on generator support, I'm > wondering how they should be implemented. Sorry, I see them as natural (and relatively straightforward) extensions of closures. > I kind of imagine that once closures are in place, it might be enough to put > all local variables of a generator function into a closure to keep their > state, and then put a label behind each yield statement and store the current > label in the closure whenever we return from the function. That would allow us > to just jump to the label on re-entry and continue the execution. This is exactly how I was planning on implementing them. > I do see that there might be a problem with deallocation of the closure, > though, as it's hard to tell when the generator is at an end. So an extended > way of doing it would be a transformation of the function into an extension > class, where local variables become class attributes. In order to avoid having > to split up the function body, it could be transformed into a single __next__ > method, and the execution could follow the goto-label scheme as described > above. That would even be independent of the closure support, although we > might want to avoid code duplication here. > > Would that make sense? > > Stefan Yes. This is in fact very much in line with how closures would be implemented--there would be an extension class that would hold the state (and garbage collection would happen when this object went out of scope, so no issues for generators or closures). The difference would be that closures get the function body put into the __call__ method and generators get the function body put into the __next__ method (with the jump table of course). One hitch that we ran into with closures (and the main reason IMHO that we didn't have time to finish them up like planned) is that variables are not bound at creation time, rather the scope is bound. Thus a function containing a closure needs to have special code as well as the inner function itself. - Robert From stefan_ml at behnel.de Thu Jul 10 09:30:26 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 10 Jul 2008 09:30:26 +0200 Subject: [Cython] Pex / 64bit issues (?) In-Reply-To: <4874F5CA.2020100@gmx.de> References: <4874F5CA.2020100@gmx.de> Message-ID: <4875BA92.8030203@behnel.de> Hi, Nikolas Tautenhahn wrote: > File "/home/btmwtaut/src/pex-0.8/pex/pimport.py", line 224, in my_import > mod = __import__(name,global_dict) > File "c_heur_solve.pyx", line 46, in c_heur_solve > assert sizeof(long)==4 # use these for our checksum > AssertionError This obviously can't work on a 64 bit machine. Must be PEX generating it, and it looks like a bug to me, as your code doesn't suggest any such assertion. Stefan From dagss at student.matnat.uio.no Thu Jul 10 10:11:01 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 10 Jul 2008 10:11:01 +0200 Subject: [Cython] generators In-Reply-To: References: <4875AEC8.9070006@behnel.de> Message-ID: <4875C415.6010706@student.matnat.uio.no> Robert Bradshaw wrote: > One hitch that we ran into with closures (and the main reason IMHO that we > didn't have time to finish them up like planned) is that variables are not > bound at creation time, rather the scope is bound. Thus a function > containing a closure needs to have special code as well as the inner > function itself. To be even more specific, it would help if self.result_code everywhere was calculated during code generation rather than type analysis (because then you could modify the scope after type analysis to move around the variables like you want to). This will clean up a lot of things IMO. I looked at it and it didn't look too difficult in principle to do that refactoring, perhaps a day or two. I suggest this as a dependency for closures (I think it was possible without but more of a mess code-wise that way...). -- Dag Sverre From parlar at gmail.com Thu Jul 10 14:49:15 2008 From: parlar at gmail.com (Jay Parlar) Date: Thu, 10 Jul 2008 08:49:15 -0400 Subject: [Cython] cimport across packages In-Reply-To: <4875A077.1030404@behnel.de> References: <4875A077.1030404@behnel.de> Message-ID: On 7/10/08, Stefan Behnel wrote: > Hi, > > I assume you are using Cython 0.9.8? Yes. I wasn't (I'd forgotten to upgrade this machine), but I'm getting the same behaviour with 0.9.8 > > > Jay Parlar wrote: > > I'm having trouble with .pxd files across packages. > > > > My main program directory has a few packages in it, one called 'sm' > > and one called 'filters'. > > > Are these really packages? I.e. do they have an __init__.py file? Yep. They've been performing fine as packages before, but now I'm trying to add the .pxd files so I can use cdef functions between modules, and thus the beginning of my pain. Jay P. From jim-crow at rambler.ru Thu Jul 10 16:40:47 2008 From: jim-crow at rambler.ru (Anatoly A. Kazantsev) Date: Thu, 10 Jul 2008 21:40:47 +0700 Subject: [Cython] cimport across packages In-Reply-To: References: <20080710113943.2abd5584.jim-crow@rambler.ru> Message-ID: <20080710214047.21eaf098.jim-crow@rambler.ru> On Thu, 10 Jul 2008 01:02:48 -0400 "Jay Parlar" wrote: > > Rename 'dc.pxd' to 'filters.dc.pxd'. > > That's works for me. > > > > > > That doesn't seem to make a difference for me... I tried renaming it > to filters.dc.pxd, and putting it in both the main directory, and the > filters directory. Oh, I don't split sources to different directories. -- Anatoly A. Kazantsev Protect your digital freedom and privacy, eliminate DRM, learn more at http://www.defectivebydesign.org/what_is_drm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080710/df4672c6/attachment.pgp From parlar at gmail.com Thu Jul 10 16:49:53 2008 From: parlar at gmail.com (Jay Parlar) Date: Thu, 10 Jul 2008 10:49:53 -0400 Subject: [Cython] cimport across packages In-Reply-To: <20080710214047.21eaf098.jim-crow@rambler.ru> References: <20080710113943.2abd5584.jim-crow@rambler.ru> <20080710214047.21eaf098.jim-crow@rambler.ru> Message-ID: On 7/10/08, Anatoly A. Kazantsev wrote: > Oh, I don't split sources to different directories. Ok, so I *kind* of have it working, in that it works, but I'm not happy about it. It seems that if I put 'filters.dc.pxd' in *both* the 'filters' directory AND the 'sm' directory, then it happily compiles. But there must be a better way than copy-pasting. Jay P. From jim-crow at rambler.ru Thu Jul 10 17:24:15 2008 From: jim-crow at rambler.ru (Anatoly A. Kazantsev) Date: Thu, 10 Jul 2008 22:24:15 +0700 Subject: [Cython] cimport across packages In-Reply-To: References: <20080710113943.2abd5584.jim-crow@rambler.ru> <20080710214047.21eaf098.jim-crow@rambler.ru> Message-ID: <20080710222415.e38e9c49.jim-crow@rambler.ru> On Thu, 10 Jul 2008 10:49:53 -0400 "Jay Parlar" wrote: > On 7/10/08, Anatoly A. Kazantsev wrote: > > Oh, I don't split sources to different directories. > > Ok, so I *kind* of have it working, in that it works, but I'm not > happy about it. It seems that if I put 'filters.dc.pxd' in *both* the > 'filters' directory AND the 'sm' directory, then it happily compiles. > But there must be a better way than copy-pasting. Why do you want to split 2 files to different directories? Just put them in one root directory of your project sources. -- Anatoly A. Kazantsev Protect your digital freedom and privacy, eliminate DRM, learn more at http://www.defectivebydesign.org/what_is_drm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080710/5c492e36/attachment.pgp From parlar at gmail.com Thu Jul 10 17:50:33 2008 From: parlar at gmail.com (Jay Parlar) Date: Thu, 10 Jul 2008 11:50:33 -0400 Subject: [Cython] cimport across packages In-Reply-To: <20080710222415.e38e9c49.jim-crow@rambler.ru> References: <20080710113943.2abd5584.jim-crow@rambler.ru> <20080710214047.21eaf098.jim-crow@rambler.ru> <20080710222415.e38e9c49.jim-crow@rambler.ru> Message-ID: On 7/10/08, Anatoly A. Kazantsev wrote: > Why do you want to split 2 files to different directories? > Just put them in one root directory of your project sources. Why do people ever separate their files into different directories? It's a better, more logical layout, given our project. There are a lot more files involved than the ones I'm talking about here, these just happen to be the Cython files. Jay P. From languitar at semipol.de Thu Jul 10 18:17:51 2008 From: languitar at semipol.de (Johannes Wienke) Date: Thu, 10 Jul 2008 18:17:51 +0200 Subject: [Cython] Function signature does not match previous declaration In-Reply-To: <4873C2D4.3060706@semipol.de> References: <48739AB1.8080900@semipol.de> <4873AF79.8000804@semipol.de> <4873B0EE.7030508@student.matnat.uio.no> <4873B142.2010802@semipol.de> <4873B369.8090703@student.matnat.uio.no> <4873B5B6.1080905@semipol.de> <4873C2D4.3060706@semipol.de> Message-ID: <4876362F.90802@semipol.de> Am 07/08/2008 09:41 PM schrieb Johannes Wienke: > Am 07/08/2008 09:18 PM schrieb Carl Witty: >> On Tue, Jul 8, 2008 at 11:45 AM, Johannes Wienke wrote: >>> Ok, nevertheless I use the approach with the pxi files for all other >>> modules and definitions in my program and only Gimage.pxi generates >>> these warnings. Furthermore only some of the functions have those >>> warnings, not all in the file. >> I suspect that with multiple inclusions, either iwType or iwImage (or >> both) defines a "new" type each time it's loaded. So the first >> declaration doesn't match the second declaration, because it's got >> "different" argument types. >> >> If you look at which functions have warnings: is it, perhaps, only the >> functions that take an argument of type iwType? If not, is there some >> other pattern of that sort in which functions have warnings? > > It seems that all functions, that use mapped enums are the problem. No one got an idea why enums are the problem? :( - Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080710/216e1395/attachment.pgp From robertwb at math.washington.edu Thu Jul 10 20:30:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 10 Jul 2008 11:30:31 -0700 (PDT) Subject: [Cython] cimport across packages In-Reply-To: References: <20080710113943.2abd5584.jim-crow@rambler.ru> <20080710214047.21eaf098.jim-crow@rambler.ru> Message-ID: For the Sage project we have code split accross dozens of directories, with relative and local imports throughtout as if they were Python fils, so this does work. Perhaps you could post your code (or at least a non-working example) somewhere, and I could take a look later to see if you're configuring somethign wrong (or it's a bug in Cython). - Robert On Thu, 10 Jul 2008, Jay Parlar wrote: > On 7/10/08, Anatoly A. Kazantsev wrote: >> Oh, I don't split sources to different directories. > > Ok, so I *kind* of have it working, in that it works, but I'm not > happy about it. It seems that if I put 'filters.dc.pxd' in *both* the > 'filters' directory AND the 'sm' directory, then it happily compiles. > But there must be a better way than copy-pasting. > > Jay P. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From parlar at gmail.com Fri Jul 11 00:12:36 2008 From: parlar at gmail.com (Jay Parlar) Date: Thu, 10 Jul 2008 18:12:36 -0400 Subject: [Cython] cimport across packages In-Reply-To: References: <20080710113943.2abd5584.jim-crow@rambler.ru> <20080710214047.21eaf098.jim-crow@rambler.ru> Message-ID: I've put up a minimal example at http://django.jayparlar.com/example.tar Unless I have filters.dc.pxd in both the 'filters' and 'sm' directories, it won't compile. If I place it in both directories, it compiles and runs fine, but I'd rather not copy-paste. Thanks, Jay P. On 7/10/08, Robert Bradshaw wrote: > For the Sage project we have code split accross dozens of directories, > with relative and local imports throughtout as if they were Python fils, > so this does work. > > Perhaps you could post your code (or at least a non-working example) > somewhere, and I could take a look later to see if you're configuring > somethign wrong (or it's a bug in Cython). > > > - Robert > > > On Thu, 10 Jul 2008, Jay Parlar wrote: > > > On 7/10/08, Anatoly A. Kazantsev wrote: > >> Oh, I don't split sources to different directories. > > > > Ok, so I *kind* of have it working, in that it works, but I'm not > > happy about it. It seems that if I put 'filters.dc.pxd' in *both* the > > 'filters' directory AND the 'sm' directory, then it happily compiles. > > But there must be a better way than copy-pasting. > > > > Jay P. > > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Fri Jul 11 09:53:37 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Jul 2008 09:53:37 +0200 Subject: [Cython] Running the CPython regression tests Message-ID: <48771181.90609@behnel.de> Hi, I extended the test runner script to let it run the CPython regression test suite. All you have to do is take a source distribution of CPython and copy the "Lib/test" directory over to Cython's "tests/" directory, naming it "pyregr". The test runner will compile each module and run all unit tests it finds. Currently, most of them fail, but I have about 160 out of 205 compiling from the CPython 2.5.1 test suite. Almost of all the failures are due to three things: - inner classes and functions - lambda - print >> I don't mind the latter since it's gone in Py3, and I'm happy to know that the rest is underways. There are also generator expressions, but they should be simple to implement once the function-to-class transformations work. Stefan From stefan_ml at behnel.de Fri Jul 11 17:04:16 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Jul 2008 17:04:16 +0200 Subject: [Cython] generators In-Reply-To: References: <4875AEC8.9070006@behnel.de> Message-ID: <48777670.7030907@behnel.de> Hi, Robert Bradshaw wrote: > This is in fact very much in line with how closures would be > implemented--there would be an extension class that would hold the state > (and garbage collection would happen when this object went out of scope, > so no issues for generators or closures). Just coming up with a couple of corner cases to make sure they will work in this solution. Could you comment on how they are handled? 1) def f(): x = 1 def a(): print x def b(): print x return (a,b) 2) for i in range(10): def a(): print i a() (outputs 0-9 in Python) 3) for i in range(10): if i == 0: def a(): print i a() (outputs 0-9 in Python) Stefan From dalcinl at gmail.com Fri Jul 11 17:55:01 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 11 Jul 2008 12:55:01 -0300 Subject: [Cython] problems with current cython-devel repo Message-ID: Hi all, after some time of inactivity, I'm back. I've just pulled cython-devel repo and I'm having some trouble. 1) It seems that the transforms code does not play nicely with the compiled Scanners.so. After installing Cython, I have to manually remove Scanners.so to get Cython running. 2) There is a typo, trivial to fix (just change a 'p' to a 's'), the diff pasted below: diff -r ebe5f9fae217 Cython/Compiler/Parsing.py --- a/Cython/Compiler/Parsing.py Fri Jul 11 16:52:31 2008 +0200 +++ b/Cython/Compiler/Parsing.py Fri Jul 11 12:17:34 2008 -0300 @@ -1615,7 +1615,7 @@ def p_c_simple_base_type(s, self_flag, n # Treat trailing [] on type as buffer access if s.sy == '[': if is_basic: - p.error("Basic C types do not support buffer access") + s.error("Basic C types do not support buffer access") return p_buffer_access(s, type_node) else: return type_node 3) But now, the code above seems to do not plat nice with the following: Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef extern from *: int foo(int x[]) int bla(int[]) ^ ------------------------------------------------------------ /tmp/qq.pyx:3:15: Basic C types do not support buffer access -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Jul 11 18:23:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Jul 2008 18:23:27 +0200 Subject: [Cython] problems with current cython-devel repo In-Reply-To: References: Message-ID: <487788FF.6040707@behnel.de> Hi, Lisandro Dalcin wrote: > Hi all, after some time of inactivity, I'm back. I've just pulled > cython-devel repo and I'm having some trouble. Yes, so do I. :-/ > 1) It seems that the transforms code does not play nicely with the > compiled Scanners.so. After installing Cython, I have to manually > remove Scanners.so to get Cython running. Not sure what the problem is, I'll have to dig into that. > 2) There is a typo, trivial to fix (just change a 'p' to a 's'), the Thanks. > 3) But now, the code above seems to do not plat nice with the following: > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef extern from *: > int foo(int x[]) > int bla(int[]) > ^ > ------------------------------------------------------------ > /tmp/qq.pyx:3:15: Basic C types do not support buffer access Hmm, ok, I commented that code out for now and added a test case. Should be back up working (at least for this part). Stefan From dalcinl at gmail.com Fri Jul 11 18:55:58 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 11 Jul 2008 13:55:58 -0300 Subject: [Cython] problems with current cython-devel repo In-Reply-To: <487788FF.6040707@behnel.de> References: <487788FF.6040707@behnel.de> Message-ID: Now I've got another failure with my code. The following works (): def winsize(size): try: w, h = size except TypeError: w = size h = size return w, h But the following not: def winsize(size): try: w, h = size except TypeError: w = h = size return w, h So it seems there is a problem with a parallel assignement appearing in the body of an 'except' clause. On 7/11/08, Stefan Behnel wrote: > Hi, > > > Lisandro Dalcin wrote: > > Hi all, after some time of inactivity, I'm back. I've just pulled > > cython-devel repo and I'm having some trouble. > > > Yes, so do I. :-/ > > > > > 1) It seems that the transforms code does not play nicely with the > > compiled Scanners.so. After installing Cython, I have to manually > > remove Scanners.so to get Cython running. > > > Not sure what the problem is, I'll have to dig into that. > > > > > 2) There is a typo, trivial to fix (just change a 'p' to a 's'), the > > > Thanks. > > > > > 3) But now, the code above seems to do not plat nice with the following: > > > > Error converting Pyrex file to C: > > ------------------------------------------------------------ > > ... > > cdef extern from *: > > int foo(int x[]) > > int bla(int[]) > > ^ > > ------------------------------------------------------------ > > /tmp/qq.pyx:3:15: Basic C types do not support buffer access > > > Hmm, ok, I commented that code out for now and added a test case. > > Should be back up working (at least for this part). > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Jul 11 19:06:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Jul 2008 19:06:07 +0200 Subject: [Cython] problems with current cython-devel repo In-Reply-To: References: <487788FF.6040707@behnel.de> Message-ID: <487792FF.5020407@behnel.de> Lisandro Dalcin wrote: > Now I've got another failure with my code. > > The following works (): > > def winsize(size): > try: > w, h = size > except TypeError: > w = size > h = size > return w, h > > But the following not: > > def winsize(size): > try: > w, h = size > except TypeError: > w = h = size > return w, h > > > So it seems there is a problem with a parallel assignement appearing > in the body of an 'except' clause. Not only there. Parallel assignments seem to be pretty much broken currently. I'm working on it. Try revision 771 if you need something working now. Stefan From stefan_ml at behnel.de Sat Jul 12 06:16:53 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 12 Jul 2008 06:16:53 +0200 Subject: [Cython] Interesting Py3k thread on generator expressions Message-ID: <48783035.5070802@behnel.de> Hi, there's an interesting thread on genexps on the Py3k list, in case someone is interested in implementing them. :) http://mail.python.org/pipermail/python-3000/2008-July/014311.html Stefan From robertwb at math.washington.edu Sat Jul 12 07:47:46 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 11 Jul 2008 22:47:46 -0700 Subject: [Cython] generators In-Reply-To: <48777670.7030907@behnel.de> References: <4875AEC8.9070006@behnel.de> <48777670.7030907@behnel.de> Message-ID: <9328F0F5-9347-4100-ABCC-95DA9F49B966@math.washington.edu> On Jul 11, 2008, at 8:04 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> This is in fact very much in line with how closures would be >> implemented--there would be an extension class that would hold the >> state >> (and garbage collection would happen when this object went out of >> scope, >> so no issues for generators or closures). > > Just coming up with a couple of corner cases to make sure they will > work in > this solution. Could you comment on how they are handled? > > 1) > > def f(): > x = 1 > def a(): print x > def b(): print x > return (a,b) There will be a cdef class created for a and b, as well as one representing the scope of f, with a __call__ method containing the body of the function. The def a() and def b() lines would instantiate this class and assign them to local variable a and b (which have a reference to the containing scope). When both of these are collected, the scope will be as well. > 2) > > for i in range(10): > def a(): print i > a() > > (outputs 0-9 in Python) This isn't an inner function at all. To fix the scoping rules, "def" would create a function at the top level, and then do an assignment to the variable "a." > 3) > > for i in range(10): > if i == 0: > def a(): print i > a() > > (outputs 0-9 in Python) Exactly the same as (2). There is nothing special about the scoping here, it's "outer" scope is the module scope, just as in the standard def function. The reason Cython has this restriction is that functions can only be created once. - Robert From stefan_ml at behnel.de Sat Jul 12 08:06:12 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 12 Jul 2008 08:06:12 +0200 Subject: [Cython] generators In-Reply-To: <9328F0F5-9347-4100-ABCC-95DA9F49B966@math.washington.edu> References: <4875AEC8.9070006@behnel.de> <48777670.7030907@behnel.de> <9328F0F5-9347-4100-ABCC-95DA9F49B966@math.washington.edu> Message-ID: <487849D4.5080006@behnel.de> Hi, Robert Bradshaw wrote: > On Jul 11, 2008, at 8:04 AM, Stefan Behnel wrote: >> 1) >> >> def f(): >> x = 1 >> def a(): print x >> def b(): print x >> return (a,b) > > There will be a cdef class created for a and b, as well as one > representing the scope of f, with a __call__ method containing the > body of the function. The def a() and def b() lines would instantiate > this class and assign them to local variable a and b (which have a > reference to the containing scope). When both of these are collected, > the scope will be as well. Yes, that sounds correct to me. >> 2) >> >> for i in range(10): >> def a(): print i >> a() >> >> (outputs 0-9 in Python) > > This isn't an inner function at all. To fix the scoping rules, "def" > would create a function at the top level, and then do an assignment > to the variable "a." I actually meant it to be an inner function of a function, so this wasn't a clear example. But I can see that this would be handled the same way as above. So, yes, I'm happy with that implementation. I just came up with these examples when I was discussing the topic with some people at EuroPython, and I couldn't really tell them how we would handle that. Stefan From languitar at semipol.de Mon Jul 14 13:25:04 2008 From: languitar at semipol.de (Johannes Wienke) Date: Mon, 14 Jul 2008 13:25:04 +0200 Subject: [Cython] Raising exceptions under python 2.5 Message-ID: <487B3790.2090801@semipol.de> Hi again, I've developed my project under python 2.4 and now I got to get it running under 2.5. The main problem I notice at the moment with the cython party is raising exceptions. Running the project I get constant errors of the form: Traceback (most recent call last): File "/tmp/shIP/test/shiptests/datatest.py", line 49, in testStartData self.__dataStore.addDataObserver(DataStore.START_IDENT, plugin1) File "data.pyx", line 535, in ship.data.DataStore.addDataObserver (build/temp.linux-i686-2.5/pyrex/data.c:4314) File "data.pyx", line 472, in ship.data.DataStore.__addToObserverDict (build/temp.linux-i686-2.5/pyrex/data.c:3908) File "data.pyx", line 558, in ship.data.DataStore.__findObserverItem (build/temp.linux-i686-2.5/pyrex/data.c:4478) TypeError: raise: exception must be an old-style class or instance The line contains: raise KeyError("No _ObserverItem found belonging to '%s'." % observer) I thought this is the preferred way of raising exceptions. Why is cython / python complaining about this? Thanks for the help Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080714/1e042005/attachment.pgp From jwienke at techfak.uni-bielefeld.de Mon Jul 14 15:27:06 2008 From: jwienke at techfak.uni-bielefeld.de (Johannes Wienke) Date: Mon, 14 Jul 2008 15:27:06 +0200 Subject: [Cython] Raising exceptions under python 2.5 In-Reply-To: <487B3790.2090801@semipol.de> References: <487B3790.2090801@semipol.de> Message-ID: <487B542A.8040407@techfak.uni-bielefeld.de> Am 07/14/2008 01:25 PM schrieb Johannes Wienke: > Hi again, > > I've developed my project under python 2.4 and now I got to get it > running under 2.5. The main problem I notice at the moment with the > cython party is raising exceptions. Running the project I get constant > errors of the form: > > Traceback (most recent call last): > File "/tmp/shIP/test/shiptests/datatest.py", line 49, in testStartData > self.__dataStore.addDataObserver(DataStore.START_IDENT, plugin1) > File "data.pyx", line 535, in ship.data.DataStore.addDataObserver > (build/temp.linux-i686-2.5/pyrex/data.c:4314) > File "data.pyx", line 472, in ship.data.DataStore.__addToObserverDict > (build/temp.linux-i686-2.5/pyrex/data.c:3908) > File "data.pyx", line 558, in ship.data.DataStore.__findObserverItem > (build/temp.linux-i686-2.5/pyrex/data.c:4478) > TypeError: raise: exception must be an old-style class or instance > > The line contains: > raise KeyError("No _ObserverItem found belonging to '%s'." % observer) > > I thought this is the preferred way of raising exceptions. Why is cython > / python complaining about this? I'm sorry, my fault, compiled against python 2.4. - Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080714/49b10a1e/attachment.pgp From stefan_ml at behnel.de Mon Jul 14 21:20:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 14 Jul 2008 21:20:02 +0200 Subject: [Cython] Raising exceptions under python 2.5 In-Reply-To: <487B3790.2090801@semipol.de> References: <487B3790.2090801@semipol.de> Message-ID: <487BA6E2.90305@behnel.de> Hi, just a quick comment. Johannes Wienke wrote: > raise KeyError("No _ObserverItem found belonging to '%s'." % observer) it's actually faster to write raise KeyError, "No _ObserverItem found belonging to '%s'." % observer even if that's not the preferred way of doing it in Python. Stefan From greg.ewing at canterbury.ac.nz Tue Jul 15 01:52:59 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 Jul 2008 11:52:59 +1200 Subject: [Cython] generators In-Reply-To: <4875C415.6010706@student.matnat.uio.no> References: <4875AEC8.9070006@behnel.de> <4875C415.6010706@student.matnat.uio.no> Message-ID: <487BE6DB.6000501@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > To be even more specific, it would help if self.result_code everywhere > was calculated during code generation rather than type analysis You might want to take a look at the latest Pyrex, as I've more or less done this already. The code generation phase now calls the result() method of a node to get its result code instead of accessing the result_code attribute directly (and not all nodes even have a result_code any more). -- Greg From greg.ewing at canterbury.ac.nz Tue Jul 15 02:26:40 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 Jul 2008 12:26:40 +1200 Subject: [Cython] Raising exceptions under python 2.5 In-Reply-To: <487BA6E2.90305@behnel.de> References: <487B3790.2090801@semipol.de> <487BA6E2.90305@behnel.de> Message-ID: <487BEEC0.4000802@canterbury.ac.nz> Stefan Behnel wrote: > it's actually faster to write > > raise KeyError, "No _ObserverItem found belonging to '%s'." % observer But only if the exception is both raised and caught in Pyrex or Cython code (because no exception instance is created). Otherwise it makes little difference. -- Greg From cswiercz at gmail.com Wed Jul 16 00:53:47 2008 From: cswiercz at gmail.com (Chris Swierczewski) Date: Tue, 15 Jul 2008 15:53:47 -0700 Subject: [Cython] Wrapping C++ Classes Message-ID: Hello, I'm having a compile issue with wrapping a certain C++ class. Allow me to describe the situation first before stating the issue. The Situation: the opentick API is a collection of cross-platform, cross-language libraries (and headers in the C++ flavor) for receiving live streams of financial data. (See www.opentick.com) Among the API/source languages available, there is a collection of cross-platform C++ headers along with a static library. What is meant by cross-platform? Well, depending on which platform you're using, different parts of the code are called and different macro definitions are made. For example, when compiling on a Linux system one must write #define __LINUX at the top of their code before calling the appropriate headers. Now, the main class is defined in the following way: class DLL_EXP OTClient: { public: ... } When running in the windows environment (that is, if you #define __WIN), DLL_EXP is set to "__declspec(dllexport)" which, to my understanding, has to do with using .dll files for callbacks akin to how twisted works. However, when compiling in the Linux environment, (#define __LINUX) the definition of the macro is "empty". That is, at some point before the definition of the class, we simply have #define DLL_EXP implying that no .dll calls need to be made. My Question (Finally): How do I go about wrapping this C++ class with this strange little define hanging around the class declaration? Methinks I'm having compile issues precisely because of this addition. I checked the Cython wiki and the Sage Wiki on wrapping C++ classes and couldn't find anything that looked like this. Thank you very much in advance for any help! (And thank you for such an awesome technology!) -- Chris Swierczewski cswiercz at gmail.com mobile: 253 2233721 From michael.abshoff at googlemail.com Wed Jul 16 00:59:25 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Tue, 15 Jul 2008 15:59:25 -0700 Subject: [Cython] Wrapping C++ Classes In-Reply-To: References: Message-ID: <487D2BCD.5060103@gmail.com> Chris Swierczewski wrote: > Hello, > > I'm having a compile issue with wrapping a certain C++ class. Allow me > to describe the situation first before stating the issue. > > The Situation: the opentick API is a collection of cross-platform, > cross-language libraries (and headers in the C++ flavor) for receiving > live streams of financial data. (See www.opentick.com) Among the > API/source languages available, there is a collection of > cross-platform C++ headers along with a static library. What is meant > by cross-platform? Well, depending on which platform you're using, > different parts of the code are called and different macro definitions > are made. For example, when compiling on a Linux system one must write > > #define __LINUX > > at the top of their code before calling the appropriate headers. Now, > the main class is defined in the following way: > > class DLL_EXP OTClient: > { > public: > ... > } > > When running in the windows environment (that is, if you #define > __WIN), DLL_EXP is set to "__declspec(dllexport)" which, to my > understanding, has to do with using .dll files for callbacks akin to > how twisted works. However, when compiling in the Linux environment, > (#define __LINUX) the definition of the macro is "empty". That is, at > some point before the definition of the class, we simply have > > #define DLL_EXP > > implying that no .dll calls need to be made. > > My Question (Finally): How do I go about wrapping this C++ class with > this strange little define hanging around the class declaration? > Methinks I'm having compile issues precisely because of this addition. > I checked the Cython wiki and the Sage Wiki on wrapping C++ classes > and couldn't find anything that looked like this. Thank you very much > in advance for any help! (And thank you for such an awesome > technology!) > Hi, Have you tried passing -D__Linux at the build time of the extension? I saw your discussion with Glenn and the removal of the __declspec(dllexport) constructs as suggested by him will bite us in the ass down the road since we intent to support MSVC for which the above constructs are needed. Cheers, Michael From cswiercz at gmail.com Wed Jul 16 01:05:06 2008 From: cswiercz at gmail.com (Chris Swierczewski) Date: Tue, 15 Jul 2008 16:05:06 -0700 Subject: [Cython] Wrapping C++ Classes In-Reply-To: <487D2BCD.5060103@gmail.com> References: <487D2BCD.5060103@gmail.com> Message-ID: Michael, > Have you tried passing -D__Linux at the build time of the extension? I didn't know that you could pass those kinds of things at the Sage Extension declaration. What I did was make a copy of the header and hard-coded the definition. (OTClient.h ==> OTClient__LINUX.h containing #define __LINUX) I'll have to try that right now, though I'm not sure if it will make a difference. > I saw your discussion with Glenn and the removal of the > __declspec(dllexport) constructs as suggested by him will bite us in the > ass down the road since we intent to support MSVC for which the above > constructs are needed. Yeah, I was a bit weary of taking that route. The opentick libraries look way too intricate to begin mucking around with that. -- Chris From cswiercz at gmail.com Wed Jul 16 01:18:58 2008 From: cswiercz at gmail.com (Chris Swierczewski) Date: Tue, 15 Jul 2008 16:18:58 -0700 Subject: [Cython] Wrapping C++ Classes In-Reply-To: <487D2BCD.5060103@gmail.com> References: <487D2BCD.5060103@gmail.com> Message-ID: Michael, > Have you tried passing -D__Linux at the build time of the extension? Just to be sure, does that mean adding opentick = Extension('sage.finance.opentick', ... extra_compile_args = ["-D __Linux"] ) If so, then I still get the following error: Updating Cython code.... sage/finance/opentick.pyx --> /Users/cswiercz/sage/local//lib/python/site-packages//sage/finance/opentick.pyx Building sage/finance/opentick.cpp because it depends on sage/finance/opentick.pyx. python2.5 `which cython` --embed-positions --incref-local-binop -I/Users/cswiercz/sage/devel/sage-opentick -o sage/finance/opentick.cpp sage/finance/opentick.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef class Opentick: cdef c_OTClient *otclient ^ ------------------------------------------------------------ /Users/cswiercz/sage/devel/sage-opentick/sage/finance/opentick.pxd:2:20: Syntax error in C variable declaration sage: Error running cython. sage: There was an error installing modified sage library code. when I have the following in opentick.pxd cdef extern from "OTClient.h": ctypedef struct c_OTClient "OTClient": pass c_OTClient *new_OTClient "new OTClient" () void del_OTClient "delete" (c_OTClient *otclient) -- Chris From greg.ewing at canterbury.ac.nz Wed Jul 16 02:57:59 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Jul 2008 12:57:59 +1200 Subject: [Cython] Wrapping C++ Classes In-Reply-To: References: Message-ID: <487D4797.5050708@canterbury.ac.nz> Chris Swierczewski wrote: > For example, when compiling on a Linux system one must write > > #define __LINUX I would write a small .h file containing the appropriate #define and use a 'cdef extern from' declaration to include it. -- Greg From robertwb at math.washington.edu Wed Jul 16 08:56:58 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 15 Jul 2008 23:56:58 -0700 Subject: [Cython] Wrapping C++ Classes In-Reply-To: References: <487D2BCD.5060103@gmail.com> Message-ID: <5F6C6AF3-C60C-4C20-BDEB-6514CDF84E84@math.washington.edu> On Jul 15, 2008, at 4:18 PM, Chris Swierczewski wrote: > Michael, > >> Have you tried passing -D__Linux at the build time of the extension? > > Just to be sure, does that mean adding > > opentick = Extension('sage.finance.opentick', > ... > extra_compile_args = ["-D > __Linux"] ) > > If so, then I still get the following error: > > Updating Cython code.... > sage/finance/opentick.pyx --> > /Users/cswiercz/sage/local//lib/python/site-packages//sage/finance/ > opentick.pyx > > Building sage/finance/opentick.cpp because it depends on > sage/finance/opentick.pyx. > python2.5 `which cython` --embed-positions --incref-local-binop > -I/Users/cswiercz/sage/devel/sage-opentick -o > sage/finance/opentick.cpp sage/finance/opentick.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef class Opentick: > cdef c_OTClient *otclient > ^ > ------------------------------------------------------------ > > /Users/cswiercz/sage/devel/sage-opentick/sage/finance/opentick.pxd: > 2:20: > Syntax error in C variable declaration > sage: Error running cython. > sage: There was an error installing modified sage library code. > > when I have the following in opentick.pxd > > cdef extern from "OTClient.h": > ctypedef struct c_OTClient "OTClient": > pass > c_OTClient *new_OTClient "new OTClient" () > void del_OTClient "delete" (c_OTClient *otclient) I am unable to reproduce this. What I have is: ----------------- opentick.pxd --------------- cdef extern from "OTClient.h": ctypedef struct c_OTClient "OTClient": pass c_OTClient *new_OTClient "new OTClient" () void del_OTClient "delete" (c_OTClient *otclient) cdef class Opentick: pass ----------------- opentick.pyx --------------- cdef class Opentick: pass --------------------------------------------------- which compiles fine to c. Perhaps you're redefining something somewhere? - Robert From robertwb at math.washington.edu Wed Jul 16 09:15:55 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 16 Jul 2008 00:15:55 -0700 Subject: [Cython] Mapping a really strange typedef in Cython In-Reply-To: <48676F15.1090607@semipol.de> References: <48676F15.1090607@semipol.de> Message-ID: <22E10F70-3084-4B0C-B0EE-7B38B2BDE23B@math.washington.edu> On Jun 29, 2008, at 4:16 AM, Johannes Wienke wrote: > Hi again, > > I'm struggling to get a really strange typedef working in Cython. The > original C (preprocessor) code looks like this: > > #define IW_CTAB_SIZE 256 > typedef guchar (*iwColtab)[3]; > > extern iwColtab iw_def_col_tab; > > /* Color formats of the data for prev_render...() */ > #define IW_RGB ((iwColtab)0) > > // some more formats > > #define IW_BW ((iwColtab)7) > #define IW_GRAY IW_BW > #define IW_INDEX (iw_def_col_tab) > > #define IW_COLFORMAT_MAX IW_BW /* last 'special' definition */ > > The purpose of this strange construct was that one can use normal > color > spaces without struggling, but one is also able to use a custom > color table. > > This definition was originally used like this: > > void prev_colConvert (iwColtab ctab, gint r, gint g, gint b, > gint *rd, gint *gd, gint *bd) > { > if (ctab == IW_RGB) { > *rd = r; > *gd = g; > *bd = b; > } else if (ctab == IW_BGR) { > *rd = b; > *gd = g; > *bd = r; > } else if (ctab > IW_COLFORMAT_MAX && r>=0 && r < IW_CTAB_SIZE){ > *rd = ctab[r][0]; > *gd = ctab[r][1]; > *bd = ctab[r][2]; > } > > } > > I need to be able to do the same thing in cython with this definition. > My current approach was simply using a enum to emulate it: > > cdef extern from "gui/Gimage.h": > > ctypedef enum iwColtab: > IW_RGB > # ... > IW_GRAY > IW_CTAB_SIZE > IW_COLFORMAT_MAX > > The normal color modes work like this, but using the color table > doesn't > work: > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > elif ctab == IW_YUV: > return Color(1.164 * (r - 16) + 1.596 * (b - 128), > 1.164 * (r - 16) - 0.813 * (b - 128) - 0.391 * > (g - > 128), > 1.164 * (r - 16) + 2.018 * (g - 128)) > elif (ctab > IW_COLFORMAT_MAX and r >= 0 and r < IW_CTAB_SIZE): > return Color(ctab[r][0], ctab[r][1], ctab[r][2]) > ^ > ------------------------------------------------------------ > > image_utils.pyx:144:25: Attempting to index non-array type 'iwColtab' > > I really have no idea how to solve this. Has anyone got a solution > for this? > > Thanks for the help > Johannes I'm not sure--what is the definition of iwColtab? An enum won't work here because one can't index into it (in Cython or in C) so I think the only thing to do here is to have two separate types (the enum, and iwColtab) and do some casting. - Robert -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080716/faba004e/attachment.pgp From robertwb at math.washington.edu Wed Jul 16 09:21:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 16 Jul 2008 00:21:20 -0700 Subject: [Cython] Make the Cython workflow easier In-Reply-To: References: <20080619144328.GC32587@phare.normalesup.org> <485A92F9.8040203@behnel.de> <20080619183809.GB10875@phare.normalesup.org> <96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu> <20080619184426.GC10875@phare.normalesup.org> <485B3E5A.8030605@behnel.de> <20080620122525.GC3651@phare.normalesup.org> <485BF242.80408@student.matnat.uio.no> Message-ID: <8018EF36-0E09-4E0B-A7EF-1BE24511AFB0@math.washington.edu> On Jun 20, 2008, at 11:13 AM, Fernando Perez wrote: > On Fri, Jun 20, 2008 at 11:09 AM, Dag Sverre Seljebotn > wrote: >> Lisandro Dalcin wrote: >>> Gael, not sure if this will help you, but I wrote this script >>> based in >>> distutils for building *.pyx files on the fly (perhaps with others >>> companion C/C++ sources). You basically use it as this: >>> >> Also remember that Pex already has this feature (as a wrapper/build >> system for Cython): >> >> pexlang.sourceforge.net > > This is just a generic request: when the dust settles a bit and you > guys have a clearer picture of where the whole pex/cython integration > regarding numpy is going to go, could you provide a little summary? > I'm super interested in this (and I can guarantee I'm not the only > one) but right now things seem to be in a strong state of flux, so I'm > OK waiting. The big part is Dag's GSoC project on buffer support. There are a lot of other small improvements which we would like to incorporate which can be found at http://wiki.cython.org/enhancements/pex . > Furthermore, it would be great if you think some of this will be in > reasonable state (even alpha, but solid enough for knowing where it > will go) by August, to present it at scipy 2008: That is certainly the goal. > > http://conference.scipy.org/ > > Robert B. is already coming to present a cython tutorial, so perhaps a > short talk on this would be OK for him to give. It would be very well > received. > > Cheers, > > f > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Fri Jul 18 09:28:28 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 18 Jul 2008 00:28:28 -0700 Subject: [Cython] Object not released when calling cpdef method from inherited class ? In-Reply-To: <1215488230.20373.14.camel@nikopol> References: <1215488230.20373.14.camel@nikopol> Message-ID: <62C56E05-9EC6-4095-AFCD-7DA5D5E7DB72@math.washington.edu> Fixed, see http://trac.cython.org/cython_trac/ticket/24 On Jul 7, 2008, at 8:37 PM, Guillaume Chereau wrote: > Hello all, > > I am currently using cython for a project related to openmoko cell > phone : http://charlie137-2.blogspot.com/2008/07/introducing- > tichy.html > > I noticed that the memory is constantly increasing when I use > cython. I > tracked down the problem to the case where I create a subclass of a > cython class and then redefine a cpdef method, asking it to call the > parent method, but ONLY if the cpdef method then call an other cpdef > method ! > > Here is the smallest example I could come with that fails : > > > == test.pyx == > > cdef class A: > cpdef func(self): > return > > cpdef test(self): > self.func() > > > == main.py == > > import test > import gc > > class B(test.A): > def test(self): > test.A.test(self) > > b = B() > > for i in range(10): > b.test() > gc.collect() > print len(gc.get_objects()) > > > The output will show that some objects are not released. > > Is it a known bug ? Is there a way to avoid it ? > > I tried with both Cython 0.9.6 and Cython 0.9.8 > > cheers, > -Charlie > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080718/619d97af/attachment.pgp From charlie at openmoko.org Fri Jul 18 10:27:41 2008 From: charlie at openmoko.org (Guillaume Chereau) Date: Fri, 18 Jul 2008 16:27:41 +0800 Subject: [Cython] Object not released when calling cpdef method from inherited class ? In-Reply-To: <62C56E05-9EC6-4095-AFCD-7DA5D5E7DB72@math.washington.edu> References: <1215488230.20373.14.camel@nikopol> <62C56E05-9EC6-4095-AFCD-7DA5D5E7DB72@math.washington.edu> Message-ID: <1216369661.7529.48.camel@nikopol> Ah excellent ! Thanks a lot ! -Charlie On Fri, 2008-07-18 at 00:28 -0700, Robert Bradshaw wrote: > Fixed, see http://trac.cython.org/cython_trac/ticket/24 > > On Jul 7, 2008, at 8:37 PM, Guillaume Chereau wrote: > > > Hello all, > > > > I am currently using cython for a project related to openmoko cell > > phone : http://charlie137-2.blogspot.com/2008/07/introducing- > > tichy.html > > > > I noticed that the memory is constantly increasing when I use > > cython. I > > tracked down the problem to the case where I create a subclass of a > > cython class and then redefine a cpdef method, asking it to call the > > parent method, but ONLY if the cpdef method then call an other cpdef > > method ! > > > > Here is the smallest example I could come with that fails : > > > > > > == test.pyx == > > > > cdef class A: > > cpdef func(self): > > return > > > > cpdef test(self): > > self.func() > > > > > > == main.py == > > > > import test > > import gc > > > > class B(test.A): > > def test(self): > > test.A.test(self) > > > > b = B() > > > > for i in range(10): > > b.test() > > gc.collect() > > print len(gc.get_objects()) > > > > > > The output will show that some objects are not released. > > > > Is it a known bug ? Is there a way to avoid it ? > > > > I tried with both Cython 0.9.6 and Cython 0.9.8 > > > > cheers, > > -Charlie > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/cython-dev/attachments/20080718/2578dbdc/attachment.pgp From stefan_ml at behnel.de Sat Jul 19 10:58:35 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 19 Jul 2008 10:58:35 +0200 Subject: [Cython] problems with current cython-devel repo In-Reply-To: References: <487788FF.6040707@behnel.de> Message-ID: <4881ACBB.4030207@behnel.de> Hi, Lisandro Dalcin wrote: > Now I've got another failure with my code. > > The following works (): > > def winsize(size): > try: > w, h = size > except TypeError: > w = size > h = size > return w, h > > But the following not: > > def winsize(size): > try: > w, h = size > except TypeError: > w = h = size > return w, h This should be fixed now. Stefan From stefan_ml at behnel.de Sat Jul 19 11:37:00 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 19 Jul 2008 11:37:00 +0200 Subject: [Cython] rev 809 - cdef exception propagation Message-ID: <4881B5BC.3030402@behnel.de> Hi, revision 809 implements a nice feature which lets cdef functions propagate exceptions by default. However, the change also introduced a problem with nogil functions, when called without the GIL being held. I added a run/nogil.pyx test case for now which crashes for me. Looking through the change I get the impression that the signature modification is done too late in the processing. To me, it feels more like this should be something that should be handled at the tree level and reflected directly in the function signature. What do you think? Stefan From dagss at student.matnat.uio.no Sat Jul 19 11:53:48 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 19 Jul 2008 11:53:48 +0200 Subject: [Cython] Standard include path Message-ID: <4881B9AC.3060909@student.matnat.uio.no> Two issues that may or may not overlap: My test case has a call to malloc (tips to get around this are welcome but I'd still like to have a look at include paths). So I'm cimporting stdlib, meaning I have to export INCLUDE=Includes prior to running tests (which I'd like to avoid as apparently no other tests relies on this?) There are two ways of going about this: - Have runtests.py always include the Includes dir - Have Cython itself always include the Includes dir (i.e. always ../../Includes relative to Cython/Compiler/Main.py). I prefer the latter one; thoughts? I suppose this might also mean installing Includes "somewhere" on setup.py (or is that already done?) Also there's another import issue: For more sophisticated transforms, I've (or Robert, can't remember) come up with a __cython__ namespace which basically contains "any symbol the compiler might like to use". For instance: TreeFragment(u"__cython__.PyObject_GetBuffer(<__cython__.PyObject*>TMP, &BUFINFO, 0)") Eventually all the __Pyx-utils might be declared in this one too (though I'm certainly not going to do it for the sake of it). Then the idea is that __cython__ is *always* cimported (and available to the user as well). Any strong feelings against simply making this a __cython__.pxd in the aforementioned standard include path? Note the consequences: The compiler will be hard-wired to work with a certain pxd file. The alternative is to create a cython-scope similar to builtin-scope programatically (or stick everything that is needed in symtab.py directly). -- Dag Sverre From dagss at student.matnat.uio.no Sat Jul 19 14:14:18 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 19 Jul 2008 14:14:18 +0200 Subject: [Cython] More concerns over extern ctypedefs Message-ID: <4881DA9A.3070406@student.matnat.uio.no> Consider this code: cdef extern from "t.h": ctypedef long hullo def foo(hullo h): pass I noticed that it creates code like this: PyArg_ParseTupleAndKeywords(__pyx_args, __pyx_kwds, "l", __pyx_argnames, &__pyx_v_h)) Which could probably destroy the stack if we don't have the exact typedef, right? Which means: One should either write our own parsing code instead (generate for each parameter signature), or generate the typestring at runtime, or insert C checks for the sizeof(the typedefs). The latter will destroy the current approach for numpy.int64 and friends and remove what I consider a nice feature. Just as I took all that care to allow approximate typedefs for buffers :-) Just noting it down, I suppose for now we'll live with it. But need to be more careful about recommending approximate typedefs. -- Dag Sverre From dagss at student.matnat.uio.no Sat Jul 19 14:21:06 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 19 Jul 2008 14:21:06 +0200 Subject: [Cython] More concerns over extern ctypedefs In-Reply-To: <4881DA9A.3070406@student.matnat.uio.no> References: <4881DA9A.3070406@student.matnat.uio.no> Message-ID: <4881DC32.3030103@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Consider this code: > > cdef extern from "t.h": > ctypedef long hullo > > def foo(hullo h): pass > > I noticed that it creates code like this: > PyArg_ParseTupleAndKeywords(__pyx_args, __pyx_kwds, "l", __pyx_argnames, > &__pyx_v_h)) > > Which could probably destroy the stack if we don't have the exact > typedef, right? Which means: One should either write our own parsing > code instead (generate for each parameter signature), or generate the > typestring at runtime, or insert C checks for the sizeof(the typedefs). Just realized that a dynamic typestring could be generated at module load time, which should be perfectly unnoticeable. So this might be the way to go. Each extern ctypedef would grow a function similar to what is mentioned in the "buffer must know exact C types thread", but switching on size instead: switch (sizeof(T)) { case sizeof(int): return 'i'; ... The thing is, I could then use this for my buffer format checking as well. (Not a priority though). -- Dag Sverre From ondrej at certik.cz Sun Jul 20 13:14:49 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 20 Jul 2008 13:14:49 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <4852169A.1050706@behnel.de> References: <4852169A.1050706@behnel.de> Message-ID: <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> On Fri, Jun 13, 2008 at 8:41 AM, Stefan Behnel wrote: > Hi all, > > Cython 0.9.8 has been released. A lot of work by Robert, Greg and others > (inluding myself) went into this release to fix many bugs and add tons of > great features. > > This release features > > - everything that was done in Pyrex 0.9.8.x that wasn't in Cython already > - faster compilation > - more optimisations > - stricter warnings about potential coding problems (e.g. when using nogil) > - support for Python 2.6 > - __future__ imports for unicode literals > > Due to very recent changes in the Python code base, this release does not yet > compile code for Python 3.0beta1, but it's well prepared so that when 3.0beta1 > finally comes out, we can provide a quick update to make your code compile > with anything from Python 2.3 to 3.0. Why is cython now compiling Cython.Plex.Scanners with gcc but not installing it with setup.py install? Is this a bug with Cython or on my side? I am just using the same Debian package as I was using for all previous cython versions, but suddenly it fails to build, because it depends on python-dev. So I added python-dev to build-depends, it builds, it works, but the Cython/Plex/Scanners.so is not installed in the final package. What's wrong? Should I upload the package as it is, i.e. without the Scanners.so file? I tested it on some simple .pyx files and it seems to work. I'd like to get this resolved in about 5 hours, so that I can upload before today's dinstall run, so that Tim can build Sage in Debian with it. Thanks, Ondrej From stefan_ml at behnel.de Sun Jul 20 13:31:00 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jul 2008 13:31:00 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> Message-ID: <488321F4.4050007@behnel.de> Hi, Ondrej Certik wrote: > Why is cython now compiling Cython.Plex.Scanners with gcc but not > installing it with setup.py install? It should get installed automatically by distutils. It gets compiled because that makes it faster. It's a purely optional feature. > Is this a bug with Cython or on my side? I am just using the same > Debian package as I was using for all previous cython versions, but > suddenly it fails to build, because it depends on python-dev. So I > added python-dev to build-depends, it builds, it works, but the > Cython/Plex/Scanners.so is not installed in the final package. What's > wrong? Any error output? > Should I upload the package as it is, i.e. without the Scanners.so > file? I tested it on some simple .pyx files and it seems to work. You can, but I would prefer having that in. > I'd like to get this resolved in about 5 hours, so that I can upload > before today's dinstall run, so that Tim can build Sage in Debian with > it. Please provide some more details. Stefan From ondrej at certik.cz Sun Jul 20 14:02:12 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 20 Jul 2008 14:02:12 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> Message-ID: <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> On Sun, Jul 20, 2008 at 1:58 PM, Ondrej Certik wrote: > On Sun, Jul 20, 2008 at 1:31 PM, Stefan Behnel wrote: >> Hi, >> >> Ondrej Certik wrote: >>> Why is cython now compiling Cython.Plex.Scanners with gcc but not >>> installing it with setup.py install? >> >> It should get installed automatically by distutils. >> >> It gets compiled because that makes it faster. It's a purely optional feature. >> >> >>> Is this a bug with Cython or on my side? I am just using the same >>> Debian package as I was using for all previous cython versions, but >>> suddenly it fails to build, because it depends on python-dev. So I >>> added python-dev to build-depends, it builds, it works, but the >>> Cython/Plex/Scanners.so is not installed in the final package. What's >>> wrong? >> >> Any error output? > > No, all is fine. It just doesn't install. > >> >> >>> Should I upload the package as it is, i.e. without the Scanners.so >>> file? I tested it on some simple .pyx files and it seems to work. >> >> You can, but I would prefer having that in. > > Yes, me too. > >> >> >>> I'd like to get this resolved in about 5 hours, so that I can upload >>> before today's dinstall run, so that Tim can build Sage in Debian with >>> it. >> >> Please provide some more details. > > The build log attached. When installing the package, here is the list of files: Ok, I just tried that manually with setup.py install --home=. and it does install the .so file. So the packaging just needs to be changed. Here is the debian/rules: #!/usr/bin/make -f # -*- makefile -*- DEB_PYTHON_SYSTEM=pysupport include /usr/share/cdbs/1/rules/debhelper.mk include /usr/share/cdbs/1/class/python-distutils.mk install/cython:: dh_installman debian/cython.1 Let me investigate some options. BTW, if you want to help maintaining the Debian package, here is the svn: Vcs-Svn: svn://svn.debian.org/svn/python-apps/packages/cython/trunk Vcs-Browser: http://svn.debian.org/wsvn/python-apps/packages/cython/trunk/?op=log Ondrej From ondrej at certik.cz Sun Jul 20 14:09:45 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 20 Jul 2008 14:09:45 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> Message-ID: <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> On Sun, Jul 20, 2008 at 2:02 PM, Ondrej Certik wrote: > On Sun, Jul 20, 2008 at 1:58 PM, Ondrej Certik wrote: >> On Sun, Jul 20, 2008 at 1:31 PM, Stefan Behnel wrote: >>> Hi, >>> >>> Ondrej Certik wrote: >>>> Why is cython now compiling Cython.Plex.Scanners with gcc but not >>>> installing it with setup.py install? >>> >>> It should get installed automatically by distutils. >>> >>> It gets compiled because that makes it faster. It's a purely optional feature. >>> >>> >>>> Is this a bug with Cython or on my side? I am just using the same >>>> Debian package as I was using for all previous cython versions, but >>>> suddenly it fails to build, because it depends on python-dev. So I >>>> added python-dev to build-depends, it builds, it works, but the >>>> Cython/Plex/Scanners.so is not installed in the final package. What's >>>> wrong? >>> >>> Any error output? >> >> No, all is fine. It just doesn't install. >> >>> >>> >>>> Should I upload the package as it is, i.e. without the Scanners.so >>>> file? I tested it on some simple .pyx files and it seems to work. >>> >>> You can, but I would prefer having that in. >> >> Yes, me too. >> >>> >>> >>>> I'd like to get this resolved in about 5 hours, so that I can upload >>>> before today's dinstall run, so that Tim can build Sage in Debian with >>>> it. >>> >>> Please provide some more details. >> >> The build log attached. When installing the package, here is the list of files: > > Ok, I just tried that manually with > > setup.py install --home=. > > and it does install the .so file. So the packaging just needs to be > changed. Here is the debian/rules: > > #!/usr/bin/make -f > # -*- makefile -*- > > DEB_PYTHON_SYSTEM=pysupport > > include /usr/share/cdbs/1/rules/debhelper.mk > include /usr/share/cdbs/1/class/python-distutils.mk > > install/cython:: > dh_installman debian/cython.1 > > > > Let me investigate some options. BTW, if you want to help maintaining > the Debian package, here is the svn: > > Vcs-Svn: svn://svn.debian.org/svn/python-apps/packages/cython/trunk > Vcs-Browser: http://svn.debian.org/wsvn/python-apps/packages/cython/trunk/?op=log I think this patch needs to be applied: Index: debian/control =================================================================== --- debian/control (revision 1584) +++ debian/control (working copy) @@ -3,7 +3,7 @@ Priority: optional Maintainer: Python Applications Packaging Team Uploaders: Ondrej Certik -Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= 2.4.4-6) +Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= 2.4.4-6), python-dev (>= 2.4.4-6) Build-Depends-Indep: python-support (>= 0.7.5) Standards-Version: 3.8.0 Homepage: http://cython.org/ @@ -12,7 +12,7 @@ XS-DM-Upload-Allowed: yes Package: cython -Architecture: all +Architecture: any Depends: ${python:Depends}, ${misc:Depends} Suggests: gcc XB-Python-Version: ${python:Versions} However, the cython build fails then: cd . && python2.4 setup.py build --build-base="/tmp/buildd/cython-0.9.8/./build" /bin/sh: python2.4: command not found make: *** [python-build-stamp-2.4] Error 127 Now I am investigating why. python2.4 is not called from Cython, so it's seems some Debian thing. CCing to Tim -- I am quite busy today, I really need to work on my thesis, so I'd appreciate any help with this. Ondrej From dagss at student.matnat.uio.no Sun Jul 20 14:51:52 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 20 Jul 2008 14:51:52 +0200 Subject: [Cython] Assignments in cdefs Message-ID: <488334E8.7070601@student.matnat.uio.no> Just want to inform you about some particularily intrusive changes on cdef variables (see the latest dagss changes). I'll discuss it thoroughly because it is a nice example of reducing the number of variable assignment cases (more on that in another post). I'd really like to have this change go in, but it is not trivial and I can understand it if you resist it. 1) Bug fix: We generated erronous C code for cdef assignments in class bodies, I now raise a compiler error on this. 2) Basically I turn "cdef int x = 4" into "cdef int x; x = 4". 3) Also I added the "first" attribute to SingleAssignmentNode to generalize the concept of first assignment (which takes care of optimal refcounting). Optimizing first assignments in the more general case is now as simple as running through the tree and setting that attribute (after analysis, prior to generation), though I didn't do this. This kind of clashes with Robert's flow control endaveours, but I feel that the beginnings of flow control is overkill for this specific case, a single pass with a transform should do it... Results: - C code looks exactly like before (I diffed it), except for comments. - The buffer just started working without any extra efforts (and it made it very easy to optimize some refcounting for buffers, though that needed five lines). Of course, this is why I bothered. - No CVarDefNode in the tree after analyse_declarations As for the code: - Removed about 30 lines and added about 40. However I do feel that the transform code is less "copy-and-paste-with-modifications" and that new code is more obvious to beginners (Stefan can pull out his article URL by now :-) I'll follow up with another post to discuss assignments in general. -- Dag Sverre From dagss at student.matnat.uio.no Sun Jul 20 15:30:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 20 Jul 2008 15:30:30 +0200 Subject: [Cython] Assignment nodes, buffers and transforms Message-ID: <48833DF6.9080403@student.matnat.uio.no> I've had some skirmishes with Robert and Stefan on this and I think it is time for a thread. Currently, for buffers, I must somehow write code for two different cases that have to do with assignment: 1. When buffers variables are somehow assigned to 2. When buffers have items assigned to The question is whether to write more code in Nodes.py to do this, or make it easier for transforms to react to assignments. The latter seem more intrusive to how you usually do things, however I also see more usecases than just buffers (especially for optimizing refcount in a better way while keeping the algorithmic code simple, as well as for parts of type inference, as well as compile-time evaluation, and just overall transform complexity in number of node types to deal with). I'm not sure if I'm suggesting that there should only be one type of assignment node, but I'd love for them to be a bit different: - SetItemNode - SetAttributeNode - SetCVarNode ...so that the node type starts corresponding to what is going on in C without having to examine the lhs. I feel that would simplify a lot of code, also existing code (and it could be made "backwards-compatible" through inheritance anyway). However that is a second step that relies on reducing to a single assignment node first (to avoid combinatorial explosion). Current ways of assigning a variable x: a) def f(x): b) cdef type x = 2 c) x = 2 d) x, (y, z) = (2, (3, 4)) e) x = y = 2 f) except Exception, x g) for x in C: That's all I can think of (with is turned into c) already). All of these cases must be covered for both 1 and 2 above. What I've done so far is support a) and c) directly, and transformed b) into c) in the parsing stage. What I *could* do now is move on and turn d) and e) into c) as well. The effect of all these things is to i) reduce the number of different node types, and ii) remove code from Nodes.py that requires you to know about how all code generation/analysis work and instead add code that requires you to know about writing tree transforms (I think that even if the latter has usually more lines of code, they all follow a fundamental principle and results in code that is more robust to changes other places in the code, more "isolated" code). f) and g) can also be turned into c) but with a bit extra work and in less obvious ways. If you think about SetItemNode, SetAttributeNode etc. you see that it isn't wholly unatural for a loop to reduce into such instructions (that is of course what is happening today too, but less explicit). All in all I'm unsure about this point -- I have to transform them all, or write BufferNodes to wrap namenodes in the tree anyway, in which case the transforms weren't really necesarry. So I ask for input. (Help me counter the exhilirating feeling I get from having Buffer.py "just work" in more cases by refactoring unrelated pieces of code :-) ) From my own experiences, whenever I have to do something in Nodes.py or ExprNodes.py I invariably introduce bugs that it takes hours to find, because I do not really understand it. Of course I could learn, but I'd rather write the transforms _if that is a preferred direction anyway_. So the ideal preferred direction is what I'd like input on, and then I'll take the final call on work amount myself (and on how smart things I can cook up for f) and g), if they end up being hopeless then I'll drop it). -- Dag Sverre From stefan_ml at behnel.de Sun Jul 20 18:27:14 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jul 2008 18:27:14 +0200 Subject: [Cython] Standard include path In-Reply-To: <4881B9AC.3060909@student.matnat.uio.no> References: <4881B9AC.3060909@student.matnat.uio.no> Message-ID: <48836762.2010404@behnel.de> Hi, Dag Sverre Seljebotn wrote: > My test case has a call to malloc (tips to get around this are welcome > but I'd still like to have a look at include paths). So I'm cimporting > stdlib, meaning I have to export INCLUDE=Includes prior to running tests > (which I'd like to avoid as apparently no other tests relies on this?) > > There are two ways of going about this: > - Have runtests.py always include the Includes dir > - Have Cython itself always include the Includes dir (i.e. always > ../../Includes relative to Cython/Compiler/Main.py). > > I prefer the latter one; thoughts? I'm fine with the second one as long as it goes last on the include path, so that people can override it. > I suppose this might also mean > installing Includes "somewhere" on setup.py (or is that already done?) It should go into the "package_data" setup. Note sure where it should be installed, but Cython/Includes sounds good to me (i.e. package data of the "Cython" package). > Also there's another import issue: For more sophisticated transforms, > I've (or Robert, can't remember) come up with a __cython__ namespace > which basically contains "any symbol the compiler might like to use". > For instance: > > TreeFragment(u"__cython__.PyObject_GetBuffer(<__cython__.PyObject*>TMP, > &BUFINFO, 0)") > > Eventually all the __Pyx-utils might be declared in this one too (though > I'm certainly not going to do it for the sake of it). Then the idea is > that __cython__ is *always* cimported (and available to the user as well). > > Any strong feelings against simply making this a __cython__.pxd in the > aforementioned standard include path? Note the consequences: The > compiler will be hard-wired to work with a certain pxd file. That's not an issue, most compilers come with some kind of external files, such as header files or a standard or runtime library. But what it would really mean is that this __cython__.pxd would become available to users. Not sure that is a good idea. > The alternative is to create a cython-scope similar to builtin-scope > programatically (or stick everything that is needed in symtab.py directly). Personally, I would prefer a builtin scope that gets defined in a module somewhere. Will that only be a plain .pxd or will there be any additional generated code involved? If the latter, I'd certainly want to keep it as part of the Cython compiler code. Stefan From stefan_ml at behnel.de Sun Jul 20 19:09:23 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jul 2008 19:09:23 +0200 Subject: [Cython] More concerns over extern ctypedefs In-Reply-To: <4881DA9A.3070406@student.matnat.uio.no> References: <4881DA9A.3070406@student.matnat.uio.no> Message-ID: <48837143.2040509@behnel.de> Hi Dag, Dag Sverre Seljebotn wrote: > Consider this code: > > cdef extern from "t.h": > ctypedef long hullo > > def foo(hullo h): pass > > I noticed that it creates code like this: > PyArg_ParseTupleAndKeywords(__pyx_args, __pyx_kwds, "l", __pyx_argnames, > &__pyx_v_h)) > > Which could probably destroy the stack if we don't have the exact > typedef, right? Which means: One should either write our own parsing > code instead (generate for each parameter signature) Robert and I were discussing to finish up the parameter unpacking code ages ago and never really got down to it. We already have a lot of specialised code for arg tuple unpacking and keyword checking. I think all that is missing is an implementation for keyword unpacking and some code to prevent that arguments are passed as both positional and keyword arguments. The problem above might finally be a reason to do this. :) Stefan From ondrej at certik.cz Sun Jul 20 19:39:17 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 20 Jul 2008 19:39:17 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> Message-ID: <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> On Sun, Jul 20, 2008 at 7:27 PM, Timothy G Abbott wrote: > On Sun, 20 Jul 2008, Ondrej Certik wrote: >> >> I think this patch needs to be applied: >> >> Index: debian/control >> =================================================================== >> --- debian/control (revision 1584) >> +++ debian/control (working copy) >> @@ -3,7 +3,7 @@ >> Priority: optional >> Maintainer: Python Applications Packaging Team >> >> Uploaders: Ondrej Certik >> -Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= >> 2.4.4-6) >> +Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= >> 2.4.4-6), python-dev (>= 2.4.4-6) >> Build-Depends-Indep: python-support (>= 0.7.5) >> Standards-Version: 3.8.0 >> Homepage: http://cython.org/ >> @@ -12,7 +12,7 @@ >> XS-DM-Upload-Allowed: yes >> >> Package: cython >> -Architecture: all >> +Architecture: any >> Depends: ${python:Depends}, ${misc:Depends} >> Suggests: gcc >> XB-Python-Version: ${python:Versions} >> >> >> However, the cython build fails then: >> >> cd . && python2.4 setup.py build >> --build-base="/tmp/buildd/cython-0.9.8/./build" >> /bin/sh: python2.4: command not found >> make: *** [python-build-stamp-2.4] Error 127 >> >> Now I am investigating why. python2.4 is not called from Cython, so >> it's seems some Debian thing. CCing to Tim -- I am quite busy today, I >> really need to work on my thesis, so I'd appreciate any help with >> this. >> >> Ondrej > > I think you want python-all-dev (>= 2.4.4-6), rather than just python-dev > (>= 2.4.4-6). What's going on is that Debian build extensions modules for > all current versions of python (python2.4 and python2.5) when you're using > python-support; python-all-dev is the right way to include that as a > dependency. You are right. Let me check and if all is ok, upload. Ondrej From ondrej at certik.cz Sun Jul 20 19:46:03 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 20 Jul 2008 19:46:03 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> Message-ID: <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> On Sun, Jul 20, 2008 at 7:39 PM, Ondrej Certik wrote: > On Sun, Jul 20, 2008 at 7:27 PM, Timothy G Abbott wrote: >> On Sun, 20 Jul 2008, Ondrej Certik wrote: >>> >>> I think this patch needs to be applied: >>> >>> Index: debian/control >>> =================================================================== >>> --- debian/control (revision 1584) >>> +++ debian/control (working copy) >>> @@ -3,7 +3,7 @@ >>> Priority: optional >>> Maintainer: Python Applications Packaging Team >>> >>> Uploaders: Ondrej Certik >>> -Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= >>> 2.4.4-6) >>> +Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= >>> 2.4.4-6), python-dev (>= 2.4.4-6) >>> Build-Depends-Indep: python-support (>= 0.7.5) >>> Standards-Version: 3.8.0 >>> Homepage: http://cython.org/ >>> @@ -12,7 +12,7 @@ >>> XS-DM-Upload-Allowed: yes >>> >>> Package: cython >>> -Architecture: all >>> +Architecture: any >>> Depends: ${python:Depends}, ${misc:Depends} >>> Suggests: gcc >>> XB-Python-Version: ${python:Versions} >>> >>> >>> However, the cython build fails then: >>> >>> cd . && python2.4 setup.py build >>> --build-base="/tmp/buildd/cython-0.9.8/./build" >>> /bin/sh: python2.4: command not found >>> make: *** [python-build-stamp-2.4] Error 127 >>> >>> Now I am investigating why. python2.4 is not called from Cython, so >>> it's seems some Debian thing. CCing to Tim -- I am quite busy today, I >>> really need to work on my thesis, so I'd appreciate any help with >>> this. >>> >>> Ondrej >> >> I think you want python-all-dev (>= 2.4.4-6), rather than just python-dev >> (>= 2.4.4-6). What's going on is that Debian build extensions modules for >> all current versions of python (python2.4 and python2.5) when you're using >> python-support; python-all-dev is the right way to include that as a >> dependency. > > You are right. Let me check and if all is ok, upload. Ok, the package builds now, but it doesn't include the .so file. I don't have time at the moment to investigate, sorry. I can upload the package as pure python if you want. BTW, feel free to fix it in PAPT debian repo and upload, or ping me and I'll upload, that's not a problem. Ondrej From stefan_ml at behnel.de Sun Jul 20 19:51:17 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jul 2008 19:51:17 +0200 Subject: [Cython] Assignments in cdefs In-Reply-To: <488334E8.7070601@student.matnat.uio.no> References: <488334E8.7070601@student.matnat.uio.no> Message-ID: <48837B15.4010007@behnel.de> Hi Dag, Dag Sverre Seljebotn wrote: > 1) Bug fix: We generated erronous C code for cdef assignments in class > bodies, I now raise a compiler error on this. Good catch. Could you push that to cython-devel? (although an actual implementation to make this work would obviously be much better :) In general, I would appreciate it if trivial/obvious/necessary things went there directly. > 2) Basically I turn "cdef int x = 4" into "cdef int x; x = 4". > > 3) Also I added the "first" attribute to SingleAssignmentNode to > generalize the concept of first assignment (which takes care of optimal > refcounting). As long as these two go together, I absolutely see them as a gain. > Optimizing first assignments in the more general case is > now as simple as running through the tree and setting that attribute > (after analysis, prior to generation), though I didn't do this. What about code like this: def test(a): cdef int c if a: c = 1 if not a: c = 2 How would you know what is a "first assignment" here? And what would you gain from your ref-counting optimisation in this case? > This > kind of clashes with Robert's flow control endaveours, but I feel that > the beginnings of flow control is overkill for this specific case, a > single pass with a transform should do it... The current flow-control implementation is somewhat limited and even has a couple of issues. One thing I ran into recently was that it can crash with a stack overflow for functions with tons of conditionals. If this can be done any simpler and/or more reliable, I'm open to suggestions. > Results: > - C code looks exactly like before (I diffed it), except for comments. Nice. > - The buffer just started working without any extra efforts (and it made > it very easy to optimize some refcounting for buffers, though that > needed five lines). Of course, this is why I bothered. Happy to hear that. > - No CVarDefNode in the tree after analyse_declarations Fine with me. Stefan From dagss at student.matnat.uio.no Sun Jul 20 20:22:47 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 20 Jul 2008 20:22:47 +0200 Subject: [Cython] Assignments in cdefs In-Reply-To: <48837B15.4010007@behnel.de> References: <488334E8.7070601@student.matnat.uio.no> <48837B15.4010007@behnel.de> Message-ID: <48838277.7090700@student.matnat.uio.no> Stefan Behnel wrote: > What about code like this: > > def test(a): > cdef int c > if a: > c = 1 > if not a: > c = 2 > > How would you know what is a "first assignment" here? And what would you gain > from your ref-counting optimisation in this case? I didn't do the general optimization, I simply set first = True on the assignment I generated when transforming the cdef statement "manually", so this is only half-way to the general thing (why? Because I'd need to _track assignments_, that's another usecase for my assignment refactoring proposal :-) With that in place I would have started done a simple one, would have taken 15 minutes [1]). As for the example, it is a "best-effort" thing. If no assignments are flagged as first, then the optimization just doesn't happen, which is the likely outcome of your example. However, if you come up with an algorithm to check this, then in this case you can flag them both as first. I should perhaps be more precise in my claim: I don't actually have an algorithm for doing this -- what I'm saying is that *if* one uses a fancy flow control algorithm it can be put into a self-contained visitor that goes over the node graph and flags assignments as it sees fit. It just seems easier to do this kind of stuff by having the "call thread" belong to the flow control algorithm rather than rely on setting side effects everywhere. At any rate, [1] would provide enough information to reconstruct the information I now discarded from the existing flow control, which should do for having the patch accepted :-) [1] The typical simple algorithm (which I'll do if assignment refactorings happen) would be like this: - Any assignments not on top indent level would never be first assignments - Any assignments on the top indent level which is the first ("vertically" and regardless of indentation level) is flagged as first. -- Dag Sverre From dagss at student.matnat.uio.no Sun Jul 20 20:29:33 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 20 Jul 2008 20:29:33 +0200 Subject: [Cython] Assignments in cdefs In-Reply-To: <48838277.7090700@student.matnat.uio.no> References: <488334E8.7070601@student.matnat.uio.no> <48837B15.4010007@behnel.de> <48838277.7090700@student.matnat.uio.no> Message-ID: <4883840D.8030800@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> What about code like this: >> >> def test(a): >> cdef int c >> if a: >> c = 1 >> if not a: >> c = 2 >> > > As for the example, it is a "best-effort" thing. If no assignments are > flagged as first, then the optimization just doesn't happen, which is > the likely outcome of your example. However, if you come up with an > algorithm to check this, then in this case you can flag them both as first. In fact, in this example there is no first assignment, as you could overload __not__ and have True in both cases :-) In general such sophisticated control only works for c-typed branch tests which makes me doubt the value of it...? But it doesn't look horrendously difficult to write code tackling it either...but that's another GSoC :-) -- Dag Sverre From stefan_ml at behnel.de Sun Jul 20 20:40:23 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jul 2008 20:40:23 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> Message-ID: <48838697.4070905@behnel.de> Ondrej Certik wrote: > Ok, the package builds now, but it doesn't include the .so file. I > don't have time at the moment to investigate, sorry. I can upload the > package as pure python if you want. Pure Python is better than no Cython. :) As I said, it's only an optional optimisation. (although I'd really like to know why this causes problems). > BTW, feel free to fix it in PAPT > debian repo and upload, or ping me and I'll upload, that's not a > problem. I'd be happy if you could include this patch. http://hg.cython.org/cython-devel/rev/e21391d5f23a It didn't make it into 0.9.8, but it makes Cython generate C code that compiles with Py3 (beta 1/2). Stefan From stefan_ml at behnel.de Sun Jul 20 20:44:25 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jul 2008 20:44:25 +0200 Subject: [Cython] Assignments in cdefs In-Reply-To: <48838277.7090700@student.matnat.uio.no> References: <488334E8.7070601@student.matnat.uio.no> <48837B15.4010007@behnel.de> <48838277.7090700@student.matnat.uio.no> Message-ID: <48838789.3070407@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> What about code like this: >> >> def test(a): >> cdef int c >> if a: >> c = 1 >> if not a: >> c = 2 >> >> How would you know what is a "first assignment" here? And what would you gain >> from your ref-counting optimisation in this case? > > As for the example, it is a "best-effort" thing. If no assignments are > flagged as first, then the optimization just doesn't happen, which is > the likely outcome of your example. Then what exactly is your "ref-counting optimisation" doing? Can you give me a code example? Stefan From dagss at student.matnat.uio.no Sun Jul 20 21:02:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 20 Jul 2008 21:02:03 +0200 Subject: [Cython] Assignments in cdefs In-Reply-To: <48838789.3070407@behnel.de> References: <488334E8.7070601@student.matnat.uio.no> <48837B15.4010007@behnel.de> <48838277.7090700@student.matnat.uio.no> <48838789.3070407@behnel.de> Message-ID: <48838BAB.7000606@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> What about code like this: >>> >>> def test(a): >>> cdef int c >>> if a: >>> c = 1 >>> if not a: >>> c = 2 >>> >>> How would you know what is a "first assignment" here? And what would you gain >>> from your ref-counting optimisation in this case? >> As for the example, it is a "best-effort" thing. If no assignments are >> flagged as first, then the optimization just doesn't happen, which is >> the likely outcome of your example. > > Then what exactly is your "ref-counting optimisation" doing? Can you give me a > code example? It makes "cdef int x = 3" behave as before, no more or less. Just refactoring the code so that buffer worked. I needed to put the stuff that handled the difference in refcounting somewhere, and that somewhere leaves a hole in which to plug further improvements, so it was a natural name, but it doesn't mean that it is more optimized than it used to be. -- Dag Sverre From dagss at student.matnat.uio.no Sun Jul 20 21:04:52 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 20 Jul 2008 21:04:52 +0200 Subject: [Cython] Assignments in cdefs In-Reply-To: <48838BAB.7000606@student.matnat.uio.no> References: <488334E8.7070601@student.matnat.uio.no> <48837B15.4010007@behnel.de> <48838277.7090700@student.matnat.uio.no> <48838789.3070407@behnel.de> <48838BAB.7000606@student.matnat.uio.no> Message-ID: <48838C54.3020309@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> What about code like this: >>>> >>>> def test(a): >>>> cdef int c >>>> if a: >>>> c = 1 >>>> if not a: >>>> c = 2 >>>> >>>> How would you know what is a "first assignment" here? And what would you gain >>>> from your ref-counting optimisation in this case? >>> As for the example, it is a "best-effort" thing. If no assignments are >>> flagged as first, then the optimization just doesn't happen, which is >>> the likely outcome of your example. >> Then what exactly is your "ref-counting optimisation" doing? Can you give me a >> code example? > > It makes "cdef int x = 3" behave as before, no more or less. Just > refactoring the code so that buffer worked. I needed to put the stuff > that handled the difference in refcounting somewhere, and that somewhere > leaves a hole in which to plug further improvements, so it was a natural > name, but it doesn't mean that it is more optimized than it used to be. > Sorry! I meant cdef MyExtensionsType a = x (as int isn't refcounted...). Hmm, I guess that's where the confusion comes from; would tend to make everything I've said unintelligible. -- Dag Sverre From stefan_ml at behnel.de Sun Jul 20 21:33:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jul 2008 21:33:02 +0200 Subject: [Cython] Assignments in cdefs In-Reply-To: <48838C54.3020309@student.matnat.uio.no> References: <488334E8.7070601@student.matnat.uio.no> <48837B15.4010007@behnel.de> <48838277.7090700@student.matnat.uio.no> <48838789.3070407@behnel.de> <48838BAB.7000606@student.matnat.uio.no> <48838C54.3020309@student.matnat.uio.no> Message-ID: <488392EE.3060009@behnel.de> Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Then what exactly is your "ref-counting optimisation" doing? Can you give me a >>> code example? >> It makes "cdef int x = 3" behave as before, no more or less. Just >> refactoring the code so that buffer worked. I needed to put the stuff >> that handled the difference in refcounting somewhere, and that somewhere >> leaves a hole in which to plug further improvements, so it was a natural >> name, but it doesn't mean that it is more optimized than it used to be. >> > > Sorry! I meant > > cdef MyExtensionsType a = x > > (as int isn't refcounted...). Ok, I just checked that there's a test case for this (run/cdefassign), and it still seems to generate the correct and optimised code on your branch. So I'm fine with your change. Stefan From dagss at student.matnat.uio.no Mon Jul 21 00:33:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 21 Jul 2008 00:33:00 +0200 Subject: [Cython] GSoC status page updated, at a crossroad Message-ID: <4883BD1C.2000907@student.matnat.uio.no> http://wiki.cython.org/DagSverreSeljebotn/status I'm at a crossroad now -- as I saw what it would take for how I would like it, I ended up in the same hole as closures: Things would be easier (possible..) if result_code was refactored. In some ways this is good, because it would tend to pull out a blocker for a lot of features. However, I have to admit that it is not the speediest way to complete the basic featureset of my GSoC. However, if somebody else has the time to step up and do the result_code refactoring (Greg has started on some of it in Pyrex but more is needed I think) I think it would benefit Cython more (i.e. more robust temporary handling, closures, more robust forinstatnode, etc...). My time estimate is about one day, so it probably takes three. (YMMV, I don't know ExprNodes.py that well yet.) -- Dag Sverre From greg.ewing at canterbury.ac.nz Mon Jul 21 02:12:48 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 21 Jul 2008 12:12:48 +1200 Subject: [Cython] More concerns over extern ctypedefs In-Reply-To: <4881DC32.3030103@student.matnat.uio.no> References: <4881DA9A.3070406@student.matnat.uio.no> <4881DC32.3030103@student.matnat.uio.no> Message-ID: <4883D480.6080205@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > Just realized that a dynamic typestring could be generated at module > load time, which should be perfectly unnoticeable. So this might be the > way to go. A simpler approach would be to disallow the use of typedefed types to declare arguments to Python functions, and require the use of explicit C types. -- Greg From stefan_ml at behnel.de Mon Jul 21 09:22:05 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 21 Jul 2008 09:22:05 +0200 (CEST) Subject: [Cython] Cython 0.9.8 released In-Reply-To: <85b5c3130807201310o22b565am16661cf1493060d2@mail.gmail.com> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> <48838697.4070905@behnel.de> <85b5c3130807201310o22b565am16661cf1493060d2@mail.gmail.com> Message-ID: <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Ondrej Certik wrote: > On Sun, Jul 20, 2008 at 8:40 PM, Stefan Behnel wrote: >> I'd be happy if you could include this patch. >> >> http://hg.cython.org/cython-devel/rev/e21391d5f23a >> >> It didn't make it into 0.9.8, but it makes Cython generate C code that >> compiles with Py3 (beta 1/2). > > I didn't put it in there yet. Why not to make a new upstream release? > That way it will help a lot more users. Yes, that would be the clean way. I was just thinking that since you were pushing a Debian package anyway, it'd be nice to have something that people can use for a little longer. Robert, could we have rev 719 as an intermediate 0.9.8.1? It would really help people who want to get their Cython code ready for Py3 (not that I heard anyone shouting, but I wouldn't want to be called a bottleneck when the code is already there). Thanks, Stefan From ondrej at certik.cz Mon Jul 21 10:42:48 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 21 Jul 2008 10:42:48 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <4852169A.1050706@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> <48838697.4070905@behnel.de> <85b5c3130807201310o22b565am16661cf1493060d2@mail.gmail.com> <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <85b5c3130807210142p27ef2cf2v67420bb9b3555db@mail.gmail.com> On Mon, Jul 21, 2008 at 9:22 AM, Stefan Behnel wrote: > Ondrej Certik wrote: >> On Sun, Jul 20, 2008 at 8:40 PM, Stefan Behnel wrote: >>> I'd be happy if you could include this patch. >>> >>> http://hg.cython.org/cython-devel/rev/e21391d5f23a >>> >>> It didn't make it into 0.9.8, but it makes Cython generate C code that >>> compiles with Py3 (beta 1/2). >> >> I didn't put it in there yet. Why not to make a new upstream release? >> That way it will help a lot more users. > > Yes, that would be the clean way. I was just thinking that since you were > pushing a Debian package anyway, it'd be nice to have something that > people can use for a little longer. I would have create debian/patches, put the patch under quilt control, inlude the quilt makefile in debian/rules and test it. Then remove everything. I would do it if the patch was going to stay there for a long time or was really critical. But since this is broken in the latest version of cython for everyone, I just prefer if you fix it upstream and I just reupload a new version, which is a matter of a few minutes (unless you change from noncompiled to compiled like this time:). Ondrej From dalcinl at gmail.com Mon Jul 21 16:04:42 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 21 Jul 2008 11:04:42 -0300 Subject: [Cython] problems with current cython-devel repo In-Reply-To: <4881ACBB.4030207@behnel.de> References: <487788FF.6040707@behnel.de> <4881ACBB.4030207@behnel.de> Message-ID: Many thanks, Stefan. Now all is working for me with current cython-devel head. On 7/19/08, Stefan Behnel wrote: > Hi, > > > Lisandro Dalcin wrote: > > Now I've got another failure with my code. > > > > The following works (): > > > > def winsize(size): > > try: > > w, h = size > > except TypeError: > > w = size > > h = size > > return w, h > > > > But the following not: > > > > def winsize(size): > > try: > > w, h = size > > except TypeError: > > w = h = size > > return w, h > > > This should be fixed now. > > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From tolot at solar-empire.de Mon Jul 21 14:07:48 2008 From: tolot at solar-empire.de (Marc Christiansen) Date: Mon, 21 Jul 2008 14:07:48 +0200 Subject: [Cython] Cython 0.9.8 released References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> <48838697.4070905@behnel.de> Message-ID: Stefan Behnel wrote: > > Ondrej Certik wrote: >> Ok, the package builds now, but it doesn't include the .so file. I >> don't have time at the moment to investigate, sorry. I can upload the >> package as pure python if you want. > > Pure Python is better than no Cython. :) > > As I said, it's only an optional optimisation. (although I'd really like to > know why this causes problems). Hi, I think I can shed some light on the problem. The observed behaviour (building Scanners.so, but not installing it) can be reproduced with the following two commands: python setup.py build python setup.py install --no-compile --root /tmp build uses build/lib.linux-x86_64-2.5/ as target directory, install does a new build into build/lib/ and installs the contents of this. The cause of the problem is that setup.py adds the Scanners.so extension to the setup() arguments only if the '--no-compile' flag is not given. --no-compile is an option for install, not for build. Which means, when using the mentioned command sequence (which gentoo does, and I guess debian too), the extension gets created anyways. It just doesn't get installed. Thinking about it, this part try: sys.argv.remove("--no-compile") except ValueError: #try to build the extension is an error, because it causes the .py files to be compiled. setup.py install --help says: --no-compile don't compile .py files Ciao Marc From stefan_ml at behnel.de Mon Jul 21 19:15:15 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 21 Jul 2008 19:15:15 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> <48838697.4070905@behnel.de> Message-ID: <4884C423.2000605@behnel.de> Hi, Marc Christiansen wrote: > Thinking about it, this part > try: > sys.argv.remove("--no-compile") > except ValueError: > #try to build the extension > > is an error, because it causes the .py files to be compiled. setup.py > install --help says: > --no-compile don't compile .py files Ah, thanks. I guess that was just too obvious a name to expect that it isn't in use already... We can change that to "--no-cython-compile" or something like that for our purpose, so that we no longer interfere with the normal distutils arguments. Stefan From stefan_ml at behnel.de Mon Jul 21 20:25:26 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 21 Jul 2008 20:25:26 +0200 Subject: [Cython] Standard include path In-Reply-To: <48836762.2010404@behnel.de> References: <4881B9AC.3060909@student.matnat.uio.no> <48836762.2010404@behnel.de> Message-ID: <4884D496.4010001@behnel.de> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> There are two ways of going about this: >> - Have runtests.py always include the Includes dir >> - Have Cython itself always include the Includes dir (i.e. always >> ../../Includes relative to Cython/Compiler/Main.py). >> >> I prefer the latter one; thoughts? > > I'm fine with the second one as long as it goes last on the include path, so > that people can override it. > >> I suppose this might also mean >> installing Includes "somewhere" on setup.py (or is that already done?) > > It should go into the "package_data" setup. Note sure where it should be > installed, but Cython/Includes sounds good to me (i.e. package data of the > "Cython" package). I changed the setup.py to install it as Cython/Includes. It's not used as import path yet, but that should be trivial. Stefan From tolot at solar-empire.de Mon Jul 21 21:07:55 2008 From: tolot at solar-empire.de (Marc Christiansen) Date: Mon, 21 Jul 2008 21:07:55 +0200 Subject: [Cython] Problem with multiple assignment Message-ID: Hi, I'm not sure if multiple assignment is the right name, but have a look at this code: cdef f1(): cdef char a, b a = b = False Cython 0.9.8 creates this: [...] * * a = b = False # <<<<<<<<<<<<<< */ __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} __pyx_v_a = __pyx_1; __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} __pyx_v_b = __pyx_1; [...] If I write a = b = 0 (or a = False; b = False), the result is the expected __pyx_v_a = 0; __pyx_v_b = 0; Ciao Marc From stefan_ml at behnel.de Tue Jul 22 07:41:08 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Jul 2008 07:41:08 +0200 Subject: [Cython] Problem with multiple assignment In-Reply-To: References: Message-ID: <488572F4.3010706@behnel.de> Hi, thanks for the report. Marc Christiansen wrote: > I'm not sure if multiple assignment is the right name, but have a look > at this code: > > cdef f1(): > cdef char a, b > > a = b = False > > Cython 0.9.8 creates this: > > [...] > * > * a = b = False # <<<<<<<<<<<<<< > */ > __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} > __pyx_v_a = __pyx_1; > __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} > __pyx_v_b = __pyx_1; > > [...] > > If I write a = b = 0 (or a = False; b = False), the result is the expected > __pyx_v_a = 0; > __pyx_v_b = 0; This is not really a problem, more of a missing optimisation for a case that's handled differently internally. It sounds like it could go away with Dag's work on the assignment handling anyway. Dag? Stefan From dagss at student.matnat.uio.no Tue Jul 22 10:29:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 22 Jul 2008 10:29:00 +0200 Subject: [Cython] Problem with multiple assignment Message-ID: <3299567351.716273@smtp.netcom.no> From: Stefan Behnel Date: Tuesday, Jul 22, 2008 7:41 am Subject: Re: [Cython] Problem with multiple assignment To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hi, > >thanks for the report. > >Marc Christiansen wrote: > I'm not sure if multiple assignment is the right name, but have a look > at this code: > > cdef f1(): > cdef char a, b > > a = b = False > > Cython 0.9.8 creates this: > > [...] > * > * a = b = False # <<<<<<<<<<<<<< > */ > __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} > __pyx_v_a = __pyx_1; > __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} > __pyx_v_b = __pyx_1; > > [...] > > If I write a = b = 0 (or a = False; b = False), the result is the expected > __pyx_v_a = 0; > __pyx_v_b = 0; > >This is not really a problem, more of a missing optimisation for a case that's handled differently internally. It sounds like it could go away with Dag's work on the assignment handling anyway. Dag? > The way I visioned it it would turn into cdef object tmp .. tmp = False a = tmp b = tmp However I suppose I could handle "rhs is a simple NameNode referring to something which is not in the global module dict etc etc" as an optimizezable special case which would give a = None b = None instead. So the answer is "I didn't think about it, but it is possible, but needs extra attention". (The assignment work will probably end up on pause this time around though, but if you want it I might get back to it at a later time. The way I would prefer to do it it depends on result_code refactoring, same as closures). Dag Sverre From dagss at student.matnat.uio.no Tue Jul 22 12:16:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 22 Jul 2008 12:16:00 +0200 Subject: [Cython] Problem with multiple assignment Message-ID: <3299573810.772000@smtp.netcom.no> From: "Dag Sverre Seljebotn" Date: Tuesday, Jul 22, 2008 10:34 am Subject: Re: [Cython] Problem with multiple assignment To: Reply-To: cython-dev at codespeak.net >From: Stefan Behnel >Date: Tuesday, Jul 22, 2008 7:41 am >Subject: Re: [Cython] Problem with multiple assignment >To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > >Hi, > >>thanks for the report. > >>Marc Christiansen wrote: > I'm not sure if multiple assignment is the right name, but have a look > at this code: > > cdef f1(): > cdef char a, b > > a = b = False > > Cython 0.9.8 creates this: > > [...] > * > * a = b = False # <<<<<<<<<<<<<< > */ > __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} > __pyx_v_a = __pyx_1; > __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} > __pyx_v_b = __pyx_1; > > [...] > > If I write a = b = 0 (or a = False; b = False), the result is the expected > __pyx_v_a = 0; > __pyx_v_b = 0; > >>This is not really a problem, more of a missing optimisation for a case that's handled differently internally. It sounds like it could go away with Dag's work on the assignment handling anyway. Dag? > > >The way I visioned it it would turn into > >cdef object tmp >.. >tmp = False >a = tmp >b = tmp > >However I suppose I could handle "rhs is a simple NameNode referring to something which is not in the global module dict etc etc" as an optimizezable special case which would give > >a = None >b = None > >instead. I meant ...= False. Dag Sverre From stefan_ml at behnel.de Tue Jul 22 16:05:42 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Jul 2008 16:05:42 +0200 (CEST) Subject: [Cython] Problem with multiple assignment In-Reply-To: <3299567351.716273@smtp.netcom.no> References: <3299567351.716273@smtp.netcom.no> Message-ID: <61181.213.61.181.86.1216735542.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > The way I visioned it it would turn into > > cdef object tmp > .. > tmp = False > a = tmp > b = tmp What about this code: cdef int i cdef float f cdef object o o = i = f = 1 I think the current way to split the assignments and handle them separately isn't bad at all. This gives us the pretty code for my example. > However I suppose I could handle "rhs is a simple NameNode referring to > something which is not in the global module dict etc etc" as an > optimizezable special case which would give > > a = False > b = False That's most likely something that the C compiler would do anyway, but that aside, there already is some special handling code for True, False and None that should be easy to adopt here. I think doing the same thing in the case of cascading assignments is the right fix for now. Stefan From dagss at student.matnat.uio.no Tue Jul 22 16:36:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 22 Jul 2008 16:36:00 +0200 Subject: [Cython] Problem with multiple assignment Message-ID: <3299589389.992820@smtp.netcom.no> Stefan wrote: >What about this code: > > cdef int i > cdef float f > cdef object o > o = i = f = 1 > >I think the current way to split the assignments and handle them >separately isn't bad at all. This gives us the pretty code for my example. I think (?) the rhs is either always an expression with one exact type (which can be further coerced, but still), or a constant literal. So I don't see a problem here, literals can be copied and expressions stored in temps. Or, it might work to simply analyse the temp as int, which coerces into the right thing on the individual assignments. The thing is, we're going to hit C at some point anyway, and the C code should be identical for any approach. Whatever way the current cascaded ass works, you just take it and try to break it up into smaller, more powerful concepts. It is all just about the numbers of concepts we are willing to carry across the whole way to code generation, and why... Currently I think all the different node types (or rather, the way they gravitate around Python constructs rather than CPython/C) makes it more difficult to increase how smart we can be in any significant way (with all kinds of optimizations, though especially refcounting perhaps). I'm starting to sound like a broken record though. I wrote a post a while ago called "a high level discussion" or something like that, I think anything on this subject should be taken up in that thread. I am not contesting that what works now works now, I am contesting whether it is the best model for future Cython development. (which also depends on the ambitions and goals I suppose). > ... I think doing the same thing in the case of cascading assignments is the right fix for now. Agreed (though it doesn't look very important, if you have a char just type 0...). As I said I've paused it. (And if I'm the only one to think it would lead to better code design then it should perhaps not be done) Dag Sverre From dagss at student.matnat.uio.no Tue Jul 22 18:27:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 22 Jul 2008 18:27:00 +0200 Subject: [Cython] Standard include path Message-ID: <3299596049.1096671@smtp.netcom.no> Stefan wrote: > >I changed the setup.py to install it as Cython/Includes. It's not used as import path yet, but that should be trivial. > Thanks! Dag Sverre From greg.ewing at canterbury.ac.nz Wed Jul 23 02:20:57 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 23 Jul 2008 12:20:57 +1200 Subject: [Cython] Problem with multiple assignment In-Reply-To: <3299589389.992820@smtp.netcom.no> References: <3299589389.992820@smtp.netcom.no> Message-ID: <48867969.8010909@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > Or, it might work to simply analyse the temp as int, which coerces > into the right thing on the individual assignments. The way I think I would approach this is to consider True and False to be of a special 'boolean' type, which is a kind of C int that coerces into Py_True or Py_False when used in a Python context. -- Greg From stefan_ml at behnel.de Wed Jul 23 07:36:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Jul 2008 07:36:57 +0200 Subject: [Cython] Problem with multiple assignment In-Reply-To: <48867969.8010909@canterbury.ac.nz> References: <3299589389.992820@smtp.netcom.no> <48867969.8010909@canterbury.ac.nz> Message-ID: <4886C379.8040802@behnel.de> Hi, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> Or, it might work to simply analyse the temp as int, which coerces >> into the right thing on the individual assignments. > > The way I think I would approach this is to consider True > and False to be of a special 'boolean' type, which is a > kind of C int that coerces into Py_True or Py_False when > used in a Python context. Cython has the "bint" C type for this. Stefan From robertwb at math.washington.edu Wed Jul 23 09:15:44 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 00:15:44 -0700 Subject: [Cython] Problem with multiple assignment In-Reply-To: <3299567351.716273@smtp.netcom.no> References: <3299567351.716273@smtp.netcom.no> Message-ID: <2A0F7DB4-77AE-462A-B59E-140511BC37A3@math.washington.edu> On Jul 22, 2008, at 1:29 AM, Dag Sverre Seljebotn wrote: > From: Stefan Behnel > Date: Tuesday, Jul 22, 2008 7:41 am > Subject: Re: [Cython] Problem with multiple assignment > To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > > Hi, >> >> thanks for the report. Yes, very interesting. >> >> Marc Christiansen wrote: >> I'm not sure if multiple assignment is the right name, but have a >> look >> at this code: >> >> cdef f1(): >> cdef char a, b >> >> a = b = False >> >> Cython 0.9.8 creates this: >> >> [...] >> * >> * a = b = False # <<<<<<<<<<<<<< >> */ >> __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == >> (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; >> __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} >> __pyx_v_a = __pyx_1; >> __pyx_1 = __pyx_PyInt_char(Py_False); if (unlikely((__pyx_1 == >> (char)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; >> __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1;} >> __pyx_v_b = __pyx_1; >> >> [...] >> >> If I write a = b = 0 (or a = False; b = False), the result is the >> expected >> __pyx_v_a = 0; >> __pyx_v_b = 0; >> >> This is not really a problem, more of a missing optimisation for a >> case that's handled differently internally. It sounds like it >> could go away with Dag's work on the assignment handling anyway. Dag? >> > > The way I visioned it it would turn into > > cdef object tmp > .. > tmp = False > a = tmp > b = tmp This is in fact exactly what it turns into, though perhaps more implicitly then you are thinking of doing, so you are right that some special casing may be needed here. However, I think a better optimization is that (as has been hinted) True and False start as bints, and then get coerced to PyObjects as needed....that was easy. Now we have * a = b = False # <<<<<<<<<<<<<< * */ __pyx_v_a = 0; __pyx_v_b = 0; as desired. Note that for Python objects * x = False # <<<<<<<<<<<<<< * */ __pyx_1 = __Pyx_PyBool_FromLong(0); if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 16; __pyx_clineno = __LINE__; goto __pyx_L1;} Py_DECREF(__pyx_v_x); __pyx_v_x = __pyx_1; Where most of this gets optimized away as #define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False)) (though it perhaps could be made to look prettier). - Robert From robertwb at math.washington.edu Wed Jul 23 09:26:23 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 00:26:23 -0700 Subject: [Cython] Standard include path In-Reply-To: <48836762.2010404@behnel.de> References: <4881B9AC.3060909@student.matnat.uio.no> <48836762.2010404@behnel.de> Message-ID: On Jul 20, 2008, at 9:27 AM, Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> My test case has a call to malloc (tips to get around this are >> welcome >> but I'd still like to have a look at include paths). So I'm >> cimporting >> stdlib, meaning I have to export INCLUDE=Includes prior to running >> tests >> (which I'd like to avoid as apparently no other tests relies on >> this?) >> >> There are two ways of going about this: >> - Have runtests.py always include the Includes dir >> - Have Cython itself always include the Includes dir (i.e. always >> ../../Includes relative to Cython/Compiler/Main.py). >> >> I prefer the latter one; thoughts? > > I'm fine with the second one as long as it goes last on the include > path, so > that people can override it. Yep, I think this is a good idea too (in fact, it's what I envisioned happening with that directory). >> Also there's another import issue: For more sophisticated transforms, >> I've (or Robert, can't remember) come up with a __cython__ namespace >> which basically contains "any symbol the compiler might like to use". >> For instance: >> >> TreeFragment(u"__cython__.PyObject_GetBuffer >> (<__cython__.PyObject*>TMP, >> &BUFINFO, 0)") >> >> Eventually all the __Pyx-utils might be declared in this one too >> (though >> I'm certainly not going to do it for the sake of it). Then the >> idea is >> that __cython__ is *always* cimported (and available to the user >> as well). >> >> Any strong feelings against simply making this a __cython__.pxd in >> the >> aforementioned standard include path? Note the consequences: The >> compiler will be hard-wired to work with a certain pxd file. > > That's not an issue, most compilers come with some kind of external > files, > such as header files or a standard or runtime library. > > But what it would really mean is that this __cython__.pxd would become > available to users. Not sure that is a good idea. Being available to the user comes with being able to use tree fragments in this way (unless we explicitly disallow it somehow). Seems reasonable to me. Should we use submodules of __cython__ rather than sticking everything we come up with in that single module? >> The alternative is to create a cython-scope similar to builtin-scope >> programatically (or stick everything that is needed in symtab.py >> directly). > > Personally, I would prefer a builtin scope that gets defined in a > module > somewhere. Will that only be a plain .pxd or will there be any > additional > generated code involved? If the latter, I'd certainly want to keep > it as part > of the Cython compiler code. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Wed Jul 23 09:37:08 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 00:37:08 -0700 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> <48838697.4070905@behnel.de> <85b5c3130807201310o22b565am16661cf1493060d2@mail.gmail.com> <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <230B7B30-E1BF-4132-91EB-DD49335862FA@math.washington.edu> On Jul 21, 2008, at 12:22 AM, Stefan Behnel wrote: > Ondrej Certik wrote: >> On Sun, Jul 20, 2008 at 8:40 PM, Stefan Behnel wrote: >>> I'd be happy if you could include this patch. >>> >>> http://hg.cython.org/cython-devel/rev/e21391d5f23a >>> >>> It didn't make it into 0.9.8, but it makes Cython generate C code >>> that >>> compiles with Py3 (beta 1/2). >> >> I didn't put it in there yet. Why not to make a new upstream release? >> That way it will help a lot more users. > > Yes, that would be the clean way. I was just thinking that since > you were > pushing a Debian package anyway, it'd be nice to have something that > people can use for a little longer. > > Robert, could we have rev 719 as an intermediate 0.9.8.1? It would > really > help people who want to get their Cython code ready for Py3 (not > that I > heard anyone shouting, but I wouldn't want to be called a > bottleneck when > the code is already there). Sure, I'd be fine with that. It's been a while since we've done a release, but the devel branch seems to have been a bit unstable lately (more so than usual). Any thoughts on how far away it would be to just release tip? - Robert From robertwb at math.washington.edu Wed Jul 23 09:45:12 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 00:45:12 -0700 Subject: [Cython] More concerns over extern ctypedefs In-Reply-To: <4883D480.6080205@canterbury.ac.nz> References: <4881DA9A.3070406@student.matnat.uio.no> <4881DC32.3030103@student.matnat.uio.no> <4883D480.6080205@canterbury.ac.nz> Message-ID: On Jul 20, 2008, at 5:12 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: >> Just realized that a dynamic typestring could be generated at module >> load time, which should be perfectly unnoticeable. So this might >> be the >> way to go. I like this idea. Alternatively, one could pass an actual long (say) into the PyArg_ParseTupleAndKeywords parameter list, and then cast the result, but I like your idea better. > A simpler approach would be to disallow the use of > typedefed types to declare arguments to Python functions, > and require the use of explicit C types. I fear this would be a step backwards in usability... - Robert From robertwb at math.washington.edu Wed Jul 23 10:15:59 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 01:15:59 -0700 Subject: [Cython] Assignments in cdefs In-Reply-To: <488334E8.7070601@student.matnat.uio.no> References: <488334E8.7070601@student.matnat.uio.no> Message-ID: On Jul 20, 2008, at 5:51 AM, Dag Sverre Seljebotn wrote: > Just want to inform you about some particularily intrusive changes on > cdef variables (see the latest dagss changes). I'll discuss it > thoroughly because it is a nice example of reducing the number of > variable assignment cases (more on that in another post). I'd really > like to have this change go in, but it is not trivial and I can > understand it if you resist it. > > 1) Bug fix: We generated erronous C code for cdef assignments in class > bodies, I now raise a compiler error on this. Good. This is actually related to the recent feature request that we had on default cdef values... > 2) Basically I turn "cdef int x = 4" into "cdef int x; x = 4". Nice. This is almost the poster example of turning complex Python code into simpler statements--well worth it in this case. > 3) Also I added the "first" attribute to SingleAssignmentNode to > generalize the concept of first assignment (which takes care of > optimal > refcounting). Optimizing first assignments in the more general case is > now as simple as running through the tree and setting that attribute > (after analysis, prior to generation), though I didn't do this. This > kind of clashes with Robert's flow control endaveours, but I feel that > the beginnings of flow control is overkill for this specific case, a > single pass with a transform should do it... I'll be the first to say it, the control flow as it is now is quite the unfinished hack, and I'm not even sure it's the right way to go. Really, the only reason it is there is so it can detect when it can save on increfing the arguments (in other words, detecting if they're ever assigned to). Real control flow was one of the things I was most sad about loosing the other GSoC project... > Results: > - C code looks exactly like before (I diffed it), except for comments. > - The buffer just started working without any extra efforts (and it > made > it very easy to optimize some refcounting for buffers, though that > needed five lines). Of course, this is why I bothered. > - No CVarDefNode in the tree after analyse_declarations Yeah, all those decelerator nodes really should go away after the types are figured out from them. > As for the code: > - Removed about 30 lines and added about 40. However I do feel that > the > transform code is less "copy-and-paste-with-modifications" and that > new > code is more obvious to beginners (Stefan can pull out his article URL > by now :-) > > I'll follow up with another post to discuss assignments in general. > > > -- > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Wed Jul 23 11:03:48 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 02:03:48 -0700 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <48833DF6.9080403@student.matnat.uio.no> References: <48833DF6.9080403@student.matnat.uio.no> Message-ID: <0F76A17E-5DCD-4F70-AE67-10D866C33913@math.washington.edu> We have talked about this already, but I think this is valuable for public record and input. On Jul 20, 2008, at 6:30 AM, Dag Sverre Seljebotn wrote: > I've had some skirmishes with Robert and Stefan on this and I think it > is time for a thread. > > Currently, for buffers, I must somehow write code for two different > cases that have to do with assignment: > 1. When buffers variables are somehow assigned to > 2. When buffers have items assigned to > > The question is whether to write more code in Nodes.py to do this, or > make it easier for transforms to react to assignments. The latter seem > more intrusive to how you usually do things, however I also see more > usecases than just buffers (especially for optimizing refcount in a > better way while keeping the algorithmic code simple, as well as for > parts of type inference, as well as compile-time evaluation, and just > overall transform complexity in number of node types to deal with). > > I'm not sure if I'm suggesting that there should only be one type of > assignment node, but I'd love for them to be a bit different: > > - SetItemNode > - SetAttributeNode > - SetCVarNode > > ...so that the node type starts corresponding to what is going on in C > without having to examine the lhs. I feel that would simplify a lot of > code, also existing code (and it could be made "backwards-compatible" > through inheritance anyway). However that is a second step that relies > on reducing to a single assignment node first (to avoid combinatorial > explosion). > > Current ways of assigning a variable x: > a) def f(x): > b) cdef type x = 2 > c) x = 2 > d) x, (y, z) = (2, (3, 4)) > e) x = y = 2 > f) except Exception, x > g) for x in C: And, (not fully supported yet) h) def x(...): > > That's all I can think of (with is turned into c) already). All of > these > cases must be covered for both 1 and 2 above. What I've done so far is > support a) and c) directly, and transformed b) into c) in the > parsing stage. > > What I *could* do now is move on and turn d) and e) into c) as well. (d) is more subtle then the example you give suggests, as the RHS could be an iterator (would you expand it to a loop) or a tuple (which is common and produces highly-optimized c). Of course it's possible that two could be expressed in tree form before outputting the C code. For (e) (and probably several of the others, one should keep in mind that using temp variables is better than creating (uniquely-named of course) local variables that last the whole scope. > The effect of all these things is to i) reduce the number of > different node > types, and ii) remove code from Nodes.py that requires you to know > about > how all code generation/analysis work and instead add code that > requires > you to know about writing tree transforms (I think that even if the > latter has usually more lines of code, they all follow a fundamental > principle and results in code that is more robust to changes other > places in the code, more "isolated" code). > > f) and g) can also be turned into c) but with a bit extra work and in > less obvious ways. If you think about SetItemNode, SetAttributeNode > etc. > you see that it isn't wholly unatural for a loop to reduce into such > instructions (that is of course what is happening today too, but less > explicit). I am curious if C compilers are better at optimizing/unrolling for loops (especially with static bounds) then while loops with an implicit counter... > All in all I'm unsure about this point -- I have to transform them > all, > or write BufferNodes to wrap namenodes in the tree anyway, in which > case > the transforms weren't really necesarry. So I ask for input. (Help me > counter the exhilirating feeling I get from having Buffer.py "just > work" > in more cases by refactoring unrelated pieces of code :-) ) > > From my own experiences, whenever I have to do something in > Nodes.py or > ExprNodes.py I invariably introduce bugs that it takes hours to find, > because I do not really understand it. Of course I could learn, but > I'd > rather write the transforms _if that is a preferred direction anyway_. > So the ideal preferred direction is what I'd like input on, and then > I'll take the final call on work amount myself (and on how smart > things > I can cook up for f) and g), if they end up being hopeless then I'll > drop it). I think transformations to simplify/consolidate some of the assignments is a good direction, but it seems things get increasingly messy and unnatural (for example with the exception catching). We are also potentially loosing information that could be used for optimization (which can be recovered, e.g. as you did for (b), but what about cdef object x cdef int a, b a = b = x # currently x gets converted twice, but one could imagine wanting to optimize so that it happens only once... I do, however, agree with you that using transformations can make for more "isolated" code, and a smaller set of Nodes to understand at the end. Switching gears a bit, what you state you (might) want is the nodes - SetItemNode - SetAttributeNode - SetCVarNode Now we already have - ItemNode - AttributeNode - NameNode And each of them comes with a analyse_target_types(), generate_assignment_code(), and generate_post_assignment_code() method. All types of assignments call these methods, so you only need to worry about putting the buffer code there, instead of worrying about all the current (or possibly future) ways of making an assignment. Another way to go is to treat it like a type test with side effects (so stuff like "(x)[0,0]" could possibly work), though ignore that if you're onto a working solution. - Robert From greg.ewing at canterbury.ac.nz Wed Jul 23 11:02:50 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 23 Jul 2008 21:02:50 +1200 Subject: [Cython] Assignments in cdefs In-Reply-To: References: <488334E8.7070601@student.matnat.uio.no> Message-ID: <4886F3BA.209@canterbury.ac.nz> Robert Bradshaw wrote: > Yeah, all those decelerator nodes really should go away You mean "declarator" nodes? I hope Cython doesn't really include nodes designed to slow your code down. :-) -- Greg From dagss at student.matnat.uio.no Wed Jul 23 15:04:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 23 Jul 2008 15:04:00 +0200 Subject: [Cython] Cython 0.9.8 released Message-ID: <3299670278.48254@smtp.netcom.no> Quick response: I'd really prefer it if you waited a week at least, preferably two, as some of my stuff was commited there and I therefore think a new merge of my branch should be done first (but that must wait). (This wasn't all bad though as it made it possible for Stefan to do decorators). This is exactly the kind of thing the "cython" tree should be used for IMO: backporting important bugfix-like patches to the current stable release and do a new stable patch release without having to feeeze devel. For instance, I found a probable bug in code used by the with transform yesterday (which I will patch when I have better net access). Dag Sverre Seljebotn -----Original Message----- From: Robert Bradshaw Date: Wednesday, Jul 23, 2008 9:37 am Subject: Re: [Cython] Cython 0.9.8 released To: Stefan Behnel CC: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Jul 21, 2008, at 12:22 AM, Stefan Behnel wrote: > >> Ondrej Certik wrote: >> On Sun, Jul 20, 2008 at 8:40 PM, Stefan Behnel wrote: >>> I'd be happy if you could include this patch. >>> >>> http://hg.cython.org/cython-devel/rev/e21391d5f23a >>> >>> It didn't make it into 0.9.8, but it makes Cython generate C code >>> that >>> compiles with Py3 (beta 1/2). >> >> I didn't put it in there yet. Why not to make a new upstream release? >> That way it will help a lot more users. > >> Yes, that would be the clean way. I was just thinking that since > you were > pushing a Debian package anyway, it'd be nice to have something that > people can use for a little longer. > >> Robert, could we have rev 719 as an intermediate 0.9.8.1? It would > really > help people who want to get their Cython code ready for Py3 (not > that I > heard anyone shouting, but I wouldn't want to be called a > bottleneck when > the code is already there). > >Sure, I'd be fine with that. It's been a while since we've done a >release, but the devel branch seems to have been a bit unstable >lately (more so than usual). Any thoughts on how far away it would be >to just release tip? > >- Robert > > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Wed Jul 23 16:07:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 23 Jul 2008 16:07:00 +0200 Subject: [Cython] Assignment nodes, buffers and transforms Message-ID: <3299674044.354273@smtp.netcom.no> Note (forgot where you touched upon it) that I consider the current temp handling in transforms a temporary solution - temps (also those written as cdef in my examples for notation) should be regular allocate_temps things. In fact I have something locally and a proposal for that which I will post when I get around to it (tomorrow). Robert Bradshaw wrote: > We have talked about this already, but I think this is valuable for >public record and input. > >On Jul 20, 2008, at 6:30 AM, Dag Sverre Seljebotn wrote: > >> I've had some skirmishes with Robert and Stefan on this and I think it > is time for a thread. > >> Currently, for buffers, I must somehow write code for two different > cases that have to do with assignment: > 1. When buffers variables are somehow assigned to > 2. When buffers have items assigned to > >> The question is whether to write more code in Nodes.py to do this, or > make it easier for transforms to react to assignments. The latter seem > more intrusive to how you usually do things, however I also see more > usecases than just buffers (especially for optimizing refcount in a > better way while keeping the algorithmic code simple, as well as for > parts of type inference, as well as compile-time evaluation, and just > overall transform complexity in number of node types to deal with). > >> I'm not sure if I'm suggesting that there should only be one type of > assignment node, but I'd love for them to be a bit different: > >> - SetItemNode > - SetAttributeNode > - SetCVarNode > >> ...so that the node type starts corresponding to what is going on in C > without having to examine the lhs. I feel that would simplify a lot of > code, also existing code (and it could be made "backwards-compatible" > through inheritance anyway). However that is a second step that relies > on reducing to a single assignment node first (to avoid combinatorial > explosion). > >> Current ways of assigning a variable x: > a) def f(x): > b) cdef type x = 2 > c) x = 2 > d) x, (y, z) = (2, (3, 4)) > e) x = y = 2 > f) except Exception, x > g) for x in C: > >And, (not fully supported yet) > >h) def x(...): > >> > That's all I can think of (with is turned into c) already). All of > these > cases must be covered for both 1 and 2 above. What I've done so far is > support a) and c) directly, and transformed b) into c) in the > parsing stage. > >> What I *could* do now is move on and turn d) and e) into c) as well. > >(d) is more subtle then the example you give suggests, as the RHS >could be an iterator (would you expand it to a loop) or a tuple >(which is common and produces highly-optimized c). Of course it's >possible that two could be expressed in tree form before outputting >the C code. > >For (e) (and probably several of the others, one should keep in mind >that using temp variables is better than creating (uniquely-named of >course) local variables that last the whole scope. > >> The effect of all these things is to i) reduce the number of > different node > types, and ii) remove code from Nodes.py that requires you to know > about > how all code generation/analysis work and instead add code that > requires > you to know about writing tree transforms (I think that even if the > latter has usually more lines of code, they all follow a fundamental > principle and results in code that is more robust to changes other > places in the code, more "isolated" code). > >> f) and g) can also be turned into c) but with a bit extra work and in > less obvious ways. If you think about SetItemNode, SetAttributeNode > etc. > you see that it isn't wholly unatural for a loop to reduce into such > instructions (that is of course what is happening today too, but less > explicit). > >I am curious if C compilers are better at optimizing/unrolling for >loops (especially with static bounds) then while loops with an >implicit counter... Yes, For*From*StatNode, which would be the target node for that kind of thing. So add that to the list as a seperate case... and it probably shouldn't turn into singleassignment. It doesn't include object assignments so tracking it is less crucial, a lot of algorithms could do without it. And it cannot assign to module scope etc either. >> All in all I'm unsure about this point -- I have to transform them > all, > or write BufferNodes to wrap namenodes in the tree anyway, in which > case > the transforms weren't really necesarry. So I ask for input. (Help me > counter the exhilirating feeling I get from having Buffer.py "just > work" > in more cases by refactoring unrelated pieces of code :-) ) > >> From my own experiences, whenever I have to do something in > Nodes.py or > ExprNodes.py I invariably introduce bugs that it takes hours to find, > because I do not really understand it. Of course I could learn, but > I'd > rather write the transforms _if that is a preferred direction anyway_. > So the ideal preferred direction is what I'd like input on, and then > I'll take the final call on work amount myself (and on how smart > things > I can cook up for f) and g), if they end up being hopeless then I'll > drop it). > >I think transformations to simplify/consolidate some of the >assignments is a good direction, but it seems things get increasingly >messy and unnatural (for example with the exception catching). We are Good point. >also potentially loosing information that could be used for >optimization (which can be recovered, e.g. as you did for (b), but >what about I prefer to think about it as "refined" rather than recovered :-) In general, with each transform you loose some and get some info, and each algorithm that needs the information must figure out where in the pipeline it wants to be. I didn't say these transforms would be done very early, just that I wanted buffers to be after them. You just have added flexibility rather than having to look at them as seperate concepts.. > >cdef object x >cdef int a, b >a = b = x # currently x gets converted twice, but one could imagine >wanting to optimize so that it happens only once... Two options: you either do it before/as part of the splitting up, as in: cdef int i, j x = i = j = somecall() turns into cdef object tmp1 cdef int tmp2 tmp1 = somecall() x = tmp1 tmp2 = tmp1 i = tmp2 j = tmp2 Option 2: you realize that your time is better spent at optimizing the general case that could arise if one wrote the thing using seperate assignments (it is still optimizeable in theory) and stick a (much more advanced) algorithm after the splitting up, where it can benefit more assignment types. > >I do, however, agree with you that using transformations can make for >more "isolated" code, and a smaller set of Nodes to understand at the >end. > > >Switching gears a bit, what you state you (might) want is the nodes > >- SetItemNode >- SetAttributeNode >- SetCVarNode > >Now we already have > >- ItemNode >- AttributeNode >- NameNode > >And each of them comes with a analyse_target_types(), >generate_assignment_code(), and generate_post_assignment_code() >method. All types of assignments call these methods, so you only need >to worry about putting the buffer code there, instead of worrying >about all the current (or possibly future) ways of making an >assignment. Yes, this is what I do now. (and as you know at this point this discussion is not for gsoc any longer) My point is still that you cannot so easily write a visitor to do general code analysis. Also, each of these function you mention often read like a lots of top-level if-tests to see which situation we are in (setting a module dict entry or a local var? And so on. So there is no correspondance in the tree with what is vastly different operations in CPython). Consider the old (obsolete?) idea of Cython-inlineable compile-time operator overloads. Then SetItem and GetItem could be transformed into method call nodes. And so on (of course, it is entirely possible to just add yet another if test and duplicate/call into some SimpleCallNode code instead). > Another way to go is to treat it like a type test with >side effects (so stuff like "(x)[0,0]" could possibly >work), though ignore that if you're onto a working solution. On a tangential note, this would mean that all the auxiliary vars and buffer acquisiton would happen again for each such cast, so I'd rather not support it because it should be avoided :-) which is why I've said that I think of this as acting on the variable rather than the type (though there is BufferType similar to TypedefType now). Dag Sverre From stefan_ml at behnel.de Wed Jul 23 17:00:20 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Jul 2008 17:00:20 +0200 (CEST) Subject: [Cython] Assignments in cdefs In-Reply-To: <4886F3BA.209@canterbury.ac.nz> References: <488334E8.7070601@student.matnat.uio.no> <4886F3BA.209@canterbury.ac.nz> Message-ID: <49730.213.61.181.86.1216825220.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Greg Ewing wrote: > Robert Bradshaw wrote: > >> Yeah, all those decelerator nodes really should go away > > You mean "declarator" nodes? > > I hope Cython doesn't really include nodes designed > to slow your code down. :-) We actually added those when we found out that the standard infinite while loop implemention in Cython terminated after 2 seconds on a C64. Now it takes 5! Stefan From stefan_ml at behnel.de Wed Jul 23 18:49:12 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Jul 2008 18:49:12 +0200 Subject: [Cython] Problem with multiple assignment In-Reply-To: <2A0F7DB4-77AE-462A-B59E-140511BC37A3@math.washington.edu> References: <3299567351.716273@smtp.netcom.no> <2A0F7DB4-77AE-462A-B59E-140511BC37A3@math.washington.edu> Message-ID: <48876108.5080002@behnel.de> Hi, Robert Bradshaw wrote: > I think a better > optimization is that (as has been hinted) True and False start as > bints, and then get coerced to PyObjects as needed....that was easy. Nice. I love it when good ideas turn out to be so low-hanging fruit that you can eat them without even whetting your hands. Stefan From stefan_ml at behnel.de Wed Jul 23 19:07:08 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Jul 2008 19:07:08 +0200 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <3299674044.354273@smtp.netcom.no> References: <3299674044.354273@smtp.netcom.no> Message-ID: <4887653C.5030309@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> cdef object x >> cdef int a, b >> a = b = x # currently x gets converted twice, but one could imagine >> wanting to optimize so that it happens only once... > > Two options: you either do it before/as part of the splitting up, as in: > cdef int i, j > x = i = j = somecall() > > turns into > cdef object tmp1 > cdef int tmp2 > tmp1 = somecall() > x = tmp1 > tmp2 = tmp1 > i = tmp2 > j = tmp2 > > Option 2: you realize that your time is better spent at optimizing the > general case that could arise if one wrote the thing using seperate > assignments (it is still optimizeable in theory) and stick a (much more > advanced) algorithm after the splitting up, where it can benefit more > assignment types. I disagree with option 2. A statement like a = b = xyz is very clearly asking for "xyz" being evaluated once, while a = xyz b = xyz is explicit about evaluating "xyz" twice (whatever xyz actually is). The code we currently generate for the first statement (at least in some cases) is surprising, while in the second case it matches my expectations. Also, I expect the logic required to determine that two statements assign the same thing to different names to be non-trivial. Why should the above be optimised, but not a = xyz c = zyx b = xyz As usual: hard to draw the line once you get started on the "general" case... Stefan From jwienke at techfak.uni-bielefeld.de Wed Jul 23 19:15:44 2008 From: jwienke at techfak.uni-bielefeld.de (Johannes Wienke) Date: Wed, 23 Jul 2008 19:15:44 +0200 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <4887653C.5030309@behnel.de> References: <3299674044.354273@smtp.netcom.no> <4887653C.5030309@behnel.de> Message-ID: <48876740.9030002@techfak.uni-bielefeld.de> Am 07/23/2008 07:07 PM schrieb Stefan Behnel: > I disagree with option 2. A statement like > > a = b = xyz > > is very clearly asking for "xyz" being evaluated once To my mind such a statement should be a syntax error. It's like a roulette game figuring out what's going on here and it isn't much more complicated to write this in two readable lines. Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080723/c09469dd/attachment.pgp From dagss at student.matnat.uio.no Wed Jul 23 19:23:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 23 Jul 2008 19:23:00 +0200 Subject: [Cython] Assignment nodes, buffers and transforms Message-ID: <3299685839.556157@smtp.netcom.no> Stefan Behnel wrote: >Hi, > >Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> cdef object x >> cdef int a, b >> a = b = x # currently x gets converted twice, but one could imagine >> wanting to optimize so that it happens only once... > > Two options: you either do it before/as part of the splitting up, as in: > cdef int i, j > x = i = j = somecall() > > turns into > cdef object tmp1 > cdef int tmp2 > tmp1 = somecall() > x = tmp1 > tmp2 = tmp1 > i = tmp2 > j = tmp2 > > Option 2: you realize that your time is better spent at optimizing the > general case that could arise if one wrote the thing using seperate > assignments (it is still optimizeable in theory) and stick a (much more > advanced) algorithm after the splitting up, where it can benefit more > assignment types. > >I disagree with option 2. A statement like > > a = b = xyz > >is very clearly asking for "xyz" being evaluated once, while > > a = xyz > b = xyz > >is explicit about evaluating "xyz" twice (whatever xyz actually is). The code we currently generate for the first statement (at least in some cases) is surprising, while in the second case it matches my expectations. > >Also, I expect the logic required to determine that two statements assign the same thing to different names to be non-trivial. Why should the above be optimised, but not > > a = xyz > c = zyx > b = xyz > :-) This was mostly to demonstrate how you would pick different places in the pipeline for different kinds of approaches (to counter the "loose information" argument) rather than an actual suggestion. (I realize there's a lot of examples, but when discussing this kind of grand overreaching design I think it is ok, these kind of scenarios help pushing the design and see how the design itself "performs"). (But I meant to (hypothetically) imply that your latter example would also be optimized.. I am assuming we are talking about calls to PyxFromLong etc. here, not anything with possible side-effects.) Dag Sverre From stefan_ml at behnel.de Wed Jul 23 19:30:37 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Jul 2008 19:30:37 +0200 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <48876740.9030002@techfak.uni-bielefeld.de> References: <3299674044.354273@smtp.netcom.no> <4887653C.5030309@behnel.de> <48876740.9030002@techfak.uni-bielefeld.de> Message-ID: <48876ABD.7090309@behnel.de> Hi, Johannes Wienke wrote: > Am 07/23/2008 07:07 PM schrieb Stefan Behnel: >> I disagree with option 2. A statement like >> >> a = b = xyz >> >> is very clearly asking for "xyz" being evaluated once > > To my mind such a statement should be a syntax error. It's like a > roulette game figuring out what's going on here and it isn't much more > complicated to write this in two readable lines. Wrong list, try python-ideas. (although I doubt you'll get much applause over there...) Stefan :) From jwienke at techfak.uni-bielefeld.de Wed Jul 23 19:37:30 2008 From: jwienke at techfak.uni-bielefeld.de (Johannes Wienke) Date: Wed, 23 Jul 2008 19:37:30 +0200 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <48876ABD.7090309@behnel.de> References: <3299674044.354273@smtp.netcom.no> <4887653C.5030309@behnel.de> <48876740.9030002@techfak.uni-bielefeld.de> <48876ABD.7090309@behnel.de> Message-ID: <48876C5A.8050707@techfak.uni-bielefeld.de> Am 07/23/2008 07:30 PM schrieb Stefan Behnel: > Johannes Wienke wrote: >> Am 07/23/2008 07:07 PM schrieb Stefan Behnel: >>> I disagree with option 2. A statement like >>> >>> a = b = xyz >>> >>> is very clearly asking for "xyz" being evaluated once >> To my mind such a statement should be a syntax error. It's like a >> roulette game figuring out what's going on here and it isn't much more >> complicated to write this in two readable lines. > > Wrong list, try python-ideas. > > (although I doubt you'll get much applause over there...) Was only my first thought while this discussion that could simply be avoided. ;) But if python has this feature, then there should already be a semantic that cython should implement? Johannes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/cython-dev/attachments/20080723/8bdd2afa/attachment.pgp From stefan_ml at behnel.de Wed Jul 23 20:23:34 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Jul 2008 20:23:34 +0200 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <48876C5A.8050707@techfak.uni-bielefeld.de> References: <3299674044.354273@smtp.netcom.no> <4887653C.5030309@behnel.de> <48876740.9030002@techfak.uni-bielefeld.de> <48876ABD.7090309@behnel.de> <48876C5A.8050707@techfak.uni-bielefeld.de> Message-ID: <48877726.1060505@behnel.de> Hi, Johannes Wienke wrote: > Am 07/23/2008 07:30 PM schrieb Stefan Behnel: >> Johannes Wienke wrote: >>> Am 07/23/2008 07:07 PM schrieb Stefan Behnel: >>>> I disagree with option 2. A statement like >>>> >>>> a = b = xyz >>>> >>>> is very clearly asking for "xyz" being evaluated once >>> To my mind such a statement should be a syntax error. It's like a >>> roulette game figuring out what's going on here and it isn't much more >>> complicated to write this in two readable lines. >> Wrong list, try python-ideas. >> >> (although I doubt you'll get much applause over there...) > > Was only my first thought while this discussion that could simply be > avoided. ;) But if python has this feature, then there should already be > a semantic that cython should implement? :) this isn't about semantics (which Python defines for us). It's about deciding which code should be generated out of the options that provide equivalent semantics (aka optimisation). In my above quote, "evaluated" does not refer to user code but to type conversions that Cython injects at the C level. Stefan From ellisonbg.net at gmail.com Wed Jul 23 23:26:39 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 23 Jul 2008 15:26:39 -0600 Subject: [Cython] Simple class hierarchy problems Message-ID: <6ce0ac130807231426p6a15118rcfacc9b968c9a232@mail.gmail.com> Hi, The following code is giving me problems: # foo.pyx cdef class Foo: cdef fooit(self): return 10 cdef class Bar(Foo): pass cdef class Bam(Bar): pass When I do "cython foo.pyx" and compile as c I get a warning: foo.c: In function '__pyx_tp_new_3foo_Bam': foo.c:608: warning: assignment from incompatible pointer type But, when I do "cython --cplus" and compile with c++ I get a failure: foo.cpp: In function 'PyObject* __pyx_tp_new_3foo_Bam(PyTypeObject*, PyObject*, PyObject*)': foo.cpp:608: error: cannot convert '__pyx_vtabstruct_3foo_Bar*' to '__pyx_vtabstruct_3foo_Foo*' in a I swear this code used to work fine. I am using Cython 0.9.8, but the problem is there if I try it in earlier versions. What am I missing? Thanks Brian From dagss at student.matnat.uio.no Wed Jul 23 23:59:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 23 Jul 2008 23:59:00 +0200 (CEST) Subject: [Cython] Temp framework proposal Message-ID: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> I'd like some feedback on this idea. The idea is simple, it is only the explanation that is complex. I am becoming very traditional in my code :-), but am making one non-traditional turn though, in how I use temps. I believe the following temp refactoring is a) conceptually simple and not a big break with current ways of doing things, b) can be made very incrementally (though there may be inconsistencies in "how things are done" along the way), c) is useful (at least to me). Seperating out the analyse_types phase in a seperate pipeline phase would have been an option; however this alternative proposal has the advantage of being more transform friendly -- allowing temps to be allocated and used at any time. Proposal (implementation cannot be uploaded until the end of week because of internet connection restriction but I have it locally): - There will be two temp systems in parallell for now. This means each function can have at worst twice the number of temps as the temp allocation algorithms would indicate, which I consider OK myself. (See below for how this can be remedied). - Temps can be retrieved as a typed Entry instance, using the following call: Symtab.new_temp(type). No releasing is needed, and it can be used as a regular Entry. *However*, it is required that each node registers the temp entries it is using in a list self.temps. (If you wondered, this requires that the temp currently stored in result_code for temp exprnodes would show up in the temps list of the _parent_. However we can add more conventions so we don't have to do this (like, a combination of self.is_temp and self.result_entry would count as a registration in parent(self).temps, all that is required is that the transform mentioned below can somehow understand what is going on.) - The "cname" on the temps is NOT available before code generation time. The idea is that you hold on to and pass around the entry until code generation phase and then look at cname. Neither are the temps available for lookup in any Scope, they merely exist by convention in the scope of the node using them (though if this really is a problem this could be fixed). - An "AnchorTemp" (need better name) transform is run right prior to code generation. This transform is responsible for figuring out what temps should actually be generated (it sets the cnames, registers the needed amount of temps in the scope etc.; currently it does this simply by wiping out free_temp_entries and using allocate/release_temps on the "node.temps" attributes, very simple code). Also (importantly), it modifies TryExceptNode and friends to make them clean up these temps. - An immediate consequence is that except clauses get an exact list of temps to decref (within the set of temps using the new system) rather than the current shotgun approach which is suboptimal (as if it mattered...) if you have a deep nested expression right prior to a try block with only simple expressions (and to get it to the theoretical minimum one could start decrefing the exact used temps on the raise statements rather than in the except/finally clause...leaving that for now though). The current implementation is already useful to me, however it cannot be used in most cases as a lot of current code store temps in result_code at analysis time (at which point cname is not available). This can be fixed either by refactoring or by the following hack: Let cname be easily recognizeable magic strings ("$!idx!$", and have Code.py run a substitution on all input strings, when the cname is available (or is this too ugly? It would bring closures closer as well I think, and I think it is ok if we agree that result_code should be refactored at some later point if one gets the time and resources; i.e. one stores "result_entry" and other entry references instead at analysis time and only calculate result_code at code generation time). Given such a hack or refactor, this temp system can easily be used by TreeFragment so that WithStatementTransform automatically stop using non-temp local variables. (This has been in my drawer for some time, I am focusing on GSoC now, but I thought I should get it out there now that it was mentioned.) Dag Sverre From cswiercz at gmail.com Thu Jul 24 00:48:05 2008 From: cswiercz at gmail.com (Chris Swierczewski) Date: Wed, 23 Jul 2008 15:48:05 -0700 Subject: [Cython] Wrapping Virtual C++ Class Functions Message-ID: Hello, The Situation: I created an instance of a C++ class which contains several virtual functions. The important thing going on here is that the class contains function combinations looking like the following: void callEvent() virtual void onEvent(object eventData) where callEvent essentially does some stuff and stores data in the eventData object which is then passed to the onEvent(). In fact, onEvent() is _called_ by the class upon the successful completion of callEvent(). The point is that when you overload onEvent(), you write your own instructions for what to do with eventData. The function itself in the class contains no actual code aside from what you overload. Question: How do I wrap this intricate situation in Cython? It needs to be something more than just. ------ .pxd ------ cdef extern from "the_class.h": ctypedef struct the_class: void callEvent() void onEvent(object eventData) cdef class MYWrapper: cdef the_class wrapped_class ------ .pyx ------ cdef class MyWrapper: def my_callEvent(self): self.wrapped_class.callEvent() def my_onEvent(self, object eventData): self.wrapped_class.onEvent(eventData) since the main problem is in handling the wrapped_class.onEvent() situation. I don't need to call onEvent() since the completion of callEvent() does so for me. I just want to overload it, I suppose, and have my code be executed upon the calling of on Event(). Any hints as to what to do? I appreciate any help. Thanks in advance. -- Chris Swierczewski cswiercz at gmail.com mobile: 253 2233721 From greg.ewing at canterbury.ac.nz Thu Jul 24 03:30:20 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 24 Jul 2008 13:30:20 +1200 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <4887653C.5030309@behnel.de> References: <3299674044.354273@smtp.netcom.no> <4887653C.5030309@behnel.de> Message-ID: <4887DB2C.5060902@canterbury.ac.nz> Stefan Behnel wrote: > As usual: hard to draw the line once you get started on the "general" case... Yes, this is really getting into the realm of common subexpression elimination, which can only be done safely if you can be sure the expression has no side effects and its result isn't affected by anything that happens in the meantime. Sounds very hard indeed to me. On the other hand, sorting the multiple assignment targets into groups depending on what conversion needs to be done probably wouldn't be too hard. -- Greg From greg.ewing at canterbury.ac.nz Thu Jul 24 03:44:31 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 24 Jul 2008 13:44:31 +1200 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <48877726.1060505@behnel.de> References: <3299674044.354273@smtp.netcom.no> <4887653C.5030309@behnel.de> <48876740.9030002@techfak.uni-bielefeld.de> <48876ABD.7090309@behnel.de> <48876C5A.8050707@techfak.uni-bielefeld.de> <48877726.1060505@behnel.de> Message-ID: <4887DE7F.4030203@canterbury.ac.nz> Stefan Behnel wrote: > In my above quote, "evaluated" does > not refer to user code but to type conversions that Cython injects at the C level. Indeed, and there's no equivalent to that in the Python case, so Python gives no guidance as to the semantics. The only case where it could make a difference to the result is if the conversion itself has a side effect, which could happen if, e.g., an object has a user defined __str__ method (although it would be a very odd thing for a __str__ method to do). In that case, the user will probably expect the side effect to only happen once, although I'd be reluctant to promise that. -- Greg From greg.ewing at canterbury.ac.nz Thu Jul 24 05:48:05 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 24 Jul 2008 15:48:05 +1200 Subject: [Cython] Wrapping Virtual C++ Class Functions In-Reply-To: References: Message-ID: <4887FB75.3040809@canterbury.ac.nz> Chris Swierczewski wrote: > void callEvent() > virtual void onEvent(object eventData) > Question: How do I wrap this intricate situation in Cython? You may need to create a C++ subclass of the class to be wrapped, with an onEvent() implementation that calls a C function, passing it "this" as a parameter. Then implement that C function in the .pyx file (declaring it "public" so that it can be called from the C++ code) so that it locates the Python wrapper object and calls a Python method of it. To map the C++ object back to the Python wrapper, you could cast a reference to the wrapper to void * and store it in a member of the C++ object (this effectively makes it a borrowed reference, so that it won't keep the wrapper object alive). -- Greg From robertwb at math.washington.edu Thu Jul 24 06:38:04 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 21:38:04 -0700 Subject: [Cython] Assignment nodes, buffers and transforms In-Reply-To: <3299674044.354273@smtp.netcom.no> References: <3299674044.354273@smtp.netcom.no> Message-ID: <8C51908F-8541-4234-B276-D96D3F524DF2@math.washington.edu> On Jul 23, 2008, at 7:07 AM, Dag Sverre Seljebotn wrote: > Note (forgot where you touched upon it) that I consider the current > temp handling in transforms a temporary solution - temps (also > those written as cdef in my examples for notation) should be > regular allocate_temps things. OK, great you've already been thinking about this. > In fact I have something locally and a proposal for that which I > will post when I get around to it (tomorrow). >>> f) and g) can also be turned into c) but with a bit extra work >>> and in >> less obvious ways. If you think about SetItemNode, SetAttributeNode >> etc. >> you see that it isn't wholly unatural for a loop to reduce into such >> instructions (that is of course what is happening today too, but less >> explicit). >> >> I am curious if C compilers are better at optimizing/unrolling for >> loops (especially with static bounds) then while loops with an >> implicit counter... > > Yes, For*From*StatNode, which would be the target node for that > kind of thing. So add that to the list as a seperate case... and it > probably shouldn't turn into singleassignment. Yep. > It doesn't include object assignments so tracking it is less > crucial, a lot of algorithms could do without it. And it cannot > assign to module scope etc either. It may, and it can assign to module scope too. > >>> All in all I'm unsure about this point -- I have to transform them >> all, >> or write BufferNodes to wrap namenodes in the tree anyway, in which >> case >> the transforms weren't really necesarry. So I ask for input. (Help me >> counter the exhilirating feeling I get from having Buffer.py "just >> work" >> in more cases by refactoring unrelated pieces of code :-) ) >> >>> From my own experiences, whenever I have to do something in >> Nodes.py or >> ExprNodes.py I invariably introduce bugs that it takes hours to find, >> because I do not really understand it. Of course I could learn, but >> I'd >> rather write the transforms _if that is a preferred direction >> anyway_. >> So the ideal preferred direction is what I'd like input on, and then >> I'll take the final call on work amount myself (and on how smart >> things >> I can cook up for f) and g), if they end up being hopeless then I'll >> drop it). >> >> I think transformations to simplify/consolidate some of the >> assignments is a good direction, but it seems things get increasingly >> messy and unnatural (for example with the exception catching). We are > > Good point. > >> also potentially loosing information that could be used for >> optimization (which can be recovered, e.g. as you did for (b), but >> what about > > I prefer to think about it as "refined" rather than recovered :-) > > In general, with each transform you loose some and get some info, > and each algorithm that needs the information must figure out where > in the pipeline it wants to be. I didn't say these transforms would > be done very early, just that I wanted buffers to be after them. > You just have added flexibility rather than having to look at them > as seperate concepts.. >> Switching gears a bit, what you state you (might) want is the nodes >> >> - SetItemNode >> - SetAttributeNode >> - SetCVarNode >> >> Now we already have >> >> - ItemNode >> - AttributeNode >> - NameNode >> >> And each of them comes with a analyse_target_types(), >> generate_assignment_code(), and generate_post_assignment_code() >> method. All types of assignments call these methods, so you only need >> to worry about putting the buffer code there, instead of worrying >> about all the current (or possibly future) ways of making an >> assignment. > > Yes, this is what I do now. (and as you know at this point this > discussion is not for gsoc any longer) Yep, but this GSoC or not it brings up a lot of interesting points. :-) > My point is still that you cannot so easily write a visitor to do > general code analysis. Also, each of these function you mention > often read like a lots of top-level if-tests to see which situation > we are in (setting a module dict entry or a local var? And so on. > So there is no correspondance in the tree with what is vastly > different operations in CPython). > > Consider the old (obsolete?) idea of Cython-inlineable compile-time > operator overloads. Then SetItem and GetItem could be transformed > into method call nodes. And so on (of course, it is entirely > possible to just add yet another if test and duplicate/call into > some SimpleCallNode code instead). Yes, this idea certainly has merits. >> Another way to go is to treat it like a type test with >> side effects (so stuff like "(x)[0,0]" could possibly >> work), though ignore that if you're onto a working solution. > > On a tangential note, this would mean that all the auxiliary vars > and buffer acquisiton would happen again for each such cast, so I'd > rather not support it because it should be avoided :-) which is why > I've said that I think of this as acting on the variable rather > than the type (though there is BufferType similar to TypedefType > now). True, but it could still be cheaper than going through Python and more versatile than creating a variable of the right type, sticking it in, then clearing the variable (though harder to implement too). - Robert From robertwb at math.washington.edu Thu Jul 24 06:42:51 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 21:42:51 -0700 Subject: [Cython] Wrapping Virtual C++ Class Functions In-Reply-To: <4887FB75.3040809@canterbury.ac.nz> References: <4887FB75.3040809@canterbury.ac.nz> Message-ID: On Jul 23, 2008, at 8:48 PM, Greg Ewing wrote: > Chris Swierczewski wrote: > >> void callEvent() >> virtual void onEvent(object eventData) > >> Question: How do I wrap this intricate situation in Cython? > > You may need to create a C++ subclass of the class to > be wrapped, with an onEvent() implementation that calls > a C function, passing it "this" as a parameter. > > Then implement that C function in the .pyx file > (declaring it "public" so that it can be called from > the C++ code) so that it locates the Python wrapper > object and calls a Python method of it. > > To map the C++ object back to the Python wrapper, you > could cast a reference to the wrapper to void * and > store it in a member of the C++ object (this effectively > makes it a borrowed reference, so that it won't keep > the wrapper object alive). Yep, I agree. Another related option would be to have the C++ subclass take a function pointer and a void* (which, again, may hold a pointer to the Python object) which would get filled in in Cython. - Robert From robertwb at math.washington.edu Thu Jul 24 07:05:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Jul 2008 22:05:42 -0700 Subject: [Cython] Temp framework proposal In-Reply-To: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> Message-ID: On Jul 23, 2008, at 2:59 PM, Dag Sverre Seljebotn wrote: > I'd like some feedback on this idea. Sure. > The idea is simple, it is only the explanation that is complex. > > I am becoming very traditional in my code :-), but am making one > non-traditional turn though, in how I use temps. I believe the > following > temp refactoring is a) conceptually simple and not a big break with > current ways of doing things, b) can be made very incrementally > (though > there may be inconsistencies in "how things are done" along the > way), c) > is useful (at least to me). > > Seperating out the analyse_types phase in a seperate pipeline phase > would > have been an option; however this alternative proposal has the > advantage > of being more transform friendly -- allowing temps to be allocated and > used at any time. > > Proposal (implementation cannot be uploaded until the end of week > because > of internet connection restriction but I have it locally): > > - There will be two temp systems in parallell for now. This means each > function can have at worst twice the number of temps as the temp > allocation algorithms would indicate, which I consider OK myself. (See > below for how this can be remedied). I'm fine for a transitional overlap. Sure beats trying to rip out the old one and fix everything at once :-). > - Temps can be retrieved as a typed Entry instance, using the > following > call: Symtab.new_temp(type). No releasing is needed, and it can be > used as > a regular Entry. *However*, it is required that each node registers > the > temp entries it is using in a list self.temps. (If you wondered, this > requires that the temp currently stored in result_code for temp > exprnodes > would show up in the temps list of the _parent_. However we can add > more > conventions so we don't have to do this (like, a combination of > self.is_temp and self.result_entry would count as a registration in > parent(self).temps, all that is required is that the transform > mentioned > below can somehow understand what is going on.) This is starting to sound a little confusing (not that the current system is totally straightforward, and I really don't like the fact that we're just passing strings around). Also, in general I'd rather have things worry about going down the tree instead of up. > - The "cname" on the temps is NOT available before code generation > time. > The idea is that you hold on to and pass around the entry until code > generation phase and then look at cname. Neither are the temps > available > for lookup in any Scope, they merely exist by convention in the > scope of > the node using them (though if this really is a problem this could be > fixed). > > - An "AnchorTemp" (need better name) transform is run right prior > to code > generation. This transform is responsible for figuring out what temps > should actually be generated (it sets the cnames, registers the needed > amount of temps in the scope etc.; currently it does this simply by > wiping > out free_temp_entries and using allocate/release_temps on the > "node.temps" > attributes, very simple code). Also (importantly), it modifies > TryExceptNode and friends to make them clean up these temps. > > - An immediate consequence is that except clauses get an exact list of > temps to decref (within the set of temps using the new system) > rather than > the current shotgun approach which is suboptimal (as if it > mattered...) Right now it's pretty tight, it only tries to decref the temps that could have been set in the block, depending on where the error is. > if you have a deep nested expression right prior to a try block > with only > simple expressions (and to get it to the theoretical minimum one could > start decrefing the exact used temps on the raise statements rather > than > in the except/finally clause...leaving that for now though). This idea has been brought up before. Trying to dynamically keep track of what temps need to be cleared isn't going to be faster than using xdecref, and putting specialized special code under each raise will do really bad things to code size (an important consideration-- and its much better to optimized the non-error path). > The current implementation is already useful to me, however it > cannot be > used in most cases as a lot of current code store temps in > result_code at > analysis time (at which point cname is not available). This can be > fixed > either by refactoring or by the following hack: Let cname be easily > recognizeable magic strings ("$!idx!$", and have Code.py run a > substitution on all input strings, when the cname is available (or > is this > too ugly? IMHO, too ugly. > It would bring closures closer as well I think, and I think it > is ok if we agree that result_code should be refactored at some later > point if one gets the time and resources; i.e. one stores > "result_entry" > and other entry references instead at analysis time and only calculate > result_code at code generation time). > > Given such a hack or refactor, this temp system can easily be used by > TreeFragment so that WithStatementTransform automatically stop using > non-temp local variables. > > (This has been in my drawer for some time, I am focusing on GSoC > now, but > I thought I should get it out there now that it was mentioned.) Sure. You've probably thought about this, but I'm not seeing in your proposal how you're planing on handling deep expressions. - Robert From greg.ewing at canterbury.ac.nz Thu Jul 24 08:40:48 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 24 Jul 2008 18:40:48 +1200 Subject: [Cython] Temp framework proposal In-Reply-To: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> Message-ID: <488823F0.4040301@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > I believe the following > temp refactoring is a) conceptually simple and not a big break with > current ways of doing things, b) can be made very incrementally (though > there may be inconsistencies in "how things are done" along the way), c) > is useful (at least to me). I don't think it should be necessary to go to this much upheaval in order to make temps easier to use. All you should really need is some suitable node types to use as building blocks during your tree transformation phases. I'm thinking of something like the following: * LocalNode - an ExprNode representing a temporary local variable that is allocated using the temps mechanism. * LetNode - a StatNode that holds a list of LocalNodes and a StatListNode. It functions like a "let" statement in Lisp, i.e. binds some local variables, executes the body and then disposes of the locals. The code for the LetNode would work in a similar way to the existing code in the for-statement node and other places that use temporary locals, except that instead of a fixed set of locals, it would do it for whatever was in its list of LocalNodes. In the body of the LetNode, wherever you wanted to refer to one of the locals, you would insert a CloneNode (an already-existing node type) that references the relevant LocalNode. Then the existing machinery should do the right thing. Not knowing the result_code won't be an issue, because you're not dealing with things at that level, you're just plugging nodes together. > I think it > is ok if we agree that result_code should be refactored at some later > point if one gets the time and resources; i.e. one stores "result_entry" > and other entry references instead at analysis time and only calculate > result_code at code generation time). You really ought to look at how this is in the current Pyrex before doing anything about this. The result code is now retrieved by calling result_as(), or something else that ends up calling it, during code generation. So there's no longer a requirement to calculate the result code at analysis time -- it can be constructed on the fly at code generation time if need be. This means it would, I think, be fairly straightforward now to merge all the temp allocation/release stuff into the code generation phase, although I haven't looked in detail at what would be required. -- Greg From dagss at student.matnat.uio.no Thu Jul 24 12:35:53 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 24 Jul 2008 12:35:53 +0200 (CEST) Subject: [Cython] Temp framework proposal In-Reply-To: References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> Message-ID: <34139.85.166.116.245.1216895753.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Jul 23, 2008, at 2:59 PM, Dag Sverre Seljebotn wrote: > >> - Temps can be retrieved as a typed Entry instance, using the >> following >> call: Symtab.new_temp(type). No releasing is needed, and it can be >> used as >> a regular Entry. *However*, it is required that each node registers >> the >> temp entries it is using in a list self.temps. (If you wondered, this >> requires that the temp currently stored in result_code for temp >> exprnodes >> would show up in the temps list of the _parent_. However we can add >> more >> conventions so we don't have to do this (like, a combination of >> self.is_temp and self.result_entry would count as a registration in >> parent(self).temps, all that is required is that the transform >> mentioned >> below can somehow understand what is going on.) > > This is starting to sound a little confusing (not that the current > system is totally straightforward, and I really don't like the fact > that we're just passing strings around). Also, in general I'd rather > have things worry about going down the tree instead of up. Yes...this was supposed to be the nested expressions explanation you sought... I'll give an example (and start again, so strike the above): print A * B where A and B are further nested expressions (all Python types). Then the MulNode would require two distinct PyObject temps to do its stuff, since B can't very well go on to use the temps that A stored its result in, but it can use any other temps that A used as part of a temporary calculation. Something like this would be the simplest: - self.temps contains the temps that occure in the calculation that single node performs. During resolving, these temps are locked so that children nodes cannot used them, but unlocked so that sibling nodes can use them. - This requires that the caller node allocates the temps the children need for their result. However it could be tricky for the caller to do this last step. So we can do something like this: - self.temps: Temps needed to do "my own" calculations. Temps here are allocated for the subtree that self is the root of. - self.result_temp: A temp I allocated to store my result in, which I give up and which the parent is going to use. These temps are allocated for the subtree which the parent is the root of. However, if we can refactor stuff so that the second line here isn't needed, great! Anyway how we do it, the important core principle is the following: - Temps should be inspectable/modifyable data, rather than acquired through call-chain-based calls; in order to make transforms using temps feel very natural and in order to allow "code analysis algorithms" to inspect and juggle with the temps. As long as we get from "imperative" to "declarative" temp programming, I don't care particularily much which conventions for declaring them are chosen. >> - An immediate consequence is that except clauses get an exact list of >> temps to decref (within the set of temps using the new system) >> rather than >> the current shotgun approach which is suboptimal (as if it >> mattered...) > > Right now it's pretty tight, it only tries to decref the temps that > could have been set in the block, depending on where the error is. No, this is wrong, unless you improved this recently. Try def f(): x = 4 x = (x ** (x * (x - (x - 2))) try: print x except: pass The point is, the except clause takes all free temps after the temp analysis of the body, which might well come from code before it. (Of course this is possible to fix within the existing framework so this is not important.) Dag Sverre From dagss at student.matnat.uio.no Thu Jul 24 15:40:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 24 Jul 2008 15:40:30 +0200 (CEST) Subject: [Cython] Temp framework proposal In-Reply-To: <488823F0.4040301@canterbury.ac.nz> References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> <488823F0.4040301@canterbury.ac.nz> Message-ID: <33086.85.166.116.245.1216906830.squirrel@webmail.uio.no> I was thinking in very similary ways to what you suggest in order to arrive at this proposal, so this ought to be interesting (and it is making me see some weaknesses in my proposal too). > Dag Sverre Seljebotn wrote: >> I believe the following >> temp refactoring is a) conceptually simple and not a big break with >> current ways of doing things, b) can be made very incrementally (though >> there may be inconsistencies in "how things are done" along the way), c) >> is useful (at least to me). > > I don't think it should be necessary to go to this > much upheaval in order to make temps easier to use. > All you should really need is some suitable node types > to use as building blocks during your tree transformation > phases. > > I'm thinking of something like the following: > > * LocalNode - an ExprNode representing a temporary local > variable that is allocated using the temps mechanism. > > * LetNode - a StatNode that holds a list of LocalNodes > and a StatListNode. It functions like a "let" statement > in Lisp, i.e. binds some local variables, executes the > body and then disposes of the locals. I was thinking about LetNode too, which I think is definitely needed -- but I think it is orthogonal as such; the issue is that we are discussing how such a LetNode (and everything else using temps) should be implemented. Because LetNodes can be introduced after what is currently the allocate_temps phase, the allocate_temps phase must then be moved (if we keep it; I'm basically saying "turn it into a pre-generation transform). But then I have a feeling that a lot of code will follow this pattern: - During analysis/transformation, you know "what you are doing" and how many temps you will need. - Saved this information for later in various ways. - Read this information and allocate temps in the allocate_temps phase. This seems like a reusable pattern that could eliminated with another design. LetNode would be one particular instance of this pattern; with your suggestion it would have "custom" (non-reusable, in one sense of the word) code to remember which temps it should allocate; then other custom nodes have custom code again to pull them out. In a similar way, ExprNode would need to remember that it should allocate its result as a temporary, and so on. This "duplication" won't disappear overnight with my approach, but I think it opens up for going away from it in another way. I am even so still considering your approach! -- just noting my arguments against it. A big weakness with my approach is this: Consider a node that does something like the following: process the A node output calculation code involving a temporary process the B node My proposal would then assume that the temp is needed for the whole thing. So perhaps an imperative approach is right afterall. Even though it is always possible to create really complicated (but *optional*) declarations (like "keep these temps during A and these during B and these for local processing and these for the result"; but imperative tend to express this more nicely). An imperative approach (still having methods in each node calling allocate/release) for "fixing/anchoring" the entries, but still being able to get hold of entries rather than strings (at a stage earlier than the anchoring occurs) might be a compromise. > The code for the LetNode would work in a similar way to > the existing code in the for-statement node and other > places that use temporary locals, except that instead of > a fixed set of locals, it would do it for whatever was in > its list of LocalNodes. > > In the body of the LetNode, wherever you wanted to refer > to one of the locals, you would insert a CloneNode (an > already-existing node type) that references the relevant > LocalNode. Entry seems like a more natural concept for this to me -- because it is a more "refined", more basic concept of a handle to a variable in the scope. This could then be used in many situations. For this case, I would use NameNode. It already accesses lots of different symbols; I have looked into making some trivial changes to NameNode so that if you provided node.entry on construction it wouldn't go looking for its node.name, so you basically do this to, say, unpack a cascaded assignment (node is the cascaded assignment, and this is psuedocodes, the details are not right): def visit_CascadedAssignmentNode(self, node): let = LetNode(temp_types=[node.rhs.type]) tmp = let.entries[0] let.body = StatementListNode(stats=[ SingleAssignmentNode(lhs=NameNode(entry=tmp), rhs=node.rhs) ] + [ SingleAssignmentNode(lhs=lhs, rhs=NameNode(tmp)) for lhs in node.lhses ]) return let Note that one reason for this is that if we start transforming into more basic operations (consider this hypothethical!), then one thing one might want to do is unrolling nested expressions. So you have a transform that basically unpacks every nested expression into assignments to temporaries (this might be a good pre-step to some code analysis-algorithms that would benefit from being able to insert if-statements in the middle of nested expressions for instance).. For the "primitive" statements that are left I would then consider it unnatural that NameNode, CloneNode and so on where exceptions to the non-nesting rule, rather each statement would contain references to entries rather than expressions. Again, this was just designed to push your approach a bit. Getting something that >> I think it >> is ok if we agree that result_code should be refactored at some later >> point if one gets the time and resources; i.e. one stores "result_entry" >> and other entry references instead at analysis time and only calculate >> result_code at code generation time). > > You really ought to look at how this is in the current > Pyrex before doing anything about this. The result code > is now retrieved by calling result_as(), or something else > that ends up calling it, during code generation. So there's > no longer a requirement to calculate the result code at > analysis time -- it can be constructed on the fly at code > generation time if need be. True, but you then need to remember what to generate, as I said above! But I will definitely not touch result_code without looking at what you have done. Do you have any "commits" or similar that would contain these changes relatively self-contained, or would the best thing to do be taking a diff between different Pyrex versions? > This means it would, I think, be fairly straightforward > now to merge all the temp allocation/release stuff into > the code generation phase, although I haven't looked in > detail at what would be required. Unless you cache the function bodies it would need to be done right prior, so that you know which local variables to declare. If it weren't for that, imperative temp allocation interleaved with code.put-statements wouldn't be that bad... Dag Sverre From greg.ewing at canterbury.ac.nz Fri Jul 25 06:54:38 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 25 Jul 2008 16:54:38 +1200 Subject: [Cython] Temp framework proposal In-Reply-To: <33086.85.166.116.245.1216906830.squirrel@webmail.uio.no> References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> <488823F0.4040301@canterbury.ac.nz> <33086.85.166.116.245.1216906830.squirrel@webmail.uio.no> Message-ID: <48895C8E.4010309@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > Because LetNodes can be introduced after what is currently the > allocate_temps phase, the allocate_temps phase must then be moved (if we > keep it; I'm basically saying "turn it into a pre-generation transform). One of the undesirable things about the current scheme is that there is a subtle coupling between the temp allocation and code generation phase. Not only is it a source of error when modifying things, it also leads to some bizarre-looking code in the allocate_temps phase, such as allocating a temp and then immediately releasing it, which looks redundant, but is actually necessary. Simply moving the allocate_temps phase to a different place wouldn't do anything to remove the dependency. If it's folded into the code generation phase, however, the general pattern would become 1. Generate code to evaluate subexpressions. 2. Allocate temp for the result of this node if needed. 3. Generate code to calculate result based on results of subexpressions. 4. Release temps holding results of subexpressions. which actually makes sense. Note that for most expression nodes, most of this pattern could be implemented by a general method in ExprNode which calls a node-specific method to implement (3) (as is done now, except it's split between the temp allocation and code generation phases). > - During analysis/transformation, you know "what you are doing" and how > many temps you will need. I'm not convinced you really need to know about temps during analysis. To my mind, temps are analogous to registers in a conventional compiler -- something very low-level that you only deal with late in the pipeline. Tree transformations should operate at a higher level of abstraction. The abstraction I would choose is "extra local variable", which is what the LocalNode represents. Whether it uses the same mechanism as that for the intermediate results of expressions is an implementation detail, as is whether it makes use of an Entry in the symbol table. > LetNode would be one particular instance of this pattern; with > your suggestion it would have "custom" (non-reusable, in one sense of the > word) code to remember which temps it should allocate; Not sure what you mean by that. The way you re-use it is by using a LetNode wherever you want extra locals. I'm expecting that various things such as for-loops will be implementable using LetNodes with suitable internal plumbing, so the custom nodes for these statements would disappear, along with the custom code in them implementing the temp allocation and code generation patterns. That's an increase in re-use, to my way of thinking. > In a similar way, ExprNode > would need to remember that it should allocate its result as a temporary, > and so on. Still not following. If temp allocation is folded into code generation, I don't think it will be necessary for a node to remember when it's using a temp across passes -- it can be figured out at the point where that information is needed (i.e. at (2) above). Some nodes might still want to do so, since it usually depends on things that are figured out during type analysis, such as whether the result is a Python reference. But this would be a private matter for each node to decide. I don't see why any other node should need to know during tree transformations. > Entry seems like a more natural concept for this to me -- because it is a > more "refined", more basic concept of a handle to a variable in the scope. I don't see what you gain by doing this. Under my proposal, there would only be one kind of thing for tree transformations to deal with, i.e. Nodes. Under yours, there would be two kinds of things -- Nodes and Entries. Also, I think it unnecessarily limits the strategy for implementing temps. The only reason the temp list is kept in the symbol table at the moment is because that's the only piece of state being passed around during the phase when temp allocation is being done. If I move temp allocation into the code generation phase, instead of the symbol table, the object being passed around is the code object. So I would move the temp list from the symbol table to the code object, at which point involving an Entry makes little sense. > one thing one might > want to do is unrolling nested expressions. So you have a transform that > basically unpacks every nested expression into assignments to temporaries I'm not sure this would be a good thing to do. Currently, temps are only used when they're really needed -- most expressions involving non-Python results just turn into an equivalent nested C expression in the generated code. Putting every result into a temp would lead to a much greater use of temps, most of them unnecessary. The C compiler *might* be able to optimise them away, but I wouldn't like to rely on it. I'd worry about many things that would normally be kept in registers spilling over into local variables. > (this might be a good pre-step to some code analysis-algorithms that would > benefit from being able to insert if-statements in the middle of nested > expressions for instance) I'd address that by adding a conditional expression node. More generally, there might be benefit in removing or reducing the distinction between statement and expression nodes. Instead of trying to turn all expressions into statements, try to turn all statements into expressions. Part of this might involve implementing if-elsif-else statements as a chain of conditional expression nodes. > Do you have any "commits" or similar that would contain these > changes relatively self-contained, I can't remember exactly; have a look at the mercurial commit comments. I've been trying to keep the changes lined up with commits, but I don't always completely succeed. > Unless you cache the function bodies it would need to be done right prior, > so that you know which local variables to declare. I would split the code stream for the function body in two, one for the declarations and one for the statements. That wouldn't be hard to do -- I'm already doing something similar with the module-level declarations (the 'h' attribute of the code object refers to another code object that collects the declarations in a StringIO). -- Greg From dagss at student.matnat.uio.no Fri Jul 25 10:16:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 25 Jul 2008 10:16:04 +0200 (CEST) Subject: [Cython] Temp framework proposal In-Reply-To: <48895C8E.4010309@canterbury.ac.nz> References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> <488823F0.4040301@canterbury.ac.nz> <33086.85.166.116.245.1216906830.squirrel@webmail.uio.no> <48895C8E.4010309@canterbury.ac.nz> Message-ID: <33322.85.166.116.245.1216973764.squirrel@webmail.uio.no> Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> Because LetNodes can be introduced after what is currently the >> allocate_temps phase, the allocate_temps phase must then be moved (if we >> keep it; I'm basically saying "turn it into a pre-generation transform). > > One of the undesirable things about the current scheme is that > there is a subtle coupling between the temp allocation and > code generation phase. Not only is it a source of error when > modifying things, it also leads to some bizarre-looking > code in the allocate_temps phase, such as allocating a temp > and then immediately releasing it, which looks redundant, > but is actually necessary. > > Simply moving the allocate_temps phase to a different place > wouldn't do anything to remove the dependency. If it's folded > into the code generation phase, however, the general patter Yes, this is what I was worried about (and explained rather poorly, and I made some wrong assumptions about your proposal as well, so I won't attempt to clean up the parts where I was unclear). > would become > > 1. Generate code to evaluate subexpressions. > 2. Allocate temp for the result of this node if needed. > 3. Generate code to calculate result based on results > of subexpressions. > 4. Release temps holding results of subexpressions. > > which actually makes sense. 5. Dispose result temp when parent calls generate_disposal_code? (For the record and checking of understanding). Great!, I'm +1 now -- integrating temp allocation with C code generation makes it very easy to get the most optimal temp usage possible. Also, this can also be implemented without affecting existing code -- simply have Code.py (and FuncDefNode) support these new temps in addition (with another prefix perhaps), then move on to slowly migrating existing code as-needed. And once that is done it is much easier to move the result_code generation of temporary exprnodes. (Note: Yes, the nested subexpression unrolling was a particularily silly one. And it definitely should not happen for C code.) > Note that for most expression nodes, most of this pattern > could be implemented by a general method in ExprNode which > calls a node-specific method to implement (3) (as is done now, > except it's split between the temp allocation and code > generation phases). > >> - During analysis/transformation, you know "what you are doing" and how >> many temps you will need. > > I'm not convinced you really need to know about temps during > analysis. To my mind, temps are analogous to registers in a > conventional compiler -- something very low-level that you > only deal with late in the pipeline. Tree transformations > should operate at a higher level of abstraction. > > The abstraction I would choose is "extra local variable", > which is what the LocalNode represents. Whether it uses > the same mechanism as that for the intermediate results of > expressions is an implementation detail, as is whether it > makes use of an Entry in the symbol table. Yes, these turn orthogonal things and the interface of LetNode functionality doesn't depend on how temps are done. Shifting over to LetNode; during analyse_declarations stuff like CVarDeclNode (and probably CArgDeclNode etc. when we get around to it) is stripped from the tree and no longer considered in the transforms, as the information has been moved over to the scope. So to be consistent with this one could think that LetNode should have a LocalNode prior to analyse_declarations; and after that it would have entries, as that is how other variables work (not necesarrily registered in a scope; if they were, one would have to create a new LetScope etc. which is kind of a pain). Note that after analyse_declarations, transforms have to (and currently does) modify the scopes directly to do their stuff with variables. (For this purpose, ModuleNode and FuncDefNode has scope and local_scope variables added that can be followed by transforms and inspected and modified). I suppose LetNode should be left until there is a temp system available that let one generate. I may very well end up supporting your approach in a few days. > I'm expecting that various things such as for-loops will > be implementable using LetNodes with suitable internal > plumbing, so the custom nodes for these statements would > disappear, along with the custom code in them implementing > the temp allocation and code generation patterns. That's > an increase in re-use, to my way of thinking. Indeed (me and Robert just discussed this in the other current thread, though without the needed LetNode principle). >> Entry seems like a more natural concept for this to me -- because it is >> a >> more "refined", more basic concept of a handle to a variable in the >> scope. > > I don't see what you gain by doing this. Under my > proposal, there would only be one kind of thing for tree > transformations to deal with, i.e. Nodes. Under yours, > there would be two kinds of things -- Nodes and Entries. Already talked about this some...just wanted to give another example: I recently wanted to do something at the beginning of the function for each argument of a certain type -- my transform then iterated the funcdefnode.local_scope.arg_entries, and added its statements to the beginning of funcdefnode.body. In transforms look at Scope and Entry somewhat like a fancy dictionary which the function or module use for storing information about themselves (although it is at the same time a parallel tree following namespace nesting rather than code structure nesting...) But -- if treating LetNode-level-temps (as opposed to C-"register"-level-temps) as Entries only adds overhead and no benefits I will let go of the idea... I think it is mostly the CloneNode that scares me, that doesn't look very intuitive to use from a transform perspective. >> Unless you cache the function bodies it would need to be done right >> prior, >> so that you know which local variables to declare. > > I would split the code stream for the function body > in two, one for the declarations and one for the statements. > That wouldn't be hard to do -- I'm already doing something > similar with the module-level declarations (the 'h' > attribute of the code object refers to another code > object that collects the declarations in a StringIO). Excellent. Dag Sverre From dagss at student.matnat.uio.no Fri Jul 25 17:25:47 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 25 Jul 2008 17:25:47 +0200 (CEST) Subject: [Cython] Specification proposal for extern ctypedef exactness Message-ID: <33576.85.166.116.245.1216999547.squirrel@webmail.uio.no> I've been touching upon this before, I'm now wondering whether we can make an exact documented specification of how extern ctypedefs can work (if there is already then I apologize..). Proposal: - It is *not* necesarry that the size (long/short or float/double/long double) matches with the real type in C - However it *is* required that "unsigned" vs "signed" is correctly declared, and that "int" vs. "float" is correctly declared. (Failing to do this leads to undefined behaviour for now, but a module-loading-time exception could be arranged for.) Furthermore, it is recommended (just to have a guideline somewhere) to only use the types "int", "unsigned int" and "float" in extern ctypedefs. (The new thing here is relying on unsigned vs. signed, which I want to use for dropping support for negative buffer indexing in situations where there can be no negative indices (I'm sure gcc could do it for me but it helps not confusing the users too)). Dag Sverre From jek-gmane at kleckner.net Fri Jul 25 17:51:43 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Fri, 25 Jul 2008 08:51:43 -0700 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> <48838697.4070905@behnel.de> <85b5c3130807201310o22b565am16661cf1493060d2@mail.gmail.com> <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > Ondrej Certik wrote: >> On Sun, Jul 20, 2008 at 8:40 PM, Stefan Behnel wrote: >>> I'd be happy if you could include this patch. >>> >>> http://hg.cython.org/cython-devel/rev/e21391d5f23a >>> >>> It didn't make it into 0.9.8, but it makes Cython generate C code that >>> compiles with Py3 (beta 1/2). >> I didn't put it in there yet. Why not to make a new upstream release? >> That way it will help a lot more users. > > Yes, that would be the clean way. I was just thinking that since you were > pushing a Debian package anyway, it'd be nice to have something that > people can use for a little longer. > > Robert, could we have rev 719 as an intermediate 0.9.8.1? It would really > help people who want to get their Cython code ready for Py3 (not that I > heard anyone shouting, but I wouldn't want to be called a bottleneck when > the code is already there). Does that patch interact or help with Ptyhon 2.6 code generation? Thanks - Jim From robertwb at math.washington.edu Fri Jul 25 17:52:24 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 25 Jul 2008 08:52:24 -0700 Subject: [Cython] Specification proposal for extern ctypedef exactness In-Reply-To: <33576.85.166.116.245.1216999547.squirrel@webmail.uio.no> References: <33576.85.166.116.245.1216999547.squirrel@webmail.uio.no> Message-ID: On Jul 25, 2008, at 8:25 AM, Dag Sverre Seljebotn wrote: > I've been touching upon this before, I'm now wondering whether we > can make > an exact documented specification of how extern ctypedefs can work (if > there is already then I apologize..). > > Proposal: > > - It is *not* necesarry that the size (long/short or float/double/long > double) matches with the real type in C > - However it *is* required that "unsigned" vs "signed" is correctly > declared, and that "int" vs. "float" is correctly declared. > (Failing to do > this leads to undefined behaviour for now, but a module-loading-time > exception could be arranged for.) > > Furthermore, it is recommended (just to have a guideline somewhere) to > only use the types "int", "unsigned int" and "float" in extern > ctypedefs. > > (The new thing here is relying on unsigned vs. signed, which I want > to use > for dropping support for negative buffer indexing in situations where > there can be no negative indices (I'm sure gcc could do it for me > but it > helps not confusing the users too)). Yes, I believe this is already the case. One may be able to get away with wrongly declaring unsigned vs. signed, but there are no promises as to the correctness if one does that. - Robert From stefan_ml at behnel.de Fri Jul 25 17:59:51 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 25 Jul 2008 17:59:51 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> <85b5c3130807201039j44ee396w5d0e711e26fba492@mail.gmail.com> <85b5c3130807201046w493532cdqfd6a1ed285993883@mail.gmail.com> <48838697.4070905@behnel.de> <85b5c3130807201310o22b565am16661cf1493060d2@mail.gmail.com> <55460.213.61.181.86.1216624925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4889F877.2010706@behnel.de> Hi, Jim Kleckner wrote: > Stefan Behnel wrote: >> Ondrej Certik wrote: >>> On Sun, Jul 20, 2008 at 8:40 PM, Stefan Behnel wrote: >>>> I'd be happy if you could include this patch. >>>> >>>> http://hg.cython.org/cython-devel/rev/e21391d5f23a >>>> >>>> It didn't make it into 0.9.8, but it makes Cython generate C code that >>>> compiles with Py3 (beta 1/2). >>> I didn't put it in there yet. Why not to make a new upstream release? >>> That way it will help a lot more users. >> Yes, that would be the clean way. I was just thinking that since you were >> pushing a Debian package anyway, it'd be nice to have something that >> people can use for a little longer. >> >> Robert, could we have rev 719 as an intermediate 0.9.8.1? It would really >> help people who want to get their Cython code ready for Py3 (not that I >> heard anyone shouting, but I wouldn't want to be called a bottleneck when >> the code is already there). > > Does that patch interact or help with Ptyhon 2.6 code generation? No, the code generated by 0.9.8 should already work with 2.6 (beta2, that is). Stefan From robertwb at math.washington.edu Fri Jul 25 18:14:27 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 25 Jul 2008 09:14:27 -0700 Subject: [Cython] Temp framework proposal In-Reply-To: <488823F0.4040301@canterbury.ac.nz> References: <1142.195.159.192.226.1216850340.squirrel@webmail.uio.no> <488823F0.4040301@canterbury.ac.nz> Message-ID: <56B8FB26-D3AD-40D7-B20F-ABC7E7A5291F@math.washington.edu> On Jul 23, 2008, at 11:40 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: >> I believe the following >> temp refactoring is a) conceptually simple and not a big break with >> current ways of doing things, b) can be made very incrementally >> (though >> there may be inconsistencies in "how things are done" along the >> way), c) >> is useful (at least to me). > > I don't think it should be necessary to go to this > much upheaval in order to make temps easier to use. > All you should really need is some suitable node types > to use as building blocks during your tree transformation > phases. > > I'm thinking of something like the following: > > * LocalNode - an ExprNode representing a temporary local > variable that is allocated using the temps mechanism. > > * LetNode - a StatNode that holds a list of LocalNodes > and a StatListNode. It functions like a "let" statement > in Lisp, i.e. binds some local variables, executes the > body and then disposes of the locals. > > The code for the LetNode would work in a similar way to > the existing code in the for-statement node and other > places that use temporary locals, except that instead of > a fixed set of locals, it would do it for whatever was in > its list of LocalNodes. > > In the body of the LetNode, wherever you wanted to refer > to one of the locals, you would insert a CloneNode (an > already-existing node type) that references the relevant > LocalNode. > > Then the existing machinery should do the right thing. > Not knowing the result_code won't be an issue, because > you're not dealing with things at that level, you're > just plugging nodes together. > >> I think it >> is ok if we agree that result_code should be refactored at some later >> point if one gets the time and resources; i.e. one stores >> "result_entry" >> and other entry references instead at analysis time and only >> calculate >> result_code at code generation time). > > You really ought to look at how this is in the current > Pyrex before doing anything about this. The result code > is now retrieved by calling result_as(), or something else > that ends up calling it, during code generation. So there's > no longer a requirement to calculate the result code at > analysis time -- it can be constructed on the fly at code > generation time if need be. > > This means it would, I think, be fairly straightforward > now to merge all the temp allocation/release stuff into > the code generation phase, although I haven't looked in > detail at what would be required. I just want to pipe up and say that putting the temp allocation/ release into the code generation phase seems extremely natural for me--more so than anything else that has been proposed. Specifically, you request a temp, use it, then release it when you're done. None of this trying to sync temps between multiple phases. - Robert From jek-gmane at kleckner.net Fri Jul 25 18:15:52 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Fri, 25 Jul 2008 09:15:52 -0700 Subject: [Cython] Cython hg repo problem Message-ID: I tried cloning http://hg.cython.org/cython/ and got the error message: $ hg clone http://hg.cython.org/cython/ destination directory: cython requesting all changes adding changesets adding manifests adding file changes added 720 changesets with 2133 changes to 619 files updating working directory abort: data/tests/compile/cpdef.pyx.i at 25c6bf8ecd4d: no match found! There seems to be something wrong with the repository for the release. Try browsing this page to show the problem as well: http://hg.cython.org/cython/file/4e4808271bdf/tests/compile/ From robertwb at math.washington.edu Fri Jul 25 18:20:50 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 25 Jul 2008 09:20:50 -0700 Subject: [Cython] Cython hg repo problem In-Reply-To: References: Message-ID: On Jul 25, 2008, at 9:15 AM, Jim Kleckner wrote: > I tried cloning http://hg.cython.org/cython/ > and got the error message: > > $ hg clone http://hg.cython.org/cython/ > destination directory: cython > requesting all changes > adding changesets > adding manifests > adding file changes > added 720 changesets with 2133 changes to 619 files > updating working directory > abort: data/tests/compile/cpdef.pyx.i at 25c6bf8ecd4d: no match found! > > > There seems to be something wrong with the repository for the release. > Try browsing this page to show the problem as well: > http://hg.cython.org/cython/file/4e4808271bdf/tests/compile/ That's not a good sign :(. I'm going to be away for the weekend, but I'll try and see what's up with this. - Robert From dalcinl at gmail.com Fri Jul 25 22:25:51 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 25 Jul 2008 17:25:51 -0300 Subject: [Cython] Simple class hierarchy problems In-Reply-To: <6ce0ac130807231426p6a15118rcfacc9b968c9a232@mail.gmail.com> References: <6ce0ac130807231426p6a15118rcfacc9b968c9a232@mail.gmail.com> Message-ID: I believe you are not missing anything, It's just a bug in Cython !!. The following patch (also attached) against cython-devel repo seems to fix the problem. Of course, perhaps the patch is not completely OK because I'm missing something about the implementation of virtual tables. --- a/Cython/Compiler/ModuleNode.py Wed Jul 23 23:02:37 2008 -0700 +++ b/Cython/Compiler/ModuleNode.py Fri Jul 25 17:18:04 2008 -0300 @@ -836,8 +836,11 @@ class ModuleNode(Nodes.Node, Nodes.Block #if need_self_cast: # self.generate_self_cast(scope, code) if type.vtabslot_cname: - if base_type: - struct_type_cast = "(struct %s*)" % base_type.vtabstruct_cname + vtab_base_type = type + while vtab_base_type.base_type: + vtab_base_type = vtab_base_type.base_type + if vtab_base_type is not type: + struct_type_cast = "(struct %s*)" % vtab_base_type.vtabstruct_cname else: struct_type_cast = "" code.putln("p->%s = %s%s;" % ( On Wed, Jul 23, 2008 at 6:26 PM, Brian Granger wrote: > Hi, > > The following code is giving me problems: > > # foo.pyx > cdef class Foo: > > cdef fooit(self): > return 10 > > cdef class Bar(Foo): > pass > > cdef class Bam(Bar): > pass > > When I do "cython foo.pyx" and compile as c I get a warning: > > foo.c: In function '__pyx_tp_new_3foo_Bam': > foo.c:608: warning: assignment from incompatible pointer type > > But, when I do "cython --cplus" and compile with c++ I get a failure: > > foo.cpp: In function 'PyObject* __pyx_tp_new_3foo_Bam(PyTypeObject*, > PyObject*, PyObject*)': > foo.cpp:608: error: cannot convert '__pyx_vtabstruct_3foo_Bar*' to > '__pyx_vtabstruct_3foo_Foo*' in a > > I swear this code used to work fine. I am using Cython 0.9.8, but the > problem is there if I try it in earlier versions. > > What am I missing? > > Thanks > > Brian > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: vtab.diff Type: text/x-patch Size: 833 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080725/c49d7741/attachment-0001.bin From stefan_ml at behnel.de Sun Jul 27 08:28:24 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 27 Jul 2008 08:28:24 +0200 Subject: [Cython] Fwd: [issue3443] crash on badly initialised AttributeError Message-ID: <488C1588.7080102@behnel.de> Talking about differences betweeen Cython and Python... I got a crash in lxml's test suite with Python 3.0beta2, so I reported that to the Python bug tracker. Looks like ours would have been more appropriate. http://bugs.python.org/issue3443 Stefan -------- Original-Message -------- Subject: [issue3443] crash on badly initialised AttributeError Date: Fri, 25 Jul 2008 21:03:50 +0000 From: Amaury Forgeot d'Arc I reproduced the problem on Windows. The exception shown is the AttributeError('next') raised and caught in lxml._elementpath.find(). The "args" atribute is cleared in the BaseException_clear, during a call to gc.collect() (in some tearDown() method). Unfortunately this exception is still shomehow stored in the thread state structure... After some researches, I think that the problem is in Cython. Cython tries to simulate a python frame without using the interpreter loop. But at least an invariant is not respected: when a frame exits without an exception set, the tstate->exc_type (and friends) should be NULL. For example, at the end of the PyInit_etree() function (called by an "import lxml.etree"), the tstate->exc_value contains an AttributeError('module' object has no attribute 'unicode'). Now, the consequence may be that all these exceptions are "implicitely chained" together. And there may be a subtle refcounting bug that shows up only if this invariant is not respected. From greg.ewing at canterbury.ac.nz Sun Jul 27 09:09:47 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 27 Jul 2008 19:09:47 +1200 Subject: [Cython] Specification proposal for extern ctypedef exactness In-Reply-To: <33576.85.166.116.245.1216999547.squirrel@webmail.uio.no> References: <33576.85.166.116.245.1216999547.squirrel@webmail.uio.no> Message-ID: <488C1F3B.5000204@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > I'm now wondering whether we can make > an exact documented specification of how extern ctypedefs can work At one time I was thinking of having a way of being explicitly vague about the type, e.g. ctypedef some int foo Such vague types could then be disallowed in places where the exact type matters. -- Greg From cb at mit.edu Thu Jul 17 23:15:47 2008 From: cb at mit.edu (Chuck Blake) Date: Thu, 17 Jul 2008 17:15:47 -0400 Subject: [Cython] inspection of compile-time macro/etc. namespace Message-ID: <20080717211547.GA11007@pdos.lcs.mit.edu> Hey, guys. Thanks for the hard work on Cython. It's been far superior to Pyrex for my needs. Anyway, I have .pxi files that I like to use for actual gcc-level function inlining. I need conditional compilation directives that make multiple inclusion idempotent. The usual guards in the C/C++ world would be: #ifndef FOO_H #define FOO_H ... #endif In Cython, I have been doing DEF FOO_1 = 1 DEF FOO_2 = 1 IF FOO_1 == FOO_2: DEF FOO_2 = 2 ... That seems kind of ugly, though. Beyond aesthetics, it is probably nice to test for a name before using it/logically fork/etc. based on that. Just as one e.g., if there is any evolution of __builtin__ names populating the environment. So, I propose a tiny addition to add a new compile- time builtin called DEFINED() that just evaluates to a boolean based on whether its string argument is in the compile-time environment. This allows the above construct to be written both more simply and more clearly: IF not DEFINED("FOO"): DEF FOO = 1 ... Name choice follows the principle of least surprise since the lowercase cpp name has similar functionality, just like the other conditional compilation features. cb -------------- next part -------------- diff -ruw Cython-0.9.8.orig/Cython/Compiler/Scanning.py Cython-0.9.8/Cython/Compiler/Scanning.py --- Cython-0.9.8.orig/Cython/Compiler/Scanning.py 2008-06-11 14:25:34.000000000 -0400 +++ Cython-0.9.8/Cython/Compiler/Scanning.py 2008-07-17 16:39:27.691882798 -0400 @@ -186,6 +186,9 @@ else: raise + def defined(self, name): + return name in self.entries + def initial_compile_time_env(): benv = CompileTimeScope() names = ('UNAME_SYSNAME', 'UNAME_NODENAME', 'UNAME_RELEASE', @@ -201,6 +204,7 @@ for name in names: benv.declare(name, getattr(__builtin__, name)) denv = CompileTimeScope(benv) + denv.declare('DEFINED', denv.defined) return denv #------------------------------------------------------------------ From mabshoff at googlemail.com Fri Jul 18 09:51:57 2008 From: mabshoff at googlemail.com (Michael Abshoff) Date: Fri, 18 Jul 2008 00:51:57 -0700 Subject: [Cython] Fwd: [sage-devel] cython installation with python 2.3 In-Reply-To: References: Message-ID: Hi, this was not answered on sage-devel, but I assume you guys would want to look into this. IIRC numpy/scipy is still supported on Python 2.3, so this would likely come up sooner or later. Cheers, Michael ---------- Forwarded message ---------- From: Utpal Sarkar Date: Tue, Jul 15, 2008 at 8:09 AM Subject: [sage-devel] cython installation with python 2.3 To: sage-devel Hi, I'm not sure this is the correct place to post this. I just installed cython 0.9.8 on my computer, which runs python 2.3. I got two error that could be fixed quite easily, and which may have to do with the fact that my python is quite old. The first is in the file Cython/Compiler/TreeFragment.py, which in line 136 reads minindent = min(len(INDENT_RE.match(x).group(0)) for x in lines), but for my version of python has to be changed to minindent = min([len(INDENT_RE.match(x).group(0)) for x in lines]) (explicit list comprehension). This should also work in newer versions of python. The other is in setup.py, where lines 15-16 are setup_args['data_files'] = [ {compiler_dir, ['Cython/Compiler/Lexicon.pickle']}] which I changed to setup_args['data_files'] = [ (compiler_dir, ['Cython/Compiler/Lexicon.pickle'])] because I saw that (my installed version of) distutils expected a tuple, not a dictionary. I don't know if this would still work with later or other versions of distutils. Greetings, Utpal --~--~---------~--~----~------------~-------~--~----~ To post to this group, send email to sage-devel at googlegroups.com To unsubscribe from this group, send email to sage-devel-unsubscribe at googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~----------~----~----~----~------~----~------~--~--- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080718/b3be15ae/attachment.htm From cb at pdos.cail.mit.edu Fri Jul 18 15:48:30 2008 From: cb at pdos.cail.mit.edu (Chuck Blake) Date: Fri, 18 Jul 2008 09:48:30 -0400 Subject: [Cython] inspection of compile-time macro/etc. namespace Message-ID: <20080718134830.GA31449@pdos.lcs.mit.edu> Hey, guys. Thanks for the hard work on Cython. It's been far superior to Pyrex for my needs. Anyway, I have .pxi files that I like to use for actual gcc-level function inlining. I need conditional compilation directives that make multiple inclusion idempotent. The usual guards in the C/C++ world would be: #ifndef FOO_H #define FOO_H ... #endif In Cython, I have been doing DEF FOO_1 = 1 DEF FOO_2 = 1 IF FOO_1 == FOO_2: DEF FOO_2 = 2 ... That seems kind of ugly, though. Beyond aesthetics, it is probably nice to test for a name before using it/logically fork/etc. based on that. Just as one e.g., if there is any evolution of __builtin__ names populating the environment. So, I propose a tiny addition to add a new compile- time builtin called DEFINED() that just evaluates to a boolean based on whether its string argument is in the compile-time environment. This allows the above construct to be written both more simply and more clearly: IF not DEFINED("FOO"): DEF FOO = 1 ... Name choice follows the principle of least surprise since the lowercase cpp name has similar functionality, just like the other conditional compilation features. cb From tabbott at MIT.EDU Sun Jul 20 19:27:34 2008 From: tabbott at MIT.EDU (Timothy G Abbott) Date: Sun, 20 Jul 2008 13:27:34 -0400 (EDT) Subject: [Cython] Cython 0.9.8 released In-Reply-To: <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> <85b5c3130807200502t3215485frd3c448fafafc46@mail.gmail.com> <85b5c3130807200509gacfd384r8bbdf5d1b54d480d@mail.gmail.com> Message-ID: On Sun, 20 Jul 2008, Ondrej Certik wrote: > > I think this patch needs to be applied: > > Index: debian/control > =================================================================== > --- debian/control (revision 1584) > +++ debian/control (working copy) > @@ -3,7 +3,7 @@ > Priority: optional > Maintainer: Python Applications Packaging Team > > Uploaders: Ondrej Certik > -Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= 2.4.4-6) > +Build-Depends: cdbs (>= 0.4.49), debhelper (>= 5.0.38), python (>= > 2.4.4-6), python-dev (>= 2.4.4-6) > Build-Depends-Indep: python-support (>= 0.7.5) > Standards-Version: 3.8.0 > Homepage: http://cython.org/ > @@ -12,7 +12,7 @@ > XS-DM-Upload-Allowed: yes > > Package: cython > -Architecture: all > +Architecture: any > Depends: ${python:Depends}, ${misc:Depends} > Suggests: gcc > XB-Python-Version: ${python:Versions} > > > However, the cython build fails then: > > cd . && python2.4 setup.py build --build-base="/tmp/buildd/cython-0.9.8/./build" > /bin/sh: python2.4: command not found > make: *** [python-build-stamp-2.4] Error 127 > > Now I am investigating why. python2.4 is not called from Cython, so > it's seems some Debian thing. CCing to Tim -- I am quite busy today, I > really need to work on my thesis, so I'd appreciate any help with > this. > > Ondrej I think you want python-all-dev (>= 2.4.4-6), rather than just python-dev (>= 2.4.4-6). What's going on is that Debian build extensions modules for all current versions of python (python2.4 and python2.5) when you're using python-support; python-all-dev is the right way to include that as a dependency. -Tim Abbott From stefan_ml at behnel.de Sun Jul 27 09:19:59 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 27 Jul 2008 09:19:59 +0200 Subject: [Cython] Fwd: [issue3443] crash on badly initialised AttributeError In-Reply-To: <488C1588.7080102@behnel.de> References: <488C1588.7080102@behnel.de> Message-ID: <488C219F.3070809@behnel.de> Amaury Forgeot d'Arc wrote: > After some researches, I think that the problem is in Cython. Cython > tries to simulate a python frame without using the interpreter loop. > But at least an invariant is not respected: when a frame exits without > an exception set, the tstate->exc_type (and friends) should be NULL. > For example, at the end of the PyInit_etree() function (called by an > "import lxml.etree"), the tstate->exc_value contains an > AttributeError('module' object has no attribute 'unicode'). > > Now, the consequence may be that all these exceptions are "implicitely > chained" together. And there may be a subtle refcounting bug that shows > up only if this invariant is not respected. This explanation sounds like the right thing to do would be to clear the exception state when a function exists cleanly but an exception was raised and caught during its execution. So the exception state would only stay available within the function itself. We could also try to emulate the Py3 behaviour as outlined in PEP 3110. http://python.org/dev/peps/pep-3110/ Opinions? Stefan From ondrej at certik.cz Sun Jul 20 13:58:26 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 20 Jul 2008 13:58:26 +0200 Subject: [Cython] Cython 0.9.8 released In-Reply-To: <488321F4.4050007@behnel.de> References: <4852169A.1050706@behnel.de> <85b5c3130807200414p5f038b46t6c37d3f72785b165@mail.gmail.com> <488321F4.4050007@behnel.de> Message-ID: <85b5c3130807200458u4212cc51od85991f7c85cf29f@mail.gmail.com> On Sun, Jul 20, 2008 at 1:31 PM, Stefan Behnel wrote: > Hi, > > Ondrej Certik wrote: >> Why is cython now compiling Cython.Plex.Scanners with gcc but not >> installing it with setup.py install? > > It should get installed automatically by distutils. > > It gets compiled because that makes it faster. It's a purely optional feature. > > >> Is this a bug with Cython or on my side? I am just using the same >> Debian package as I was using for all previous cython versions, but >> suddenly it fails to build, because it depends on python-dev. So I >> added python-dev to build-depends, it builds, it works, but the >> Cython/Plex/Scanners.so is not installed in the final package. What's >> wrong? > > Any error output? No, all is fine. It just doesn't install. > > >> Should I upload the package as it is, i.e. without the Scanners.so >> file? I tested it on some simple .pyx files and it seems to work. > > You can, but I would prefer having that in. Yes, me too. > > >> I'd like to get this resolved in about 5 hours, so that I can upload >> before today's dinstall run, so that Tim can build Sage in Debian with >> it. > > Please provide some more details. The build log attached. When installing the package, here is the list of files: $ wajig list-files cython /. /usr /usr/bin /usr/bin/cython /usr/share /usr/share/man /usr/share/man/man1 /usr/share/man/man1/cython.1.gz /usr/share/python-support /usr/share/python-support/cython /usr/share/python-support/cython/Cython /usr/share/python-support/cython/Cython/Tests /usr/share/python-support/cython/Cython/Tests/TestCodeWriter.py /usr/share/python-support/cython/Cython/Tests/__init__.py /usr/share/python-support/cython/Cython/CodeWriter.py /usr/share/python-support/cython/Cython/Plex /usr/share/python-support/cython/Cython/Plex/Scanners.py /usr/share/python-support/cython/Cython/Plex/Traditional.py /usr/share/python-support/cython/Cython/Plex/Errors.py /usr/share/python-support/cython/Cython/Plex/test_tm.py /usr/share/python-support/cython/Cython/Plex/Transitions.py /usr/share/python-support/cython/Cython/Plex/Regexps.py /usr/share/python-support/cython/Cython/Plex/__init__.py /usr/share/python-support/cython/Cython/Plex/Lexicons.py /usr/share/python-support/cython/Cython/Plex/DFA.py /usr/share/python-support/cython/Cython/Plex/Machines.py /usr/share/python-support/cython/Cython/Plex/Actions.py /usr/share/python-support/cython/Cython/Plex/Timing.py /usr/share/python-support/cython/Cython/Utils.py /usr/share/python-support/cython/Cython/Distutils /usr/share/python-support/cython/Cython/Distutils/build_ext.py /usr/share/python-support/cython/Cython/Distutils/extension.py /usr/share/python-support/cython/Cython/Distutils/__init__.py /usr/share/python-support/cython/Cython/Unix /usr/share/python-support/cython/Cython/Unix/__init__.py /usr/share/python-support/cython/Cython/Unix/LinuxSystem.py /usr/share/python-support/cython/Cython/Compiler /usr/share/python-support/cython/Cython/Compiler/Tests /usr/share/python-support/cython/Cython/Compiler/Tests/TestTreeFragment.py /usr/share/python-support/cython/Cython/Compiler/Tests/__init__.py /usr/share/python-support/cython/Cython/Compiler/TreeFragment.py /usr/share/python-support/cython/Cython/Compiler/Visitor.py /usr/share/python-support/cython/Cython/Compiler/Version.py /usr/share/python-support/cython/Cython/Compiler/Lexicon.pickle /usr/share/python-support/cython/Cython/Compiler/ModuleNode.py /usr/share/python-support/cython/Cython/Compiler/Code.py /usr/share/python-support/cython/Cython/Compiler/PyrexTypes.py /usr/share/python-support/cython/Cython/Compiler/Annotate.py /usr/share/python-support/cython/Cython/Compiler/Lexicon.py /usr/share/python-support/cython/Cython/Compiler/Main.py /usr/share/python-support/cython/Cython/Compiler/Errors.py /usr/share/python-support/cython/Cython/Compiler/ExprNodes.py /usr/share/python-support/cython/Cython/Compiler/Optimize.py /usr/share/python-support/cython/Cython/Compiler/CmdLine.py /usr/share/python-support/cython/Cython/Compiler/Parsing.py /usr/share/python-support/cython/Cython/Compiler/ControlFlow.py /usr/share/python-support/cython/Cython/Compiler/__init__.py /usr/share/python-support/cython/Cython/Compiler/Builtin.py /usr/share/python-support/cython/Cython/Compiler/Naming.py /usr/share/python-support/cython/Cython/Compiler/TypeSlots.py /usr/share/python-support/cython/Cython/Compiler/Future.py /usr/share/python-support/cython/Cython/Compiler/Nodes.py /usr/share/python-support/cython/Cython/Compiler/Scanning.py /usr/share/python-support/cython/Cython/Compiler/Symtab.py /usr/share/python-support/cython/Cython/Compiler/Options.py /usr/share/python-support/cython/Cython/Compiler/DebugFlags.py /usr/share/python-support/cython/Cython/__init__.py /usr/share/python-support/cython/Cython/TestUtils.py /usr/share/python-support/cython/Cython/Debugging.py /usr/share/python-support/cython/Cython/Mac /usr/share/python-support/cython/Cython/Mac/MacSystem.py /usr/share/python-support/cython/Cython/Mac/DarwinSystem.py /usr/share/python-support/cython/Cython/Mac/TS_Misc_Suite.py /usr/share/python-support/cython/Cython/Mac/MacUtils.py /usr/share/python-support/cython/Cython/Mac/__init__.py /usr/share/python-support/cython/Cython-0.9.8.egg-info /usr/share/doc /usr/share/doc/cython /usr/share/doc/cython/copyright /usr/share/doc/cython/changelog.Debian.gz /usr/share/doc/cython/README.Debian Ondrej -------------- next part -------------- A non-text attachment was scrubbed... Name: cython.build.log Type: text/x-log Size: 40027 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080720/f1a89f66/attachment-0001.bin From stefan_ml at behnel.de Sun Jul 27 21:58:40 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 27 Jul 2008 21:58:40 +0200 Subject: [Cython] Fwd: [sage-devel] cython installation with python 2.3 In-Reply-To: References: Message-ID: <488CD370.6010101@behnel.de> Hi, in general, Cython should support 2.3, so it's a bug if it doesn't. Michael Abshoff wrote: > The first is in the file Cython/Compiler/TreeFragment.py, which in > line 136 reads > minindent = min(len(INDENT_RE.match(x).group(0)) for x in lines), but > for my version of python has to be changed to > minindent = min([len(INDENT_RE.match(x).group(0)) for x in lines]) > (explicit list comprehension). This should also work in newer versions > of python. Yep, that's already fixed in the developer version. > The other is in setup.py, where lines 15-16 are > setup_args['data_files'] = [ > {compiler_dir, ['Cython/Compiler/Lexicon.pickle']}] > which I changed to > setup_args['data_files'] = [ > (compiler_dir, ['Cython/Compiler/Lexicon.pickle'])] > because I saw that (my installed version of) distutils expected a > tuple, not a dictionary. I don't know if this would still work with > later or other versions of distutils. I checked, and that's what the Py2.3 docs tell me as well. This is code specifically for Py2.3, so I have no idea how this could ever work... Thanks for the report, Stefan From dagss at student.matnat.uio.no Tue Jul 29 11:25:11 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 11:25:11 +0200 Subject: [Cython] -dagss merge? + what next Message-ID: <488EE1F7.8060905@student.matnat.uio.no> I'm at a stage where I think we can consider a new merge of -dagss. What I say below applies to revision 876:5df94de34ff7; the -dagss branch will promptly go unstable again so make sure you pull this one. Buffers are basically working (I am reasonably confident it is working but I still have ideas for testcases that I haven't written so the testsuite is not exhaustive). In particular I'd like to put it out as an "available beta component" in the next Cython release (I can write some release notes for it). What next? Comments welcome on this: - I'd like to leave buffers in the global scope or as attributes out of it for now. It is really non-trivial and I think it would be wrong to prioritize it above all the other stuff I could do. - Given that, I think these three tasks are important: - Supporting structs (currently only ints and floats are supported). Complex numbers are part of that. - Temp allocation gets quite hairy with buffers and I'd like to implement Greg Ewing's proposal to clean it up. But there is a potential for cleaning it more than it is within the existing framework as well. - "final" functions in pxd file, in order to implement numpy.ndarray.__getbuffer__ in the pxd file. (I won't do any inlining in Cython, it is just a way of having code automatically copied from the pxd to the pyx (and potentially skip vtable lookups if I get around to it)). Those are the big tasks, but there are also a few smaller nice-haves I can go for (a "cython.shape" builtin that allows you to retrieve the shape information of a buffer, and so on). Notes: - Includes is now always included. (Though I am not any longer using the __cython__ namespace, my test cases will like to cimport things). - Get/ReleaseBuffer is no longer defined when not needed - Temp allocation is rather nasty at the moment (using my prototype which we ended up rejecting!). I am thinking that I might rather spend half a day to a day implementing Greg Ewing's ideas and use those than fix it. (I'm thinking along the lines of first implementing "code.fork()" which forks the output stream, using a cStringIO "tree" so that no buffer copying is needed). But it is working, and even though I'll definitely remove it I don't think it is blocking a merge. - Overall strategy: I've modified FuncDefNode, NameNode and IndexNode to do what I need to do. No new nodes. I ended up thinking this was much cleaner than inserting new nodes: - IndexNode already supports two different concepts (C arrays and Python Get/SetItem), adding a third seemed to be consistent with this. - NameNode: The way things are implemented, buffer auxiliary variables (a lot of them) are declared and connected to the local variable that is of the buffer type. So a typecasting-style node that primarily work through temporaries would have been less clean. NameNode also already support many different kinds of operations and so adding another one seemed ok. I'm ok with splitting into seperate buffer nodes later, but I think it should be done while also splitting the rest, i.e. transform IndexNode into CArrayIndexNode, PythonIndexNode and BufferIndexNode and so on. (But, seeing as that is only for abstract purity at the moment I definitely won't touch it.) -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 11:36:07 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 11:36:07 +0200 Subject: [Cython] Cython and complex floats Message-ID: <488EE487.90102@student.matnat.uio.no> Buffers and Python has the complex datatype, while I'm reasonably sure I haven't found anything about it in Cython (?). So here's the three options I see ordered by amount of work, comments welcome. These are incremental so it is not about picking one, but about whether anyone has any objections to 2) or 3) going into Cython, as well as supporting non-C99-compilers is either a) mandatory or b) a waste of time. (Also I don't know whether I'll get around to more than 1) this time around, will have to see what you say.) 1) Complex numbers can be pulled out into a struct consisting of two floats. Like this: cdef struct cdouble: double real double imag def f(object[cdouble] buf): ... Operations on the fields must happen manually. 2) In addition, Cython gets support for the C99 "complex" float modifier. def complex double sum(object[complex double] buf): cdef complex double result = 0 + 0j ... Here, operations between complex datatypes are done natively by the compiler. However the code generated is illegal on non-C99-compilers. 3) Also support older compilers by doing all operations through conditional macros where complex operations are defined for other compilers as well. (I guess, given 2) this is not much work) -- Dag Sverre From robertwb at math.washington.edu Tue Jul 29 11:43:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jul 2008 02:43:07 -0700 Subject: [Cython] -dagss merge? + what next In-Reply-To: <488EE1F7.8060905@student.matnat.uio.no> References: <488EE1F7.8060905@student.matnat.uio.no> Message-ID: <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote: > I'm at a stage where I think we can consider a new merge of -dagss. > What > I say below applies to revision 876:5df94de34ff7; the -dagss branch > will > promptly go unstable again so make sure you pull this one. Great. I was actually just wondering earlier today when a good stable pull would be. > Buffers are basically working (I am reasonably confident it is working > but I still have ideas for testcases that I haven't written so the > testsuite is not exhaustive). In particular I'd like to put it out > as an > "available beta component" in the next Cython release (I can write > some > release notes for it). Sure. I want to do a release again soon. This is safe for an alpha as it is a completely new feature with new (formerly illegal) syntax, so it shouldn't impact old working code, and hopefully we'll get a lot of feedback. > What next? Comments welcome on this: > - I'd like to leave buffers in the global scope or as attributes > out of > it for now. It is really non-trivial and I think it would be wrong to > prioritize it above all the other stuff I could do. I agree. See my (very loose) priorities below. > - Given that, I think these three tasks are important: > - Supporting structs (currently only ints and floats are > supported). > Complex numbers are part of that. 3. > - Temp allocation gets quite hairy with buffers and I'd like to > implement Greg Ewing's proposal to clean it up. But there is a > potential > for cleaning it more than it is within the existing framework as well. 1. > - "final" functions in pxd file, in order to implement > numpy.ndarray.__getbuffer__ in the pxd file. (I won't do any > inlining in > Cython, it is just a way of having code automatically copied from the > pxd to the pyx (and potentially skip vtable lookups if I get around > to it)). 2. (Though this should be really easy compared to the temps stuff). > Those are the big tasks, but there are also a few smaller nice-haves I > can go for (a "cython.shape" builtin that allows you to retrieve the > shape information of a buffer, and so on). I saw you mention shape (and other "magic" functions), and I'm curious. Could you elaborate? > > Notes: > > - Includes is now always included. (Though I am not any longer > using the > __cython__ namespace, my test cases will like to cimport things). > > - Get/ReleaseBuffer is no longer defined when not needed Excellent. > - Temp allocation is rather nasty at the moment (using my prototype > which we ended up rejecting!). I am thinking that I might rather spend > half a day to a day implementing Greg Ewing's ideas and use those than > fix it. (I'm thinking along the lines of first implementing > "code.fork()" which forks the output stream, using a cStringIO > "tree" so > that no buffer copying is needed). Sounds good. > But it is working, and even though I'll definitely remove it I don't > think it is blocking a merge. > > - Overall strategy: I've modified FuncDefNode, NameNode and > IndexNode to > do what I need to do. No new nodes. I ended up thinking this was much > cleaner than inserting new nodes: > - IndexNode already supports two different concepts (C arrays and > Python Get/SetItem), adding a third seemed to be consistent with this. > - NameNode: The way things are implemented, buffer auxiliary > variables (a lot of them) are declared and connected to the local > variable that is of the buffer type. So a typecasting-style node that > primarily work through temporaries would have been less clean. > NameNode > also already support many different kinds of operations and so adding > another one seemed ok. This sounds good and relatively clean, as well as consistent with what's there now (i.e. if someone wants to know how indexing works, there's one place to look). > I'm ok with splitting into seperate buffer nodes later, but I think it > should be done while also splitting the rest, i.e. transform IndexNode > into CArrayIndexNode, PythonIndexNode and BufferIndexNode and so on. > (But, seeing as that is only for abstract purity at the moment I > definitely won't touch it.) Yeah, sounds like another task for another day (though hopefully not too far off). Good work! - Robert From dagss at student.matnat.uio.no Tue Jul 29 12:57:25 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 12:57:25 +0200 Subject: [Cython] -dagss merge? + what next In-Reply-To: <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> Message-ID: <488EF795.7060405@student.matnat.uio.no> Robert Bradshaw wrote: > On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote: > >> Those are the big tasks, but there are also a few smaller nice-haves I >> can go for (a "cython.shape" builtin that allows you to retrieve the >> shape information of a buffer, and so on). > > I saw you mention shape (and other "magic" functions), and I'm > curious. Could you elaborate? When acquiring a buffer, you get hold of a lot of information that is not currently exported -- shape is most important. For NumPy this is ok as ndarray exports its shape field for efficient access, but without another mechanism you can't write generic algorithms that work with all buffers without passing the shape explicitly. This even leads to less clean test case code... I haven't thought much about the list (add as needed) but something like: - shape(buf) -- Returns a tuple containing the shape of the buffer. However, shape(buf)[0] should compile directly to __pyx_bshape0_buf. (I thought about shape(buf, 0) as well but decided against, how about you?) This would be the primary for-loop mechanism: from cython import shape ... cdef int i for i in range(shape(buf)[0]): buf[i] = 9 - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to allow low-level access (buffer gets a void* directly, strides and suboffsets probably). For contiguous arrays this could be more efficient (the proper high-level approach for such things are iterators). The idea is that even if you write low-level code, you could use the buffer syntax so that you don't need to worry about acquisition. - ndim(buf) -- If the ndim-at-compile-time-restriction is lifted (because you have low-level access or iterators you can use instead of indexing) then this will be useful. Finally, if I get as far as iterators (I won't!), cython.ndenumerate could create very efficient iteration code. (Probably a NumPy-specific iterator could be less work though...if I can't just copy the approach of NumPy iterators directly...NumPy comes with a seperate n-dimensional iterator class that allows you to write ndim-generic code.) -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 13:00:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 13:00:03 +0200 Subject: [Cython] -dagss merge? + what next In-Reply-To: <488EF795.7060405@student.matnat.uio.no> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> <488EF795.7060405@student.matnat.uio.no> Message-ID: <488EF833.7020408@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote: >> >>> Those are the big tasks, but there are also a few smaller nice-haves I >>> can go for (a "cython.shape" builtin that allows you to retrieve the >>> shape information of a buffer, and so on). >> I saw you mention shape (and other "magic" functions), and I'm >> curious. Could you elaborate? > > When acquiring a buffer, you get hold of a lot of information that is > not currently exported -- shape is most important. For NumPy this is ok > as ndarray exports its shape field for efficient access, but without > another mechanism you can't write generic algorithms that work with all > buffers without passing the shape explicitly. This even leads to less > clean test case code... > > I haven't thought much about the list (add as needed) but something like: > > - shape(buf) -- Returns a tuple containing the shape of the buffer. > However, shape(buf)[0] should compile directly to __pyx_bshape0_buf. (I > thought about shape(buf, 0) as well but decided against, how about you?) > > This would be the primary for-loop mechanism: > > from cython import shape > ... > cdef int i > for i in range(shape(buf)[0]): buf[i] = 9 > > - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to > allow low-level access (buffer gets a void* directly, strides and > suboffsets probably). For contiguous arrays this could be more efficient ...strides and suboffsets probably Py_ssize_t*. Sorry. -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 13:51:35 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 13:51:35 +0200 Subject: [Cython] On GetBuffer/ReleaseBuffer (attn: Stefan) Message-ID: <488F0447.5030601@student.matnat.uio.no> First, Stefan: In tests/run/buffer.pyx I saw references to "obj == NULL" for locking the buffer. I can't seem find any references to this anywhere, any hints? Secondly, Robert answered a question I noted down for "asking later" on the wiki so I'll respond to his response here. Me: Does any exceptions ReleaseBuffer might set matter? Should I start calling PyErr_Occurred? Robert: Don't call PyErr_Occurred, check the return value for -1 (which is what it is supposed to return on failure). Me again: Note that PyObject_ReleaseBuffer is defined like this: PyAPI_FUNC(void) PyObject_ReleaseBuffer(PyObject *obj, Py_buffer *view); Actually, it says in the PEP that the function "always succeeds". But I would guess that we don't actually enforce that for Cython-implemented __releasebuffer__? So ideally one would raise a compiler error if there was a chance that an exception would be propagated from ReleaseBuffer (much like Java does for some exceptions)... -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 15:04:20 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 15:04:20 +0200 Subject: [Cython] -dagss and executable bit warning Message-ID: <488F1554.9040009@student.matnat.uio.no> Due to transferring my repository via a FAT usb stick, I accidentally commited all files in -dagss as executable. I have fixed this (all files but bin/cython non-executable) in 877:cd8ec8228670, which is also considered stable for merge. That will teach me to tar my files... -- Dag Sverre From robertwb at math.washington.edu Tue Jul 29 18:34:24 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jul 2008 09:34:24 -0700 Subject: [Cython] -dagss merge? + what next In-Reply-To: <488EF795.7060405@student.matnat.uio.no> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> <488EF795.7060405@student.matnat.uio.no> Message-ID: On Jul 29, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote: >> >>> Those are the big tasks, but there are also a few smaller nice- >>> haves I >>> can go for (a "cython.shape" builtin that allows you to retrieve the >>> shape information of a buffer, and so on). >> >> I saw you mention shape (and other "magic" functions), and I'm >> curious. Could you elaborate? > > When acquiring a buffer, you get hold of a lot of information that is > not currently exported -- shape is most important. For NumPy this > is ok > as ndarray exports its shape field for efficient access, but without > another mechanism you can't write generic algorithms that work with > all > buffers without passing the shape explicitly. This even leads to less > clean test case code... > > I haven't thought much about the list (add as needed) but something > like: > > - shape(buf) -- Returns a tuple containing the shape of the buffer. > However, shape(buf)[0] should compile directly to > __pyx_bshape0_buf. (I > thought about shape(buf, 0) as well but decided against, how about > you?) > > This would be the primary for-loop mechanism: > > from cython import shape > ... > cdef int i > for i in range(shape(buf)[0]): buf[i] = 9 > > - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to > allow low-level access (buffer gets a void* directly, strides and > suboffsets probably). For contiguous arrays this could be more > efficient > (the proper high-level approach for such things are iterators). The > idea > is that even if you write low-level code, you could use the buffer > syntax so that you don't need to worry about acquisition. > > - ndim(buf) -- If the ndim-at-compile-time-restriction is lifted > (because you have low-level access or iterators you can use instead of > indexing) then this will be useful. > > Finally, if I get as far as iterators (I won't!), cython.ndenumerate > could create very efficient iteration code. (Probably a NumPy-specific > iterator could be less work though...if I can't just copy the approach > of NumPy iterators directly...NumPy comes with a seperate n- > dimensional > iterator class that allows you to write ndim-generic code.) Ah, I see. These could be very useful, though I'm not sure we should be augmenting the namespace to provide them (perhaps make them magic methods of a buffer object instead?) - Robert From robertwb at math.washington.edu Tue Jul 29 18:38:34 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jul 2008 09:38:34 -0700 Subject: [Cython] On GetBuffer/ReleaseBuffer (attn: Stefan) In-Reply-To: <488F0447.5030601@student.matnat.uio.no> References: <488F0447.5030601@student.matnat.uio.no> Message-ID: <046ADA9E-DB6A-44EB-B683-561728480B90@math.washington.edu> On Jul 29, 2008, at 4:51 AM, Dag Sverre Seljebotn wrote: > First, Stefan: In tests/run/buffer.pyx I saw references to "obj == > NULL" > for locking the buffer. I can't seem find any references to this > anywhere, any hints? > > Secondly, Robert answered a question I noted down for "asking > later" on > the wiki so I'll respond to his response here. > > Me: Does any exceptions ReleaseBuffer might set matter? Should I start > calling PyErr_Occurred? > > Robert: Don't call PyErr_Occurred, check the return value for -1 > (which > is what it is supposed to return on failure). > > Me again: > > Note that PyObject_ReleaseBuffer is defined like this: > > PyAPI_FUNC(void) PyObject_ReleaseBuffer(PyObject *obj, Py_buffer > *view); Ah, what I wrote was only relevant to PyObject_GetBuffer. > Actually, it says in the PEP that the function "always succeeds". > But I > would guess that we don't actually enforce that for Cython-implemented > __releasebuffer__? So ideally one would raise a compiler error if > there > was a chance that an exception would be propagated from ReleaseBuffer > (much like Java does for some exceptions)... I think the thing to do here is use write_unraisable (or whatever that method is called). Since the C/API assumes that the function always succeeds, setting an exception here wouldn't get cleaned up correctly in Python code. - Robert From dagss at student.matnat.uio.no Tue Jul 29 19:33:26 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 19:33:26 +0200 Subject: [Cython] On forking CCodeWriters Message-ID: <488F5466.5030209@student.matnat.uio.no> I just implemented forking codewriters. I'll use them for temporaries, but they have other usecases as well (I used it to clean up ModuleNode.generate_c_code as well, and I might want to add (in addition) utility code registration in the code writer). The concept is this: Whenever you have a codewriter, you can call fork() on it: declarations_code = code.fork() and that will "drop off" a codewriter at the current location (there shouldn't be any perforamance hit compared with what is in Cython today, everything was buffered anyway and buffers are never copied). One can go on writing to code, and once one is done, one can write to declarations_code to insert something at the point it was forked off. Example: h_code = code.fork() ... self.generate_filename_table(code) self.generate_utility_functions(env, code, h_code) ... f = open_new_file(result.c_file) # automatically includes what was written to h_code at right location: code.copyto(f) f.close() How does this handle state? There is no general rule, this depends on the state being handled and the amounts of synchronization one is willing to go to. For "local function state" (labels and soon temporaries) I've added enter/exit_cfunc_scope which sets up a label and temp context, and this context is not available to the forked code writer, only the "main" one. This is enough as it allows me to leave off a non-temp-using code writer for the function declarations, and you cannot really synchronize the temps either. For other state (such as perhaps utility code some day) it is easier to make a global state, so that the root codewriter sets up a utility code context which is shared by every forked codewriter. Especially the AnnotationCodeWriter might need special attention in this process -- it work ok now, but that might be because I haven't really started forking yet. -- Dag Sverre From stefan_ml at behnel.de Tue Jul 29 19:48:18 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jul 2008 19:48:18 +0200 Subject: [Cython] On GetBuffer/ReleaseBuffer (attn: Stefan) In-Reply-To: <488F0447.5030601@student.matnat.uio.no> References: <488F0447.5030601@student.matnat.uio.no> Message-ID: <488F57E2.9040905@behnel.de> Hi, Dag Sverre Seljebotn wrote: > First, Stefan: In tests/run/buffer.pyx I saw references to "obj == NULL" > for locking the buffer. I can't seem find any references to this > anywhere, any hints? Ah, yes, that's a left-over from the original buffer proposal as implemented before 3.0 beta. I specifically pushed for a change here (and I'm very happy I started implementing this in time). In the resulting discussion, Greg finally got Trevis convinced that read/write locking was too complex a thing to include in the basic buffer protocol API. The fact that the test is still passing isn't a good sign, though. The implementation in Py3 is still not finished, and they're getting close to the last beta release. > PyAPI_FUNC(void) PyObject_ReleaseBuffer(PyObject *obj, Py_buffer *view); > > Actually, it says in the PEP that the function "always succeeds". But I > would guess that we don't actually enforce that for Cython-implemented > __releasebuffer__? So ideally one would raise a compiler error if there > was a chance that an exception would be propagated from ReleaseBuffer > (much like Java does for some exceptions)... I would prefer a visible compiler warning *whenever* we generate a "writeUnraisable()", not only for a buffer case, as "errors should never pass silently, unless explicitly silenced". I'm against an error, though. If someone wants to ignore the warning, we'd still end up with valid code. Stefan From stefan_ml at behnel.de Tue Jul 29 20:00:38 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jul 2008 20:00:38 +0200 Subject: [Cython] On forking CCodeWriters In-Reply-To: <488F5466.5030209@student.matnat.uio.no> References: <488F5466.5030209@student.matnat.uio.no> Message-ID: <488F5AC6.9060402@behnel.de> Hi Dag, Dag Sverre Seljebotn wrote: > I just implemented forking codewriters. I'll use them for temporaries, > but they have other usecases as well (I used it to clean up > ModuleNode.generate_c_code as well, and I might want to add (in > addition) utility code registration in the code writer). > > The concept is this: Whenever you have a codewriter, you can call fork() > on it: > > declarations_code = code.fork() > > and that will "drop off" a codewriter at the current location (there > shouldn't be any perforamance hit compared with what is in Cython today, > everything was buffered anyway and buffers are never copied). One can go > on writing to code, and once one is done, one can write to > declarations_code to insert something at the point it was forked off. If I understand this correctly, "forking" isn't the right word (and it got me very confused when I read your description). It's rather a kind of stub mechanism or a placeholder, something that keeps an insertion point alive and allows you to add code at this point even if you have already continued appending code after that point. This reminds me more of a cone, or a Deferred in Twisted terminology. Is there any other use case besides the header-/c-file overlap? Stefan From stefan_ml at behnel.de Tue Jul 29 20:29:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jul 2008 20:29:57 +0200 Subject: [Cython] -dagss merge? + what next In-Reply-To: <488EF795.7060405@student.matnat.uio.no> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> <488EF795.7060405@student.matnat.uio.no> Message-ID: <488F61A5.3050003@behnel.de> Hi, Dag Sverre Seljebotn wrote: > - shape(buf) -- Returns a tuple containing the shape of the buffer. > However, shape(buf)[0] should compile directly to __pyx_bshape0_buf. Do you mean any index access here or is the first item special in any way? > This would be the primary for-loop mechanism: > > from cython import shape import? not cimport? > - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to > allow low-level access (buffer gets a void* directly, strides and > suboffsets probably). "buffer" is actually a builtin in Py2 and was replaced by "memoryview" in Py3 (so I would consider it free for future use, especially since it's namespaced here). I assume that you meant it to provide direct access to the plain buffer "void*", right? Stefan From stefan_ml at behnel.de Tue Jul 29 20:32:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jul 2008 20:32:02 +0200 Subject: [Cython] -dagss merge? + what next In-Reply-To: <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> Message-ID: <488F6222.4000007@behnel.de> Hi, Robert Bradshaw wrote: > Good work! +1 here. :) Stefan From dagss at student.matnat.uio.no Tue Jul 29 20:42:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 20:42:00 +0200 Subject: [Cython] On GetBuffer/ReleaseBuffer (attn: Stefan) In-Reply-To: <488F57E2.9040905@behnel.de> References: <488F0447.5030601@student.matnat.uio.no> <488F57E2.9040905@behnel.de> Message-ID: <488F6478.1000604@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> First, Stefan: In tests/run/buffer.pyx I saw references to "obj == NULL" >> for locking the buffer. I can't seem find any references to this >> anywhere, any hints? > > Ah, yes, that's a left-over from the original buffer proposal as implemented > before 3.0 beta. I specifically pushed for a change here (and I'm very happy I > started implementing this in time). In the resulting discussion, Greg finally > got Trevis convinced that read/write locking was too complex a thing to > include in the basic buffer protocol API. > > The fact that the test is still passing isn't a good sign, though. The > implementation in Py3 is still not finished, and they're getting close to the > last beta release. Thanks, I'll happily ignore it then. > I would prefer a visible compiler warning *whenever* we generate a > "writeUnraisable()", not only for a buffer case, as "errors should never pass > silently, unless explicitly silenced". I'm against an error, though. If > someone wants to ignore the warning, we'd still end up with valid code. Well, you can always require that one writes try: ... except: pass and so it is explicit what is going on in the code.... (if I'm not mistaken, I actually haven't heard of writeUnraisable until now so haven't looked into it). -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 20:42:10 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 20:42:10 +0200 Subject: [Cython] -dagss merge? + what next In-Reply-To: <488F61A5.3050003@behnel.de> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> <488EF795.7060405@student.matnat.uio.no> <488F61A5.3050003@behnel.de> Message-ID: <488F6482.7030404@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> - shape(buf) -- Returns a tuple containing the shape of the buffer. >> However, shape(buf)[0] should compile directly to __pyx_bshape0_buf. > > Do you mean any index access here or is the first item special in any way? Any index (and compilation error if you use an index >= ndim). > > >> This would be the primary for-loop mechanism: >> >> from cython import shape > > import? not cimport? cimport. > > >> - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to >> allow low-level access (buffer gets a void* directly, strides and >> suboffsets probably). > > "buffer" is actually a builtin in Py2 and was replaced by "memoryview" in Py3 > (so I would consider it free for future use, especially since it's namespaced > here). > > I assume that you meant it to provide direct access to the plain buffer > "void*", right? Yes. -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 20:56:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 20:56:44 +0200 Subject: [Cython] On forking CCodeWriters In-Reply-To: <488F5AC6.9060402@behnel.de> References: <488F5466.5030209@student.matnat.uio.no> <488F5AC6.9060402@behnel.de> Message-ID: <488F67EC.3030904@student.matnat.uio.no> Stefan Behnel wrote: > Hi Dag, > > Dag Sverre Seljebotn wrote: >> I just implemented forking codewriters. I'll use them for temporaries, >> but they have other usecases as well (I used it to clean up >> ModuleNode.generate_c_code as well, and I might want to add (in >> addition) utility code registration in the code writer). >> >> The concept is this: Whenever you have a codewriter, you can call fork() >> on it: >> >> declarations_code = code.fork() >> >> and that will "drop off" a codewriter at the current location (there >> shouldn't be any perforamance hit compared with what is in Cython today, >> everything was buffered anyway and buffers are never copied). One can go >> on writing to code, and once one is done, one can write to >> declarations_code to insert something at the point it was forked off. > > If I understand this correctly, "forking" isn't the right word (and it got me > very confused when I read your description). It's rather a kind of stub > mechanism or a placeholder, something that keeps an insertion point alive and > allows you to add code at this point even if you have already continued > appending code after that point. You're probably right. It is a concept that developed and what I started with was more like a fork than this :-) Hmm. It is kind of similar though .. a lot of state is copied in ways similar to process forking. But I see that the name is not good. Suggestions about the exact call name very welcome .. I'll leave it for a few days and see what feels natural before I go changing it. > This reminds me more of a cone, or a Deferred in Twisted terminology. Just to be clear, there's no callback here. > Is there any other use case besides the header-/c-file overlap? Yes, my primary one: Add allocate_temp and release_temp to CCodeWriter (see thread with Greg on temps). allocate_temp and release_temp calls are interleaved with putln calls and is held on to for the exact right lines of code. As you need to know how many temps and of which type they are before entering a function...this was the primary motivation for this addition to CCodeWriter. (This temp system will coexist with the one in the analysis phase for now. The two are not totally overlapping concepts, there must be something left prior to code generation...see discussion with Greg). This will make my buffer code (and much other code, if one wants to) a bit cleaner, but more importantly get the door open for moving the calculation of result_code to the code generation phase. Which will make a lot of stuff easier (closures for one thing). Also I tended to create a lot of "utility code" (other functions) on the fly in the buffer code (as they are instantiated once per dtype), and it would be natural to a) be able to call use_utility_code during code generation rather than on the scope, and b) use a CodeWriter instance to write dynamic "utility code" with, rather than the cruder string concatanation I currently do. This work will be useful for this as well, but doing this is much less of a priority. -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 21:16:29 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 21:16:29 +0200 Subject: [Cython] On forking CCodeWriters In-Reply-To: <488F67EC.3030904@student.matnat.uio.no> References: <488F5466.5030209@student.matnat.uio.no> <488F5AC6.9060402@behnel.de> <488F67EC.3030904@student.matnat.uio.no> Message-ID: <488F6C8D.5020706@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi Dag, >> >> Dag Sverre Seljebotn wrote: >>> I just implemented forking codewriters. I'll use them for temporaries, >>> but they have other usecases as well (I used it to clean up >>> ModuleNode.generate_c_code as well, and I might want to add (in >>> addition) utility code registration in the code writer). >>> >>> The concept is this: Whenever you have a codewriter, you can call fork() >>> on it: >>> >>> declarations_code = code.fork() >>> >>> and that will "drop off" a codewriter at the current location (there >>> shouldn't be any perforamance hit compared with what is in Cython today, >>> everything was buffered anyway and buffers are never copied). One can go >>> on writing to code, and once one is done, one can write to >>> declarations_code to insert something at the point it was forked off. >> If I understand this correctly, "forking" isn't the right word (and it got me >> very confused when I read your description). It's rather a kind of stub >> mechanism or a placeholder, something that keeps an insertion point alive and >> allows you to add code at this point even if you have already continued >> appending code after that point. > > You're probably right. It is a concept that developed and what I started > with was more like a fork than this :-) > > Hmm. It is kind of similar though .. a lot of state is copied in ways > similar to process forking. But I see that the name is not good. > > Suggestions about the exact call name very welcome .. I'll leave it for > a few days and see what feels natural before I go changing it. This is no longer true, it started irritating me incredibly when I started thinking about it. It is now code.delayed_writing(). But I can change it again. Also I will probably add code.new_fragment() and code.put_fragment() as well (fragments will not inherit indentation state; these can be used for utility code generation). -- Dag Sverre From stefan_ml at behnel.de Tue Jul 29 21:23:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jul 2008 21:23:02 +0200 Subject: [Cython] On forking CCodeWriters In-Reply-To: <488F67EC.3030904@student.matnat.uio.no> References: <488F5466.5030209@student.matnat.uio.no> <488F5AC6.9060402@behnel.de> <488F67EC.3030904@student.matnat.uio.no> Message-ID: <488F6E16.2050602@behnel.de> Hi, Dag Sverre Seljebotn wrote: > would be natural to a) be able to call use_utility_code during code > generation rather than on the scope This is the one convincing argument I needed. :) Stefan From dagss at student.matnat.uio.no Tue Jul 29 21:56:17 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 21:56:17 +0200 Subject: [Cython] -dagss merge? + what next In-Reply-To: References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> <488EF795.7060405@student.matnat.uio.no> Message-ID: <488F75E1.7070409@student.matnat.uio.no> Robert Bradshaw wrote: > On Jul 29, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote: >>> >>>> Those are the big tasks, but there are also a few smaller nice- >>>> haves I >>>> can go for (a "cython.shape" builtin that allows you to retrieve the >>>> shape information of a buffer, and so on). >>> I saw you mention shape (and other "magic" functions), and I'm >>> curious. Could you elaborate? >> When acquiring a buffer, you get hold of a lot of information that is >> not currently exported -- shape is most important. For NumPy this >> is ok >> as ndarray exports its shape field for efficient access, but without >> another mechanism you can't write generic algorithms that work with >> all >> buffers without passing the shape explicitly. This even leads to less >> clean test case code... >> >> I haven't thought much about the list (add as needed) but something >> like: >> >> - shape(buf) -- Returns a tuple containing the shape of the buffer. >> However, shape(buf)[0] should compile directly to >> __pyx_bshape0_buf. (I >> thought about shape(buf, 0) as well but decided against, how about >> you?) >> >> This would be the primary for-loop mechanism: >> >> from cython import shape >> ... >> cdef int i >> for i in range(shape(buf)[0]): buf[i] = 9 >> >> - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to >> allow low-level access (buffer gets a void* directly, strides and >> suboffsets probably). For contiguous arrays this could be more >> efficient >> (the proper high-level approach for such things are iterators). The >> idea >> is that even if you write low-level code, you could use the buffer >> syntax so that you don't need to worry about acquisition. >> >> - ndim(buf) -- If the ndim-at-compile-time-restriction is lifted >> (because you have low-level access or iterators you can use instead of >> indexing) then this will be useful. >> >> Finally, if I get as far as iterators (I won't!), cython.ndenumerate >> could create very efficient iteration code. (Probably a NumPy-specific >> iterator could be less work though...if I can't just copy the approach >> of NumPy iterators directly...NumPy comes with a seperate n- >> dimensional >> iterator class that allows you to write ndim-generic code.) > > Ah, I see. These could be very useful, though I'm not sure we should > be augmenting the namespace to provide them (perhaps make them magic > methods of a buffer object instead?) What are the disadvantages of adding a cimportable cython namespace for this? I don't think we can get around it personally -- in fact I was thinking of using it for the pragma code [1]. Meanwhile, attributes can always collide. Especially having "object[int]" not do an attribute lookup for every attribute access would be a big loss I feel. What about adding syntax in "cdef extern class ndarray" to provide these functions in addition, so that the class author is in control and can make the decision. That is in line with another feature I'd like feedback on, starting another thread. [1]: - cython.unchecked_getitem(buf, i, j) and cython.unchecked_cast(...) - with cython.noboundschecking: ... - @cython.noboundschecking def f(): ... etc. Also it could be a primary vehicle (in a future far off) of making Cython more namespace-compatible with Python: with cython.nogil: -- Dag Sverre From dagss at student.matnat.uio.no Tue Jul 29 22:00:57 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 29 Jul 2008 22:00:57 +0200 Subject: [Cython] Buffer default options Message-ID: <488F76F9.4090308@student.matnat.uio.no> There's a lot of email today but as I finished one part I'm staking out the course again. I'm thinking about letting the class author specify default buffer options. Something like this: cdef extern class Image ...: __cythonbufferdefaults__ = {'ndim' : 2} __cythonbuffermandatory__ = {'dtype': unsigned char, indirect=True} __cythonbufferalways__ = True [1] I think this makes a lot of sense, as the author of the Image class might export exactly this kind of buffer and nothing else. With the above settings, one would automatically get efficient indexing on all "cdef Image" instances. NumPy is kind of special in the flexibility it provides, but even there, setting indirect=False can provide a speedup transparently. Once the indirect option is implemented at all, that is :-) but that *must* happen to make NumPy as efficient as possible. As for priority, I guess this is below most of what I've talked about so far. Though it might suddenly look like a low hanging fruit that I go for when I need a break from other stuff. [1] (Well, I consider custom parsing for that last line better than inventing a wholly new syntax...and we've been talking about new Python-style type references as well, which fits in here) -- Dag Sverre From greg.ewing at canterbury.ac.nz Wed Jul 30 02:28:59 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 30 Jul 2008 12:28:59 +1200 Subject: [Cython] On GetBuffer/ReleaseBuffer (attn: Stefan) In-Reply-To: <488F0447.5030601@student.matnat.uio.no> References: <488F0447.5030601@student.matnat.uio.no> Message-ID: <488FB5CB.6090505@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > So ideally one would raise a compiler error if there > was a chance that an exception would be propagated from ReleaseBuffer I think it would be better to insert a call to Pyx_WriteUnraisable in that case. -- Greg From robertwb at math.washington.edu Wed Jul 30 03:45:44 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jul 2008 18:45:44 -0700 Subject: [Cython] On GetBuffer/ReleaseBuffer (attn: Stefan) In-Reply-To: <488FB5CB.6090505@canterbury.ac.nz> References: <488F0447.5030601@student.matnat.uio.no> <488FB5CB.6090505@canterbury.ac.nz> Message-ID: On Jul 29, 2008, at 5:28 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> So ideally one would raise a compiler error if there >> was a chance that an exception would be propagated from ReleaseBuffer > > I think it would be better to insert a call to > Pyx_WriteUnraisable in that case. I am of the same opinion--though I like the idea of a warning every time Pyx_WriteUnraisable is needed. - Robert From robertwb at math.washington.edu Wed Jul 30 06:05:38 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jul 2008 21:05:38 -0700 Subject: [Cython] On forking CCodeWriters In-Reply-To: <488F6C8D.5020706@student.matnat.uio.no> References: <488F5466.5030209@student.matnat.uio.no> <488F5AC6.9060402@behnel.de> <488F67EC.3030904@student.matnat.uio.no> <488F6C8D.5020706@student.matnat.uio.no> Message-ID: <3B24C13B-C408-46D5-AC54-4E8B19D803A4@math.washington.edu> On Jul 29, 2008, at 12:16 PM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Hi Dag, >>> >>> Dag Sverre Seljebotn wrote: >>>> I just implemented forking codewriters. I'll use them for >>>> temporaries, >>>> but they have other usecases as well (I used it to clean up >>>> ModuleNode.generate_c_code as well, and I might want to add (in >>>> addition) utility code registration in the code writer). >>>> >>>> The concept is this: Whenever you have a codewriter, you can >>>> call fork() >>>> on it: >>>> >>>> declarations_code = code.fork() >>>> >>>> and that will "drop off" a codewriter at the current location >>>> (there >>>> shouldn't be any perforamance hit compared with what is in >>>> Cython today, >>>> everything was buffered anyway and buffers are never copied). >>>> One can go >>>> on writing to code, and once one is done, one can write to >>>> declarations_code to insert something at the point it was forked >>>> off. >>> If I understand this correctly, "forking" isn't the right word >>> (and it got me >>> very confused when I read your description). It's rather a kind >>> of stub >>> mechanism or a placeholder, something that keeps an insertion >>> point alive and >>> allows you to add code at this point even if you have already >>> continued >>> appending code after that point. >> >> You're probably right. It is a concept that developed and what I >> started >> with was more like a fork than this :-) >> >> Hmm. It is kind of similar though .. a lot of state is copied in ways >> similar to process forking. But I see that the name is not good. >> >> Suggestions about the exact call name very welcome .. I'll leave >> it for >> a few days and see what feels natural before I go changing it. > > This is no longer true, it started irritating me incredibly when I > started thinking about it. It is now code.delayed_writing(). But I can > change it again. I think the idea of forking the code writer is a great one. In terms of naming, perhaps "code.insertion_point()" or something like that. I'm not sure with delayed_writing() because it's unclear whether or not the returned object is delayed, or the current object is delayed. - Robert From robertwb at math.washington.edu Wed Jul 30 06:11:49 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jul 2008 21:11:49 -0700 Subject: [Cython] -dagss merge? + what next In-Reply-To: <488F75E1.7070409@student.matnat.uio.no> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> <488EF795.7060405@student.matnat.uio.no> <488F75E1.7070409@student.matnat.uio.no> Message-ID: <42D740AF-702F-46AA-BEB7-1CA4477E239E@math.washington.edu> On Jul 29, 2008, at 12:56 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Jul 29, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> On Jul 29, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote: >>>> >>>>> Those are the big tasks, but there are also a few smaller nice- >>>>> haves I >>>>> can go for (a "cython.shape" builtin that allows you to >>>>> retrieve the >>>>> shape information of a buffer, and so on). >>>> I saw you mention shape (and other "magic" functions), and I'm >>>> curious. Could you elaborate? >>> When acquiring a buffer, you get hold of a lot of information >>> that is >>> not currently exported -- shape is most important. For NumPy this >>> is ok >>> as ndarray exports its shape field for efficient access, but without >>> another mechanism you can't write generic algorithms that work with >>> all >>> buffers without passing the shape explicitly. This even leads to >>> less >>> clean test case code... >>> >>> I haven't thought much about the list (add as needed) but something >>> like: >>> >>> - shape(buf) -- Returns a tuple containing the shape of the buffer. >>> However, shape(buf)[0] should compile directly to >>> __pyx_bshape0_buf. (I >>> thought about shape(buf, 0) as well but decided against, how about >>> you?) >>> >>> This would be the primary for-loop mechanism: >>> >>> from cython import shape >>> ... >>> cdef int i >>> for i in range(shape(buf)[0]): buf[i] = 9 >>> >>> - buffer(buf), strides(buf), suboffsets(buf) -- Could have these to >>> allow low-level access (buffer gets a void* directly, strides and >>> suboffsets probably). For contiguous arrays this could be more >>> efficient >>> (the proper high-level approach for such things are iterators). The >>> idea >>> is that even if you write low-level code, you could use the buffer >>> syntax so that you don't need to worry about acquisition. >>> >>> - ndim(buf) -- If the ndim-at-compile-time-restriction is lifted >>> (because you have low-level access or iterators you can use >>> instead of >>> indexing) then this will be useful. >>> >>> Finally, if I get as far as iterators (I won't!), cython.ndenumerate >>> could create very efficient iteration code. (Probably a NumPy- >>> specific >>> iterator could be less work though...if I can't just copy the >>> approach >>> of NumPy iterators directly...NumPy comes with a seperate n- >>> dimensional >>> iterator class that allows you to write ndim-generic code.) >> >> Ah, I see. These could be very useful, though I'm not sure we should >> be augmenting the namespace to provide them (perhaps make them magic >> methods of a buffer object instead?) > > What are the disadvantages of adding a cimportable cython namespace > for > this? I don't think we can get around it personally -- in fact I was > thinking of using it for the pragma code [1]. OK, for some reason I was missing the fact that you were cimporting them (I thought you were suggesting they be builtins like len). I think it should somehow be clear that these operate on buffers, either by placing them in cython.buffer or calling them buffer_shape, etc. (I prefer the former). > Meanwhile, attributes can always collide. Especially having > "object[int]" not do an attribute lookup for every attribute access > would be a big loss I feel. > > What about adding syntax in "cdef extern class ndarray" to provide > these > functions in addition, so that the class author is in control and can > make the decision. > > That is in line with another feature I'd like feedback on, starting > another thread. > > [1]: > > - cython.unchecked_getitem(buf, i, j) and cython.unchecked_cast(...) > - with cython.noboundschecking: ... > - @cython.noboundschecking > def f(): ... > > etc. Also it could be a primary vehicle (in a future far off) of > making > Cython more namespace-compatible with Python: > > with cython.nogil: Yeah, I like this general direction, though your first example is a bit verbose. Again, something to be explicit about the fact that these are buffer settings would be good. - Robert From robertwb at math.washington.edu Wed Jul 30 06:15:57 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jul 2008 21:15:57 -0700 Subject: [Cython] Buffer default options In-Reply-To: <488F76F9.4090308@student.matnat.uio.no> References: <488F76F9.4090308@student.matnat.uio.no> Message-ID: <301FA75B-1CCF-41C7-A0A3-9A58ABBB8928@math.washington.edu> On Jul 29, 2008, at 1:00 PM, Dag Sverre Seljebotn wrote: > There's a lot of email today but as I finished one part I'm staking > out > the course again. > > I'm thinking about letting the class author specify default buffer > options. Something like this: > > cdef extern class Image ...: > __cythonbufferdefaults__ = {'ndim' : 2} > __cythonbuffermandatory__ = {'dtype': unsigned char, indirect=True} > __cythonbufferalways__ = True > > [1] > > I think this makes a lot of sense, as the author of the Image class > might export exactly this kind of buffer and nothing else. With the > above settings, one would automatically get efficient indexing on all > "cdef Image" instances. > > NumPy is kind of special in the flexibility it provides, but even > there, > setting indirect=False can provide a speedup transparently. Once the > indirect option is implemented at all, that is :-) but that *must* > happen to make NumPy as efficient as possible. > > As for priority, I guess this is below most of what I've talked > about so > far. Though it might suddenly look like a low hanging fruit that I go > for when I need a break from other stuff. > > [1] (Well, I consider custom parsing for that last line better than > inventing a wholly new syntax...and we've been talking about new > Python-style type references as well, which fits in here) This almost seems to magical for me, if one wants to use a buffer perhaps it is better to be explicit. But I'm curious to hear what other people think. - Robert From stefan_ml at behnel.de Wed Jul 30 08:05:35 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jul 2008 08:05:35 +0200 Subject: [Cython] -dagss merge? + what next In-Reply-To: <42D740AF-702F-46AA-BEB7-1CA4477E239E@math.washington.edu> References: <488EE1F7.8060905@student.matnat.uio.no> <13F84849-5F22-49C8-99EA-12E37342FE23@math.washington.edu> <488EF795.7060405@student.matnat.uio.no> <488F75E1.7070409@student.matnat.uio.no> <42D740AF-702F-46AA-BEB7-1CA4477E239E@math.washington.edu> Message-ID: <489004AF.3000306@behnel.de> Hi, Robert Bradshaw wrote: > On Jul 29, 2008, at 12:56 PM, Dag Sverre Seljebotn wrote: >>>> - shape(buf) >>>> - buffer(buf), strides(buf), suboffsets(buf) >>>> - ndim(buf) > > OK, for some reason I was missing the fact that you were cimporting > them (I thought you were suggesting they be builtins like len). I > think it should somehow be clear that these operate on buffers, > either by placing them in cython.buffer or calling them buffer_shape, > etc. (I prefer the former). +1, this looks perfectly fine to me: from cython.buffer cimport shape or from cython cimport buffer buffer.shape(...) Although from cython cimport buffer buffer.buffer(...) looks a little funny. Maybe "buffer.plainbuffer()" is clearer - "buffer" isn't the struct field name anyway. >> - cython.unchecked_getitem(buf, i, j) and cython.unchecked_cast(...) >> - with cython.noboundschecking: ... >> - @cython.noboundschecking >> def f(): ... >> >> etc. Also it could be a primary vehicle (in a future far off) of >> making >> Cython more namespace-compatible with Python: >> >> with cython.nogil: > > Yeah, I like this general direction, though your first example is a > bit verbose. It's pretty explicit, though. I don't think it would help readability if you made it shorter. And if users really think they are smart enough, there's always from cython cimport unchecked_getitem as ugi :) Stefan From stefan_ml at behnel.de Wed Jul 30 08:07:33 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jul 2008 08:07:33 +0200 Subject: [Cython] On forking CCodeWriters In-Reply-To: <3B24C13B-C408-46D5-AC54-4E8B19D803A4@math.washington.edu> References: <488F5466.5030209@student.matnat.uio.no> <488F5AC6.9060402@behnel.de> <488F67EC.3030904@student.matnat.uio.no> <488F6C8D.5020706@student.matnat.uio.no> <3B24C13B-C408-46D5-AC54-4E8B19D803A4@math.washington.edu> Message-ID: <48900525.1090607@behnel.de> Hi, Robert Bradshaw wrote: > I think the idea of forking the code writer is a great one. In terms > of naming, perhaps "code.insertion_point()" or something like that. +1, reads well. Stefan From dagss at student.matnat.uio.no Wed Jul 30 09:14:53 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 30 Jul 2008 09:14:53 +0200 (CEST) Subject: [Cython] Buffer default options In-Reply-To: <301FA75B-1CCF-41C7-A0A3-9A58ABBB8928@math.washington.edu> References: <488F76F9.4090308@student.matnat.uio.no> <301FA75B-1CCF-41C7-A0A3-9A58ABBB8928@math.washington.edu> Message-ID: <54679.193.157.229.67.1217402093.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Jul 29, 2008, at 1:00 PM, Dag Sverre Seljebotn wrote: > >> There's a lot of email today but as I finished one part I'm staking >> out >> the course again. >> >> I'm thinking about letting the class author specify default buffer >> options. Something like this: >> >> cdef extern class Image ...: >> __cythonbufferdefaults__ = {'ndim' : 2} >> __cythonbuffermandatory__ = {'dtype': unsigned char, indirect=True} >> __cythonbufferalways__ = True >> >> [1] >> >> I think this makes a lot of sense, as the author of the Image class >> might export exactly this kind of buffer and nothing else. With the >> above settings, one would automatically get efficient indexing on all >> "cdef Image" instances. >> >> NumPy is kind of special in the flexibility it provides, but even >> there, >> setting indirect=False can provide a speedup transparently. Once the >> indirect option is implemented at all, that is :-) but that *must* >> happen to make NumPy as efficient as possible. >> >> As for priority, I guess this is below most of what I've talked >> about so >> far. Though it might suddenly look like a low hanging fruit that I go >> for when I need a break from other stuff. >> >> [1] (Well, I consider custom parsing for that last line better than >> inventing a wholly new syntax...and we've been talking about new >> Python-style type references as well, which fits in here) > > This almost seems to magical for me, if one wants to use a buffer > perhaps it is better to be explicit. But I'm curious to hear what > other people think. Let me just give one more concrete example for NumPy then. If you do cdef ndarray[int, 2] buf currently, then it is going to create code for checking whether it should do indirect access, which is one if-test per dimension per lookup -- and you *know* that for ndarrays, you can always get around with strided access, i.e. something like cdef ndarray[int, 2, 'strided'] buf Now, I would kind of like to be able to tell people to use "object[int, 2]" to write a generic buffer algorithm, but "ndarray[int, 2]" to write something that only works with NumPy in an optimized fashion, and that is it -- and the proposal kind of grew out of that, it is a way of letting the users not have to type mode="strided" all the time. (BTW, do you like mode=different strings for this, or should I go with "strided=True", "c=True", "fortran=True", etc? There will be two modes at first: "full" and "strided", although if cython.buffer.bufptr is introduced than "c", "fortran", "contig" will be useful as well.) Of course I could only do this for the mode, but this doesn't seem to be a special case -- though most buffer usecases seems to be even more fixed (if you have a JPEG library why bother to specify the ndim and so on). As for an option for automatically retrieving a buffer, I might agree. Also I see the downside that it allows syntax like "JPEGImage[]" and "MultiDimImage[3]". Dag Sverre From robertwb at math.washington.edu Wed Jul 30 09:22:12 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 30 Jul 2008 00:22:12 -0700 Subject: [Cython] long literal integers Message-ID: <25BC3B81-9325-4D59-9CCD-6F47D617B434@math.washington.edu> Currently there is no way to output literal C long, long long values, or unsigned long values. Also, large integer values used in a python context are truncated. It does, however, parse a trailing L to create a Python long. I propose that we update this to pass the trailing L (and LL) to the underlying C code, as well as accepting the unsigned suffix (also passed to C the code), and covert large integer literals into Python longs without truncating. This is slightly backwards incompatible, so I wanted to get people's feelings on this before making the change. - Robert From robertwb at math.washington.edu Wed Jul 30 09:33:23 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 30 Jul 2008 00:33:23 -0700 Subject: [Cython] Buffer default options In-Reply-To: <54679.193.157.229.67.1217402093.squirrel@webmail.uio.no> References: <488F76F9.4090308@student.matnat.uio.no> <301FA75B-1CCF-41C7-A0A3-9A58ABBB8928@math.washington.edu> <54679.193.157.229.67.1217402093.squirrel@webmail.uio.no> Message-ID: <09EC153B-52DB-4F32-9889-1076714D89BB@math.washington.edu> On Jul 30, 2008, at 12:14 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Jul 29, 2008, at 1:00 PM, Dag Sverre Seljebotn wrote: >> >>> There's a lot of email today but as I finished one part I'm staking >>> out >>> the course again. >>> >>> I'm thinking about letting the class author specify default buffer >>> options. Something like this: >>> >>> cdef extern class Image ...: >>> __cythonbufferdefaults__ = {'ndim' : 2} >>> __cythonbuffermandatory__ = {'dtype': unsigned char, >>> indirect=True} >>> __cythonbufferalways__ = True >>> >>> [1] >>> >>> I think this makes a lot of sense, as the author of the Image class >>> might export exactly this kind of buffer and nothing else. With the >>> above settings, one would automatically get efficient indexing on >>> all >>> "cdef Image" instances. >>> >>> NumPy is kind of special in the flexibility it provides, but even >>> there, >>> setting indirect=False can provide a speedup transparently. Once the >>> indirect option is implemented at all, that is :-) but that *must* >>> happen to make NumPy as efficient as possible. >>> >>> As for priority, I guess this is below most of what I've talked >>> about so >>> far. Though it might suddenly look like a low hanging fruit that >>> I go >>> for when I need a break from other stuff. >>> >>> [1] (Well, I consider custom parsing for that last line better than >>> inventing a wholly new syntax...and we've been talking about new >>> Python-style type references as well, which fits in here) >> >> This almost seems to magical for me, if one wants to use a buffer >> perhaps it is better to be explicit. But I'm curious to hear what >> other people think. > > Let me just give one more concrete example for NumPy then. If you do > > cdef ndarray[int, 2] buf > > currently, then it is going to create code for checking whether it > should > do indirect access, which is one if-test per dimension per lookup > -- and > you *know* that for ndarrays, you can always get around with strided > access, i.e. something like > > cdef ndarray[int, 2, 'strided'] buf > > Now, I would kind of like to be able to tell people to use "object > [int, > 2]" to write a generic buffer algorithm, but "ndarray[int, 2]" to > write > something that only works with NumPy in an optimized fashion, and > that is > it -- and the proposal kind of grew out of that, it is a way of > letting > the users not have to type mode="strided" all the time. > > (BTW, do you like mode=different strings for this, or should I go with > "strided=True", "c=True", "fortran=True", etc? I like just providing strings, which I am assuming map to access flags. > There will be two modes at > first: "full" and "strided", although if cython.buffer.bufptr is > introduced than "c", "fortran", "contig" will be useful as well.) > > Of course I could only do this for the mode, but this doesn't seem > to be a > special case -- > though most buffer usecases seems to be even more fixed (if you have a > JPEG library why bother to specify the ndim and so on). > > As for an option for automatically retrieving a buffer, I might agree. > Also I see the downside that it allows syntax like "JPEGImage[]" and > "MultiDimImage[3]". You have me convinced that providing defaults is a good thing (and I agree many (most?) libraries/classes will have a fixed dimension/ type). The __cythonbuffermandatory__ just to turn what would be a runtime error into a compile time error, right? It may fail to be true for subclasses. __cythonbufferalways__ can be assumed--if there is enough (default) information to provide a buffer, then do it, otherwise don't. - Robert From stefan_ml at behnel.de Wed Jul 30 09:54:32 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jul 2008 09:54:32 +0200 (CEST) Subject: [Cython] long literal integers In-Reply-To: <25BC3B81-9325-4D59-9CCD-6F47D617B434@math.washington.edu> References: <25BC3B81-9325-4D59-9CCD-6F47D617B434@math.washington.edu> Message-ID: <37863.213.61.181.86.1217404472.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Currently there is no way to output literal C long, long long values, > or unsigned long values. Also, large integer values used in a python > context are truncated. It does, however, parse a trailing L to create > a Python long. > > I propose that we update this to pass the trailing L (and LL) to the > underlying C code, as well as accepting the unsigned suffix (also > passed to C the code), and covert large integer literals into Python > longs without truncating. This is slightly backwards incompatible, so > I wanted to get people's feelings on this before making the change. It may be incompatible, but it's definitely the expected behaviour. I would consider code pretty much broken if it relies on truncated literals. I think it's generally a good idea to have literals start off as C literals and convert them at need (either at compile time or runtime). This seems to apply here, too. Stefan From dagss at student.matnat.uio.no Wed Jul 30 12:23:27 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 30 Jul 2008 12:23:27 +0200 Subject: [Cython] Temps in CCodewriter Message-ID: <4890411F.9080302@student.matnat.uio.no> OK, temps in CCodeWriter is now available in -dagss. (As for status, it was Greg's idea, Robert seemed to approve and I haven't heard anything from Stefan on it but he hasn't opposed it either.) Temps created during code generation live in another namespace than temps created during analysis, the two don't overlap. Example: tmp = code.func.allocate_temp(PyrexTypes.c_int_type) code.putln('%s = %s;' % (tmp, ...)) .... code.func.release_temp(tmp) The reason I want "code.func" is to make it explicit that this belongs to a seperate function context that is a) *not* given on to created insertion_points (there would be no way of having this always work), b) must be allocated using code.enter/exit_cfunc_scope. I.e b = code.insertion_point() b.func.allocate_temp(...) # will raise error, b.func is None (In FuncDefNode.generate_function_definition I then have: tempvardecl_code = code.insertion_point() ... do everything that will allocat temps on code ... tempvardecl_code.put_temp_declarations(code.func) ) Labels also live in code.func but I've created backwards-compatible redirectors for those so no code needs to change. I won't do any actual result_code refactoring at this stage as I don't need it, though it is a natural next step. First one should try to merge in Greg's work. I think this is stable for a pull as well -- I'll just start to mark the latest stable revision on my status wiki page. -- Dag Sverre From stefan_ml at behnel.de Wed Jul 30 13:01:14 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jul 2008 13:01:14 +0200 (CEST) Subject: [Cython] Temps in CCodewriter In-Reply-To: <4890411F.9080302@student.matnat.uio.no> References: <4890411F.9080302@student.matnat.uio.no> Message-ID: <60467.213.61.181.86.1217415674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi Dag, Dag Sverre Seljebotn wrote: > OK, temps in CCodeWriter is now available in -dagss. (As for status, it > was Greg's idea, Robert seemed to approve and I haven't heard anything > from Stefan on it but he hasn't opposed it either.) I never really managed to understand the details of temp allocation, and I was always happy to get code working that required them. So I'm totally +1 for getting this simplified. And (as Robert said already), moving temp allocations into the code generation phase makes perfect sense to me. I assume that this also uses your code insertion modification to insert temp declarations at the top of the function code? Is there actually any support for block-local temps? I know, this is non-trivial as it impacts exception handling (e.g. duplicated error lables with and without temp cleanup), but it would make sense e.g. for nogil functions. Stefan From dagss at student.matnat.uio.no Wed Jul 30 13:10:07 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 30 Jul 2008 13:10:07 +0200 Subject: [Cython] Temps in CCodewriter In-Reply-To: <60467.213.61.181.86.1217415674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <4890411F.9080302@student.matnat.uio.no> <60467.213.61.181.86.1217415674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48904C0F.2090800@student.matnat.uio.no> Stefan Behnel wrote: > Hi Dag, > > Dag Sverre Seljebotn wrote: >> OK, temps in CCodeWriter is now available in -dagss. (As for status, it >> was Greg's idea, Robert seemed to approve and I haven't heard anything >> from Stefan on it but he hasn't opposed it either.) > > I never really managed to understand the details of temp allocation, and I > was always happy to get code working that required them. So I'm totally +1 > for getting this simplified. And (as Robert said already), moving temp > allocations into the code generation phase makes perfect sense to me. > > I assume that this also uses your code insertion modification to insert > temp declarations at the top of the function code? Yes. > Is there actually any support for block-local temps? I know, this is > non-trivial as it impacts exception handling (e.g. duplicated error lables > with and without temp cleanup), but it would make sense e.g. for nogil > functions. No, but there's no obstacles to implement it from the perspective of CCodeWriter if one decides that it is worth the effort. -- Dag Sverre From nicole at cats-muvva.net Wed Jul 30 14:06:53 2008 From: nicole at cats-muvva.net (Nicole King) Date: Wed, 30 Jul 2008 13:06:53 +0100 Subject: [Cython] Opaque Pointers in Wrapping a C API Message-ID: <200807301306.53534.nicole@cats-muvva.net> Please forgive me if this is an old question. I have been unable to make any progress on this despite Google. I have a small program: cdef extern from *: ctypedef struct opaque_pointer: pass opaque_pointer* init(char *tablename) void finish(opaque_pointer** h) cdef class a_class: cdef opaque_pointer* the_handle def __init__(table_name): the_handle = init(table_name) def __dealloc__(self): finish(&self.the_handle) On compiling it, this happens: nicole at hosts-2:4:~/ $ cython libiptc.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... void finish(opaque_pointer** h) cdef class a_class: cdef opaque_pointer* the_handle def __init__(table_name): the_handle = init(table_name) ^ ------------------------------------------------------------ /home/nicole/firewall/libiptc.pyx:10:21: Cannot convert 'opaque_pointer *' to Python object nicole at hosts-2:4:~/ $ I've obviously misunderstood something, but I'm blessed if I can understand what is going wrong. I'm a pretty experienced programmer; picked up python in a couple of days, so it must be a pretty fundamental lack on my part. I'd be grateful for any assistance you could give. Best regards Nicole King From stefan_ml at behnel.de Wed Jul 30 14:56:25 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jul 2008 14:56:25 +0200 (CEST) Subject: [Cython] Opaque Pointers in Wrapping a C API In-Reply-To: <200807301306.53534.nicole@cats-muvva.net> References: <200807301306.53534.nicole@cats-muvva.net> Message-ID: <55004.213.61.181.86.1217422585.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Nicole King schrieb: > Please forgive me if this is an old question. I have been unable to make > any progress on this despite Google. > > I have a small program: > > cdef extern from *: > ctypedef struct opaque_pointer: > pass > opaque_pointer* init(char *tablename) > void finish(opaque_pointer** h) > > cdef class a_class: > cdef opaque_pointer* the_handle > def __init__(table_name): > the_handle = init(table_name) This should read: self.the_handle = ... Stefan From dagss at student.matnat.uio.no Wed Jul 30 16:49:28 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 30 Jul 2008 16:49:28 +0200 Subject: [Cython] Buffer default options In-Reply-To: <09EC153B-52DB-4F32-9889-1076714D89BB@math.washington.edu> References: <488F76F9.4090308@student.matnat.uio.no> <301FA75B-1CCF-41C7-A0A3-9A58ABBB8928@math.washington.edu> <54679.193.157.229.67.1217402093.squirrel@webmail.uio.no> <09EC153B-52DB-4F32-9889-1076714D89BB@math.washington.edu> Message-ID: <48907F78.2020008@student.matnat.uio.no> Robert Bradshaw wrote: > On Jul 30, 2008, at 12:14 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On Jul 29, 2008, at 1:00 PM, Dag Sverre Seljebotn wrote: >>> >>>> There's a lot of email today but as I finished one part I'm staking >>>> out >>>> the course again. >>>> >>>> I'm thinking about letting the class author specify default buffer >>>> options. Something like this: >>>> >>>> cdef extern class Image ...: >>>> __cythonbufferdefaults__ = {'ndim' : 2} >>>> __cythonbuffermandatory__ = {'dtype': unsigned char, >>>> indirect=True} >>>> __cythonbufferalways__ = True >>>> >>>> [1] >>>> >>>> I think this makes a lot of sense, as the author of the Image class >>>> might export exactly this kind of buffer and nothing else. With the >>>> above settings, one would automatically get efficient indexing on >>>> all >>>> "cdef Image" instances. >>>> >>>> NumPy is kind of special in the flexibility it provides, but even >>>> there, >>>> setting indirect=False can provide a speedup transparently. Once the >>>> indirect option is implemented at all, that is :-) but that *must* >>>> happen to make NumPy as efficient as possible. >>>> >>>> As for priority, I guess this is below most of what I've talked >>>> about so >>>> far. Though it might suddenly look like a low hanging fruit that >>>> I go >>>> for when I need a break from other stuff. >>>> >>>> [1] (Well, I consider custom parsing for that last line better than >>>> inventing a wholly new syntax...and we've been talking about new >>>> Python-style type references as well, which fits in here) >>> This almost seems to magical for me, if one wants to use a buffer >>> perhaps it is better to be explicit. But I'm curious to hear what >>> other people think. >> Let me just give one more concrete example for NumPy then. If you do >> >> cdef ndarray[int, 2] buf >> >> currently, then it is going to create code for checking whether it >> should >> do indirect access, which is one if-test per dimension per lookup >> -- and >> you *know* that for ndarrays, you can always get around with strided >> access, i.e. something like >> >> cdef ndarray[int, 2, 'strided'] buf >> >> Now, I would kind of like to be able to tell people to use "object >> [int, >> 2]" to write a generic buffer algorithm, but "ndarray[int, 2]" to >> write >> something that only works with NumPy in an optimized fashion, and >> that is >> it -- and the proposal kind of grew out of that, it is a way of >> letting >> the users not have to type mode="strided" all the time. >> >> (BTW, do you like mode=different strings for this, or should I go with >> "strided=True", "c=True", "fortran=True", etc? > > I like just providing strings, which I am assuming map to access flags. More or less, but there may not always be a direct mapping, even though at first there will be. I am mostly considering usability and efficiency and making use of the PEP for that end, not cloning the semantics of the PEP directly. There is at least one case (the most common form of indirect indexing, where the last dimension is strided and the rest indirect... a "lines" mode) where I could introduce new assumptions that you cannot express in the buffer flags in order to get more optimal code in that case. (I'm wondering if this might be wanted for PIL, but I'll think of that at some later point -- if PIL is always 2D, introducing a suboffsets variable and setting suboffsets=(0,-1) by default will suffice). -- Dag Sverre From dalcinl at gmail.com Wed Jul 30 17:38:17 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 30 Jul 2008 12:38:17 -0300 Subject: [Cython] vtables: bad code generation, incorrect casts. Message-ID: Brian Granger posted some days ago about problems with multiple inheritance: http://codespeak.net/pipermail/cython-dev/2008-July/001814.html It seems that Cython is generating bad code. A C compiler (gcc) just warn about it, but a C++ compiler (g++) generates an error. A tentative patch for this issue is posted here: http://codespeak.net/pipermail/cython-dev/2008-July/001835.html If you have a bit of time, please look at this issue. Regards, -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Jul 30 18:01:13 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 30 Jul 2008 18:01:13 +0200 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: References: Message-ID: <48909049.9050506@student.matnat.uio.no> Lisandro Dalcin wrote: > Brian Granger posted some days ago about problems with multiple inheritance: > > http://codespeak.net/pipermail/cython-dev/2008-July/001814.html > > It seems that Cython is generating bad code. A C compiler (gcc) just > warn about it, but a C++ compiler (g++) > generates an error. > > A tentative patch for this issue is posted here: > > http://codespeak.net/pipermail/cython-dev/2008-July/001835.html > > If you have a bit of time, please look at this issue. Even with the patch, the end-result seems a bit strange: static PyObject *__pyx_tp_new_4vtab_Bam(PyTypeObject *t, PyObject *a, PyObject *k) { struct __pyx_obj_4vtab_Bam *p; PyObject *o = __pyx_tp_new_4vtab_Foo(t, a, k); if (!o) return 0; p = ((struct __pyx_obj_4vtab_Bam *)o); p->__pyx_base.__pyx_base.__pyx_vtab = (struct __pyx_vtabstruct_4vtab_Foo*)__pyx_vtabptr_4vtab_Bam; return o; } I.e. on vtable generation it seems to skip its immediate ancestor. Something about symmetry just doesn't seem right about this code with or without your patch... I can't figure out more right now, gotta go. Perhaps tomorrow, if nobody beats me to it. Just letting you know that this might need more care than that patch alone. -- Dag Sverre From stefan_ml at behnel.de Wed Jul 30 19:32:50 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jul 2008 19:32:50 +0200 Subject: [Cython] Buffer default options In-Reply-To: <488F76F9.4090308@student.matnat.uio.no> References: <488F76F9.4090308@student.matnat.uio.no> Message-ID: <4890A5C2.7060301@behnel.de> Hi, Dag Sverre Seljebotn wrote: > I'm thinking about letting the class author specify default buffer > options. Something like this: > > cdef extern class Image ...: > __cythonbufferdefaults__ = {'ndim' : 2} > __cythonbuffermandatory__ = {'dtype': unsigned char, indirect=True} > __cythonbufferalways__ = True I think default buffer options (mainly a default buffer layout) makes sense in general. It's basically saying "when I return a buffer, it will look like this". I wonder about the syntax, though. Wouldn't one special name + a couple of keyword arguments be enough? What about specifying it like this: cdef extern class Image ...: buffer[...buffer args...] __getbuffer__(...) ? Stefan From stefan_ml at behnel.de Wed Jul 30 22:39:25 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jul 2008 22:39:25 +0200 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: <48909049.9050506@student.matnat.uio.no> References: <48909049.9050506@student.matnat.uio.no> Message-ID: <4890D17D.6020200@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Brian Granger posted some days ago about problems with multiple inheritance: >> >> http://codespeak.net/pipermail/cython-dev/2008-July/001814.html >> >> It seems that Cython is generating bad code. A C compiler (gcc) just >> warn about it, but a C++ compiler (g++) >> generates an error. >> >> A tentative patch for this issue is posted here: >> >> http://codespeak.net/pipermail/cython-dev/2008-July/001835.html The original patch made a test case fail, the attached patch works better (not sure why this is required, though). > Even with the patch, the end-result seems a bit strange: > > static PyObject *__pyx_tp_new_4vtab_Bam(PyTypeObject *t, PyObject *a, > PyObject *k) { > struct __pyx_obj_4vtab_Bam *p; > PyObject *o = __pyx_tp_new_4vtab_Foo(t, a, k); > if (!o) return 0; > p = ((struct __pyx_obj_4vtab_Bam *)o); > p->__pyx_base.__pyx_base.__pyx_vtab = (struct > __pyx_vtabstruct_4vtab_Foo*)__pyx_vtabptr_4vtab_Bam; > return o; > } > > I.e. on vtable generation it seems to skip its immediate ancestor. It still seems to be the right thing to do. Maybe Greg can enlighten us here. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: baseclass-vtab.patch Type: text/x-patch Size: 880 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080730/1b00450a/attachment.bin From dalcinl at gmail.com Thu Jul 31 00:14:21 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 30 Jul 2008 19:14:21 -0300 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: <4890D17D.6020200@behnel.de> References: <48909049.9050506@student.matnat.uio.no> <4890D17D.6020200@behnel.de> Message-ID: On Wed, Jul 30, 2008 at 5:39 PM, Stefan Behnel wrote: > > The original patch made a test case fail, the attached patch works better (not > sure why this is required, though). Of Course! Your version then works even in this case below, where the class first introducing a vtable is in turn inheriting from a cdef class without cdef methods (that the reason why your modification is required!): cdef class Base0: pass cdef class Base(Base0): pass cdef class Foo(Base): cdef fooit(self): return 0 cdef class Bar(Foo): pass cdef class Bam(Bar): pass cdef class Zoo(Bam): pass Your patch then generate the right code for the case above. > >> Even with the patch, the end-result seems a bit strange: >> >> I.e. on vtable generation it seems to skip its immediate ancestor. > > It still seems to be the right thing to do. Maybe Greg can enlighten us here. > I agree with you, Stefan. Let's wait for Greeg to comment on this... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From nicole at cats-muvva.net Wed Jul 30 20:17:09 2008 From: nicole at cats-muvva.net (Nicole King) Date: Wed, 30 Jul 2008 19:17:09 +0100 Subject: [Cython] Opaque Pointers in Wrapping a C API Message-ID: <200807301917.10429.nicole@cats-muvva.net> Dear Stefan, Thank you for your reply. Your suggestion seems quite reasonable. I had, indeed, tried it before. It causes a different message that should, to someone with more wits than I, suggest the origin of my problem. I'm still baffled. nicole at hosts-2:4:~/firewall $cython libiptc.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... void finish(opaque_pointer** h) cdef class a_class: cdef opaque_pointer* the_handle def __init__(table_name): self.the_handle = init(table_name) ^ ------------------------------------------------------------ /home/nicole/firewall/libiptc.pyx:10:8: undeclared name not builtin: self nicole at hosts-2:4:~/firewall $ I'm wondering if I should abandon the idea of using Cython/Pyrex. Many thanks Nicole From robertwb at math.washington.edu Thu Jul 31 04:17:37 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 30 Jul 2008 19:17:37 -0700 Subject: [Cython] Opaque Pointers in Wrapping a C API In-Reply-To: <200807301917.10429.nicole@cats-muvva.net> References: <200807301917.10429.nicole@cats-muvva.net> Message-ID: On Jul 30, 2008, at 11:17 AM, Nicole King wrote: > Dear Stefan, > > Thank you for your reply. > > Your suggestion seems quite reasonable. I had, indeed, tried it > before. It causes a different message > that should, to someone with more wits than I, suggest the origin > of my problem. I'm still baffled. > > nicole at hosts-2:4:~/firewall $cython libiptc.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > void finish(opaque_pointer** h) > > cdef class a_class: > cdef opaque_pointer* the_handle > def __init__(table_name): > self.the_handle = init(table_name) > ^ > ------------------------------------------------------------ > > /home/nicole/firewall/libiptc.pyx:10:8: undeclared name not > builtin: self > nicole at hosts-2:4:~/firewall $ Unlike some languages, Python requires that the "self" parameter be explicitly named everywhere, including in the method arguments. Thus one would have to write cdef class a_class: cdef opaque_pointer* the_handle def __init__(self, table_name): self.the_handle = init(table_name) > I'm wondering if I should abandon the idea of using Cython/Pyrex. I hope Cython works for you. If you don't know Python yet it will probably be a bit of a steeper learning curve than otherwise, but sure beats writing everything by hand. - Robert From ellisonbg.net at gmail.com Thu Jul 31 06:16:35 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 30 Jul 2008 21:16:35 -0700 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: References: Message-ID: <6ce0ac130807302116h47feda89h4d5d4b875f20eead@mail.gmail.com> Thanks for looking into this. The really confusing thing is that I swear this code used for work. I haven't had the time to chase down what version of Cython this began with, but it seems like something that should work ideally. I am in the middle of moving across the county right now though, so I won't have time to look at this for a few weeks though. Cheers, Brian On Wed, Jul 30, 2008 at 8:38 AM, Lisandro Dalcin wrote: > Brian Granger posted some days ago about problems with multiple inheritance: > > http://codespeak.net/pipermail/cython-dev/2008-July/001814.html > > It seems that Cython is generating bad code. A C compiler (gcc) just > warn about it, but a C++ compiler (g++) > generates an error. > > A tentative patch for this issue is posted here: > > http://codespeak.net/pipermail/cython-dev/2008-July/001835.html > > If you have a bit of time, please look at this issue. > > Regards, > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Thu Jul 31 07:44:14 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 31 Jul 2008 07:44:14 +0200 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: <6ce0ac130807302116h47feda89h4d5d4b875f20eead@mail.gmail.com> References: <6ce0ac130807302116h47feda89h4d5d4b875f20eead@mail.gmail.com> Message-ID: <4891512E.2040106@behnel.de> Hi, Brian Granger wrote: > Thanks for looking into this. The really confusing thing is that I > swear this code used for work. I haven't had the time to chase down > what version of Cython this began with, but it seems like something > that should work ideally. Is it possible that you added a C method somewhere in the hierarchy since the last time you tried? That could trigger such a problem. Stefan From greg.ewing at canterbury.ac.nz Thu Jul 31 08:02:10 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 31 Jul 2008 18:02:10 +1200 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: <4890D17D.6020200@behnel.de> References: <48909049.9050506@student.matnat.uio.no> <4890D17D.6020200@behnel.de> Message-ID: <48915562.9010602@canterbury.ac.nz> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > > > I.e. on vtable generation it seems to skip its immediate ancestor. > > It still seems to be the right thing to do. Maybe Greg can enlighten us here. There are a couple of things going on: * When one type inherits from another, the struct for the ancestor type is embedded as a member at the beginning of the derived type, named __pyx_base. * There is at most one vtable pointer in each instance, and it lives in the struct for the most ancestral type that has any C methods. So the chain of __pyx_base references is drilling down through the layers of the onion to the struct where the vtable pointer is declared. An alternative would be just to cast the object pointer to the appropriate ancestor type, but at the time it seemed cleaner to generate code that avoided casts if possible. -- Greg From stefan_ml at behnel.de Thu Jul 31 08:14:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 31 Jul 2008 08:14:27 +0200 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: <48915562.9010602@canterbury.ac.nz> References: <48909049.9050506@student.matnat.uio.no> <4890D17D.6020200@behnel.de> <48915562.9010602@canterbury.ac.nz> Message-ID: <48915843.6030101@behnel.de> Hi, Greg Ewing wrote: > Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >> >>> I.e. on vtable generation it seems to skip its immediate ancestor. >> It still seems to be the right thing to do. Maybe Greg can enlighten us here. > > There are a couple of things going on: > > * When one type inherits from another, the struct for the > ancestor type is embedded as a member at the beginning of > the derived type, named __pyx_base. > > * There is at most one vtable pointer in each instance, > and it lives in the struct for the most ancestral type > that has any C methods. Sounds good to me. > So the chain of __pyx_base references is drilling down > through the layers of the onion to the struct where the > vtable pointer is declared. Ok, then the patch is actually correct in that it also drills down to the ancestor type that defines the vtable and casts the pointer to that type. Stefan From greg.ewing at canterbury.ac.nz Thu Jul 31 08:11:37 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 31 Jul 2008 18:11:37 +1200 Subject: [Cython] vtables: bad code generation, incorrect casts. In-Reply-To: <48915843.6030101@behnel.de> References: <48909049.9050506@student.matnat.uio.no> <4890D17D.6020200@behnel.de> <48915562.9010602@canterbury.ac.nz> <48915843.6030101@behnel.de> Message-ID: <48915799.10202@canterbury.ac.nz> Stefan Behnel wrote: > > Ok, then the patch is actually correct in that it also drills down to the > ancestor type that defines the vtable and casts the pointer to that type. Yes. The vtable pointer itself needs to be cast, because the vtable of the derived type may have extra C methods added to it, so it's not the same type as the vtable of the ancestor type. -- Greg From dagss at student.matnat.uio.no Thu Jul 31 10:07:20 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 31 Jul 2008 10:07:20 +0200 Subject: [Cython] Buffer default options In-Reply-To: <4890A5C2.7060301@behnel.de> References: <488F76F9.4090308@student.matnat.uio.no> <4890A5C2.7060301@behnel.de> Message-ID: <489172B8.1020502@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> I'm thinking about letting the class author specify default buffer >> options. Something like this: >> >> cdef extern class Image ...: >> __cythonbufferdefaults__ = {'ndim' : 2} >> __cythonbuffermandatory__ = {'dtype': unsigned char, indirect=True} >> __cythonbufferalways__ = True > > I think default buffer options (mainly a default buffer layout) makes sense in > general. It's basically saying "when I return a buffer, it will look like this". > > I wonder about the syntax, though. Wouldn't one special name + a couple of > keyword arguments be enough? In the discussion with Robert we kind of boiled it down to only defaults being needed; at least mandatory has problems and does not really gain anything. My preference now would be for __cythonbufferopts__ with a tuple or dict giving "defaults", "mandatory" etc. As for "being enough", in syntax discussions pretty much every syntax proposed is possible :-) My personal taste for new syntax is this: a) Extends the existing language grammar as little as possible b) Won't make it harder to get "Python-runnable-Cython-code" in the future, c) Is somewhat Pythonic (see below). For usability these may be overriden (as with the buffer access syntax), but this is going to be used so seldom that I don't see usability as a concern at all -- these are going to be written once or twice per library. Python has a long tradition of this (like __metaclass__; you and me would have thought of all kinds of funky syntax like class A(B, meta=C): ... or perhaps class A[C](B) but Guido went with a simple __metaclass__ attribute, meaning a quite drastical change in how the class works can be had without any new grammar. -- Dag Sverre From stefan_ml at behnel.de Thu Jul 31 11:08:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 31 Jul 2008 11:08:36 +0200 (CEST) Subject: [Cython] Buffer default options In-Reply-To: <489172B8.1020502@student.matnat.uio.no> References: <488F76F9.4090308@student.matnat.uio.no> <4890A5C2.7060301@behnel.de> <489172B8.1020502@student.matnat.uio.no> Message-ID: <49859.213.61.181.86.1217495316.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> I'm thinking about letting the class author specify default buffer >>> options. Something like this: >>> >>> cdef extern class Image ...: >>> __cythonbufferdefaults__ = {'ndim' : 2} >>> __cythonbuffermandatory__ = {'dtype': unsigned char, indirect=True} >>> __cythonbufferalways__ = True >> >> I think default buffer options (mainly a default buffer layout) makes >> sense in >> general. It's basically saying "when I return a buffer, it will look >> like this". >> >> I wonder about the syntax, though. Wouldn't one special name + a couple >> of >> keyword arguments be enough? > > [...] My preference now would be for __cythonbufferopts__ with a > tuple or dict giving "defaults", "mandatory" etc. Something like that, yes. It could reuse the existing compile-time "DEF" infrastructure, thus resulting in a real tuple that can be evaluated at parse time and disappear later on. Not sure if a dict works here, though. Stefan From robertwb at math.washington.edu Thu Jul 31 18:08:32 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 31 Jul 2008 09:08:32 -0700 Subject: [Cython] Buffer default options In-Reply-To: <489172B8.1020502@student.matnat.uio.no> References: <488F76F9.4090308@student.matnat.uio.no> <4890A5C2.7060301@behnel.de> <489172B8.1020502@student.matnat.uio.no> Message-ID: On Jul 31, 2008, at 1:07 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> Dag Sverre Seljebotn wrote: >>> I'm thinking about letting the class author specify default buffer >>> options. Something like this: >>> >>> cdef extern class Image ...: >>> __cythonbufferdefaults__ = {'ndim' : 2} >>> __cythonbuffermandatory__ = {'dtype': unsigned char, >>> indirect=True} >>> __cythonbufferalways__ = True >> >> I think default buffer options (mainly a default buffer layout) >> makes sense in >> general. It's basically saying "when I return a buffer, it will >> look like this". >> >> I wonder about the syntax, though. Wouldn't one special name + a >> couple of >> keyword arguments be enough? > > In the discussion with Robert we kind of boiled it down to only > defaults > being needed; at least mandatory has problems and does not really gain > anything. My preference now would be for __cythonbufferopts__ with a > tuple or dict giving "defaults", "mandatory" etc. I am for keeping it very simple, so I would propose a single dictionary __cythonbufferdefaults__. > As for "being enough", in syntax discussions pretty much every syntax > proposed is possible :-) My personal taste for new syntax is this: > > a) Extends the existing language grammar as little as possible > b) Won't make it harder to get "Python-runnable-Cython-code" in the > future, > c) Is somewhat Pythonic (see below). A counter-argument is that if you misspell it, it will silently fail to work. (You still have me convinced though.) > For usability these may be overriden (as with the buffer access > syntax), > but this is going to be used so seldom that I don't see usability as a > concern at all -- these are going to be written once or twice per > library. > > Python has a long tradition of this (like __metaclass__; you and me > would have thought of all kinds of funky syntax like > > class A(B, meta=C): ... > > or perhaps > > class A[C](B) > > but Guido went with a simple __metaclass__ attribute, meaning a quite > drastical change in how the class works can be had without any new > grammar. > > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Thu Jul 31 18:35:28 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 31 Jul 2008 18:35:28 +0200 Subject: [Cython] Buffer default options In-Reply-To: References: <488F76F9.4090308@student.matnat.uio.no> <4890A5C2.7060301@behnel.de> <489172B8.1020502@student.matnat.uio.no> Message-ID: <4891E9D0.5070203@student.matnat.uio.no> Robert Bradshaw wrote: > On Jul 31, 2008, at 1:07 AM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Hi, >>> >>> Dag Sverre Seljebotn wrote: >>>> I'm thinking about letting the class author specify default buffer >>>> options. Something like this: >>>> >>>> cdef extern class Image ...: >>>> __cythonbufferdefaults__ = {'ndim' : 2} >>>> __cythonbuffermandatory__ = {'dtype': unsigned char, >>>> indirect=True} >>>> __cythonbufferalways__ = True >>> I think default buffer options (mainly a default buffer layout) >>> makes sense in >>> general. It's basically saying "when I return a buffer, it will >>> look like this". >>> >>> I wonder about the syntax, though. Wouldn't one special name + a >>> couple of >>> keyword arguments be enough? >> In the discussion with Robert we kind of boiled it down to only >> defaults >> being needed; at least mandatory has problems and does not really gain >> anything. My preference now would be for __cythonbufferopts__ with a >> tuple or dict giving "defaults", "mandatory" etc. > > I am for keeping it very simple, so I would propose a single > dictionary __cythonbufferdefaults__. > >> As for "being enough", in syntax discussions pretty much every syntax >> proposed is possible :-) My personal taste for new syntax is this: >> >> a) Extends the existing language grammar as little as possible >> b) Won't make it harder to get "Python-runnable-Cython-code" in the >> future, >> c) Is somewhat Pythonic (see below). > > A counter-argument is that if you misspell it, it will silently fail > to work. (You still have me convinced though.) Well, this is Python after all :-) self.finshied = True -- Dag Sverre From dagss at student.matnat.uio.no Thu Jul 31 21:04:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 31 Jul 2008 21:04:44 +0200 Subject: [Cython] Refactor or surgery on global constants? Message-ID: <48920CCC.4070301@student.matnat.uio.no> In making it possible to have runnable code in pxds, I need to make a choice. The problem is that interned strings, objects etc. live in seperate scopes and you get double copies, basically now I get the following code twice in my c-file: static char __pyx_k___getbuffer__[] = "__getbuffer__"; static PyObject *__pyx_kp___getbuffer__; which gcc isn't too thrilled about. 1) Do drastic surgery on the scopes, so that interned strings, cached constants etc. from the pxd "module" scope is merged over to the pyx module scope, then relink the pxd function scopes to the pyx scopes instead. 2) Move these things from the scopes to code.global (a global context of the CCodeWriter). This fits nicely with utility_code going over as well. This means that for now each scope simply pipe their things into code where they are merged, while in time then e.g. StringNode could intern a string during code generation rather than bothering with it during analysis. (I think this is a viable way forward, but may take an hour or three longer for me to do.) 3) I suppose, linking the pxd to the pyx during analysis. But I see this as a step backward, we want pxds to be reusable between pyx-es (and they basically are today), so I'm very against this. 4) ? Any preferences welcome. If I don't hear anything, odds are I'll start on 2), but that would be a waste if people felt strongly about interned strings etc. belonging at analysis. -- Dag Sverre