From stefan_ml at behnel.de Sun Mar 2 12:24:30 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Mar 2008 12:24:30 +0100 Subject: [Cython] Cython test coverage Message-ID: <47CA8E6E.4040107@behnel.de> Hi, I just ran coverage.py over the test suite. http://nedbatchelder.com/code/modules/coverage.html $ python -m coverage -x runtests.py [...] $ python -m coverage -r Cython/Compiler/*.py Name Stmts Exec Cover ------------------------------------------------ Cython/Compiler/Annotate 107 19 17% Cython/Compiler/Builtin 17 17 100% Cython/Compiler/CmdLine 68 0 0% Cython/Compiler/Code 246 226 91% Cython/Compiler/DebugFlags 3 3 100% Cython/Compiler/Errors 68 41 60% Cython/Compiler/ExprNodes 1969 1245 63% Cython/Compiler/Lexicon 39 2 5% Cython/Compiler/Main 212 137 64% Cython/Compiler/ModuleNode 1048 914 87% Cython/Compiler/Naming 67 66 98% Cython/Compiler/Nodes 2038 1412 69% Cython/Compiler/Options 11 11 100% Cython/Compiler/Parsing 1556 1078 69% Cython/Compiler/PyrexTypes 589 484 82% Cython/Compiler/Scanning 260 186 71% Cython/Compiler/Symtab 803 666 82% Cython/Compiler/TypeSlots 264 259 98% Cython/Compiler/Version 1 1 100% Cython/Compiler/__init__ 0 0 100% ------------------------------------------------ TOTAL 9366 6767 72% Quite ok, but also quite a bit missing. Especially Parsing.py, Nodes.py and ExprNodes.py do not make me feel very comfortable. I would assume error testing to be a major factor here. We test a lot of working code, but little broken code. Stefan From stefan_ml at behnel.de Sun Mar 2 12:25:34 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Mar 2008 12:25:34 +0100 Subject: [Cython] Cython and Python 3k Message-ID: <47CA8EAE.4030904@behnel.de> Hi, just starting a little thread here to collect things to remember when we migrate to Python 3k one day. Robert Bradshaw wrote: > Stefan Behnel wrote: >>>> Also, there's Python 3, where especially the C-API is still in flux. >>>> Not sure >>>> if it's worth to start thinking about that yet, but it should at least >>>> stay on the list. >>> >>> Yep. >> >> I asked on cython-dev to see how far the C-API is. Answer I got is >> that it may >> take a bit to stabilize, including renaming types (str etc.) and >> fixing the >> outdated docs. So we'll have to see what the status will be in June. >> It might really be worth a sprint by then. >> >> http://comments.gmane.org/gmane.comp.python.devel/92255?set_lines=100000 >> >> Two additional things I see here: 1) porting Cython itself, and 2) >> working on >> better optimisation support in general, to keep users from doing >> direct C-API >> calls in Cython code, as this might lead to reduced portability. > > I think Python 2.x will be around for quite a while, even after P3k > comes out, but if the API is starting to stabilize (and even if not) I > think it is good to start looking at it. Porting Cython itself should be > pretty easy, 3.0 is mostly backwards compatible, and I don't think we > use much that isn't. It will almost certainly require a branch, though. We might be lucky as most of Cython's code is still pretty old style, without using unicode, generators and the like. I added bundles that fix at least the obvious "print()" and "<>" stuff. Things like "print >> sys.stderr" will need fixing, no idea what else is waiting... > Ideally, I would like to be able to run (the same) Cython with Python > 2.x or 3.x, and have it produce code that compiles (using macros) for > inclusion in either 2.x or 3.x runtimes. We can do this for 2.3-2.5. > Whether or not this is feasible across such a major revision remains to > be seen, but I'm optimistic. So am I (still). We will have to figure out some transition semantics for str/unicode/bytes, though, also at the Cython syntax level. But I guess that has time till the first Py3k beta release. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: package-upstream.bundle Type: application/octet-stream Size: 2904 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080302/f5d0a648/attachment-0002.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: upstream.bundle Type: application/octet-stream Size: 3532 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080302/f5d0a648/attachment-0003.obj From stefan_ml at behnel.de Sun Mar 2 12:30:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Mar 2008 12:30:36 +0100 Subject: [Cython] Cython and Python 3k In-Reply-To: <47CA8EAE.4030904@behnel.de> References: <47CA8EAE.4030904@behnel.de> Message-ID: <47CA8FDC.5040208@behnel.de> Stefan Behnel wrote: > I added bundles that fix at least the obvious "print()" and "<>" stuff. Sorry, incomplete bundle. Here we go. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: upstream.bundle Type: application/octet-stream Size: 8202 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080302/9c9823f7/attachment.obj From ndbecker2 at gmail.com Mon Mar 3 13:54:50 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 03 Mar 2008 07:54:50 -0500 Subject: [Cython] make extension type accesible from python? Message-ID: I'm trying to learn cython. As a simple first try, I have some generic c++ code 'my_sum'. I want to pass it an intvec (std::vector). It seems to work OK when accessed from c-code: cdef extern from "vector": pass cdef extern from "test1.hpp": ctypedef struct intvec "std::vector": void (* push_back)(int elem) intvec intvec_factory "std::vector"(int len) int my_sum "my_sum >"(intvec vec) cdef intvec v = intvec_factory(2) v.push_back(2) print my_sum (v) I have no idea how to expose intvec to python, and then be able to call my_sum from python. I assume there is some way to make extension types that can be used from python? Of course, I'd want to be able to add other methods, so that, for example, intvec can be constructed and manipulated from python. Any simple examples I could look at? From stefan_ml at behnel.de Mon Mar 3 14:14:09 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 3 Mar 2008 14:14:09 +0100 (CET) Subject: [Cython] make extension type accesible from python? In-Reply-To: References: Message-ID: <11469.194.114.62.68.1204550049.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: [real problem stripped] > I assume there is some way to make extension types that can be used from > python? Of course, I'd want to be able to add other methods, so that, for > example, intvec can be constructed and manipulated from python. Yes, wrapping it in a Python extension type is the right approach here. > Any simple examples I could look at? The Pyrex docs, for example: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/ http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/extension_types.html Stefan From ndbecker2 at gmail.com Mon Mar 3 19:01:25 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 03 Mar 2008 13:01:25 -0500 Subject: [Cython] make extension type accesible from python? References: <11469.194.114.62.68.1204550049.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > Neal Becker wrote: > [real problem stripped] >> I assume there is some way to make extension types that can be used from >> python? Of course, I'd want to be able to add other methods, so that, >> for example, intvec can be constructed and manipulated from python. > > Yes, wrapping it in a Python extension type is the right approach here. > > >> Any simple examples I could look at? > > The Pyrex docs, for example: > > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/ > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/extension_types.html > > Stefan I've been studying these, but I haven't seen my question addressed. Are there any examples I could look at? From ndbecker2 at gmail.com Tue Mar 4 13:39:02 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 04 Mar 2008 07:39:02 -0500 Subject: [Cython] wrapping c++ Message-ID: I've made some progress on learning to wrap std::vector. I've got a __getitem__, but I don't know what to do about __setitem__. cdef extern from "vector": pass cdef extern from "test1.hpp": ctypedef struct intvec "std::vector": void (* push_back)(int elem) inline int get "operator[]" (int i) intvec intvec_factory "std::vector"(int len) int my_sum "my_sum >"(intvec vec) cdef intvec v = intvec_factory(2) v.push_back(2) #print my_sum (v) cdef class intvec_wrap: cdef intvec vec def __init__(self): self.vec = intvec_factory (2) def __getitem__(self, int i): return self.vec.get (i) Problem is, operator[] (or at()) returns an int&. I don't have a clue what to do with this. From wstein at gmail.com Tue Mar 4 15:54:11 2008 From: wstein at gmail.com (William Stein) Date: Tue, 4 Mar 2008 08:54:11 -0600 Subject: [Cython] wrapping c++ In-Reply-To: References: Message-ID: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> On Tue, Mar 4, 2008 at 6:39 AM, Neal Becker wrote: > I've made some progress on learning to wrap std::vector. I've got a > __getitem__, but I don't know what to do about __setitem__. > cdef extern from "vector": > pass > > cdef extern from "test1.hpp": > ctypedef struct intvec "std::vector": > void (* push_back)(int elem) > inline int get "operator[]" (int i) > > intvec intvec_factory "std::vector"(int len) > > int my_sum "my_sum >"(intvec vec) > > > > cdef intvec v = intvec_factory(2) > v.push_back(2) > > #print my_sum (v) > > cdef class intvec_wrap: > cdef intvec vec > def __init__(self): > self.vec = intvec_factory (2) > def __getitem__(self, int i): > return self.vec.get (i) > > Problem is, operator[] (or at()) returns an int&. I don't have a clue what > to do with this. There is an example of doing setitem directly in Cython with std::vector using C name specifiers in my talk on Cython at Enthought from two days ago. See the talk listed at the bottom of this page: http://wiki.sagemath.org/days8/schedule Here's the example: cdef extern from "vector.h": ctypedef unsigned long size_type ctypedef struct intvec "std::vector": int get_entry "operator[]"(int n) size_type size() intvec intvec_factory "std::vector"(int len) cdef extern from "": void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() void SET(intvec, int, int) DEFINE_SET() cdef intvec v = intvec_factory(10) SET(v,2,5) print v.get_entry(2) print v.size() ---------------- The key thing is the evil dirty trick to define a SET macro directly in Cython. -- William From stefan_ml at behnel.de Tue Mar 4 17:09:39 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 4 Mar 2008 17:09:39 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> Message-ID: <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> William Stein wrote: > cdef extern from "": > void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() > void SET(intvec, int, int) > DEFINE_SET() > > The key thing is the evil dirty trick to define a SET macro directly in > Cython. "Evil" and "dirty" were exactly the two words that came to my mind before I even got to reading your own comment on this. :) I would really put something like this into an external header file, and just declare "SET()" as being external... Stefan From ndbecker2 at gmail.com Tue Mar 4 17:21:06 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 04 Mar 2008 11:21:06 -0500 Subject: [Cython] wrapping c++ References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > William Stein wrote: >> cdef extern from "": >> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >> void SET(intvec, int, int) >> DEFINE_SET() >> >> The key thing is the evil dirty trick to define a SET macro directly in >> Cython. > > "Evil" and "dirty" were exactly the two words that came to my mind before > I even got to reading your own comment on this. :) > > I would really put something like this into an external header file, and > just declare "SET()" as being external... > > Stefan Any thoughts on what it would take to support a non-evil implementation? From stefan_ml at behnel.de Tue Mar 4 18:07:55 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 4 Mar 2008 18:07:55 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Stefan Behnel wrote: > >> William Stein wrote: >>> cdef extern from "": >>> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >>> void SET(intvec, int, int) >>> DEFINE_SET() >>> >>> The key thing is the evil dirty trick to define a SET macro directly in >>> Cython. >> >> "Evil" and "dirty" were exactly the two words that came to my mind >> before I even got to reading your own comment on this. :) >> >> I would really put something like this into an external header file, and >> just declare "SET()" as being external... >> > Any thoughts on what it would take to support a non-evil implementation? The above is ok, except that I wouldn't write something like this into a Cython file. You will likely need a header file for a couple of low-level things anyway, so I would just create one with the "#define" above and keep the 'cdef extern from "theheader.h"' for the "SET()" function (read: macro). Stefan From robertwb at math.washington.edu Tue Mar 4 20:30:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 11:30:33 -0800 Subject: [Cython] wrapping c++ In-Reply-To: <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <122F497C-84FA-4BA2-A8E3-F1AEFF9A6391@math.washington.edu> On Mar 4, 2008, at 9:07 AM, Stefan Behnel wrote: > Neal Becker wrote: >> Stefan Behnel wrote: >> >>> William Stein wrote: >>>> cdef extern from "": >>>> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >>>> void SET(intvec, int, int) >>>> DEFINE_SET() >>>> >>>> The key thing is the evil dirty trick to define a SET macro >>>> directly in >>>> Cython. >>> >>> "Evil" and "dirty" were exactly the two words that came to my mind >>> before I even got to reading your own comment on this. :) >>> >>> I would really put something like this into an external header >>> file, and >>> just declare "SET()" as being external... >>> >> Any thoughts on what it would take to support a non-evil >> implementation? > > The above is ok, except that I wouldn't write something like this > into a > Cython file. You will likely need a header file for a couple of low- > level > things anyway, so I would just create one with the "#define" above and > keep the 'cdef extern from "theheader.h"' for the "SET()" function > (read: > macro). I've been thinking about adding C++ operator support to Cython. I'm not sure of the best syntax to use for this, but it should be possible to tell Cython that a certain type has overloaded operators, and also to be able to specify a specific function to use for such operators (for example, this would be useful in numpy for fast indexing of arrays--one would define a macro that does the dirty work and it would be like the PEX a{i,j}. - Robert From wstein at gmail.com Tue Mar 4 20:32:14 2008 From: wstein at gmail.com (William Stein) Date: Tue, 4 Mar 2008 13:32:14 -0600 Subject: [Cython] Fwd: [sage-support] cython and filenames In-Reply-To: <0311e1c9-04dd-49d7-89fd-d8311010682e@d4g2000prg.googlegroups.com> References: <0311e1c9-04dd-49d7-89fd-d8311010682e@d4g2000prg.googlegroups.com> Message-ID: <85e81ba30803041132o7d31ef5cn149840a65b7b7efe@mail.gmail.com> Hi, I'm forwarding this message from sage-support to Cython-devel, where it points out a probable bug in the Cython compiler (with fix). -- william (from Sage Days 8) ---------- Forwarded message ---------- From: Kate Date: Tue, Mar 4, 2008 at 10:44 AM Subject: [sage-support] cython and filenames To: sage-support Here is a snippet of code from cython.py in the main tree (SAGE_ROOT/devel/sage/sage/misc/cython.py) def cython(filename, verbose=False, compile_message=False, use_cache=False, create_local_c_file=False, annotate=True, sage_namespace=True): if not filename.endswith('pyx'): print "File (=%s) should have extension .pyx"%filename clean_filename = sanitize(filename) ====> base = os.path.split(os.path.splitext(clean_filename)[0])[1] ====> abs_base = os.path.abspath(os.path.split(os.path.splitext(clean_filename)[0])[0]) It seems to me that the marked lines, in addition to being hard to parse, do not have the intended effect. This is because in the creation of clean_filename, the call to the sanitize function converts all slashes and dots into underscores, which are not 'picked up' by the calls to the os.path.* functions. Is this really what is intended? Kate --~--~---------~--~----~------------~-------~--~----~ To post to this group, send email to sage-support at googlegroups.com To unsubscribe from this group, send email to sage-support-unsubscribe at googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URLs: http://www.sagemath.org -~----------~----~----~----~------~----~------~--~--- -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From robertwb at math.washington.edu Tue Mar 4 20:39:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 11:39:14 -0800 Subject: [Cython] Fwd: [sage-support] cython and filenames In-Reply-To: <85e81ba30803041132o7d31ef5cn149840a65b7b7efe@mail.gmail.com> References: <0311e1c9-04dd-49d7-89fd-d8311010682e@d4g2000prg.googlegroups.com> <85e81ba30803041132o7d31ef5cn149840a65b7b7efe@mail.gmail.com> Message-ID: <2F102241-C47B-4E84-A0CC-15C371A90BB9@math.washington.edu> This actually isn't the cython compiler code, it's in the Sage library. I'm actually not sure what the goal was here with the clean_filename stuff (didn't write it myself, and I'm wary of changing it because I'm guessing there was some reason to do it this way)... - Robert On Mar 4, 2008, at 11:32 AM, William Stein wrote: > Hi, > > I'm forwarding this message from sage-support to Cython-devel, where > it points out a probable bug in the Cython compiler (with fix). > > -- william (from Sage Days 8) > > ---------- Forwarded message ---------- > From: Kate > Date: Tue, Mar 4, 2008 at 10:44 AM > Subject: [sage-support] cython and filenames > To: sage-support > > > > Here is a snippet of code from cython.py in the main tree > (SAGE_ROOT/devel/sage/sage/misc/cython.py) > > def cython(filename, verbose=False, compile_message=False, > use_cache=False, create_local_c_file=False, annotate=True, > sage_namespace=True): > if not filename.endswith('pyx'): > print "File (=%s) should have extension .pyx"%filename > > clean_filename = sanitize(filename) > ====> base = os.path.split(os.path.splitext(clean_filename)[0])[1] > ====> abs_base = > os.path.abspath(os.path.split(os.path.splitext(clean_filename)[0]) > [0]) > > It seems to me that the marked lines, in addition to being > hard to parse, do not have the intended effect. This is > because in the creation of clean_filename, the call to > the sanitize function converts all slashes and dots into > underscores, which are not 'picked up' by the calls to > the os.path.* functions. > > Is this really what is intended? > > Kate > > --~--~---------~--~----~------------~-------~--~----~ > To post to this group, send email to sage-support at googlegroups.com > To unsubscribe from this group, send email to > sage-support-unsubscribe at googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-support > URLs: http://www.sagemath.org > -~----------~----~----~----~------~----~------~--~--- > > > > > -- > William Stein > Associate Professor of Mathematics > University of Washington > http://wstein.org > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Tue Mar 4 21:54:34 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 04 Mar 2008 21:54:34 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support Message-ID: <47CDB70A.7080903@student.matnat.uio.no> Since Robert mentioned NumPy in relation with adding operator support I thought about sharing my more thoughts about NumPy - I'm very new to Cython so I guess take it for what it is worth - however what I've seen so far looks so promising for me that I might want to spend some time in a few months working on implementing some of this, which perhaps may make my thoughts more intereseting :-) Currently, Cython is mostly geared towards wrapping C code, but it is also an excellent foundation for being a numerical tool - but the rough edges are still prohibitive. A few relatively small steps (in terms of man-hours needed) would improve the situation a lot I think - not perfect, but perhaps in a few years we can have something that will finally kill FORTRAN :-) Three suggestions comes briefly here, if anyone's interested and it is not already discussed and decided I might flesh them out in "PEP-style" in the coming month? Note that a) is what is important for me, b) and c) is just something I throw along... a) numpy.ndarray syntax candy. Really, what one should implement is syntax support for PEP-3118: http://www.python.org/dev/peps/pep-3118/ Because this protocol will be shared between NumPy, PIL etc. in Python 3 it could make sense to simply have "native"/hard-coded support for this aspect without necesarrily making it a generic operator feature, and one can then use the same approach as will be needed for buffers in Python 3 for NumPy in Python 2? Example (where "array" is considered a new, Cython-native type that will have automatic conversion from any NumPy arrays and Python 3 buffers): def myfunc(array<2, unsigned char> arr): arr[4, 5] = 10 might be translated to the equivalent of the currently legal: def myfunc(numpy.ndarray arr): if arr.nd != 2 or arr.dtype != numpy.dtype(numpy.uint8): raise ValueError("Must pass 2-dimensional uint8 array.") cdef unsigned char* arr_buf = arr.data arr.data[4 * arr.strides[0] + 5 * arr.strides[1]] = 10 (Probably caching the strides in local variables etc.). That should do as a first implementation -- it is always possible to be more sophisticated, but this little will allow NumPyers to simply dive in. Specifically, the number of dimensions must be declared first and only direct access in that many dimensions are allowed. Slices etc. should be less important (they can be done on the Python object instead). Moving on from here, one should probably instead define bufferinfo from PEP-3118 and make it say def myfunc(bufferinfo arr): if arr.ndim != 2 or arr.format != "B") or arr.readonly: raise ValueError("Must pass writeable 2-dimensional buffer with format 'B'.") ... with automatic conversion from NumPy arrays to bufferinfo. b) Allow numpy types? Basically, make it possible to say "cdef uint8 myvar", at least for in-function-variables that is not interfacing with C code, so that for numerical use one doesn't need to learn C. This can be in addition, so it should not break existing code, though I can understand resentment against the idea as well. c) Probably controversial: More Pythonic syntax. A syntax for decoration of function arguments is decided upon (at least in Python 3), so to align with that one could allow for stuff like @Compile def myfunc(a: uint8, b: array(2, uint8), c: int = 10): d: ptr(int) = &a print a, b, c, d Which is "almost" Python - only the definition of d is different, but consistency talks for change there as well. This can also be in addition to the existing syntax so it should not break anything (allowing, say, only one type of syntax per function). But a) is what is interesting here... -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 4 22:02:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 04 Mar 2008 22:02:30 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <47CDB70A.7080903@student.matnat.uio.no> References: <47CDB70A.7080903@student.matnat.uio.no> Message-ID: <47CDB8E6.2010200@student.matnat.uio.no> Correction: > def myfunc(numpy.ndarray arr): > if arr.nd != 2 or arr.dtype != numpy.dtype(numpy.uint8): > raise ValueError("Must pass 2-dimensional uint8 array.") > cdef unsigned char* arr_buf = arr.data > arr.data[4 * arr.strides[0] + 5 * arr.strides[1]] = 10 > Last line should index arr_buf, not arr.data. BTW, > perfect, but perhaps in a few years we can have something that will > finally kill FORTRAN :-) > In case you were wondering, this is definitely a joke. One can hope though. -- Dag Sverre From ndbecker2 at gmail.com Tue Mar 4 22:28:15 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 04 Mar 2008 16:28:15 -0500 Subject: [Cython] wrapping c++ References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <122F497C-84FA-4BA2-A8E3-F1AEFF9A6391@math.washington.edu> Message-ID: Robert Bradshaw wrote: > On Mar 4, 2008, at 9:07 AM, Stefan Behnel wrote: > >> Neal Becker wrote: >>> Stefan Behnel wrote: >>> >>>> William Stein wrote: >>>>> cdef extern from "": >>>>> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >>>>> void SET(intvec, int, int) >>>>> DEFINE_SET() >>>>> >>>>> The key thing is the evil dirty trick to define a SET macro >>>>> directly in >>>>> Cython. >>>> >>>> "Evil" and "dirty" were exactly the two words that came to my mind >>>> before I even got to reading your own comment on this. :) >>>> >>>> I would really put something like this into an external header >>>> file, and >>>> just declare "SET()" as being external... >>>> >>> Any thoughts on what it would take to support a non-evil >>> implementation? >> >> The above is ok, except that I wouldn't write something like this >> into a >> Cython file. You will likely need a header file for a couple of low- >> level >> things anyway, so I would just create one with the "#define" above and >> keep the 'cdef extern from "theheader.h"' for the "SET()" function >> (read: >> macro). > > > I've been thinking about adding C++ operator support to Cython. I'm > not sure of the best syntax to use for this, but it should be > possible to tell Cython that a certain type has overloaded operators, > and also to be able to specify a specific function to use for such > operators (for example, this would be useful in numpy for fast > indexing of arrays--one would define a macro that does the dirty work > and it would be like the PEX a{i,j}. > > - Robert Is operator the real issue here? In the std::vector example, there is 'at', which does (almost) the same thing as operator[]. My problem is that both 'at' and 'operator[]' return int&, and to use them you need to say 'at[i]=j', which I don't know how to do in cython (except for the macro version above, or to write additional wrapper code in more c++ headers). From dagss at student.matnat.uio.no Tue Mar 4 22:53:02 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 04 Mar 2008 22:53:02 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> Message-ID: <47CDC4BE.7050608@student.matnat.uio.no> > buffers, then this could be provided automatically. Overloading the = > operator would work for the one direction, and perhaps something > similar would be provided to do an "convert self to other type" > declaration. Right. This will keep me thinking :-) Has there been any thoughts on doing something as brute as allowing Python code in pxd files that is simply called compile-time and returns strings of C code? Or if not there, as a kind of plugin-architecture for the Cython compiler engine? Like this (apologies for bad knowledge of this aspect of Cython): (Lot's of stuff that doesn't work, just scetching wildly...) cdef class std_vector_int: def __compile_index_assign__(lefthand, indices, righthand): return "%s[%s] = %s;" % (as_c(lefthand), as_c(indices), as_c(righthand)) def __compile_index_lookup__(lefthand, indices): return "%s[%s];" % (as_c(lefthand), as_c(indices)) def __compile_assign__(lefthand, rightand):... cdef class numpy.numarray ...: def __compile_index_assign__(lefthand, indices, righthand): l = indices.split(":") if len(l) != int(lefthand.type_params[0]): raise ValueError(...) idx = " + ".join([as_c(x) + " * stride[%d]" % d for x in zip(l, range(len(l))]) return "%s[%s] = %s;" % (as_c(lefthand), idx, as_c(righthand)) With the "macro as cdef"-trick earlier in the mailing list this certainly doesn't seem *too* far-fetched for me... > > These data types can be declared, right now, by doing. > > cdef extern from "numpy.h": > cdeftype int uint8 Ah, right :-) >> >> @Compile >> def myfunc(a: uint8, b: array(2, uint8), c: int = 10): >> d: ptr(int) = &a >> print a, b, c, d >> > > One thing I really like about the cdef keyword is that it makes it > very clear what is special Cython commands, and what is not. I also > find this harder to read (and parse). (BTW, what's the @Compile > supposed to mean?) However, if people want to move in this direction I > wouldn't protest against it as an alternative. The idea was to focus on making Cython usable for numerical computation -- and for that use (as opposed to making wrappers for C code) you do NOT want people to feel that it is not part of Python, but rather treat it as a "typed Python that will run fast". The use case is simply somebody that uses NumPy for everything, but comes over this one operation that a) isn't in the libraries, b) can't be vectorized but must be implemented as for-loops. If they can simply drop in a @Compile decorator and add some types and get on with their work, the pure convenience of it might make Cython a regular tool for these kind of things... @Compile simply uses the standard Python function decorator syntax. The @Compile might be @Cython instead actually. The idea is that while Python syntax allows for decorating function arguments, it doesn't say anything about what they're for (documentation? something else?), so you would often have a function decorator that acts on them, and this serves to document that the function argument decorators are Cython types. Long term, @Compile might be used to replace the function with calls to compiled code in Python-land, and to extract functions to compile from a py-file in Cython-land, so that one can mix Cython and Python in one source file. Also, it can take the options that Cython needs (ie @Compile(native=True, except=1) instead of cdef ... except ). Dag Sverre From robertwb at math.washington.edu Tue Mar 4 23:15:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 14:15:20 -0800 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <47CDC4BE.7050608@student.matnat.uio.no> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> <47CDC4BE.7050608@student.matnat.uio.no> Message-ID: <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> On Mar 4, 2008, at 1:53 PM, Dag Sverre Seljebotn wrote: > >> buffers, then this could be provided automatically. Overloading the = >> operator would work for the one direction, and perhaps something >> similar would be provided to do an "convert self to other type" >> declaration. > Right. This will keep me thinking :-) > > Has there been any thoughts on doing something as brute as allowing > Python code in pxd files that is simply called compile-time and > returns > strings of C code? Or if not there, as a kind of plugin- > architecture for > the Cython compiler engine? Like this (apologies for bad knowledge of > this aspect of Cython): > > (Lot's of stuff that doesn't work, just scetching wildly...) > > cdef class std_vector_int: > def __compile_index_assign__(lefthand, indices, righthand): > return "%s[%s] = %s;" % (as_c(lefthand), as_c(indices), as_c > (righthand)) > def __compile_index_lookup__(lefthand, indices): > return "%s[%s];" % (as_c(lefthand), as_c(indices)) > def __compile_assign__(lefthand, rightand):... > > cdef class numpy.numarray ...: > def __compile_index_assign__(lefthand, indices, righthand): > l = indices.split(":") > if len(l) != int(lefthand.type_params[0]): raise ValueError(...) > idx = " + ".join([as_c(x) + " * stride[%d]" % d for x in zip(l, > range(len(l))]) > return "%s[%s] = %s;" % (as_c(lefthand), idx, as_c(righthand)) No, nothing like that has been attempted (or even discussed before). It is an interesting idea. I see potential for lots of subtle bugs (e.g. refcounting errors). I think if we can reduce operators (and conversions) to a specified function call (pass the operator to C++) there will be little, if any, need for something like this. I may be overly optimistic for things like making sure dimensions match however. > With the "macro as cdef"-trick earlier in the mailing list this > certainly doesn't seem *too* far-fetched for me... That code is simply evil. > > >> >> These data types can be declared, right now, by doing. >> >> cdef extern from "numpy.h": >> cdeftype int uint8 > Ah, right :-) > > >>> >>> @Compile >>> def myfunc(a: uint8, b: array(2, uint8), c: int = 10): >>> d: ptr(int) = &a >>> print a, b, c, d >>> >> >> One thing I really like about the cdef keyword is that it makes it >> very clear what is special Cython commands, and what is not. I also >> find this harder to read (and parse). (BTW, what's the @Compile >> supposed to mean?) However, if people want to move in this >> direction I >> wouldn't protest against it as an alternative. > The idea was to focus on making Cython usable for numerical > computation > -- and for that use (as opposed to making wrappers for C code) you do > NOT want people to feel that it is not part of Python, but rather > treat > it as a "typed Python that will run fast". The use case is simply > somebody that uses NumPy for everything, but comes over this one > operation that a) isn't in the libraries, b) can't be vectorized but > must be implemented as for-loops. If they can simply drop in a > @Compile > decorator and add some types and get on with their work, the pure > convenience of it might make Cython a regular tool for these kind of > things... This is a good point. > @Compile simply uses the standard Python function decorator syntax. > The > @Compile might be @Cython instead actually. The idea is that while > Python syntax allows for decorating function arguments, it doesn't say > anything about what they're for (documentation? something else?), > so you > would often have a function decorator that acts on them, and this > serves > to document that the function argument decorators are Cython types. > > Long term, @Compile might be used to replace the function with > calls to > compiled code in Python-land, and to extract functions to compile > from a > py-file in Cython-land, so that one can mix Cython and Python in one > source file. Also, it can take the options that Cython needs (ie > @Compile(native=True, except=1) instead of cdef ... except ). If we can come up with a good syntax for using decorators to express all Cython options, that could have some advantages (though I would still keep the old way as well). For example, one could debug the algorithm as a pure Python program which has a faster turnaround time and more tools (pylint, prun, etc) and then compile it when one is fairly confident that all is working well. Perhaps something could even be done with ctypes to actually simulate library wrapping and everything in python mode. Mixing Python and Cython code in the same file is far in the future (as I see it) and seems to have little advantage over just having the whole file be compiled. - Robert From ondrej at certik.cz Wed Mar 5 00:04:45 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 5 Mar 2008 00:04:45 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> <47CDC4BE.7050608@student.matnat.uio.no> <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> Message-ID: <85b5c3130803041504h3124e5dcmbf93c48cf7f5d398@mail.gmail.com> On Tue, Mar 4, 2008 at 11:15 PM, Robert Bradshaw wrote: > > On Mar 4, 2008, at 1:53 PM, Dag Sverre Seljebotn wrote: > > > > >> buffers, then this could be provided automatically. Overloading the = > >> operator would work for the one direction, and perhaps something > >> similar would be provided to do an "convert self to other type" > >> declaration. > > Right. This will keep me thinking :-) > > > > Has there been any thoughts on doing something as brute as allowing > > Python code in pxd files that is simply called compile-time and > > returns > > strings of C code? Or if not there, as a kind of plugin- > > architecture for > > the Cython compiler engine? Like this (apologies for bad knowledge of > > this aspect of Cython): > > > > (Lot's of stuff that doesn't work, just scetching wildly...) > > > > cdef class std_vector_int: > > def __compile_index_assign__(lefthand, indices, righthand): > > return "%s[%s] = %s;" % (as_c(lefthand), as_c(indices), as_c > > (righthand)) > > def __compile_index_lookup__(lefthand, indices): > > return "%s[%s];" % (as_c(lefthand), as_c(indices)) > > def __compile_assign__(lefthand, rightand):... > > > > cdef class numpy.numarray ...: > > def __compile_index_assign__(lefthand, indices, righthand): > > l = indices.split(":") > > if len(l) != int(lefthand.type_params[0]): raise ValueError(...) > > idx = " + ".join([as_c(x) + " * stride[%d]" % d for x in zip(l, > > range(len(l))]) > > return "%s[%s] = %s;" % (as_c(lefthand), idx, as_c(righthand)) > > No, nothing like that has been attempted (or even discussed before). > It is an interesting idea. I see potential for lots of subtle bugs > (e.g. refcounting errors). I think if we can reduce operators (and > conversions) to a specified function call (pass the operator to C++) > there will be little, if any, need for something like this. I may be > overly optimistic for things like making sure dimensions match however. > > > > With the "macro as cdef"-trick earlier in the mailing list this > > certainly doesn't seem *too* far-fetched for me... > > That code is simply evil. > > > > > > > > >> > >> These data types can be declared, right now, by doing. > >> > >> cdef extern from "numpy.h": > >> cdeftype int uint8 > > Ah, right :-) > > > > > >>> > >>> @Compile > >>> def myfunc(a: uint8, b: array(2, uint8), c: int = 10): > >>> d: ptr(int) = &a > >>> print a, b, c, d > >>> > >> > >> One thing I really like about the cdef keyword is that it makes it > >> very clear what is special Cython commands, and what is not. I also > >> find this harder to read (and parse). (BTW, what's the @Compile > >> supposed to mean?) However, if people want to move in this > >> direction I > >> wouldn't protest against it as an alternative. > > The idea was to focus on making Cython usable for numerical > > computation > > -- and for that use (as opposed to making wrappers for C code) you do > > NOT want people to feel that it is not part of Python, but rather > > treat > > it as a "typed Python that will run fast". The use case is simply > > somebody that uses NumPy for everything, but comes over this one > > operation that a) isn't in the libraries, b) can't be vectorized but > > must be implemented as for-loops. If they can simply drop in a > > @Compile > > decorator and add some types and get on with their work, the pure > > convenience of it might make Cython a regular tool for these kind of > > things... > > This is a good point. > > > > @Compile simply uses the standard Python function decorator syntax. > > The > > @Compile might be @Cython instead actually. The idea is that while > > Python syntax allows for decorating function arguments, it doesn't say > > anything about what they're for (documentation? something else?), > > so you > > would often have a function decorator that acts on them, and this > > serves > > to document that the function argument decorators are Cython types. > > > > Long term, @Compile might be used to replace the function with > > calls to > > compiled code in Python-land, and to extract functions to compile > > from a > > py-file in Cython-land, so that one can mix Cython and Python in one > > source file. Also, it can take the options that Cython needs (ie > > @Compile(native=True, except=1) instead of cdef ... except ). > > If we can come up with a good syntax for using decorators to express > all Cython options, that could have some advantages (though I would > still keep the old way as well). For example, one could debug the > algorithm as a pure Python program which has a faster turnaround time > and more tools (pylint, prun, etc) and then compile it when one is > fairly confident that all is working well. Perhaps something could > even be done with ctypes to actually simulate library wrapping and > everything in python mode. Yes, we were discussing exactly this with Travis Oliphant at Sage Days 8, it would be very cool, if a "pure python" code could be compiled. You can use decorators, or even python comments for specifying the types. Definitely it would be very useful, but it's quite a lot of work to get everything right. Ondrej From jek-gmane at kleckner.net Wed Mar 5 00:47:59 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Tue, 04 Mar 2008 15:47:59 -0800 Subject: [Cython] Embedding Pyrex/Cython tutorial Message-ID: I just ran across David McNab's tutorial about embedding Pyrex here: http://www.freenet.org.nz/python/embeddingpyrex/ Very interesting, thanks. David, would it make sense to put that into the Cython wiki? It would be front and center with more visibility. Cheers - Jim From robertwb at math.washington.edu Wed Mar 5 00:53:58 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 15:53:58 -0800 Subject: [Cython] Embedding Pyrex/Cython tutorial In-Reply-To: References: Message-ID: Yes, please put it on the wiki. - Robert On Mar 4, 2008, at 3:47 PM, Jim Kleckner wrote: > I just ran across David McNab's tutorial about embedding Pyrex here: > http://www.freenet.org.nz/python/embeddingpyrex/ > > Very interesting, thanks. > > David, would it make sense to put that into the Cython wiki? > > It would be front and center with more visibility. > > Cheers - Jim > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From jek-gmane at kleckner.net Wed Mar 5 01:29:37 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Tue, 04 Mar 2008 16:29:37 -0800 Subject: [Cython] Documentation link broken In-Reply-To: <9FC12853-9BC7-4E38-82EA-8002FBE9B195@math.washington.edu> References: <479D3921.9030006@martincmartin.com> <9FC12853-9BC7-4E38-82EA-8002FBE9B195@math.washington.edu> Message-ID: Robert Bradshaw wrote: > Thanks for the note--looks like he renamed the file. I just fixed it. This front page duplicates the wiki front page and is subject to divergence: ( compare http://www.cython.org/ with http://wiki.cython.org/Cython ) It would probably be better to concentrate on one or the other rather than keeping both consistent and choosing the wiki page means that others can fix it when they see it as well. From dagss at student.matnat.uio.no Wed Mar 5 01:31:36 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 5 Mar 2008 01:31:36 +0100 (CET) Subject: [Cython] Ideas developed: Code generator plugin architecture In-Reply-To: <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> <47CDC4BE.7050608@student.matnat.uio.no> <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> Message-ID: <54349.193.157.237.39.1204677096.squirrel@webmail.uio.no> The ideas have now evolved so much that a new subject was necesarry. I've written up this (but not linked to it anywhere, perhaps a "Far-out ideas" section could be added or something)... http://wiki.cython.org/CodeGeneratorPluginArchitecture It is simply an expansion of the last idea of indexing operators. Now, I haven't even looked at Cython source so I have no idea if this is at all feasible, if it requires a full redesign or reimplementation or whatever. However, what I do see is that it would solve any problems with wrapping any C++, STL, NumPy, ... code for ever, and it doesn't look like an awfully far fetch from what Cython is already doing. (As I say in the document, it is a very imperative rather than declarative approach, but if it can solve the problems quickly then it should be forward-compatible with more powerful declarative approaches anyway, so whatever gives the best results...) I'll calm down now. :-) Dag Sverre From robertwb at math.washington.edu Wed Mar 5 01:36:08 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 16:36:08 -0800 Subject: [Cython] Documentation link broken In-Reply-To: References: <479D3921.9030006@martincmartin.com> <9FC12853-9BC7-4E38-82EA-8002FBE9B195@math.washington.edu> Message-ID: Yes. I would like to greatly simplify the front page, and have it be mostly static stuff (and a "safe" link to download the source). It's just a matter of finding time. Timothy Clemens came up with a proposed design. http:// cython.timothyclemans.com/ I'd think there's little enough stuff that we should make it all available without any clicking though. - Robert On Mar 4, 2008, at 4:29 PM, Jim Kleckner wrote: > Robert Bradshaw wrote: >> Thanks for the note--looks like he renamed the file. I just fixed it. > > This front page duplicates the wiki front page and is subject to > divergence: > ( compare http://www.cython.org/ with http://wiki.cython.org/ > Cython ) > > It would probably be better to concentrate on one or the other rather > than keeping both consistent and choosing the wiki page means that > others can fix it when they see it as well. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Wed Mar 5 07:34:00 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 22:34:00 -0800 Subject: [Cython] [sage-devel] Cython benchmarks? In-Reply-To: <46ba3d380803042221j103b73bn22239d3fc37036d3@mail.gmail.com> References: <46ba3d380803042221j103b73bn22239d3fc37036d3@mail.gmail.com> Message-ID: That all depends on how it is written. If you are using a lot of python data types, it will be slower. If not, you can write Cython that translates to nearly pure C (and runs at that speed). - Robert On Mar 4, 2008, at 10:21 PM, Carlo Hamalainen wrote: > > Hi, > > I have this request for Cython benchmarks on a blog post of mine - is > there any data out there, specifically comparing Cython to C/C++? > > -- > Carlo Hamalainen > http://carlo-hamalainen.net > > > > ---------- Forwarded message ---------- > http://carlo-hamalainen.net/blog/?p=19 > Author : Robert Samal > E-mail : rsamal at sfu.ca > > Does somebody else have some experience in how cython compares > with C/C++? Every once in a while I need to do some computation > (something NP-complete or worse in general, so it > usually ends up as an ugly backtracking). I'd be happy to do > everything from within Sage (and using python/cython), but I'm not > sure, if it is fast enough (or if it getting fast enough, I suppose > that cython is improving gradually). > > --~--~---------~--~----~------------~-------~--~----~ > To post to this group, send email to sage-devel at googlegroups.com > To unsubscribe from this group, send email to sage-devel- > unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/ > group/sage-devel > URLs: http://www.sagemath.org > -~----------~----~----~----~------~----~------~--~--- From stefan_ml at behnel.de Wed Mar 5 08:40:05 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 5 Mar 2008 08:40:05 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <122F497C-84FA-4BA2-A8E3-F1AEFF9A6391@math.washington.edu> Message-ID: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Is operator the real issue here? In the std::vector example, there is > 'at', > which does (almost) the same thing as operator[]. My problem is that > both 'at' and 'operator[]' return int&, and to use them you need to > say 'at[i]=j', which I don't know how to do in cython (except for the > macro > version above, or to write additional wrapper code in more c++ headers). If you declare a variable cdef int* myvar Cython allows you to say myvar[5] = 1 Is that what you want? Stefan From stefan_ml at behnel.de Wed Mar 5 09:17:19 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 5 Mar 2008 09:17:19 +0100 (CET) Subject: [Cython] Cython benchmarks? In-Reply-To: References: <46ba3d380803042221j103b73bn22239d3fc37036d3@mail.gmail.com> Message-ID: <30539.194.114.62.67.1204705039.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Mar 4, 2008, at 10:21 PM, Carlo Hamalainen wrote: >> I have this request for Cython benchmarks on a blog post of mine - is >> there any data out there, specifically comparing Cython to C/C++? >> >> ---------- Forwarded message ---------- >> Does somebody else have some experience in how cython compares >> with C/C++? Every once in a while I need to do some computation >> (something NP-complete or worse in general, so it >> usually ends up as an ugly backtracking). I'd be happy to do >> everything from within Sage (and using python/cython), but I'm not >> sure, if it is fast enough (or if it getting fast enough, I suppose >> that cython is improving gradually). > > That all depends on how it is written. If you are using a lot of > python data types, it will be slower. If not, you can write Cython > that translates to nearly pure C (and runs at that speed). I think the best way to do this is to let Cython generate the C code and then look through the time critical parts to check for obvious problems, such as multiple back and forth conversion between C types and Python types or integer loops that run with Python run variables. Usually, this can be fixed by a couple of "cdef" declarations in the Cython code, which then translates to more optimal C code. I don't think a general benchmark is of much value here, as it would not catch the most common problems like the ones above. And if you want to optimise, there is no better way than timing your own code, which the timeit module in Python's stdlib makes easy enough to do. Just write a Cython module with a Python ("def") function that does exactly what you want to benchmark, and then call it through timeit. That's a one-liner. I added a quick blog entry on this (currently untested, but I'll fix that as soon as I get to it). You should get the idea anyway. http://behnel.de/cgi-bin/weblog_basic/index.php?p=17 Stefan From stefan_ml at behnel.de Wed Mar 5 13:36:28 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 5 Mar 2008 13:36:28 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: <200803050609.34160.ndbecker2@gmail.com> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <200803050609.34160.ndbecker2@gmail.com> Message-ID: <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: >> Neal Becker wrote: >> > Is operator the real issue here? In the std::vector example, there is >> > 'at', >> > which does (almost) the same thing as operator[]. My problem is that >> > both 'at' and 'operator[]' return int&, and to use them you need to >> > say 'at[i]=j', which I don't know how to do in cython (except for the >> > macro >> > version above, or to write additional wrapper code in more c++ >> headers). > > Let's say we have a c++ class intvec, it has > int& at(int index); I just looked that up, int& is a C++ reference here (didn't know that notation). What I would do is: create a header file and use it to define macros that do C++ operations on references in a function-like style. Things like the SET() 'function' that William proposed. Then, you define these macros as being external functions in a .pxd file, and cimport them into your .pyx file. Then you can use them in your Cython code as normal functions, and Cython will generate the macro calls for you, that the C preprocessor will translate into the correct operations. As Robert suggested, enhancing Cython's syntax to understand references would be helpful in the long term. As for the return type of your at() function, have you tried declaring it as returning "int*" instead? What does Cython do in this case? Stefan From dagss at student.matnat.uio.no Wed Mar 5 13:53:35 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 Mar 2008 13:53:35 +0100 Subject: [Cython] wrapping c++ In-Reply-To: <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <200803050609.34160.ndbecker2@gmail.com> <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <47CE97CF.7050004@student.matnat.uio.no> > > As for the return type of your at() function, have you tried declaring it > as returning "int*" instead? What does Cython do in this case? > C++ references doesn't work that way, I don't think that would work. C++ code would look like this: vector v(10); v.at(3) = 4; If you declare the return type to be int* you simply have a problem... Dag Sverre From dagss at student.matnat.uio.no Wed Mar 5 13:56:01 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 Mar 2008 13:56:01 +0100 Subject: [Cython] wrapping c++ In-Reply-To: <47CE97CF.7050004@student.matnat.uio.no> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <200803050609.34160.ndbecker2@gmail.com> <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <47CE97CF.7050004@student.matnat.uio.no> Message-ID: <47CE9861.8060301@student.matnat.uio.no> > > If you declare the return type to be int* you simply have a problem... Aw...sorry, hit the send combination while in the middle of typing! What I meant to say is that it won't solve the problem - Cython will treat it as a pointer type. AFAIK the only way to assign to an lvalue in C is if you dereference it, ie *lvalueReturningFunc() = rvalue; But references have automatic dereferencing, so the syntax above can never work, and the needed syntax: lvalueReturningFunc() = rvalue is not legal C IIRC. So macros are the way to go. -- Dag Sverre From stefan_ml at behnel.de Wed Mar 5 20:18:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Mar 2008 20:18:07 +0100 Subject: [Cython] pybench shootout between Cython 0.9.6.12+ and Python 2.5.1 Message-ID: <47CEF1EF.1080701@behnel.de> Hi, I got most of pybench running (as shipped with Python 2.5.1), just had to rename the .py files to .pyx (except Setup.py) and fix some "*" imports in the configuration, Cython does not allow that. Benchmark suites that run: - Arithmetic - Constructs - Lists - Tuples - Dict - Exceptions - Imports - Strings - Numbers - Unicode Benchmark suites that do not run: - Calls - Lookups - Instances - NewInstances The main problem is that they define classes and functions dynamically, which is not currently supported by Cython. The attached text file compares the results of a stock Python 2.5.1 from Ubuntu Gutsy to a Cython compiled version of the benchmarks. Positive differences mean that Python is slower than Cython. (I couldn't manage to get the opposite comparison running due to pickle problems). The end result is that Cython is almost 1/3 faster than the Python interpreter for plain Python code. Especially control structures like if/else and loops gain a lot from compilation, even for plain Python loops. BIG FAT WARNING: Please do not overrate these numbers, this is an unfair comparison between an interpreter and a compiler that do not even understand the same language. Also, I didn't compile Python myself, so the compiler optimisations differ between the two. However, I did not alter the benchmarks themselves in any way, so all benchmark functions use exactly the same Python source code in both runs. The "0ms" that Cython gives you in a couple of tests are actually real. They mean that the timings were not measurable compared to the overhead of the benchmark call itself, which is credible for an empty try-except, for example, or the comparison of numbers. Have fun, Stefan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cybench-compare.txt Url: http://codespeak.net/pipermail/cython-dev/attachments/20080305/eec8b962/attachment-0001.txt From wstein at gmail.com Wed Mar 5 20:54:17 2008 From: wstein at gmail.com (William Stein) Date: Wed, 5 Mar 2008 11:54:17 -0800 Subject: [Cython] pybench shootout between Cython 0.9.6.12+ and Python 2.5.1 In-Reply-To: <47CEF1EF.1080701@behnel.de> References: <47CEF1EF.1080701@behnel.de> Message-ID: <85e81ba30803051154k75908489j6a1db8a2ec5a16c7@mail.gmail.com> On Wed, Mar 5, 2008 at 11:18 AM, Stefan Behnel wrote: > Hi, > > I got most of pybench running (as shipped with Python 2.5.1), just had to > rename the .py files to .pyx (except Setup.py) and fix some "*" imports in the > configuration, Cython does not allow that. > > Benchmark suites that run: > - Arithmetic > - Constructs > - Lists > - Tuples > - Dict > - Exceptions > - Imports > - Strings > - Numbers > - Unicode > > Benchmark suites that do not run: > - Calls > - Lookups > - Instances > - NewInstances > > The main problem is that they define classes and functions dynamically, which > is not currently supported by Cython. > > The attached text file compares the results of a stock Python 2.5.1 from > Ubuntu Gutsy to a Cython compiled version of the benchmarks. Positive > differences mean that Python is slower than Cython. (I couldn't manage to get > the opposite comparison running due to pickle problems). > > The end result is that Cython is almost 1/3 faster than the Python interpreter > for plain Python code. Especially control structures like if/else and loops > gain a lot from compilation, even for plain Python loops. Just out of curiosity could you use Pyrex (instead of Cython) to run all the same benchmarks? I'm very curious how much the optimizations that were added to Cython have impacted performance on these sorts of things. > > BIG FAT WARNING: Please do not overrate these numbers, this is an unfair > comparison between an interpreter and a compiler that do not even understand > the same language. Also, I didn't compile Python myself, so the compiler > optimisations differ between the two. However, I did not alter the benchmarks > themselves in any way, so all benchmark functions use exactly the same Python > source code in both runs. > > The "0ms" that Cython gives you in a couple of tests are actually real. They > mean that the timings were not measurable compared to the overhead of the > benchmark call itself, which is credible for an empty try-except, for example, > or the comparison of numbers. > > Have fun, > Stefan > > > ************************************************************************ > **This file compares Python 2.5.1 (this) with Cython 0.9.6.12 (other) ** > ************************************************************************ > > ------------------------------------------------------------------------------- > PYBENCH 2.0 > ------------------------------------------------------------------------------- > * using Python 2.5.1 > * disabled garbage collection > * system check interval set to maximum: 2147483647 > * using timer: time.time > > Calibrating tests. Please wait... > > Running 10 round(s) of the suite at warp factor 10: > > * Round 1 done in 6.541 seconds. > * Round 2 done in 6.499 seconds. > * Round 3 done in 6.634 seconds. > * Round 4 done in 6.501 seconds. > * Round 5 done in 6.558 seconds. > * Round 6 done in 6.581 seconds. > * Round 7 done in 6.521 seconds. > * Round 8 done in 6.567 seconds. > * Round 9 done in 6.503 seconds. > * Round 10 done in 6.502 seconds. > > ------------------------------------------------------------------------------- > Benchmark: 2008-03-05 17:01:02 > ------------------------------------------------------------------------------- > > Rounds: 10 > Warp: 10 > Timer: time.time > > Machine Details: > Platform ID: Linux-2.6.22-14-generic-i686-with-debian-lenny-sid > Processor: > > Python: > Executable: /usr/bin/python > Version: 2.5.1 > Compiler: GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2) > Bits: 32bit > Build: Oct 5 2007 13:36:32 (#r251:54863) > Unicode: UCS4 > > > ------------------------------------------------------------------------------- > Comparing with: ../cybench/cybench.log > ------------------------------------------------------------------------------- > > Rounds: 10 > Warp: 10 > Timer: time.time > > Machine Details: > Platform ID: Linux-2.6.22-14-generic-i686-with-debian-lenny-sid > Processor: > > Python: > Executable: /usr/bin/python > Version: 2.5.1 > Compiler: GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2) > Bits: 32bit > Build: Oct 5 2007 13:36:32 (#r251:54863) > Unicode: UCS4 > > > Test minimum run-time average run-time > this other diff this other diff > ------------------------------------------------------------------------------- > CompareFloats: 160ms 0ms +7486272.6% 161ms 0ms +3534624.7% > CompareFloatsIntegers: 155ms 0ms +37093705.7% 155ms 0ms +6358029.8% > CompareIntegers: 152ms 0ms +32709538.5% 152ms 0ms +3928831.4% > CompareInternedStrings: 162ms 50ms +223.3% 168ms 51ms +226.4% > CompareLongs: 125ms 601ms -79.1% 126ms 603ms -79.2% > CompareStrings: 164ms 87ms +89.3% 164ms 90ms +83.0% > CompareUnicode: 165ms 107ms +54.6% 166ms 108ms +54.3% > ConcatStrings: 257ms 224ms +14.9% 279ms 243ms +15.1% > ConcatUnicode: 244ms 223ms +9.4% 245ms 224ms +9.6% > CreateNewInstances: 141ms 0ms n/a 142ms 0ms n/a > CreateStringsWithConcat: 189ms 153ms +23.1% 189ms 154ms +22.9% > CreateUnicodeWithConcat: 169ms 142ms +19.0% 172ms 146ms +17.9% > DictCreation: 137ms 63ms +116.3% 138ms 64ms +116.9% > DictWithFloatKeys: 180ms 190ms -5.4% 180ms 193ms -6.4% > DictWithIntegerKeys: 144ms 65ms +120.1% 145ms 66ms +119.3% > DictWithStringKeys: 142ms 51ms +180.1% 142ms 51ms +180.4% > ForLoops: 131ms 22ms +501.1% 132ms 22ms +501.9% > IfThenElse: 139ms 47ms +198.0% 139ms 47ms +197.2% > ListSlicing: 123ms 121ms +1.7% 123ms 121ms +1.5% > NestedForLoops: 154ms 17ms +832.3% 156ms 17ms +843.6% > SecondImport: 136ms 159ms -14.7% 137ms 161ms -15.0% > SecondPackageImport: 141ms 166ms -15.6% 142ms 168ms -15.5% > SecondSubmoduleImport: 174ms 199ms -12.9% 175ms 202ms -13.5% > SimpleComplexArithmetic: 168ms 139ms +20.7% 168ms 139ms +20.8% > SimpleDictManipulation: 150ms 85ms +76.6% 150ms 85ms +75.7% > SimpleFloatArithmetic: 180ms 85ms +112.5% 192ms 86ms +123.4% > SimpleIntFloatArithmetic: 134ms 52ms +157.0% 134ms 52ms +156.7% > SimpleIntegerArithmetic: 134ms 51ms +162.7% 134ms 51ms +162.5% > SimpleListManipulation: 137ms 37ms +271.2% 137ms 37ms +268.5% > SimpleLongArithmetic: 142ms 107ms +32.3% 142ms 108ms +31.6% > SmallLists: 148ms 78ms +89.2% 149ms 79ms +89.1% > SmallTuples: 142ms 75ms +88.0% 143ms 76ms +87.7% > StringMappings: 143ms 124ms +15.1% 144ms 126ms +14.6% > StringPredicates: 169ms 116ms +45.8% 169ms 119ms +42.3% > StringSlicing: 153ms 94ms +62.8% 157ms 99ms +58.3% > TryExcept: 122ms 0ms +10754178.9% 122ms 0ms +2514522.4% > TryRaiseExcept: 125ms 81ms +53.6% 125ms 83ms +51.2% > TupleSlicing: 144ms 116ms +23.5% 144ms 117ms +23.6% > UnicodeMappings: 142ms 140ms +1.8% 142ms 141ms +1.1% > UnicodePredicates: 146ms 108ms +35.8% 146ms 109ms +33.8% > UnicodeProperties: 134ms 93ms +43.9% 134ms 93ms +43.9% > UnicodeSlicing: 176ms 131ms +34.4% 178ms 135ms +31.8% > ------------------------------------------------------------------------------- > Totals: 6468ms 4398ms n/a 6541ms 4464ms n/a > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From ndbecker2 at gmail.com Tue Mar 4 16:00:04 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 4 Mar 2008 10:00:04 -0500 Subject: [Cython] wrapping c++ In-Reply-To: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> Message-ID: <200803041000.04266.ndbecker2@gmail.com> On Tuesday 04 March 2008, William Stein wrote: > On Tue, Mar 4, 2008 at 6:39 AM, Neal Becker wrote: > > I've made some progress on learning to wrap std::vector. I've got a > > __getitem__, but I don't know what to do about __setitem__. > > cdef extern from "vector": > > pass > > > > cdef extern from "test1.hpp": > > ctypedef struct intvec "std::vector": > > void (* push_back)(int elem) > > inline int get "operator[]" (int i) > > > > intvec intvec_factory "std::vector"(int len) > > > > int my_sum "my_sum >"(intvec vec) > > > > > > > > cdef intvec v = intvec_factory(2) > > v.push_back(2) > > > > #print my_sum (v) > > > > cdef class intvec_wrap: > > cdef intvec vec > > def __init__(self): > > self.vec = intvec_factory (2) > > def __getitem__(self, int i): > > return self.vec.get (i) > > > > Problem is, operator[] (or at()) returns an int&. I don't have a clue > > what to do with this. > > There is an example of doing setitem directly in Cython with > std::vector > using C name specifiers in my talk on Cython at Enthought from two > days ago. See > the talk listed at the bottom of this page: > http://wiki.sagemath.org/days8/schedule > > Here's the example: > > cdef extern from "vector.h": > ctypedef unsigned long size_type > > ctypedef struct intvec "std::vector": > int get_entry "operator[]"(int n) > size_type size() > > intvec intvec_factory "std::vector"(int len) > > cdef extern from "": > void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() > void SET(intvec, int, int) > DEFINE_SET() > > > cdef intvec v = intvec_factory(10) > SET(v,2,5) > print v.get_entry(2) > print v.size() > > > ---------------- > > The key thing is the evil dirty trick to define a SET macro directly in > Cython. -- William Wow, that is ugly. From ndbecker2 at gmail.com Wed Mar 5 12:09:33 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 5 Mar 2008 06:09:33 -0500 Subject: [Cython] wrapping c++ In-Reply-To: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <200803050609.34160.ndbecker2@gmail.com> On Wednesday 05 March 2008, Stefan Behnel wrote: > Neal Becker wrote: > > Is operator the real issue here? In the std::vector example, there is > > 'at', > > which does (almost) the same thing as operator[]. My problem is that > > both 'at' and 'operator[]' return int&, and to use them you need to > > say 'at[i]=j', which I don't know how to do in cython (except for the > > macro > > version above, or to write additional wrapper code in more c++ headers). > > If you declare a variable > > cdef int* myvar > > Cython allows you to say > > myvar[5] = 1 > > Is that what you want? > > Stefan Does that help for a function that returns int&? Let's say we have a c++ class intvec, it has int& at(int index); From stefan_ml at behnel.de Wed Mar 5 22:13:44 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Mar 2008 22:13:44 +0100 Subject: [Cython] pybench shootout between Cython 0.9.6.12+ and Python 2.5.1 and Pyrex 0.9.6.4 In-Reply-To: <85e81ba30803051154k75908489j6a1db8a2ec5a16c7@mail.gmail.com> References: <47CEF1EF.1080701@behnel.de> <85e81ba30803051154k75908489j6a1db8a2ec5a16c7@mail.gmail.com> Message-ID: <47CF0D08.9040302@behnel.de> Hi, William Stein wrote: > On Wed, Mar 5, 2008 at 11:18 AM, Stefan Behnel wrote: >> I got most of pybench running (as shipped with Python 2.5.1), just had to >> rename the .py files to .pyx (except Setup.py) and fix some "*" imports in the >> configuration, Cython does not allow that. > > Just out of curiosity could you use Pyrex (instead of Cython) to run > all the same benchmarks? I'm very curious how much the optimizations > that were added to Cython have impacted performance on these > sorts of things. Want a number? 11% :) I also attached the sources I used. Stefan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pyrex-compare.txt Url: http://codespeak.net/pipermail/cython-dev/attachments/20080305/b7cf0764/attachment-0001.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: cybench.tgz Type: application/x-compressed-tar Size: 27088 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080305/b7cf0764/attachment-0001.bin From dg at pnylab.com Thu Mar 6 16:26:18 2008 From: dg at pnylab.com (Dan Gindikin) Date: Thu, 06 Mar 2008 10:26:18 -0500 Subject: [Cython] Thoughts on numerical computing/NumPy support Message-ID: <47D00D1A.8030603@pnylab.com> Hi Dag, I've just finished v1 of a preprocessor for pyrex/cython, called pex, that does (among other things) precisely what you describe for numpy ndarrays, you can say: cdef matmult(ndarray A, ndarray B): cdef int i,j,q cdef ndarray ans for i from 0<=i Nice!, very interesting. Myself I couldn't quite put my thoughts about this away and have spent too many hours the past days hacking on Cython source, and software architecture-wise it's been very stimulating. If my interest doesn't fade (which it better if I am to hand in my uni assignments in time...) I might have a prototype of my own ideas in a week, taking a few hours in-between now and then. The end-result should be the same thing, however my approach is very different. While pex is probably the way forward I want to pursue my own attempt a little further because I believe a preprocessor can only do so much before it ends up either lacking wanted capabilities because of missing type knowledge etc. (adding overloaded funtion support?), or duplicating much of what Cython already has. So my approach is instead adding another layer so that it is possible to program Cython at a "higher level" and create plugins for the kinds of thing you would want to preprocess for. (I didn't look up pex yet though....I don't have webaccess now...so I might have jumped to wrong conclusions.) My approach is of course very intrusive and the chances of a high enough success for inclusion in Cython is perhaps 10 percent or so (to make something up, and I'm not familiar with Cython processes...). But if you want to have a look it would still be very interesting to discuss with somebody who have taken a different approach...the notes are on the cython wiki (look on recentlychanged for my name...it is a lot of brain dump but should give an idea.) Dag Sverre Seljebotn -----Original Message----- From: Dan Gindikin Date: Thursday, Mar 6, 2008 4:26 pm Subject: Re: [Cython] Thoughts on numerical computing/NumPy support To: cython-dev at codespeak.net Hi Dag, > >I've just finished v1 of a preprocessor for pyrex/cython, called pex, that does >(among other things) precisely what you describe for numpy ndarrays, you can say: > >cdef matmult(ndarray A, ndarray B): > cdef int i,j,q > > cdef ndarray ans > > for i from 0<=i for j from 0<=j for q from 0<=q ans{i,j}=ans{i,j}+A{i,q}*B{q,j} > > return ans > >All of the {} accesses occur at C speed. We've been talking with Robert about moving this, and perhaps other features of pex into cython. > >-- >Dan >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From wstein at gmail.com Thu Mar 6 23:36:58 2008 From: wstein at gmail.com (William Stein) Date: Thu, 6 Mar 2008 14:36:58 -0800 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <3287687773.2188279@smtp.netcom.no> References: <3287687773.2188279@smtp.netcom.no> Message-ID: <85e81ba30803061436o5586264dwc560dd6aa17c1d4e@mail.gmail.com> On 06 Mar 2008 22:36:00 +0100, Dag Sverre Seljebotn wrote: > Nice!, very interesting. > > Myself I couldn't quite put my thoughts about this away and have spent too many hours the past days hacking on Cython source, and software architecture-wise it's been very stimulating. If my interest doesn't fade (which it better if I am to hand in my uni assignments in time...) I might have a prototype of my own ideas in a week, taking a few hours in-between now and then. The end-result should be the same thing, however my approach is very different. > > While pex is probably the way forward I want to pursue my own attempt a little further because I believe a preprocessor can only do so much before it ends up either lacking wanted capabilities because of missing type knowledge etc. My impression talking to everybody and especially the PEX people is that *your* approach (not a preprocessor) is probably the way forward. PEX (and the examples and tests it includes) is excellent because it lays down a nice implementation that is actually extremely useful for real work of what we want to be able to do. But the actual implementation was basically the easiest way for the PEX people to get something up and running quickly without having to change Cython itself (which was an initial design goal for them for their first version, I think). > (adding overloaded funtion support?), or duplicating much of what Cython already has. So my approach is instead adding another layer so that it is possible to program Cython at a "higher level" and create plugins for the kinds of thing you would want to preprocess for. (I didn't look up pex yet though....I don't have webaccess now...so I might have jumped to wrong conclusions.) > > My approach is of course very intrusive and the chances of a high enough success for inclusion in Cython is perhaps 10 percent or so (to make something up, and I'm not familiar with Cython processes...). But if you want to have a look it would still be very interesting to discuss with somebody who have taken a different approach...the notes are on the cython wiki (look on recentlychanged for my name...it is a lot of brain dump but should give an idea.) > Maybe the first thing you do won't get in as is, but whatever you do I think it will be very helpful somehow. In any case, there are currently not enough Cython developers, so I'm very happy that you are getting involved! -- William From dg at pnylab.com Fri Mar 7 15:16:28 2008 From: dg at pnylab.com (Dan Gindikin) Date: Fri, 07 Mar 2008 09:16:28 -0500 Subject: [Cython] Thoughts on numerical computing/NumPy support Message-ID: <47D14E3C.3070509@pnylab.com> Dag, you are absolutely right, since Pex only does shallow type analysis, and has no access to Cython's internal state during compilation, there is a rather hard stop on what it is able to do. As William said, we wanted to get something up quickly and did not want to muck around Cython internals. I too think this sort functionality clearly belongs in the language proper, as you are describing. -- Dan From dagss at student.matnat.uio.no Fri Mar 7 21:08:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 07 Mar 2008 21:08:00 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" Message-ID: <47D1A0A0.9000105@student.matnat.uio.no> During my work to investigate how NumPy, C++ template etc. support could best be added I discovered a way of integrating such code that has very low impact on changing existing code and flow, yet does (I believe) have the capacity to give quite powerful results. I've made a prototype: - A framework in a 300 line diff - An example using the framework: A reimplementation of the good old for i in range(x, y) => for i from i <= ... transform, in 40 lines The whole thing and a long description is here: http://wiki.cython.org/enhancements/parsetreetransforms -- Dag Sverre From robertwb at math.washington.edu Fri Mar 7 21:31:25 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 7 Mar 2008 12:31:25 -0800 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D1A0A0.9000105@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> Message-ID: <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> This looks very, very interesting and promising. I was thinking about code transformations like this quite a while ago for optimization reasons. I think this is the way to go. - Robert On Mar 7, 2008, at 12:08 PM, Dag Sverre Seljebotn wrote: > During my work to investigate how NumPy, C++ template etc. support > could > best be added I discovered a way of integrating such code that has > very > low impact on changing existing code and flow, yet does (I believe) > have > the capacity to give quite powerful results. > > I've made a prototype: > - A framework in a 300 line diff > - An example using the framework: A reimplementation of the good old > for i in range(x, y) => for i from i <= ... transform, in 40 lines > > The whole thing and a long description is here: > > http://wiki.cython.org/enhancements/parsetreetransforms > > > -- > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From gfurnish at indirectproof.net Sat Mar 8 00:02:37 2008 From: gfurnish at indirectproof.net (Gary Furnish) Date: Fri, 7 Mar 2008 16:02:37 -0700 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> Message-ID: <706850310803071502m24e38116t64323549546d273b@mail.gmail.com> I agree. This is very well designed and should hopefully be the framework for building many of the numpy things without having to alter the core of cython. Why did you end up using > for relation2 in your example though? On 3/7/08, Robert Bradshaw wrote: > This looks very, very interesting and promising. I was thinking about > code transformations like this quite a while ago for optimization > reasons. I think this is the way to go. > > > - Robert > > > On Mar 7, 2008, at 12:08 PM, Dag Sverre Seljebotn wrote: > > > During my work to investigate how NumPy, C++ template etc. support > > could > > best be added I discovered a way of integrating such code that has > > very > > low impact on changing existing code and flow, yet does (I believe) > > have > > the capacity to give quite powerful results. > > > > I've made a prototype: > > - A framework in a 300 line diff > > - An example using the framework: A reimplementation of the good old > > for i in range(x, y) => for i from i <= ... transform, in 40 lines > > > > The whole thing and a long description is here: > > > > http://wiki.cython.org/enhancements/parsetreetransforms > > > > > > -- > > Dag Sverre > > > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Sat Mar 8 11:26:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 11:26:04 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <706850310803071502m24e38116t64323549546d273b@mail.gmail.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <706850310803071502m24e38116t64323549546d273b@mail.gmail.com> Message-ID: <47D269BC.2060109@student.matnat.uio.no> Gary Furnish wrote: > I agree. This is very well designed and should hopefully be the > framework for building many of the numpy things without having to > alter the core of cython. Why did you end up using > for relation2 in > your example though? > Apparently you read my example more carefully than I did :-) It should be < of course. (I didn't actually run any code generated with it so I missed it). Regarding NumPy: - I think type parameters should also be introduced in addition to Cython core. It's an easy job, one avoids a lot of hacks, and they have many potential uses (like C++ templates). - Then, this framework can be used to implement NumPy support. And I think it should be, because it is necasarry to gain experience in this before doing step d), - However, this has a drawback: One still has to know a lot of Cython internals, and when Cython internals change then the transforms will sometimes have to change as well, since the Cython parse tree is exported in "raw" form. - Ideally, one would therefore have yet another layer (built, of course, using this transform layer) that's more geared to specifically doing the transforms necesarry for special support for types. My (now a bit outdated) thoughts on that are on: http://wiki.cython.org/CodeGeneratorPluginArchitecture. But as I said I don't think NumPy support should wait for this layer, rather building NumPy support will show us how such a layer should look like. - Finally, having such a "type specific behaviour" layer will allow NumPy code to be seperated out of Cython development and allow it to live in the NumPy codebase instead (if they want to). Dag Sverre From dagss at student.matnat.uio.no Sat Mar 8 13:43:42 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 13:43:42 +0100 Subject: [Cython] Patch ready (was: Re: Done with prototype for "parse tree transforms") In-Reply-To: <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> Message-ID: <47D289FE.10108@student.matnat.uio.no> Robert Bradshaw wrote: > This looks very, very interesting and promising. I was thinking about > code transformations like this quite a while ago for optimization > reasons. I think this is the way to go. In that case, I've cleaned it up a little bit and now submit it as a patch against the current cython-devel repo (see attachment); I believe it is ready for inclusion in the cython-devel branch if you approve. It doesn't *do* anything currently, and really shouldn't affect behaviour if one doesn't use the new -T switch (so regression testing for this patch can be done simply by checking that the same code is generated for a lot of examples -- I didn't take time for this though). However placing it in the main repository means that more people than myself can start playing with applying transforms that actually does something. Changes: A little code has been added all over the place, but most of it doesn't "interact" at all with existing code, unless you count declaring new symbols. The diff is small enough that you can simply read it .. the "worst" parts: - Node.generate_function_definitions got an extra argument everywhere (but I grepped for usages and am pretty sure I fixed all calls, at least within the Compiler module). - FuncDefNode.generate_function_definitions has calls to the transforms added: self.declare_arguments(lenv) + transforms.run('before_analyse_function', self, env=env, lenv=lenv, genv=genv) self.body.analyse_declarations(lenv) self.body.analyse_expressions(lenv) + transforms.run('after_analyse_function', self, env=env, lenv=lenv, genv=genv) (Without program arguments, these will iterate an empty list and return.) Known problems: - Only two cut-points/phases for transforms are provided, and more are neeeded, especially one in-between type deduction and type coercion, but this requires more intrusive refactoring that is better left for a seperate commit. This gives a start though. - The lists of which attributes for each node constitute "children" in the tree can probably be tuned a bit, however this is best left for when one actually have a use-case (transform) making use of the node type in question. Dag Sverre -------------- next part -------------- A non-text attachment was scrubbed... Name: treetransform.diff Type: text/x-patch Size: 30685 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080308/2b779953/attachment-0001.bin From dagss at student.matnat.uio.no Sat Mar 8 14:11:13 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 14:11:13 +0100 Subject: [Cython] Patch ready In-Reply-To: <47D289FE.10108@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> Message-ID: <47D29071.5080207@student.matnat.uio.no> A new patch (towards the original repo, so just forget about the original one), it simply contains a little more documentation in Transform.py. (What's the proper way to submit patches anyway .. through the bug tracker?) Dag Sverre -------------- next part -------------- A non-text attachment was scrubbed... Name: treetransform2.diff Type: text/x-patch Size: 32455 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080308/bccecb42/attachment-0001.bin From wstein at gmail.com Sat Mar 8 16:00:57 2008 From: wstein at gmail.com (William Stein) Date: Sat, 8 Mar 2008 07:00:57 -0800 Subject: [Cython] [sage-devel] linear algebra in Cython In-Reply-To: References: Message-ID: <85e81ba30803080700p1134b3d7x7198fce382ceb833@mail.gmail.com> On Fri, Mar 7, 2008 at 6:52 PM, Vincent wrote: > > I would like to implement some Bayesian routines using Cython. I have > looked at some of the Cython examples but they don't really provide > enough information for a newbie like me. As a starting point i'd like > to implement bootstrapping of OLS parameter standard errors. That > would give me an idea of how to implement (1) linear algebra in > Cython, i assume by calling routines from Numpy in some way, Much Python code will "just work" in Cython. You an just directly use numpy as you would from python, and it will work but be potentially slow depending on what you're doing. There is very active work going on in the Cython project to make it easy to write code that works blazingly quickly directly with numpy arrays, but that is _not_ in Cython yet, unfortunately, and we're not even sure what form it will take. > (2) > random number generators for sampling, numpy/scipy have a lot of those. > and (3) running loops. See David Roe's example from the previous email and browse wiki.cython.org > > I anyone has any (simple) examples of how to do something like this in > Cython i'd love to see them. Also, if there are better ways to do this > than through Cython i'd luv to hear your suggestions. > > Thanks, > > Vincent > > --~--~---------~--~----~------------~-------~--~----~ > To post to this group, send email to sage-devel at googlegroups.com > To unsubscribe from this group, send email to sage-devel-unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/group/sage-devel > URLs: http://www.sagemath.org > -~----------~----~----~----~------~----~------~--~--- > > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From dagss at student.matnat.uio.no Sat Mar 8 16:48:32 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 16:48:32 +0100 Subject: [Cython] Prototype for parametrized types Message-ID: <47D2B550.9060709@student.matnat.uio.no> Now that pipeline transforms are in place it is at least theoretically possible to take advantage of this: http://wiki.cython.org/enhancements/typeparameters Don't apply this to cython-dev just now, it's meant mostly for prototype + if somebody else wants to take over and finish it. This might mark the end of my coding sprint for now -- I won't break my neck by trying to go all the way to numpy arrays right away. -- Dag Sverre From martin at martincmartin.com Sat Mar 8 17:09:01 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 11:09:01 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D1A0A0.9000105@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> Message-ID: <47D2BA1D.1040202@martincmartin.com> Great! How does it compare to Lisp's macros? Best, Martin Dag Sverre Seljebotn wrote: > During my work to investigate how NumPy, C++ template etc. support could > best be added I discovered a way of integrating such code that has very > low impact on changing existing code and flow, yet does (I believe) have > the capacity to give quite powerful results. > > I've made a prototype: > - A framework in a 300 line diff > - An example using the framework: A reimplementation of the good old > for i in range(x, y) => for i from i <= ... transform, in 40 lines > > The whole thing and a long description is here: > > http://wiki.cython.org/enhancements/parsetreetransforms > > From dagss at student.matnat.uio.no Sat Mar 8 17:18:19 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 17:18:19 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BA1D.1040202@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> Message-ID: <47D2BC4B.7040008@student.matnat.uio.no> Martin C. Martin wrote: > Great! > > How does it compare to Lisp's macros? I'll try to answer, but I'm guessing about your assumptions so do ask again if I miss. This is work that is only relevant to the internal Cython compiler, so it doesn't compare (or compares very poorly) with LISP. Ie when transforming the parse tree one is transforming the input Cython source to output C source, but the transform itself doesn't live within that parse tree at all. If what you are asking instead is: How easy is it to write transforms compared to writing LISP macros, then the answer is still that it compares poorly, but then the goals are rather different too (the number of transforms will be max ~30 I think, and the number of LISP programs are... :-) ) Dag Sverre From martin at martincmartin.com Sat Mar 8 17:23:16 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 11:23:16 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BC4B.7040008@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> Message-ID: <47D2BD74.7030104@martincmartin.com> Ok, thanks a lot. My day job is programming Lisp, and since I started two years ago, I've seen the light about a number of things, including macros. It would be cool to have access to the Abstract Syntax Tree of a Cython expression/statement/function/etc., and be able to manipulate it at compile time. But it sounds like that's a whole other project. :) Best, Martin Dag Sverre Seljebotn wrote: > Martin C. Martin wrote: >> Great! >> >> How does it compare to Lisp's macros? > I'll try to answer, but I'm guessing about your assumptions so do ask > again if I miss. > > This is work that is only relevant to the internal Cython compiler, so > it doesn't compare (or compares very poorly) with LISP. Ie when > transforming the parse tree one is transforming the input Cython source > to output C source, but the transform itself doesn't live within that > parse tree at all. > > If what you are asking instead is: How easy is it to write transforms > compared to writing LISP macros, then the answer is still that it > compares poorly, but then the goals are rather different too (the number > of transforms will be max ~30 I think, and the number of LISP programs > are... :-) ) > > > Dag Sverre From dagss at student.matnat.uio.no Sat Mar 8 17:28:34 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 17:28:34 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BD74.7030104@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> Message-ID: <47D2BEB2.2080106@student.matnat.uio.no> Martin C. Martin wrote: > Ok, thanks a lot. > > My day job is programming Lisp, and since I started two years ago, > I've seen the light about a number of things, including macros. It > would be cool to have access to the Abstract Syntax Tree of a Cython > expression/statement/function/etc., and be able to manipulate it at > compile time. But it sounds like that's a whole other project. :) Hmm. I'm still not sure whether I've come across right :-) Accessing the abstract syntax tree is exactly what this is about, and you can do that (however the syntax for doing would seem rather complex and messy compared to with LISP, though a lot better I believe than what has been going on until now). The problem is that the code that accesses the syntax tree has to be written in Python and inserted into the compiler at launch time -- you cannot analyse yourself. In Python syntax, it is possible to do something remotely similar to: part-of-project.py: def myfunc(...): ... part-of-compiler.py: def modify_syntax_tree(tree): ... modify_syntax_tree(myfunc) However, you can NOT do modify_syntax_tree(modify_syntax_tree) or anything like that, which is why I believe it doesn't compare with LISP. Dag Sverre From ndbecker2 at gmail.com Sat Mar 8 18:27:47 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 08 Mar 2008 12:27:47 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> Message-ID: Martin C. Martin wrote: > Ok, thanks a lot. > > My day job is programming Lisp, and since I started two years ago, I've > seen the light about a number of things, including macros. It would be > cool to have access to the Abstract Syntax Tree of a Cython > expression/statement/function/etc., and be able to manipulate it at > compile time. But it sounds like that's a whole other project. :) > Wow! Someone is paid to program lisp? From stefan_ml at behnel.de Sat Mar 8 18:37:17 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Mar 2008 18:37:17 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D2B550.9060709@student.matnat.uio.no> References: <47D2B550.9060709@student.matnat.uio.no> Message-ID: <47D2CECD.40104@behnel.de> Dag Sverre Seljebotn wrote: > http://wiki.cython.org/enhancements/typeparameters Why do you think a new syntax for type parametrisation would be better than reusing (parts of) PEP 3107? http://www.python.org/dev/peps/pep-3107/ Stefan From robertwb at math.washington.edu Sat Mar 8 21:08:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 Mar 2008 12:08:22 -0800 Subject: [Cython] Enhancements proposals Message-ID: As some of you have probably noticed, I put up a new page on the wiki summarizing many of the enhancements that have been discussed here, and would be good to think about in light of the upcoming Sage development days (http://wiki.sagemath.org/dev1 a big part of which will be focused on improving Cython). There are also some very good proposals by Dag Seljebotn. The proposals are in varying degrees of completion--ranging from vague wishlist items to actual implementations. Feedback and input would be very beneficial, I would like these items to approach the specificity of PEP's and think that many if not most of them could be hammered out and implemented by/during the coding springs this June. http://wiki.cython.org/enhancements - Robert From robertwb at math.washington.edu Sat Mar 8 21:29:06 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 Mar 2008 12:29:06 -0800 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D2CECD.40104@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> Message-ID: <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> I think using parentheses for type parameterization is the most natural thing to do as well. (The only other viable syntax that comes to mind is angle brackets, and that's only natural to C++ people). This makes it valid Python syntax as well. I think this is orthogonal to PEP 3107, which specifies where the type specification should go (and I agree we should support this syntax too). The proposal is that anywhere the compiler expects a valid type, a parameterized type would work as well. The more I think about it, the more the plugin approach seems to be the right one to take, though I hope a lot can be specified in a .pxd without having to write custom plugins. These could be distributed with Cython, though not as part of the core compiler (see http:// wiki.cython.org/declaration ) - Robert On Mar 8, 2008, at 9:37 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> http://wiki.cython.org/enhancements/typeparameters > > Why do you think a new syntax for type parametrisation would be > better than > reusing (parts of) PEP 3107? > > http://www.python.org/dev/peps/pep-3107/ > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Sat Mar 8 21:57:04 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Mar 2008 21:57:04 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> Message-ID: <47D2FDA0.2030403@behnel.de> Hi, Robert Bradshaw wrote: > I think using parentheses for type parameterization is the most natural > thing to do as well. (The only other viable syntax that comes to mind is > angle brackets, and that's only natural to C++ people). Ok, I buy that. It would even resemble decorators to a certain extent. > This makes it valid Python syntax as well. How is that? > I think this is orthogonal to PEP 3107, which specifies where the type > specification should go (and I agree we should support this syntax too). > The proposal is that anywhere the compiler expects a valid type, a > parameterized type would work as well. Would that also allow you to distinguish things like "list" and "any subtype of list"? > The more I think about it, the more the plugin approach seems to be the > right one to take, though I hope a lot can be specified in a .pxd > without having to write custom plugins. These could be distributed with > Cython, though not as part of the core compiler (see > http://wiki.cython.org/declaration ) You could even ship them with Numpy and other projects instead of integrating them with Cython. Cython will nicely do without Numpy, but if you want to access Numpy through Cython, you will need Numpy installed first, so that's the right place to put a numpy.pxd for Cython. Stefan (BTW, you really should stop top-posting ...) From robertwb at math.washington.edu Sat Mar 8 22:09:26 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 Mar 2008 13:09:26 -0800 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D2FDA0.2030403@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> Message-ID: On Mar 8, 2008, at 12:57 PM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> I think using parentheses for type parameterization is the most >> natural >> thing to do as well. (The only other viable syntax that comes to >> mind is >> angle brackets, and that's only natural to C++ people). > > Ok, I buy that. It would even resemble decorators to a certain extent. > > >> This makes it valid Python syntax as well. > > How is that? foo(arg) is a valid cython expression, foo is not. The PEP specifies any valid expression can follow the : >> I think this is orthogonal to PEP 3107, which specifies where the >> type >> specification should go (and I agree we should support this syntax >> too). >> The proposal is that anywhere the compiler expects a valid type, a >> parameterized type would work as well. > > Would that also allow you to distinguish things like "list" and > "any subtype > of list"? Yes, but what good would it do to know something is a subtype of list? >> The more I think about it, the more the plugin approach seems to >> be the >> right one to take, though I hope a lot can be specified in a .pxd >> without having to write custom plugins. These could be distributed >> with >> Cython, though not as part of the core compiler (see >> http://wiki.cython.org/declaration ) > > You could even ship them with Numpy and other projects instead of > integrating > them with Cython. Cython will nicely do without Numpy, but if you > want to > access Numpy through Cython, you will need Numpy installed first, > so that's > the right place to put a numpy.pxd for Cython. Yep. But if we distributed with Cython then one wouldn't even have to tell the Cython compiler where to find the pxd files. Numpy is a bit of a special case, as they are actually going to use Cython and probably help write their own .pxd files. But for most libraries I think distributing them with Cython makes perfect sense (For example, I bet gmp is not about to shop Cython bindings to their library, for instance). > (BTW, you really should stop top-posting ...) Sorry... From dagss at student.matnat.uio.no Sun Mar 9 01:52:21 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 9 Mar 2008 01:52:21 +0100 (CET) Subject: [Cython] Patch ready (was: Re: Done with prototype for "parse tree transforms") In-Reply-To: <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> Message-ID: <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> > It would be nice to figure out a way that pluggins could be written > to be less brittle (i.e. not so dependent on the inner workings and > names of the Cython compiler) too, eventually. Totally agreed (and I've written about that too some places, especially that the -T switch will only ever be a developer tool). It's important for me that this transform framework isn't as much a plugin architecture as a change in the modularity of the compiler core itself. Currently (if I dare) the ambitions seems to be too high for the current code architecture and (in all modesty :-) ) I believe this approach goes a long way solving that problem. So this is the way I see it (and I'd guess you agree but I want it on the mailing list): - In an ideal world, it is simply wrong to have the methods acting on the parse tree implemented as methods on the parse tree itself. (See example below). Rather you want the parse tree to be "data" and seperate, componentized and well-defined transforms to be the vehicles of making the compilation happen. So ideally, the parse tree shouldn't be that brittle because there should be "nothing there" but information that is spec-ed according to the phases one is in. (However for plugin use it is definetely too complex still). - Working towards such an ideal code world is not the business one is in and is also very boring, so one makes compromises: Whenever new functionality is introduced, write it as transforms, whenever something new is impossible to write as transforms, make the minimal necesarry modifications towards making it possible to write is as transforms, but leave it at that. - I would however consider that type coercion should be split into a seperate CoercionTransform descending from the Transform class, rather than making a new coercion phase within the existing code-in-nodes paradigm. - At some point a new layer should definitely be written that allows simpler types of plugins. However (but I'm less sure of this than other things) I believe NumPy support and C++ template support can be written as transforms directly, because they are high priorities and to gain experience with what kind of plugin support would be most beneficial. - We should keep in the back of our heads that the transform problem has been solved in many ways in varying degrees of generalness (ick..too late in the night to look up the proper phrase). For instance it would be perfectly possible to export the tree to XML, run an XSLT transform, and parse the XML back, and then one could use XSLT for writing all the transforms (which would be much better in some cases than Python, and much worse in others). (I'm personally not that happy about XSLT in this particular case, though for many other uses it is perfect, but I'd do some research and look for libraries in general that can help write transforms. Something like doing XPath on this thing would help. Hmm...in fact there probably are Python XPath implementations that one could fit on our DOM, only question is whether it is worth the effort. If you don't know these just look them up). The important thing is that having a basic support for transforms doesn't say anything about different creative ways the transforms can be implemented in. Example for demonstration only: What if you want to switch your output to Java for Jython support? A lot of work, sure, but there's no reason it should be impossible *in principle* to fit something like that into Cython (there would be a lot of overlap in functionality, increasing as we get smarter with what Cython does). If C code generation is moved to a tree transform instead, it is simply a matter of replacing that one (big!) component, instead of a matter of breaking the code all over the place. (Not saying we ever attempt that though :-) better not, but contemplating it kind of makes me decide for which direction I like and don't like...) Dag Sverre From martin at martincmartin.com Sun Mar 9 02:01:46 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 20:01:46 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BEB2.2080106@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> Message-ID: <47D336FA.4030407@martincmartin.com> Dag Sverre Seljebotn wrote: > Martin C. Martin wrote: >> Ok, thanks a lot. >> >> My day job is programming Lisp, and since I started two years ago, >> I've seen the light about a number of things, including macros. It >> would be cool to have access to the Abstract Syntax Tree of a Cython >> expression/statement/function/etc., and be able to manipulate it at >> compile time. But it sounds like that's a whole other project. :) > Hmm. I'm still not sure whether I've come across right :-) > > Accessing the abstract syntax tree is exactly what this is about, and > you can do that (however the syntax for doing would seem rather complex > and messy compared to with LISP, though a lot better I believe than what > has been going on until now). > > The problem is that the code that accesses the syntax tree has to be > written in Python and inserted into the compiler at launch time -- you > cannot analyse yourself. So, how hard would it be to provide a mechanism to do this in the same file? There's a large subset of use cases for Lisp's macros which need the &body parameter. To do the equivalent in Python/Cython, you'd need a way to specify a block of code, much like a closure that doesn't take any parameters. Interesting... Best, Martin From dagss at student.matnat.uio.no Sun Mar 9 02:55:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 9 Mar 2008 02:55:39 +0100 (CET) Subject: [Cython] Compile-time macros In-Reply-To: <47D336FA.4030407@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> Message-ID: <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> Continuining the LISP thread... >> The problem is that the code that accesses the syntax tree has to be >> written in Python and inserted into the compiler at launch time -- you >> cannot analyse yourself. > > So, how hard would it be to provide a mechanism to do this in the same > file? There's a large subset of use cases for Lisp's macros which need Heh, ok, starting to understand you. Well, I can think of a way to have compile-time macro support in Cython that would make sense and be a natural part of other optimizations, however that is *far* in the future. This framework allows kind of the same thing, but it is so complicated and will easily break if Cython changes so it would practically never be worth the effort, especially when there is ample support for this in Python at run-time already. Python have for a long time allowed "run-time" macros, ie code generation, of course (though perhaps not as powerful as LISP, and gives a slower implementation): def make_adder(howmuch): def adder(x): return x + howmuch return adder add10_func = make_adder(10) print add10_func(z) # prints the value of z+10 Now, adding two seperate (hard!) features to Cython: 1) Supporting inner functions, ie closures, at all 2) Automatically collapse function calls where one can know that the code doesn't depend on anything that isn't known at compile time. Once this is done, the make_adder call above should automatically collapse and the add10 function be made directly compile-time. Try to read the clock or anything similar in make_adder though and it will be delayed to run-time. Of course one could have a @Macro annotation to make sure you got compiler errors if it couldn't be collapsed. (But NOTE: This is hard, and might never be done, I'm only dreaming. I don't see any real showstoppers for this but some hard work is needed and perhaps not as much return-on-investment as other exciting features one could add.) Dag Sverre From martin at martincmartin.com Sun Mar 9 03:25:31 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 21:25:31 -0500 Subject: [Cython] OT: programming Lisp for a living In-Reply-To: References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> Message-ID: <47D34A9B.3010406@martincmartin.com> Neal Becker wrote: > Martin C. Martin wrote: > >> Ok, thanks a lot. >> >> My day job is programming Lisp, and since I started two years ago, I've >> seen the light about a number of things, including macros. It would be >> cool to have access to the Abstract Syntax Tree of a Cython >> expression/statement/function/etc., and be able to manipulate it at >> compile time. But it sounds like that's a whole other project. :) >> > Wow! Someone is paid to program lisp? Yep, and coincidently I just found this blog tonight: http://lispjobs.wordpress.com/ Before starting there, I'd only used Lisp for classes, and a tiny little bit in grad school for simplifying some expressions. Things I've learned about Lisp since programming in it: 1. Lisp is *not* a functional language, it's a multi-paradigm language. That is, you can program in an imperative style if you want, you don't need any weird monads or anything. Most of our code is imperative and we don't use recursion much, only in places where it makes things clearer. 2. Lisp is the only dynamically typed language (that I know of) that cares about efficiency. It has optional static typing for efficiency, so you can say "this sub-expression will be of type X, so at runtime don't even bother looking at its type, just call the right method based the the type I'm telling you." It's about as fast as C/C++ if you avoid a few poorly thought out parts of the language. 3. Macros, which are basically functions that run at compile time that produce Lisp code, are really great. You can use them for efficiency, to make little domain languages, and other sorts of abstractions that are awkward/impossible in other languages. We're a very successful company that has a great culture; our managers don't see their jobs as checking up on you, but as getting problems out of your way so you can concentrate on your work; we worry about programmer's motivations and try hard to match projects to their interests, even letting them do their own projects if they're relevant and well thought out; and everyone is very friendly and relaxed. Check us out if you're interested, www.itasoftware.com , no Lisp knowledge required. Tell them I sent you. [I've never plugged like that to a general mailing list before, but since someone asked about getting paid to program Lisp... Sorry if it's a breach of ediquite.] Best, Martin From martin at martincmartin.com Sun Mar 9 03:42:22 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 21:42:22 -0500 Subject: [Cython] Compile-time macros In-Reply-To: <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> Message-ID: <47D34E8E.20705@martincmartin.com> Dag Sverre Seljebotn wrote: > Continuining the LISP thread... > >>> The problem is that the code that accesses the syntax tree has to be >>> written in Python and inserted into the compiler at launch time -- you >>> cannot analyse yourself. >> So, how hard would it be to provide a mechanism to do this in the same >> file? There's a large subset of use cases for Lisp's macros which need > > Heh, ok, starting to understand you. Well, I can think of a way to have > compile-time macro support in Cython that would make sense and be a > natural part of other optimizations, however that is *far* in the future. As you say later on, this is just dreaming. But dreaming is fun. :) Actually, I think there's a version, completely separate from the transform code that started this thread, that might not be hard at all. The basic idea of Lisp macros is that: 1. Lisp makes it easy to manipulate lists, including lists of lists. 2. In Lisp, all code is represented as lists. 3. So, it's pretty easy to write code which produces Lisp code. Of course, Python/Cython programs aren't represented as lists. However, programs can be naturally represented as an abstract syntax tree. In fact, that's a standard representation inside a compiler. It follows naturally from how a language is structured: a block contains 1 or more statements, the statements contain expressions, which contain sub-expressions, etc. It's essentially a Document Object Model (DOM) for source code, which is already hierarchically structured. So it would be easy to come up with a representation of Python/Cython program using standard Python data types, where elements correspond 1-1 with the parts of a program. So that takes care of 2, and Python has lots of ways to work easily with lists as well as many other data structures, which takes care of 1. So if we could introduce the concept of a special kind of function, which is run at compile time, and which takes the (unevaluated) parse trees of its arguments, instead of the runtime values of them, we could probably have something equivalent to Lisp macros. A macro would be a lot like transforming an XML DOM into another DOM, except instead of XML it's a parse tree. > Python have for a long time allowed "run-time" macros, ie code generation, > of course (though perhaps not as powerful as LISP, and gives a slower > implementation): > > def make_adder(howmuch): > def adder(x): return x + howmuch > return adder > > add10_func = make_adder(10) > print add10_func(z) # prints the value of z+10 > > Now, adding two seperate (hard!) features to Cython: > 1) Supporting inner functions, ie closures, at all > 2) Automatically collapse function calls where one can know that the code > doesn't depend on anything that isn't known at compile time. As you say, this is significantly different from Lisp macros, and the compiler needs to be very clever to make it efficient. A basic "defmacro," as outlined above, should be easy to implement, and to implement efficiently. Just a thought, Martin From robertwb at math.washington.edu Sun Mar 9 10:20:00 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 9 Mar 2008 01:20:00 -0800 Subject: [Cython] Patch ready (was: Re: Done with prototype for "parse tree transforms") In-Reply-To: <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> Message-ID: <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> On Mar 8, 2008, at 4:52 PM, Dag Sverre Seljebotn wrote: >> It would be nice to figure out a way that pluggins could be written >> to be less brittle (i.e. not so dependent on the inner workings and >> names of the Cython compiler) too, eventually. > > Totally agreed (and I've written about that too some places, > especially > that the -T switch will only ever be a developer tool). It's > important for > me that this transform framework isn't as much a plugin > architecture as a > change in the modularity of the compiler core itself. This is a worthy goal. > Currently (if I dare) the ambitions seems to be too high for the > current code architecture > and (in all modesty :-) ) I believe this approach goes a long way > solving > that problem. Yep. > So this is the way I see it (and I'd guess you agree but I want it > on the > mailing list): > - In an ideal world, it is simply wrong to have the methods acting > on the > parse tree implemented as methods on the parse tree itself. (See > example > below). Rather you want the parse tree to be "data" and seperate, > componentized and well-defined transforms to be the vehicles of > making the > compilation happen. So ideally, the parse tree shouldn't be that > brittle > because there should be "nothing there" but information that is > spec-ed > according to the phases one is in. (However for plugin use it is > definetely too complex still). > > - Working towards such an ideal code world is not the business one > is in > and is also very boring, so one makes compromises: Whenever new > functionality is introduced, write it as transforms, whenever > something > new is impossible to write as transforms, make the minimal necesarry > modifications towards making it possible to write is as transforms, > but > leave it at that. > > - I would however consider that type coercion should be split into a > seperate CoercionTransform descending from the Transform class, rather > than making a new coercion phase within the existing code-in-nodes > paradigm. This makes a lot of sense. I would like to move information about when the coercions are possible to the types themselves too. > - At some point a new layer should definitely be written that allows > simpler types of plugins. However (but I'm less sure of this than > other > things) I believe NumPy support and C++ template support can be > written as > transforms directly, because they are high priorities and to gain > experience with what kind of plugin support would be most beneficial. I think parameterized and intelligent types will play a key role here as well, and seem more modular than transforms looking for types with the string "numpy" in them. > - We should keep in the back of our heads that the transform > problem has > been solved in many ways in varying degrees of generalness > (ick..too late > in the night to look up the proper phrase). For instance it would be > perfectly possible to export the tree to XML, run an XSLT > transform, and > parse the XML back, and then one could use XSLT for writing all the > transforms (which would be much better in some cases than Python, > and much > worse in others). (I'm personally not that happy about XSLT in this > particular case, though for many other uses it is perfect, but I'd > do some > research and look for libraries in general that can help write > transforms. > Something like doing XPath on this thing would help. Hmm...in fact > there > probably are Python XPath implementations that one could fit on our > DOM, > only question is whether it is worth the effort. If you don't know > these > just look them up). Speaking specifically of XSLT, in your framework it would be possible to write a Transform that dumps out the tree as XML (like your printer, but fancier), runs some XSLT on it, then reads it back in. Most of my personal experience with XML has been unimpressive (usually 'cause it's way to bloated of a tool than needed for the task at hand, and slow). > The important thing is that having a basic support for > transforms doesn't say anything about different creative ways the > transforms can be implemented in. > > Example for demonstration only: What if you want to switch your > output to > Java for Jython support? A lot of work, sure, but there's no reason it > should be impossible *in principle* to fit something like that into > Cython > (there would be a lot of overlap in functionality, increasing as we > get > smarter with what Cython does). If C code generation is moved to a > tree > transform instead, it is simply a matter of replacing that one (big!) > component, instead of a matter of breaking the code all over the > place. > (Not saying we ever attempt that though :-) better not, but > contemplating > it kind of makes me decide for which direction I like and don't > like...) That would be one massively crazy transform to try and understand (especially because many relationships are very non-local), but in theory it could be viewed that way. I made a couple of changes to get it to run on Python 2.3 and merged it into the development repo. Something else I noticed was that you are looking at parse trees of single functions, rather than the module as a whole. (I'm guessing this is because the processing phases aren't in sync). It'd be good to change this. - Robert From dagss at student.matnat.uio.no Sun Mar 9 12:23:19 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 09 Mar 2008 12:23:19 +0100 Subject: [Cython] Compile-time macros In-Reply-To: <47D34E8E.20705@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> <47D34E8E.20705@martincmartin.com> Message-ID: <47D3C8A7.6000302@student.matnat.uio.no> > > So if we could introduce the concept of a special kind of function, > which is run at compile time, and which takes the (unevaluated) parse > trees of its arguments, instead of the runtime values of them, we > could probably have something equivalent to Lisp macros. A macro > would be a lot like transforming an XML DOM into another DOM, except > instead of XML it's a parse tree. > > A basic "defmacro," as outlined above, should be easy to implement, > and to implement efficiently. Hmm. I believe you are right. A problem though is that the parse tree is currently very subject to change all the time as Cython develops, so one would need to insert an "API" tree instead of a raw export I think, which would make it a bit more work. But not very much. This is certainly one perspective to take in the development of (perhaps parts of) NumPy support and so on, it could be based on macros. I don't think I'd vote for such an approach but the idea certainly has some merits and it might be worth it to pursue it a bit. I know that some kind of compile-time execution of code would be nice to have (proof: I once came across a mathematical C++ library that did extensive compile-time optimization of your code. How did they write it? Using C++ templates for compile-time meta-programming. Ugh. Imagine finding the square root of a number at compile time using C++ templates...) For the usecases I can think of the "compile-time evaluation of closures" would be more user-friendly, but of course one approach doesn't rule out doing the other later. Write a proposal in http://wiki.cython.org/enhancements perhaps? -- Dag Sverre From dagss at student.matnat.uio.no Sun Mar 9 12:49:02 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 09 Mar 2008 12:49:02 +0100 Subject: [Cython] Patch ready In-Reply-To: <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> Message-ID: <47D3CEAE.5030209@student.matnat.uio.no> > > Speaking specifically of XSLT, in your framework it would be possible > to write a Transform that dumps out the tree as XML (like your > printer, but fancier), runs some XSLT on it, then reads it back in. > Most of my personal experience with XML has been unimpressive (usually > 'cause it's way to bloated of a tool than needed for the task at hand, > and slow). True. But working a little on making transforms easy to write one might find some middle ground, rough example: class CoercionTransform(TemplateMatchTransform): match=lambda x: isinstance(x, AssignmentNode) and x.lhs.type != x.rhs.type def coerce_equals_assignment(node): other_usages_of_lhs = self.lookup_functionwide(lambda x: x.name == node.lhs.name) .... coerce_equals_assignment = make_template(coerce_equals_assignment, match) (Dropping 2.3 support would make this nicer.) -- Dag Sverre From stefan_ml at behnel.de Sun Mar 9 13:40:14 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Mar 2008 13:40:14 +0100 Subject: [Cython] XSLT based optimisation In-Reply-To: <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> Message-ID: <47D3DAAE.1060906@behnel.de> Hi, Robert Bradshaw wrote: > Speaking specifically of XSLT, in your framework it would be possible > to write a Transform that dumps out the tree as XML (like your > printer, but fancier), runs some XSLT on it, then reads it back in. > Most of my personal experience with XML has been unimpressive > (usually 'cause it's way to bloated of a tool than needed for the > task at hand, and slow). Hmm, slow, is it? Tried lxml lately? It's actually a funny idea to optimise code with a tool that is itself written in the compiled language. That way, lxml would basically optimise itself. :) Also, you could easily represent the parse tree in custom classes in lxml (although not the ones that Cython currently uses). And, lxml now has XSLT extension elements, meaning, you can write your own XSLT commands in Python and do stuff that you can't do in XSLT in plain Python code, like calculating static expressions, for example. How is that for a tool? Stefan From stefan_ml at behnel.de Sun Mar 9 13:46:47 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Mar 2008 13:46:47 +0100 Subject: [Cython] Patch ready In-Reply-To: <47D3CEAE.5030209@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> <47D3CEAE.5030209@student.matnat.uio.no> Message-ID: <47D3DC37.2060207@behnel.de> Hi, Dag Sverre Seljebotn wrote: >> Speaking specifically of XSLT, in your framework it would be possible >> to write a Transform that dumps out the tree as XML (like your >> printer, but fancier), runs some XSLT on it, then reads it back in. >> Most of my personal experience with XML has been unimpressive (usually >> 'cause it's way to bloated of a tool than needed for the task at hand, >> and slow). > > True. But working a little on making transforms easy to write one might > find some middle ground, rough example: > > class CoercionTransform(TemplateMatchTransform): > match=lambda x: isinstance(x, AssignmentNode) and x.lhs.type != x.rhs.type > def coerce_equals_assignment(node): > other_usages_of_lhs = self.lookup_functionwide(lambda x: x.name == > node.lhs.name) > .... > coerce_equals_assignment = make_template(coerce_equals_assignment, match) I would prefer a domain specific language for node selection here. Something like XPath, but it would do pattern matching against the AST of Cython to see which optimisations can be applied. As a starting point, here is a very simple XPath-like language: http://codespeak.net/svn/lxml/trunk/src/lxml/_elementpath.py > (Dropping 2.3 support would make this nicer.) I think that should still be avoided for now, although C code compatibility with Python 2.3 is more important than having the compiler itself run under Py2.3. Stefan From stefan_ml at behnel.de Sun Mar 9 13:52:43 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Mar 2008 13:52:43 +0100 Subject: [Cython] OT: programming Lisp for a living In-Reply-To: <47D34A9B.3010406@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D34A9B.3010406@martincmartin.com> Message-ID: <47D3DD9B.4040502@behnel.de> Hi, Martin C. Martin wrote: > 2. Lisp is the only dynamically typed language (that I know of) that > cares about efficiency. It has optional static typing for efficiency, > so you can say "this sub-expression will be of type X, so at runtime > don't even bother looking at its type, just call the right method based > the the type I'm telling you." It's about as fast as C/C++ if you avoid > a few poorly thought out parts of the language. There are very efficient /implicitly/ typed languages, though. Think of ML, for example. It has compile time type inference, so it's actually statically typed, you just don't see it (and you don't care either). Stefan From stefan_ml at behnel.de Sun Mar 9 13:58:50 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Mar 2008 13:58:50 +0100 Subject: [Cython] Enhancements proposals In-Reply-To: References: Message-ID: <47D3DF0A.2020704@behnel.de> Hi, Robert Bradshaw wrote: > The proposals are in varying degrees of completion > I would like these items to approach the specificity of PEP's I assume you meant CEP's here, right? :) Stefan From martin at martincmartin.com Sun Mar 9 14:59:08 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sun, 09 Mar 2008 09:59:08 -0400 Subject: [Cython] Compile-time macros In-Reply-To: <47D3C8A7.6000302@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> <47D34E8E.20705@martincmartin.com> <47D3C8A7.6000302@student.matnat.uio.no> Message-ID: <47D3ED2C.7030405@martincmartin.com> Dag Sverre Seljebotn wrote: > >> >> So if we could introduce the concept of a special kind of function, >> which is run at compile time, and which takes the (unevaluated) parse >> trees of its arguments, instead of the runtime values of them, we >> could probably have something equivalent to Lisp macros. A macro >> would be a lot like transforming an XML DOM into another DOM, except >> instead of XML it's a parse tree. >> >> A basic "defmacro," as outlined above, should be easy to implement, >> and to implement efficiently. > > Hmm. I believe you are right. A problem though is that the parse tree is > currently very subject to change all the time as Cython develops, so one > would need to insert an "API" tree instead of a raw export I think, > which would make it a bit more work. But not very much. Right, we might want to have two parse trees: an initial one for macros, that matches user concepts 1:1 with a stable API, which is then transformed into a second one, used internally by the compiler. Of course, I expect the translation from the first to the second to be just a few tweaks here and there, and so would be somewhat trivial. Maybe even a no-op. > This is certainly one perspective to take in the development of (perhaps > parts of) NumPy support and so on, it could be based on macros. I don't > think I'd vote for such an approach but the idea certainly has some > merits and it might be worth it to pursue it a bit. I think it's too early to recommend for or against it, but I think it's worth exploring. > I know that some kind of compile-time execution of code would be nice to > have (proof: I once came across a mathematical C++ library that did > extensive compile-time optimization of your code. How did they write it? > Using C++ templates for compile-time meta-programming. Ugh. Imagine > finding the square root of a number at compile time using C++ > templates...) Someone at work mentioned that, after programming in Lisp for many years, they had to learn their first purely functional language: C++ template meta-programming. It's Turing complete, but you don't even have loops, you have to implement them using recursion. > For the usecases I can think of the "compile-time > evaluation of closures" would be more user-friendly, but of course one > approach doesn't rule out doing the other later. > > Write a proposal in http://wiki.cython.org/enhancements perhaps? Will do. Best, Martin From stefan_ml at behnel.de Sun Mar 9 15:39:30 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Mar 2008 15:39:30 +0100 Subject: [Cython] Compile-time macros In-Reply-To: <47D34E8E.20705@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> <47D34E8E.20705@martincmartin.com> Message-ID: <47D3F6A2.9040308@behnel.de> Hi, Martin C. Martin wrote: > So if we could introduce the concept of a special kind of function, > which is run at compile time, and which takes the (unevaluated) parse > trees of its arguments, instead of the runtime values of them, we could > probably have something equivalent to Lisp macros. A macro would be a > lot like transforming an XML DOM into another DOM, except instead of XML > it's a parse tree. That reminds me of compile-time decorators. Imagine you could write something like @cython.memoize def dostuff(a,b): ... (special casing the 'cython' prefix here) and Cython would call memoize() with the *parse tree* of dostuff() as parameter. That would obviously use the same infrastructure as compiler plugins, but it would allow you to explicitly run plugins on your code. Stefan From stefan_ml at behnel.de Sun Mar 9 21:57:55 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 Mar 2008 21:57:55 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> Message-ID: <47D44F53.20305@behnel.de> Hi, Robert Bradshaw wrote: > On Mar 8, 2008, at 12:57 PM, Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> I think using parentheses for type parameterization is the most natural >>> thing to do as well. (The only other viable syntax that comes to mind is >>> angle brackets, and that's only natural to C++ people). >>> This makes it valid Python syntax as well. >> >> How is that? > > foo(arg) is a valid cython expression, foo is not. The PEP > specifies any valid expression can follow the : Regarding function parameters, yes, so that's definitely an advantage. But that does not yet give you type annotations for variables. If I understand the original proposal right, those would become cdef type.param(1,2,3) varname I don't see how you could embed this kind of declaration in pure Python. Also, a "ctypedef" might often be better than a parametrisation per use (but I guess that's up to the programmers). > Numpy is a bit of > a special case, as they are actually going to use Cython and probably > help write their own .pxd files. But for most libraries I think > distributing them with Cython makes perfect sense (For example, I bet > gmp is not about to shop Cython bindings to their library, for instance). Agreed. Stefan From dagss at student.matnat.uio.no Sun Mar 9 22:25:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 09 Mar 2008 22:25:04 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D44F53.20305@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> <47D44F53.20305@behnel.de> Message-ID: <47D455B0.90407@student.matnat.uio.no> > Regarding function parameters, yes, so that's definitely an advantage. But > that does not yet give you type annotations for variables. If I understand the > original proposal right, those would become > > cdef type.param(1,2,3) varname > This is going to be a repeat of what Robert already said, but I feel it is important that the point gets through: The original proposal did not say anything about how to declare a variable. It does a) say something about how to declare/use a type, anywhere a type might be needed, b) use the current syntax for the examples, because using a non-current syntax for examples would be silly (where would one stop? type inference is a real possibility, should I assume that for the examples?) We all agree that decorating variables in real Python is a real problem, but this proposal doesn't change anything for better or worse considering that. What this proposal says is that, given a hyopthetical CEP102 that proposes the following syntax for declaring types of in-function variables: i = typed(int) then a consequence of my proposal is that it is legal to also say v = typed(cpp.vector(int)) On the other hand, if the hypothetical CEP103 gets accepted instead which says "screw Python syntax" and makes it i: int also inside functions, then a consequence of my proposal is that it is legal to also say v: cpp.vector(int) inside functions. See how they are orthogonal, to use Robert's phrase? I'm sorry if the proposal was unclear about this. Dag Sverre From stefan_ml at behnel.de Mon Mar 10 08:10:20 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 10 Mar 2008 08:10:20 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D455B0.90407@student.matnat.uio.no> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> <47D44F53.20305@behnel.de> <47D455B0.90407@student.matnat.uio.no> Message-ID: <47D4DEDC.2000600@behnel.de> Hi, Dag Sverre Seljebotn wrote: > The original proposal did not say anything about how to declare a > variable. It does a) say something about how to declare/use a type, > anywhere a type might be needed, b) use the current syntax for the > examples, because using a non-current syntax for examples would be silly Sure. > We all agree that decorating variables in real Python is a real problem, > but this proposal doesn't change anything for better or worse > considering that. What this proposal says is that, given a hyopthetical > CEP102 that proposes the following syntax for declaring types of > in-function variables: > > i = typed(int) > > then a consequence of my proposal is that it is legal to also say > > v = typed(cpp.vector(int)) I hope it won't be that syntax, as this is what an assignment from a function call looks like, this is not a declaration. Declarative syntax should not be easily confused with syntax that results in real code doing something. > On the other hand, if the hypothetical CEP103 gets accepted instead > which says "screw Python syntax" and makes it > > i: int > > also inside functions, then a consequence of my proposal is that it is > legal to also say > > v: cpp.vector(int) > > inside functions. I hope it won't be that syntax either, as a ':' already has a very important meaning in Python: it starts a block. for i: int in range(100): print i I don't want that to become reality. I understand that it makes sense to use it in signatures, as it doesn't have a special meaning there. But it does have a very clear meaning in code. > See how they are orthogonal, to use Robert's phrase? > I'm sorry if the proposal was unclear about this. I wasn't arguing about the proposal itself. I said that prefixes and a function call like syntax look decorator-like, so they match the current way declarations work in the Python language. But I was suggesting that there is currently an asymmetry between how we can declare types in interfaces (i.e. function signatures) and inside code fragments. Pyrex has that already, as you don't need a cdef in signatures, but it doesn't get any better with PEP 3107. So the only advantage that a Python-like signature gives us is that it would be valid Python code if used in signatures. The more important place where type declarations are used (i.e. inside the code), is not impacted. Stefan From dagss at student.matnat.uio.no Mon Mar 10 09:13:19 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 10 Mar 2008 09:13:19 +0100 (CET) Subject: [Cython] Python syntax for type declarations In-Reply-To: <47D4DEDC.2000600@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> <47D44F53.20305@behnel.de> <47D455B0.90407@student.matnat.uio.no> <47D4DEDC.2000600@behnel.de> Message-ID: <63034.193.157.243.12.1205136799.squirrel@webmail.uio.no> Ok, I see where you are coming from now, sorry. > I hope it won't be that syntax either, as a ':' already has a very > important > meaning in Python: it starts a block. That's true. Even thought it shouold be inambiguous I think that fact alone will make sure Guido will never accept something like that into the language... I guess, if one already has a syntax and can't make up a better one it stays as it is... (perhaps optionally dropping the cdef keyword and allowing typing within for-loops would make it more friendly though). On the other hand: Automatic type inference within functions is a real possibility. I don't see type inference on the function arguments though (not of exportable functions at least), in fact, unless the arguments are typed one often couldn't infer the types of the body variables either. So using the PEP on the function arguments and leaving all variables in the body without type declaration might do the trick. Note: This is for the case the one uses Cython to speed up calculations etc.. when wrapping C code you probably want to manually type the code for type-checking purposes etc, but in that case the C syntax is "most natural" anyway because the audience will be C coders... Dag Sverre From robertwb at math.washington.edu Mon Mar 10 18:05:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 Mar 2008 10:05:20 -0700 Subject: [Cython] Compile-time macros In-Reply-To: <47D3C8A7.6000302@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> <47D34E8E.20705@martincmartin.com> <47D3C8A7.6000302@student.matnat.uio.no> Message-ID: <93A155E3-C1AC-4F91-A111-5A84377BA0F7@math.washington.edu> On Mar 9, 2008, at 4:23 AM, Dag Sverre Seljebotn wrote: >> So if we could introduce the concept of a special kind of function, >> which is run at compile time, and which takes the (unevaluated) parse >> trees of its arguments, instead of the runtime values of them, we >> could probably have something equivalent to Lisp macros. A macro >> would be a lot like transforming an XML DOM into another DOM, except >> instead of XML it's a parse tree. >> >> A basic "defmacro," as outlined above, should be easy to implement, >> and to implement efficiently. > > Hmm. I believe you are right. A problem though is that the parse > tree is > currently very subject to change all the time as Cython develops, > so one > would need to insert an "API" tree instead of a raw export I think, > which would make it a bit more work. But not very much. Fortunately for us, the parse tree follows the python grammar very closely, which is pretty static. This (plus a bit of extras like coercion and type declarations) is very close to what we would like to expose. The most brittle part is making new nodes (e.g. one has to know exactly all the right attributes). > This is certainly one perspective to take in the development of > (perhaps > parts of) NumPy support and so on, it could be based on macros. I > don't > think I'd vote for such an approach but the idea certainly has some > merits and it might be worth it to pursue it a bit. The full range of numpy indexing and slicing would probably be hard to implement in macros alone. > I know that some kind of compile-time execution of code would be > nice to > have (proof: I once came across a mathematical C++ library that did > extensive compile-time optimization of your code. How did they > write it? > Using C++ templates for compile-time meta-programming. Ugh. Imagine > finding the square root of a number at compile time using C++ > templates...) For the usecases I can think of the "compile-time > evaluation of closures" would be more user-friendly, but of course one > approach doesn't rule out doing the other later. One can keep in mind that our output is being passed to a C/C++ compiler, which are typically pretty good at eliminating many compile- time resolvable expressions and inlining functions. Our job is to write code that makes it easy for the C compiler to do so. - Robert From robertwb at math.washington.edu Mon Mar 10 18:08:00 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 Mar 2008 10:08:00 -0700 Subject: [Cython] Enhancements proposals In-Reply-To: <47D3DF0A.2020704@behnel.de> References: <47D3DF0A.2020704@behnel.de> Message-ID: On Mar 9, 2008, at 5:58 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> The proposals are in varying degrees of completion >> I would like these items to approach the specificity of PEP's > > I assume you meant CEP's here, right? :) I mean that I would like our CEPs to become as good as PEPs. I was trying to write a dozen of them in one sitting so didn't have time to flesh them all out. - Robert From robertwb at math.washington.edu Mon Mar 10 18:11:58 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 Mar 2008 10:11:58 -0700 Subject: [Cython] XSLT based optimisation In-Reply-To: <47D3DAAE.1060906@behnel.de> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> <47D3DAAE.1060906@behnel.de> Message-ID: <2B211105-BBD8-44DB-B112-A90B6FC4763F@math.washington.edu> On Mar 9, 2008, at 5:40 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> Speaking specifically of XSLT, in your framework it would be possible >> to write a Transform that dumps out the tree as XML (like your >> printer, but fancier), runs some XSLT on it, then reads it back in. >> Most of my personal experience with XML has been unimpressive >> (usually 'cause it's way to bloated of a tool than needed for the >> task at hand, and slow). > > Hmm, slow, is it? Tried lxml lately? :-). No, actually most of my excursions into XML were a long time ago, with Java and PHP, and for tasks better suited to a SQL database rather than a huge XML file. Being the wrong tool for the job left a bad taste in my mouth, but that's not the fault (I'm willing to believe) of XML. > It's actually a funny idea to optimise code with a tool that is > itself written > in the compiled language. That way, lxml would basically optimise > itself. :) > > Also, you could easily represent the parse tree in custom classes > in lxml > (although not the ones that Cython currently uses). And, lxml now has > XSLT extension elements, meaning, you can write your own XSLT > commands in > Python and do stuff that you can't do in XSLT in plain Python code, > like > calculating static expressions, for example. How is that for a tool? If we decide to use XML/XSLT, lxml certainly seems like the fitting tool :). - Robert From robertwb at math.washington.edu Mon Mar 10 18:23:52 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 Mar 2008 10:23:52 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D4DEDC.2000600@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> <47D44F53.20305@behnel.de> <47D455B0.90407@student.matnat.uio.no> <47D4DEDC.2000600@behnel.de> Message-ID: <59F77EA4-B446-443B-AD81-737803696DD5@math.washington.edu> On Mar 10, 2008, at 12:10 AM, Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >> The original proposal did not say anything about how to declare a >> variable. It does a) say something about how to declare/use a type, >> anywhere a type might be needed, b) use the current syntax for the >> examples, because using a non-current syntax for examples would be >> silly > > Sure. > > >> We all agree that decorating variables in real Python is a real >> problem, >> but this proposal doesn't change anything for better or worse >> considering that. What this proposal says is that, given a >> hyopthetical >> CEP102 that proposes the following syntax for declaring types of >> in-function variables: >> >> i = typed(int) >> >> then a consequence of my proposal is that it is legal to also say >> >> v = typed(cpp.vector(int)) > > I hope it won't be that syntax, as this is what an assignment from > a function > call looks like, this is not a declaration. Declarative syntax > should not be > easily confused with syntax that results in real code doing something. My suggestion would be something of the form i = cython.types.int v = cython.types.cpp.vector(int) This would also have the right feel to it if type inference is done (though in the this case, assignment to i of a non-int would not change the type of i). > But I was suggesting that there is currently an asymmetry between > how we can > declare types in interfaces (i.e. function signatures) and inside code > fragments. Pyrex has that already, as you don't need a cdef in > signatures, but > it doesn't get any better with PEP 3107. So the only advantage that a > Python-like signature gives us is that it would be valid Python > code if used > in signatures. The more important place where type declarations are > used (i.e. > inside the code), is not impacted. True. Function signatures are a special case because the syntax of a function declaration is constrained enough that type information is not easily confused with anything else (for both humans and computers). As it has been mentioned, I'm hopeful type inference within a function, with typed inputs, can go a long way towards making local type declarations unnecessary. - Robert From stefan_ml at behnel.de Mon Mar 10 19:44:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 10 Mar 2008 19:44:57 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <59F77EA4-B446-443B-AD81-737803696DD5@math.washington.edu> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> <47D44F53.20305@behnel.de> <47D455B0.90407@student.matnat.uio.no> <47D4DEDC.2000600@behnel.de> <59F77EA4-B446-443B-AD81-737803696DD5@math.washington.edu> Message-ID: <47D581A9.3090902@behnel.de> Hi, Robert Bradshaw wrote: > On Mar 10, 2008, at 12:10 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> We all agree that decorating variables in real Python is a real problem, >>> but this proposal doesn't change anything for better or worse >>> considering that. What this proposal says is that, given a hyopthetical >>> CEP102 that proposes the following syntax for declaring types of >>> in-function variables: >>> >>> i = typed(int) >>> >>> then a consequence of my proposal is that it is legal to also say >>> >>> v = typed(cpp.vector(int)) >> >> I hope it won't be that syntax, as this is what an assignment from a >> function >> call looks like, this is not a declaration. Declarative syntax should >> not be >> easily confused with syntax that results in real code doing something. > > My suggestion would be something of the form > > i = cython.types.int > v = cython.types.cpp.vector(int) > > This would also have the right feel to it if type inference is done > (though in the this case, assignment to i of a non-int would not change > the type of i). I'm still not convinced that assignment is a good way of dealing with type declarations. Wouldn't something like this work as well: def dostuff(a : int, b : str): type(c, char*) c = b or def dostuff(a : int, b : str): assert type(c, char*) c = b or maybe def dostuff(a : int, b : str): cython.assert_type(c, char*) c = b This /would/ be valid Python syntax as long as you use only plain Python types. And it's definitely more intuitive than an assignment. Alternatively, I could imagine typing values at the first assignment. So, what about this: def dostuff(a : int, b : str): c = cython.assigntype(char*, b) Again, plain Python if you stick to Python types. But this time, it really *is* an assignment. Stefan From stefan_ml at behnel.de Mon Mar 10 19:55:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 10 Mar 2008 19:55:57 +0100 Subject: [Cython] Compile-time macros In-Reply-To: <93A155E3-C1AC-4F91-A111-5A84377BA0F7@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> <47D34E8E.20705@martincmartin.com> <47D3C8A7.6000302@student.matnat.uio.no> <93A155E3-C1AC-4F91-A111-5A84377BA0F7@math.washington.edu> Message-ID: <47D5843D.5030201@behnel.de> Hi, Robert Bradshaw wrote: > On Mar 9, 2008, at 4:23 AM, Dag Sverre Seljebotn wrote: > Fortunately for us, the parse tree follows the python grammar very > closely, which is pretty static. This (plus a bit of extras like > coercion and type declarations) is very close to what we would like > to expose. The most brittle part is making new nodes (e.g. one has to > know exactly all the right attributes). If it's only for a public plugin API, we could hide the real tree classes behind factory functions in a separate module that would have the same name as the tree nodes, but that would use explicit keyword arguments. That way, you will at least know when you pass an unsupported keyword, and you can generate epydoc documentation for the public API. The only problem I see is that you couldn't just say "isinstance(node, MyFactoryNode)" (MyFactoryNode being the factory function here). But hacking something in the line of "MyFactoryNode.isinstance(node)" isn't that bad either, is it? Stefan From robertwb at math.washington.edu Mon Mar 10 21:29:44 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 Mar 2008 13:29:44 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D581A9.3090902@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> <47D44F53.20305@behnel.de> <47D455B0.90407@student.matnat.uio.no> <47D4DEDC.2000600@behnel.de> <59F77EA4-B446-443B-AD81-737803696DD5@math.washington.edu> <47D581A9.3090902@behnel.de> Message-ID: <700FA5E7-A5E5-4FEB-9A06-0FAADCB26959@math.washington.edu> On Mar 10, 2008, at 11:44 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> On Mar 10, 2008, at 12:10 AM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> We all agree that decorating variables in real Python is a real >>>> problem, >>>> but this proposal doesn't change anything for better or worse >>>> considering that. What this proposal says is that, given a >>>> hyopthetical >>>> CEP102 that proposes the following syntax for declaring types of >>>> in-function variables: >>>> >>>> i = typed(int) >>>> >>>> then a consequence of my proposal is that it is legal to also say >>>> >>>> v = typed(cpp.vector(int)) >>> >>> I hope it won't be that syntax, as this is what an assignment from a >>> function >>> call looks like, this is not a declaration. Declarative syntax >>> should >>> not be >>> easily confused with syntax that results in real code doing >>> something. >> >> My suggestion would be something of the form >> >> i = cython.types.int >> v = cython.types.cpp.vector(int) >> >> This would also have the right feel to it if type inference is done >> (though in the this case, assignment to i of a non-int would not >> change >> the type of i). > > I'm still not convinced that assignment is a good way of dealing > with type > declarations. > > Wouldn't something like this work as well: > > def dostuff(a : int, b : str): > type(c, char*) > c = b > > or > > def dostuff(a : int, b : str): > assert type(c, char*) > c = b > > or maybe > > def dostuff(a : int, b : str): > cython.assert_type(c, char*) > c = b > > This /would/ be valid Python syntax as long as you use only plain > Python > types. And it's definitely more intuitive than an assignment. Perhaps. This would give you a runtime error in python though. > Alternatively, I could imagine typing values at the first > assignment. So, what > about this: > > def dostuff(a : int, b : str): > c = cython.assigntype(char*, b) > > Again, plain Python if you stick to Python types. But this time, it > really > *is* an assignment. Looking at stuff like this makes me realize how nice the cdef keyword really is... - Robert From dagss at student.matnat.uio.no Mon Mar 10 23:03:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 10 Mar 2008 23:03:00 +0100 Subject: [Cython] Prototype for parametrized types Message-ID: <3288034995.3535600@smtp.netcom.no> (I take the liberty to top-post when I'm on cellular...) I like Robert's proposal, but with a twist: It is NOT a declaration, but an automatically inferred variable and a cast. So if foo() returns a Python int (object), then you can not write a = cython.types.int a = foo() but have to write a = cython.types.int(foo()) The former example is very confusing to Pythoners, while the second is very consistent: Some *classes*/types have compile-time optimizations when they are operated on (whether numpy.ndarray or cython.types.int), but variable semantic actually follow Python's. We compile actions on *types* to C, not actions on variables. Declaring types on arguments are then simply a short-hand for "always run argument through the given factory/type constructor", and can be implemented for Python types as well, so def foo(a : unicode): would make sure int and str params are converted to unicode. I believe this to be very in the spirit of rest of Python, and that it wouldn't be that much more verbose. The cost is lack of compiler warnings - but one share that with the rest of Python, and I believe we type for speed, not design? Perhaps easy type asserts can then be used in addition if one wants to check that the code is compiled efficiently... Dag Sverre Dag Sverre Seljebotn -----Original Message----- From: Robert Bradshaw Date: Monday, Mar 10, 2008 9:30 pm Subject: Re: [Cython] Prototype for parametrized types To: Cc: Dag Sverre Seljebotn , cython-dev at codespeak.net Stefan Behnel CC: Dag Sverre Seljebotn , cython-dev at codespeak.netTo: Stefan Behnel On Mar 10, 2008, at 11:44 AM, Stefan Behnel wrote: > >> Hi, > >> Robert Bradshaw wrote: >> On Mar 10, 2008, at 12:10 AM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> We all agree that decorating variables in real Python is a real >>>> problem, >>>> but this proposal doesn't change anything for better or worse >>>> considering that. What this proposal says is that, given a >>>> hyopthetical >>>> CEP102 that proposes the following syntax for declaring types of >>>> in-function variables: >>>> >>>> i = typed(int) >>>> >>>> then a consequence of my proposal is that it is legal to also say >>>> >>>> v = typed(cpp.vector(int)) >>> >>> I hope it won't be that syntax, as this is what an assignment from a >>> function >>> call looks like, this is not a declaration. Declarative syntax >>> should >>> not be >>> easily confused with syntax that results in real code doing >>> something. >> >> My suggestion would be something of the form >> >> i = cython.types.int >> v = cython.types.cpp.vector(int) >> >> This would also have the right feel to it if type inference is done >> (though in the this case, assignment to i of a non-int would not >> change >> the type of i). > >> I'm still not convinced that assignment is a good way of dealing > with type > declarations. > >> Wouldn't something like this work as well: > >> def dostuff(a : int, b : str): > type(c, char*) > c = b > >> or > >> def dostuff(a : int, b : str): > assert type(c, char*) > c = b > >> or maybe > >> def dostuff(a : int, b : str): > cython.assert_type(c, char*) > c = b > >> This /would/ be valid Python syntax as long as you use only plain > Python > types. And it's definitely more intuitive than an assignment. > >Perhaps. This would give you a runtime error in python though. > >> Alternatively, I could imagine typing values at the first > assignment. So, what > about this: > >> def dostuff(a : int, b : str): > c = cython.assigntype(char*, b) > >> Again, plain Python if you stick to Python types. But this time, it > really > *is* an assignment. > >Looking at stuff like this makes me realize how nice the cdef keyword >really is... > >- Robert > > From dagss at student.matnat.uio.no Mon Mar 10 23:16:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 10 Mar 2008 23:16:00 +0100 Subject: [Cython] Prototype for parametrized types Message-ID: <3288035817.3617704@smtp.netcom.no> I can see that my syntax will conflict with optionally parametrized types. An approach (but I'll admit ot is getting a bit convoluted): - numpy.ndarray(numpy.uint8, 2, arr) converts arr to the given type. Ie mix xompile-time and runtime params (but only the Cython developers need to care, an UnsupportedAtRuntimeError will be customary anyway...) - if missing the value parameter, a constructor os returned instead, so def foo(a : ndarray(uint8, 2)) has the same meaning as the (also legal) def foo(a): tmptype = ndarray(uint8, 2) a = tmptype(a) BTW, there's my suggestion for a Pythonic typedef as well. - The auto-converted argument mechanism can always pass "value" by name for extra safety. This really make it feel like a Python library and has no learning curve for experienced Python users? C library writers can keep the old syntax of course. Dag Sverre From dagss at student.matnat.uio.no Mon Mar 10 23:26:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 10 Mar 2008 23:26:00 +0100 Subject: [Cython] Another type idea Message-ID: <3288036381.3674092@smtp.netcom.no> Getting ideas after posting... One could also declare known return type of existing ptyhon lib functions like this: assumetype(cython.types.int, sum) def foo(arr): s = sum(arr) # s is now native C int leading to eliminating all the casts. Dag Sverre Seljebotn From robertwb at math.washington.edu Mon Mar 10 23:52:35 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 Mar 2008 15:52:35 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <3288034995.3535600@smtp.netcom.no> References: <3288034995.3535600@smtp.netcom.no> Message-ID: <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> On Mar 10, 2008, at 3:03 PM, Dag Sverre Seljebotn wrote: > (I take the liberty to top-post when I'm on cellular...) > > I like Robert's proposal, but with a twist: It is NOT a > declaration, but an automatically inferred variable and a cast. Handling explicit casts is easy, and I like the syntax below, but I am trying to come up with a way to do an actual type declaration (which may or may not be part of an assignment). For example, think of code that uses temporary mpz_t values (if you're not familiar with gmp, see http://trac.sagemath.org/sage_trac/attachment/ticket/2155/ trac-2155.patch for a concrete example). Also, one of the things that makes Cython so great is that it handles type conversions automatically--having to do so manually every time would be a big step back. Ideally, I'd like some syntax that could be run as valid Python (no syntax or name errors) but declare a specific variable to be a type, which is more powerful than using assignment + type inference (for example, one could add several python ints to a and it would do the right thing). > So if foo() returns a Python int (object), then you can not write > > a = cython.types.int > a = foo() > > but have to write > > a = cython.types.int(foo()) > > The former example is very confusing to Pythoners, while the second > is very consistent: Some *classes*/types have compile-time > optimizations when they are operated on (whether numpy.ndarray or > cython.types.int), but variable semantic actually follow Python's. > We compile actions on *types* to C, not actions on variables. Yes. But to discover types, a variable <-> type mapping is used. (In the distant future different variables could even be typed differently in different parts of a function, but that would be more complicated to write and more difficult to understand as a user). > Declaring types on arguments are then simply a short-hand for > "always run argument through the given factory/type constructor", > and can be implemented for Python types as well, so > > def foo(a : unicode): > > would make sure int and str params are converted to unicode. Yes. > I believe this to be very in the spirit of rest of Python, and that > it wouldn't be that much more verbose. The cost is lack of compiler > warnings - but one share that with the rest of Python, and I > believe we type for speed, not design? Compiler warnings are very, very nice (and more valuable in Cython as the edit-compile-run cycle takes much longer. > Perhaps easy type asserts can then be used in addition if one wants > to check that the code is compiled efficiently... This is a good idea. - Robert From robertwb at math.washington.edu Mon Mar 10 23:55:48 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 Mar 2008 15:55:48 -0700 Subject: [Cython] Another type idea In-Reply-To: <3288036381.3674092@smtp.netcom.no> References: <3288036381.3674092@smtp.netcom.no> Message-ID: <266AA584-28F9-4F2B-B9E7-C217010B8FFA@math.washington.edu> On Mar 10, 2008, at 3:26 PM, Dag Sverre Seljebotn wrote: > Getting ideas after posting... > > One could also declare known return type of existing ptyhon lib > functions like this: > > assumetype(cython.types.int, sum) > > def foo(arr): > s = sum(arr) > # s is now native C int > > leading to eliminating all the casts. I'm not so sure about this--overriding the declared types of builtin functions, the non-locality of the declaration (assumetype could be anywhere in the file, or imported from somewhere else, but the conversion happens inside the function). - Robert From dagss at student.matnat.uio.no Tue Mar 11 00:48:58 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 11 Mar 2008 00:48:58 +0100 (CET) Subject: [Cython] Prototype for parametrized types In-Reply-To: <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> Message-ID: <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> > Ideally, I'd like some syntax that could be run as valid Python (no > syntax or name errors) but declare a specific variable to be a type, > which is more powerful than using assignment + type inference (for > example, one could add several python ints to a and it would do the > right thing). Overloading += and + would help, but I know that's not the point :-) Perhaps then a variant on the SAGE approach with vars("x") or similar? Using a string for the variable name makes it more natural for Python users since they know x is not declared yet, and you can already do locals()["x"] which is almost as "wierd". So typed("x", cython.types.int, "s,w", cython.types.ptr(cython.types.char)) w = "ok" x = "compiler error" If consistency is important then even @cython(args=("x", cython.types.int), ret=cython.types.int, except=-1) def foo(x): typed("sq", cython.types.int) square = x*x return sq But it might be too verbose. Similar example not typing, for comparison: def bar(x: cython.types.int): assert fullycompiled # True if no object references made in block and sub-blocks sq = x * x assert native(sq) # Always True because of fully-compiled, but... return sq Doesn't seem to be anything wrong with combining both and let the user pick (guess that is the meaning of "type inference" - but I see the issues clearer now, guess I'm lagging behind a bit). Still - if there are two modes of operations: 1) Don't type variables but use inference heavily 2) Explicitly typed variable names then perhaps it isn't critical that 2) has a valid Python syntax if 1) has it? (Can always add some candy to current syntax, like allowing "for int i in range(..." and make the "cdef" keyword optional). Dag Sverre From dagss at student.matnat.uio.no Tue Mar 11 00:52:26 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 11 Mar 2008 00:52:26 +0100 (CET) Subject: [Cython] Lock the wiki front page Message-ID: <65028.193.157.243.12.1205193146.squirrel@webmail.uio.no> Looks like somebody with the permissions for it should lock the wiki front page, it's been heavily spammed all day. Oddly they don't do anything with sub-pages... Dag Sverre From stefan_ml at behnel.de Tue Mar 11 09:03:50 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 09:03:50 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <700FA5E7-A5E5-4FEB-9A06-0FAADCB26959@math.washington.edu> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> <47D44F53.20305@behnel.de> <47D455B0.90407@student.matnat.uio.no> <47D4DEDC.2000600@behnel.de> <59F77EA4-B446-443B-AD81-737803696DD5@math.washington.edu> <47D581A9.3090902@behnel.de> <700FA5E7-A5E5-4FEB-9A06-0FAADCB26959@math.washington.edu> Message-ID: <47D63CE6.60101@behnel.de> Hi Robert, Robert Bradshaw wrote: > On Mar 10, 2008, at 11:44 AM, Stefan Behnel wrote: >> Alternatively, I could imagine typing values at the first assignment. >> So, what about this: >> >> def dostuff(a : int, b : str): >> c = cython.assigntype(char*, b) >> >> Again, plain Python if you stick to Python types. But this time, it >> really *is* an assignment. > > Looking at stuff like this makes me realize how nice the cdef keyword > really is... :) I have to agree here. I even think a simple code translator script that removes the cdefs and outputs clean Python code would do more good than trying to squeeze Cython's type annotations into Python-looking or even Python-parsable expressions. After all, cdefs are two things: a) a way to interact with C code, in which case it's not Python code anyway. b) a way to optimise your code by hand. And if you want your code to run fast when compiled to C, you probably do not care much about running it in plain interpreted Python (especially if you can provide an equivalent pure Python module by passing your code to a simple translator...) Stefan From stefan_ml at behnel.de Tue Mar 11 09:08:32 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 09:08:32 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <3288034995.3535600@smtp.netcom.no> References: <3288034995.3535600@smtp.netcom.no> Message-ID: <47D63E00.1030307@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Declaring types on arguments are then simply a short-hand for "always run > argument through the given factory/type constructor", and can be > implemented for Python types as well, so > > def foo(a : unicode): > > would make sure int and str params are converted to unicode. No way. Converting a byte string to unicode is something that has to be done explicitly. How could you know the input encoding that is used for the byte sequence? And while automatic conversion of numbers to strings is less problematic, I would not expect my language to do that for me (except for printing, maybe). Stefan From robertwb at math.washington.edu Tue Mar 11 09:13:03 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Mar 2008 01:13:03 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> Message-ID: <54A4034B-2375-425D-B403-CCA830121939@math.washington.edu> On Mar 10, 2008, at 4:48 PM, Dag Sverre Seljebotn wrote: >> Ideally, I'd like some syntax that could be run as valid Python (no >> syntax or name errors) but declare a specific variable to be a type, >> which is more powerful than using assignment + type inference (for >> example, one could add several python ints to a and it would do the >> right thing). > > Overloading += and + would help, but I know that's not the point :-) > > Perhaps then a variant on the SAGE approach with vars("x") or similar? > Using a string for the variable name makes it more natural for Python > users since they know x is not declared yet, and you can already do > locals()["x"] which is almost as "wierd". So > > typed("x", cython.types.int, "s,w", cython.types.ptr > (cython.types.char)) > w = "ok" > x = "compiler error" I agree with the wierdness here, and I'm not sure it even works sage: def foo(): locals()['x'] = 3; print x, locals()['x'] ....: sage: foo() > If consistency is important then even > > @cython(args=("x", cython.types.int), ret=cython.types.int, except=-1) > def foo(x): > typed("sq", cython.types.int) > square = x*x > return sq > > But it might be too verbose. IMHO, yes, this is getting way to verbose, but it could be accepted (though the P3K ways is preferred). Perhaps there could be a locals dict that is passed in as well, i.e. @cython(arg_types={"x": cython.types.int}, local_types={"square", cython.types.int}, return_type=cython.types.int, except=-1) def foo(x): square = x*x return square > Similar example not typing, for comparison: > > def bar(x: cython.types.int): > assert fullycompiled # True if no object references made in block > and > sub-blocks > sq = x * x > assert native(sq) # Always True because of fully-compiled, but... > return sq > > Doesn't seem to be anything wrong with combining both and let the user > pick (guess that is the meaning of "type inference" - but I see the > issues > clearer now, guess I'm lagging behind a bit). > > Still - if there are two modes of operations: > 1) Don't type variables but use inference heavily > 2) Explicitly typed variable names > > then perhaps it isn't critical that 2) has a valid Python syntax if > 1) has > it? (1) can be done on pure Python code, but without any types to start with it is fairly limited. I think specifying types in function declarations can go a long way to make it powerful though. I think there should be a valid way to do (2) as well, but it might not be as nice. > (Can always add some candy to current syntax, like allowing "for int i > in range(..." and make the "cdef" keyword optional). Even removing the cdef keyword doesn't make it valid Python (and its presence makes parsing so much easier), so I don't see much of a gain there. - Robert From stefan_ml at behnel.de Tue Mar 11 09:36:04 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 09:36:04 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <54A4034B-2375-425D-B403-CCA830121939@math.washington.edu> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <54A4034B-2375-425D-B403-CCA830121939@math.washington.edu> Message-ID: <47D64474.7030203@behnel.de> Hi, Robert Bradshaw wrote: > On Mar 10, 2008, at 4:48 PM, Dag Sverre Seljebotn wrote: >> @cython(args=("x", cython.types.int), ret=cython.types.int, except=-1) >> def foo(x): >> >> But it might be too verbose. > > IMHO, yes, this is getting way to verbose, but it could be accepted > (though the P3K ways is preferred). Perhaps there could be a locals > dict that is passed in as well, i.e. > > @cython(arg_types={"x": cython.types.int}, local_types={"square", > cython.types.int}, return_type=cython.types.int, except=-1) > def foo(x): > square = x*x > return square And it would be trivial to implement with the decorator plugin infrastructure. Remember my proposal to let compile time decorators operate on the parse tree? In this case, it would traverse the tree of the function, and just annotate all occurrences of parameters ("x"), the return value, exception raising, and the named local variables ("square") with their provided types, *before* running any further type inference or analysis steps over the tree. Then, the type analysis step would already know the expected types and could raise errors on assignments of unknown or unexpected types. So we'd have a couple of interception points where decorators and plugins can run: post-parse, pre-type-analysis, post-type-analysis, ... A plugin could just set that up on registration and Cython would call it at the right time and then continue working on the modified syntax tree. One drawback of the decorator notation: I don't think you can express pointers and external C types (structs etc.) in any sensible way, but if I understand this right, this proposal is only for type annotation of otherwise plain Python code, right? Stefan From robertwb at math.washington.edu Tue Mar 11 09:41:35 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Mar 2008 01:41:35 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D64474.7030203@behnel.de> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <54A4034B-2375-425D-B403-CCA830121939@math.washington.edu> <47D64474.7030203@behnel.de> Message-ID: <4A02EA2E-53C1-48D3-B309-81876ED6B92E@math.washington.edu> On Mar 11, 2008, at 1:36 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> On Mar 10, 2008, at 4:48 PM, Dag Sverre Seljebotn wrote: >>> @cython(args=("x", cython.types.int), ret=cython.types.int, >>> except=-1) >>> def foo(x): >>> >>> But it might be too verbose. >> >> IMHO, yes, this is getting way to verbose, but it could be accepted >> (though the P3K ways is preferred). Perhaps there could be a locals >> dict that is passed in as well, i.e. >> >> @cython(arg_types={"x": cython.types.int}, local_types={"square", >> cython.types.int}, return_type=cython.types.int, except=-1) >> def foo(x): >> square = x*x >> return square > > And it would be trivial to implement with the decorator plugin > infrastructure. > Remember my proposal to let compile time decorators operate on the > parse tree? > In this case, it would traverse the tree of the function, and just > annotate > all occurrences of parameters ("x"), the return value, exception > raising, and > the named local variables ("square") with their provided types, > *before* > running any further type inference or analysis steps over the tree. > > Then, the type analysis step would already know the expected types > and could > raise errors on assignments of unknown or unexpected types. > > So we'd have a couple of interception points where decorators and > plugins can > run: post-parse, pre-type-analysis, post-type-analysis, ... A > plugin could > just set that up on registration and Cython would call it at the > right time > and then continue working on the modified syntax tree. > > One drawback of the decorator notation: I don't think you can > express pointers > and external C types (structs etc.) in any sensible way, Yeah, it could get ugly: cython.types.pointer(cython.types.pointer (cython.types.int))." At this point string processing might not be bad, i.e. cython.type("int**") or even "int**" if it's clear from context that we're specifying a type. (I do want to avoid putting to much semantic information into "black box" strings though). > but if I understand > this right, this proposal is only for type annotation of otherwise > plain > Python code, right? Yep. - Robert From stefan_ml at behnel.de Tue Mar 11 09:58:18 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 09:58:18 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <4A02EA2E-53C1-48D3-B309-81876ED6B92E@math.washington.edu> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <54A4034B-2375-425D-B403-CCA830121939@math.washington.edu> <47D64474.7030203@behnel.de> <4A02EA2E-53C1-48D3-B309-81876ED6B92E@math.washington.edu> Message-ID: <47D649AA.2030801@behnel.de> Hi, Robert Bradshaw wrote: > On Mar 11, 2008, at 1:36 AM, Stefan Behnel wrote: >>> On Mar 10, 2008, at 4:48 PM, Dag Sverre Seljebotn wrote: >>>> @cython(args=("x", cython.types.int), ret=cython.types.int, except=-1) >>>> def foo(x): >> >> One drawback of the decorator notation: I don't think you can express >> pointers and external C types (structs etc.) in any sensible way, > > Yeah, it could get ugly: > cython.types.pointer(cython.types.pointer(cython.types.int))." At this > point string processing might not be bad, i.e. cython.type("int**") or > even "int**" if it's clear from context that we're specifying a type. (I > do want to avoid putting to much semantic information into "black box" > strings though). We could provide standard pointer and array types (p_int, int_array), but there are always limits. Something like cython.parse_type("int**") would work nicely, though, and would obviously reuse the parser, so you'd still get compile time syntax checking. And since the decorator annotations would run before the type analysis, undeclared types can be caught as well. Stefan From dagss at student.matnat.uio.no Tue Mar 11 09:59:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 11 Mar 2008 09:59:39 +0100 (CET) Subject: [Cython] Prototype for parametrized types In-Reply-To: <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> Message-ID: <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> I'm combining many emails into one. Robert: > Yeah, it could get ugly: cython.types.pointer(cython.types.pointer > (cython.types.int))." I think "cython.types" is too longwinded anyway, in practice it would be: import cython.types as c c.ptr(c.ptr(c.int)) Which isn't *that* bad? And one could provide all kind of convenience pre-declared names: c.pint = c.ptr(c.int) c.ppint = ... c.pchar = ... c.ppchar = ... Perhaps just make a convention of providing up to three "ppp"-types for every type, and then have a parametrized type c.ptrs(6, c.int) if you really want to do int******. PChar is already a well-known synonym for char* in some codebases. >> 1) Don't type variables but use inference heavily >> 2) Explicitly typed variable names > > (1) can be done on pure Python code, but without any types to start > with it is fairly limited. I think specifying types in function > declarations can go a long way to make it powerful though. I think Yes, I meant type inference with: - Type constructors as conversion operators (a la C++) - A function decorator that passes the arguments to their decorators I.e. 1) has every possibility to be let the types be explicit, but through the normal Python way of doing that (ie not binding the variables to a type but making sure in code that one knows what type the objects are). This has the most Python feel to it and would make people quickly be able to use it - while the non-standard declare type syntax always is around if one prefers it. A last (desperate) syntax for type declarations: with cython.types.int as x, cython.types.int as y: (Some time since I read the with PEP so don't know if this is entirely right, think multiple statements are a proposed future feature or something.) Stephan wrote: >> >> def foo(a : unicode): >> >> would make sure int and str params are converted to unicode. > > No way. Converting a byte string to unicode is something that has to be > done > explicitly. How could you know the input encoding that is used for the > byte > sequence? It wasn't a proposed or wanted functionality, it was only a demonstration of the fact that as a Python user, if def f(a: cython.types.int): ... automatically construct a Cython int on entering, I kind of expect to be able to do: def twice_factory(x): return cython.types.int(x) * 2 def f(a: twice_factory): ... And have f automatically have its argument doubled Dag Sverre From stefan_ml at behnel.de Tue Mar 11 10:18:34 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 10:18:34 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> Message-ID: <47D64E6A.2010403@behnel.de> Hi, Dag Sverre Seljebotn wrote: > I'm combining many emails into one. Separating them again. :) > Stefan wrote: (mind the -f-) >>> def foo(a : unicode): >>> >>> would make sure int and str params are converted to unicode. >> No way. Converting a byte string to unicode is something that has to be >> done >> explicitly. How could you know the input encoding that is used for the >> byte sequence? > > It wasn't a proposed or wanted functionality, it was only a demonstration > of the fact that as a Python user, if > > def f(a: cython.types.int): > ... > > automatically construct a Cython int on entering, I kind of expect to be > able to do: > > def twice_factory(x): return cython.types.int(x) * 2 > > def f(a: twice_factory): > ... > > And have f automatically have its argument doubled No. Honestly. If you want such a functionality, write a decorator, but don't stuff the language with it. It's perfectly fine and a great feature that Cython can do C-to-Python-and-back-again type conversion on the fly. But allowing anything beyond that would be too much magic without any compelling use case. And I actually think that the syntax that we choose should not suggest stupid things. That's one reason why I dislike the assignment syntax. Your example > def twice_factory(x): return cython.types.int(x) * 2 is a *very* good example for something completely meaningless that this syntax suggests to users. Stefan From stefan_ml at behnel.de Tue Mar 11 10:30:39 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 10:30:39 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> Message-ID: <47D6513F.2070705@behnel.de> Hi, Dag Sverre Seljebotn wrote: >> Yeah, it could get ugly: cython.types.pointer(cython.types.pointer >> (cython.types.int))." > > I think "cython.types" is too longwinded anyway, in practice it would be: > > import cython.types as c wouldn't that have to be cimport cython.types as c ? Although that would break Python compatibility again, right? That's somewhat unfortunate... BTW, what would be provide to keep up Python compatibility anyway? Would there be a fake importable package like this just to make the code work in Python? You'd have to ship that with you Cython/Python code, then ... > c.ptr(c.ptr(c.int)) > > Which isn't *that* bad? And one could provide all kind of convenience > pre-declared names: > > c.pint = c.ptr(c.int) > c.ppint = ... > c.pchar = ... > c.ppchar = ... > > Perhaps just make a convention of providing up to three "ppp"-types for > every type, and then have a parametrized type > > c.ptrs(6, c.int) > > if you really want to do int******. > > PChar is already a well-known synonym for char* in some codebases. +1 here, although I'd call it "p_int" and "pp_int" instead of a good ol' "pint". :) And c.ptrs() is better written as "c.pointer(c.int, levels=6)" and maybe DEF-assigned to a name somewhere in the code. Stefan From robertwb at math.washington.edu Tue Mar 11 10:55:15 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Mar 2008 02:55:15 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D6513F.2070705@behnel.de> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> Message-ID: <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> On Mar 11, 2008, at 2:30 AM, Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: >>> Yeah, it could get ugly: cython.types.pointer(cython.types.pointer >>> (cython.types.int))." >> >> I think "cython.types" is too longwinded anyway, in practice it >> would be: >> >> import cython.types as c > > wouldn't that have to be > > cimport cython.types as c > > ? > > Although that would break Python compatibility again, right? > > That's somewhat unfortunate... > > BTW, what would be provide to keep up Python compatibility anyway? > Would there > be a fake importable package like this just to make the code work > in Python? > You'd have to ship that with you Cython/Python code, then ... Yep, there'd be a "fake" cython module that one could import. It would be cool if it could actually do ctypes to emulate some of the stuff before actually compiling. I don't really see any way of providing the rich set of names we need to make executable python code that can also be compiled. >> c.ptr(c.ptr(c.int)) >> >> Which isn't *that* bad? And one could provide all kind of convenience >> pre-declared names: >> >> c.pint = c.ptr(c.int) >> c.ppint = ... >> c.pchar = ... >> c.ppchar = ... >> >> Perhaps just make a convention of providing up to three "ppp"- >> types for >> every type, and then have a parametrized type >> >> c.ptrs(6, c.int) >> >> if you really want to do int******. >> >> PChar is already a well-known synonym for char* in some codebases. > > +1 here, although I'd call it "p_int" and "pp_int" instead of a > good ol' > "pint". :) > > And c.ptrs() is better written as "c.pointer(c.int, levels=6)" and > maybe > DEF-assigned to a name somewhere in the code. I see all this stuff as syntactic sugar that wouldn't usually get run, rather they would be compiler hints. - Robert From stefan_ml at behnel.de Tue Mar 11 11:24:40 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 11:24:40 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> Message-ID: <47D65DE8.6080708@behnel.de> Hi, Robert Bradshaw wrote: > On Mar 11, 2008, at 2:30 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> c.ptr(c.ptr(c.int)) >>> >>> Which isn't *that* bad? And one could provide all kind of convenience >>> pre-declared names: >>> >>> c.pint = c.ptr(c.int) >>> c.ppint = ... >>> c.pchar = ... >>> c.ppchar = ... >>> >>> Perhaps just make a convention of providing up to three "ppp"-types for >>> every type, and then have a parametrized type >>> >>> c.ptrs(6, c.int) >>> >>> if you really want to do int******. >>> >>> PChar is already a well-known synonym for char* in some codebases. >> >> +1 here, although I'd call it "p_int" and "pp_int" instead of a good ol' >> "pint". :) >> >> And c.ptrs() is better written as "c.pointer(c.int, levels=6)" and maybe >> DEF-assigned to a name somewhere in the code. > > I see all this stuff as syntactic sugar that wouldn't usually get run, > rather they would be compiler hints. Hmm, I think it should be more. I was considering them to be type references that Cython would look up in a separate type module at compile time. You could even integrate compile-time factories that would return Cython types when called by the compiler. The decorator notation gives you all sorts of possibilities to interact with Plugins already, so this would just be one step further. Stefan From robertwb at math.washington.edu Tue Mar 11 11:33:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Mar 2008 03:33:20 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D65DE8.6080708@behnel.de> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> <47D65DE8.6080708@behnel.de> Message-ID: On Mar 11, 2008, at 3:24 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> On Mar 11, 2008, at 2:30 AM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> c.ptr(c.ptr(c.int)) >>>> >>>> Which isn't *that* bad? And one could provide all kind of >>>> convenience >>>> pre-declared names: >>>> >>>> c.pint = c.ptr(c.int) >>>> c.ppint = ... >>>> c.pchar = ... >>>> c.ppchar = ... >>>> >>>> Perhaps just make a convention of providing up to three "ppp"- >>>> types for >>>> every type, and then have a parametrized type >>>> >>>> c.ptrs(6, c.int) >>>> >>>> if you really want to do int******. >>>> >>>> PChar is already a well-known synonym for char* in some codebases. >>> >>> +1 here, although I'd call it "p_int" and "pp_int" instead of a >>> good ol' >>> "pint". :) >>> >>> And c.ptrs() is better written as "c.pointer(c.int, levels=6)" >>> and maybe >>> DEF-assigned to a name somewhere in the code. >> >> I see all this stuff as syntactic sugar that wouldn't usually get >> run, >> rather they would be compiler hints. > > Hmm, I think it should be more. I was considering them to be type > references > that Cython would look up in a separate type module at compile > time. You could > even integrate compile-time factories that would return Cython > types when > called by the compiler. The decorator notation gives you all sorts of > possibilities to interact with Plugins already, so this would just > be one step > further. I agree that types wold be a good thing to look up that could be the same module in several cases. As for the decorator notation, in one case it would be passed a function object, and in another it would be passed a parse tree. One could write them to take either, but at the lowest level they still get turned into compiler hints (tree transformations to run being seen as one such hint). - Robert From stefan_ml at behnel.de Tue Mar 11 11:54:47 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 11:54:47 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> <47D65DE8.6080708@behnel.de> Message-ID: <47D664F7.9070500@behnel.de> Hi, Robert Bradshaw wrote: > As for the decorator notation, in one case > it would be passed a function object, and in another it would be passed > a parse tree. "case" meaning: a) when running in the Python interpreter and b) when running inside Cython? Or do you mean at different stages of the compile process? Or at compile time and run time of Cython code? I can't think of a case right now where you'd pass the function object. Stefan From robertwb at math.washington.edu Tue Mar 11 12:32:18 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Mar 2008 04:32:18 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D664F7.9070500@behnel.de> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> <47D65DE8.6080708@behnel.de> <47D664F7.9070500@behnel.de> Message-ID: <28925B8E-E860-4338-A32A-FCBA7C174F6F@math.washington.edu> On Mar 11, 2008, at 3:54 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> As for the decorator notation, in one case >> it would be passed a function object, and in another it would be >> passed >> a parse tree. > > "case" meaning: a) when running in the Python interpreter and b) > when running > inside Cython? Yes, this is what I mean. > Or do you mean at different stages of the compile process? Or > at compile time and run time of Cython code? > > I can't think of a case right now where you'd pass the function > object. > > Stefan From stefan_ml at behnel.de Tue Mar 11 13:06:06 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 11 Mar 2008 13:06:06 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <28925B8E-E860-4338-A32A-FCBA7C174F6F@math.washington.edu> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> <47D65DE8.6080708@behnel.de> <47D664F7.9070500@behnel.de> <28925B8E-E860-4338-A32A-FCBA7C174F6F@math.washington.edu> Message-ID: <47D675AE.8000201@behnel.de> Robert Bradshaw wrote: > On Mar 11, 2008, at 3:54 AM, Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> As for the decorator notation, in one case >>> it would be passed a function object, and in another it would be passed >>> a parse tree. >> >> "case" meaning: a) when running in the Python interpreter and b) when >> running >> inside Cython? > > Yes, this is what I mean. Hmmm, ok, but we already agreed that they'd use different code bases, mainly a fake implementation for the Python runtime case (at least for types). And I actually don't see much use for the functionality provided by Cython plugins inside the Python interpreter. Most decorator plugins would likely be replaced by def my_decorator(*args, **kwargs): def dummy(func): return func return dummy in their Python implementation, just to make decorated Cython code work under Python. Anyway, maybe there really are things that Cython plugins may provide one day, that similarly apply to Python code. I don't know... Stefan From dagss at student.matnat.uio.no Tue Mar 11 13:19:50 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 11 Mar 2008 13:19:50 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D675AE.8000201@behnel.de> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> <47D65DE8.6080708@behnel.de> <47D664F7.9070500@behnel.de> <28925B8E-E860-4338-A32A-FCBA7C174F6F@math.washington.edu> <47D675AE.8000201@behnel.de> Message-ID: <47D678E6.6020000@student.matnat.uio.no> I feel like reaching something of a summary or conclusion before withdrawing from the discussion ... I feel that what has happened now is more of a brainstorming session and that it may be possible to identify more fundamental issues about how the language should function. Here's my conclusions: - This is complicated enough that development using the current syntax should perhaps simply continue, and that it intuitively looks like (though perhaps it should be explored in more detail) that most features one would implement (special type support, operator overloading, type inference to a degree) can be developed within the current syntax and ported to future approaches with little problem. - Doing all the nice stuff (compile time macros, Python-valid syntax, look closer on variable declaration syntax) etc. should perhaps be done through identifying a few key principles in how Cython and Python should operate together and see where those principles lead us. (I now think that one such principle might be that Python syntax should work like a Python users expect it to in every way, and that other types of behaviour requires extension syntax - regarding typing variables for instance.) I think I personally need a week or so pause at this point to mature those thoughts, so I won't make more syntax comments for now. (But I'm not trying to stop you!, just letting you know.) Dag Sverre From robertwb at math.washington.edu Wed Mar 12 01:37:47 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Mar 2008 17:37:47 -0700 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D678E6.6020000@student.matnat.uio.no> References: <3288034995.3535600@smtp.netcom.no> <30A0BE51-A50A-4015-A1EB-E05234EF30F4@math.washington.edu> <65003.193.157.243.12.1205192938.squirrel@webmail.uio.no> <8A5E196A-13A2-469B-A53C-2D9D488D2A3C@math.washington.edu> <65277.193.157.243.12.1205225979.squirrel@webmail.uio.no> <47D6513F.2070705@behnel.de> <80B7AEB3-AFF5-489D-98E8-9C02CBFB24AD@math.washington.edu> <47D65DE8.6080708@behnel.de> <47D664F7.9070500@behnel.de> <28925B8E-E860-4338-A32A-FCBA7C174F6F@math.washington.edu> <47D675AE.8000201@behnel.de> <47D678E6.6020000@student.matnat.uio.no> Message-ID: On Mar 11, 2008, at 5:19 AM, Dag Sverre Seljebotn wrote: > I feel like reaching something of a summary or conclusion before > withdrawing from the discussion ... I feel that what has happened > now is > more of a brainstorming session and that it may be possible to > identify > more fundamental issues about how the language should function. Yep. Brainstorming was exactly what I was hoping to happen. > Here's my conclusions: > - This is complicated enough that development using the current syntax > should perhaps simply continue, and that it intuitively looks like > (though perhaps it should be explored in more detail) that most > features > one would implement (special type support, operator overloading, type > inference to a degree) can be developed within the current syntax and > ported to future approaches with little problem. I agree--the current syntax is very good at what it does and I don't see it going away any time soon, if ever. Being able to do (nearly) everything in a pure-python way will be nice, but I would imagine whatever we come up with some of it will be more cumbersome than the "cdef" way. With the exception of compile-time macros, they are simply clues to the compiler, and would be handled at the parser level to give identical trees. > - Doing all the nice stuff (compile time macros, Python-valid syntax, > look closer on variable declaration syntax) etc. should perhaps be > done > through identifying a few key principles in how Cython and Python > should > operate together and see where those principles lead us. (I now think > that one such principle might be that Python syntax should work like a > Python users expect it to in every way, and that other types of > behaviour requires extension syntax - regarding typing variables for > instance.) Yes, Python syntax has Python behavior has been a guiding principle for Cython (and Pyrex) since its inception. > I think I personally need a week or so pause at this point to mature > those thoughts, so I won't make more syntax comments for now. (But I'm > not trying to stop you!, just letting you know.) Me too, and I'm going to be away at a conference later this week too. Though we have covered several topics, one thing we haven't come up with even any ideas for is how to indicate types support operator overloading (e.g. in a pxd file). I'm curious to see what anyone can come up with for that, as I haven't had any ideas that I like yet. - Robert From robertwb at math.washington.edu Wed Mar 12 20:39:47 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 12 Mar 2008 12:39:47 -0700 Subject: [Cython] [Pyrex] switch statement In-Reply-To: <20080312143212.8bd28718.simon@arrowtheory.com> References: <20080312143212.8bd28718.simon@arrowtheory.com> Message-ID: <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> I did some experiments with this, and the results seemed to indicated it does not. Also, nested ifs performed poorly compared to a flat list of ifs (for a medium-sized (5-50) set of conditions). It would be nice to have some syntax for switch statements. Several alternatives have been rejected for Python: http://www.python.org/dev/ peps/pep-3103/ - Robert On Mar 12, 2008, at 11:32 AM, Simon Burton wrote: > Does anyone know if gcc can optimise a switch done with nested if's > as well as it optimises a real switch ? (eg. using computed-gotos) From dagss at student.matnat.uio.no Thu Mar 13 10:10:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 13 Mar 2008 10:10:44 +0100 Subject: [Cython] [Pyrex] switch statement In-Reply-To: <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> References: <20080312143212.8bd28718.simon@arrowtheory.com> <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> Message-ID: <47D8EF94.2000909@student.matnat.uio.no> Robert Bradshaw wrote: > I did some experiments with this, and the results seemed to indicated > it does not. Also, nested ifs performed poorly compared to a flat > list of ifs (for a medium-sized (5-50) set of conditions). > > It would be nice to have some syntax for switch statements. Several > alternatives have been rejected for Python: http://www.python.org/dev/ > peps/pep-3103/ > Wrote an idea for this at http://wiki.cython.org/enhancements/switch although it is probably in the category "too busy dreaming rather than getting real work done". Dag Sverre From stefan_ml at behnel.de Thu Mar 13 09:55:52 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 13 Mar 2008 09:55:52 +0100 Subject: [Cython] [Pyrex] switch statement In-Reply-To: <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> References: <20080312143212.8bd28718.simon@arrowtheory.com> <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> Message-ID: <47D8EC18.6080802@behnel.de> Hi, Robert Bradshaw top-posted: > On Mar 12, 2008, at 11:32 AM, Simon Burton wrote: >> Does anyone know if gcc can optimise a switch done with nested if's >> as well as it optimises a real switch ? (eg. using computed-gotos) > > I did some experiments with this, and the results seemed to indicate > it does not. Also, nested ifs performed poorly compared to a flat > list of ifs (for a medium-sized (5-50) set of conditions). Interesting. Does that also hold for boolean "and/or" expressions? Cython translates those to nested ifs. (I guess that's difficult to compare, though, as it short-circuits.) > It would be nice to have some syntax for switch statements. Several > alternatives have been rejected for Python: http://www.python.org/dev/ > peps/pep-3103/ That's because they have Python semantics, not C semantics. Python's language semantics make it a bit harder to come up with a meaningful "switch" design than a plain C language "check this integer value against a list of other integers" switch statement. Regarding syntax: my opinion on this is that we should just do without any new syntax, take our cool plugin optimiser infrastructure and just replace the most obvious if c_int_val == 1: code1() elif c_int_val == 2: code2() elif c_int_val == some_other_constant_c_int: code3() else: code4() by a straight C switch statement. No special casing if it involves variable values, Python values, or otherwise non compile-time known values (i.e. DEF is allowed), or any other comparison operators than "==". Just an optimisation for the most simple case that a C switch statement can handle: - if-elif-else chain with exactly one operator at each step: '==' -> decidable right after parsing (or even in the parser) - lhs always the same plain C integer variable -> "same name" decidable right after parsing, "type" after type annalysis - rhs a compile-time constant C integer value -> "type" decidable after type annalysis This should be easy and fast enough to decide. That way, people can decide what trade-off between Python semantics and optimisation they want. And if we find more cases that we can support, we can improve the plugin instead of the syntax. Stefan From robertwb at math.washington.edu Fri Mar 14 08:05:15 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 14 Mar 2008 00:05:15 -0700 Subject: [Cython] [Pyrex] switch statement In-Reply-To: <47D8EC18.6080802@behnel.de> References: <20080312143212.8bd28718.simon@arrowtheory.com> <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> <47D8EC18.6080802@behnel.de> Message-ID: On Mar 13, 2008, at 1:55 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw top-posted: >> On Mar 12, 2008, at 11:32 AM, Simon Burton wrote: >>> Does anyone know if gcc can optimise a switch done with nested if's >>> as well as it optimises a real switch ? (eg. using computed-gotos) >> >> I did some experiments with this, and the results seemed to indicate >> it does not. Also, nested ifs performed poorly compared to a flat >> list of ifs (for a medium-sized (5-50) set of conditions). > > Interesting. Does that also hold for boolean "and/or" expressions? > Cython > translates those to nested ifs. (I guess that's difficult to > compare, though, > as it short-circuits.) The thing about Cython's nested ifs is that (in Python) the cost of evaluating the truth statement is non-trivial (e.g. a function call at least), so short-circuits are nearly always a gain. Because the statements may have side effects, this is mandatory to enforce. >> It would be nice to have some syntax for switch statements. Several >> alternatives have been rejected for Python: http://www.python.org/ >> dev/ >> peps/pep-3103/ > > That's because they have Python semantics, not C semantics. > Python's language > semantics make it a bit harder to come up with a meaningful > "switch" design > than a plain C language "check this integer value against a list of > other > integers" switch statement. > > Regarding syntax: my opinion on this is that we should just do > without any new > syntax, take our cool plugin optimiser infrastructure and just > replace the > most obvious > > if c_int_val == 1: > code1() > elif c_int_val == 2: > code2() > elif c_int_val == some_other_constant_c_int: > code3() > else: > code4() > > by a straight C switch statement. No special casing if it involves > variable > values, Python values, or otherwise non compile-time known values > (i.e. DEF is > allowed), or any other comparison operators than "==". Just an > optimisation > for the most simple case that a C switch statement can handle: > > - if-elif-else chain with exactly one operator at each step: '==' > -> decidable right after parsing (or even in the parser) > - lhs always the same plain C integer variable > -> "same name" decidable right after parsing, "type" after type > annalysis > - rhs a compile-time constant C integer value > -> "type" decidable after type annalysis > > This should be easy and fast enough to decide. > > That way, people can decide what trade-off between Python semantics > and > optimisation they want. And if we find more cases that we can > support, we can > improve the plugin instead of the syntax. Dag Sverre wrote a similar writeup: http://wiki.cython.org/ enhancements/switch . I think it's a good idea. For small lists the savings will be minimal, but for larger ones there can be significant gains. - Robert From dagss at student.matnat.uio.no Fri Mar 14 09:47:32 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 14 Mar 2008 09:47:32 +0100 Subject: [Cython] Offtopic (was: Re: [Pyrex] switch statement) In-Reply-To: References: <20080312143212.8bd28718.simon@arrowtheory.com> <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> <47D8EC18.6080802@behnel.de> Message-ID: <47DA3BA4.5060805@student.matnat.uio.no> > Dag Sverre wrote a similar writeup: http://wiki.cython.org/ > enhancements/switch . I think it's a good idea. For small lists the > savings will be minimal, but for larger ones there can be significant > gains. > I often question switch statements though, I'm actually quite happy that Guido didn't make them part of Python. Often something like this is more appropriate: handlers = { 500: http_ok, 404: http_not_found } handlers[retcode](request, response) Of course, since we are talking performance of the GCC compiler here, I suppose it's one of the cases where performance for a O(1) operation do count, and then it's a different matter. All I'm saying is that most code I see using switch statements would often be better off without it. *runs and ducks* -- Dag Sverre From robertwb at math.washington.edu Fri Mar 14 10:02:01 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 14 Mar 2008 02:02:01 -0700 Subject: [Cython] Offtopic (was: Re: [Pyrex] switch statement) In-Reply-To: <47DA3BA4.5060805@student.matnat.uio.no> References: <20080312143212.8bd28718.simon@arrowtheory.com> <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> <47D8EC18.6080802@behnel.de> <47DA3BA4.5060805@student.matnat.uio.no> Message-ID: On Mar 14, 2008, at 1:47 AM, Dag Sverre Seljebotn wrote: >> Dag Sverre wrote a similar writeup: http://wiki.cython.org/ >> enhancements/switch . I think it's a good idea. For small lists >> the savings will be minimal, but for larger ones there can be >> significant gains. >> > I often question switch statements though, I'm actually quite happy > that Guido didn't make them part of Python. Often something like > this is more appropriate: > > handlers = { > 500: http_ok, > 404: http_not_found > } > handlers[retcode](request, response) Yes, this is certainly better form. Of course, if request/response were code blocks, one would have to do something a bit more clever (e.g. make a dictionary of functions and then call one). > Of course, since we are talking performance of the GCC compiler > here, I suppose it's one of the cases where performance for a O(1) > operation do count, and then it's a different matter. All I'm > saying is that most code I see using switch statements would often > be better off without it. :) Here's the latest use case that I wanted: http://hg.sagemath.org/sage-main/file/ff0875e07b4e/sage/ext/ fast_eval.pyx (starting at line 294--ignore that last case, this is because the function gets called tens of thousands of times in a row, and I didn't want an uncommon case to make everything slower). An if- elif translation would work great here. You can see that I used the dictionary paradigm in that same file as well. - Robert From dagss at student.matnat.uio.no Fri Mar 14 10:30:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 14 Mar 2008 10:30:12 +0100 Subject: [Cython] Offtopic In-Reply-To: References: <20080312143212.8bd28718.simon@arrowtheory.com> <835A03D8-8BAD-45C7-B1B0-5598DF7E5AE1@math.washington.edu> <47D8EC18.6080802@behnel.de> <47DA3BA4.5060805@student.matnat.uio.no> Message-ID: <47DA45A4.20107@student.matnat.uio.no> > > Here's the latest use case that I wanted: > http://hg.sagemath.org/sage-main/file/ff0875e07b4e/sage/ext/fast_eval.pyx > (starting at line 294--ignore that last case, this is because the > function gets called tens of thousands of times in a row, and I didn't > want an uncommon case to make everything slower). An if-elif > translation would work great here. Interesting. That's in a tight situation and a switch is right, sure. If I understand it correctly (it's an interpreter for a custom stack-based language?) then seeing it kind of makes me want to write a compiler compiling that stack language to native machine code instead, and then one could still use dictionaries as part of the compile phase and get rid of the switch :-) I'm sure there are better ways to spend our time though. (But it does make me think about perhaps have a real brainstorming some time about possible macro approaches for Cython, and then using macros could actually be a run-time feature of Cython on systems having a C compiler (and one could compile that operation stack using such run-time macros...after getting a ready parse-tree, just call "func = cython.compile(mytree)" and func would point into compiled code). One would not need to compile a new module per macro, a ctypes-approach would be fine for getting the code loaded after compilation.) This is a feature one would think seriously about in 2010 though, not now. What one can start to think about is making some slight changes in the coupling between Main.py and CmdLine.py (in effect reverse the control, have CmdLine.py drive Main.py and not the other way) so that it is possible to use Cython as a library rather than only a program. API and everything would be very unstable for now but just inverting that control flow could be an important mentality change. -- Dag Sverre From stefan_ml at behnel.de Fri Mar 14 10:43:12 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 14 Mar 2008 10:43:12 +0100 Subject: [Cython] distutils rebuild on updated dependencies Message-ID: <47DA48B0.3000909@behnel.de> Hi, based on a problem description by Arc Riley, here's a simple patch that adds dependency support to Cython's build_ext implementation (distutils). It allows you to simply add all .pxd and .pxi files that your .pyx depends on to the list of module sources, and have Cython rebuild your module when the dependencies are updated. It does, however, not deal with the case of multiple .pyx files with separate dependencies that build a single module. But that could be added through a convention in the source order. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: distutils-dependencies.patch Type: text/x-patch Size: 1880 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080314/b042c7bd/attachment.bin From robertwb at math.washington.edu Fri Mar 14 10:48:45 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 14 Mar 2008 02:48:45 -0700 Subject: [Cython] distutils rebuild on updated dependencies In-Reply-To: <47DA48B0.3000909@behnel.de> References: <47DA48B0.3000909@behnel.de> Message-ID: Is this just by name? I don't see in here how B.pyx that references A.pxd and C.pxi gets its dependancies updated. - Robert On Mar 14, 2008, at 2:43 AM, Stefan Behnel wrote: > Hi, > > based on a problem description by Arc Riley, here's a simple patch > that adds > dependency support to Cython's build_ext implementation (distutils). > > It allows you to simply add all .pxd and .pxi files that your .pyx > depends on > to the list of module sources, and have Cython rebuild your module > when the > dependencies are updated. > > It does, however, not deal with the case of multiple .pyx files > with separate > dependencies that build a single module. But that could be added > through a > convention in the source order. > > Stefan > # HG changeset patch > # User Stefan Behnel > # Date 1205487293 -3600 > # Node ID 1abce3b0dd95220bb0b9b6a86dbe891c5923cc9c > # Parent d49c041d08535698109742a2cc4fe9e78eede234 > support adding pxd/pxi dependencies to Extension sources and force > rebuild on dependency updates > > diff -r d49c041d0853 -r 1abce3b0dd95 Distutils/build_ext.py > --- a/Distutils/build_ext.py Sun Mar 09 00:40:34 2008 -0800 > +++ b/Distutils/build_ext.py Fri Mar 14 10:34:53 2008 +0100 > @@ -155,6 +155,7 @@ class build_ext(_build_ext.build_ext): > else: > target_dir = None > > + newest_dependency = None > for source in sources: > (base, ext) = os.path.splitext(os.path.basename(source)) > if ext == ".pyx": # Cython source file > @@ -162,6 +163,10 @@ class build_ext(_build_ext.build_ext): > new_sources.append(os.path.join(output_dir, base + > target_ext)) > pyrex_sources.append(source) > pyrex_targets[source] = new_sources[-1] > + elif ext == '.pxi' or ext == '.pxd': > + if newest_dependency is None \ > + or newer(source, newest_dependency): > + newest_dependency = source > else: > new_sources.append(source) > > @@ -172,7 +177,10 @@ class build_ext(_build_ext.build_ext): > > for source in pyrex_sources: > target = pyrex_targets[source] > - if self.force or newer(source, target): > + rebuild = self.force or newer(source, target) > + if not rebuild and newest_dependency is not None: > + rebuild = newer(newest_dependency, target) > + if rebuild: > log.info("cythoning %s to %s", source, target) > self.mkpath(os.path.dirname(target)) > options = CompilationOptions(pyrex_default_options, > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Fri Mar 14 10:57:26 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 14 Mar 2008 10:57:26 +0100 Subject: [Cython] distutils rebuild on updated dependencies In-Reply-To: References: <47DA48B0.3000909@behnel.de> Message-ID: <47DA4C06.4090905@behnel.de> Hi, Robert Bradshaw wrote: > Is this just by name? I don't see in here how B.pyx that references > A.pxd and C.pxi gets its dependancies updated. Read my post. :) > On Mar 14, 2008, at 2:43 AM, Stefan Behnel wrote: >> based on a problem description by Arc Riley, here's a simple patch >> that adds >> dependency support to Cython's build_ext implementation (distutils). >> >> It allows you to simply add all .pxd and .pxi files that your .pyx >> depends on >> to the list of module sources, and have Cython rebuild your module >> when the dependencies are updated. This means, you can write distutils.core.Extension( "my.module", sources = [ "my.module.pyx", "other.pxi", "stuff.pxi" ], ...) So this is explicit in your setup.py, not implicit. You can obviously improve on this by actually *tracking* dependencies (which my patch does *not* do). Stefan From stefan_ml at behnel.de Fri Mar 14 17:06:26 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 14 Mar 2008 17:06:26 +0100 Subject: [Cython] [Pyrex] Cython development (was: array setting and math function annoyance) In-Reply-To: References: <8B993AA4-2CA8-487A-8238-D1ADCA93E674@math.washington.edu> <07CD8AA9-35E7-45E4-955F-C65DD9B9EF66@math.washington.edu> <47DA20F6.2000808@behnel.de> <47DA3BEE.6060604@behnel.de> Message-ID: <47DAA282.5020302@behnel.de> Hi, Arc Riley wrote: >> It's always better to improve things than to branch for trivial changes > > I agree. It's troubling to see this proliferation of project-specific Pyrex > branches each with only minor changes. > > In contrast, however, we've been at the mercy of Greg to decide when to > accept patches, his available time to rewrite them, then later release, and > finally for a new release to be packaged by distros. I understand this is a > frustration you guys had too, behind your reason to form Cython, but if we > used Cython we would be in the same situation with you than we are with > Pyrex. I see the situation a bit different. I think we have learned from the problems with Pyrex, which were the reason why so many people started their own Pyrex branch somewhere. Cython is a very open project, and we really try to attract people who want to contribute, instead of creating new branches for no good reason. Cython has a much (much!) faster release cycle than Pyrex, and it has an open development repository (Pyrex also has one by now, BTW). So you can use developer versions with the fixes/features you need even before a new release comes out. http://hg.cython.org/ > We're not interested in the language pre-processor politics, we've stayed > out of them so far, but given all our logistics we really need greater > control over that part of our own project. No idea what you mean here. If you are referring to the recent discussions on the mailing list, then I would say that patches will be reviewed and good ideas will eventually be implemented depending on our resources, but big changes take time, so don't "fear" them by tomorrow. In any case, we do care about not breaking existing code without very good reason, so I would say you're on the safe side there. >> You can't just ignore the most obvious way to build a C extension for Python >> (i.e. using distutils), and then come and complain that your hand-written >> setup doesn't work. > > Oh, we're using Python's distutils to build the C source. > > That 100 lines in our setup.py is a workaround for Pyrex.Distutils failure > of detecting source changes and rebuilding .c files only when necessary. > That part would be better generalized and in Distutils rather than in our > setup.py since any project could take advantage of it I wrote a patch so that you can do that with Cython. You will just have to care for the dependency tracking yourself, and provide your distutils Extension with the complete list of sources (including .pxi and .pxd files). Cython (or rather distutils) will then build only those extensions where any of the source files in that list has changed. >> improve Cython's distutils helper (build_ext.py) to handle the problem of >> calculating dependencies better. > > That's what we're looking at, yes. > > I think the most pragmatic solution for us right now is a branch, given our > release time table and Cython's level of recent commit activity is that good or bad? > and that these changes will undoubtedly require discussion. > We'll of course provide patches. Is there anything in addition to my patch that you need here? Stefan From stefan_ml at behnel.de Fri Mar 14 18:51:01 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 14 Mar 2008 18:51:01 +0100 Subject: [Cython] [Pyrex] Cython development In-Reply-To: References: <8B993AA4-2CA8-487A-8238-D1ADCA93E674@math.washington.edu> <07CD8AA9-35E7-45E4-955F-C65DD9B9EF66@math.washington.edu> <47DA20F6.2000808@behnel.de> <47DA3BEE.6060604@behnel.de> <47DAA282.5020302@behnel.de> Message-ID: <47DABB05.9050504@behnel.de> Hi, Arc Riley wrote: > Again, I don't want this to come across negative, but we're limited to > what's been *released* and in many cases *packaged* for distros because it's > a dependency to build from source. It was a pain for people when we > upgraded to Pyrex 0.9.6.x since it broke compatibility with Pyrex > 0.9.5.1awith all the __cinit__'s May I ask one question: why are your users impacted by Cython/Pyrex? Don't you ship with pre-generated .c files? (and how do you do quality assurance in your users use their own version of Pyrex?) Stefan From stefan_ml at behnel.de Fri Mar 14 19:02:20 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 14 Mar 2008 19:02:20 +0100 Subject: [Cython] [Pyrex] Cython development In-Reply-To: References: <8B993AA4-2CA8-487A-8238-D1ADCA93E674@math.washington.edu> <07CD8AA9-35E7-45E4-955F-C65DD9B9EF66@math.washington.edu> <47DA20F6.2000808@behnel.de> <47DA3BEE.6060604@behnel.de> <47DAA282.5020302@behnel.de> Message-ID: <47DABDAC.2020508@behnel.de> Hi, Arc Riley wrote: >> I wrote a patch so that you can do that with Cython. You will just have to >> care for the dependency tracking yourself, and provide your distutils >> Extension with the complete list of sources (including .pxi and .pxd >> files). >> Cython (or rather distutils) will then build only those extensions where >> any of the source files in that list has changed. > > Thank you for this, but note that it's minor compared to the various > functions which search for cimports and the like. We've worked around what > it builds a package as (I hope) using the full_module_name argument to > compile but this is only the beginning. Fair enough. > Setting the package directory '':'src' would, as I understand it, cause > Cython to compile our extensions not as (ie) src.colors.soy.colors but as > colors.soy.colors; closer, but still a no-go. Ah, and I noticed that you do not seem to provide your fully qualified package names as first parameter to Extension(). I guess, when I dig deeper, I would find even more things like this. You really can't blame Cython if you provide it with incomplete information all over the place. > Honestly I think a distutils.extension.Extension parameter for psources= > (etc), treating the Pyrex/Cython/etc sources in a style akin to compiling C > sources, compiling each source to a separate .c file, and appending those > generated .c files to sources= is a better direction to go That would just unnecessarily complicate things IMHO. There's one parameter for any kind of sources now, that should be enough. >> Is there anything in addition to my patch that you need here? > > Immediately, the most direct and painless solution would be to handle > cimport name searching by only it's file name, rather than it's directory > path, to match Pyrex's behavior. ... or, you could provide a directory structure that actually matches your package structure, rather than trying to push Cython into working on inconsistent package directories (lacking the main package, for example), by providing an incomplete view on the package structure. Fixing your package_dir parameter and your Extension() name should get you pretty far, I assume. Stefan From mmartin at itasoftware.com Sun Mar 9 02:00:59 2008 From: mmartin at itasoftware.com (Martin C. Martin) Date: Sat, 08 Mar 2008 20:00:59 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BEB2.2080106@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> Message-ID: <47D336CB.9000306@itasoftware.com> Dag Sverre Seljebotn wrote: > Martin C. Martin wrote: >> Ok, thanks a lot. >> >> My day job is programming Lisp, and since I started two years ago, >> I've seen the light about a number of things, including macros. It >> would be cool to have access to the Abstract Syntax Tree of a Cython >> expression/statement/function/etc., and be able to manipulate it at >> compile time. But it sounds like that's a whole other project. :) > Hmm. I'm still not sure whether I've come across right :-) > > Accessing the abstract syntax tree is exactly what this is about, and > you can do that (however the syntax for doing would seem rather complex > and messy compared to with LISP, though a lot better I believe than what > has been going on until now). > > The problem is that the code that accesses the syntax tree has to be > written in Python and inserted into the compiler at launch time -- you > cannot analyse yourself. So, how hard would it be to provide a mechanism to do this in the same file? There's a large subset of use cases for Lisp's macros which need the &body parameter. To do the equivalent in Python/Cython, you'd need a way to specify a block of code, much like a closure that doesn't take any parameters. Interesting... Best, Martin From arcriley at gmail.com Fri Mar 14 18:10:46 2008 From: arcriley at gmail.com (Arc Riley) Date: Fri, 14 Mar 2008 13:10:46 -0400 Subject: [Cython] [Pyrex] Cython development (was: array setting and math function annoyance) In-Reply-To: <47DAA282.5020302@behnel.de> References: <8B993AA4-2CA8-487A-8238-D1ADCA93E674@math.washington.edu> <07CD8AA9-35E7-45E4-955F-C65DD9B9EF66@math.washington.edu> <47DA20F6.2000808@behnel.de> <47DA3BEE.6060604@behnel.de> <47DAA282.5020302@behnel.de> Message-ID: > Cython is a very open project, and we really try to attract people who > want to > contribute, instead of creating new branches for no good reason. The text at the top of the site should be changed from " Development of Cython is mainly motivated by the needs of SAGE" in that case :-) Cython has a much (much!) faster release cycle than Pyrex, and it has an > open > development repository (Pyrex also has one by now, BTW). Last I checked, Pyrex has a tarball of a mercurial image. This does not equate to an open development repository, IMHO. > So you can use developer versions with the fixes/features you need even > before a new release > comes out. Again, I don't want this to come across negative, but we're limited to what's been *released* and in many cases *packaged* for distros because it's a dependency to build from source. It was a pain for people when we upgraded to Pyrex 0.9.6.x since it broke compatibility with Pyrex 0.9.5.1awith all the __cinit__'s I wrote a patch so that you can do that with Cython. You will just have to > care for the dependency tracking yourself, and provide your distutils > Extension with the complete list of sources (including .pxi and .pxd > files). > Cython (or rather distutils) will then build only those extensions where > any > of the source files in that list has changed. Thank you for this, but note that it's minor compared to the various functions which search for cimports and the like. We've worked around what it builds a package as (I hope) using the full_module_name argument to compile but this is only the beginning. Setting the package directory '':'src' would, as I understand it, cause Cython to compile our extensions not as (ie) src.colors.soy.colors but as colors.soy.colors; closer, but still a no-go. Honestly I think a distutils.extension.Extension parameter for psources= (etc), treating the Pyrex/Cython/etc sources in a style akin to compiling C sources, compiling each source to a separate .c file, and appending those generated .c files to sources= is a better direction to go Is there anything in addition to my patch that you need here? Immediately, the most direct and painless solution would be to handle cimport name searching by only it's file name, rather than it's directory path, to match Pyrex's behavior. I'm a huge fan of minimizing hacks, so ideally I'd like a more direct behavior akin to the above suggestion. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080314/5c17278d/attachment.htm From arcriley at gmail.com Fri Mar 14 19:19:09 2008 From: arcriley at gmail.com (Arc Riley) Date: Fri, 14 Mar 2008 14:19:09 -0400 Subject: [Cython] [Pyrex] Cython development In-Reply-To: <47DABB05.9050504@behnel.de> References: <07CD8AA9-35E7-45E4-955F-C65DD9B9EF66@math.washington.edu> <47DA20F6.2000808@behnel.de> <47DA3BEE.6060604@behnel.de> <47DAA282.5020302@behnel.de> <47DABB05.9050504@behnel.de> Message-ID: > May I ask one question: why are your users impacted by Cython/Pyrex? Don't > you > ship with pre-generated .c files? (and how do you do quality assurance in > your > users use their own version of Pyrex?) Up until the last release we were distributing pre-processed .c files. We've received at least a dozen complaints (which means a t least a few dozen more) about this, that the .c source in our tarballs are unparseable by any sane human, that we haven't actually complied with the GPL's requirements in the "source" release since the .c sources are not what is intended to be modified, and of course Windows users needing a separate source zip since we have OS-specific building going on the Pyrex level. Since the .c sources are OS-specific and add a great deal to the overall package size, we've reached a rough consensus that starting with our next release our source tarball will be pre-processed only; basically just Trunk snapshotted to a versioned tag and a few version strings set in it. Even beside this, we currently have 11 active developers, 20 total contributors according to ohloh. According to my logs, 197 anonymous full checkouts (based on access to a file that hasn't changed) have been made by non-bots since our last release, compared to 1449 downloads of our last release (12%), and we know at least a dozen of those are regulars. I include these stats to show that pre-generating the *.c is an inadequate solution. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080314/b1136cff/attachment.htm From stefan_ml at behnel.de Fri Mar 14 19:48:39 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 14 Mar 2008 19:48:39 +0100 Subject: [Cython] [Pyrex] Cython development In-Reply-To: References: <07CD8AA9-35E7-45E4-955F-C65DD9B9EF66@math.washington.edu> <47DA20F6.2000808@behnel.de> <47DA3BEE.6060604@behnel.de> <47DAA282.5020302@behnel.de> <47DABB05.9050504@behnel.de> Message-ID: <47DAC887.8020806@behnel.de> Hi, Arc Riley wrote: > Up until the last release we were distributing pre-processed .c files. > > We've received at least a dozen complaints (which means a t least a few > dozen more) about this, that the .c source in our tarballs are unparseable > by any sane human Then put a clear warning into the docs/FAQ: "the C files are generated, please ignore them when reading the source". > that we haven't actually complied with the GPL's > requirements in the "source" release since the .c sources are not what is > intended to be modified Distributing both the .c files *and* the Pyrex source would help here. > and of course Windows users needing a separate > source zip since we have OS-specific building going on the Pyrex level. Interesting. lxml supports various different versions of libxml2/libxslt, each with their own bugs and quirks etc. I guess that's comparable to platform specific code. However, we can distribute our generated .c files together with the normal sources, as all version specific stuff is handled in a separate header file. There, we #define a couple of names that we declare as ints in a .pxd file. Our Cython code is then based on those names, for example: cimport config # this defines the names from the header file ... if config.HAS_WHATEVER: dostuff() else: dootherstuff() The header would be something like #ifdef STUFF #define HAS_WHATEVER 1 #else #define HAS_WHATEVER 0 #endif And the .pxd goes: cdef extern from "config.h": cdef int HAS_WHATEVER The C compiler then takes whatever branch is selected by the #defines in the header. That way, we avoid any platform specific differences in the generated C code. Obviously, you may or may not be able to do something similar. Stefan From dagss at student.matnat.uio.no Sat Mar 15 10:25:51 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 10:25:51 +0100 (CET) Subject: [Cython] Draft for compile-time calculation inliner Message-ID: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> http://wiki.cython.org/enhancements/inlining As I think this is possible in a one or two week timeframe, I ask of you all to tell me what obvious things I am missing completely and where I did a wrong turn in my reasoning :-) I copy and paste: Why? - It leaves the way open for a hassle-free operator overloading syntax - use the same operator overloading conventions that Python use even for wrapped C library types without worrying about performance. - A convenient tool for doing compile-time calculations and having parametrized code written at compile-time. The creative use C++'s template syntax has gotten over the years should demonstrate that the inlining the GCC compiler does is not sufficient for advanced use cases (see http://ubiety.uwaterloo.ca/~tveldhui/papers/Expression-Templates/exprtmpl.html, Blitz++ etc., basically using the C++ template system to inline vector multiplications etc. that's written dynamically according to vector sizes). Dag Sverre From gfurnish at indirectproof.net Sat Mar 15 10:52:42 2008 From: gfurnish at indirectproof.net (Gary Furnish) Date: Sat, 15 Mar 2008 03:52:42 -0600 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> Message-ID: <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> While I've liked your previous ideas, parse trees are not the correct way to do optimizations in my opinion. This would dramatically increase Cython compile time by orders of magnitude for one. If we really wanted to do this we should build dependency graphs (I'm currently working on code that does this for classes) for all functions and then use some more standard algorithms. (It is easy to optimize your example with graphs, it is significantly more complicated using the parse tree). Doing this properly is no small undertaking as what you are essentially asking for is a full optimizing compiler (without register allocation, granted). Furthermore we should probably only attempt this after cython can bootstrap itself, as its going to be tough to keep compile times reasonable while cython is still written in python. While this task is simple to describe, there are thousands of devils in the details. I'm hesitant to call this a good idea without someone experienced in writing compilers willing to work on it close to full time. On 3/15/08, Dag Sverre Seljebotn wrote: > http://wiki.cython.org/enhancements/inlining > > As I think this is possible in a one or two week timeframe, I ask of you > all to tell me what obvious things I am missing completely and where I did > a wrong turn in my reasoning :-) > > I copy and paste: > > Why? > > - It leaves the way open for a hassle-free operator overloading syntax - > use the same operator overloading conventions that Python use even for > wrapped C library types without worrying about performance. > > - A convenient tool for doing compile-time calculations and having > parametrized code written at compile-time. The creative use C++'s template > syntax has gotten over the years should demonstrate that the inlining the > GCC compiler does is not sufficient for advanced use cases (see > http://ubiety.uwaterloo.ca/~tveldhui/papers/Expression-Templates/exprtmpl.html, > Blitz++ etc., basically using the C++ template system to inline vector > multiplications etc. that's written dynamically according to vector > sizes). > > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Sat Mar 15 11:34:33 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 15 Mar 2008 11:34:33 +0100 Subject: [Cython] When to run tree transformations Message-ID: <47DBA639.2070109@behnel.de> Hi, as I suggested before, I think it would be helpful if plugins could tell the compiler directly when they were to run (without the current -T option). I think there would at least be an analysis step in addition to the existing tree processing step for the plugin lifecycle, but the plugin might also need to run at different times of the compiler lifecycle, and treat different syntax structures. I see the following lifecycle hooks here: - pre analyse types - post analyse types - pre generate code - post generate code I would suggest mapping these to method names in the plugins. You could then write class MyTransform(Transform): def pre_analyse_types(self, node, name): ... instead of the current "process_node()" method. I could also imagine keeping a set of node types in the plugin that would be used for selecting appropriate transformations for the current node. In the transformation example, there is this line if isinstance(node, Nodes.ForInStatNode) ... that determines when to run the plugin. If the plugin provided a set of interesting nodes instead, the infrastructure could store them in a dict (mapping node types to a list of plugins) and directly select the transformations that can run on that node. Question in that case: how to deal with subtypes? Would they be expected to match? Or should the plugin just specify all node types, including their expected subtypes? Stefan From dagss at student.matnat.uio.no Sat Mar 15 11:49:33 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 11:49:33 +0100 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> Message-ID: <47DBA9BD.8030600@student.matnat.uio.no> Thanks, this is exactly the response I was hoping for. It would have been my own - my own reasoning about this idea has been: - I found an approach to do this easily - This should be hard - => I'm wrong...somewhere... and in the end I put it out for public ridicule instead of thinking more about which one of my assumptions are wrong. > While I've liked your previous ideas, parse trees are not the correct > way to do optimizations in my opinion. This would dramatically > increase Cython compile time by orders of magnitude for one. If we > I don't expect a full optimizing compiler. Note specifically that it will do *nothing* about optimizing run-time expressions: a - b + b will be left as-is without any change, UNLESS they are all known compile-time (and some code for evaluating compile-time expressions and marking nodes as known compile-time are already in place thanks to the DEF syntax). If you think your comment still apply then just say a simple "yes it does" and I'll think harder about it. An optimizing compiler tries to optimize things like the above, which is not what this is about. Compilation times depends on what you use it on. I think it would be a manual opt-in for starters, possibly for ever, and whoever opts in (with @cython.inline for instance) have to eat their own cake! The idea is that with this one can have a convenient run-time syntax for operator overloading, while without it the syntax for operator overloading I'm thinking about is a no-go (specifically talking about how to generate a dynamic operator overload for NumPy arrays here). For starters I would plan to use it only for implementing NumPy array access in a nice way, effectively collapsing arr[i,j] through this overload: @cython.inline def __getitem__(self, index): cdef int bufidx = 0 if len(index) != type(self).nd: raise ValueError("...") if not isinstance(index, list): if typeof(self).checkbounds: if index < 0: raise ValueError("...") ...etc... bufidx = index else: for subindex, dim in zip(index, range(len(index)): if len(subindex) > 1: raise ValueError("slices not supported") if typeof(self).checkbounds: if subindex < 0: raise ValueError("...") ...and so on... bufidx += self.strides[dim] // sizeof(typeof(self).dtype) * subindex return (self.data)[bufidx] into (arr.data)[i * arr.strides[0]//4 + j * arr.strides[1]//4] for 2-D array lookup. This adds O(nm) to the compilation time where n is the number of NumPy array lookups and m is the number of dimensions of the array being looked up. Adding memoization to this is where the smart dependency graphs needs to enter I guess, and that's a -- Dag Sverre From dagss at student.matnat.uio.no Sat Mar 15 12:06:38 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 12:06:38 +0100 Subject: [Cython] When to run tree transformations In-Reply-To: <47DBA639.2070109@behnel.de> References: <47DBA639.2070109@behnel.de> Message-ID: <47DBADBE.5030709@student.matnat.uio.no> > I would suggest mapping these to method names in the plugins. You could then write > > class MyTransform(Transform): > def pre_analyse_types(self, node, name): > ... > > instead of the current "process_node()" method. > I respectfully disagree strongly :-) From a simple design standpoint, mixing information about which phase one is in and the simple contract that a Transform must obey to doesn't make sense. Also, it kinds of breaks IoC: http://en.wikipedia.org/wiki/Inversion_of_Control Apart from a puristic perspective, it has practical consequenceses: In time a lot of transform will run in the same phase, especially between the type deduction and type coercion phase, and this binds down written transforms so that it is difficult to later divide into sub-phases and so on to get them all run in the right order. I always assumed one would use a standard plugin pattern like this: MyPlugin.py: import MyTransform def register_cython_module(controller): controller.add_transform_to_phase("myphase", MyTransform.MyTransform) controller.add_commandline_argument_i_need(....) Basically, Cython on startup loads a list of modules, and if they contain a register_cython_module call it is called with something wrapping the Cython compiler options object as parameter. > > that determines when to run the plugin. If the plugin provided a set of > interesting nodes instead, the infrastructure could store them in a dict > (mapping node types to a list of plugins) and directly select the > transformations that can run on that node. Again I think this is wrong design. The functionality you want should definitely be provided, however it would simply be either a base class or a wrapper class (leaning towards the former) for a transform: class MyTransform(NodeFilterTransform): def __init__(self): NodeFilterTransform.__init__(self, [ForStatNode, ForListNode,]) However, again, I think this will probably be more generic, doing something like import cython.dom as d class MyTransform(TemplateMatchTransform): match = d.if(cond=d.bool, body=d.any, else=d.any) def process_if(self, node, name): ... process_if = register_template(match, process_if) (Or even match = "some-xpath-inspired-thing"). Basically, both approaches (+ another one using XPath, and another one and so on) can be used, live in seperate base classes, and transforms pick the most convenient to extend. Building type filtering directly into the core however locks you to The One Right approach, while different transforms will have different needs. I cannot see any performance benefits either. -- Dag Sverre From dagss at student.matnat.uio.no Sat Mar 15 12:16:21 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 12:16:21 +0100 Subject: [Cython] When to run tree transformations In-Reply-To: <47DBADBE.5030709@student.matnat.uio.no> References: <47DBA639.2070109@behnel.de> <47DBADBE.5030709@student.matnat.uio.no> Message-ID: <47DBB005.7080406@student.matnat.uio.no> An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080315/ecd91351/attachment.htm From dagss at student.matnat.uio.no Sat Mar 15 12:20:15 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 12:20:15 +0100 Subject: [Cython] When to run tree transformations In-Reply-To: <47DBADBE.5030709@student.matnat.uio.no> References: <47DBA639.2070109@behnel.de> <47DBADBE.5030709@student.matnat.uio.no> Message-ID: <47DBB0EF.9080509@student.matnat.uio.no> > Building type filtering directly into the core however locks you to The > One Right approach, while different transforms will have different > needs. I cannot see any performance benefits either. > Also, how would you build type filtering into the core at all, if not by using transforms to implement it? Certainly not by hard-wiring it into the current code-processing-in-methods-in-nodes? As I said earlier, I'm a strong believer of Transform.py simply being about creating a more modular and maintainble compiler core, not about building a plugin framework. Transforms and plugins are two conceptually different things, the former being a necessity for the latter. I'll calm down now :-) Thanks for raising the point, I'm curious to hear your thoughts... -- Dag Sverre From stefan_ml at behnel.de Sat Mar 15 17:24:52 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 15 Mar 2008 17:24:52 +0100 Subject: [Cython] When to run tree transformations In-Reply-To: <47DBB005.7080406@student.matnat.uio.no> References: <47DBA639.2070109@behnel.de> <47DBADBE.5030709@student.matnat.uio.no> <47DBB005.7080406@student.matnat.uio.no> Message-ID: <47DBF854.4040307@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Basically, write encapsulated, stand-alone code that can be unit-tested easily, > and don't assume that the compiler works this or that way (ie, the transform > shouldn't have knowledge about various phase > > What it does need knowledge about is what kind of contract/state the tree is in: > "I expect there to be type information there"), which is something different > from "I know there will be type information there because we are in state B or C > or D but not A". The former is a contract, the latter makes the transform assume > something about the compiler operation. I generally agree with the IoC principle, however, the current way of using cmd line options is pretty much out of question. So, yes, plugins should register themselves, and they should register for a specific state of the tree. I expect that to mean: before or after type analysis (i.e. the plugin either depends on types being analysed, does not depend on types being known or does its own stuff to analyse types), or before or after generating code for the node (i.e. the plugin has to add code to the output). I think that state should be reflected by the syntax tree, so it should be possible to avoid having plugins that have to do stuff before *and* after phases (although that might still lead to a cleaner design - not sure). If you want to distinguish between a contract and a compiler phase here, fine. I think the type analysis case makes for some fine contract, and the code generation just isn't a contract, it's a phase. You'll likely need both. Stefan From robertwb at math.washington.edu Sat Mar 15 17:33:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 15 Mar 2008 09:33:22 -0700 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> Message-ID: <680EB43A-21E6-400E-934C-154F552EAC89@math.washington.edu> I disagree, I think parse trees are a great place to put many optimizations--for instance GCC performs many of its optimizations at that level. - Robert On Mar 15, 2008, at 2:52 AM, Gary Furnish wrote: > While I've liked your previous ideas, parse trees are not the correct > way to do optimizations in my opinion. This would dramatically > increase Cython compile time by orders of magnitude for one. If we > really wanted to do this we should build dependency graphs (I'm > currently working on code that does this for classes) for all > functions and then use some more standard algorithms. (It is easy to > optimize your example with graphs, it is significantly more > complicated using the parse tree). Doing this properly is no small > undertaking as what you are essentially asking for is a full > optimizing compiler (without register allocation, granted). > Furthermore we should probably only attempt this after cython can > bootstrap itself, as its going to be tough to keep compile times > reasonable while cython is still written in python. While this task > is simple to describe, there are thousands of devils in the details. > I'm hesitant to call this a good idea without someone experienced in > writing compilers willing to work on it close to full time. > > On 3/15/08, Dag Sverre Seljebotn wrote: >> http://wiki.cython.org/enhancements/inlining >> >> As I think this is possible in a one or two week timeframe, I ask >> of you >> all to tell me what obvious things I am missing completely and >> where I did >> a wrong turn in my reasoning :-) >> >> I copy and paste: >> >> Why? >> >> - It leaves the way open for a hassle-free operator overloading >> syntax - >> use the same operator overloading conventions that Python use >> even for >> wrapped C library types without worrying about performance. >> >> - A convenient tool for doing compile-time calculations and having >> parametrized code written at compile-time. The creative use C++'s >> template >> syntax has gotten over the years should demonstrate that the >> inlining the >> GCC compiler does is not sufficient for advanced use cases (see >> http://ubiety.uwaterloo.ca/~tveldhui/papers/Expression-Templates/ >> exprtmpl.html, >> Blitz++ etc., basically using the C++ template system to inline >> vector >> multiplications etc. that's written dynamically according to vector >> sizes). >> >> Dag Sverre >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From gfurnish at indirectproof.net Sat Mar 15 19:18:09 2008 From: gfurnish at indirectproof.net (Gary Furnish) Date: Sat, 15 Mar 2008 12:18:09 -0600 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <680EB43A-21E6-400E-934C-154F552EAC89@math.washington.edu> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> <680EB43A-21E6-400E-934C-154F552EAC89@math.washington.edu> Message-ID: <706850310803151118q44cb05fcq44f4fdf3f89eb9de@mail.gmail.com> I'll agree that some simple optimizations may be appropriate at the parse tree level (for loop range replacement, which we already do, is a great example). However, none of the examples on the wiki are things I would consider good things to do at the parse tree level. Having a full python interperter around to get around the fact that we don't have decent dependencies is almost certainly not the way to do this. My dislike of parse tree optimizations mainly centers around using it for very complicated things that can be much better served by other algorithms. On 3/15/08, Robert Bradshaw wrote: > I disagree, I think parse trees are a great place to put many > optimizations--for instance GCC performs many of its optimizations at > that level. > > > - Robert > > > On Mar 15, 2008, at 2:52 AM, Gary Furnish wrote: > > > While I've liked your previous ideas, parse trees are not the correct > > way to do optimizations in my opinion. This would dramatically > > increase Cython compile time by orders of magnitude for one. If we > > really wanted to do this we should build dependency graphs (I'm > > currently working on code that does this for classes) for all > > functions and then use some more standard algorithms. (It is easy to > > optimize your example with graphs, it is significantly more > > complicated using the parse tree). Doing this properly is no small > > undertaking as what you are essentially asking for is a full > > optimizing compiler (without register allocation, granted). > > Furthermore we should probably only attempt this after cython can > > bootstrap itself, as its going to be tough to keep compile times > > reasonable while cython is still written in python. While this task > > is simple to describe, there are thousands of devils in the details. > > I'm hesitant to call this a good idea without someone experienced in > > writing compilers willing to work on it close to full time. > > > > On 3/15/08, Dag Sverre Seljebotn wrote: > >> http://wiki.cython.org/enhancements/inlining > >> > >> As I think this is possible in a one or two week timeframe, I ask > >> of you > >> all to tell me what obvious things I am missing completely and > >> where I did > >> a wrong turn in my reasoning :-) > >> > >> I copy and paste: > >> > >> Why? > >> > >> - It leaves the way open for a hassle-free operator overloading > >> syntax - > >> use the same operator overloading conventions that Python use > >> even for > >> wrapped C library types without worrying about performance. > >> > >> - A convenient tool for doing compile-time calculations and having > >> parametrized code written at compile-time. The creative use C++'s > >> template > >> syntax has gotten over the years should demonstrate that the > >> inlining the > >> GCC compiler does is not sufficient for advanced use cases (see > >> http://ubiety.uwaterloo.ca/~tveldhui/papers/Expression-Templates/ > >> exprtmpl.html, > >> Blitz++ etc., basically using the C++ template system to inline > >> vector > >> multiplications etc. that's written dynamically according to vector > >> sizes). > >> > >> Dag Sverre > >> > >> _______________________________________________ > >> Cython-dev mailing list > >> Cython-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/cython-dev > >> > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > From dagss at student.matnat.uio.no Sat Mar 15 19:38:24 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 19:38:24 +0100 (CET) Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <706850310803151118q44cb05fcq44f4fdf3f89eb9de@mail.gmail.com> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> <680EB43A-21E6-400E-934C-154F552EAC89@math.washington.edu> <706850310803151118q44cb05fcq44f4fdf3f89eb9de@mail.gmail.com> Message-ID: <33253.85.165.149.126.1205606304.squirrel@webmail.uio.no> > Having a full python interperter around to get around the fact that we > don't have decent dependencies is almost certainly not the way to do > this. My dislike of parse tree optimizations mainly centers around > using it for very complicated things that can be much better served by > other algorithms. I'll admit (as you've probably concluded) that I'm out of my field of knowledge here. Still, seeing that you don't think my hack isn't ideal, what do you think would be the best way forward to get clean NumPy integration in, say, six weeks of full-time work by somebody who has no experience with compilers, but knows programming and text-book algorithms well? a) Custom plugin doing custom NumPy rewrites on the parse tree. (Easily done in one week for that matter...) b) Find a (very contrived) declarative syntax instead of Cython function calls for making the kind of code we need c) Just forget it, and start the long route of figuring out dependencies etc. towards a full-fledged optimizing compiler. I sort of figured my hack could provide a quick way forward... (BTW, reading books on compiler design would be an option.) Dag Sverre From gfurnish at indirectproof.net Sat Mar 15 19:46:02 2008 From: gfurnish at indirectproof.net (Gary Furnish) Date: Sat, 15 Mar 2008 12:46:02 -0600 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <33253.85.165.149.126.1205606304.squirrel@webmail.uio.no> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <706850310803150252x6822fce9k9875f70d6008c252@mail.gmail.com> <680EB43A-21E6-400E-934C-154F552EAC89@math.washington.edu> <706850310803151118q44cb05fcq44f4fdf3f89eb9de@mail.gmail.com> <33253.85.165.149.126.1205606304.squirrel@webmail.uio.no> Message-ID: <706850310803151146w48616bado78e1a390afbf276a@mail.gmail.com> I think A is quite acceptable (and probably a good idea). Anytime I hear "very contrived" in the design phase I get worried, so I don't like B. I think C might make a good topic to talk about eventually, but I think it is far more important to get numpy support in now (without worrying about optimization at all). Making C happen will probably require some design changes in Cython, so it is important in my book to define a public API to the parse tree and not let plugins poke around and do whatever they want. On 3/15/08, Dag Sverre Seljebotn wrote: > > > Having a full python interperter around to get around the fact that we > > don't have decent dependencies is almost certainly not the way to do > > this. My dislike of parse tree optimizations mainly centers around > > using it for very complicated things that can be much better served by > > other algorithms. > > > I'll admit (as you've probably concluded) that I'm out of my field of > knowledge here. Still, seeing that you don't think my hack isn't ideal, > what do you think would be the best way forward to get clean NumPy > integration in, say, six weeks of full-time work by somebody who has no > experience with compilers, but knows programming and text-book algorithms > well? > > a) Custom plugin doing custom NumPy rewrites on the parse tree. (Easily > done in one week for that matter...) > b) Find a (very contrived) declarative syntax instead of Cython function > calls for making the kind of code we need > c) Just forget it, and start the long route of figuring out dependencies > etc. towards a full-fledged optimizing compiler. > > I sort of figured my hack could provide a quick way forward... > > (BTW, reading books on compiler design would be an option.) > > > Dag Sverre > > From robertwb at math.washington.edu Sat Mar 15 22:04:35 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 15 Mar 2008 14:04:35 -0700 Subject: [Cython] When to run tree transformations In-Reply-To: <47DBA639.2070109@behnel.de> References: <47DBA639.2070109@behnel.de> Message-ID: <19BB1603-FF19-4D47-8248-743345783AA3@math.washington.edu> On Mar 15, 2008, at 3:34 AM, Stefan Behnel wrote: > Hi, > > as I suggested before, I think it would be helpful if plugins could > tell the > compiler directly when they were to run (without the current -T > option). I > think there would at least be an analysis step in addition to the > existing > tree processing step for the plugin lifecycle, but the plugin might > also need > to run at different times of the compiler lifecycle, and treat > different > syntax structures. I see the following lifecycle hooks here: > > - pre analyse types > - post analyse types - post type conversion phase? > - pre generate code > - post generate code What would happen in this last one? > I would suggest mapping these to method names in the plugins. You > could then write > > class MyTransform(Transform): > def pre_analyse_types(self, node, name): > ... > > instead of the current "process_node()" method. I think process_node offers more flexibility of phases, which may change/become more granular later on. Some transforms may need to know what the current phase is though. > I could also imagine keeping a set of node types in the plugin that > would be > used for selecting appropriate transformations for the current > node. In the > transformation example, there is this line > > if isinstance(node, Nodes.ForInStatNode) ... > > that determines when to run the plugin. If the plugin provided a > set of > interesting nodes instead, the infrastructure could store them in a > dict > (mapping node types to a list of plugins) and directly select the > transformations that can run on that node. Question in that case: > how to deal > with subtypes? Would they be expected to match? Or should the > plugin just > specify all node types, including their expected subtypes? Perhaps. One would have to come up with a reasonable way to filter things. The current approach seems the simplest way to get something up and running. - Robert - Robert From robertwb at math.washington.edu Sat Mar 15 22:12:54 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 15 Mar 2008 14:12:54 -0700 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> Message-ID: On Mar 15, 2008, at 2:25 AM, Dag Sverre Seljebotn wrote: > http://wiki.cython.org/enhancements/inlining > > As I think this is possible in a one or two week timeframe, I ask > of you > all to tell me what obvious things I am missing completely and > where I did > a wrong turn in my reasoning :-) > > I copy and paste: > > Why? > > - It leaves the way open for a hassle-free operator overloading > syntax - > use the same operator overloading conventions that Python use even for > wrapped C library types without worrying about performance. More comments on this in a separate email. > - A convenient tool for doing compile-time calculations and having > parametrized code written at compile-time. The creative use C++'s > template > syntax has gotten over the years should demonstrate that the > inlining the > GCC compiler does is not sufficient for advanced use cases (see > http://ubiety.uwaterloo.ca/~tveldhui/papers/Expression-Templates/ > exprtmpl.html, > Blitz++ etc., basically using the C++ template system to inline vector > multiplications etc. that's written dynamically according to vector > sizes). This mostly seems to be a proposal for compile-time calculations, and says very little about inlining and loop unrolling. I think compile- time calculations are a valuable optimization, but I think inlining (which one can explicitly request in the C output) and loop unrolling are well handled by GCC and is probably best handled at the this level for most things (for now at least). Keeping track of the contents of variables (e.g. has a variable been initialized, set to a certain value, within a certain range) would be very useful for getting rid of a lot of checks (especially if we want to raise LocalVariableErrors rather than treating all uninitialized variables as None). - Robert From stefan_ml at behnel.de Sat Mar 15 22:23:25 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 15 Mar 2008 22:23:25 +0100 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> Message-ID: <47DC3E4D.8000508@behnel.de> Hi, Robert Bradshaw wrote: > I think inlining > (which one can explicitly request in the C output) and loop unrolling > are well handled by GCC and is probably best handled at this > level for most things (for now at least). In general, yes. There is one thing that GCC can't do, however, and that's dropping the INCREF/DECREF of parameters in an inlined function. So, especially for very short functions (where inlining makes the biggest difference), having Cython do the inlining would actually result in a performance gain. But I agree that that's neither low-enough hanging fruit nor such a major gain that it's worth doing it right now... Stefan From robertwb at math.washington.edu Sat Mar 15 22:26:56 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 15 Mar 2008 14:26:56 -0700 Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: <47DC3E4D.8000508@behnel.de> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <47DC3E4D.8000508@behnel.de> Message-ID: On Mar 15, 2008, at 2:23 PM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> I think inlining >> (which one can explicitly request in the C output) and loop unrolling >> are well handled by GCC and is probably best handled at this >> level for most things (for now at least). > > In general, yes. There is one thing that GCC can't do, however, and > that's > dropping the INCREF/DECREF of parameters in an inlined function. So, > especially for very short functions (where inlining makes the biggest > difference), having Cython do the inlining would actually result in a > performance gain. Or, even better, detect when the incref/decref at the top and bottom of a function are unneeded, and remove them for all functions. > > But I agree that that's neither low-enough hanging fruit nor such a > major gain > that it's worth doing it right now... > > Stefan From stefan_ml at behnel.de Sat Mar 15 22:37:03 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 15 Mar 2008 22:37:03 +0100 Subject: [Cython] When to run tree transformations In-Reply-To: <19BB1603-FF19-4D47-8248-743345783AA3@math.washington.edu> References: <47DBA639.2070109@behnel.de> <19BB1603-FF19-4D47-8248-743345783AA3@math.washington.edu> Message-ID: <47DC417F.3010004@behnel.de> Hi, Robert Bradshaw wrote: > On Mar 15, 2008, at 3:34 AM, Stefan Behnel wrote: > >> Hi, >> >> as I suggested before, I think it would be helpful if plugins could >> tell the >> compiler directly when they were to run (without the current -T >> option). I >> think there would at least be an analysis step in addition to the >> existing >> tree processing step for the plugin lifecycle, but the plugin might >> also need >> to run at different times of the compiler lifecycle, and treat different >> syntax structures. I see the following lifecycle hooks here: >> >> - pre analyse types >> - post analyse types > > - post type conversion phase? Sure, good catch. >> - pre generate code >> - post generate code > > What would happen in this last one? Assuming that plugins are called during the tree traversal, the first would output code before the normal code generated for the node, and the second would add code behind it. Think of it as a very simple form of AOP join points. >> I would suggest mapping these to method names in the plugins. You >> could then write >> >> class MyTransform(Transform): >> def pre_analyse_types(self, node, name): >> ... >> >> instead of the current "process_node()" method. > > I think process_node offers more flexibility of phases, which may > change/become more granular later on. Some transforms may need to know > what the current phase is though. As Dag suggested, we could let a plugin register its transformations for specific phases. That makes it more general and explicit enough. >> I could also imagine keeping a set of node types in the plugin that >> would be >> used for selecting appropriate transformations for the current node. >> In the >> transformation example, there is this line >> >> if isinstance(node, Nodes.ForInStatNode) ... >> >> that determines when to run the plugin. If the plugin provided a set of >> interesting nodes instead, the infrastructure could store them in a dict >> (mapping node types to a list of plugins) and directly select the >> transformations that can run on that node. Question in that case: how >> to deal >> with subtypes? Would they be expected to match? Or should the plugin just >> specify all node types, including their expected subtypes? > > Perhaps. One would have to come up with a reasonable way to filter > things. The current approach seems the simplest way to get something up > and running. I'm fine with that - but not with the '-T' command line option: a) it doesn't work with distutils and b) it would be stupid to require users to take case that transformations run in the right phase. Activating/deactivating plugins is ok, but the configuration must be an explicit part of the plugin itself (be it code or declaration). Stefan From dagss at student.matnat.uio.no Sat Mar 15 23:31:09 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 23:31:09 +0100 (CET) Subject: [Cython] About putting my money where my mouth is Message-ID: <32970.85.165.149.126.1205620269.squirrel@webmail.uio.no> Just an explanation for my behaviour recently: I really do like to come with code rather than only opinions (like I did with the transform patch), so I'm sorry for how I've acted here lately. The reason I've only come with opinions is basically that this fascinated me enough that I'm going to attempt applying for a Google Summer of Code for Cython, for "Developing Cython towards better NumPy integration" (Note that the status of this is of course very uncertain, just let it serve as background for explaining my behaviour.) Because a good application must be written for that, it basically means that I have to only think and discuss and plan at the moment, as I don't have more time than I need for this before application deadline, meaning no coding or experimenting or "just getting there eventually" just now for me... (This doesn't mean that my application will contain lots of schetchy and little thought out subprojects, hopefully it will be very concise and manageable, but I want to do some exploration on possibilities etc. before handing the application in.) Thanks for any patience. Dag Sverre From dagss at student.matnat.uio.no Sat Mar 15 23:52:18 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 15 Mar 2008 23:52:18 +0100 (CET) Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> Message-ID: <33491.85.165.149.126.1205621538.squirrel@webmail.uio.no> > time calculations are a valuable optimization, but I think inlining > (which one can explicitly request in the C output) and loop unrolling > are well handled by GCC and is probably best handled at the this > level for most things (for now at least). Yes, I explained my rationale for that badly. The reason I wanted to do loop unrolling is because I think it would look very bad if the C code was littered with extra for-loops for every NumPy lookup, see: http://wiki.cython.org/enhancements/operators/ambitious which I have updated and made clearer. About inlining, as long as it wouldn't affect GCC's caching or noncaching of the stride calculations, I'm fine without. Details: ctypdef class numpy ... def __getitem__(self, index): return (self.data)[self.strides[0] // 4 * index[0] + self.strides[1] // 4 * index[0]] The above is the code as it would look in the parse-tree *after* compile-time optimization with the knowledge that type type is int and the dimensions 2. Note that the loop has been unrolled, resulting in that list of +.... also note that those stride calculations must either be cached by GCC or must also happen...hmm... I really should do some experiments I guess... Dag Sverre From dagss at student.matnat.uio.no Sun Mar 16 01:38:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 16 Mar 2008 01:38:04 +0100 (CET) Subject: [Cython] Draft for compile-time calculation inliner In-Reply-To: References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> Message-ID: <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> > > More comments on this in a separate email. Don't think I recieved it? Anyway, I think I finally understand your concerns (about time, I simply overlooked something crucial...), and my response is in a completely rewritten and much shorter and less complex http://wiki.cython.org/enhancements/operators/ambitious (Sorry about spamming you in the middle of your conference, just let it be until you have time...) Dag Sverre From robertwb at math.washington.edu Sun Mar 16 08:19:12 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 16 Mar 2008 00:19:12 -0700 Subject: [Cython] Draft for operator overloading In-Reply-To: <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> Message-ID: <0F62CBBD-B2FC-46A4-B952-27EEB2E8A03A@math.washington.edu> On Mar 15, 2008, at 5:38 PM, Dag Sverre Seljebotn wrote: >> More comments on this in a separate email. > > Don't think I recieved it? No. I didn't have time to post a full response then, and I'm just finding time to think about this now :). > Anyway, I think I finally understand your concerns (about time, I > simply > overlooked something crucial...), and my response is in a completely > rewritten and much shorter and less complex > > http://wiki.cython.org/enhancements/operators/ambitious It is certainly easier to understand, and seems much more concrete to me. (I could actually see this getting implemented.) Essentially, you still will have a transformation that converts the binary operations to a function call (which may or may not be inlined, either by Cython or by gcc (I would say the latter). The function body comes from the .pxd file. This is a bit odd, but I think the right place to put it. If the given type is a parameterized type, how would the "body" of the method refer to this compile-time data? You had a typeof(x) pseudo function that might work--i.e. typeof(x).foo() would call the method foo of the type x (as known at compile time), call foo() on it which by would return a note that would be inserted into the tree at that point. > (Sorry about spamming you in the middle of your conference, just > let it be > until you have time...) No problem. - Robert From dagss at student.matnat.uio.no Sun Mar 16 11:42:09 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 16 Mar 2008 11:42:09 +0100 (CET) Subject: [Cython] Draft for operator overloading In-Reply-To: <0F62CBBD-B2FC-46A4-B952-27EEB2E8A03A@math.washington.edu> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> <0F62CBBD-B2FC-46A4-B952-27EEB2E8A03A@math.washington.edu> Message-ID: <33484.85.165.149.126.1205664129.squirrel@webmail.uio.no> >> http://wiki.cython.org/enhancements/operators/ambitious > > Essentially, you still will have a transformation that converts the > binary operations to a function call (which may or may not be > inlined, either by Cython or by gcc (I would say the latter). The > function body comes from the .pxd file. This is a bit odd, but I > think the right place to put it. I'll tackle who inlines at the bottom, but I'll agree it should usually be gcc. Bear in mind that I don't have much experience wrapping C code with Cython, I've read the Pyrex docs but I didn't understand everything fully. I don't believe it has to be in the .pxd file but because I know little about the internals there it was a fuzzy point. Basically, the C++-style operators should be in the PXD file as they only are declarations and don't contain any code. However the Python-style operators should be in .pyx. Yes, a transform that translates operators to function calls is required. In the case that a C-style operator is declared (this will only happen on C++ classes and the native types) it is simply let through to C code (so it replaces whatever is currently syntax checking operator use), in the case that a Python-operator is declared it is translated to .__operator__(), which in turn gets translated to a normal C function call. I'm not sure whether I want to extend the struct so that it is possible to declare member functions (non-polymorphic!, basically all they would do is pass the subject as self, and make sure the function is mangled and implemented); or whether I want to require NumPy to basically implement a Python class (with fancy operators) wrapping the C type (without operators). > If the given type is a parameterized type, how would the "body" of > the method refer to this compile-time data? You had a typeof(x) > pseudo function that might work--i.e. typeof(x).foo() would call the > method foo of the type x (as known at compile time), call foo() on it > which by would return a note that would be inserted into the tree at > that point. I don't believe it is necesarry wth type-only functions. It is all standard Python stuff. However for a type with arguments one would want to (in a Python operator) access the type arguments, which could be done by type(self).arg). Then there's the issue of dealing with C++ templates (because "T" might be a type name in the C++-style operator+ declaration) but I think that should actually come in the form of native C++ template support and special keywords for that. About who inlines then: NumPy is an excemption because GCC isn't smart enough (at least on -O3). Consider accessing a numpy array in a tight for-loop. The [] operator must contain lookup and division of the stride which can be cached outside of the for-loop; however with If Cython inlines, then it might be possible for another transform coming *after* the inlining to "lift" out the recomputed variables (detect that they cannot change within the loop). But, it might be a bit far-fetched. Anyway, inlining would be controlled with a decorator. An alternative is a new functionality, I'll call it "scope", but I'll have to write a new draft about it I think. Dag Sverre From dagss at student.matnat.uio.no Sun Mar 16 11:44:25 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 16 Mar 2008 11:44:25 +0100 (CET) Subject: [Cython] Draft for operator overloading In-Reply-To: <33484.85.165.149.126.1205664129.squirrel@webmail.uio.no> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> <0F62CBBD-B2FC-46A4-B952-27EEB2E8A03A@math.washington.edu> <33484.85.165.149.126.1205664129.squirrel@webmail.uio.no> Message-ID: <33421.85.165.149.126.1205664265.squirrel@webmail.uio.no> > about the internals there it was a fuzzy point. Basically, the C++-style > operators should be in the PXD file as they only are declarations and > don't contain any code. However the Python-style operators should be in > .pyx. ...and merged with the declarations in the .pxd filed, like under "Sharing extension types" here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/sharing.html Dag Sverre From dagss at student.matnat.uio.no Sun Mar 16 12:58:42 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 16 Mar 2008 12:58:42 +0100 (CET) Subject: [Cython] NumPy code generation performance notes Message-ID: <33827.85.165.149.126.1205668722.squirrel@webmail.uio.no> This outlines the problem I'm dealing with these days in a more direct way, so that you can see the connection with all the stuff I been suggesting (Cython inlining and optimizer, operator overloading syntax etc.): http://wiki.cython.org/enhancements/numpy The scopevars idea is a bit underspecified, I don't have big hopes for it but I'll write it down when I get the time... Dag Sverre From robertwb at math.washington.edu Sun Mar 16 13:08:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 16 Mar 2008 05:08:07 -0700 Subject: [Cython] Draft for operator overloading In-Reply-To: <33484.85.165.149.126.1205664129.squirrel@webmail.uio.no> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> <0F62CBBD-B2FC-46A4-B952-27EEB2E8A03A@math.washington.edu> <33484.85.165.149.126.1205664129.squirrel@webmail.uio.no> Message-ID: <90F6816C-577F-4227-B3E6-83C383F728E3@math.washington.edu> On Mar 16, 2008, at 3:42 AM, Dag Sverre Seljebotn wrote: >>> http://wiki.cython.org/enhancements/operators/ambitious >> >> Essentially, you still will have a transformation that converts the >> binary operations to a function call (which may or may not be >> inlined, either by Cython or by gcc (I would say the latter). The >> function body comes from the .pxd file. This is a bit odd, but I >> think the right place to put it. > > I'll tackle who inlines at the bottom, but I'll agree it should > usually be > gcc. > > Bear in mind that I don't have much experience wrapping C code with > Cython, I've read the Pyrex docs but I didn't understand everything > fully. > > I don't believe it has to be in the .pxd file but because I know > little > about the internals there it was a fuzzy point. Basically, the C++- > style > operators should be in the PXD file as they only are declarations and > don't contain any code. However the Python-style operators should > be in > .pyx. The .pxd files should contain everything needed to cimport a class. This is the whole point of .pxd files, and since it is the module *using* the code that needs to know how to set/get items, etc. it needs to be in the .pxd file. An example that makes this clear is numpy--there isn't even a .pyx file to put the code into (just the .c files, and a .pxd file that goes along with them). > Yes, a transform that translates operators to function calls is > required. > In the case that a C-style operator is declared (this will only > happen on > C++ classes and the native types) it is simply let through to C > code (so > it replaces whatever is currently syntax checking operator use), in > the > case that a Python-operator is declared it is translated to > .__operator__(), which in turn gets translated to a normal C function > call. And then the default __operator__() would get translated to the Python operation (i.e. PyNumber_Add)? > I'm not sure whether I want to extend the struct so that it is > possible to > declare member functions (non-polymorphic!, basically all they > would do is > pass the subject as self, and make sure the function is mangled and > implemented); or whether I want to require NumPy to basically > implement a > Python class (with fancy operators) wrapping the C type (without > operators). Not quite following you here. >> If the given type is a parameterized type, how would the "body" of >> the method refer to this compile-time data? You had a typeof(x) >> pseudo function that might work--i.e. typeof(x).foo() would call the >> method foo of the type x (as known at compile time), call foo() on it >> which by would return a note that would be inserted into the tree at >> that point. > > I don't believe it is necesarry wth type-only functions. It is all > standard Python stuff. However for a type with arguments one would > want to > (in a Python operator) access the type arguments, which could be > done by > type(self).arg). The compile-time type and runtime type are two distinct objects, and will have different methods and members. There may not even be a 1-1 correspondence. And it would be unsafe at compile time to evaluate type(x) for most x (e.g. one could get a subclass). > Then there's the issue of dealing with C++ templates (because "T" > might be > a type name in the C++-style operator+ declaration) but I think that > should actually come in the form of native C++ template support and > special keywords for that. We'll have to deal with this before treating templates--if a cython array is type float, the return value of indexing should be the same. > About who inlines then: > > NumPy is an excemption because GCC isn't smart enough (at least on - > O3). > Consider accessing a numpy array in a tight for-loop. The [] > operator must > contain lookup and division of the stride which can be cached > outside of > the for-loop; however with > > If Cython inlines, then it might be possible for another transform > coming > *after* the inlining to "lift" out the recomputed variables (detect > that > they cannot change within the loop). But, it might be a bit far- > fetched. > Anyway, inlining would be controlled with a decorator. This is a good point--but I'm not sure if we should worry about this too much for the first iteration. > An alternative is a new functionality, I'll call it "scope", but > I'll have > to write a new draft about it I think. Sounds a lot like control flow analysis we kicked around back in December/January. At the time I thought it'd be too ambitious, but it seems more reasonable now. - Robert From dagss at student.matnat.uio.no Sun Mar 16 13:28:46 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 16 Mar 2008 13:28:46 +0100 (CET) Subject: [Cython] Draft for operator overloading In-Reply-To: <90F6816C-577F-4227-B3E6-83C383F728E3@math.washington.edu> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> <0F62CBBD-B2FC-46A4-B952-27EEB2E8A03A@math.washington.edu> <33484.85.165.149.126.1205664129.squirrel@webmail.uio.no> <90F6816C-577F-4227-B3E6-83C383F728E3@math.washington.edu> Message-ID: <33361.85.165.149.126.1205670526.squirrel@webmail.uio.no> (Leaving pxd/pyx to be dealt with later, it's becoming too much at once.) > And then the default __operator__() would get translated to the > Python operation (i.e. PyNumber_Add)? Note: I didn't intend the C++ syntax and Python syntax to be competing alternatives, I believe them to be complementary. I believe the reason we couldn't make any headway on operator syntax is because there's a distinction between declaring what is available on types in C++ directly (not C, no operator overloading there...) and thus must have C++ access code generated, versus "extra" Cython functionality (including redirecting operators to function calls in that functionality). The full chain of dealing with " a + b" would be: - If it is am "untyped" Python object, do as today, whatever that is. - Otherwise: - See if a.__add__ exists, if so, generate code to call it (as a regular C member function, whether __add__ has been declare cdef or def or pdef or whatever - have the same semantics as with other member functions, also with respect to pxd and pyx) - Then, see if a native operator is supported. This can be because it is a pointer or number, or because "operator+" has been declared as available on the type (latter in C++ mode only) - Otherwise, FAIL > The compile-time type and runtime type are two distinct objects, and > will have different methods and members. There may not even be a 1-1 > correspondence. And it would be unsafe at compile time to evaluate > type(x) for most x (e.g. one could get a subclass). Good point. Need to think more. C++ templates does deal with this exact problem, but the syntax might not fit us... Basically, that's what all this is about right... NumPy support needs template support. Was just hoping to avoid the functional programming style (and especially hoping to avoid creating a full template compiler..) Dag Sverre From robertwb at math.washington.edu Sun Mar 16 13:43:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 16 Mar 2008 05:43:14 -0700 Subject: [Cython] Draft for operator overloading In-Reply-To: <33361.85.165.149.126.1205670526.squirrel@webmail.uio.no> References: <58800.193.157.243.12.1205573151.squirrel@webmail.uio.no> <32987.85.165.149.126.1205627884.squirrel@webmail.uio.no> <0F62CBBD-B2FC-46A4-B952-27EEB2E8A03A@math.washington.edu> <33484.85.165.149.126.1205664129.squirrel@webmail.uio.no> <90F6816C-577F-4227-B3E6-83C383F728E3@math.washington.edu> <33361.85.165.149.126.1205670526.squirrel@webmail.uio.no> Message-ID: <0528B9A1-90E4-4D6E-8D47-60A1E07BCA1D@math.washington.edu> On Mar 16, 2008, at 5:28 AM, Dag Sverre Seljebotn wrote: > (Leaving pxd/pyx to be dealt with later, it's becoming too much at > once.) > >> And then the default __operator__() would get translated to the >> Python operation (i.e. PyNumber_Add)? > > Note: I didn't intend the C++ syntax and Python syntax to be competing > alternatives, I believe them to be complementary. My thoughts too. > I believe the reason we couldn't make any headway on operator > syntax is > because there's a distinction between declaring what is available > on types > in C++ directly (not C, no operator overloading there...) and thus > must > have C++ access code generated, versus "extra" Cython functionality > (including redirecting operators to function calls in that > functionality). > > The full chain of dealing with " a + b" would be: > > - If it is am "untyped" Python object, do as today, whatever that is. > - Otherwise: > - See if a.__add__ exists, if so, generate code to call it (as a > regular C > member function, whether __add__ has been declare cdef or def or > pdef or > whatever - have the same semantics as with other member functions, > also > with respect to pxd and pyx) > - Then, see if a native operator is supported. This can be because > it is a > pointer or number, or because "operator+" has been declared as > available > on the type (latter in C++ mode only) > - Otherwise, FAIL __add__ won't be a member function, it'll be a c function. I was suggesting that PyObject's default __add__ is PyObject_Add. This is your "whatever that is" for untyped objects. >> The compile-time type and runtime type are two distinct objects, and >> will have different methods and members. There may not even be a 1-1 >> correspondence. And it would be unsafe at compile time to evaluate >> type(x) for most x (e.g. one could get a subclass). > > Good point. Need to think more. C++ templates does deal with this > exact > problem, but the syntax might not fit us... > > Basically, that's what all this is about right... NumPy support needs > template support. Was just hoping to avoid the functional programming > style (and especially hoping to avoid creating a full template > compiler..) I think we can avoid creating a full template compiler, but we'll need some way to access the parameters of a parameterized type in a sensible way. - Robert From martin at martincmartin.com Mon Mar 17 23:04:34 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Mon, 17 Mar 2008 18:04:34 -0400 Subject: [Cython] --cplus option to the "cython" command line script Message-ID: <47DEEAF2.6070805@martincmartin.com> Hi, the --cplus option doesn't appear in the usage, and the comments in the code say its only supported on MacOS X. But that isn't true, it's used by the distutils when you specify langauge="c++". Can it be added to the usage in Cython/Compiler/CmdLine.py ? Best, Martin From jek-gmane at kleckner.net Tue Mar 18 01:08:23 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Mon, 17 Mar 2008 17:08:23 -0700 Subject: [Cython] Small patch for __stdcall Message-ID: The following small patch gets rid of some warnings on the cygwin platform (it it not WIN32 but does have definitions of __stdcall resulting in warnings). Please let me know if it will or won't get into the upstream release. Thanks. --- ModuleNode.py.orig 2008-03-17 16:55:53.946851100 -0700 +++ ModuleNode.py 2008-03-17 16:58:21.900687900 -0700 @@ -272,8 +272,10 @@ code.putln("#if PY_VERSION_HEX < 0x02040000") code.putln(" #define METH_COEXIST 0") code.putln("#endif") - code.putln("#ifndef WIN32") + code.putln("#ifndef __stdcall") code.putln(" #define __stdcall") + code.putln("#endif") + code.putln("#ifndef __cdecl") code.putln(" #define __cdecl") code.putln("#endif") self.generate_extern_c_macro_definition(code) From f.guerrieri at gmail.com Tue Mar 18 17:34:18 2008 From: f.guerrieri at gmail.com (Francesco Guerrieri) Date: Tue, 18 Mar 2008 17:34:18 +0100 Subject: [Cython] broken link on the homepage Message-ID: <79b79e730803180934y68a966b0n8d3ab32f988ca973@mail.gmail.com> Hi, this is my first post to the list :) The link to download Cython on the homepage is not correct, and returns a 404 Not Found. This happens both for the zip and the tar.gz format: The requested URL /cython-0.9.6.12.zip was not found on this server. The name should be Cython-0.9.6.12.zip with a capital C. Thanks for all the work, I'm beginning to use cython for my projects, so probably I will be back with some easy question... bye, Francesco From robertwb at math.washington.edu Tue Mar 18 23:36:45 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Mar 2008 15:36:45 -0700 Subject: [Cython] [Pyrex] Cython development In-Reply-To: <200803181824.14943.faltet@carabos.com> References: <47DB18C9.7030308@canterbury.ac.nz> <4866bea60803180820rc5151fbydeee4e2990ae0fe1@mail.gmail.com> <200803181824.14943.faltet@carabos.com> Message-ID: On Mar 18, 2008, at 10:24 AM, Francesc Altet wrote: > A Tuesday 18 March 2008, Chris Mellon escrigu?: >> On Fri, Mar 14, 2008 at 7:31 PM, Greg Ewing > wrote: >>> Stefan Behnel wrote: >>>> Then put a clear warning into the docs/FAQ: "the C files are >>>> generated, please ignore them when reading the source". >>> >>> Pyrex already puts a note at the top of the .c file >>> indicating that it was generated. I could perhaps >>> enhance it to include a warning against editing by >>> hand, although I think to most people it would be >>> fairly obvious from the appearance of the code that >>> you're not meant to do that! >>> >>> -- >> >> And really, as generated C source goes, Pyrex/Cython produce some of >> the easiest to read I've ever seen. The inclusion of the source (with >> line numbers and context!) for each chunk of the generated C code is >> great. > > Yeah, i definitely agree with this. When things go wrong, it is quite > easy to have a look at C sources. Thanks for that! If you're using Cython, you can use the -a option and it will produced an html of your source (colorized according to how much Python is being used, and the lines are clickable (expanding to the C). See, for example http://sage.math.washington.edu/home/robertwb/ cython/polynomial_element.pyx.html - Robert From robertwb at math.washington.edu Tue Mar 18 23:41:45 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Mar 2008 15:41:45 -0700 Subject: [Cython] [Pyrex] exposing extension class attributes In-Reply-To: <20080318135557.6194090f.simon@arrowtheory.com> References: <20080318135557.6194090f.simon@arrowtheory.com> Message-ID: On Mar 18, 2008, at 10:55 AM, Simon Burton wrote: > > I am considering using some kind of name convention to distinguish > between > extension class access at the python and at the c level: > > cdef class Foo: > > cdef int _foo > > def __init__(self, int foo): > self._foo = foo > > property foo: > def __get__(self): > return self._foo > > > Ie. the c attribute has an underscore in front of it, and > from python the attribute is without underscore. > > As this is likely to drive me moderately crazy (keeping track > of underscores) I am wondering how other people handle this issue ? Declare foo as public or readonly (e.g. cdef public int foo or cdef readonly int foo). Pyrex/Cython will then create the get/set property methods for you. > I guess I am just trying to write C code without writing C code (or > C++ code). > Perhaps I can abandon (exposing these classes to) the python > interpreter altogether, > and stick to writing pyrex code. > > A similar problem occurs with methods. I gave up on duplicating > every method > (a cdef version and a python version), and am just using python > methods for now. Cython has "cpdef" methods that does this for you--if you call it from C it's a cdef method, if you call it from Python it's a def method. These can be overridden in the usual way and it does all the right stuff. - Robert From robertwb at math.washington.edu Wed Mar 19 01:15:01 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Mar 2008 17:15:01 -0700 Subject: [Cython] [Pyrex] exposing extension class attributes In-Reply-To: <47E05879.4050605@canterbury.ac.nz> References: <20080318135557.6194090f.simon@arrowtheory.com> <47E05879.4050605@canterbury.ac.nz> Message-ID: On Mar 18, 2008, at 5:04 PM, Greg Ewing wrote: > Simon Burton wrote: >> Ie. the c attribute has an underscore in front of it, and >> from python the attribute is without underscore. >> >> As this is likely to drive me moderately crazy (keeping track >> of underscores) I am wondering how other people handle this issue ? > > Would calling it "c_foo" make you feel any better? > > I'm not sure what could be done to make this any easier, > short of having some magic by which the same name refers > to the C attribute in Pyrex and the Python property in > Python. But you need to be able to refer to the property > in Pyrex as well, so I can't see how that would work. > >> A similar problem occurs with methods. I gave up on duplicating >> every method >> (a cdef version and a python version), and am just using python >> methods for now. > > Cython has a mechanism for making C methods callable from > Python as well as Pyrex, but last I heard, the implementation > is incorrect in the presence of overriding+inheritance, and > the Cython people are unwilling to pay the performance cost > required to make it work properly. Overriding and inheritance work for cpdef methods exactly as they do in Python. This has a slight speed penalty (one dictionary lookup) in some cases, but in the most common case it is essentially as fast as a cdef method. - Robert From robertwb at math.washington.edu Wed Mar 19 03:45:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Mar 2008 19:45:33 -0700 Subject: [Cython] [Pyrex] exposing extension class attributes In-Reply-To: <47E0739A.4050704@canterbury.ac.nz> References: <20080318135557.6194090f.simon@arrowtheory.com> <47E05879.4050605@canterbury.ac.nz> <47E0739A.4050704@canterbury.ac.nz> Message-ID: <2E6A2EF1-7B88-4797-B046-2D71A65CB7FB@math.washington.edu> On Mar 18, 2008, at 6:59 PM, Greg Ewing wrote: > Robert Bradshaw wrote: > >> Overriding and inheritance work for cpdef methods exactly as they >> do in Python. > > Does that mean you've fixed the inherited method call > problem? What approach did you use? Yes. If the object has a dictionary, it does a lookup to see if the method has been overridden. It also checks a global flag that is set to skip this dispatching step in some cases. - Robert From jek-gmane at kleckner.net Wed Mar 19 06:04:23 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Tue, 18 Mar 2008 22:04:23 -0700 Subject: [Cython] cpdef wrapping of char* functions Message-ID: I tried some wrapping of a function that returns a char* by using cpdef and got an error: File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", line 897, in call_self_node cfunc = ExprNodes.NameNode(self.pos, name=self.declarator.base.name) AttributeError: CFuncDeclaratorNode instance has no attribute 'name' Should this work? It might be because it is at the module level and not in a class perhaps? This is using 9.6.12. Test case: cdef extern from "foo.h": cdef int c_baz "baz"() cdef char* c_bar "bar"() cpdef int baz(): return c_baz() cpdef char* bar(): return c_bar() From robertwb at math.washington.edu Wed Mar 19 06:07:41 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Mar 2008 22:07:41 -0700 Subject: [Cython] cpdef wrapping of char* functions In-Reply-To: References: Message-ID: <719C7F28-66DD-40A7-906D-94BFAED1BCE0@math.washington.edu> Yes, this should work. I'll look into what's going wrong (cpdef at the module level is rather new, but I wasn't expecting surprises like this). - Robert On Mar 18, 2008, at 10:04 PM, Jim Kleckner wrote: > I tried some wrapping of a function that returns a char* by > using cpdef and got an error: > > File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", > line 897, in call_self_node > cfunc = ExprNodes.NameNode(self.pos, > name=self.declarator.base.name) > AttributeError: CFuncDeclaratorNode instance has no attribute 'name' > > Should this work? > It might be because it is at the module level and not in a class > perhaps? > > This is using 9.6.12. > > > Test case: > > cdef extern from "foo.h": > cdef int c_baz "baz"() > cdef char* c_bar "bar"() > > cpdef int baz(): > return c_baz() > > cpdef char* bar(): > return c_bar() > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From f.guerrieri at gmail.com Wed Mar 19 12:23:32 2008 From: f.guerrieri at gmail.com (Francesco Guerrieri) Date: Wed, 19 Mar 2008 12:23:32 +0100 Subject: [Cython] newbie question on dynamic size for array Message-ID: <79b79e730803190423i6b047f48k8415c32d480f8a12@mail.gmail.com> Hi :) I'm just beginning to experiment with Cython. It involves a slight (or not-so-slight!) rethinking of some approaches, but it seems rather smooth to me. I think that I like it :) Now I come to my question: I'm trying to understand how to manage memory, which is definitely something which you don't do in your python coding of everyday :-) I am trying to dynamically size an array. Could you link me to some example (the sage codebase will be fine) or link? As I expected, cdef int size = 10 cdef int p[size] doesn't work. I found the following link http://permalink.gmane.org/gmane.comp.python.cython.devel/162 which seemed very promising :-) but I don't see any reply to it... I have found the following code in one of the Demos (numeric_demo.pyx): cdef float *elems elems = a.data but a.data (a is an Array) is not initialized in the example :) I have found the following page on the wiki http://www.sagemath.org:9001/MallocReplacements but doesn't answer my basic question. I _know_ am that I am missing something very basic. Ah, unfortunately I have no working knowledge of the C API, while probably the answer lies there.. thanks in advance, Francesco From robertwb at math.washington.edu Wed Mar 19 21:41:37 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Mar 2008 13:41:37 -0700 Subject: [Cython] newbie question on dynamic size for array In-Reply-To: <79b79e730803190423i6b047f48k8415c32d480f8a12@mail.gmail.com> References: <79b79e730803190423i6b047f48k8415c32d480f8a12@mail.gmail.com> Message-ID: <95BF7911-C495-43BE-97CF-8A98F6986F5A@math.washington.edu> On Mar 19, 2008, at 4:23 AM, Francesco Guerrieri wrote: > Hi :) > I'm just beginning to experiment with Cython. It involves a slight (or > not-so-slight!) rethinking of some > approaches, but it seems rather smooth to me. I think that I like > it :) > Now I come to my question: I'm trying to understand how to manage > memory, > which is definitely something which you don't do in your python coding > of everyday :-) > > I am trying to dynamically size an array. > Could you link me to some example (the sage codebase will be fine) > or link? > > As I expected, > cdef int size = 10 > cdef int p[size] > > doesn't work. I found the following link > http://permalink.gmane.org/gmane.comp.python.cython.devel/162 > which seemed very promising :-) but I don't see any reply to it... > > I have found the following code in one of the Demos > (numeric_demo.pyx): > > cdef float *elems > elems = a.data > > but a.data (a is an Array) is not initialized in the example :) > > I have found the following page on the wiki > http://www.sagemath.org:9001/MallocReplacements > but doesn't answer my basic question. I _know_ am that I am missing > something very basic. > Ah, unfortunately I have no working knowledge of the C API, > while probably the answer lies there.. Have you ever written code in C? Cython, with respect to pointer memory management, is exactly the same. What you should do is include stdlib.pxi (inside the cython/includes directory). Alternatively, you can just paste that code at the top of your file. You now have 3 functions--request memory, resize memory, and free memory. A typical example would be cdef int size = 10 cdef float *elms = malloc(size * sizeof(float)) # returns a chunk of memory that's size * sizeof(float) bytes big [do some stuff with elms] size *= 2 cdef float *elms = realloc(elms, size * sizeof(float)) # makes your chunk bigger, possibly moving (and copying) the old data [some more stuff] free(*elms) # tell the system you no longer need this chunk of memory. If you don't do this, you will have a memory leak If you're putting this array in cdef classes, a good place to put malloc is in the __cinit__ (aka __new__) method, and put a free in the __dealloc__ method. That way you know whenever you request memory you will give it back when you're done. - Robert From jek-gmane at kleckner.net Wed Mar 19 23:05:44 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Wed, 19 Mar 2008 15:05:44 -0700 Subject: [Cython] Declaring and using call-by-reference arguments Message-ID: Has anyone wrapped functions with call-by-reference syntax? What do you recommend? Thanks. From jek-gmane at kleckner.net Wed Mar 19 23:55:46 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Wed, 19 Mar 2008 15:55:46 -0700 Subject: [Cython] Declaring and using call-by-reference arguments In-Reply-To: References: Message-ID: Jim Kleckner wrote: > Has anyone wrapped functions with call-by-reference syntax? > What do you recommend? One workaround is to declare to cython that the function requires an object and then dereference a pointer with ptr[0]. Per this page: http://www.sagemath.org/doc/html/prog/node63.html From f.guerrieri at gmail.com Thu Mar 20 09:54:28 2008 From: f.guerrieri at gmail.com (Francesco Guerrieri) Date: Thu, 20 Mar 2008 09:54:28 +0100 Subject: [Cython] newbie question on dynamic size for array In-Reply-To: <95BF7911-C495-43BE-97CF-8A98F6986F5A@math.washington.edu> References: <79b79e730803190423i6b047f48k8415c32d480f8a12@mail.gmail.com> <95BF7911-C495-43BE-97CF-8A98F6986F5A@math.washington.edu> Message-ID: <79b79e730803200154r57450573ub484da49f337ac75@mail.gmail.com> On Wed, Mar 19, 2008 at 9:41 PM, Robert Bradshaw wrote: > Have you ever written code in C? Cython, with respect to pointer > memory management, is exactly the same. Thanks Robert. Luckily I know C (and even more luckily I try do to most of my work in Python) :-) I didn't find a reference to stdlib.pxi in the tutorials... surely it was there and escaped my attention! I will try your code and will probably be back with other questions in a short time :) Francesco From f.guerrieri at gmail.com Thu Mar 20 11:57:34 2008 From: f.guerrieri at gmail.com (Francesco Guerrieri) Date: Thu, 20 Mar 2008 11:57:34 +0100 Subject: [Cython] from arrays to python objects Message-ID: <79b79e730803200357l782d3fe3oc9aee1071dc6da8c@mail.gmail.com> Hi, it's always me :-) I've been looking for a standard way for passing from a c array to a python object, eg a list. In the primes example, this is solved by creating both the array (with a fixed size, but now I know how to handle it :-) ) and the list, and then appending the relevant prime numbers to the list. Is this the idiomatic approach? I have looked for something similar in the sage codebase but didn't find something relevant. So if I have an array, do I have always to do something like this? for i from 0 <= i < size_of_array: my_list.append(array[i]) return my_list thanks in advance, Francesco From dagss at student.matnat.uio.no Thu Mar 20 13:35:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 20 Mar 2008 13:35:44 +0100 (CET) Subject: [Cython] from arrays to python objects In-Reply-To: <79b79e730803200357l782d3fe3oc9aee1071dc6da8c@mail.gmail.com> References: <79b79e730803200357l782d3fe3oc9aee1071dc6da8c@mail.gmail.com> Message-ID: <1068.195.159.185.73.1206016544.squirrel@webmail.uio.no> > So if I have an array, do I have always to do something like this? You can consider using NumPy arrays. Then you construct a NumPy array the Python way, and only access it's internal buffer from Cython, meaning that you always have a full, valid Python object. NumPy contains methods for converting that array to a list if you need to (but it is more efficient to simply use the array in your Python code). http://new.scipy.org/Wiki/Cookbook/Pyrex_and_NumPy Dag Sverre From f.guerrieri at gmail.com Thu Mar 20 14:36:56 2008 From: f.guerrieri at gmail.com (Francesco Guerrieri) Date: Thu, 20 Mar 2008 14:36:56 +0100 Subject: [Cython] from arrays to python objects In-Reply-To: <1068.195.159.185.73.1206016544.squirrel@webmail.uio.no> References: <79b79e730803200357l782d3fe3oc9aee1071dc6da8c@mail.gmail.com> <1068.195.159.185.73.1206016544.squirrel@webmail.uio.no> Message-ID: <79b79e730803200636k305f6f07hdee217a979d81d16@mail.gmail.com> On Thu, Mar 20, 2008 at 1:35 PM, Dag Sverre Seljebotn wrote: > > So if I have an array, do I have always to do something like this? > > You can consider using NumPy arrays. Then you construct a NumPy array the > Python way, and only access it's internal buffer from Cython, meaning that > you always have a full, valid Python object. NumPy contains methods for > converting that array to a list if you need to (but it is more efficient > to simply use the array in your Python code). > > http://new.scipy.org/Wiki/Cookbook/Pyrex_and_NumPy > > > Dag Sverre This seems a very sensible suggestion. I can keep it as a NumPy array, no need to convert it to a list. Thanks Dag. Francesco From robertwb at math.washington.edu Thu Mar 20 14:49:04 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 20 Mar 2008 06:49:04 -0700 Subject: [Cython] [Pyrex] How to deal with a block of memory when pickling? In-Reply-To: References: Message-ID: <35CE5837-E89C-4BA7-8729-217CE8C59173@math.washington.edu> On Mar 20, 2008, at 6:42 AM, Simon King wrote: > Good afternoon! > > I guess that the following has a standard solution, but i don't > know it. > > Using cython (hope that the pyrex list also answers cython questions), cython-dev at codespeak.net is the official Cython mailing list, but many of us lurk here too (as there are often common questions). > I cdefined some class MTX (it is a wrapper for C-MeatAxe matrices). > One of > the attributes is a pointer p to a block of memory that defines the > coefficients of the matrix. > > Now i want to pickle it. One obvious way is to get a list of matrix > entries and work with it. But this is too slow. Is there a more > direct way > to handle the block of memory p is pointing at? > > I tried to make a type cast of p to char* and to read this as a > string. > In some cases it worked, and going the way back (string -> char*) > reconstructed the memory block p was pointing at. > However, sometimes apparently it was not clear to python how long the > string is supposed to be; so i got an empty string, and the > reconstruction > failed. > > What can i do? Use the PyString_FromStringAndSize function. (Otherwise it assumes a null character is the end of the string, per C convention.) - Robert From stefan_ml at behnel.de Thu Mar 20 08:09:49 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 20 Mar 2008 08:09:49 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd Message-ID: <47E20DBD.5000208@behnel.de> Hi, since William brought up this topic lately, I think there are a couple of very valid cases where you want to define a macro in C and use it as a function in Cython. This currently means: 1) write a macros.h file to hold the macro 2) add a 'cdef extern from "macros.h"' to your favourite .pxd file to define your macros in Cython So this requires you to keep track of two files for things that may just be a couple of lines in C. I think this is so common that it would help if you could write inline C code in a .pxd file and have it written into your generated C file automatically. I could imagine a syntax like this: # macros.pxd MACROS = """ #define _isString(obj) (PyString_CheckExact(obj) || \ PyUnicode_CheckExact(obj) || \ PyObject_TypeCheck(obj, &PyBaseString_Type)) #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) """ cdef inline from MACROS: cdef int _isString(object obj) cdef char* _fqtypename(object t) Instead of an '#include "macros.h"', this would write the string given by MACROS verbatimly into the C file, directly behind the #include section. It could also keep track of the strings in a set to make sure they are only written out once, even when cimported redundantly at various places. Any comments on this? Stefan From dagss at student.matnat.uio.no Fri Mar 21 09:22:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 21 Mar 2008 09:22:00 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd Message-ID: <3288936139.869343@smtp.netcom.no> I have been thinking the same thing myself. Just wanted to also point in the direction of inlineable functions in pxd-file in my draft on the wiki (under enhancements). The more features Cython gets, the more cases can be solved by inlineable functions written in Cython instead of with macros (the benefit being of course that one can stick to Cython as the language). No problems with doing both though... Dag Sverre Seljebotn -----Original Message----- From: Stefan Behnel Date: Friday, Mar 21, 2008 8:38 am Subject: [Cython] Syntax for writing C macros in a Cython .pxd To: Cython-dev Hi, since William brought up this topic lately, I think there are a couple of very valid cases where you want to define a macro in C and use it as a function in Cython. This currently means: 1) write a macros.h file to hold the macro 2) add a 'cdef extern from "macros.h"' to your favourite .pxd file to define your macros in Cython So this requires you to keep track of two files for things that may just be a couple of lines in C. I think this is so common that it would help if you could write inline C code in a .pxd file and have it written into your generated C file automatically. I could imagine a syntax like this: # macros.pxd MACROS = """ #define _isString(obj) (PyString_CheckExact(obj) || \ PyUnicode_CheckExact(obj) || \ PyObject_TypeCheck(obj, &PyBaseString_Type)) #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) """ cdef inline from MACROS: cdef int _isString(object obj) cdef char* _fqtypename(object t) Instead of an '#include "macros.h"', this would write the string given by MACROS verbatimly into the C file, directly behind the #include section. It could also keep track of the strings in a set to make sure they are only written out once, even when cimported redundantly at various places. Any comments on this? Stefan _______________________________________________ Cython-dev mailing list Cython-dev at codespeak.net http://codespeak.net/mailman/listinfo/cython-dev From martin at martincmartin.com Fri Mar 21 15:16:50 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Fri, 21 Mar 2008 10:16:50 -0400 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: <47E20DBD.5000208@behnel.de> References: <47E20DBD.5000208@behnel.de> Message-ID: <47E3C352.2060305@martincmartin.com> Macros in C have a lot of problems, so I think it's best to avoid them whenever possible. In particular: - The symbols they define don't show up in the debugger, memory profiler, or other tools. - They work at the textual level, so it's easy to get incorrect behaviour if you forget to add extra parenthesis that aren't needed in normal C. - When using them to define an inline "function," they can't always be used all places a function can. In your example: #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) _fqtypename(a + b) expands to: (((PyTypeObject*)a + b)->ob_type->tp_name) And by C's precedence rules, the cast applies to "a" *before* "b" is added. If you want an inline function, why not just declare it inline? The use cases discussed on the list recently were to get around the fact that Cython doesn't know all that much about C or C++. Why not just teach it more? No language that learned from C, such as Java and Python, include macros. C++ tries hard to provide language support for all idioms that C needs them for, and there's even some talk of deprecating macros in C++ (IIRC). So, I think the response should be to teach Cython about references, rather than macros. What do people think? Best, Martin Stefan Behnel wrote: > Hi, > > since William brought up this topic lately, I think there are a couple of very > valid cases where you want to define a macro in C and use it as a function in > Cython. This currently means: > > 1) write a macros.h file to hold the macro > 2) add a 'cdef extern from "macros.h"' to your favourite .pxd file to define > your macros in Cython > > So this requires you to keep track of two files for things that may just be a > couple of lines in C. > > I think this is so common that it would help if you could write inline C code > in a .pxd file and have it written into your generated C file automatically. I > could imagine a syntax like this: > > # macros.pxd > MACROS = """ > #define _isString(obj) (PyString_CheckExact(obj) || \ > PyUnicode_CheckExact(obj) || \ > PyObject_TypeCheck(obj, &PyBaseString_Type)) > > #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) > """ > > cdef inline from MACROS: > cdef int _isString(object obj) > cdef char* _fqtypename(object t) > > Instead of an '#include "macros.h"', this would write the string given by > MACROS verbatimly into the C file, directly behind the #include section. It > could also keep track of the strings in a set to make sure they are only > written out once, even when cimported redundantly at various places. > > Any comments on this? > > Stefan > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From ndbecker2 at gmail.com Fri Mar 21 15:18:42 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 21 Mar 2008 10:18:42 -0400 Subject: [Cython] Syntax for writing C macros in a Cython .pxd References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> Message-ID: Martin C. Martin wrote: > Macros in C have a lot of problems, so I think it's best to avoid them > whenever possible. In particular: > > - The symbols they define don't show up in the debugger, memory > profiler, or other tools. > - They work at the textual level, so it's easy to get incorrect > behaviour if you forget to add extra parenthesis that aren't needed in > normal C. > - When using them to define an inline "function," they can't always be > used all places a function can. > > In your example: > > #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) > > _fqtypename(a + b) expands to: > > (((PyTypeObject*)a + b)->ob_type->tp_name) > > And by C's precedence rules, the cast applies to "a" *before* "b" is > added. > > If you want an inline function, why not just declare it inline? > > The use cases discussed on the list recently were to get around the fact > that Cython doesn't know all that much about C or C++. Why not just > teach it more? > > No language that learned from C, such as Java and Python, include > macros. C++ tries hard to provide language support for all idioms that > C needs them for, and there's even some talk of deprecating macros in > C++ (IIRC). > > So, I think the response should be to teach Cython about references, > rather than macros. What do people think? > > Best, > Martin > > Stefan Behnel wrote: >> Hi, >> >> since William brought up this topic lately, I think there are a couple of >> very valid cases where you want to define a macro in C and use it as a >> function in Cython. This currently means: >> >> 1) write a macros.h file to hold the macro >> 2) add a 'cdef extern from "macros.h"' to your favourite .pxd file to >> define your macros in Cython >> >> So this requires you to keep track of two files for things that may just >> be a couple of lines in C. >> >> I think this is so common that it would help if you could write inline C >> code in a .pxd file and have it written into your generated C file >> automatically. I could imagine a syntax like this: >> >> # macros.pxd >> MACROS = """ >> #define _isString(obj) (PyString_CheckExact(obj) || \ >> PyUnicode_CheckExact(obj) || \ >> PyObject_TypeCheck(obj, >> &PyBaseString_Type)) >> >> #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) >> """ >> >> cdef inline from MACROS: >> cdef int _isString(object obj) >> cdef char* _fqtypename(object t) >> >> Instead of an '#include "macros.h"', this would write the string given by >> MACROS verbatimly into the C file, directly behind the #include section. >> It could also keep track of the strings in a set to make sure they are >> only written out once, even when cimported redundantly at various places. >> >> Any comments on this? >> Ideally, I agree - but I have no idea how hard it is or how long it would take. From stefan_ml at behnel.de Fri Mar 21 19:24:56 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 21 Mar 2008 19:24:56 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: <47E3C352.2060305@martincmartin.com> References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> Message-ID: <47E3FD78.9060803@behnel.de> Hi, Martin C. Martin to-posted: > Stefan Behnel wrote: >> since William brought up this topic lately, I think there are a couple >> of very valid cases where you want to define a macro in C and use it as a >> function in Cython. >> >> I think this is so common that it would help if you could write inline >> C code in a .pxd file and have it written into your generated C file >> automatically. I could imagine a syntax like this: >> >> # macros.pxd >> MACROS = """ >> #define _isString(obj) (PyString_CheckExact(obj) || \ >> PyUnicode_CheckExact(obj) || \ >> PyObject_TypeCheck(obj, >> &PyBaseString_Type)) >> >> #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) >> """ >> >> cdef inline from MACROS: >> cdef int _isString(object obj) >> cdef char* _fqtypename(object t) > > Macros in C have a lot of problems, so I think it's best to avoid them > whenever possible. In particular: > > - The symbols they define don't show up in the debugger, memory > profiler, or other tools. Well, in most cases macros are rather short code snippets, not function replacing code blocks. So I don't care about their symbols. > - They work at the textual level, so it's easy to get incorrect > behaviour if you forget to add extra parenthesis that aren't needed in > normal C. > - When using them to define an inline "function," they can't always be > used all places a function can. Sure, that's why you have to take care when you use them. In my first sentence, I said "I think there are a couple of very valid cases", I think that makes the focus clear. > In your example: > > #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) > > _fqtypename(a + b) expands to: > > (((PyTypeObject*)a + b)->ob_type->tp_name) And that's definitely not a valid use case of _fqtypename(). > If you want an inline function, why not just declare it inline? Because for small code snippets that exist solely for performance reasons, the overhead of a function defined in Cython may be too high. > The use cases discussed on the list recently were to get around the fact > that Cython doesn't know all that much about C or C++. Why not just > teach it more? Does that imply a 'real' syntax for C macros? I would deny the need. As you noted, macros are defined at the text level. Stefan From robertwb at math.washington.edu Fri Mar 21 19:31:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 21 Mar 2008 11:31:33 -0700 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: <47E3C352.2060305@martincmartin.com> References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> Message-ID: <75898B2A-DB20-4ACA-84B1-A3787ACEEF54@math.washington.edu> On Mar 21, 2008, at 7:16 AM, Martin C. Martin wrote: > Macros in C have a lot of problems, so I think it's best to avoid them > whenever possible. In particular: > > - The symbols they define don't show up in the debugger, memory > profiler, or other tools. > - They work at the textual level, so it's easy to get incorrect > behaviour if you forget to add extra parenthesis that aren't needed in > normal C. > - When using them to define an inline "function," they can't always be > used all places a function can. > > In your example: > > #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) > > _fqtypename(a + b) expands to: > > (((PyTypeObject*)a + b)->ob_type->tp_name) > > And by C's precedence rules, the cast applies to "a" *before* "b" > is added. > > If you want an inline function, why not just declare it inline? > > The use cases discussed on the list recently were to get around the > fact > that Cython doesn't know all that much about C or C++. Why not just > teach it more? > > No language that learned from C, such as Java and Python, include > macros. C++ tries hard to provide language support for all idioms > that > C needs them for, and there's even some talk of deprecating macros in > C++ (IIRC). > > So, I think the response should be to teach Cython about references, > rather than macros. What do people think? > > Best, > Martin > > Stefan Behnel wrote: >> Hi, >> >> since William brought up this topic lately, I think there are a >> couple of very >> valid cases where you want to define a macro in C and use it as a >> function in >> Cython. This currently means: >> >> 1) write a macros.h file to hold the macro >> 2) add a 'cdef extern from "macros.h"' to your favourite .pxd file >> to define >> your macros in Cython >> >> So this requires you to keep track of two files for things that >> may just be a >> couple of lines in C. >> >> I think this is so common that it would help if you could write >> inline C code >> in a .pxd file and have it written into your generated C file >> automatically. I >> could imagine a syntax like this: >> >> # macros.pxd >> MACROS = """ >> #define _isString(obj) (PyString_CheckExact(obj) || \ >> PyUnicode_CheckExact(obj) || \ >> PyObject_TypeCheck(obj, >> &PyBaseString_Type)) >> >> #define _fqtypename(o) (((PyTypeObject*)o)->ob_type- >> >tp_name) >> """ >> >> cdef inline from MACROS: >> cdef int _isString(object obj) >> cdef char* _fqtypename(object t) >> >> Instead of an '#include "macros.h"', this would write the string >> given by >> MACROS verbatimly into the C file, directly behind the #include >> section. It >> could also keep track of the strings in a set to make sure they >> are only >> written out once, even when cimported redundantly at various places. >> >> Any comments on this? Cython already has inline functions and will soon support many more C+ + idioms, which should cover 90% or more what macros are used for, and in a much cleaner way. In fact, both of the examples you give can be done just fine this way. Allowing arbitrary chunks of C in strings in pxd files seems ugly as well as encouraging bad hacks. If one really wants to code raw C using another file seems much more natural. - Robert From martin at martincmartin.com Fri Mar 21 19:39:24 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Fri, 21 Mar 2008 14:39:24 -0400 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: <47E3FD78.9060803@behnel.de> References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> Message-ID: <47E400DC.1080402@martincmartin.com> Hi Stefan, Stefan Behnel wrote: > Hi, > > Martin C. Martin to-posted: >> Stefan Behnel wrote: >>> since William brought up this topic lately, I think there are a couple >>> of very valid cases where you want to define a macro in C and use it as a >>> function in Cython. >>> >>> I think this is so common that it would help if you could write inline >>> C code in a .pxd file and have it written into your generated C file >>> automatically. I could imagine a syntax like this: >>> >>> # macros.pxd >>> MACROS = """ >>> #define _isString(obj) (PyString_CheckExact(obj) || \ >>> PyUnicode_CheckExact(obj) || \ >>> PyObject_TypeCheck(obj, >>> &PyBaseString_Type)) >>> >>> #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) >>> """ >>> >>> cdef inline from MACROS: >>> cdef int _isString(object obj) >>> cdef char* _fqtypename(object t) >> Macros in C have a lot of problems, so I think it's best to avoid them >> whenever possible. In particular: >> >> - The symbols they define don't show up in the debugger, memory >> profiler, or other tools. > > Well, in most cases macros are rather short code snippets, not function > replacing code blocks. So I don't care about their symbols. That's surprising to me. Don't you think you'd want to step through even short snippets in a debugger? To have them show up as a separate item in a profile, rather than spread around all the places that call them? Same with memory leaks or corruption? > >> - They work at the textual level, so it's easy to get incorrect >> behaviour if you forget to add extra parenthesis that aren't needed in >> normal C. >> - When using them to define an inline "function," they can't always be >> used all places a function can. > > Sure, that's why you have to take care when you use them. In my first > sentence, I said "I think there are a couple of very valid cases", I think > that makes the focus clear. > > >> In your example: >> >> #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) >> >> _fqtypename(a + b) expands to: >> >> (((PyTypeObject*)a + b)->ob_type->tp_name) > > And that's definitely not a valid use case of _fqtypename(). Why is that? Isn't pointer arithmetic a valid use case? >> If you want an inline function, why not just declare it inline? > > Because for small code snippets that exist solely for performance reasons, the > overhead of a function defined in Cython may be too high. But if it's declared inline, there's no function call overhead at runtime. Unless you're in a debug build, in which case you'd rather have the overhead to help track down bugs, leaks, etc. Do you mean compile time overhead? Or something else? >> The use cases discussed on the list recently were to get around the fact >> that Cython doesn't know all that much about C or C++. Why not just >> teach it more? > > Does that imply a 'real' syntax for C macros? No, it implies having Cython distinguish between references & values where appropriate. Best, Martin From dagss at student.matnat.uio.no Sat Mar 22 11:40:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 22 Mar 2008 11:40:12 +0100 (CET) Subject: [Cython] My SoC plan Message-ID: <54224.85.165.153.170.1206182412.squirrel@webmail.uio.no> I've written over the last few weeks on my project proposal for a Google Summer of Code (SoC) project. Note that I do not yet have a SoC for this, I'm going to apply in the coming week. Still I post this to the mailing list because a) feedback is always welcome, and b) the plan might be worth discussing in its own right without considering my SoC status (it is very relevant to the last days' macro discussions!) In short, it involves implementing some C++-like features but with a Python twist (limited template support, perhaps function overloading, operator overloading) so that NumPy support can be added through only using Cython language features. (This focus on Cython features rather than simply using a NumPy specific transform was an explicitly requested goal for the project; while the NumPy aim will help keep development narrow and focused.) Note: I unfortunately don't have more time than I need for discussing this the coming days. Therefore I cannot myself participate in any low-level discussion before after the submission deadline (all syntax definitely fall within this category, and in fact most within-spec things). I'll probably only have time to participate in discussions on very high-level stuff and the project plan itself, as well as clear up things that readers found unclear. The project plan is linked to below, and it links to my relevant writings all over the wiki. If you feel like it you're more than welcome to edit the plan, I'll use revision history to see who added what and why. (It goes without saying that the drafts linked to is freely editable, they can't really be considered "mine", in fact I'd be very happy if I didn't turn up as last editor myself everywhere). http://wiki.cython.org/DagSverreSeljebotn/soc/details (The application itself that I plan to submit can also be found here: http://wiki.cython.org/DagSverreSeljebotn/soc but is probably less relevant to you people than the above). Dag Sverre From gfurnish at indirectproof.net Sat Mar 22 13:31:52 2008 From: gfurnish at indirectproof.net (Gary Furnish) Date: Sat, 22 Mar 2008 06:31:52 -0600 Subject: [Cython] My SoC plan In-Reply-To: <54224.85.165.153.170.1206182412.squirrel@webmail.uio.no> References: <54224.85.165.153.170.1206182412.squirrel@webmail.uio.no> Message-ID: <706850310803220531o56c35a5bq184e9ff3b1001e5f@mail.gmail.com> >From skimming this, it looks pretty good, although I worry that some of the time estimates may be overly optimistic. I'd strongly consider figuring out which of these you consider most important and concentrating on doing one or two well and then doing the rest if time allows. I also strongly encourage you to drop by the Sage IRC channel at #sage on irc.freenode.net if your interested in discussing it in more detail (and I also note that to a very large extent Sage serves as a very strong regression test Cython). On Sat, Mar 22, 2008 at 4:40 AM, Dag Sverre Seljebotn wrote: > I've written over the last few weeks on my project proposal for a Google > Summer of Code (SoC) project. > > Note that I do not yet have a SoC for this, I'm going to apply in the > coming week. Still I post this to the mailing list because a) feedback is > always welcome, and b) the plan might be worth discussing in its own right > without considering my SoC status (it is very relevant to the last days' > macro discussions!) > > In short, it involves implementing some C++-like features but with a > Python twist (limited template support, perhaps function overloading, > operator overloading) so that NumPy support can be added through only > using Cython language features. (This focus on Cython features rather than > simply using a NumPy specific transform was an explicitly requested goal > for the project; while the NumPy aim will help keep development narrow and > focused.) > > Note: I unfortunately don't have more time than I need for discussing this > the coming days. Therefore I cannot myself participate in any low-level > discussion before after the submission deadline (all syntax definitely > fall within this category, and in fact most within-spec things). I'll > probably only have time to participate in discussions on very high-level > stuff and the project plan itself, as well as clear up things that readers > found unclear. > > The project plan is linked to below, and it links to my relevant writings > all over the wiki. If you feel like it you're more than welcome to edit > the plan, I'll use revision history to see who added what and why. (It > goes without saying that the drafts linked to is freely editable, they > can't really be considered "mine", in fact I'd be very happy if I didn't > turn up as last editor myself everywhere). > > http://wiki.cython.org/DagSverreSeljebotn/soc/details > > (The application itself that I plan to submit can also be found here: > http://wiki.cython.org/DagSverreSeljebotn/soc > but is probably less relevant to you people than the above). > > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From martin at martincmartin.com Sat Mar 22 15:28:44 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 22 Mar 2008 10:28:44 -0400 Subject: [Cython] What's the difference between the different repositories? Message-ID: <47E5179C.2090805@martincmartin.com> Hi, There are four repositories in mercurial. What's the difference between them? If I want to mock up some changes and possibly submit a patch, which one should I use? Best, Martin From dagss at student.matnat.uio.no Sat Mar 22 15:52:11 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 22 Mar 2008 15:52:11 +0100 (CET) Subject: [Cython] My SoC plan In-Reply-To: <706850310803220531o56c35a5bq184e9ff3b1001e5f@mail.gmail.com> References: <54224.85.165.153.170.1206182412.squirrel@webmail.uio.no> <706850310803220531o56c35a5bq184e9ff3b1001e5f@mail.gmail.com> Message-ID: <55884.85.165.153.170.1206197531.squirrel@webmail.uio.no> Thanks for helpful feedback! Update: > From skimming this, it looks pretty good, although I worry that some > of the time estimates may be overly optimistic. I'd strongly consider > figuring out which of these you consider most important and > concentrating on doing one or two well and then doing the rest if time > allows. I ended up hinting that the phase seperation will be a joint effort (it really must be, and at least Robert and Stefan has voiced approval of getting this done) and should be done within the end of dev1 (which I hope to attend) or I will go for the backup plan (Of course I'll get this somehow confirmed by Robert before I deliver it - Robert?) Then I made the performance optimization part an optional bonus. The significant part is then really only about parametrized types, overloading and function templates, which should be possible to aim for I feel. Though it might still look much, I don't know. This is really a sacrifice of the best possible NumPy support to the advantage of generic Cython development. The only way to be positively certain of having excellent NumPy support within the timeframe is plan B, but I've got signals that NumPy-only code isn't what is wanted. > I also strongly encourage you to drop by the Sage IRC channel > at #sage on irc.freenode.net if your interested in discussing it in > more detail I would, but I just don't have time (as in, I don't have time even if it breaks my SoC). Also note that I will primarily apply through the Python Foundation. Dag Sverre From robertwb at math.washington.edu Sat Mar 22 18:51:46 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 22 Mar 2008 10:51:46 -0700 Subject: [Cython] What's the difference between the different repositories? In-Reply-To: <47E5179C.2090805@martincmartin.com> References: <47E5179C.2090805@martincmartin.com> Message-ID: There are currently 2 repositories in Cython--the "top level" package one, and the actual Cython source. These should probably be merged. For each of these two, there is a -dev branch that is the development repository, and the non-dev one that is always synced with the current release. If you're working on the acutal code, you should patch against cython- dev. - Robert On Mar 22, 2008, at 7:28 AM, Martin C. Martin wrote: > Hi, > > There are four repositories in mercurial. What's the difference > between > them? If I want to mock up some changes and possibly submit a patch, > which one should I use? > > Best, > Martin > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sat Mar 22 19:24:46 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 22 Mar 2008 11:24:46 -0700 Subject: [Cython] Fast matrices over GF(p) In-Reply-To: References: <20080320095215.b309f8fb.simon@arrowtheory.com> Message-ID: On Mar 21, 2008, at 3:40 PM, Simon King wrote: > Dear Simon, dear Robert, > >> Use these functions instead: >> >> cdef extern from "Python.h": >> >> object PyString_FromStringAndSize(char *s, Py_ssize_t len) >> char* PyString_AsString(object string) > > Again, thank you for your valuable help! > > As part of a project on the computation of cohomology rings (in > terms of > minimal generators of a graded commutative algebra and a minimal > set of > algebraic relations between them) for finite p-groups, i wrote a > Pyrex/Cython class "MTX" wrapping Ringe's MeatAxe matrices. > > Disadvantage of MTX: > It is only for small fields (order <256). But i don't need more. > > Advantage: > In some tests (dense matrices over GF(7), size 200x200), addition of > matrices, multiplication by scalars, inversion and various basic > operations (e.g. transposition) turned out to be faster with MTX > than with > the matrices used by Sage (addition and inversion only slightly, > but the > rest by factors of more than 100). > > And, with your help, pickling/unpickling is faster as well, namely > by a > factor of >200 compared with Sage-matrices. > > Do you know another piece of free software that is good in dealing > with > matrices over small fields? Perhaps there is some further improvement > possible, specifically if the matrices are of large size (i need > kernels > of matrices with >1000 rows and columns)? Any hint is welcome! Sounds very interesting! Sage has been mostly optimized to deal with very large matrices, and GF(p) for p barely fitting into a word (to facilitate linear algebra over Z and Q). It sounds like your code does much better for small p. Do you mind if I forward your email onto sage-devel? - Robert From stefan_ml at behnel.de Sun Mar 23 09:25:22 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 23 Mar 2008 09:25:22 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: <47E400DC.1080402@martincmartin.com> References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> Message-ID: <47E613F2.4050303@behnel.de> Hi, Martin C. Martin wrote: > Stefan Behnel wrote: >> Well, in most cases macros are rather short code snippets, not function >> replacing code blocks. So I don't care about their symbols. > > That's surprising to me. Don't you think you'd want to step through > even short snippets in a debugger? To have them show up as a separate > item in a profile, rather than spread around all the places that call > them? Same with memory leaks or corruption? Again: there are valid use cases for macros. They are mostly for speed, not for improved debugability. >>> In your example: >>> >>> #define _fqtypename(o) (((PyTypeObject*)o)->ob_type->tp_name) >>> >>> _fqtypename(a + b) expands to: >>> >>> (((PyTypeObject*)a + b)->ob_type->tp_name) >> >> And that's definitely not a valid use case of _fqtypename(). > > Why is that? Isn't pointer arithmetic a valid use case? How do you do pointer arithmetic on Python objects? Remember the declaration: cdef inline from MACROS: cdef char* _fqtypename(object t) >>> If you want an inline function, why not just declare it inline? >> >> Because for small code snippets that exist solely for performance >> reasons, the overhead of a function defined in Cython may be too high. > > But if it's declared inline, there's no function call overhead at > runtime. Unless you're in a debug build, in which case you'd rather > have the overhead to help track down bugs, leaks, etc. Do you mean > compile time overhead? Or something else? There currently is a ref-count overhead. When that is removed, I wouldn't mind seeing inline functions as a macro replacement in many cases. But I'm pretty sure even then I see a couple of cases where I'd use macros. >>> The use cases discussed on the list recently were to get around the fact >>> that Cython doesn't know all that much about C or C++. Why not just >>> teach it more? >> >> Does that imply a 'real' syntax for C macros? > > No, it implies having Cython distinguish between references & values > where appropriate. Which, admittedly, is the case William brought up, but it's not the only use case for macros that I am talking about here. I would prefer keeping that a separate topic. Stefan From dagss at student.matnat.uio.no Sun Mar 23 10:06:32 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 23 Mar 2008 10:06:32 +0100 (CET) Subject: [Cython] Syntax for writing C macros in a Cython .pxd] Message-ID: <58176.85.165.153.170.1206263192.squirrel@webmail.uio.no> > How do you do pointer arithmetic on Python objects? Remember the If you have Python objects stored in an array? Your macro casts the parameter to a pointer types, hence, you can do pointer arithmetic. (Perhaps you meant that it won't arise in practice, but that's different.) > There currently is a ref-count overhead. When that is removed, I wouldn't > mind > seeing inline functions as a macro replacement in many cases. But I'm > pretty > sure even then I see a couple of cases where I'd use macros. Short-term, macros are very useful. Long-term, I don't think they will be necesarry (as C++ demonstrates very well that they are not necessarry in principle). If the period until the long-term goals are several years (which I think it might be) then there is a case for including macro support in pxd files even if it is not "ideal", I think. How about this syntax instead for "purists": @cython.inline def mymacro(x): return raw_c("((MyType*)x)->field") Ie allow raw C in any function using some syntax (could be a "with-block" as well), the same way C compilers can allow native assembler code as a propriotary extension. If a function consists of only a single raw_c instruction, we turn it into a macro instead, automatically paranthezing all arguments...something like that. Dag Sverre From stefan_ml at behnel.de Sun Mar 23 10:48:42 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 23 Mar 2008 10:48:42 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: <58163.85.165.153.170.1206263155.squirrel@webmail.uio.no> References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> <58163.85.165.153.170.1206263155.squirrel@webmail.uio.no> Message-ID: <47E6277A.8050400@behnel.de> Hi, Dag Sverre Seljebotn wrote: >> How do you do pointer arithmetic on Python objects? Remember the > > If you have Python objects stored in an array? Your macro casts the > parameter to a pointer types, hence, you can do pointer arithmetic. Show me an example of valid Cython code that uses this function cdef char* _fqtypename(object t) together with pointer arithmetic in a way that breaks its macro definition. >> There currently is a ref-count overhead. When that is removed, I wouldn't >> mind >> seeing inline functions as a macro replacement in many cases. But I'm >> pretty sure even then I see a couple of cases where I'd use macros. > > Short-term, macros are very useful. Long-term, I don't think they will be > necesarry (as C++ demonstrates very well that they are not necessarry in > principle). > > If the period until the long-term goals are several years (which I think > it might be) then there is a case for including macro support in pxd files > even if it is not "ideal", I think. > > How about this syntax instead for "purists": > > @cython.inline > def mymacro(x): > return raw_c("((MyType*)x)->field") That would be "cython.raw_c()". That's a syntax that allows basically any interaction with C code. However, I would prefer a syntax that stops at the function level so that Cython sees what goes in and out. That's the kind of macro I would want to have support for. It would also support this kind of macro code: #ifdef __GNUC__ /* Test for GCC > 2.95 */ #if __GNUC__ > 2 || (__GNUC__ == 2 && (__GNUC_MINOR__ > 95)) #define unlikely_condition(x) __builtin_expect((x), 0) #else /* __GNUC__ > 2 ... */ #define unlikely_condition(x) (x) #endif /* __GNUC__ > 2 ... */ #else /* __GNUC__ */ #define unlikely_condition(x) (x) #endif /* __GNUC__ */ I don't think you can do much better, even with the huge overhead of adding a Cython specific syntax. BTW, what is the "cython.inline" needed for in your example? Or is that just meant as a replacement for the current "inline" keyword? > Ie allow raw C in any function using some syntax (could be a "with-block" > as well) I don't think a "with" block works here. I wouldn't want to allow real C syntax in Cython. A string should be the highest level of raw-C integration. > the same way C compilers can allow native assembler code as a > propriotary extension. If a function consists of only a single raw_c > instruction, we turn it into a macro instead, automatically paranthezing > all arguments...something like that. Too much magic for a simple purpose? Stefan From ndbecker2 at gmail.com Sun Mar 23 12:13:08 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 23 Mar 2008 07:13:08 -0400 Subject: [Cython] Syntax for writing C macros in a Cython .pxd References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> Message-ID: Stefan Behnel wrote: > Hi, > > Martin C. Martin wrote: >> Stefan Behnel wrote: >>> Well, in most cases macros are rather short code snippets, not function >>> replacing code blocks. So I don't care about their symbols. >> >> That's surprising to me. Don't you think you'd want to step through >> even short snippets in a debugger? To have them show up as a separate >> item in a profile, rather than spread around all the places that call >> them? Same with memory leaks or corruption? > > Again: there are valid use cases for macros. They are mostly for speed, > not for improved debugability. > I don't see this argument. There is no speed advantage of macros over inline, providing your compiler supports inline. Maybe you're talking about C compilers that don't support inline? From dagss at student.matnat.uio.no Sun Mar 23 12:26:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 23 Mar 2008 12:26:00 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd Message-ID: <3289119981.211735@smtp.netcom.no> yes, cython.inline was pointless in this context and my suggestion was too magic. Still, until one is able to deal with it automatically, perhaps a decorator or keyword saying that "don't bother to refcount my parameters, I'll take the consequences" that can be used on an inline function would fix some cases? (It is certainly not more "dangerous" than allowing raw C macros). But I'm really only brainstorming here, have nothing against your suggestion (just hope it will become obsolete in time -- DEF might be extended to access GCC defines and so on, but it is not something that should be a priority...). Dag Sverre Seljebotn -----Original Message----- From: Stefan Behnel Date: Sunday, Mar 23, 2008 10:48 am Subject: Re: [Cython] Syntax for writing C macros in a Cython .pxd To: Dag Sverre Seljebotn CC: Cython-dev Hi, Dag Sverre Seljebotn wrote: > How do you do pointer arithmetic on Python objects? Remember the If you have Python objects stored in an array? Your macro casts the parameter to a pointer types, hence, you can do pointer arithmetic. Show me an example of valid Cython code that uses this function cdef char* _fqtypename(object t) together with pointer arithmetic in a way that breaks its macro definition. >> There currently is a ref-count overhead. When that is removed, I wouldn't > mind > seeing inline functions as a macro replacement in many cases. But I'm > pretty sure even then I see a couple of cases where I'd use macros. Short-term, macros are very useful. Long-term, I don't think they will be necesarry (as C++ demonstrates very well that they are not necessarry in principle). If the period until the long-term goals are several years (which I think it might be) then there is a case for including macro support in pxd files even if it is not "ideal", I think. How about this syntax instead for "purists": @cython.inline def mymacro(x): return raw_c("((MyType*)x)->field") That would be "cython.raw_c()". That's a syntax that allows basically any interaction with C code. However, I would prefer a syntax that stops at the function level so that Cython sees what goes in and out. That's the kind of macro I would want to have support for. It would also support this kind of macro code: #ifdef __GNUC__ /* Test for GCC > 2.95 */ #if __GNUC__ > 2 || (__GNUC__ == 2 && (__GNUC_MINOR__ > 95)) #define unlikely_condition(x) __builtin_expect((x), 0) #else /* __GNUC__ > 2 ... */ #define unlikely_condition(x) (x) #endif /* __GNUC__ > 2 ... */ #else /* __GNUC__ */ #define unlikely_condition(x) (x) #endif /* __GNUC__ */ I don't think you can do much better, even with the huge overhead of adding a Cython specific syntax. BTW, what is the "cython.inline" needed for in your example? Or is that just meant as a replacement for the current "inline" keyword? > Ie allow raw C in any function using some syntax (could be a "with-block" as well) I don't think a "with" block works here. I wouldn't want to allow real C syntax in Cython. A string should be the highest level of raw-C integration. > the same way C compilers can allow native assembler code as a propriotary extension. If a function consists of only a single raw_c instruction, we turn it into a macro instead, automatically paranthezing all arguments...something like that. Too much magic for a simple purpose? Stefan From stefan_ml at behnel.de Sun Mar 23 18:12:58 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 23 Mar 2008 18:12:58 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> Message-ID: <47E68F9A.7010204@behnel.de> Hi, Neal Becker wrote: > Stefan Behnel wrote: >> Martin C. Martin wrote: >>> Stefan Behnel wrote: >>>> Well, in most cases macros are rather short code snippets, not function >>>> replacing code blocks. So I don't care about their symbols. >>> That's surprising to me. Don't you think you'd want to step through >>> even short snippets in a debugger? To have them show up as a separate >>> item in a profile, rather than spread around all the places that call >>> them? Same with memory leaks or corruption? >> Again: there are valid use cases for macros. They are mostly for speed, >> not for improved debugability. >> > > I don't see this argument. There is no speed advantage of macros over > inline, providing your compiler supports inline. Maybe you're talking > about C compilers that don't support inline? Maybe a different example would help. How would you write this in Cython: // --- FILE defs.h #define PY_NEW(T) \ (((PyTypeObject*)(T))->tp_new( \ (PyTypeObject*)(T), __pyx_empty_tuple, NULL)) # --- FILE some.pyx cdef extern from "defs.h": # macro call to 't->tp_new()' for fast instantiation cdef DTD NEW_DTD "PY_NEW" (object t) dtd = NEW_DTD(DTD) Stefan From robertwb at math.washington.edu Mon Mar 24 18:45:29 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 24 Mar 2008 10:45:29 -0700 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: <47E68F9A.7010204@behnel.de> References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> <47E68F9A.7010204@behnel.de> Message-ID: On Mar 23, 2008, at 10:12 AM, Stefan Behnel wrote: > Hi, > > Neal Becker wrote: >> Stefan Behnel wrote: >>> Martin C. Martin wrote: >>>> Stefan Behnel wrote: >>>>> Well, in most cases macros are rather short code snippets, not >>>>> function >>>>> replacing code blocks. So I don't care about their symbols. >>>> That's surprising to me. Don't you think you'd want to step >>>> through >>>> even short snippets in a debugger? To have them show up as a >>>> separate >>>> item in a profile, rather than spread around all the places that >>>> call >>>> them? Same with memory leaks or corruption? >>> Again: there are valid use cases for macros. They are mostly for >>> speed, >>> not for improved debugability. >>> >> >> I don't see this argument. There is no speed advantage of macros >> over >> inline, providing your compiler supports inline. Maybe you're >> talking >> about C compilers that don't support inline? > > Maybe a different example would help. How would you write this in > Cython: > > // --- FILE defs.h > #define PY_NEW(T) \ > (((PyTypeObject*)(T))->tp_new( \ > (PyTypeObject*)(T), __pyx_empty_tuple, NULL)) One would declare the type object as having a function pointer field tp_new with the right signature. (In fact, when we support builtin- types, this should be available by default at T.tp_new((), NULL)). > # --- FILE some.pyx > cdef extern from "defs.h": > # macro call to 't->tp_new()' for fast instantiation > cdef DTD NEW_DTD "PY_NEW" (object t) > > dtd = NEW_DTD(DTD) This can only be done because one is dealing with two separate namespaces. I do think there are valid uses for C macros/arbitrary C in some Cython code. I also hope that the need for such special use cases goes down as Cython improves. But if one wants to muck around with stuff like this I think one should write an actual .c or .h file and include stuff from there, rather than provide an easy (and IMHO ugly) way to embed raw C code strings in Cython files. - Robert From robertwb at math.washington.edu Mon Mar 24 18:55:40 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 24 Mar 2008 10:55:40 -0700 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> <47E68F9A.7010204@behnel.de> Message-ID: [somehow this got detached from the original thread...] On Mar 23, 2008, at 2:06 AM, Dag Sverre Seljebotn wrote: > >> How do you do pointer arithmetic on Python objects? Remember the > > If you have Python objects stored in an array? Your macro casts the > parameter to a pointer types, hence, you can do pointer arithmetic. > > (Perhaps you meant that it won't arise in practice, but that's > different.) One can already do this in Cython. or >> There currently is a ref-count overhead. When that is removed, I >> wouldn't mind >> seeing inline functions as a macro replacement in many cases. But >> I'm pretty >> sure even then I see a couple of cases where I'd use macros. Removing the ref-count overhead should be easy to do. Perhaps I'll do it this afternoon. > Short-term, macros are very useful. Long-term, I don't think they > will be > necesarry (as C++ demonstrates very well that they are not > necessarry in > principle). > > If the period until the long-term goals are several years (which I > think > it might be) then there is a case for including macro support in > pxd files > even if it is not "ideal", I think. > > How about this syntax instead for "purists": > > @cython.inline > def mymacro(x): > return raw_c("((MyType*)x)->field") > > Ie allow raw C in any function using some syntax (could be a "with- > block" > as well), the same way C compilers can allow native assembler code > as a > propriotary extension. If a function consists of only a single raw_c > instruction, we turn it into a macro instead, automatically > paranthezing > all arguments...something like that. This avoid the whole issue of name mangling... and to automatically parenthesis arguments we would have to (essentially) write a C parser. - Robert From robertwb at math.washington.edu Mon Mar 24 19:00:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 24 Mar 2008 11:00:33 -0700 Subject: [Cython] Fast matrices over GF(p) In-Reply-To: References: <20080320095215.b309f8fb.simon@arrowtheory.com> Message-ID: <35024ECB-A2B3-41AA-8690-2843859CC7F9@math.washington.edu> On Mar 22, 2008, at 1:08 PM, Simon King wrote: > Dear Robert, > > On Sat, 22 Mar 2008, Robert Bradshaw wrote: >> >> Sounds very interesting! Sage has been mostly optimized to deal >> with very >> large matrices, and GF(p) for p barely fitting into a word (to >> facilitate >> linear algebra over Z and Q). It sounds like your code does much >> better for >> small p. Do you mind if I forward your email onto sage-devel? > > I am on sage-devel, myself. So i wouldn't mind. > > But it is of course not *my* code. MeatAxe can be found at > http://www.math.rwth-aachen.de/~MTX/. My contribution is just a > wrapper, i > didn't improve the arithmetics. Actually i use a rather old version of > C-MeatAxe. This is because my boss (David Green) wrote C-code for > computing cohomology, relying on the old C-MeatAxe. Some of us ran into MeatAxe before, but no one had any experience with it or knew anyone who used it, so nothing ever happened of it. I was thinking since you actually use it (and have a Cython wrapper) you would know what it's strengths/weaknesses are and whether or not it would be worth including in Sage. - Robert From stefan_ml at behnel.de Mon Mar 24 20:37:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 24 Mar 2008 20:37:36 +0100 Subject: [Cython] Syntax for writing C macros in a Cython .pxd In-Reply-To: References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> <47E68F9A.7010204@behnel.de> Message-ID: <47E80300.1080502@behnel.de> Hi, Robert Bradshaw wrote: >> // --- FILE defs.h >> #define PY_NEW(T) \ >> (((PyTypeObject*)(T))->tp_new( \ >> (PyTypeObject*)(T), __pyx_empty_tuple, NULL)) > > One would declare the type object as having a function pointer field > tp_new with the right signature. (In fact, when we support > builtin-types, this should be available by default at T.tp_new((), NULL)). Ok, I like that. >> # --- FILE some.pyx >> cdef extern from "defs.h": >> # macro call to 't->tp_new()' for fast instantiation >> cdef DTD NEW_DTD "PY_NEW" (object t) >> >> dtd = NEW_DTD(DTD) > > This can only be done because one is dealing with two separate namespaces. Which was part of my proposal. > I do think there are valid uses for C macros/arbitrary C in some Cython > code. I also hope that the need for such special use cases goes down as > Cython improves. But if one wants to muck around with stuff like this I > think one should write an actual .c or .h file and include stuff from > there, rather than provide an easy (and IMHO ugly) way to embed raw C > code strings in Cython files. Admittedly, that's only one more file to keep track of, and you might need it anyway... Stefan From gfurnish at gfurnish.net Tue Mar 25 19:17:06 2008 From: gfurnish at gfurnish.net (Gary Furnish) Date: Tue, 25 Mar 2008 12:17:06 -0600 Subject: [Cython] Cython circular cdef import patch Message-ID: <8f8f8530803251117x4efeeafch7243af2b9b302f2e@mail.gmail.com> This patch adds extra logic to code generation to sort dependencies to guarantee that C code is generated in the right order for circular cdef imports. It does NOT solve python circular import issues. a.pyx: cimport b b.pyx: cimport a Is thus legal, but you can not reference a or b in the global namespace of b.pyx or a.pyx (such as to instantiate a class). This patch also modifies cython to output the exact line in the C code where an exception was thrown in addition to the currently displayed pyx file and line. This enables significantly faster developmental debugging. Finally it splits module initialization into two phases: one that initiates types and handles imports, and another that executes python commands at the global namespace level. This will be more useful as Cython starts to assume more advanced optimization and code generation features The patch is available at: http://trac.sagemath.org/sage_trac/ticket/2655(the third attachment only) and is based against 0.9.6.12, although I can rebase if needed. I am hoping this can be merged into Cython. --Gary Furnish -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080325/55c89ee5/attachment.htm From beach at verinet.com Tue Mar 25 21:05:14 2008 From: beach at verinet.com (David J. C. Beach) Date: Tue, 25 Mar 2008 14:05:14 -0600 Subject: [Cython] Extension Type Question Message-ID: <21145031-8630-4FEF-B3A1-33F5DB5AA51E@verinet.com> Cython developers, I knew about Pyrex a few years ago but only recently discovered Cython. I'm very glad to see that there is an active community involved in adding these features to Pyrex. I've been trying to cut my teeth on Pyrex/Cython by writing a few extension types, and have encountered some confusion about whether the "self" argument needs to be given an explicit type. This short example illustrates my confusion: cdef class IntBox: cdef int _data def __init__(self, int value=0): self._data = value def __add__(IntBox self, IntBox other not None): cdef IntBox result = IntBox(self._data + other._data) return result def __repr__(self): return str(self._data) In this code, the __init__() and __repr__() methods have no problem accessing self._data. However, if I exclude the "IntBox" type before the self argument in the __add__() method, the code doesn't run. (It fails at runtime by saying that it is unable to access the member "_data". (Presumably, this is because Cython compiled it as a python dictionary attribute lookup, instead of using the cdef'd storage.) My questions are: - Why am I seeing this behavior? - What is the best practice concerning the "typing" of the self parameter? Should I always write "IntBox self" as the first argument to each method of the IntBox class? (Even though it appears to only be *sometimes* necessary?) Thank you for your help in this confusing matter. And thank you for your development of Cython. David From robertwb at math.washington.edu Tue Mar 25 23:06:40 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Mar 2008 15:06:40 -0700 Subject: [Cython] Extension Type Question In-Reply-To: <21145031-8630-4FEF-B3A1-33F5DB5AA51E@verinet.com> References: <21145031-8630-4FEF-B3A1-33F5DB5AA51E@verinet.com> Message-ID: On Mar 25, 2008, at 1:05 PM, David J. C. Beach wrote: > Cython developers, > > I knew about Pyrex a few years ago but only recently discovered > Cython. I'm very glad to see that there is an active community > involved in adding these features to Pyrex. Thanks. > I've been trying to cut my teeth on Pyrex/Cython by writing a few > extension types, and have encountered some confusion about whether the > "self" argument needs to be given an explicit type. This short > example illustrates my confusion: > > cdef class IntBox: > > cdef int _data > > def __init__(self, int value=0): > self._data = value > > def __add__(IntBox self, IntBox other not None): > cdef IntBox result = IntBox(self._data + other._data) > return result > > def __repr__(self): > return str(self._data) > > > In this code, the __init__() and __repr__() methods have no problem > accessing self._data. > > However, if I exclude the "IntBox" type before the self argument in > the __add__() method, the code doesn't run. (It fails at runtime by > saying that it is unable to access the member "_data". (Presumably, > this is because Cython compiled it as a python dictionary attribute > lookup, instead of using the cdef'd storage.) > > My questions are: > > - Why am I seeing this behavior? > - What is the best practice concerning the "typing" of the self > parameter? Should I always write "IntBox self" as the first argument > to each method of the IntBox class? (Even though it appears to only > be *sometimes* necessary?) > > Thank you for your help in this confusing matter. See the "arithmatic operators" at http://www.cosc.canterbury.ac.nz/ greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html . This is because of the nature of the Python/C API, and though there's talk of emulating the __add__/__radd__ paradigm of Python it's unclear how one would do so. Self is always the right type otherwise. - Robert From mistobaan at gmail.com Wed Mar 26 21:04:36 2008 From: mistobaan at gmail.com (Fabrizio Milo aka misto) Date: Wed, 26 Mar 2008 21:04:36 +0100 Subject: [Cython] Summer Of Code Message-ID: Hi to everyone, I am interested in working on cython during the summer of Code. The objective would be to improve the support of Cython for Python 2.5 and 3000. I have some ideas on how to do that but not a detailed plan. I would like to talk with someone of the possible mentor in order to draft a plan of the steps to accomplish. I logged many time on the #sage channel but no one shows up. Looking forward for a response :D Thanks Fabrizio -------------------------- Luck favors the prepared mind. (Pasteur) From robertwb at math.washington.edu Wed Mar 26 21:55:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 26 Mar 2008 13:55:20 -0700 Subject: [Cython] Summer Of Code In-Reply-To: References: Message-ID: On Mar 26, 2008, at 1:04 PM, Fabrizio Milo aka misto wrote: > Hi to everyone, > > I am interested in working on cython during the summer of Code. > The objective would be to improve the support of Cython for Python > 2.5 and 3000. > I have some ideas on how to do that but not a detailed plan. That would be great. I assume you've already take a look at http:// wiki.cython.org/enhancements/ (especially the "Full Python Support" and 3000 sections). If not that would be a great place to start. > I would like to talk with someone of the possible mentor in order to > draft a plan of the steps to accomplish. > I logged many time on the #sage channel but no one shows up. I could be a possible mentor, as could several other people in the Sage group. The official sage irc channel is sage-devel, which is pretty active (though I'm not on there at the moment, maybe I should hop on). > Looking forward for a response :D Look forward to working with you, - Robert From stefan_ml at behnel.de Thu Mar 27 14:31:01 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 27 Mar 2008 14:31:01 +0100 Subject: [Cython] ref-count overhead of function parameters In-Reply-To: References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> <47E68F9A.7010204@behnel.de> Message-ID: <47EBA195.6010000@behnel.de> Hi, Robert Bradshaw wrote: >>> There currently is a ref-count overhead. When that is removed, I >>> wouldn't mind >>> seeing inline functions as a macro replacement in many cases. But >>> I'm pretty >>> sure even then I see a couple of cases where I'd use macros. > > Removing the ref-count overhead should be easy to do. Perhaps I'll do > it this afternoon. In what cases would that be removed? When there is no assignment to a parameter inside the function body? Stefan From stefan_ml at behnel.de Thu Mar 27 18:50:17 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 27 Mar 2008 18:50:17 +0100 Subject: [Cython] PEP 361: Python 2.6/3.0 release schedule In-Reply-To: References: Message-ID: <47EBDE59.5060404@behnel.de> Hi all, looks like the first Python 3 beta could make it before the dev1 get-together. > Executive summary: Python 2.6 and 3.0 finals are planned for September > 3, 2008. > > PEP 361 contains all the gory details; from the PEP: > > Feb 29 2008: Python 2.6a1 and 3.0a3 are released > Apr 02 2008: Python 2.6a2 and 3.0a4 planned > May 07 2008: Python 2.6a3 and 3.0a5 planned > Jun 04 2008: Python 2.6b1 and 3.0b1 planned > Jul 02 2008: Python 2.6b2 and 3.0b2 planned > Aug 06 2008: Python 2.6rc1 and 3.0rc1 planned > Aug 20 2008: Python 2.6rc2 and 3.0rc2 planned > Sep 03 2008: Python 2.6 and 3.0 final > > http://www.python.org/dev/peps/pep-0361/ The beta-1 release, expected for June 2008, is supposed to have a stable C-API, which is the point where it makes sense to think about porting Cython. Stefan From robertwb at math.washington.edu Thu Mar 27 18:54:32 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 27 Mar 2008 10:54:32 -0700 Subject: [Cython] ref-count overhead of function parameters In-Reply-To: <47EBA195.6010000@behnel.de> References: <47E20DBD.5000208@behnel.de> <47E3C352.2060305@martincmartin.com> <47E3FD78.9060803@behnel.de> <47E400DC.1080402@martincmartin.com> <47E613F2.4050303@behnel.de> <47E68F9A.7010204@behnel.de> <47EBA195.6010000@behnel.de> Message-ID: <2698CA79-BC75-4758-9336-1480DE0057DB@math.washington.edu> On Mar 27, 2008, at 6:31 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >>>> There currently is a ref-count overhead. When that is removed, I >>>> wouldn't mind >>>> seeing inline functions as a macro replacement in many cases. But >>>> I'm pretty >>>> sure even then I see a couple of cases where I'd use macros. >> >> Removing the ref-count overhead should be easy to do. Perhaps I'll do >> it this afternoon. > > In what cases would that be removed? When there is no assignment to a > parameter inside the function body? Exactly. And I finished it already. - Robert From simon at arrowtheory.com Thu Mar 27 21:17:50 2008 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 27 Mar 2008 16:17:50 -0400 Subject: [Cython] cimport inside a package Message-ID: <20080327161750.083aecce.simon@arrowtheory.com> == foo.pxd == cdef int foo() == foo.pyx == cdef int foo(): print "hi foo" return 9 == bar.pyx == cimport foo def bar(): print foo.foo() ============= All good so far. If I put the above in a regular python package called zap, and then (from outside of the zap package): $ python Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from zap import bar Traceback (most recent call last): File "", line 1, in File "bar.pyx", line 2, in bar ImportError: No module named foo >>> The C code in bar.c is trying to import module bar. What is the correct cimport invocation in bar.pyx ? I tried some obvious permutations ("from zap cimport bar"), but nothing got past the cython compiler. Simon. From robertwb at math.washington.edu Thu Mar 27 21:25:45 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 27 Mar 2008 13:25:45 -0700 Subject: [Cython] cimport inside a package In-Reply-To: <20080327161750.083aecce.simon@arrowtheory.com> References: <20080327161750.083aecce.simon@arrowtheory.com> Message-ID: <49F941B9-B09C-4B26-8751-974329FE45D2@math.washington.edu> What does your setup.py look like? On Mar 27, 2008, at 1:17 PM, Simon Burton wrote: > == foo.pxd == > > > cdef int foo() > > > == foo.pyx == > > cdef int foo(): > print "hi foo" > return 9 > > > == bar.pyx == > > cimport foo > > def bar(): > print foo.foo() > > ============= > > All good so far. > > If I put the above in a regular python package called zap, > and then (from outside of the zap package): > > $ python > Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12) > [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> from zap import bar > Traceback (most recent call last): > File "", line 1, in > File "bar.pyx", line 2, in bar > ImportError: No module named foo >>>> > > The C code in bar.c is trying to import module bar. > What is the correct cimport invocation in bar.pyx ? > I tried some obvious permutations ("from zap cimport bar"), but > nothing got past > the cython compiler. > > Simon. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From simon at arrowtheory.com Thu Mar 27 21:28:58 2008 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 27 Mar 2008 16:28:58 -0400 Subject: [Cython] cimport inside a package In-Reply-To: <49F941B9-B09C-4B26-8751-974329FE45D2@math.washington.edu> References: <20080327161750.083aecce.simon@arrowtheory.com> <49F941B9-B09C-4B26-8751-974329FE45D2@math.washington.edu> Message-ID: <20080327162858.937d684c.simon@arrowtheory.com> I have a Makefile ( the -I. doesn't seem to help): bar.so: bar.o gcc $(CFLAGS) -shared bar.o -o bar.so bar.o: bar.c gcc $(CFLAGS) -c bar.c -I/usr/include/python2.5 bar.c: bar.pyx cython -I. bar.pyx foo.so: foo.o gcc $(CFLAGS) -shared foo.o -o foo.so foo.o: foo.c gcc $(CFLAGS) -c foo.c -I/usr/include/python2.5 foo.c: foo.pyx cython -I. foo.pyx On Thu, 27 Mar 2008 13:25:45 -0700 Robert Bradshaw wrote: > What does your setup.py look like? > > On Mar 27, 2008, at 1:17 PM, Simon Burton wrote: > > > == foo.pxd == > > > > > > cdef int foo() > > > > > > == foo.pyx == > > > > cdef int foo(): > > print "hi foo" > > return 9 > > > > > > == bar.pyx == > > > > cimport foo > > > > def bar(): > > print foo.foo() > > > > ============= > > > > All good so far. > > > > If I put the above in a regular python package called zap, > > and then (from outside of the zap package): > > > > $ python > > Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12) > > [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> from zap import bar > > Traceback (most recent call last): > > File "", line 1, in > > File "bar.pyx", line 2, in bar > > ImportError: No module named foo > >>>> > > > > The C code in bar.c is trying to import module bar. > > What is the correct cimport invocation in bar.pyx ? > > I tried some obvious permutations ("from zap cimport bar"), but > > nothing got past > > the cython compiler. > > > > Simon. > > > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > From simon at arrowtheory.com Thu Mar 27 22:16:13 2008 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 27 Mar 2008 17:16:13 -0400 Subject: [Cython] module initialization Message-ID: <20080327171613.3d80aa2d.simon@arrowtheory.com> I have a module mux.pyx where the initialization is being called more than once (the initmux c function). Does anyone know why this might be happening ? Is it something to do with tricky cimport usage (python would never initialize an extension module twice..) ? Here is my gdb session: Breakpoint 1, initmux () at mux.c:6829 6829 __pyx_m = Py_InitModule4("mux", __pyx_methods, 0, 0, PYTHON_API_VERSION); (gdb) bt #0 initmux () at mux.c:6829 #1 0x080e1a34 in _PyImport_LoadDynamicModule () #2 0x080df6d5 in ?? () #3 0xbfda8807 in ?? () #4 0xbfda7797 in ?? () #5 0x081c68c8 in ?? () #6 0x08089c69 in PyObject_GetAttrString () #7 0x080df98b in ?? () #8 0x00000003 in ?? () #9 0x00000000 in ?? () (gdb) c Continuing. [New Thread -1420035184 (LWP 11098)] [New Thread -1420563568 (LWP 11099)] [New Thread -1421091952 (LWP 11100)] [New Thread -1421620336 (LWP 11101)] [New Thread -1422148720 (LWP 11102)] [New Thread -1422677104 (LWP 11103)] [New Thread -1424778352 (LWP 11104)] Breakpoint 1, initmux () at mux.c:6829 6829 __pyx_m = Py_InitModule4("mux", __pyx_methods, 0, 0, PYTHON_API_VERSION); (gdb) bt #0 initmux () at mux.c:6829 #1 0x080e1a34 in _PyImport_LoadDynamicModule () #2 0x080df6d5 in ?? () #3 0xbfdabaf7 in ?? () #4 0xbfdaaa87 in ?? () #5 0x082aafa8 in ?? () #6 0x080872b0 in PyDict_GetItemString () #7 0x080df98b in ?? () #8 0x00000003 in ?? () #9 0x00000000 in ?? () (gdb) Hmm, maybe it's thread related ? yikes. Simon. From simon at arrowtheory.com Fri Mar 28 16:38:24 2008 From: simon at arrowtheory.com (Simon Burton) Date: Fri, 28 Mar 2008 11:38:24 -0400 Subject: [Cython] module initialization In-Reply-To: <716F357B-67C4-4C1D-B93D-5A484BE9912D@math.washington.edu> References: <20080327171613.3d80aa2d.simon@arrowtheory.com> <716F357B-67C4-4C1D-B93D-5A484BE9912D@math.washington.edu> Message-ID: <20080328113824.61276100.simon@arrowtheory.com> On Thu, 27 Mar 2008 14:24:46 -0700 Robert Bradshaw wrote: > > More likely to be thread-related--IIRC Cython calls the Python/C API > to do its imports (though I haven't looked into it enough to > completely absolve Cython of any potential guilt...) > > - Robert Yes. This is the solution I found to create a module singleton: cdef void *_mux mux = None def get_mux(): global _mux, mux if _mux == NULL: mux = Mux() # create singleton _mux = mux elif mux is None: mux = _mux return mux From robertwb at math.washington.edu Sat Mar 29 00:34:55 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 28 Mar 2008 16:34:55 -0700 Subject: [Cython] Repository merge Message-ID: <0F9A0B8E-856B-4EBC-96A8-CBC2D8D5FF24@math.washington.edu> Currently, for purely historical reasons, there are two repositories for Cython, which is both annoying and confusing. I am going to merge the two in a way that preserves the history, but this may make bundles against the old repository unapplyable to the new one. (Patches should still be fine) I figure, however, this is better done sooner than later. So, if anyone has significant work in the source tree they would like to see merged in before I make this change, now is the time to speak up. - Robert From robertwb at math.washington.edu Sat Mar 29 00:51:50 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 28 Mar 2008 16:51:50 -0700 Subject: [Cython] from arrays to python objects In-Reply-To: <79b79e730803200357l782d3fe3oc9aee1071dc6da8c@mail.gmail.com> References: <79b79e730803200357l782d3fe3oc9aee1071dc6da8c@mail.gmail.com> Message-ID: <1C4DE956-A840-4A60-8C2B-D245F065A28D@math.washington.edu> On Mar 20, 2008, at 3:57 AM, Francesco Guerrieri wrote: > Hi, it's always me :-) > I've been looking for a standard way for passing from a c array to a > python object, eg a list. > In the primes example, this is solved by creating both the array > (with a fixed size, but now I know how to handle it :-) ) and the > list, > and then appending the relevant prime numbers to the list. > > Is this the idiomatic approach? > I have looked for something similar in the sage codebase but > didn't find something relevant. > > So if I have an array, do I have always to do something like this? > > for i from 0 <= i < size_of_array: > my_list.append(array[i]) > return my_list The numpy suggestion is good, but I could also suggest [array[i] for i from 0 <= i < size] which should both be faster and easier to read. - Robert From robertwb at math.washington.edu Sat Mar 29 01:57:06 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 28 Mar 2008 17:57:06 -0700 Subject: [Cython] My SoC plan In-Reply-To: <54224.85.165.153.170.1206182412.squirrel@webmail.uio.no> References: <54224.85.165.153.170.1206182412.squirrel@webmail.uio.no> Message-ID: <3028F551-D105-4E41-A6FC-BA530A5D4F62@math.washington.edu> On Mar 22, 2008, at 3:40 AM, Dag Sverre Seljebotn wrote: > I've written over the last few weeks on my project proposal for a > Google > Summer of Code (SoC) project. > > Note that I do not yet have a SoC for this, I'm going to apply in the > coming week. Still I post this to the mailing list because a) > feedback is > always welcome, and b) the plan might be worth discussing in its > own right > without considering my SoC status (it is very relevant to the last > days' > macro discussions!) > > In short, it involves implementing some C++-like features but with a > Python twist (limited template support, perhaps function overloading, > operator overloading) so that NumPy support can be added through only > using Cython language features. (This focus on Cython features > rather than > simply using a NumPy specific transform was an explicitly requested > goal > for the project; while the NumPy aim will help keep development > narrow and > focused.) > > Note: I unfortunately don't have more time than I need for > discussing this > the coming days. Therefore I cannot myself participate in any low- > level > discussion before after the submission deadline (all syntax definitely > fall within this category, and in fact most within-spec things). I'll > probably only have time to participate in discussions on very high- > level > stuff and the project plan itself, as well as clear up things that > readers > found unclear. > > The project plan is linked to below, and it links to my relevant > writings > all over the wiki. If you feel like it you're more than welcome to > edit > the plan, I'll use revision history to see who added what and why. (It > goes without saying that the drafts linked to is freely editable, they > can't really be considered "mine", in fact I'd be very happy if I > didn't > turn up as last editor myself everywhere). > > http://wiki.cython.org/DagSverreSeljebotn/soc/details > > (The application itself that I plan to submit can also be found here: > http://wiki.cython.org/DagSverreSeljebotn/soc > but is probably less relevant to you people than the above). I have finished reading all the proposals and thinking about it and I have to say I think this is going to be a great project. Can you put up a formal proposal on the google site for PSF (I haven't seen anything there yet). Your "Developing Cython for Easy NumPy Integration" is a great title, and the bulk of the text is already written. - Robert From robertwb at math.washington.edu Sat Mar 29 19:51:02 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 29 Mar 2008 11:51:02 -0700 Subject: [Cython] [Bug 208262] [NEW] Invalid typedefs on Windows In-Reply-To: <20080328142109.24766.44349.malonedeb@gandwana.canonical.com> References: <20080328142109.24766.44349.malonedeb@gandwana.canonical.com> <20080328142109.24766.44349.malonedeb@gandwana.canonical.com> Message-ID: <4796BDA2-FDEB-40C7-89C3-674AF1998099@math.washington.edu> I have got no idea on this one. Anyone using Cython/Pyrex on windows run into anything like this before? On Mar 28, 2008, at 7:21 AM, Chris Perkins wrote: > Public bug reported: > > If I compile this one-liner pyx file on Windows: > > ctypedef int Int32 > > I can get this in my generated c file, depending on how and from where > cython is invoked: > > typedef int __pyx_t_22C:\temp\cython_bug\bug_Int32; > > ** Affects: cython > Importance: Undecided > Status: New > > -- > Invalid typedefs on Windows > https://bugs.launchpad.net/bugs/208262 > You received this bug notification because you are a member of Cython- > dev, which is subscribed to Cython. From robertwb at math.washington.edu Sat Mar 29 22:37:40 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 29 Mar 2008 14:37:40 -0700 Subject: [Cython] [Pyrex] [Bug 208262] [NEW] Invalid typedefs on Windows In-Reply-To: <47EEB0F6.7030700@canterbury.ac.nz> References: <20080328142109.24766.44349.malonedeb@gandwana.canonical.com> <20080328142109.24766.44349.malonedeb@gandwana.canonical.com> <4796BDA2-FDEB-40C7-89C3-674AF1998099@math.washington.edu> <47EEB0F6.7030700@canterbury.ac.nz> Message-ID: On Mar 29, 2008, at 2:13 PM, Greg Ewing wrote: >> On Mar 28, 2008, at 7:21 AM, Chris Perkins wrote: >> >>> typedef int __pyx_t_22C:\temp\cython_bug\bug_Int32; > > I've had another thought -- this looks like it's the > result of a name mangling operation that's somehow got > hold of a pathname as a module name. > > I gather that Cython figures out module names from > pathnames differently from Pyrex. Could something have > gone wrong in that area? > > Could it be something to do with forward vs. back > slashes? Thanks for your replies. We hunted it down and it turns out your hunch was exactly right. It might affect Pyrex as well (if the filename has some weird character in it, regardless of directory structure). - Robert From robertwb at math.washington.edu Sun Mar 30 11:18:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 30 Mar 2008 02:18:42 -0700 Subject: [Cython] cpdef wrapping of char* functions In-Reply-To: <719C7F28-66DD-40A7-906D-94BFAED1BCE0@math.washington.edu> References: <719C7F28-66DD-40A7-906D-94BFAED1BCE0@math.washington.edu> Message-ID: <68925284-B5C7-4F02-B98F-0643EAE00D4E@math.washington.edu> Resolved. On Mar 18, 2008, at 10:07 PM, Robert Bradshaw wrote: > Yes, this should work. I'll look into what's going wrong (cpdef at > the module level is rather new, but I wasn't expecting surprises like > this). > > - Robert > > > On Mar 18, 2008, at 10:04 PM, Jim Kleckner wrote: > >> I tried some wrapping of a function that returns a char* by >> using cpdef and got an error: >> >> File "/usr/lib/python2.5/site-packages/Cython/Compiler/Nodes.py", >> line 897, in call_self_node >> cfunc = ExprNodes.NameNode(self.pos, >> name=self.declarator.base.name) >> AttributeError: CFuncDeclaratorNode instance has no attribute 'name' >> >> Should this work? >> It might be because it is at the module level and not in a class >> perhaps? >> >> This is using 9.6.12. >> >> >> Test case: >> >> cdef extern from "foo.h": >> cdef int c_baz "baz"() >> cdef char* c_bar "bar"() >> >> cpdef int baz(): >> return c_baz() >> >> cpdef char* bar(): >> return c_bar() >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sun Mar 30 11:26:47 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 30 Mar 2008 02:26:47 -0700 Subject: [Cython] Small patch for __stdcall In-Reply-To: References: Message-ID: <8731B3CA-63F7-4D00-B148-26404C0FF12C@math.washington.edu> Yes, it is in. Thanks. On Mar 17, 2008, at 5:08 PM, Jim Kleckner wrote: > The following small patch gets rid of some warnings on the cygwin > platform (it it not WIN32 but does have definitions of __stdcall > resulting in warnings). > > Please let me know if it will or won't get into the upstream release. > > Thanks. > > --- ModuleNode.py.orig 2008-03-17 16:55:53.946851100 -0700 > +++ ModuleNode.py 2008-03-17 16:58:21.900687900 -0700 > @@ -272,8 +272,10 @@ > code.putln("#if PY_VERSION_HEX < 0x02040000") > code.putln(" #define METH_COEXIST 0") > code.putln("#endif") > - code.putln("#ifndef WIN32") > + code.putln("#ifndef __stdcall") > code.putln(" #define __stdcall") > + code.putln("#endif") > + code.putln("#ifndef __cdecl") > code.putln(" #define __cdecl") > code.putln("#endif") > self.generate_extern_c_macro_definition(code) > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sun Mar 30 11:57:04 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 30 Mar 2008 02:57:04 -0700 Subject: [Cython] [Pyrex] char specifier in enum In-Reply-To: <20080222121500.c4bf5607.simon@arrowtheory.com> References: <20080222121500.c4bf5607.simon@arrowtheory.com> Message-ID: This has been resolved in Cython (one doesn't even need to specify c'A' as in Pyrex, though it won't accept multi-character literals). - Robert On Feb 22, 2008, at 9:15 AM, Simon Burton wrote: > > I have a simple example: > > cdef enum foo: > A = 'A' > B = 'B' > > > gcc (4.0.3) gives this error: > > test.c:46: error: ?__pyx_k1? undeclared here (not in a function) > test.c:48: error: ?__pyx_k2? undeclared here (not in a function) > > here are the offending lines: > > 42 > 43 /* Declarations from test */ > 44 > 45 enum __pyx_t_4test_foo { > 46 __pyx_e_4test_A = __pyx_k1, > 47 __pyx_e_4test_B = __pyx_k2 > 48 }; > 49 > 50 > 51 > 52 /* Implementation of test */ > 53 > 54 static char __pyx_k1[] = "A"; > 55 static char __pyx_k2[] = "B"; > 56 > > There are several problems with this code... > > I am basically translating some c code to pyrex, maybe i should > just put this enum stuff > in a header and use a cdef extern in my pyrex code. > > Any other ideas ? > > thanks, > > Simon. > > > _______________________________________________ > Pyrex mailing list > Pyrex at lists.copyleft.no > http://lists.copyleft.no/mailman/listinfo/pyrex From robertwb at math.washington.edu Sun Mar 30 12:04:58 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 30 Mar 2008 03:04:58 -0700 Subject: [Cython] cython python versions In-Reply-To: References: <85b5c3130802130925l4acebe03kad82df927463932a@mail.gmail.com> Message-ID: <4F59EE0C-DF6D-4D27-8172-CF599598AB9E@math.washington.edu> On Feb 13, 2008, at 9:39 AM, Tim Abbott wrote: > On Wed, 13 Feb 2008, Ondrej Certik wrote: > >> Hi Tim, >> >> On Feb 11, 2008 6:09 AM, Tim Abbott wrote: >>> Hello, >>> >>> If you use python-distutils to install cython, it ends up >>> replacing the #! >>> line at the start of bin/cython (#!/usr/bin/env python) with a more >>> specific version, i.e. (#!/usr/bin/python or #!/usr/bin/ >>> python2.x). I >>> assume this is undesirable; it caused problems for me building >>> SAGE on >>> Debian Lenny. >> >> Thanks very much for tracing the problem down. One option is that >> we simply add >> a patch to the Debian package to rewrite /usr/bin/python back to >> /usr/bin/env python. >> I can do this when uploading a new version of cython. >> >>> Search for "adjust" in >>> /usr/lib/python2.5/distutils/command/build_scripts.py if you want >>> to see >>> the code that does this rewriting; it seems to be controlled by >>> whether >>> cython is registered as a script in setup.py. >> >> So what is the solution to fix this? Fixing the cython's setup.py? >> Let's do that >> in the next release then. > > I think that's the right solution. > Was this issue resolved? I am planning on doing another release soon. - Robert From martin at martincmartin.com Mon Mar 31 01:41:11 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sun, 30 Mar 2008 19:41:11 -0400 Subject: [Cython] Bug in DictNode Message-ID: <47F02517.8020907@martincmartin.com> Hi, Calling get_child_accessors() on a DictNode throws an exception, because, because it doesn't define a child_attrs field (and thus inherits the value None from class Node), so get_child_attrs() checks self.subexprs to see if it's None, but the field doesn't exist. So, what's the right fix? - Change get_child_attrs() to look at self.saved_subexpr_nodes instead of self.subexprs? - Add "subexprs = None" to ExprNode? - Add "subexprs = None" to DictNode? - Something else? Best, Martin From ondrej at certik.cz Mon Mar 31 01:48:34 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 31 Mar 2008 01:48:34 +0200 Subject: [Cython] cython python versions In-Reply-To: <4F59EE0C-DF6D-4D27-8172-CF599598AB9E@math.washington.edu> References: <85b5c3130802130925l4acebe03kad82df927463932a@mail.gmail.com> <4F59EE0C-DF6D-4D27-8172-CF599598AB9E@math.washington.edu> Message-ID: <85b5c3130803301648l759ad75amada0c5e1dd0e93ef@mail.gmail.com> On Sun, Mar 30, 2008 at 12:04 PM, Robert Bradshaw wrote: > > On Feb 13, 2008, at 9:39 AM, Tim Abbott wrote: > > On Wed, 13 Feb 2008, Ondrej Certik wrote: > > > >> Hi Tim, > >> > >> On Feb 11, 2008 6:09 AM, Tim Abbott wrote: > >>> Hello, > >>> > >>> If you use python-distutils to install cython, it ends up > >>> replacing the #! > >>> line at the start of bin/cython (#!/usr/bin/env python) with a more > >>> specific version, i.e. (#!/usr/bin/python or #!/usr/bin/ > >>> python2.x). I > >>> assume this is undesirable; it caused problems for me building > >>> SAGE on > >>> Debian Lenny. > >> > >> Thanks very much for tracing the problem down. One option is that > >> we simply add > >> a patch to the Debian package to rewrite /usr/bin/python back to > >> /usr/bin/env python. > >> I can do this when uploading a new version of cython. > >> > >>> Search for "adjust" in > >>> /usr/lib/python2.5/distutils/command/build_scripts.py if you want > >>> to see > >>> the code that does this rewriting; it seems to be controlled by > >>> whether > >>> cython is registered as a script in setup.py. > >> > >> So what is the solution to fix this? Fixing the cython's setup.py? > >> Let's do that > >> in the next release then. > > > > I think that's the right solution. > > > > Was this issue resolved? I am planning on doing another release soon. Tim? As to me, all is fine. Ondrej From martin at martincmartin.com Mon Mar 31 02:07:27 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sun, 30 Mar 2008 20:07:27 -0400 Subject: [Cython] Bug in DictNode In-Reply-To: <47F02517.8020907@martincmartin.com> References: <47F02517.8020907@martincmartin.com> Message-ID: <47F02B3F.7020302@martincmartin.com> In case it makes a difference, this DictNode is the "dict" attribute of a PyClassDefNode, created from this: class Spam: pass Best, Martin Martin C. Martin wrote: > Hi, > > Calling get_child_accessors() on a DictNode throws an exception, > because, because it doesn't define a child_attrs field (and thus > inherits the value None from class Node), so get_child_attrs() checks > self.subexprs to see if it's None, but the field doesn't exist. So, > what's the right fix? > > - Change get_child_attrs() to look at self.saved_subexpr_nodes instead > of self.subexprs? > - Add "subexprs = None" to ExprNode? > - Add "subexprs = None" to DictNode? > - Something else? > > Best, > Martin > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Mon Mar 31 10:36:50 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 31 Mar 2008 10:36:50 +0200 Subject: [Cython] Bug in DictNode In-Reply-To: <47F02517.8020907@martincmartin.com> References: <47F02517.8020907@martincmartin.com> Message-ID: <47F0A2A2.5010407@student.matnat.uio.no> Martin C. Martin wrote: > Hi, > > Calling get_child_accessors() on a DictNode throws an exception, > because, because it doesn't define a child_attrs field (and thus > inherits the value None from class Node), so get_child_attrs() checks > self.subexprs to see if it's None, but the field doesn't exist. So, > what's the right fix? > > - Change get_child_attrs() to look at self.saved_subexpr_nodes instead > of self.subexprs? > - Add "subexprs = None" to ExprNode? > - Add "subexprs = None" to DictNode? > - Something else? > Two fixes are needed. If you are simply trying to get some transform code running that doesn't involve transforming DictNode then the first will suffice for now. 1) Fix the self.subexprs check (probably adding subexprs = None to ExprNode would be best) 2) Get DictNode to actually export its children to the tree. I don't know this part of the code too well so this might be wrong, but in principle this is done by adding to DictNode: child_attrs = ["key_value_pairs"] However, there is a problem with this: key_value_pairs doesn't contain tree nodes as children but rather tuples. There are many approaches to this: a) Every tree transform that works with DictNode accepts this as the manner of working, and the current tree printers etc. are modified to handle this. b) Probably better: Change the implementation of DictNode to use a list of a new DictEntryNode node class (with "key" and "value" subnodes and attributes) directly in key_value_pairs. This needs proper testing. c) Being more careful with existing code, one might override get_child_attrs in DictNode instead, and provide an adaptor object that wraps the list and returns approprioately wrapped children (basically export a list of DictEntryNode, each DictEntryNode representing each tuple in the list -- but this is complicated in that changes to the exported list should also be mirrored in , so this is probably going to be a ~70 line wrapper and not worth it over b - however it won't break existing code and so won't need the same degree of testing (for now)). -- Dag Sverre From robertwb at math.washington.edu Mon Mar 31 18:11:52 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 31 Mar 2008 09:11:52 -0700 Subject: [Cython] Bug in DictNode In-Reply-To: <47F0A2A2.5010407@student.matnat.uio.no> References: <47F02517.8020907@martincmartin.com> <47F0A2A2.5010407@student.matnat.uio.no> Message-ID: On Mar 31, 2008, at 1:36 AM, Dag Sverre Seljebotn wrote: > Martin C. Martin wrote: >> Hi, >> >> Calling get_child_accessors() on a DictNode throws an exception, >> because, because it doesn't define a child_attrs field (and thus >> inherits the value None from class Node), so get_child_attrs() checks >> self.subexprs to see if it's None, but the field doesn't exist. So, >> what's the right fix? >> >> - Change get_child_attrs() to look at self.saved_subexpr_nodes >> instead >> of self.subexprs? >> - Add "subexprs = None" to ExprNode? >> - Add "subexprs = None" to DictNode? >> - Something else? >> > Two fixes are needed. If you are simply trying to get some transform > code running that doesn't involve transforming DictNode then the first > will suffice for now. > > 1) Fix the self.subexprs check (probably adding subexprs = None to > ExprNode would be best) > > 2) Get DictNode to actually export its children to the tree. I don't > know this part of the code too well so this might be wrong, but in > principle this is done by adding to DictNode: > > child_attrs = ["key_value_pairs"] > > However, there is a problem with this: key_value_pairs doesn't contain > tree nodes as children but rather tuples. There are many approaches > to this: > > a) Every tree transform that works with DictNode accepts this as the > manner of working, and the current tree printers etc. are modified to > handle this. > > b) Probably better: Change the implementation of DictNode to use a > list > of a new DictEntryNode node class (with "key" and "value" subnodes and > attributes) directly in key_value_pairs. This needs proper testing. > > c) Being more careful with existing code, one might override > get_child_attrs in DictNode instead, and provide an adaptor object > that > wraps the list and returns approprioately wrapped children (basically > export a list of DictEntryNode, each DictEntryNode representing each > tuple in the list -- but this is complicated in that changes to the > exported list should also be mirrored in , so this is probably > going to > be a ~70 line wrapper and not worth it over b - however it won't break > existing code and so won't need the same degree of testing (for now)). Personally, I'm torn between (a) and (b) as being the best options. There are also problems with comparison nodes that I'm trying to get sorted out before releasing. - Robert From tabbott at MIT.EDU Mon Mar 31 02:26:37 2008 From: tabbott at MIT.EDU (Timothy G Abbott) Date: Sun, 30 Mar 2008 20:26:37 -0400 (EDT) Subject: [Cython] cython python versions In-Reply-To: <85b5c3130803301648l759ad75amada0c5e1dd0e93ef@mail.gmail.com> References: <85b5c3130802130925l4acebe03kad82df927463932a@mail.gmail.com> <4F59EE0C-DF6D-4D27-8172-CF599598AB9E@math.washington.edu> <85b5c3130803301648l759ad75amada0c5e1dd0e93ef@mail.gmail.com> Message-ID: On Mon, 31 Mar 2008, Ondrej Certik wrote: > On Sun, Mar 30, 2008 at 12:04 PM, Robert Bradshaw > wrote: > >> Was this issue resolved? I am planning on doing another release soon. > > Tim? As to me, all is fine. It turns out for the Debian packaging one should run 'python cython' anyway, so all is fine from my perspective as well. -Tim Abbott From king at mathematik.uni-jena.de Thu Mar 20 15:04:44 2008 From: king at mathematik.uni-jena.de (Simon King) Date: Thu, 20 Mar 2008 15:04:44 +0100 (CET) Subject: [Cython] [Pyrex] How to deal with a block of memory when pickling? In-Reply-To: <35CE5837-E89C-4BA7-8729-217CE8C59173@math.washington.edu> References: <35CE5837-E89C-4BA7-8729-217CE8C59173@math.washington.edu> Message-ID: Dear Robert, On Thu, 20 Mar 2008, Robert Bradshaw wrote: > >What can i do? > > Use the PyString_FromStringAndSize function. (Otherwise it assumes a null > character is the end of the string, per C convention.) Thank you very much! Cheers Simon From king at mathematik.uni-jena.de Sat Mar 22 21:08:11 2008 From: king at mathematik.uni-jena.de (Simon King) Date: Sat, 22 Mar 2008 21:08:11 +0100 (CET) Subject: [Cython] Fast matrices over GF(p) In-Reply-To: References: <20080320095215.b309f8fb.simon@arrowtheory.com> Message-ID: Dear Robert, On Sat, 22 Mar 2008, Robert Bradshaw wrote: > > Sounds very interesting! Sage has been mostly optimized to deal with very > large matrices, and GF(p) for p barely fitting into a word (to facilitate > linear algebra over Z and Q). It sounds like your code does much better for > small p. Do you mind if I forward your email onto sage-devel? I am on sage-devel, myself. So i wouldn't mind. But it is of course not *my* code. MeatAxe can be found at http://www.math.rwth-aachen.de/~MTX/. My contribution is just a wrapper, i didn't improve the arithmetics. Actually i use a rather old version of C-MeatAxe. This is because my boss (David Green) wrote C-code for computing cohomology, relying on the old C-MeatAxe. Yours Simon From king at mathematik.uni-jena.de Mon Mar 24 21:38:46 2008 From: king at mathematik.uni-jena.de (Simon King) Date: Mon, 24 Mar 2008 21:38:46 +0100 (CET) Subject: [Cython] Fast matrices over GF(p) In-Reply-To: <35024ECB-A2B3-41AA-8690-2843859CC7F9@math.washington.edu> References: <20080320095215.b309f8fb.simon@arrowtheory.com> <35024ECB-A2B3-41AA-8690-2843859CC7F9@math.washington.edu> Message-ID: Dear Robert, On Mon, 24 Mar 2008, Robert Bradshaw wrote: > > Some of us ran into MeatAxe before, but no one had any experience with it or > knew anyone who used it, so nothing ever happened of it. I was thinking since > you actually use it (and have a Cython wrapper) you would know what it's > strengths/weaknesses are and whether or not it would be worth including in > Sage. Ok, then i think i will write to Sage devel in a few days and post some code, so that people can test it. A few words on strengths/weaknesses: Main Strength of MeatAxe: IMO, it is a slim and simple data structure. Hence, many basic operations such as copying and (un)pickling are a lot faster with MTX than with Sage matrices. Also, the hash and test for equality of my wrapper works quite fast: Although i do not (yet) cache the hash, using a 500x500 MTX matrix as key for looking up in a dictionary is (on average) 0.57 ms, while it is (on average) 819 ms with the corresponding Sage matrix, even though its hash is already cached (if the hash isn't known, one look-up takes more than 2s)! The strength of MTX in arithmetics is IMO: - difference of two matrices (>300 times faster than Sage); - multiplication of a matrix with a scalar (actually this is a very weak point of Sage matrices, namely it is slower than the multiplication of two matrices). - Computation of nullspace is good (for a dense 1000x500 matrix over GF(7), MTX was 6 times faster than Sage). Weak points in arithmetics: - Multiplication of two matrices (slower than Sage by a factor 2.5 for 500x500 matrices over GF(7)); - exponentiation of a matrix. Conceptual weakness: - Restriction to fields of order <256 - MeatAxe (at least version 2.2.3 of 1997 that i am using) relies on multiplication tables that are stored in a file in the current directory, and whose creation relies on an executable "maketab". This, i think, is nasty. I don't know if the more recent versions have the multiplication tables in memory. Also i don't know if my wrapper would still work if i'd change to the new MeatAxe. So far my experience Cheers Simon