From stefan_ml at behnel.de Sun Mar 2 12:24:30 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Mar 2008 12:24:30 +0100 Subject: [Cython] Cython test coverage Message-ID: <47CA8E6E.4040107@behnel.de> Hi, I just ran coverage.py over the test suite. http://nedbatchelder.com/code/modules/coverage.html $ python -m coverage -x runtests.py [...] $ python -m coverage -r Cython/Compiler/*.py Name Stmts Exec Cover ------------------------------------------------ Cython/Compiler/Annotate 107 19 17% Cython/Compiler/Builtin 17 17 100% Cython/Compiler/CmdLine 68 0 0% Cython/Compiler/Code 246 226 91% Cython/Compiler/DebugFlags 3 3 100% Cython/Compiler/Errors 68 41 60% Cython/Compiler/ExprNodes 1969 1245 63% Cython/Compiler/Lexicon 39 2 5% Cython/Compiler/Main 212 137 64% Cython/Compiler/ModuleNode 1048 914 87% Cython/Compiler/Naming 67 66 98% Cython/Compiler/Nodes 2038 1412 69% Cython/Compiler/Options 11 11 100% Cython/Compiler/Parsing 1556 1078 69% Cython/Compiler/PyrexTypes 589 484 82% Cython/Compiler/Scanning 260 186 71% Cython/Compiler/Symtab 803 666 82% Cython/Compiler/TypeSlots 264 259 98% Cython/Compiler/Version 1 1 100% Cython/Compiler/__init__ 0 0 100% ------------------------------------------------ TOTAL 9366 6767 72% Quite ok, but also quite a bit missing. Especially Parsing.py, Nodes.py and ExprNodes.py do not make me feel very comfortable. I would assume error testing to be a major factor here. We test a lot of working code, but little broken code. Stefan From stefan_ml at behnel.de Sun Mar 2 12:25:34 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Mar 2008 12:25:34 +0100 Subject: [Cython] Cython and Python 3k Message-ID: <47CA8EAE.4030904@behnel.de> Hi, just starting a little thread here to collect things to remember when we migrate to Python 3k one day. Robert Bradshaw wrote: > Stefan Behnel wrote: >>>> Also, there's Python 3, where especially the C-API is still in flux. >>>> Not sure >>>> if it's worth to start thinking about that yet, but it should at least >>>> stay on the list. >>> >>> Yep. >> >> I asked on cython-dev to see how far the C-API is. Answer I got is >> that it may >> take a bit to stabilize, including renaming types (str etc.) and >> fixing the >> outdated docs. So we'll have to see what the status will be in June. >> It might really be worth a sprint by then. >> >> http://comments.gmane.org/gmane.comp.python.devel/92255?set_lines=100000 >> >> Two additional things I see here: 1) porting Cython itself, and 2) >> working on >> better optimisation support in general, to keep users from doing >> direct C-API >> calls in Cython code, as this might lead to reduced portability. > > I think Python 2.x will be around for quite a while, even after P3k > comes out, but if the API is starting to stabilize (and even if not) I > think it is good to start looking at it. Porting Cython itself should be > pretty easy, 3.0 is mostly backwards compatible, and I don't think we > use much that isn't. It will almost certainly require a branch, though. We might be lucky as most of Cython's code is still pretty old style, without using unicode, generators and the like. I added bundles that fix at least the obvious "print()" and "<>" stuff. Things like "print >> sys.stderr" will need fixing, no idea what else is waiting... > Ideally, I would like to be able to run (the same) Cython with Python > 2.x or 3.x, and have it produce code that compiles (using macros) for > inclusion in either 2.x or 3.x runtimes. We can do this for 2.3-2.5. > Whether or not this is feasible across such a major revision remains to > be seen, but I'm optimistic. So am I (still). We will have to figure out some transition semantics for str/unicode/bytes, though, also at the Cython syntax level. But I guess that has time till the first Py3k beta release. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: package-upstream.bundle Type: application/octet-stream Size: 2904 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080302/f5d0a648/attachment-0002.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: upstream.bundle Type: application/octet-stream Size: 3532 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080302/f5d0a648/attachment-0003.obj From stefan_ml at behnel.de Sun Mar 2 12:30:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Mar 2008 12:30:36 +0100 Subject: [Cython] Cython and Python 3k In-Reply-To: <47CA8EAE.4030904@behnel.de> References: <47CA8EAE.4030904@behnel.de> Message-ID: <47CA8FDC.5040208@behnel.de> Stefan Behnel wrote: > I added bundles that fix at least the obvious "print()" and "<>" stuff. Sorry, incomplete bundle. Here we go. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: upstream.bundle Type: application/octet-stream Size: 8202 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080302/9c9823f7/attachment.obj From ndbecker2 at gmail.com Mon Mar 3 13:54:50 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 03 Mar 2008 07:54:50 -0500 Subject: [Cython] make extension type accesible from python? Message-ID: I'm trying to learn cython. As a simple first try, I have some generic c++ code 'my_sum'. I want to pass it an intvec (std::vector). It seems to work OK when accessed from c-code: cdef extern from "vector": pass cdef extern from "test1.hpp": ctypedef struct intvec "std::vector": void (* push_back)(int elem) intvec intvec_factory "std::vector"(int len) int my_sum "my_sum >"(intvec vec) cdef intvec v = intvec_factory(2) v.push_back(2) print my_sum (v) I have no idea how to expose intvec to python, and then be able to call my_sum from python. I assume there is some way to make extension types that can be used from python? Of course, I'd want to be able to add other methods, so that, for example, intvec can be constructed and manipulated from python. Any simple examples I could look at? From stefan_ml at behnel.de Mon Mar 3 14:14:09 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 3 Mar 2008 14:14:09 +0100 (CET) Subject: [Cython] make extension type accesible from python? In-Reply-To: References: Message-ID: <11469.194.114.62.68.1204550049.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: [real problem stripped] > I assume there is some way to make extension types that can be used from > python? Of course, I'd want to be able to add other methods, so that, for > example, intvec can be constructed and manipulated from python. Yes, wrapping it in a Python extension type is the right approach here. > Any simple examples I could look at? The Pyrex docs, for example: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/ http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/extension_types.html Stefan From ndbecker2 at gmail.com Mon Mar 3 19:01:25 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 03 Mar 2008 13:01:25 -0500 Subject: [Cython] make extension type accesible from python? References: <11469.194.114.62.68.1204550049.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > Neal Becker wrote: > [real problem stripped] >> I assume there is some way to make extension types that can be used from >> python? Of course, I'd want to be able to add other methods, so that, >> for example, intvec can be constructed and manipulated from python. > > Yes, wrapping it in a Python extension type is the right approach here. > > >> Any simple examples I could look at? > > The Pyrex docs, for example: > > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/ > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/extension_types.html > > Stefan I've been studying these, but I haven't seen my question addressed. Are there any examples I could look at? From ndbecker2 at gmail.com Tue Mar 4 13:39:02 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 04 Mar 2008 07:39:02 -0500 Subject: [Cython] wrapping c++ Message-ID: I've made some progress on learning to wrap std::vector. I've got a __getitem__, but I don't know what to do about __setitem__. cdef extern from "vector": pass cdef extern from "test1.hpp": ctypedef struct intvec "std::vector": void (* push_back)(int elem) inline int get "operator[]" (int i) intvec intvec_factory "std::vector"(int len) int my_sum "my_sum >"(intvec vec) cdef intvec v = intvec_factory(2) v.push_back(2) #print my_sum (v) cdef class intvec_wrap: cdef intvec vec def __init__(self): self.vec = intvec_factory (2) def __getitem__(self, int i): return self.vec.get (i) Problem is, operator[] (or at()) returns an int&. I don't have a clue what to do with this. From wstein at gmail.com Tue Mar 4 15:54:11 2008 From: wstein at gmail.com (William Stein) Date: Tue, 4 Mar 2008 08:54:11 -0600 Subject: [Cython] wrapping c++ In-Reply-To: References: Message-ID: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> On Tue, Mar 4, 2008 at 6:39 AM, Neal Becker wrote: > I've made some progress on learning to wrap std::vector. I've got a > __getitem__, but I don't know what to do about __setitem__. > cdef extern from "vector": > pass > > cdef extern from "test1.hpp": > ctypedef struct intvec "std::vector": > void (* push_back)(int elem) > inline int get "operator[]" (int i) > > intvec intvec_factory "std::vector"(int len) > > int my_sum "my_sum >"(intvec vec) > > > > cdef intvec v = intvec_factory(2) > v.push_back(2) > > #print my_sum (v) > > cdef class intvec_wrap: > cdef intvec vec > def __init__(self): > self.vec = intvec_factory (2) > def __getitem__(self, int i): > return self.vec.get (i) > > Problem is, operator[] (or at()) returns an int&. I don't have a clue what > to do with this. There is an example of doing setitem directly in Cython with std::vector using C name specifiers in my talk on Cython at Enthought from two days ago. See the talk listed at the bottom of this page: http://wiki.sagemath.org/days8/schedule Here's the example: cdef extern from "vector.h": ctypedef unsigned long size_type ctypedef struct intvec "std::vector": int get_entry "operator[]"(int n) size_type size() intvec intvec_factory "std::vector"(int len) cdef extern from "": void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() void SET(intvec, int, int) DEFINE_SET() cdef intvec v = intvec_factory(10) SET(v,2,5) print v.get_entry(2) print v.size() ---------------- The key thing is the evil dirty trick to define a SET macro directly in Cython. -- William From stefan_ml at behnel.de Tue Mar 4 17:09:39 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 4 Mar 2008 17:09:39 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> Message-ID: <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> William Stein wrote: > cdef extern from "": > void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() > void SET(intvec, int, int) > DEFINE_SET() > > The key thing is the evil dirty trick to define a SET macro directly in > Cython. "Evil" and "dirty" were exactly the two words that came to my mind before I even got to reading your own comment on this. :) I would really put something like this into an external header file, and just declare "SET()" as being external... Stefan From ndbecker2 at gmail.com Tue Mar 4 17:21:06 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 04 Mar 2008 11:21:06 -0500 Subject: [Cython] wrapping c++ References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > William Stein wrote: >> cdef extern from "": >> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >> void SET(intvec, int, int) >> DEFINE_SET() >> >> The key thing is the evil dirty trick to define a SET macro directly in >> Cython. > > "Evil" and "dirty" were exactly the two words that came to my mind before > I even got to reading your own comment on this. :) > > I would really put something like this into an external header file, and > just declare "SET()" as being external... > > Stefan Any thoughts on what it would take to support a non-evil implementation? From stefan_ml at behnel.de Tue Mar 4 18:07:55 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 4 Mar 2008 18:07:55 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Stefan Behnel wrote: > >> William Stein wrote: >>> cdef extern from "": >>> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >>> void SET(intvec, int, int) >>> DEFINE_SET() >>> >>> The key thing is the evil dirty trick to define a SET macro directly in >>> Cython. >> >> "Evil" and "dirty" were exactly the two words that came to my mind >> before I even got to reading your own comment on this. :) >> >> I would really put something like this into an external header file, and >> just declare "SET()" as being external... >> > Any thoughts on what it would take to support a non-evil implementation? The above is ok, except that I wouldn't write something like this into a Cython file. You will likely need a header file for a couple of low-level things anyway, so I would just create one with the "#define" above and keep the 'cdef extern from "theheader.h"' for the "SET()" function (read: macro). Stefan From robertwb at math.washington.edu Tue Mar 4 20:30:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 11:30:33 -0800 Subject: [Cython] wrapping c++ In-Reply-To: <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <122F497C-84FA-4BA2-A8E3-F1AEFF9A6391@math.washington.edu> On Mar 4, 2008, at 9:07 AM, Stefan Behnel wrote: > Neal Becker wrote: >> Stefan Behnel wrote: >> >>> William Stein wrote: >>>> cdef extern from "": >>>> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >>>> void SET(intvec, int, int) >>>> DEFINE_SET() >>>> >>>> The key thing is the evil dirty trick to define a SET macro >>>> directly in >>>> Cython. >>> >>> "Evil" and "dirty" were exactly the two words that came to my mind >>> before I even got to reading your own comment on this. :) >>> >>> I would really put something like this into an external header >>> file, and >>> just declare "SET()" as being external... >>> >> Any thoughts on what it would take to support a non-evil >> implementation? > > The above is ok, except that I wouldn't write something like this > into a > Cython file. You will likely need a header file for a couple of low- > level > things anyway, so I would just create one with the "#define" above and > keep the 'cdef extern from "theheader.h"' for the "SET()" function > (read: > macro). I've been thinking about adding C++ operator support to Cython. I'm not sure of the best syntax to use for this, but it should be possible to tell Cython that a certain type has overloaded operators, and also to be able to specify a specific function to use for such operators (for example, this would be useful in numpy for fast indexing of arrays--one would define a macro that does the dirty work and it would be like the PEX a{i,j}. - Robert From wstein at gmail.com Tue Mar 4 20:32:14 2008 From: wstein at gmail.com (William Stein) Date: Tue, 4 Mar 2008 13:32:14 -0600 Subject: [Cython] Fwd: [sage-support] cython and filenames In-Reply-To: <0311e1c9-04dd-49d7-89fd-d8311010682e@d4g2000prg.googlegroups.com> References: <0311e1c9-04dd-49d7-89fd-d8311010682e@d4g2000prg.googlegroups.com> Message-ID: <85e81ba30803041132o7d31ef5cn149840a65b7b7efe@mail.gmail.com> Hi, I'm forwarding this message from sage-support to Cython-devel, where it points out a probable bug in the Cython compiler (with fix). -- william (from Sage Days 8) ---------- Forwarded message ---------- From: Kate Date: Tue, Mar 4, 2008 at 10:44 AM Subject: [sage-support] cython and filenames To: sage-support Here is a snippet of code from cython.py in the main tree (SAGE_ROOT/devel/sage/sage/misc/cython.py) def cython(filename, verbose=False, compile_message=False, use_cache=False, create_local_c_file=False, annotate=True, sage_namespace=True): if not filename.endswith('pyx'): print "File (=%s) should have extension .pyx"%filename clean_filename = sanitize(filename) ====> base = os.path.split(os.path.splitext(clean_filename)[0])[1] ====> abs_base = os.path.abspath(os.path.split(os.path.splitext(clean_filename)[0])[0]) It seems to me that the marked lines, in addition to being hard to parse, do not have the intended effect. This is because in the creation of clean_filename, the call to the sanitize function converts all slashes and dots into underscores, which are not 'picked up' by the calls to the os.path.* functions. Is this really what is intended? Kate --~--~---------~--~----~------------~-------~--~----~ To post to this group, send email to sage-support at googlegroups.com To unsubscribe from this group, send email to sage-support-unsubscribe at googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URLs: http://www.sagemath.org -~----------~----~----~----~------~----~------~--~--- -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From robertwb at math.washington.edu Tue Mar 4 20:39:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 11:39:14 -0800 Subject: [Cython] Fwd: [sage-support] cython and filenames In-Reply-To: <85e81ba30803041132o7d31ef5cn149840a65b7b7efe@mail.gmail.com> References: <0311e1c9-04dd-49d7-89fd-d8311010682e@d4g2000prg.googlegroups.com> <85e81ba30803041132o7d31ef5cn149840a65b7b7efe@mail.gmail.com> Message-ID: <2F102241-C47B-4E84-A0CC-15C371A90BB9@math.washington.edu> This actually isn't the cython compiler code, it's in the Sage library. I'm actually not sure what the goal was here with the clean_filename stuff (didn't write it myself, and I'm wary of changing it because I'm guessing there was some reason to do it this way)... - Robert On Mar 4, 2008, at 11:32 AM, William Stein wrote: > Hi, > > I'm forwarding this message from sage-support to Cython-devel, where > it points out a probable bug in the Cython compiler (with fix). > > -- william (from Sage Days 8) > > ---------- Forwarded message ---------- > From: Kate > Date: Tue, Mar 4, 2008 at 10:44 AM > Subject: [sage-support] cython and filenames > To: sage-support > > > > Here is a snippet of code from cython.py in the main tree > (SAGE_ROOT/devel/sage/sage/misc/cython.py) > > def cython(filename, verbose=False, compile_message=False, > use_cache=False, create_local_c_file=False, annotate=True, > sage_namespace=True): > if not filename.endswith('pyx'): > print "File (=%s) should have extension .pyx"%filename > > clean_filename = sanitize(filename) > ====> base = os.path.split(os.path.splitext(clean_filename)[0])[1] > ====> abs_base = > os.path.abspath(os.path.split(os.path.splitext(clean_filename)[0]) > [0]) > > It seems to me that the marked lines, in addition to being > hard to parse, do not have the intended effect. This is > because in the creation of clean_filename, the call to > the sanitize function converts all slashes and dots into > underscores, which are not 'picked up' by the calls to > the os.path.* functions. > > Is this really what is intended? > > Kate > > --~--~---------~--~----~------------~-------~--~----~ > To post to this group, send email to sage-support at googlegroups.com > To unsubscribe from this group, send email to > sage-support-unsubscribe at googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-support > URLs: http://www.sagemath.org > -~----------~----~----~----~------~----~------~--~--- > > > > > -- > William Stein > Associate Professor of Mathematics > University of Washington > http://wstein.org > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Tue Mar 4 21:54:34 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 04 Mar 2008 21:54:34 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support Message-ID: <47CDB70A.7080903@student.matnat.uio.no> Since Robert mentioned NumPy in relation with adding operator support I thought about sharing my more thoughts about NumPy - I'm very new to Cython so I guess take it for what it is worth - however what I've seen so far looks so promising for me that I might want to spend some time in a few months working on implementing some of this, which perhaps may make my thoughts more intereseting :-) Currently, Cython is mostly geared towards wrapping C code, but it is also an excellent foundation for being a numerical tool - but the rough edges are still prohibitive. A few relatively small steps (in terms of man-hours needed) would improve the situation a lot I think - not perfect, but perhaps in a few years we can have something that will finally kill FORTRAN :-) Three suggestions comes briefly here, if anyone's interested and it is not already discussed and decided I might flesh them out in "PEP-style" in the coming month? Note that a) is what is important for me, b) and c) is just something I throw along... a) numpy.ndarray syntax candy. Really, what one should implement is syntax support for PEP-3118: http://www.python.org/dev/peps/pep-3118/ Because this protocol will be shared between NumPy, PIL etc. in Python 3 it could make sense to simply have "native"/hard-coded support for this aspect without necesarrily making it a generic operator feature, and one can then use the same approach as will be needed for buffers in Python 3 for NumPy in Python 2? Example (where "array" is considered a new, Cython-native type that will have automatic conversion from any NumPy arrays and Python 3 buffers): def myfunc(array<2, unsigned char> arr): arr[4, 5] = 10 might be translated to the equivalent of the currently legal: def myfunc(numpy.ndarray arr): if arr.nd != 2 or arr.dtype != numpy.dtype(numpy.uint8): raise ValueError("Must pass 2-dimensional uint8 array.") cdef unsigned char* arr_buf = arr.data arr.data[4 * arr.strides[0] + 5 * arr.strides[1]] = 10 (Probably caching the strides in local variables etc.). That should do as a first implementation -- it is always possible to be more sophisticated, but this little will allow NumPyers to simply dive in. Specifically, the number of dimensions must be declared first and only direct access in that many dimensions are allowed. Slices etc. should be less important (they can be done on the Python object instead). Moving on from here, one should probably instead define bufferinfo from PEP-3118 and make it say def myfunc(bufferinfo arr): if arr.ndim != 2 or arr.format != "B") or arr.readonly: raise ValueError("Must pass writeable 2-dimensional buffer with format 'B'.") ... with automatic conversion from NumPy arrays to bufferinfo. b) Allow numpy types? Basically, make it possible to say "cdef uint8 myvar", at least for in-function-variables that is not interfacing with C code, so that for numerical use one doesn't need to learn C. This can be in addition, so it should not break existing code, though I can understand resentment against the idea as well. c) Probably controversial: More Pythonic syntax. A syntax for decoration of function arguments is decided upon (at least in Python 3), so to align with that one could allow for stuff like @Compile def myfunc(a: uint8, b: array(2, uint8), c: int = 10): d: ptr(int) = &a print a, b, c, d Which is "almost" Python - only the definition of d is different, but consistency talks for change there as well. This can also be in addition to the existing syntax so it should not break anything (allowing, say, only one type of syntax per function). But a) is what is interesting here... -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 4 22:02:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 04 Mar 2008 22:02:30 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <47CDB70A.7080903@student.matnat.uio.no> References: <47CDB70A.7080903@student.matnat.uio.no> Message-ID: <47CDB8E6.2010200@student.matnat.uio.no> Correction: > def myfunc(numpy.ndarray arr): > if arr.nd != 2 or arr.dtype != numpy.dtype(numpy.uint8): > raise ValueError("Must pass 2-dimensional uint8 array.") > cdef unsigned char* arr_buf = arr.data > arr.data[4 * arr.strides[0] + 5 * arr.strides[1]] = 10 > Last line should index arr_buf, not arr.data. BTW, > perfect, but perhaps in a few years we can have something that will > finally kill FORTRAN :-) > In case you were wondering, this is definitely a joke. One can hope though. -- Dag Sverre From ndbecker2 at gmail.com Tue Mar 4 22:28:15 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 04 Mar 2008 16:28:15 -0500 Subject: [Cython] wrapping c++ References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <122F497C-84FA-4BA2-A8E3-F1AEFF9A6391@math.washington.edu> Message-ID: Robert Bradshaw wrote: > On Mar 4, 2008, at 9:07 AM, Stefan Behnel wrote: > >> Neal Becker wrote: >>> Stefan Behnel wrote: >>> >>>> William Stein wrote: >>>>> cdef extern from "": >>>>> void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() >>>>> void SET(intvec, int, int) >>>>> DEFINE_SET() >>>>> >>>>> The key thing is the evil dirty trick to define a SET macro >>>>> directly in >>>>> Cython. >>>> >>>> "Evil" and "dirty" were exactly the two words that came to my mind >>>> before I even got to reading your own comment on this. :) >>>> >>>> I would really put something like this into an external header >>>> file, and >>>> just declare "SET()" as being external... >>>> >>> Any thoughts on what it would take to support a non-evil >>> implementation? >> >> The above is ok, except that I wouldn't write something like this >> into a >> Cython file. You will likely need a header file for a couple of low- >> level >> things anyway, so I would just create one with the "#define" above and >> keep the 'cdef extern from "theheader.h"' for the "SET()" function >> (read: >> macro). > > > I've been thinking about adding C++ operator support to Cython. I'm > not sure of the best syntax to use for this, but it should be > possible to tell Cython that a certain type has overloaded operators, > and also to be able to specify a specific function to use for such > operators (for example, this would be useful in numpy for fast > indexing of arrays--one would define a macro that does the dirty work > and it would be like the PEX a{i,j}. > > - Robert Is operator the real issue here? In the std::vector example, there is 'at', which does (almost) the same thing as operator[]. My problem is that both 'at' and 'operator[]' return int&, and to use them you need to say 'at[i]=j', which I don't know how to do in cython (except for the macro version above, or to write additional wrapper code in more c++ headers). From dagss at student.matnat.uio.no Tue Mar 4 22:53:02 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 04 Mar 2008 22:53:02 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> Message-ID: <47CDC4BE.7050608@student.matnat.uio.no> > buffers, then this could be provided automatically. Overloading the = > operator would work for the one direction, and perhaps something > similar would be provided to do an "convert self to other type" > declaration. Right. This will keep me thinking :-) Has there been any thoughts on doing something as brute as allowing Python code in pxd files that is simply called compile-time and returns strings of C code? Or if not there, as a kind of plugin-architecture for the Cython compiler engine? Like this (apologies for bad knowledge of this aspect of Cython): (Lot's of stuff that doesn't work, just scetching wildly...) cdef class std_vector_int: def __compile_index_assign__(lefthand, indices, righthand): return "%s[%s] = %s;" % (as_c(lefthand), as_c(indices), as_c(righthand)) def __compile_index_lookup__(lefthand, indices): return "%s[%s];" % (as_c(lefthand), as_c(indices)) def __compile_assign__(lefthand, rightand):... cdef class numpy.numarray ...: def __compile_index_assign__(lefthand, indices, righthand): l = indices.split(":") if len(l) != int(lefthand.type_params[0]): raise ValueError(...) idx = " + ".join([as_c(x) + " * stride[%d]" % d for x in zip(l, range(len(l))]) return "%s[%s] = %s;" % (as_c(lefthand), idx, as_c(righthand)) With the "macro as cdef"-trick earlier in the mailing list this certainly doesn't seem *too* far-fetched for me... > > These data types can be declared, right now, by doing. > > cdef extern from "numpy.h": > cdeftype int uint8 Ah, right :-) >> >> @Compile >> def myfunc(a: uint8, b: array(2, uint8), c: int = 10): >> d: ptr(int) = &a >> print a, b, c, d >> > > One thing I really like about the cdef keyword is that it makes it > very clear what is special Cython commands, and what is not. I also > find this harder to read (and parse). (BTW, what's the @Compile > supposed to mean?) However, if people want to move in this direction I > wouldn't protest against it as an alternative. The idea was to focus on making Cython usable for numerical computation -- and for that use (as opposed to making wrappers for C code) you do NOT want people to feel that it is not part of Python, but rather treat it as a "typed Python that will run fast". The use case is simply somebody that uses NumPy for everything, but comes over this one operation that a) isn't in the libraries, b) can't be vectorized but must be implemented as for-loops. If they can simply drop in a @Compile decorator and add some types and get on with their work, the pure convenience of it might make Cython a regular tool for these kind of things... @Compile simply uses the standard Python function decorator syntax. The @Compile might be @Cython instead actually. The idea is that while Python syntax allows for decorating function arguments, it doesn't say anything about what they're for (documentation? something else?), so you would often have a function decorator that acts on them, and this serves to document that the function argument decorators are Cython types. Long term, @Compile might be used to replace the function with calls to compiled code in Python-land, and to extract functions to compile from a py-file in Cython-land, so that one can mix Cython and Python in one source file. Also, it can take the options that Cython needs (ie @Compile(native=True, except=1) instead of cdef ... except ). Dag Sverre From robertwb at math.washington.edu Tue Mar 4 23:15:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 14:15:20 -0800 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <47CDC4BE.7050608@student.matnat.uio.no> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> <47CDC4BE.7050608@student.matnat.uio.no> Message-ID: <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> On Mar 4, 2008, at 1:53 PM, Dag Sverre Seljebotn wrote: > >> buffers, then this could be provided automatically. Overloading the = >> operator would work for the one direction, and perhaps something >> similar would be provided to do an "convert self to other type" >> declaration. > Right. This will keep me thinking :-) > > Has there been any thoughts on doing something as brute as allowing > Python code in pxd files that is simply called compile-time and > returns > strings of C code? Or if not there, as a kind of plugin- > architecture for > the Cython compiler engine? Like this (apologies for bad knowledge of > this aspect of Cython): > > (Lot's of stuff that doesn't work, just scetching wildly...) > > cdef class std_vector_int: > def __compile_index_assign__(lefthand, indices, righthand): > return "%s[%s] = %s;" % (as_c(lefthand), as_c(indices), as_c > (righthand)) > def __compile_index_lookup__(lefthand, indices): > return "%s[%s];" % (as_c(lefthand), as_c(indices)) > def __compile_assign__(lefthand, rightand):... > > cdef class numpy.numarray ...: > def __compile_index_assign__(lefthand, indices, righthand): > l = indices.split(":") > if len(l) != int(lefthand.type_params[0]): raise ValueError(...) > idx = " + ".join([as_c(x) + " * stride[%d]" % d for x in zip(l, > range(len(l))]) > return "%s[%s] = %s;" % (as_c(lefthand), idx, as_c(righthand)) No, nothing like that has been attempted (or even discussed before). It is an interesting idea. I see potential for lots of subtle bugs (e.g. refcounting errors). I think if we can reduce operators (and conversions) to a specified function call (pass the operator to C++) there will be little, if any, need for something like this. I may be overly optimistic for things like making sure dimensions match however. > With the "macro as cdef"-trick earlier in the mailing list this > certainly doesn't seem *too* far-fetched for me... That code is simply evil. > > >> >> These data types can be declared, right now, by doing. >> >> cdef extern from "numpy.h": >> cdeftype int uint8 > Ah, right :-) > > >>> >>> @Compile >>> def myfunc(a: uint8, b: array(2, uint8), c: int = 10): >>> d: ptr(int) = &a >>> print a, b, c, d >>> >> >> One thing I really like about the cdef keyword is that it makes it >> very clear what is special Cython commands, and what is not. I also >> find this harder to read (and parse). (BTW, what's the @Compile >> supposed to mean?) However, if people want to move in this >> direction I >> wouldn't protest against it as an alternative. > The idea was to focus on making Cython usable for numerical > computation > -- and for that use (as opposed to making wrappers for C code) you do > NOT want people to feel that it is not part of Python, but rather > treat > it as a "typed Python that will run fast". The use case is simply > somebody that uses NumPy for everything, but comes over this one > operation that a) isn't in the libraries, b) can't be vectorized but > must be implemented as for-loops. If they can simply drop in a > @Compile > decorator and add some types and get on with their work, the pure > convenience of it might make Cython a regular tool for these kind of > things... This is a good point. > @Compile simply uses the standard Python function decorator syntax. > The > @Compile might be @Cython instead actually. The idea is that while > Python syntax allows for decorating function arguments, it doesn't say > anything about what they're for (documentation? something else?), > so you > would often have a function decorator that acts on them, and this > serves > to document that the function argument decorators are Cython types. > > Long term, @Compile might be used to replace the function with > calls to > compiled code in Python-land, and to extract functions to compile > from a > py-file in Cython-land, so that one can mix Cython and Python in one > source file. Also, it can take the options that Cython needs (ie > @Compile(native=True, except=1) instead of cdef ... except ). If we can come up with a good syntax for using decorators to express all Cython options, that could have some advantages (though I would still keep the old way as well). For example, one could debug the algorithm as a pure Python program which has a faster turnaround time and more tools (pylint, prun, etc) and then compile it when one is fairly confident that all is working well. Perhaps something could even be done with ctypes to actually simulate library wrapping and everything in python mode. Mixing Python and Cython code in the same file is far in the future (as I see it) and seems to have little advantage over just having the whole file be compiled. - Robert From ondrej at certik.cz Wed Mar 5 00:04:45 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 5 Mar 2008 00:04:45 +0100 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> <47CDC4BE.7050608@student.matnat.uio.no> <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> Message-ID: <85b5c3130803041504h3124e5dcmbf93c48cf7f5d398@mail.gmail.com> On Tue, Mar 4, 2008 at 11:15 PM, Robert Bradshaw wrote: > > On Mar 4, 2008, at 1:53 PM, Dag Sverre Seljebotn wrote: > > > > >> buffers, then this could be provided automatically. Overloading the = > >> operator would work for the one direction, and perhaps something > >> similar would be provided to do an "convert self to other type" > >> declaration. > > Right. This will keep me thinking :-) > > > > Has there been any thoughts on doing something as brute as allowing > > Python code in pxd files that is simply called compile-time and > > returns > > strings of C code? Or if not there, as a kind of plugin- > > architecture for > > the Cython compiler engine? Like this (apologies for bad knowledge of > > this aspect of Cython): > > > > (Lot's of stuff that doesn't work, just scetching wildly...) > > > > cdef class std_vector_int: > > def __compile_index_assign__(lefthand, indices, righthand): > > return "%s[%s] = %s;" % (as_c(lefthand), as_c(indices), as_c > > (righthand)) > > def __compile_index_lookup__(lefthand, indices): > > return "%s[%s];" % (as_c(lefthand), as_c(indices)) > > def __compile_assign__(lefthand, rightand):... > > > > cdef class numpy.numarray ...: > > def __compile_index_assign__(lefthand, indices, righthand): > > l = indices.split(":") > > if len(l) != int(lefthand.type_params[0]): raise ValueError(...) > > idx = " + ".join([as_c(x) + " * stride[%d]" % d for x in zip(l, > > range(len(l))]) > > return "%s[%s] = %s;" % (as_c(lefthand), idx, as_c(righthand)) > > No, nothing like that has been attempted (or even discussed before). > It is an interesting idea. I see potential for lots of subtle bugs > (e.g. refcounting errors). I think if we can reduce operators (and > conversions) to a specified function call (pass the operator to C++) > there will be little, if any, need for something like this. I may be > overly optimistic for things like making sure dimensions match however. > > > > With the "macro as cdef"-trick earlier in the mailing list this > > certainly doesn't seem *too* far-fetched for me... > > That code is simply evil. > > > > > > > > >> > >> These data types can be declared, right now, by doing. > >> > >> cdef extern from "numpy.h": > >> cdeftype int uint8 > > Ah, right :-) > > > > > >>> > >>> @Compile > >>> def myfunc(a: uint8, b: array(2, uint8), c: int = 10): > >>> d: ptr(int) = &a > >>> print a, b, c, d > >>> > >> > >> One thing I really like about the cdef keyword is that it makes it > >> very clear what is special Cython commands, and what is not. I also > >> find this harder to read (and parse). (BTW, what's the @Compile > >> supposed to mean?) However, if people want to move in this > >> direction I > >> wouldn't protest against it as an alternative. > > The idea was to focus on making Cython usable for numerical > > computation > > -- and for that use (as opposed to making wrappers for C code) you do > > NOT want people to feel that it is not part of Python, but rather > > treat > > it as a "typed Python that will run fast". The use case is simply > > somebody that uses NumPy for everything, but comes over this one > > operation that a) isn't in the libraries, b) can't be vectorized but > > must be implemented as for-loops. If they can simply drop in a > > @Compile > > decorator and add some types and get on with their work, the pure > > convenience of it might make Cython a regular tool for these kind of > > things... > > This is a good point. > > > > @Compile simply uses the standard Python function decorator syntax. > > The > > @Compile might be @Cython instead actually. The idea is that while > > Python syntax allows for decorating function arguments, it doesn't say > > anything about what they're for (documentation? something else?), > > so you > > would often have a function decorator that acts on them, and this > > serves > > to document that the function argument decorators are Cython types. > > > > Long term, @Compile might be used to replace the function with > > calls to > > compiled code in Python-land, and to extract functions to compile > > from a > > py-file in Cython-land, so that one can mix Cython and Python in one > > source file. Also, it can take the options that Cython needs (ie > > @Compile(native=True, except=1) instead of cdef ... except ). > > If we can come up with a good syntax for using decorators to express > all Cython options, that could have some advantages (though I would > still keep the old way as well). For example, one could debug the > algorithm as a pure Python program which has a faster turnaround time > and more tools (pylint, prun, etc) and then compile it when one is > fairly confident that all is working well. Perhaps something could > even be done with ctypes to actually simulate library wrapping and > everything in python mode. Yes, we were discussing exactly this with Travis Oliphant at Sage Days 8, it would be very cool, if a "pure python" code could be compiled. You can use decorators, or even python comments for specifying the types. Definitely it would be very useful, but it's quite a lot of work to get everything right. Ondrej From jek-gmane at kleckner.net Wed Mar 5 00:47:59 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Tue, 04 Mar 2008 15:47:59 -0800 Subject: [Cython] Embedding Pyrex/Cython tutorial Message-ID: I just ran across David McNab's tutorial about embedding Pyrex here: http://www.freenet.org.nz/python/embeddingpyrex/ Very interesting, thanks. David, would it make sense to put that into the Cython wiki? It would be front and center with more visibility. Cheers - Jim From robertwb at math.washington.edu Wed Mar 5 00:53:58 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 15:53:58 -0800 Subject: [Cython] Embedding Pyrex/Cython tutorial In-Reply-To: References: Message-ID: Yes, please put it on the wiki. - Robert On Mar 4, 2008, at 3:47 PM, Jim Kleckner wrote: > I just ran across David McNab's tutorial about embedding Pyrex here: > http://www.freenet.org.nz/python/embeddingpyrex/ > > Very interesting, thanks. > > David, would it make sense to put that into the Cython wiki? > > It would be front and center with more visibility. > > Cheers - Jim > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From jek-gmane at kleckner.net Wed Mar 5 01:29:37 2008 From: jek-gmane at kleckner.net (Jim Kleckner) Date: Tue, 04 Mar 2008 16:29:37 -0800 Subject: [Cython] Documentation link broken In-Reply-To: <9FC12853-9BC7-4E38-82EA-8002FBE9B195@math.washington.edu> References: <479D3921.9030006@martincmartin.com> <9FC12853-9BC7-4E38-82EA-8002FBE9B195@math.washington.edu> Message-ID: Robert Bradshaw wrote: > Thanks for the note--looks like he renamed the file. I just fixed it. This front page duplicates the wiki front page and is subject to divergence: ( compare http://www.cython.org/ with http://wiki.cython.org/Cython ) It would probably be better to concentrate on one or the other rather than keeping both consistent and choosing the wiki page means that others can fix it when they see it as well. From dagss at student.matnat.uio.no Wed Mar 5 01:31:36 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 5 Mar 2008 01:31:36 +0100 (CET) Subject: [Cython] Ideas developed: Code generator plugin architecture In-Reply-To: <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> References: <47CDB70A.7080903@student.matnat.uio.no> <9AA09672-8B1E-453C-AD37-624F2C36ECD4@math.washington.edu> <47CDC4BE.7050608@student.matnat.uio.no> <5003A34A-6DCC-4C9D-9463-ACA63603C38B@math.washington.edu> Message-ID: <54349.193.157.237.39.1204677096.squirrel@webmail.uio.no> The ideas have now evolved so much that a new subject was necesarry. I've written up this (but not linked to it anywhere, perhaps a "Far-out ideas" section could be added or something)... http://wiki.cython.org/CodeGeneratorPluginArchitecture It is simply an expansion of the last idea of indexing operators. Now, I haven't even looked at Cython source so I have no idea if this is at all feasible, if it requires a full redesign or reimplementation or whatever. However, what I do see is that it would solve any problems with wrapping any C++, STL, NumPy, ... code for ever, and it doesn't look like an awfully far fetch from what Cython is already doing. (As I say in the document, it is a very imperative rather than declarative approach, but if it can solve the problems quickly then it should be forward-compatible with more powerful declarative approaches anyway, so whatever gives the best results...) I'll calm down now. :-) Dag Sverre From robertwb at math.washington.edu Wed Mar 5 01:36:08 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 16:36:08 -0800 Subject: [Cython] Documentation link broken In-Reply-To: References: <479D3921.9030006@martincmartin.com> <9FC12853-9BC7-4E38-82EA-8002FBE9B195@math.washington.edu> Message-ID: Yes. I would like to greatly simplify the front page, and have it be mostly static stuff (and a "safe" link to download the source). It's just a matter of finding time. Timothy Clemens came up with a proposed design. http:// cython.timothyclemans.com/ I'd think there's little enough stuff that we should make it all available without any clicking though. - Robert On Mar 4, 2008, at 4:29 PM, Jim Kleckner wrote: > Robert Bradshaw wrote: >> Thanks for the note--looks like he renamed the file. I just fixed it. > > This front page duplicates the wiki front page and is subject to > divergence: > ( compare http://www.cython.org/ with http://wiki.cython.org/ > Cython ) > > It would probably be better to concentrate on one or the other rather > than keeping both consistent and choosing the wiki page means that > others can fix it when they see it as well. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Wed Mar 5 07:34:00 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 4 Mar 2008 22:34:00 -0800 Subject: [Cython] [sage-devel] Cython benchmarks? In-Reply-To: <46ba3d380803042221j103b73bn22239d3fc37036d3@mail.gmail.com> References: <46ba3d380803042221j103b73bn22239d3fc37036d3@mail.gmail.com> Message-ID: That all depends on how it is written. If you are using a lot of python data types, it will be slower. If not, you can write Cython that translates to nearly pure C (and runs at that speed). - Robert On Mar 4, 2008, at 10:21 PM, Carlo Hamalainen wrote: > > Hi, > > I have this request for Cython benchmarks on a blog post of mine - is > there any data out there, specifically comparing Cython to C/C++? > > -- > Carlo Hamalainen > http://carlo-hamalainen.net > > > > ---------- Forwarded message ---------- > http://carlo-hamalainen.net/blog/?p=19 > Author : Robert Samal > E-mail : rsamal at sfu.ca > > Does somebody else have some experience in how cython compares > with C/C++? Every once in a while I need to do some computation > (something NP-complete or worse in general, so it > usually ends up as an ugly backtracking). I'd be happy to do > everything from within Sage (and using python/cython), but I'm not > sure, if it is fast enough (or if it getting fast enough, I suppose > that cython is improving gradually). > > --~--~---------~--~----~------------~-------~--~----~ > To post to this group, send email to sage-devel at googlegroups.com > To unsubscribe from this group, send email to sage-devel- > unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/ > group/sage-devel > URLs: http://www.sagemath.org > -~----------~----~----~----~------~----~------~--~--- From stefan_ml at behnel.de Wed Mar 5 08:40:05 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 5 Mar 2008 08:40:05 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> <5725.194.114.62.68.1204646979.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <17856.194.114.62.68.1204650475.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <122F497C-84FA-4BA2-A8E3-F1AEFF9A6391@math.washington.edu> Message-ID: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Is operator the real issue here? In the std::vector example, there is > 'at', > which does (almost) the same thing as operator[]. My problem is that > both 'at' and 'operator[]' return int&, and to use them you need to > say 'at[i]=j', which I don't know how to do in cython (except for the > macro > version above, or to write additional wrapper code in more c++ headers). If you declare a variable cdef int* myvar Cython allows you to say myvar[5] = 1 Is that what you want? Stefan From stefan_ml at behnel.de Wed Mar 5 09:17:19 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 5 Mar 2008 09:17:19 +0100 (CET) Subject: [Cython] Cython benchmarks? In-Reply-To: References: <46ba3d380803042221j103b73bn22239d3fc37036d3@mail.gmail.com> Message-ID: <30539.194.114.62.67.1204705039.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Mar 4, 2008, at 10:21 PM, Carlo Hamalainen wrote: >> I have this request for Cython benchmarks on a blog post of mine - is >> there any data out there, specifically comparing Cython to C/C++? >> >> ---------- Forwarded message ---------- >> Does somebody else have some experience in how cython compares >> with C/C++? Every once in a while I need to do some computation >> (something NP-complete or worse in general, so it >> usually ends up as an ugly backtracking). I'd be happy to do >> everything from within Sage (and using python/cython), but I'm not >> sure, if it is fast enough (or if it getting fast enough, I suppose >> that cython is improving gradually). > > That all depends on how it is written. If you are using a lot of > python data types, it will be slower. If not, you can write Cython > that translates to nearly pure C (and runs at that speed). I think the best way to do this is to let Cython generate the C code and then look through the time critical parts to check for obvious problems, such as multiple back and forth conversion between C types and Python types or integer loops that run with Python run variables. Usually, this can be fixed by a couple of "cdef" declarations in the Cython code, which then translates to more optimal C code. I don't think a general benchmark is of much value here, as it would not catch the most common problems like the ones above. And if you want to optimise, there is no better way than timing your own code, which the timeit module in Python's stdlib makes easy enough to do. Just write a Cython module with a Python ("def") function that does exactly what you want to benchmark, and then call it through timeit. That's a one-liner. I added a quick blog entry on this (currently untested, but I'll fix that as soon as I get to it). You should get the idea anyway. http://behnel.de/cgi-bin/weblog_basic/index.php?p=17 Stefan From stefan_ml at behnel.de Wed Mar 5 13:36:28 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 5 Mar 2008 13:36:28 +0100 (CET) Subject: [Cython] wrapping c++ In-Reply-To: <200803050609.34160.ndbecker2@gmail.com> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <200803050609.34160.ndbecker2@gmail.com> Message-ID: <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: >> Neal Becker wrote: >> > Is operator the real issue here? In the std::vector example, there is >> > 'at', >> > which does (almost) the same thing as operator[]. My problem is that >> > both 'at' and 'operator[]' return int&, and to use them you need to >> > say 'at[i]=j', which I don't know how to do in cython (except for the >> > macro >> > version above, or to write additional wrapper code in more c++ >> headers). > > Let's say we have a c++ class intvec, it has > int& at(int index); I just looked that up, int& is a C++ reference here (didn't know that notation). What I would do is: create a header file and use it to define macros that do C++ operations on references in a function-like style. Things like the SET() 'function' that William proposed. Then, you define these macros as being external functions in a .pxd file, and cimport them into your .pyx file. Then you can use them in your Cython code as normal functions, and Cython will generate the macro calls for you, that the C preprocessor will translate into the correct operations. As Robert suggested, enhancing Cython's syntax to understand references would be helpful in the long term. As for the return type of your at() function, have you tried declaring it as returning "int*" instead? What does Cython do in this case? Stefan From dagss at student.matnat.uio.no Wed Mar 5 13:53:35 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 Mar 2008 13:53:35 +0100 Subject: [Cython] wrapping c++ In-Reply-To: <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <200803050609.34160.ndbecker2@gmail.com> <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <47CE97CF.7050004@student.matnat.uio.no> > > As for the return type of your at() function, have you tried declaring it > as returning "int*" instead? What does Cython do in this case? > C++ references doesn't work that way, I don't think that would work. C++ code would look like this: vector v(10); v.at(3) = 4; If you declare the return type to be int* you simply have a problem... Dag Sverre From dagss at student.matnat.uio.no Wed Mar 5 13:56:01 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 Mar 2008 13:56:01 +0100 Subject: [Cython] wrapping c++ In-Reply-To: <47CE97CF.7050004@student.matnat.uio.no> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <200803050609.34160.ndbecker2@gmail.com> <4863.194.114.62.67.1204720588.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <47CE97CF.7050004@student.matnat.uio.no> Message-ID: <47CE9861.8060301@student.matnat.uio.no> > > If you declare the return type to be int* you simply have a problem... Aw...sorry, hit the send combination while in the middle of typing! What I meant to say is that it won't solve the problem - Cython will treat it as a pointer type. AFAIK the only way to assign to an lvalue in C is if you dereference it, ie *lvalueReturningFunc() = rvalue; But references have automatic dereferencing, so the syntax above can never work, and the needed syntax: lvalueReturningFunc() = rvalue is not legal C IIRC. So macros are the way to go. -- Dag Sverre From stefan_ml at behnel.de Wed Mar 5 20:18:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Mar 2008 20:18:07 +0100 Subject: [Cython] pybench shootout between Cython 0.9.6.12+ and Python 2.5.1 Message-ID: <47CEF1EF.1080701@behnel.de> Hi, I got most of pybench running (as shipped with Python 2.5.1), just had to rename the .py files to .pyx (except Setup.py) and fix some "*" imports in the configuration, Cython does not allow that. Benchmark suites that run: - Arithmetic - Constructs - Lists - Tuples - Dict - Exceptions - Imports - Strings - Numbers - Unicode Benchmark suites that do not run: - Calls - Lookups - Instances - NewInstances The main problem is that they define classes and functions dynamically, which is not currently supported by Cython. The attached text file compares the results of a stock Python 2.5.1 from Ubuntu Gutsy to a Cython compiled version of the benchmarks. Positive differences mean that Python is slower than Cython. (I couldn't manage to get the opposite comparison running due to pickle problems). The end result is that Cython is almost 1/3 faster than the Python interpreter for plain Python code. Especially control structures like if/else and loops gain a lot from compilation, even for plain Python loops. BIG FAT WARNING: Please do not overrate these numbers, this is an unfair comparison between an interpreter and a compiler that do not even understand the same language. Also, I didn't compile Python myself, so the compiler optimisations differ between the two. However, I did not alter the benchmarks themselves in any way, so all benchmark functions use exactly the same Python source code in both runs. The "0ms" that Cython gives you in a couple of tests are actually real. They mean that the timings were not measurable compared to the overhead of the benchmark call itself, which is credible for an empty try-except, for example, or the comparison of numbers. Have fun, Stefan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cybench-compare.txt Url: http://codespeak.net/pipermail/cython-dev/attachments/20080305/eec8b962/attachment-0001.txt From wstein at gmail.com Wed Mar 5 20:54:17 2008 From: wstein at gmail.com (William Stein) Date: Wed, 5 Mar 2008 11:54:17 -0800 Subject: [Cython] pybench shootout between Cython 0.9.6.12+ and Python 2.5.1 In-Reply-To: <47CEF1EF.1080701@behnel.de> References: <47CEF1EF.1080701@behnel.de> Message-ID: <85e81ba30803051154k75908489j6a1db8a2ec5a16c7@mail.gmail.com> On Wed, Mar 5, 2008 at 11:18 AM, Stefan Behnel wrote: > Hi, > > I got most of pybench running (as shipped with Python 2.5.1), just had to > rename the .py files to .pyx (except Setup.py) and fix some "*" imports in the > configuration, Cython does not allow that. > > Benchmark suites that run: > - Arithmetic > - Constructs > - Lists > - Tuples > - Dict > - Exceptions > - Imports > - Strings > - Numbers > - Unicode > > Benchmark suites that do not run: > - Calls > - Lookups > - Instances > - NewInstances > > The main problem is that they define classes and functions dynamically, which > is not currently supported by Cython. > > The attached text file compares the results of a stock Python 2.5.1 from > Ubuntu Gutsy to a Cython compiled version of the benchmarks. Positive > differences mean that Python is slower than Cython. (I couldn't manage to get > the opposite comparison running due to pickle problems). > > The end result is that Cython is almost 1/3 faster than the Python interpreter > for plain Python code. Especially control structures like if/else and loops > gain a lot from compilation, even for plain Python loops. Just out of curiosity could you use Pyrex (instead of Cython) to run all the same benchmarks? I'm very curious how much the optimizations that were added to Cython have impacted performance on these sorts of things. > > BIG FAT WARNING: Please do not overrate these numbers, this is an unfair > comparison between an interpreter and a compiler that do not even understand > the same language. Also, I didn't compile Python myself, so the compiler > optimisations differ between the two. However, I did not alter the benchmarks > themselves in any way, so all benchmark functions use exactly the same Python > source code in both runs. > > The "0ms" that Cython gives you in a couple of tests are actually real. They > mean that the timings were not measurable compared to the overhead of the > benchmark call itself, which is credible for an empty try-except, for example, > or the comparison of numbers. > > Have fun, > Stefan > > > ************************************************************************ > **This file compares Python 2.5.1 (this) with Cython 0.9.6.12 (other) ** > ************************************************************************ > > ------------------------------------------------------------------------------- > PYBENCH 2.0 > ------------------------------------------------------------------------------- > * using Python 2.5.1 > * disabled garbage collection > * system check interval set to maximum: 2147483647 > * using timer: time.time > > Calibrating tests. Please wait... > > Running 10 round(s) of the suite at warp factor 10: > > * Round 1 done in 6.541 seconds. > * Round 2 done in 6.499 seconds. > * Round 3 done in 6.634 seconds. > * Round 4 done in 6.501 seconds. > * Round 5 done in 6.558 seconds. > * Round 6 done in 6.581 seconds. > * Round 7 done in 6.521 seconds. > * Round 8 done in 6.567 seconds. > * Round 9 done in 6.503 seconds. > * Round 10 done in 6.502 seconds. > > ------------------------------------------------------------------------------- > Benchmark: 2008-03-05 17:01:02 > ------------------------------------------------------------------------------- > > Rounds: 10 > Warp: 10 > Timer: time.time > > Machine Details: > Platform ID: Linux-2.6.22-14-generic-i686-with-debian-lenny-sid > Processor: > > Python: > Executable: /usr/bin/python > Version: 2.5.1 > Compiler: GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2) > Bits: 32bit > Build: Oct 5 2007 13:36:32 (#r251:54863) > Unicode: UCS4 > > > ------------------------------------------------------------------------------- > Comparing with: ../cybench/cybench.log > ------------------------------------------------------------------------------- > > Rounds: 10 > Warp: 10 > Timer: time.time > > Machine Details: > Platform ID: Linux-2.6.22-14-generic-i686-with-debian-lenny-sid > Processor: > > Python: > Executable: /usr/bin/python > Version: 2.5.1 > Compiler: GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2) > Bits: 32bit > Build: Oct 5 2007 13:36:32 (#r251:54863) > Unicode: UCS4 > > > Test minimum run-time average run-time > this other diff this other diff > ------------------------------------------------------------------------------- > CompareFloats: 160ms 0ms +7486272.6% 161ms 0ms +3534624.7% > CompareFloatsIntegers: 155ms 0ms +37093705.7% 155ms 0ms +6358029.8% > CompareIntegers: 152ms 0ms +32709538.5% 152ms 0ms +3928831.4% > CompareInternedStrings: 162ms 50ms +223.3% 168ms 51ms +226.4% > CompareLongs: 125ms 601ms -79.1% 126ms 603ms -79.2% > CompareStrings: 164ms 87ms +89.3% 164ms 90ms +83.0% > CompareUnicode: 165ms 107ms +54.6% 166ms 108ms +54.3% > ConcatStrings: 257ms 224ms +14.9% 279ms 243ms +15.1% > ConcatUnicode: 244ms 223ms +9.4% 245ms 224ms +9.6% > CreateNewInstances: 141ms 0ms n/a 142ms 0ms n/a > CreateStringsWithConcat: 189ms 153ms +23.1% 189ms 154ms +22.9% > CreateUnicodeWithConcat: 169ms 142ms +19.0% 172ms 146ms +17.9% > DictCreation: 137ms 63ms +116.3% 138ms 64ms +116.9% > DictWithFloatKeys: 180ms 190ms -5.4% 180ms 193ms -6.4% > DictWithIntegerKeys: 144ms 65ms +120.1% 145ms 66ms +119.3% > DictWithStringKeys: 142ms 51ms +180.1% 142ms 51ms +180.4% > ForLoops: 131ms 22ms +501.1% 132ms 22ms +501.9% > IfThenElse: 139ms 47ms +198.0% 139ms 47ms +197.2% > ListSlicing: 123ms 121ms +1.7% 123ms 121ms +1.5% > NestedForLoops: 154ms 17ms +832.3% 156ms 17ms +843.6% > SecondImport: 136ms 159ms -14.7% 137ms 161ms -15.0% > SecondPackageImport: 141ms 166ms -15.6% 142ms 168ms -15.5% > SecondSubmoduleImport: 174ms 199ms -12.9% 175ms 202ms -13.5% > SimpleComplexArithmetic: 168ms 139ms +20.7% 168ms 139ms +20.8% > SimpleDictManipulation: 150ms 85ms +76.6% 150ms 85ms +75.7% > SimpleFloatArithmetic: 180ms 85ms +112.5% 192ms 86ms +123.4% > SimpleIntFloatArithmetic: 134ms 52ms +157.0% 134ms 52ms +156.7% > SimpleIntegerArithmetic: 134ms 51ms +162.7% 134ms 51ms +162.5% > SimpleListManipulation: 137ms 37ms +271.2% 137ms 37ms +268.5% > SimpleLongArithmetic: 142ms 107ms +32.3% 142ms 108ms +31.6% > SmallLists: 148ms 78ms +89.2% 149ms 79ms +89.1% > SmallTuples: 142ms 75ms +88.0% 143ms 76ms +87.7% > StringMappings: 143ms 124ms +15.1% 144ms 126ms +14.6% > StringPredicates: 169ms 116ms +45.8% 169ms 119ms +42.3% > StringSlicing: 153ms 94ms +62.8% 157ms 99ms +58.3% > TryExcept: 122ms 0ms +10754178.9% 122ms 0ms +2514522.4% > TryRaiseExcept: 125ms 81ms +53.6% 125ms 83ms +51.2% > TupleSlicing: 144ms 116ms +23.5% 144ms 117ms +23.6% > UnicodeMappings: 142ms 140ms +1.8% 142ms 141ms +1.1% > UnicodePredicates: 146ms 108ms +35.8% 146ms 109ms +33.8% > UnicodeProperties: 134ms 93ms +43.9% 134ms 93ms +43.9% > UnicodeSlicing: 176ms 131ms +34.4% 178ms 135ms +31.8% > ------------------------------------------------------------------------------- > Totals: 6468ms 4398ms n/a 6541ms 4464ms n/a > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From ndbecker2 at gmail.com Tue Mar 4 16:00:04 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 4 Mar 2008 10:00:04 -0500 Subject: [Cython] wrapping c++ In-Reply-To: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> References: <85e81ba30803040654j7d112395o2014cac2179d6278@mail.gmail.com> Message-ID: <200803041000.04266.ndbecker2@gmail.com> On Tuesday 04 March 2008, William Stein wrote: > On Tue, Mar 4, 2008 at 6:39 AM, Neal Becker wrote: > > I've made some progress on learning to wrap std::vector. I've got a > > __getitem__, but I don't know what to do about __setitem__. > > cdef extern from "vector": > > pass > > > > cdef extern from "test1.hpp": > > ctypedef struct intvec "std::vector": > > void (* push_back)(int elem) > > inline int get "operator[]" (int i) > > > > intvec intvec_factory "std::vector"(int len) > > > > int my_sum "my_sum >"(intvec vec) > > > > > > > > cdef intvec v = intvec_factory(2) > > v.push_back(2) > > > > #print my_sum (v) > > > > cdef class intvec_wrap: > > cdef intvec vec > > def __init__(self): > > self.vec = intvec_factory (2) > > def __getitem__(self, int i): > > return self.vec.get (i) > > > > Problem is, operator[] (or at()) returns an int&. I don't have a clue > > what to do with this. > > There is an example of doing setitem directly in Cython with > std::vector > using C name specifiers in my talk on Cython at Enthought from two > days ago. See > the talk listed at the bottom of this page: > http://wiki.sagemath.org/days8/schedule > > Here's the example: > > cdef extern from "vector.h": > ctypedef unsigned long size_type > > ctypedef struct intvec "std::vector": > int get_entry "operator[]"(int n) > size_type size() > > intvec intvec_factory "std::vector"(int len) > > cdef extern from "": > void DEFINE_SET "#define SET(a,b,c) a[b]=c; //"() > void SET(intvec, int, int) > DEFINE_SET() > > > cdef intvec v = intvec_factory(10) > SET(v,2,5) > print v.get_entry(2) > print v.size() > > > ---------------- > > The key thing is the evil dirty trick to define a SET macro directly in > Cython. -- William Wow, that is ugly. From ndbecker2 at gmail.com Wed Mar 5 12:09:33 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 5 Mar 2008 06:09:33 -0500 Subject: [Cython] wrapping c++ In-Reply-To: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <44275.194.114.62.67.1204702805.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <200803050609.34160.ndbecker2@gmail.com> On Wednesday 05 March 2008, Stefan Behnel wrote: > Neal Becker wrote: > > Is operator the real issue here? In the std::vector example, there is > > 'at', > > which does (almost) the same thing as operator[]. My problem is that > > both 'at' and 'operator[]' return int&, and to use them you need to > > say 'at[i]=j', which I don't know how to do in cython (except for the > > macro > > version above, or to write additional wrapper code in more c++ headers). > > If you declare a variable > > cdef int* myvar > > Cython allows you to say > > myvar[5] = 1 > > Is that what you want? > > Stefan Does that help for a function that returns int&? Let's say we have a c++ class intvec, it has int& at(int index); From stefan_ml at behnel.de Wed Mar 5 22:13:44 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Mar 2008 22:13:44 +0100 Subject: [Cython] pybench shootout between Cython 0.9.6.12+ and Python 2.5.1 and Pyrex 0.9.6.4 In-Reply-To: <85e81ba30803051154k75908489j6a1db8a2ec5a16c7@mail.gmail.com> References: <47CEF1EF.1080701@behnel.de> <85e81ba30803051154k75908489j6a1db8a2ec5a16c7@mail.gmail.com> Message-ID: <47CF0D08.9040302@behnel.de> Hi, William Stein wrote: > On Wed, Mar 5, 2008 at 11:18 AM, Stefan Behnel wrote: >> I got most of pybench running (as shipped with Python 2.5.1), just had to >> rename the .py files to .pyx (except Setup.py) and fix some "*" imports in the >> configuration, Cython does not allow that. > > Just out of curiosity could you use Pyrex (instead of Cython) to run > all the same benchmarks? I'm very curious how much the optimizations > that were added to Cython have impacted performance on these > sorts of things. Want a number? 11% :) I also attached the sources I used. Stefan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pyrex-compare.txt Url: http://codespeak.net/pipermail/cython-dev/attachments/20080305/b7cf0764/attachment-0001.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: cybench.tgz Type: application/x-compressed-tar Size: 27088 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080305/b7cf0764/attachment-0001.bin From dg at pnylab.com Thu Mar 6 16:26:18 2008 From: dg at pnylab.com (Dan Gindikin) Date: Thu, 06 Mar 2008 10:26:18 -0500 Subject: [Cython] Thoughts on numerical computing/NumPy support Message-ID: <47D00D1A.8030603@pnylab.com> Hi Dag, I've just finished v1 of a preprocessor for pyrex/cython, called pex, that does (among other things) precisely what you describe for numpy ndarrays, you can say: cdef matmult(ndarray A, ndarray B): cdef int i,j,q cdef ndarray ans for i from 0<=i Nice!, very interesting. Myself I couldn't quite put my thoughts about this away and have spent too many hours the past days hacking on Cython source, and software architecture-wise it's been very stimulating. If my interest doesn't fade (which it better if I am to hand in my uni assignments in time...) I might have a prototype of my own ideas in a week, taking a few hours in-between now and then. The end-result should be the same thing, however my approach is very different. While pex is probably the way forward I want to pursue my own attempt a little further because I believe a preprocessor can only do so much before it ends up either lacking wanted capabilities because of missing type knowledge etc. (adding overloaded funtion support?), or duplicating much of what Cython already has. So my approach is instead adding another layer so that it is possible to program Cython at a "higher level" and create plugins for the kinds of thing you would want to preprocess for. (I didn't look up pex yet though....I don't have webaccess now...so I might have jumped to wrong conclusions.) My approach is of course very intrusive and the chances of a high enough success for inclusion in Cython is perhaps 10 percent or so (to make something up, and I'm not familiar with Cython processes...). But if you want to have a look it would still be very interesting to discuss with somebody who have taken a different approach...the notes are on the cython wiki (look on recentlychanged for my name...it is a lot of brain dump but should give an idea.) Dag Sverre Seljebotn -----Original Message----- From: Dan Gindikin Date: Thursday, Mar 6, 2008 4:26 pm Subject: Re: [Cython] Thoughts on numerical computing/NumPy support To: cython-dev at codespeak.net Hi Dag, > >I've just finished v1 of a preprocessor for pyrex/cython, called pex, that does >(among other things) precisely what you describe for numpy ndarrays, you can say: > >cdef matmult(ndarray A, ndarray B): > cdef int i,j,q > > cdef ndarray ans > > for i from 0<=i for j from 0<=j for q from 0<=q ans{i,j}=ans{i,j}+A{i,q}*B{q,j} > > return ans > >All of the {} accesses occur at C speed. We've been talking with Robert about moving this, and perhaps other features of pex into cython. > >-- >Dan >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From wstein at gmail.com Thu Mar 6 23:36:58 2008 From: wstein at gmail.com (William Stein) Date: Thu, 6 Mar 2008 14:36:58 -0800 Subject: [Cython] Thoughts on numerical computing/NumPy support In-Reply-To: <3287687773.2188279@smtp.netcom.no> References: <3287687773.2188279@smtp.netcom.no> Message-ID: <85e81ba30803061436o5586264dwc560dd6aa17c1d4e@mail.gmail.com> On 06 Mar 2008 22:36:00 +0100, Dag Sverre Seljebotn wrote: > Nice!, very interesting. > > Myself I couldn't quite put my thoughts about this away and have spent too many hours the past days hacking on Cython source, and software architecture-wise it's been very stimulating. If my interest doesn't fade (which it better if I am to hand in my uni assignments in time...) I might have a prototype of my own ideas in a week, taking a few hours in-between now and then. The end-result should be the same thing, however my approach is very different. > > While pex is probably the way forward I want to pursue my own attempt a little further because I believe a preprocessor can only do so much before it ends up either lacking wanted capabilities because of missing type knowledge etc. My impression talking to everybody and especially the PEX people is that *your* approach (not a preprocessor) is probably the way forward. PEX (and the examples and tests it includes) is excellent because it lays down a nice implementation that is actually extremely useful for real work of what we want to be able to do. But the actual implementation was basically the easiest way for the PEX people to get something up and running quickly without having to change Cython itself (which was an initial design goal for them for their first version, I think). > (adding overloaded funtion support?), or duplicating much of what Cython already has. So my approach is instead adding another layer so that it is possible to program Cython at a "higher level" and create plugins for the kinds of thing you would want to preprocess for. (I didn't look up pex yet though....I don't have webaccess now...so I might have jumped to wrong conclusions.) > > My approach is of course very intrusive and the chances of a high enough success for inclusion in Cython is perhaps 10 percent or so (to make something up, and I'm not familiar with Cython processes...). But if you want to have a look it would still be very interesting to discuss with somebody who have taken a different approach...the notes are on the cython wiki (look on recentlychanged for my name...it is a lot of brain dump but should give an idea.) > Maybe the first thing you do won't get in as is, but whatever you do I think it will be very helpful somehow. In any case, there are currently not enough Cython developers, so I'm very happy that you are getting involved! -- William From dg at pnylab.com Fri Mar 7 15:16:28 2008 From: dg at pnylab.com (Dan Gindikin) Date: Fri, 07 Mar 2008 09:16:28 -0500 Subject: [Cython] Thoughts on numerical computing/NumPy support Message-ID: <47D14E3C.3070509@pnylab.com> Dag, you are absolutely right, since Pex only does shallow type analysis, and has no access to Cython's internal state during compilation, there is a rather hard stop on what it is able to do. As William said, we wanted to get something up quickly and did not want to muck around Cython internals. I too think this sort functionality clearly belongs in the language proper, as you are describing. -- Dan From dagss at student.matnat.uio.no Fri Mar 7 21:08:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 07 Mar 2008 21:08:00 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" Message-ID: <47D1A0A0.9000105@student.matnat.uio.no> During my work to investigate how NumPy, C++ template etc. support could best be added I discovered a way of integrating such code that has very low impact on changing existing code and flow, yet does (I believe) have the capacity to give quite powerful results. I've made a prototype: - A framework in a 300 line diff - An example using the framework: A reimplementation of the good old for i in range(x, y) => for i from i <= ... transform, in 40 lines The whole thing and a long description is here: http://wiki.cython.org/enhancements/parsetreetransforms -- Dag Sverre From robertwb at math.washington.edu Fri Mar 7 21:31:25 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 7 Mar 2008 12:31:25 -0800 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D1A0A0.9000105@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> Message-ID: <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> This looks very, very interesting and promising. I was thinking about code transformations like this quite a while ago for optimization reasons. I think this is the way to go. - Robert On Mar 7, 2008, at 12:08 PM, Dag Sverre Seljebotn wrote: > During my work to investigate how NumPy, C++ template etc. support > could > best be added I discovered a way of integrating such code that has > very > low impact on changing existing code and flow, yet does (I believe) > have > the capacity to give quite powerful results. > > I've made a prototype: > - A framework in a 300 line diff > - An example using the framework: A reimplementation of the good old > for i in range(x, y) => for i from i <= ... transform, in 40 lines > > The whole thing and a long description is here: > > http://wiki.cython.org/enhancements/parsetreetransforms > > > -- > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From gfurnish at indirectproof.net Sat Mar 8 00:02:37 2008 From: gfurnish at indirectproof.net (Gary Furnish) Date: Fri, 7 Mar 2008 16:02:37 -0700 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> Message-ID: <706850310803071502m24e38116t64323549546d273b@mail.gmail.com> I agree. This is very well designed and should hopefully be the framework for building many of the numpy things without having to alter the core of cython. Why did you end up using > for relation2 in your example though? On 3/7/08, Robert Bradshaw wrote: > This looks very, very interesting and promising. I was thinking about > code transformations like this quite a while ago for optimization > reasons. I think this is the way to go. > > > - Robert > > > On Mar 7, 2008, at 12:08 PM, Dag Sverre Seljebotn wrote: > > > During my work to investigate how NumPy, C++ template etc. support > > could > > best be added I discovered a way of integrating such code that has > > very > > low impact on changing existing code and flow, yet does (I believe) > > have > > the capacity to give quite powerful results. > > > > I've made a prototype: > > - A framework in a 300 line diff > > - An example using the framework: A reimplementation of the good old > > for i in range(x, y) => for i from i <= ... transform, in 40 lines > > > > The whole thing and a long description is here: > > > > http://wiki.cython.org/enhancements/parsetreetransforms > > > > > > -- > > Dag Sverre > > > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Sat Mar 8 11:26:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 11:26:04 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <706850310803071502m24e38116t64323549546d273b@mail.gmail.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <706850310803071502m24e38116t64323549546d273b@mail.gmail.com> Message-ID: <47D269BC.2060109@student.matnat.uio.no> Gary Furnish wrote: > I agree. This is very well designed and should hopefully be the > framework for building many of the numpy things without having to > alter the core of cython. Why did you end up using > for relation2 in > your example though? > Apparently you read my example more carefully than I did :-) It should be < of course. (I didn't actually run any code generated with it so I missed it). Regarding NumPy: - I think type parameters should also be introduced in addition to Cython core. It's an easy job, one avoids a lot of hacks, and they have many potential uses (like C++ templates). - Then, this framework can be used to implement NumPy support. And I think it should be, because it is necasarry to gain experience in this before doing step d), - However, this has a drawback: One still has to know a lot of Cython internals, and when Cython internals change then the transforms will sometimes have to change as well, since the Cython parse tree is exported in "raw" form. - Ideally, one would therefore have yet another layer (built, of course, using this transform layer) that's more geared to specifically doing the transforms necesarry for special support for types. My (now a bit outdated) thoughts on that are on: http://wiki.cython.org/CodeGeneratorPluginArchitecture. But as I said I don't think NumPy support should wait for this layer, rather building NumPy support will show us how such a layer should look like. - Finally, having such a "type specific behaviour" layer will allow NumPy code to be seperated out of Cython development and allow it to live in the NumPy codebase instead (if they want to). Dag Sverre From dagss at student.matnat.uio.no Sat Mar 8 13:43:42 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 13:43:42 +0100 Subject: [Cython] Patch ready (was: Re: Done with prototype for "parse tree transforms") In-Reply-To: <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> Message-ID: <47D289FE.10108@student.matnat.uio.no> Robert Bradshaw wrote: > This looks very, very interesting and promising. I was thinking about > code transformations like this quite a while ago for optimization > reasons. I think this is the way to go. In that case, I've cleaned it up a little bit and now submit it as a patch against the current cython-devel repo (see attachment); I believe it is ready for inclusion in the cython-devel branch if you approve. It doesn't *do* anything currently, and really shouldn't affect behaviour if one doesn't use the new -T switch (so regression testing for this patch can be done simply by checking that the same code is generated for a lot of examples -- I didn't take time for this though). However placing it in the main repository means that more people than myself can start playing with applying transforms that actually does something. Changes: A little code has been added all over the place, but most of it doesn't "interact" at all with existing code, unless you count declaring new symbols. The diff is small enough that you can simply read it .. the "worst" parts: - Node.generate_function_definitions got an extra argument everywhere (but I grepped for usages and am pretty sure I fixed all calls, at least within the Compiler module). - FuncDefNode.generate_function_definitions has calls to the transforms added: self.declare_arguments(lenv) + transforms.run('before_analyse_function', self, env=env, lenv=lenv, genv=genv) self.body.analyse_declarations(lenv) self.body.analyse_expressions(lenv) + transforms.run('after_analyse_function', self, env=env, lenv=lenv, genv=genv) (Without program arguments, these will iterate an empty list and return.) Known problems: - Only two cut-points/phases for transforms are provided, and more are neeeded, especially one in-between type deduction and type coercion, but this requires more intrusive refactoring that is better left for a seperate commit. This gives a start though. - The lists of which attributes for each node constitute "children" in the tree can probably be tuned a bit, however this is best left for when one actually have a use-case (transform) making use of the node type in question. Dag Sverre -------------- next part -------------- A non-text attachment was scrubbed... Name: treetransform.diff Type: text/x-patch Size: 30685 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080308/2b779953/attachment-0001.bin From dagss at student.matnat.uio.no Sat Mar 8 14:11:13 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 14:11:13 +0100 Subject: [Cython] Patch ready In-Reply-To: <47D289FE.10108@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> Message-ID: <47D29071.5080207@student.matnat.uio.no> A new patch (towards the original repo, so just forget about the original one), it simply contains a little more documentation in Transform.py. (What's the proper way to submit patches anyway .. through the bug tracker?) Dag Sverre -------------- next part -------------- A non-text attachment was scrubbed... Name: treetransform2.diff Type: text/x-patch Size: 32455 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080308/bccecb42/attachment-0001.bin From wstein at gmail.com Sat Mar 8 16:00:57 2008 From: wstein at gmail.com (William Stein) Date: Sat, 8 Mar 2008 07:00:57 -0800 Subject: [Cython] [sage-devel] linear algebra in Cython In-Reply-To: References: Message-ID: <85e81ba30803080700p1134b3d7x7198fce382ceb833@mail.gmail.com> On Fri, Mar 7, 2008 at 6:52 PM, Vincent wrote: > > I would like to implement some Bayesian routines using Cython. I have > looked at some of the Cython examples but they don't really provide > enough information for a newbie like me. As a starting point i'd like > to implement bootstrapping of OLS parameter standard errors. That > would give me an idea of how to implement (1) linear algebra in > Cython, i assume by calling routines from Numpy in some way, Much Python code will "just work" in Cython. You an just directly use numpy as you would from python, and it will work but be potentially slow depending on what you're doing. There is very active work going on in the Cython project to make it easy to write code that works blazingly quickly directly with numpy arrays, but that is _not_ in Cython yet, unfortunately, and we're not even sure what form it will take. > (2) > random number generators for sampling, numpy/scipy have a lot of those. > and (3) running loops. See David Roe's example from the previous email and browse wiki.cython.org > > I anyone has any (simple) examples of how to do something like this in > Cython i'd love to see them. Also, if there are better ways to do this > than through Cython i'd luv to hear your suggestions. > > Thanks, > > Vincent > > --~--~---------~--~----~------------~-------~--~----~ > To post to this group, send email to sage-devel at googlegroups.com > To unsubscribe from this group, send email to sage-devel-unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/group/sage-devel > URLs: http://www.sagemath.org > -~----------~----~----~----~------~----~------~--~--- > > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From dagss at student.matnat.uio.no Sat Mar 8 16:48:32 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 16:48:32 +0100 Subject: [Cython] Prototype for parametrized types Message-ID: <47D2B550.9060709@student.matnat.uio.no> Now that pipeline transforms are in place it is at least theoretically possible to take advantage of this: http://wiki.cython.org/enhancements/typeparameters Don't apply this to cython-dev just now, it's meant mostly for prototype + if somebody else wants to take over and finish it. This might mark the end of my coding sprint for now -- I won't break my neck by trying to go all the way to numpy arrays right away. -- Dag Sverre From martin at martincmartin.com Sat Mar 8 17:09:01 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 11:09:01 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D1A0A0.9000105@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> Message-ID: <47D2BA1D.1040202@martincmartin.com> Great! How does it compare to Lisp's macros? Best, Martin Dag Sverre Seljebotn wrote: > During my work to investigate how NumPy, C++ template etc. support could > best be added I discovered a way of integrating such code that has very > low impact on changing existing code and flow, yet does (I believe) have > the capacity to give quite powerful results. > > I've made a prototype: > - A framework in a 300 line diff > - An example using the framework: A reimplementation of the good old > for i in range(x, y) => for i from i <= ... transform, in 40 lines > > The whole thing and a long description is here: > > http://wiki.cython.org/enhancements/parsetreetransforms > > From dagss at student.matnat.uio.no Sat Mar 8 17:18:19 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 17:18:19 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BA1D.1040202@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> Message-ID: <47D2BC4B.7040008@student.matnat.uio.no> Martin C. Martin wrote: > Great! > > How does it compare to Lisp's macros? I'll try to answer, but I'm guessing about your assumptions so do ask again if I miss. This is work that is only relevant to the internal Cython compiler, so it doesn't compare (or compares very poorly) with LISP. Ie when transforming the parse tree one is transforming the input Cython source to output C source, but the transform itself doesn't live within that parse tree at all. If what you are asking instead is: How easy is it to write transforms compared to writing LISP macros, then the answer is still that it compares poorly, but then the goals are rather different too (the number of transforms will be max ~30 I think, and the number of LISP programs are... :-) ) Dag Sverre From martin at martincmartin.com Sat Mar 8 17:23:16 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 11:23:16 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BC4B.7040008@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> Message-ID: <47D2BD74.7030104@martincmartin.com> Ok, thanks a lot. My day job is programming Lisp, and since I started two years ago, I've seen the light about a number of things, including macros. It would be cool to have access to the Abstract Syntax Tree of a Cython expression/statement/function/etc., and be able to manipulate it at compile time. But it sounds like that's a whole other project. :) Best, Martin Dag Sverre Seljebotn wrote: > Martin C. Martin wrote: >> Great! >> >> How does it compare to Lisp's macros? > I'll try to answer, but I'm guessing about your assumptions so do ask > again if I miss. > > This is work that is only relevant to the internal Cython compiler, so > it doesn't compare (or compares very poorly) with LISP. Ie when > transforming the parse tree one is transforming the input Cython source > to output C source, but the transform itself doesn't live within that > parse tree at all. > > If what you are asking instead is: How easy is it to write transforms > compared to writing LISP macros, then the answer is still that it > compares poorly, but then the goals are rather different too (the number > of transforms will be max ~30 I think, and the number of LISP programs > are... :-) ) > > > Dag Sverre From dagss at student.matnat.uio.no Sat Mar 8 17:28:34 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Mar 2008 17:28:34 +0100 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BD74.7030104@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> Message-ID: <47D2BEB2.2080106@student.matnat.uio.no> Martin C. Martin wrote: > Ok, thanks a lot. > > My day job is programming Lisp, and since I started two years ago, > I've seen the light about a number of things, including macros. It > would be cool to have access to the Abstract Syntax Tree of a Cython > expression/statement/function/etc., and be able to manipulate it at > compile time. But it sounds like that's a whole other project. :) Hmm. I'm still not sure whether I've come across right :-) Accessing the abstract syntax tree is exactly what this is about, and you can do that (however the syntax for doing would seem rather complex and messy compared to with LISP, though a lot better I believe than what has been going on until now). The problem is that the code that accesses the syntax tree has to be written in Python and inserted into the compiler at launch time -- you cannot analyse yourself. In Python syntax, it is possible to do something remotely similar to: part-of-project.py: def myfunc(...): ... part-of-compiler.py: def modify_syntax_tree(tree): ... modify_syntax_tree(myfunc) However, you can NOT do modify_syntax_tree(modify_syntax_tree) or anything like that, which is why I believe it doesn't compare with LISP. Dag Sverre From ndbecker2 at gmail.com Sat Mar 8 18:27:47 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 08 Mar 2008 12:27:47 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> Message-ID: Martin C. Martin wrote: > Ok, thanks a lot. > > My day job is programming Lisp, and since I started two years ago, I've > seen the light about a number of things, including macros. It would be > cool to have access to the Abstract Syntax Tree of a Cython > expression/statement/function/etc., and be able to manipulate it at > compile time. But it sounds like that's a whole other project. :) > Wow! Someone is paid to program lisp? From stefan_ml at behnel.de Sat Mar 8 18:37:17 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Mar 2008 18:37:17 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D2B550.9060709@student.matnat.uio.no> References: <47D2B550.9060709@student.matnat.uio.no> Message-ID: <47D2CECD.40104@behnel.de> Dag Sverre Seljebotn wrote: > http://wiki.cython.org/enhancements/typeparameters Why do you think a new syntax for type parametrisation would be better than reusing (parts of) PEP 3107? http://www.python.org/dev/peps/pep-3107/ Stefan From robertwb at math.washington.edu Sat Mar 8 21:08:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 Mar 2008 12:08:22 -0800 Subject: [Cython] Enhancements proposals Message-ID: As some of you have probably noticed, I put up a new page on the wiki summarizing many of the enhancements that have been discussed here, and would be good to think about in light of the upcoming Sage development days (http://wiki.sagemath.org/dev1 a big part of which will be focused on improving Cython). There are also some very good proposals by Dag Seljebotn. The proposals are in varying degrees of completion--ranging from vague wishlist items to actual implementations. Feedback and input would be very beneficial, I would like these items to approach the specificity of PEP's and think that many if not most of them could be hammered out and implemented by/during the coding springs this June. http://wiki.cython.org/enhancements - Robert From robertwb at math.washington.edu Sat Mar 8 21:29:06 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 Mar 2008 12:29:06 -0800 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D2CECD.40104@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> Message-ID: <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> I think using parentheses for type parameterization is the most natural thing to do as well. (The only other viable syntax that comes to mind is angle brackets, and that's only natural to C++ people). This makes it valid Python syntax as well. I think this is orthogonal to PEP 3107, which specifies where the type specification should go (and I agree we should support this syntax too). The proposal is that anywhere the compiler expects a valid type, a parameterized type would work as well. The more I think about it, the more the plugin approach seems to be the right one to take, though I hope a lot can be specified in a .pxd without having to write custom plugins. These could be distributed with Cython, though not as part of the core compiler (see http:// wiki.cython.org/declaration ) - Robert On Mar 8, 2008, at 9:37 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> http://wiki.cython.org/enhancements/typeparameters > > Why do you think a new syntax for type parametrisation would be > better than > reusing (parts of) PEP 3107? > > http://www.python.org/dev/peps/pep-3107/ > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Sat Mar 8 21:57:04 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Mar 2008 21:57:04 +0100 Subject: [Cython] Prototype for parametrized types In-Reply-To: <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> Message-ID: <47D2FDA0.2030403@behnel.de> Hi, Robert Bradshaw wrote: > I think using parentheses for type parameterization is the most natural > thing to do as well. (The only other viable syntax that comes to mind is > angle brackets, and that's only natural to C++ people). Ok, I buy that. It would even resemble decorators to a certain extent. > This makes it valid Python syntax as well. How is that? > I think this is orthogonal to PEP 3107, which specifies where the type > specification should go (and I agree we should support this syntax too). > The proposal is that anywhere the compiler expects a valid type, a > parameterized type would work as well. Would that also allow you to distinguish things like "list" and "any subtype of list"? > The more I think about it, the more the plugin approach seems to be the > right one to take, though I hope a lot can be specified in a .pxd > without having to write custom plugins. These could be distributed with > Cython, though not as part of the core compiler (see > http://wiki.cython.org/declaration ) You could even ship them with Numpy and other projects instead of integrating them with Cython. Cython will nicely do without Numpy, but if you want to access Numpy through Cython, you will need Numpy installed first, so that's the right place to put a numpy.pxd for Cython. Stefan (BTW, you really should stop top-posting ...) From robertwb at math.washington.edu Sat Mar 8 22:09:26 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 Mar 2008 13:09:26 -0800 Subject: [Cython] Prototype for parametrized types In-Reply-To: <47D2FDA0.2030403@behnel.de> References: <47D2B550.9060709@student.matnat.uio.no> <47D2CECD.40104@behnel.de> <2C09E5A5-8917-4717-855D-E6085B541F1F@math.washington.edu> <47D2FDA0.2030403@behnel.de> Message-ID: On Mar 8, 2008, at 12:57 PM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: >> I think using parentheses for type parameterization is the most >> natural >> thing to do as well. (The only other viable syntax that comes to >> mind is >> angle brackets, and that's only natural to C++ people). > > Ok, I buy that. It would even resemble decorators to a certain extent. > > >> This makes it valid Python syntax as well. > > How is that? foo(arg) is a valid cython expression, foo is not. The PEP specifies any valid expression can follow the : >> I think this is orthogonal to PEP 3107, which specifies where the >> type >> specification should go (and I agree we should support this syntax >> too). >> The proposal is that anywhere the compiler expects a valid type, a >> parameterized type would work as well. > > Would that also allow you to distinguish things like "list" and > "any subtype > of list"? Yes, but what good would it do to know something is a subtype of list? >> The more I think about it, the more the plugin approach seems to >> be the >> right one to take, though I hope a lot can be specified in a .pxd >> without having to write custom plugins. These could be distributed >> with >> Cython, though not as part of the core compiler (see >> http://wiki.cython.org/declaration ) > > You could even ship them with Numpy and other projects instead of > integrating > them with Cython. Cython will nicely do without Numpy, but if you > want to > access Numpy through Cython, you will need Numpy installed first, > so that's > the right place to put a numpy.pxd for Cython. Yep. But if we distributed with Cython then one wouldn't even have to tell the Cython compiler where to find the pxd files. Numpy is a bit of a special case, as they are actually going to use Cython and probably help write their own .pxd files. But for most libraries I think distributing them with Cython makes perfect sense (For example, I bet gmp is not about to shop Cython bindings to their library, for instance). > (BTW, you really should stop top-posting ...) Sorry... From dagss at student.matnat.uio.no Sun Mar 9 01:52:21 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 9 Mar 2008 01:52:21 +0100 (CET) Subject: [Cython] Patch ready (was: Re: Done with prototype for "parse tree transforms") In-Reply-To: <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> Message-ID: <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> > It would be nice to figure out a way that pluggins could be written > to be less brittle (i.e. not so dependent on the inner workings and > names of the Cython compiler) too, eventually. Totally agreed (and I've written about that too some places, especially that the -T switch will only ever be a developer tool). It's important for me that this transform framework isn't as much a plugin architecture as a change in the modularity of the compiler core itself. Currently (if I dare) the ambitions seems to be too high for the current code architecture and (in all modesty :-) ) I believe this approach goes a long way solving that problem. So this is the way I see it (and I'd guess you agree but I want it on the mailing list): - In an ideal world, it is simply wrong to have the methods acting on the parse tree implemented as methods on the parse tree itself. (See example below). Rather you want the parse tree to be "data" and seperate, componentized and well-defined transforms to be the vehicles of making the compilation happen. So ideally, the parse tree shouldn't be that brittle because there should be "nothing there" but information that is spec-ed according to the phases one is in. (However for plugin use it is definetely too complex still). - Working towards such an ideal code world is not the business one is in and is also very boring, so one makes compromises: Whenever new functionality is introduced, write it as transforms, whenever something new is impossible to write as transforms, make the minimal necesarry modifications towards making it possible to write is as transforms, but leave it at that. - I would however consider that type coercion should be split into a seperate CoercionTransform descending from the Transform class, rather than making a new coercion phase within the existing code-in-nodes paradigm. - At some point a new layer should definitely be written that allows simpler types of plugins. However (but I'm less sure of this than other things) I believe NumPy support and C++ template support can be written as transforms directly, because they are high priorities and to gain experience with what kind of plugin support would be most beneficial. - We should keep in the back of our heads that the transform problem has been solved in many ways in varying degrees of generalness (ick..too late in the night to look up the proper phrase). For instance it would be perfectly possible to export the tree to XML, run an XSLT transform, and parse the XML back, and then one could use XSLT for writing all the transforms (which would be much better in some cases than Python, and much worse in others). (I'm personally not that happy about XSLT in this particular case, though for many other uses it is perfect, but I'd do some research and look for libraries in general that can help write transforms. Something like doing XPath on this thing would help. Hmm...in fact there probably are Python XPath implementations that one could fit on our DOM, only question is whether it is worth the effort. If you don't know these just look them up). The important thing is that having a basic support for transforms doesn't say anything about different creative ways the transforms can be implemented in. Example for demonstration only: What if you want to switch your output to Java for Jython support? A lot of work, sure, but there's no reason it should be impossible *in principle* to fit something like that into Cython (there would be a lot of overlap in functionality, increasing as we get smarter with what Cython does). If C code generation is moved to a tree transform instead, it is simply a matter of replacing that one (big!) component, instead of a matter of breaking the code all over the place. (Not saying we ever attempt that though :-) better not, but contemplating it kind of makes me decide for which direction I like and don't like...) Dag Sverre From martin at martincmartin.com Sun Mar 9 02:01:46 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 20:01:46 -0500 Subject: [Cython] Done with prototype for "parse tree transforms" In-Reply-To: <47D2BEB2.2080106@student.matnat.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> Message-ID: <47D336FA.4030407@martincmartin.com> Dag Sverre Seljebotn wrote: > Martin C. Martin wrote: >> Ok, thanks a lot. >> >> My day job is programming Lisp, and since I started two years ago, >> I've seen the light about a number of things, including macros. It >> would be cool to have access to the Abstract Syntax Tree of a Cython >> expression/statement/function/etc., and be able to manipulate it at >> compile time. But it sounds like that's a whole other project. :) > Hmm. I'm still not sure whether I've come across right :-) > > Accessing the abstract syntax tree is exactly what this is about, and > you can do that (however the syntax for doing would seem rather complex > and messy compared to with LISP, though a lot better I believe than what > has been going on until now). > > The problem is that the code that accesses the syntax tree has to be > written in Python and inserted into the compiler at launch time -- you > cannot analyse yourself. So, how hard would it be to provide a mechanism to do this in the same file? There's a large subset of use cases for Lisp's macros which need the &body parameter. To do the equivalent in Python/Cython, you'd need a way to specify a block of code, much like a closure that doesn't take any parameters. Interesting... Best, Martin From dagss at student.matnat.uio.no Sun Mar 9 02:55:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 9 Mar 2008 02:55:39 +0100 (CET) Subject: [Cython] Compile-time macros In-Reply-To: <47D336FA.4030407@martincmartin.com> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> Message-ID: <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> Continuining the LISP thread... >> The problem is that the code that accesses the syntax tree has to be >> written in Python and inserted into the compiler at launch time -- you >> cannot analyse yourself. > > So, how hard would it be to provide a mechanism to do this in the same > file? There's a large subset of use cases for Lisp's macros which need Heh, ok, starting to understand you. Well, I can think of a way to have compile-time macro support in Cython that would make sense and be a natural part of other optimizations, however that is *far* in the future. This framework allows kind of the same thing, but it is so complicated and will easily break if Cython changes so it would practically never be worth the effort, especially when there is ample support for this in Python at run-time already. Python have for a long time allowed "run-time" macros, ie code generation, of course (though perhaps not as powerful as LISP, and gives a slower implementation): def make_adder(howmuch): def adder(x): return x + howmuch return adder add10_func = make_adder(10) print add10_func(z) # prints the value of z+10 Now, adding two seperate (hard!) features to Cython: 1) Supporting inner functions, ie closures, at all 2) Automatically collapse function calls where one can know that the code doesn't depend on anything that isn't known at compile time. Once this is done, the make_adder call above should automatically collapse and the add10 function be made directly compile-time. Try to read the clock or anything similar in make_adder though and it will be delayed to run-time. Of course one could have a @Macro annotation to make sure you got compiler errors if it couldn't be collapsed. (But NOTE: This is hard, and might never be done, I'm only dreaming. I don't see any real showstoppers for this but some hard work is needed and perhaps not as much return-on-investment as other exciting features one could add.) Dag Sverre From martin at martincmartin.com Sun Mar 9 03:25:31 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 21:25:31 -0500 Subject: [Cython] OT: programming Lisp for a living In-Reply-To: References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> Message-ID: <47D34A9B.3010406@martincmartin.com> Neal Becker wrote: > Martin C. Martin wrote: > >> Ok, thanks a lot. >> >> My day job is programming Lisp, and since I started two years ago, I've >> seen the light about a number of things, including macros. It would be >> cool to have access to the Abstract Syntax Tree of a Cython >> expression/statement/function/etc., and be able to manipulate it at >> compile time. But it sounds like that's a whole other project. :) >> > Wow! Someone is paid to program lisp? Yep, and coincidently I just found this blog tonight: http://lispjobs.wordpress.com/ Before starting there, I'd only used Lisp for classes, and a tiny little bit in grad school for simplifying some expressions. Things I've learned about Lisp since programming in it: 1. Lisp is *not* a functional language, it's a multi-paradigm language. That is, you can program in an imperative style if you want, you don't need any weird monads or anything. Most of our code is imperative and we don't use recursion much, only in places where it makes things clearer. 2. Lisp is the only dynamically typed language (that I know of) that cares about efficiency. It has optional static typing for efficiency, so you can say "this sub-expression will be of type X, so at runtime don't even bother looking at its type, just call the right method based the the type I'm telling you." It's about as fast as C/C++ if you avoid a few poorly thought out parts of the language. 3. Macros, which are basically functions that run at compile time that produce Lisp code, are really great. You can use them for efficiency, to make little domain languages, and other sorts of abstractions that are awkward/impossible in other languages. We're a very successful company that has a great culture; our managers don't see their jobs as checking up on you, but as getting problems out of your way so you can concentrate on your work; we worry about programmer's motivations and try hard to match projects to their interests, even letting them do their own projects if they're relevant and well thought out; and everyone is very friendly and relaxed. Check us out if you're interested, www.itasoftware.com , no Lisp knowledge required. Tell them I sent you. [I've never plugged like that to a general mailing list before, but since someone asked about getting paid to program Lisp... Sorry if it's a breach of ediquite.] Best, Martin From martin at martincmartin.com Sun Mar 9 03:42:22 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sat, 08 Mar 2008 21:42:22 -0500 Subject: [Cython] Compile-time macros In-Reply-To: <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <47D2BA1D.1040202@martincmartin.com> <47D2BC4B.7040008@student.matnat.uio.no> <47D2BD74.7030104@martincmartin.com> <47D2BEB2.2080106@student.matnat.uio.no> <47D336FA.4030407@martincmartin.com> <60867.193.157.243.12.1205027739.squirrel@webmail.uio.no> Message-ID: <47D34E8E.20705@martincmartin.com> Dag Sverre Seljebotn wrote: > Continuining the LISP thread... > >>> The problem is that the code that accesses the syntax tree has to be >>> written in Python and inserted into the compiler at launch time -- you >>> cannot analyse yourself. >> So, how hard would it be to provide a mechanism to do this in the same >> file? There's a large subset of use cases for Lisp's macros which need > > Heh, ok, starting to understand you. Well, I can think of a way to have > compile-time macro support in Cython that would make sense and be a > natural part of other optimizations, however that is *far* in the future. As you say later on, this is just dreaming. But dreaming is fun. :) Actually, I think there's a version, completely separate from the transform code that started this thread, that might not be hard at all. The basic idea of Lisp macros is that: 1. Lisp makes it easy to manipulate lists, including lists of lists. 2. In Lisp, all code is represented as lists. 3. So, it's pretty easy to write code which produces Lisp code. Of course, Python/Cython programs aren't represented as lists. However, programs can be naturally represented as an abstract syntax tree. In fact, that's a standard representation inside a compiler. It follows naturally from how a language is structured: a block contains 1 or more statements, the statements contain expressions, which contain sub-expressions, etc. It's essentially a Document Object Model (DOM) for source code, which is already hierarchically structured. So it would be easy to come up with a representation of Python/Cython program using standard Python data types, where elements correspond 1-1 with the parts of a program. So that takes care of 2, and Python has lots of ways to work easily with lists as well as many other data structures, which takes care of 1. So if we could introduce the concept of a special kind of function, which is run at compile time, and which takes the (unevaluated) parse trees of its arguments, instead of the runtime values of them, we could probably have something equivalent to Lisp macros. A macro would be a lot like transforming an XML DOM into another DOM, except instead of XML it's a parse tree. > Python have for a long time allowed "run-time" macros, ie code generation, > of course (though perhaps not as powerful as LISP, and gives a slower > implementation): > > def make_adder(howmuch): > def adder(x): return x + howmuch > return adder > > add10_func = make_adder(10) > print add10_func(z) # prints the value of z+10 > > Now, adding two seperate (hard!) features to Cython: > 1) Supporting inner functions, ie closures, at all > 2) Automatically collapse function calls where one can know that the code > doesn't depend on anything that isn't known at compile time. As you say, this is significantly different from Lisp macros, and the compiler needs to be very clever to make it efficient. A basic "defmacro," as outlined above, should be easy to implement, and to implement efficiently. Just a thought, Martin From robertwb at math.washington.edu Sun Mar 9 10:20:00 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 9 Mar 2008 01:20:00 -0800 Subject: [Cython] Patch ready (was: Re: Done with prototype for "parse tree transforms") In-Reply-To: <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> References: <47D1A0A0.9000105@student.matnat.uio.no> <45118C83-78E8-4C8E-BF0E-CB8F5C8BE670@math.washington.edu> <47D289FE.10108@student.matnat.uio.no> <89717831-6CA6-47B8-8455-6B609F6C1950@math.washington.edu> <60724.193.157.243.12.1205023941.squirrel@webmail.uio.no> Message-ID: <962BF409-E5DF-44F4-8EC2-2F4A94E93E5D@math.washington.edu> On Mar 8, 2008, at 4:52 PM, Dag Sverre Seljebotn wrote: >> It would be nice to figure out a way that pluggins could be written >> to be less brittle (i.e. not so dependent on the inner workings and >> names of the Cython compiler) too, eventually. > > Totally agreed (and I've written about that too some places, > especially > that the -T switch will only ever be a developer tool). It's > important for > me that this transform framework isn't as much a plugin > architecture as a > change in the modularity of the compiler core itself. This is a worthy goal. > Currently (if I dare) the ambitions seems to be too high for the > current code architecture > and (in all modesty :-) ) I believe this approach goes a long way > solving > that problem. Yep. > So this is the way I see it (and I'd guess you agree but I want it > on the > mailing list): > - In an ideal world, it is simply wrong to have the methods acting > on the > parse tree implemented as methods on the parse tree itself. (See > example > below). Rather you want the parse tree to be "data" and seperate, > componentized and well-defined transforms to be the vehicles of > making the > compilation happen. So ideally, the parse tree shouldn't be that > brittle > because there should be "nothing there" but information that is > spec-ed > according to the phases one is in. (However for plugin use it is > definetely too complex still). > > - Working towards such an ideal code world is not the business one > is in > and is also very boring, so one makes compromises: Whenever new > functionality is introduced, write it as transforms, whenever > something > new is impossible to write as transforms, make the minimal necesarry > modifications towards making it possible to write is as transforms, > but > leave it at that. > > - I would however consider that type coercion should be split into a > seperate CoercionTransform descending from the Transform class, rather > than making a new coercion phase within the existing code-in-nodes > paradigm. This makes a lot of sense. I would like to move information about when the coercions are possible to the types themselves too. > - At some point a new layer should definitely be written that allows > simpler types of plugins. However (but I'm less sure of this than > other > things) I believe NumPy support and C++ template support can be > written as > transforms directly, because they are high priorities and to gain > experience with what kind of plugin support would be most beneficial. I think parameterized and intelligent types will play a key role here as well, and seem more modular than transforms looking for types with the string "numpy" in them. > - We should keep in the back of our heads that the transform > problem has > been solved in many ways in varying degrees of generalness > (ick..too late > in the night to look up the proper phrase). For instance it would be > perfectly possible to export the tree to XML, run an XSLT > transform, and > parse the XML back, and then one could use XSLT for writing all the > transforms (which would be much better in some cases than Python, > and much > worse in others). (I'm personally not that happy about XSLT in this > particular case, though for many other uses it is perfect, but I'd > do some > research and look for libraries in general that can help write > transforms. > Something like doing XPath on this thing would help. Hmm...in fact > there > probably are Python XPath implementations that one could fit on our > DOM, > only question is whether it is worth the effort. If you don't know > these > just look them up). Speaking specifically of XSLT, in your fram