From dagss at student.matnat.uio.no Tue Apr 1 20:33:06 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 20:33:06 +0200 Subject: [Cython] Robin Message-ID: <47F27FE2.8090309@student.matnat.uio.no> Just saw this: http://robin.python-hosting.com/ Seems like a good tool taking it on face value, but I didn't try it or look at the source (there is a numpy directory in the source tree though...). Perhaps there is something we can lift from it... I saw it on the Python gsoc list when somebody was looking for mentors for it, full email: Hello all, I am an M.Sc. student at Tel-Aviv University, Israel, and an open source developer of the project "Robin" (see http://corwin.amber.googlepages.com/, http://robin.python-hosting.com for a taste) which is somewhat of a SWIG alternative for automatically creating Python bindings for C++ libraries. It provides several features missing from existing tools, and is also cleaner, maintainable, and extendable. I would like to push Robin forward as part of Google's "Summer of Code", which means: 1. Create a good reference manual and tutorial (with demos) 2. Compile packages for popular OS distros 3. Stabilize Robin such that it would be capable of seamlessly generate bindings for some well-known open-source C++ libraries such as a. webkit (http://webkit.org) b. libtorrent (http://libtorrent.rakshasa.no/) And if anyone has any other libraries to "challange" Robin I would be happy to use them as test cases and refine Robin that way. I am still looking for a mentor from PSF. thank you, -- Shachar -- Dag Sverre From dagss at student.matnat.uio.no Tue Apr 1 23:34:58 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 23:34:58 +0200 Subject: [Cython] Generic Message-ID: <47F2AA82.2080102@student.matnat.uio.no> Another draft from me that is an attempt to see if my drafts and Fabrizio's drafts have common traits that can be developed together. http://wiki.cython.org/enhancements/treevisitors Stefan, we already talked about some of this... -- Dag Sverre From dagss at student.matnat.uio.no Tue Apr 1 23:36:05 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 01 Apr 2008 23:36:05 +0200 Subject: [Cython] Generic tree visitors In-Reply-To: <47F2AA82.2080102@student.matnat.uio.no> References: <47F2AA82.2080102@student.matnat.uio.no> Message-ID: <47F2AAC5.8060304@student.matnat.uio.no> ...should have been the title. -- Dag Sverre From dagss at student.matnat.uio.no Wed Apr 2 20:37:46 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Apr 2008 20:37:46 +0200 Subject: [Cython] Wiki cleanup / CEP numbering scheme Message-ID: <47F3D27A.80501@student.matnat.uio.no> I'd like to spend an hour or so cleaning up the enhancements page if this (I guess it is not a necessity, but if it should be done anyway then this is a good time). If I get thumbs up for the following I'll get at it. I propose the following scheme: Number each document with a CEP number. Numbering scheme: CEP1xx: "Frontend". User experience CEPs and meta-CEPs; high-level goals that drive development process ("better C++ support", "better numpy support") as well as build system, command-line interface, usage as library etc. CEP2xx: Better support for compiling non-extended Python code CEP3xx: New features for the Cython language that doesn't overlap with Python (templates, code in pxd files) CEP4xx: Optimizations -- things that doesn't change the Cython language but which changes C output. For instance for-in => for-from transform. CEP9xx: Cython implementation things (transformation pipelines, development of new parser, etc.) The important thing isn't that we cover every case now, but that documents that's numbered now doesn't have to be renumbered. Each document has the following fields (guidelines only): - CEP status: Anyone can submit a CEP with status to "Idea" (working on it) or "Proposed" (believe it is ready for getting accepted), project administrators can set status to "Accepted" - Implementation status: "None", "Prototyped", "Patch ready", "Committed in #...." etc. Author field is not necesarry (one can check wiki info), however if under development the developer "owning" the implementation initiative for it should be noted. Also the CEPs that goes into a GSoC application should note that in a Comment field. In the wiki: - Pages keep their current URL, but the CEP number is noted on the enhancements page and in the title - The enhancements page has a link to a description of this system, then CEPs by overall category. However the CEPs are NOT necesarrily sorted, as one might want to create subgroups of CEPs - A brainstorming section at the end of enhancements contains stuff that is on their way to become a CEP but is not specific enough (however the bar shouldn't be too high if it is evident that the idea is an isolated concept that doesn't need to be broken up, such as "better local variable handling") -- Dag Sverre From robertwb at math.washington.edu Thu Apr 3 07:13:05 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 2 Apr 2008 22:13:05 -0700 Subject: [Cython] cimport inside a package In-Reply-To: <20080327162858.937d684c.simon@arrowtheory.com> References: <20080327161750.083aecce.simon@arrowtheory.com> <49F941B9-B09C-4B26-8751-974329FE45D2@math.washington.edu> <20080327162858.937d684c.simon@arrowtheory.com> Message-ID: I think the problem is that it's getting confused between the names "foo" and "zap.foo." I'm not sure, however, what the correct solution to this is, other than giving the names via setup.py, but perhaps someone more familiar with distutils and/or compiling extensions by hand would have a better idea what's going on. - Robert On Mar 27, 2008, at 1:28 PM, Simon Burton wrote: > > > I have a Makefile ( the -I. doesn't seem to help): > > > bar.so: bar.o > gcc $(CFLAGS) -shared bar.o -o bar.so > > bar.o: bar.c > gcc $(CFLAGS) -c bar.c -I/usr/include/python2.5 > > bar.c: bar.pyx > cython -I. bar.pyx > > > > foo.so: foo.o > gcc $(CFLAGS) -shared foo.o -o foo.so > > foo.o: foo.c > gcc $(CFLAGS) -c foo.c -I/usr/include/python2.5 > > foo.c: foo.pyx > cython -I. foo.pyx > > > > > On Thu, 27 Mar 2008 13:25:45 -0700 > Robert Bradshaw wrote: > >> What does your setup.py look like? >> >> On Mar 27, 2008, at 1:17 PM, Simon Burton wrote: >> >>> == foo.pxd == >>> >>> >>> cdef int foo() >>> >>> >>> == foo.pyx == >>> >>> cdef int foo(): >>> print "hi foo" >>> return 9 >>> >>> >>> == bar.pyx == >>> >>> cimport foo >>> >>> def bar(): >>> print foo.foo() >>> >>> ============= >>> >>> All good so far. >>> >>> If I put the above in a regular python package called zap, >>> and then (from outside of the zap package): >>> >>> $ python >>> Python 2.5.1 (r251:54863, Mar 7 2008, 04:10:12) >>> [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 >>> Type "help", "copyright", "credits" or "license" for more >>> information. >>>>>> from zap import bar >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "bar.pyx", line 2, in bar >>> ImportError: No module named foo >>>>>> >>> >>> The C code in bar.c is trying to import module bar. >>> What is the correct cimport invocation in bar.pyx ? >>> I tried some obvious permutations ("from zap cimport bar"), but >>> nothing got past >>> the cython compiler. >>> >>> Simon. >>> >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Thu Apr 3 08:19:16 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 2 Apr 2008 23:19:16 -0700 Subject: [Cython] --cplus option to the "cython" command line script In-Reply-To: <47DEEAF2.6070805@martincmartin.com> References: <47DEEAF2.6070805@martincmartin.com> Message-ID: <044F70B6-0E1A-4294-874E-5AAE38447627@math.washington.edu> Done. On Mar 17, 2008, at 3:04 PM, Martin C. Martin wrote: > Hi, > > the --cplus option doesn't appear in the usage, and the comments in > the > code say its only supported on MacOS X. But that isn't true, it's > used > by the distutils when you specify langauge="c++". > > Can it be added to the usage in Cython/Compiler/CmdLine.py ? > > Best, > Martin > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From mistobaan at gmail.com Thu Apr 3 08:44:35 2008 From: mistobaan at gmail.com (Fabrizio Milo aka misto) Date: Thu, 3 Apr 2008 08:44:35 +0200 Subject: [Cython] Wiki cleanup / CEP numbering scheme In-Reply-To: <47F3D27A.80501@student.matnat.uio.no> References: <47F3D27A.80501@student.matnat.uio.no> Message-ID: Great, I had this idea in mind too. Thumbs up! Fabrizio From robertwb at math.washington.edu Fri Apr 4 01:26:59 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 3 Apr 2008 16:26:59 -0700 Subject: [Cython] [Pyrex] C++ forward declaration In-Reply-To: <47F5663B.4010100@acusim.com> References: <47F5663B.4010100@acusim.com> Message-ID: <2D8F0128-966B-4965-83E9-F069FC1EC680@math.washington.edu> On Apr 3, 2008, at 4:20 PM, Ravi Lanka wrote: > Pyrex gurus, > > I am trying to wrap two C++ classes in a Namespace called > "MyNameSpace". I was breezing through using some of the ideas > shared by > Lenard and others. I got stuck with a case of forward declaration as > below. The two classes I have, A and B, have methods that require the > objects corresponding to B and A. If I try to declare, say class B, > without all the methods before class A and then try to create the > complete code for class B, it complains that class B is re-declared. > How do I get around this problem ? > > cdef extern from "A.h": > ctypedef struct A "MyNameSpace::A": > void (*foo)( B data ) > > cdef extern from "B.h": > ctypedef struct B "MyNameSpace::B": > void (*bar)( A data ) > > thanks > Ravi I believe you have to do a forward declaration. The following works for me: cdef extern from "B.h": ctypedef struct B cdef extern from "A.h": ctypedef struct A "MyNameSpace::A": void foo(B data) cdef extern from "B.h": ctypedef struct B "MyNameSpace::B": void bar(A data) cdef A a cdef B b # this will compile...but don't run it a.foo(b) b.bar(a) - Robert From ggellner at uoguelph.ca Fri Apr 4 05:19:21 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Thu, 3 Apr 2008 23:19:21 -0400 Subject: [Cython] access to enclosing scope in a cdef Message-ID: <20080404031921.GA6772@giton> Sorry for the simple question, but I couldn't figure out how to do this myself... I am wrapping a C-code ode solver that uses a callback after each time step. So to save the output I need to append these values to a global array. Roughly I want to have a calling function that creates a numpy array that the callback will write to. The problem is I don't know how to make this kind of global variable in cython. The skeleton that I would like to be able to do (given that I have removed all the real logic): # This is the solution callback (not the model definition for those who use # ode solvers) that is the ode solver is returning the *y values after each # step. cdef void solout(int nr, double *x, double *y): # I will have a loop in the future, but as long as I can # save a single value to the array I can solve this myself. output.data[0][0] = y[0] # This the python driver def ode(t): dim = 4 # hard set to make the code short . . . # Allocate my array output = numpy.zeros((4, len(t))) # set solver values y0 = # Call the solver, (I assume func, the model, is defined elsewhere . . .) odesolve(func, y0, t) # now at this point I would want output to have its first row set I hope this is clear, if my simplification has made it worse I can give a full example of what the code would look like (that is I can get everything to work with print statements, I just am not able to save the output . . .) Any help would be greatly appreciated! Gabriel From robertwb at math.washington.edu Fri Apr 4 09:22:04 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 4 Apr 2008 00:22:04 -0700 Subject: [Cython] Cython 0.9.6.13rc1 is up. Message-ID: <46256EE6-B976-418A-99FE-73E0C2D43DF0@math.washington.edu> This has the new repository hierarchy, so you won't be able to pull from the online -devel ones. If no one reports any bugs in then I will release tomorrow. http://sage.math.washington.edu/home/robertwb/cython/ - Robert From dagss at student.matnat.uio.no Fri Apr 4 11:29:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 04 Apr 2008 11:29:04 +0200 Subject: [Cython] Wiki cleanup / CEP numbering scheme In-Reply-To: <47F3D27A.80501@student.matnat.uio.no> References: <47F3D27A.80501@student.matnat.uio.no> Message-ID: <47F5F4E0.6030207@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > I'd like to spend an hour or so cleaning up the enhancements page if > this (I guess it is not a necessity, but if it should be done anyway > then this is a good time). If I get thumbs up for the following I'll get > at it. > OK, this is done (not all the spec titles etc. are changed accordingly but we'll take that as we go). Dag Sverre From mistobaan at gmail.com Fri Apr 4 15:14:57 2008 From: mistobaan at gmail.com (Fabrizio Milo aka misto) Date: Fri, 4 Apr 2008 15:14:57 +0200 Subject: [Cython] Wiki cleanup / CEP numbering scheme In-Reply-To: <47F5F4E0.6030207@student.matnat.uio.no> References: <47F3D27A.80501@student.matnat.uio.no> <47F5F4E0.6030207@student.matnat.uio.no> Message-ID: > OK, this is done (not all the spec titles etc. are changed accordingly > but we'll take that as we go). Good Job! Fabrizio -------------------------- Luck favors the prepared mind. (Pasteur) From simon at arrowtheory.com Fri Apr 4 16:29:37 2008 From: simon at arrowtheory.com (Simon Burton) Date: Fri, 4 Apr 2008 10:29:37 -0400 Subject: [Cython] access to enclosing scope in a cdef In-Reply-To: <20080404031921.GA6772@giton> References: <20080404031921.GA6772@giton> Message-ID: <20080404102937.b7140134.simon@arrowtheory.com> On Thu, 3 Apr 2008 23:19:21 -0400 Gabriel Gellner wrote: > Sorry for the simple question, but I couldn't figure out how to do this > myself... > > I am wrapping a C-code ode solver that uses a callback after each time step. > So to save the output I need to append these values to a global array. > > Roughly I want to have a calling function that creates a numpy array that the > callback will write to. The problem is I don't know how to make this kind of > global variable in cython. > > The skeleton that I would like to be able to do (given that I have removed all > the real logic): > > # This is the solution callback (not the model definition for those who use > # ode solvers) that is the ode solver is returning the *y values after each > # step. > cdef void solout(int nr, double *x, double *y): > # I will have a loop in the future, but as long as I can > # save a single value to the array I can solve this myself. > output.data[0][0] = y[0] > > # This the python driver > def ode(t): global output > dim = 4 # hard set to make the code short . . . > > # Allocate my array > output = numpy.zeros((4, len(t))) > > # set solver values > y0 = > # Call the solver, (I assume func, the model, is defined elsewhere . . .) > odesolve(func, y0, t) > > # now at this point I would want output to have its first row set > > I hope this is clear, if my simplification has made it worse I can give a full > example of what the code would look like (that is I can get everything to work > with print statements, I just am not able to save the output . . .) > > Any help would be greatly appreciated! > > Gabriel > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Fri Apr 4 19:20:26 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 4 Apr 2008 10:20:26 -0700 Subject: [Cython] Wiki cleanup / CEP numbering scheme In-Reply-To: <47F5F4E0.6030207@student.matnat.uio.no> References: <47F3D27A.80501@student.matnat.uio.no> <47F5F4E0.6030207@student.matnat.uio.no> Message-ID: <8A34BEE8-3E20-4510-BCCE-0BE6F468589C@math.washington.edu> On Apr 4, 2008, at 2:29 AM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> I'd like to spend an hour or so cleaning up the enhancements page if >> this (I guess it is not a necessity, but if it should be done anyway >> then this is a good time). If I get thumbs up for the following >> I'll get >> at it. >> > OK, this is done (not all the spec titles etc. are changed accordingly > but we'll take that as we go). Thanks! From robertwb at math.washington.edu Sat Apr 5 03:01:37 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 4 Apr 2008 18:01:37 -0700 Subject: [Cython] [Pyrex] C++ forward declaration In-Reply-To: <47F6A9FF.1020005@telus.net> References: <47F5663B.4010100@acusim.com> <2D8F0128-966B-4965-83E9-F069FC1EC680@math.washington.edu> <47F6A9FF.1020005@telus.net> Message-ID: <2700F0D2-3FE8-447A-BE07-C1D3B0A187C0@math.washington.edu> On Apr 4, 2008, at 3:21 PM, Lenard Lindstrom wrote: > First, the Pyrex 0.9.6.4 compiler rejected the above code given by > Robert Bradshaw. Second, even when altered to use function pointers > the > generated code was wrong. Sorry, I just hastily typed this up in my email to demonstrate the concept of forward declaration (which was the missing piece). Also, you're right about needing to do function pointers if you're using Pyrex. > The global variable b was delcared as "B", not > "MyNameSpace::B". This example corrects both problems: > > cdef extern from "B.h": > ctypedef struct B "MyNameSpace::B" > > cdef extern from "A.h": > ctypedef struct A "MyNameSpace::A": > void (* foo)(B data) > > cdef extern from "B.h": > ctypedef struct B "MyNameSpace::B": > void (* bar)(A data) > > cdef A a > cdef B b > > # this will compile...but don't run it > a.foo(b) > b.bar(a) > > Note that B.h will be included before A.h in the generated C file. > > -- > Lenard Lindstrom > > > > _______________________________________________ > Pyrex mailing list > Pyrex at lists.copyleft.no > http://lists.copyleft.no/mailman/listinfo/pyrex From robertwb at math.washington.edu Sat Apr 5 03:25:19 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 4 Apr 2008 18:25:19 -0700 Subject: [Cython] Cython 0.9.6.13 Released Message-ID: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> Hi All. Cython 0.9.6.13 is available for download at http://cython.org or http://pypi.python.org/pypi/Cython/0.9.6.13. The main improvements to this release are - C++ exception handling (Felix Wu) - (optional) C line numbers in Errors (Gary Furnish) - some circular cimports (Gary Furnish) - (experimental) parse tree transforms (Dag Seljebotn) - struct member functions automatically coerced to function pointers (for easier C++ wrapping) - no unneeded incref on function arguments - allow single-character ascii literals to be used as ints (no need for c'x' notation) - better support for using arrays as pointers There are also several bugfixes and pre-Py3K changes due to Robert Bradshaw, Stefan Behnel, Jum Kleckner, and Chris Perkins. The compiler and package repositories have been merged, and while all history has been preserved it is a completely new repository now. We are looking forward to lots of development at Sage Developer Days 1 (http://wiki.sagemath.org/dev1) and hopefully some Google Summer of Code projects over the summer. - Robert From ggellner at uoguelph.ca Sat Apr 5 06:34:16 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 5 Apr 2008 00:34:16 -0400 Subject: [Cython] python callback Message-ID: <20080405043416.GA9069@basestar> Could anyone give me a tip on how to convert a python function of the form def func(x): return some_operation(x) to a C function of the form cdef void cfunc(double x, double *y): y = some_operation(x) That is I want to be able to turn a python function into a C function with pass by reference return semantics. I looked at the cheesefinder example, and I am not sure how to adapt it, as I want to be able to have the user supply the python function, and I can't think of how to make a cdef callback function that knows what this function is, that is cdef void callback(souble x, double *y): y = func(x) I am not sure how to set func in this way, without rewriting the cdef each time. In python I would just use a higher order function . . . Thanks, Gabriel From michael.abshoff at googlemail.com Sat Apr 5 06:27:08 2008 From: michael.abshoff at googlemail.com (Michael.Abshoff) Date: Sat, 05 Apr 2008 06:27:08 +0200 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> Message-ID: <47F6FF9C.5090509@gmail.com> Robert Bradshaw wrote: > Hi All. > > Cython 0.9.6.13 is available for download at http://cython.org or > http://pypi.python.org/pypi/Cython/0.9.6.13. The main improvements to > this release are > > - C++ exception handling (Felix Wu) > - (optional) C line numbers in Errors (Gary Furnish) > - some circular cimports (Gary Furnish) > - (experimental) parse tree transforms (Dag Seljebotn) > - struct member functions automatically coerced to function pointers > (for easier C++ wrapping) > - no unneeded incref on function arguments > - allow single-character ascii literals to be used as ints (no need > for c'x' notation) > - better support for using arrays as pointers > > There are also several bugfixes and pre-Py3K changes due to Robert > Bradshaw, Stefan Behnel, Jum Kleckner, and Chris Perkins. The > compiler and package repositories have been merged, and while all > history has been preserved it is a completely new repository now. > > We are looking forward to lots of development at Sage Developer Days > 1 (http://wiki.sagemath.org/dev1) and hopefully some Google Summer of > Code projects over the summer. > > - Robert Hi guys, Sage with the new Cython has a lot of leaks exposed by valgrind: ==9372== LEAK SUMMARY: ==9372== definitely lost: 232,533 bytes in 3,757 blocks. Those leaks are all considered "possibly lost" or "still reachable" with Cython 0.9.6.12, so it seems to be mostly an accounting issue [valgrind's accounting of possibly lost vs. still reachable vs. definitely lost can vary depending on other leaks, but I will spare you the details here]. While we have looked into this in the past at various Sage Days it might be a good idea to put this on the agenda for Dev Day 1 since this currently adds massive noise to the interesting bits when looking for memleaks in Sage. I can add a suppression file, but I would consider that only a last resort. Keep up the great work. Cheers, Michael > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Sat Apr 5 14:31:55 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 05 Apr 2008 14:31:55 +0200 Subject: [Cython] Refactoring Errors + test harness Message-ID: <47F7713B.8020301@student.matnat.uio.no> I've been playing with making Errors an object that is passed into functions (rather than "global variable"). I think this should happen at some point as the current approach is rather nasty. This is not a pointless excersize; the long-term aim for this would be to make it possible to export Cython cleanly as a library rather than just a command-line program, which in turn is necesarry for. Also one should already operate with seperate error contexts: Some errors (like, in initialization of Builtins.py) should be ignored (now they are recorded and subsequently deleted) while other errors (initalization in Symtab.py) should probably raise exceptions instead of reporting errors as they would signify Cython bugs. The changes needed were pretty extensive though. I ended up giving every single function in 'ExprNodes.py', 'Nodes.py', 'ModuleNode.py', 'Symtab.py' a new parameter, "ctx", specifying the "compilation context" one is in and which carry error() and warning() functions. (I use a script to do this; it goes in two passes; first extract and change all function definitions, and then change all function calls using a name used in a function definition.) The alternative would be to basically set the context parameter as attributes on the nodes themselves; I felt this was unnatural as method calls on nodes does changes on the node structure, while errors encountered while doing that is something that should be reported to the caller of the method. My question then: - Will this get into Cython if I spend more time on it and finish it? It is a quite dangerous change, although now seems the right time to do it (having just done a release one has time to discover any bugs introduced). - What's the best method to regression test easily? I've not built SAGE yet and would prefer something lighter if available; somebody has a test harness set up? Is it in the Cython repo? -- Dag Sverre From dagss at student.matnat.uio.no Sat Apr 5 14:33:52 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 05 Apr 2008 14:33:52 +0200 Subject: [Cython] Refactoring Errors + test harness In-Reply-To: <47F7713B.8020301@student.matnat.uio.no> References: <47F7713B.8020301@student.matnat.uio.no> Message-ID: <47F771B0.4010809@student.matnat.uio.no> > > This is not a pointless excersize; the long-term aim for this would be > to make it possible to export Cython cleanly as a library rather than > just a command-line program, which in turn is necesarry for. > ...necesarry for creating small experimentation programs that only plays with certain parts of Cython rather than having to always do a full compile. OK, I'll readily admit the benefits aren't *that* great. I'm easily convinced into dropping it too. -- Dag Sverre From simon at arrowtheory.com Sat Apr 5 18:58:21 2008 From: simon at arrowtheory.com (simon at arrowtheory.com) Date: Sat, 5 Apr 2008 12:58:21 -0400 (EDT) Subject: [Cython] python callback In-Reply-To: <20080405043416.GA9069@basestar> References: <20080405043416.GA9069@basestar> Message-ID: <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> > Could anyone give me a tip on how to convert a python function > of the form > > def func(x): > return some_operation(x) > > to a C function of the form > > cdef void cfunc(double x, double *y): > y = some_operation(x) y[0] = some_operation(x) > > That is I want to be able to turn a python function into > a C function with pass by reference return semantics. > > I looked at the cheesefinder example, and I am not sure how to adapt it, > as I > want to be able to have the user supply the python function, and I can't > think > of how to make a cdef callback function that knows what this function is, > that > is > > cdef void callback(souble x, double *y): > y = func(x) > > I am not sure how to set func in this way, without rewriting the cdef each > time. In python I would just use a higher order function . . . in general, you need an extra "void *" argument in the callback function (which you would use to pass a python callable). Well designed C libraries use this convention, but it sounds like you don't have this. > > Thanks, > Gabriel > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Simon. From ggellner at uoguelph.ca Sat Apr 5 19:41:03 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 5 Apr 2008 13:41:03 -0400 Subject: [Cython] python callback In-Reply-To: <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> References: <20080405043416.GA9069@basestar> <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> Message-ID: <20080405174103.GA12685@basestar> On Sat, Apr 05, 2008 at 12:58:21PM -0400, simon at arrowtheory.com wrote: > > Could anyone give me a tip on how to convert a python function > > of the form > > > > def func(x): > > return some_operation(x) > > > > to a C function of the form > > > > cdef void cfunc(double x, double *y): > > y = some_operation(x) > > y[0] = some_operation(x) > Yeah, sorry that was a typo... > > > > That is I want to be able to turn a python function into > > a C function with pass by reference return semantics. > > > > I looked at the cheesefinder example, and I am not sure how to adapt it, > > as I > > want to be able to have the user supply the python function, and I can't > > think > > of how to make a cdef callback function that knows what this function is, > > that > > is > > > > cdef void callback(souble x, double *y): > > y = func(x) > > > > I am not sure how to set func in this way, without rewriting the cdef each > > time. In python I would just use a higher order function . . . > > in general, you need an extra "void *" argument in the callback > function (which you would use to pass a python callable). > Well designed C libraries use this convention, but it > sounds like you don't have this. > Curses, I guess I will have to change the original code :-( Thanks for the help. Gabriel From robertwb at math.washington.edu Sun Apr 6 01:20:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 5 Apr 2008 16:20:22 -0700 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <47F6FF9C.5090509@gmail.com> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> <47F6FF9C.5090509@gmail.com> Message-ID: <2B328C76-3102-4199-9319-D2EDFAE22338@math.washington.edu> On Apr 4, 2008, at 9:27 PM, Michael.Abshoff wrote: > Robert Bradshaw wrote: >> Hi All. >> >> Cython 0.9.6.13 is available for download at http://cython.org or >> http://pypi.python.org/pypi/Cython/0.9.6.13. The main improvements to >> this release are >> >> - C++ exception handling (Felix Wu) >> - (optional) C line numbers in Errors (Gary Furnish) >> - some circular cimports (Gary Furnish) >> - (experimental) parse tree transforms (Dag Seljebotn) >> - struct member functions automatically coerced to function pointers >> (for easier C++ wrapping) >> - no unneeded incref on function arguments >> - allow single-character ascii literals to be used as ints (no need >> for c'x' notation) >> - better support for using arrays as pointers >> >> There are also several bugfixes and pre-Py3K changes due to Robert >> Bradshaw, Stefan Behnel, Jum Kleckner, and Chris Perkins. The >> compiler and package repositories have been merged, and while all >> history has been preserved it is a completely new repository now. >> >> We are looking forward to lots of development at Sage Developer Days >> 1 (http://wiki.sagemath.org/dev1) and hopefully some Google Summer of >> Code projects over the summer. >> >> - Robert > > Hi guys, > > Sage with the new Cython has a lot of leaks exposed by valgrind: > > ==9372== LEAK SUMMARY: > ==9372== definitely lost: 232,533 bytes in 3,757 blocks. > > Those leaks are all considered "possibly lost" or "still reachable" > with > Cython 0.9.6.12, so it seems to be mostly an accounting issue > [valgrind's accounting of possibly lost vs. still reachable vs. > definitely lost can vary depending on other leaks, but I will spare > you > the details here]. To clarify, it doesn't sound like there are significantly many new leaks, but the accounting has shifted around a bit. > While we have looked into this in the past at various Sage Days it > might > be a good idea to put this on the agenda for Dev Day 1 since this > currently adds massive noise to the interesting bits when looking for > memleaks in Sage. I can add a suppression file, but I would consider > that only a last resort. Perhaps one thing that might cut down the noise is disabling intern_names and cache_builtins in http://hg.cython.org/cython/file/b68682070c8e/Cython/Compiler/Options.py (Also, on that note, I think we want to ship c_line_in_traceback = 1 with Sage.) I also made the default cleanup level 0 again, as it was causing Segfaults in peoples code. Here is the issue: ---- a.pyx ---- import b foo = b.B() ---- b.pyx --- class B: def __del__(self): print "hi" ---- Now B *cannot* be deleted after the module b is cleaned up because the string "hi" may have been deallocated (sometimes it seems to still work, e.g. if the memory involved hasn't been dirtied by anything else). There's also the issue of the dictionaries that don't go away when they should. - Robert From michael.abshoff at googlemail.com Sun Apr 6 01:39:32 2008 From: michael.abshoff at googlemail.com (Michael.Abshoff) Date: Sun, 06 Apr 2008 01:39:32 +0200 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <2B328C76-3102-4199-9319-D2EDFAE22338@math.washington.edu> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> <47F6FF9C.5090509@gmail.com> <2B328C76-3102-4199-9319-D2EDFAE22338@math.washington.edu> Message-ID: <47F80DB4.4050807@gmail.com> Robert Bradshaw wrote: > On Apr 4, 2008, at 9:27 PM, Michael.Abshoff wrote: >> Robert Bradshaw wrote: >>> Hi All. >>> >>> Cython 0.9.6.13 is available for download at http://cython.org or >>> http://pypi.python.org/pypi/Cython/0.9.6.13. The main improvements to >>> this release are >>> >>> - C++ exception handling (Felix Wu) >>> - (optional) C line numbers in Errors (Gary Furnish) >>> - some circular cimports (Gary Furnish) >>> - (experimental) parse tree transforms (Dag Seljebotn) >>> - struct member functions automatically coerced to function pointers >>> (for easier C++ wrapping) >>> - no unneeded incref on function arguments >>> - allow single-character ascii literals to be used as ints (no need >>> for c'x' notation) >>> - better support for using arrays as pointers >>> >>> There are also several bugfixes and pre-Py3K changes due to Robert >>> Bradshaw, Stefan Behnel, Jum Kleckner, and Chris Perkins. The >>> compiler and package repositories have been merged, and while all >>> history has been preserved it is a completely new repository now. >>> >>> We are looking forward to lots of development at Sage Developer Days >>> 1 (http://wiki.sagemath.org/dev1) and hopefully some Google Summer of >>> Code projects over the summer. >>> >>> - Robert >> Hi guys, >> >> Sage with the new Cython has a lot of leaks exposed by valgrind: >> >> ==9372== LEAK SUMMARY: >> ==9372== definitely lost: 232,533 bytes in 3,757 blocks. Hi guys >> Those leaks are all considered "possibly lost" or "still reachable" >> with >> Cython 0.9.6.12, so it seems to be mostly an accounting issue >> [valgrind's accounting of possibly lost vs. still reachable vs. >> definitely lost can vary depending on other leaks, but I will spare >> you >> the details here]. > > To clarify, it doesn't sound like there are significantly many new > leaks, but the accounting has shifted around a bit. Correct. While I do not have precise figures yet it is very close to what I saw with 0.9.6.12 and Sage 2.11. >> While we have looked into this in the past at various Sage Days it >> might >> be a good idea to put this on the agenda for Dev Day 1 since this >> currently adds massive noise to the interesting bits when looking for >> memleaks in Sage. I can add a suppression file, but I would consider >> that only a last resort. > > Perhaps one thing that might cut down the noise is disabling > intern_names and cache_builtins in > > http://hg.cython.org/cython/file/b68682070c8e/Cython/Compiler/Options.py > > (Also, on that note, I think we want to ship c_line_in_traceback = 1 > with Sage.) I also made the default cleanup level 0 again, as it was > causing Segfaults in peoples code. Here is the issue: Ok, I will dial that up and see if there is any trouble in Sage. I remember some trouble with integer.pyx but we did fix those issues when we finally squashed #1337. > ---- a.pyx ---- > > import b > foo = b.B() > > ---- b.pyx --- > > class B: > def __del__(self): > print "hi" > > ---- > > Now B *cannot* be deleted after the module b is cleaned up because > the string "hi" may have been deallocated (sometimes it seems to > still work, e.g. if the memory involved hasn't been dirtied by > anything else). > > There's also the issue of the dictionaries that don't go away when > they should. Yes, hopefully we can spend some time on this before, during or after Dev1 since I will be in Seattle for about three weeks around Dev1 :) > - Robert Cheers, Michael > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From f.guerrieri at gmail.com Sun Apr 6 12:57:48 2008 From: f.guerrieri at gmail.com (Francesco Guerrieri) Date: Sun, 6 Apr 2008 12:57:48 +0200 Subject: [Cython] Installing on Windows Message-ID: <79b79e730804060357j74b04d2xb5d4959f9a13f566@mail.gmail.com> Hello, I have added on the wiki a short tutorial on installing cython on windows, based on my experience. I thought it was useful to collect the steps on the wiki. Please check for the contents to see if I misunderstood something (the steps described have actually worked for me but maybe I was in some corner case and didn't know that :) ) I put a link to the page under the section Cython installers bye Francesco From dagss at student.matnat.uio.no Sun Apr 6 13:43:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 06 Apr 2008 13:43:30 +0200 Subject: [Cython] Results of XPathTransform / W3CDOM experiments Message-ID: <47F8B762.4010700@student.matnat.uio.no> I'm still fooling around with some experiments. The following is now working (in my local repo) as a way of transforming for-froms: class ForInToForFrom(XPathTransform): @template("pyr:ForInStatNode[iterator/pyr:IteratorNode/sequence/pyr:SimpleCallNode/function/pyr:NameNode/@name = 'range']") def for_in_range_to_for_from_range(self, node): result = Nodes.ForFromStatNode(... ... return result Everything happens on the Pyrex tree, there's no translation to XML or anything like that. Example attached (though you can't run it outside me repo, it's just for demonstration). The question is: Is this a way forward for transforms? For some more examples, consider that one could for instance select all equality statements that must have some coercion by "pyr:SimpleAssignmentNode[lhs/@type != rhs/@type]" But this is contrived, coercion won't work this way. But also consider that one can select inner functions by "pyr:FuncDefNode//pyr:FuncDefNode" and outer functions only by "pyr:ModuleNode/body/pyr:FuncDefNode" and so on. The gains are highest if XPath selections are used for all transforms written, because then the finite state machines (see below) can (in principle at least) be combined so that only one tree traversal per phase is needed regardless of how the code is modularized into multiple transforms. (If combining, one must use a subset of XPath where only the descendants axis is available outside of predicates, I guess this is the same as XSLT match statements?). What I've done: - Put a subset of the W3C DOM API on top of the tree. No modifications to Cython code tree was necesarry except adding a base class (and I finally had a legitimate use for a metaclass or two. Yay!). A "side-effect" is that the tree can be streamed to XML (see example code). - Use the webpath XPath 2.0 transform to select nodes (http://sourceforge.net/projects/webpath), and act on them on traversal. Questions: - Anyone know of good DOM transformation libraries for Cython? - Does anyone think this would be useful? - Does anyone think this could be a standard for writing transforms? - Any other good uses for a W3C DOM on our parse trees? (it is a seperate component) I'll assume that streaming in and out of XSLT is not going to be convenient, but something else perhaps? Some notes: - It currently scales horribly with the number of "templates"; one full tree traversal per match. In order to fix this, one either has to find a better XPath library (which must be hacked a bit), an XSLT processor or similar implemented entirely in Python, or a full-time week is needed to improve webpath by using a Finite State Automata library (which does the standard non-deterministic automatas to deterministic automatas, there are several good ones and this is not too hard to do). Does it matter if we do 30 traversals on the tree rather than 2-3? As long as it can be optimized "in principle"? - On the other hand, once that is done, one can "combine" tree traversals so that multiple transforms work in the same traversal, meaning that the number of traversals will be reduced compared to what is in sight now. - But, the current less efficient implementation is working. I might probably leave it for now at this because the gains seem less than the effort, but if anyone thinks this is interesting then speak up and we can see. -- Dag Sverre -------------- next part -------------- A non-text attachment was scrubbed... Name: testbed.py Type: text/x-python Size: 2476 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080406/2ca2cea1/attachment-0001.py From dagss at student.matnat.uio.no Sun Apr 6 13:53:54 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 06 Apr 2008 13:53:54 +0200 Subject: [Cython] Results of XPathTransform / W3CDOM experiments In-Reply-To: <47F8B762.4010700@student.matnat.uio.no> References: <47F8B762.4010700@student.matnat.uio.no> Message-ID: <47F8B9D2.6030307@student.matnat.uio.no> > > Questions: > - Anyone know of good DOM transformation libraries for Cython? _P_ython. -- Dag Sverre From martin at martincmartin.com Sun Apr 6 14:46:56 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Sun, 06 Apr 2008 08:46:56 -0400 Subject: [Cython] Lisp inspired transforms Message-ID: <47F8C640.10901@martincmartin.com> Hi all, I've been doing some thinking and prototyping of a transform system inspired by Common Lisp macros. You can see the results as the newest CEP: http://wiki.cython.org/enhancements/metaprogramming Briefly, it allows you to define a transform in the Cython source code. The transform runs at compile time, and takes the *parse trees* of its arguments. In the examples, I define a simple symbolic differentiator "deriv" which means you can write: def eggs(a, b): return deriv(5*a+b, a) and it is translated at compile time to: def eggs(a, b): return (((5 * 1) + (0 * a)) + 0) You can use this, for example, with numerical optimization techniques like Newton's method. An example which differentiates 5 * x**3 - 10 and finds it's root is given on the above wiki page, here's the output: >>> TestTrans.myfunct(10.0) (4990.0, 1500.0) >>> TestTrans.newtons(TestTrans.myfunct, 10.0) f( 10.0 ) = 4990.0 , f'( 10.0 ) = 1500.0 f( 6.67333333333 ) = 1475.93037185 , f'( 6.67333333333 ) = 668.000666667 f( 4.46385893383 ) = 434.735082042 , f'( 4.46385893383 ) = 298.890548717 f( 3.00936301916 ) = 126.267956666 , f'( 3.00936301916 ) = 135.843986716 f( 2.07985587115 ) = 34.9852072625 , f'( 2.07985587115 ) = 64.8870066716 f( 1.54068464016 ) = 8.28568621842 , f'( 1.54068464016 ) = 35.6056374064 f( 1.3079774954 ) = 1.18847303518 , f'( 1.3079774954 ) = 25.6620769269 f( 1.26166506953 ) = 0.0415843883547 , f'( 1.26166506953 ) = 23.8769812152 f( 1.25992345957 ) = 5.73769235945e-05 , f'( 1.25992345957 ) = 23.8111068596 f( 1.2599210499 ) = 1.09734443754e-10 , f'( 1.2599210499 ) = 23.8110157797 f( 1.25992104989 ) = 0.0 , f'( 1.25992104989 ) = 23.8110157795 1.2599210498948732 It's also useful for moving computation from run time to compile time, and makes it easy for the programmer to specify what should happen where. There are many other use cases as well. I'm attaching a patch that implements the examples. It's very proof-of-concept, which means its really rough. :) I also implemented the bar minimum to get the examples working, but it gives you an idea. Best, Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: CythonMetaprogramming.tar.gz Type: application/gzip Size: 8137 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080406/de466ac3/attachment.bin From mistobaan at gmail.com Sun Apr 6 17:19:41 2008 From: mistobaan at gmail.com (Fabrizio Milo aka misto) Date: Sun, 6 Apr 2008 17:19:41 +0200 Subject: [Cython] Results of XPathTransform / W3CDOM experiments In-Reply-To: <47F8B762.4010700@student.matnat.uio.no> References: <47F8B762.4010700@student.matnat.uio.no> Message-ID: Dag, Do you want to transform the ExprTree or the DOCUMENT (i.e. the C file) produced by the ExprTree? I understood that you were working on the second point, but seems that you are working on the first point ! Matching an expr tree with XPath, seems to me like shooting to a fly with a cannon. It will be very slow and not all the possible matching rules will be possible with XPath plus I imagine that will be some trickery with recursive functions. I would prefer other forms of matching: i.e. class ForInToForFrom(Transformation): def ex_match1 (self, tree): return len(tree) >=3 and tree is ForInStatNode and tree[1] is SimpleCallNode and tree[2] is NameNode and tree[2].name == RANGE_ID def ex_match2 (self, tree): return tree == ForInStatNode ( SimpleCallNode ( NameNode (name = RANGE_ID))) def eg_match3 (self,tree): subtree = buildMatch (""" ForInStatNode: SimpleCallNode: NameNode: name = range """) return subtree.matches(tree) def transform (self, tree): return Nodes.ForFromStatNode(...) Plex have a DSL to express matching Rules we could readapt it to match Nodes instead of chars ... To better understand what kind of transformation we would like to apply I think we should have a 10 / 20 item list of possible transforms. >From this list should be derived the best form to express the matching against the ExprTree or the AST. This things could be done in N different ways. Finding the most simple and effective is the challenge. Fabrizio -------------------------- Luck favors the prepared mind. (Pasteur) From dagss at student.matnat.uio.no Sun Apr 6 18:29:15 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 06 Apr 2008 18:29:15 +0200 Subject: [Cython] Results of XPathTransform / W3CDOM experiments In-Reply-To: References: <47F8B762.4010700@student.matnat.uio.no> Message-ID: <47F8FA5B.5070907@student.matnat.uio.no> > Matching an expr tree with XPath, seems to me like shooting to a fly > with a cannon. > That appears to be my conclusion as well, I just had to explore it (and discuss it). I wonder if you overestimate how heavy XPath (or a subset) is though as the alternatives you suggest seems to me to be just as heavy! > It will be very slow and not all the possible matching rules will be > possible with XPath plus I imagine that will be some trickery with > recursive functions. > Slow: Not slower than any alternatives doing the same thing. Matching possible: Yes, one would be able to call into any Python function so... Recursive functions: Not sure what you mean. There are potential problems related with recursive transforms processes in general but that is independent of the mechanism used for node matching. > class ForInToForFrom(Transformation): > > def ex_match1 (self, tree): > return len(tree) >=3 and tree is ForInStatNode and tree[1] is > SimpleCallNode and tree[2] is NameNode and tree[2].name == RANGE_ID > I like this. From a theoretic perspective though: But how would you select nodes depending on where it is located (ie all nested functions)? So you need to access parent stack as well. And then (in theory) you have algorithmic performance worse than with XPath, because with XPath you can turn stuff like (function//function)|(function//for) into function//(function|for) and this transformation is done at Cython launch-time (and if a problem can be pickled to speed up Cython launch). > def ex_match2 (self, tree): > return tree == ForInStatNode ( SimpleCallNode ( NameNode > (name = RANGE_ID))) > This would be about the same thing as an XPath approach! But one has to write the == machinery, and since that is done manually towards our exact tree then some optimizations might creep in. > def eg_match3 (self,tree): > subtree = buildMatch (""" > ForInStatNode: > SimpleCallNode: > NameNode: > name = range > """) > return subtree.matches(tree) > Now you have exactly the XPath approach, except that we have to reimplement a custom parser and specify a custom language! > To better understand what kind of transformation we would like to > apply I think we should have a 10 / 20 item list of possible > transforms. > Well said. I think it will probably be the first one, but at least now XPath is a bit explored so that one can recognize it if it is needed... -- Dag Sverre From ggellner at uoguelph.ca Mon Apr 7 07:57:26 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Mon, 7 Apr 2008 01:57:26 -0400 Subject: [Cython] python callback In-Reply-To: <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> References: <20080405043416.GA9069@basestar> <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> Message-ID: <20080407055726.GA9347@basestar> Looking through the scipy odeint C wrapper, I realized the simple pattern to solve this (just having the callback call a global python function, that is set by the driver.) Does anyone think this is worth making a Wiki page for? If so I would be happy to write up the solution. Gabriel On Sat, Apr 05, 2008 at 12:58:21PM -0400, simon at arrowtheory.com wrote: > > Could anyone give me a tip on how to convert a python function > > of the form > > > > def func(x): > > return some_operation(x) > > > > to a C function of the form > > > > cdef void cfunc(double x, double *y): > > y = some_operation(x) > > y[0] = some_operation(x) > > > > > That is I want to be able to turn a python function into > > a C function with pass by reference return semantics. > > > > I looked at the cheesefinder example, and I am not sure how to adapt it, > > as I > > want to be able to have the user supply the python function, and I can't > > think > > of how to make a cdef callback function that knows what this function is, > > that > > is > > > > cdef void callback(souble x, double *y): > > y = func(x) > > > > I am not sure how to set func in this way, without rewriting the cdef each > > time. In python I would just use a higher order function . . . > > in general, you need an extra "void *" argument in the callback > function (which you would use to pass a python callable). > Well designed C libraries use this convention, but it > sounds like you don't have this. > > > > > Thanks, > > Gabriel > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > > > Simon. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Mon Apr 7 08:21:31 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Apr 2008 08:21:31 +0200 Subject: [Cython] Refactoring Errors + test harness In-Reply-To: <47F7713B.8020301@student.matnat.uio.no> References: <47F7713B.8020301@student.matnat.uio.no> Message-ID: <47F9BD6B.6080506@behnel.de> Hi, Dag Sverre Seljebotn wrote: > - What's the best method to regression test easily? I've not built SAGE > yet and would prefer something lighter if available; somebody has a test > harness set up? Is it in the Cython repo? Sure, seen the little "runtests.py" script? The tests are in the "tests" directory. Stefan From stefan_ml at behnel.de Mon Apr 7 11:25:21 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Apr 2008 11:25:21 +0200 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> Message-ID: <47F9E881.7080405@behnel.de> Hi Robert, thanks for this release. A few comments: The file "MANIFEST" shouldn't be in the repository as it's autogenerated from the file "MANIFEST.in" by distutils. Also, there seems to be a problem with cimporting public extension classes from another Cython module. I was away last week, so I couldn't test with the release candidate, but building lxml now gives me: ----------------- Traceback (most recent call last): File "setup.py", line 106, in **extra_options File "distutils/core.py", line 151, in setup File "distutils/dist.py", line 974, in run_commands File "distutils/dist.py", line 994, in run_command File "distutils/command/build_ext.py", line 290, in run File "/.../Cython/Distutils/build_ext.py", line 81, in build_extensions ext.sources = self.cython_sources(ext.sources, ext) File "/.../Cython/Distutils/build_ext.py", line 193, in cython_sources full_module_name=module_name) File "/.../Cython/Compiler/Main.py", line 304, in compile return context.compile(source, options, full_module_name) File "/.../Cython/Compiler/Main.py", line 201, in compile tree.process_implementation(scope, options, result) File "/.../Cython/Compiler/ModuleNode.py", line 78, in process_implementation self.generate_c_code(env, options, result) File "/.../Cython/Compiler/ModuleNode.py", line 262, in generate_c_code self.generate_declarations_for_modules(env, modules, code.h) File "/.../Cython/Compiler/ModuleNode.py", line 454, in generate_declarations_for_modules vtabslot_list = self.generate_vtabslot_list(vtabslot_dict) File "/.../Cython/Compiler/ModuleNode.py", line 418, in generate_vtabslot_list if(recurse_vtabslot_check_inheritance(vtab_list[j],vtab_list[i], vtab_dict)==1): File "/.../Cython/Compiler/ModuleNode.py", line 44, in recurse_vtabslot_check_inheritance base = dict[base.type.base_type.objstruct_cname] KeyError: 'LxmlElementBase' ----------------- This class is cimported from this definition in a .pxd: cdef extern from "lxml.etree_api.h": cdef class lxml.etree.ElementBase(_Element) \ [ object LxmlElementBase ]: ... http://codespeak.net/svn/lxml/trunk/src/lxml/etreepublic.pxd by this Cython file: http://codespeak.net/svn/lxml/trunk/src/lxml/lxml.objectify.pyx I don't think there is a test case for cimporting extension classes yet. I'll have to see when I find the time to look into this, but maybe not before the end of the week. Stefan From dagss at student.matnat.uio.no Mon Apr 7 11:35:38 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 7 Apr 2008 11:35:38 +0200 (CEST) Subject: [Cython] [Fwd: [Numpy-discussion] New project : Spyke python-to-C compiler] Message-ID: <50815.193.157.243.12.1207560938.squirrel@webmail.uio.no> I already replied saying that Cython has many of the same goals (though as I'm not on python-dev I'm not sure if my email gets through, and somebody might want to create a more informed and better response; didn't have time). Dag Sverre ---------------------------- Original Message ---------------------------- Subject: [Numpy-discussion] New project : Spyke python-to-C compiler From: "Rahul Garg" Date: Mon, April 7, 2008 02:48 To: numpy-discussion at scipy.org python-dev at python.org -------------------------------------------------------------------------- Note this message has been posted to numpy-discussion and python-dev. Sorry for the multiple posting but I thought both python devs and numpy users will be interested. If you believe your list should not receive this email, let me know. Also I just wanted to introduce myself since I may ask doubts about Python and Numpy internals from time to time :) Hi. I am a student at Univ of Alberta doing my masters in computing science. I am writing a Python-to-C compiler as one part of my thesis. The compiler, named Spyke, will be made available in a couple of weeks and is geared towards scientific applications and will therefore focus mostly on needs of scientific app developers. What is Spyke? In many performance critical projects, it is often necessary to rewrite parts of the application in C. However writing C wrappers can be time consuming. Spyke offers an alternative approach. You add annotations to your Python code as strings. These strings are discarded by the Python interpreter but these are interpreted as types by Spyke compiler to convert to C. Example : "int -> int" def f(x): return 2*x In this case the Spyke compiler will consider the string "int -> int" as a decalration that the function accepts int as parameter and returns int. Spyke will then generate a C function and a wrapper function. This idea is directly copied from PLW (Python Language Wrapper) project. Once Python3k arrives, much of these declarations will be moved to function annotations and class decorators. This way you can do all your development and debugging interactively using the standard Python interpreter. When you need to compile to C, you just add type annotations to places that you want to convert and invoke spyke on the annotated module. This is different from Pyrex because Pyrex does not accept Python code. With Spyke, your code is 100% pure python. Spyke has basic support for functions and classes. Spyke can do very basic type inference for local variables in function bodies. Spyke also has partial support for homogenous lists and dictionaries and fixed length tuples. One big advantage of Spyke is that it understands at least part of numpy. Numpy arrays are treated as fundamental types and Spyke knows what C code to generate for slicing/indexing of numpy arrays etc. This should help a lot in scientific applications. Note that Spyke can handle only a subset of Python. Exceptions, iterators, generators, runtime code generation of any kind etc is not handled. Nested functions will be added soon. I will definitely add some of these missing features based on what is actually required for real world Python codes. Currently if Spyke does not understand a function, it just leaves it as Python code. Classes can be handled but special methods are not currently supported. The support of classes is a little brittle because I am trying to resolve some issues b/w old and new style of classes. Where is Spyke? Spyke will be available as a binary only release in a couple of weeks. I intend to make it open source after a few months. Spyke is written in Python and Java and should be platform independant. I do intend to make the source open in a few months. Right now its undergoing very rapid development and has negligible amounts of documentation so the source code right now is pretty useless to anyone else anyway. I need help: However I need a bit of help. I am having a couple of problems : a) I am finding it hard to get pure Python+NumPy testing codes. I need more codes to test the compiler. Developing a compiler without a test-suite is kind of useless. If you have some pure Python codes which need better performance, please contact me. I guarantee that your codes will not be released to public without your permission but might be referenced in academic publications. I can also make the compiler available to you hopefully after 10th of April. Its kind of unstable currently. I will also need your help in annotating the provided testing codes since I probably wont know what your application is doing. b) Libraries which interface with C/C++ : Many codes in SciPy for instance have mixed language codes. Part of the code is written in C/C++. Spyke only knows how to annotated Python codes. For C/C++ libraries wrapped into Python modules, Spyke will therefore need to know at least 2 things : i) The mapping of a C function name/struct etc to Python ii) The type information of the said C function. There are many many ways that people interact with C code. People either write wrappers manually, or use autogenerated wrappers using SWIG or SIP Boost.Python etc., use Pyrex or Cython while some people use ctypes. I dont have the time or resources to support these multitude of methods. I considered trying to parse the C code implementing wrappers but its "non-trivial" to put it mildly. Parsing only SWIG generated code is another possibility but its still hard. Another approach that I am seriously considering is to support a subset of ctypes (with additional restriction) instead. But my question is : Is ctypes good enough for most of you? Ctypes cannot interface with C++ code but its pure Python. However I have not seen too many instances of people using ctypes. c) Strings as type declarations : Do you think I should use decorators instead at least for function type declarations? thanks for patiently reading this, comments and inquiries sought. rahul _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From robertwb at math.washington.edu Mon Apr 7 19:09:10 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 7 Apr 2008 10:09:10 -0700 Subject: [Cython] python callback In-Reply-To: <20080407055726.GA9347@basestar> References: <20080405043416.GA9069@basestar> <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> <20080407055726.GA9347@basestar> Message-ID: On Apr 6, 2008, at 10:57 PM, Gabriel Gellner wrote: > Looking through the scipy odeint C wrapper, I realized the simple > pattern to > solve this (just having the callback call a global python function, > that is > set by the driver.) > > Does anyone think this is worth making a Wiki page for? If so I > would be happy > to write up the solution. Yes, certainly. Thanks. > > Gabriel > > On Sat, Apr 05, 2008 at 12:58:21PM -0400, simon at arrowtheory.com wrote: >>> Could anyone give me a tip on how to convert a python function >>> of the form >>> >>> def func(x): >>> return some_operation(x) >>> >>> to a C function of the form >>> >>> cdef void cfunc(double x, double *y): >>> y = some_operation(x) >> >> y[0] = some_operation(x) >> >>> >>> That is I want to be able to turn a python function into >>> a C function with pass by reference return semantics. >>> >>> I looked at the cheesefinder example, and I am not sure how to >>> adapt it, >>> as I >>> want to be able to have the user supply the python function, and >>> I can't >>> think >>> of how to make a cdef callback function that knows what this >>> function is, >>> that >>> is >>> >>> cdef void callback(souble x, double *y): >>> y = func(x) >>> >>> I am not sure how to set func in this way, without rewriting the >>> cdef each >>> time. In python I would just use a higher order function . . . >> >> in general, you need an extra "void *" argument in the callback >> function (which you would use to pass a python callable). >> Well designed C libraries use this convention, but it >> sounds like you don't have this. >> >>> >>> Thanks, >>> Gabriel >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >> Simon. >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Mon Apr 7 19:10:05 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 7 Apr 2008 10:10:05 -0700 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <47F9E881.7080405@behnel.de> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> <47F9E881.7080405@behnel.de> Message-ID: I'm pretty sure I know what this is and how to fix it. - Robert On Apr 7, 2008, at 2:25 AM, Stefan Behnel wrote: > Hi Robert, > > thanks for this release. A few comments: > > The file "MANIFEST" shouldn't be in the repository as it's > autogenerated from > the file "MANIFEST.in" by distutils. > > Also, there seems to be a problem with cimporting public extension > classes > from another Cython module. I was away last week, so I couldn't > test with the > release candidate, but building lxml now gives me: > > ----------------- > Traceback (most recent call last): > File "setup.py", line 106, in > **extra_options > File "distutils/core.py", line 151, in setup > File "distutils/dist.py", line 974, in run_commands > File "distutils/dist.py", line 994, in run_command > File "distutils/command/build_ext.py", line 290, in run > File "/.../Cython/Distutils/build_ext.py", line 81, in > build_extensions > ext.sources = self.cython_sources(ext.sources, ext) > File "/.../Cython/Distutils/build_ext.py", line 193, in > cython_sources > full_module_name=module_name) > File "/.../Cython/Compiler/Main.py", line 304, in compile > return context.compile(source, options, full_module_name) > File "/.../Cython/Compiler/Main.py", line 201, in compile > tree.process_implementation(scope, options, result) > File "/.../Cython/Compiler/ModuleNode.py", line 78, in > process_implementation > self.generate_c_code(env, options, result) > File "/.../Cython/Compiler/ModuleNode.py", line 262, in > generate_c_code > self.generate_declarations_for_modules(env, modules, code.h) > File "/.../Cython/Compiler/ModuleNode.py", line 454, in > generate_declarations_for_modules > vtabslot_list = self.generate_vtabslot_list(vtabslot_dict) > File "/.../Cython/Compiler/ModuleNode.py", line 418, in > generate_vtabslot_list > if(recurse_vtabslot_check_inheritance(vtab_list[j],vtab_list[i], > vtab_dict)==1): > File "/.../Cython/Compiler/ModuleNode.py", line 44, in > recurse_vtabslot_check_inheritance > base = dict[base.type.base_type.objstruct_cname] > KeyError: 'LxmlElementBase' > ----------------- > > This class is cimported from this definition in a .pxd: > > cdef extern from "lxml.etree_api.h": > cdef class lxml.etree.ElementBase(_Element) \ > [ object LxmlElementBase ]: > ... > > http://codespeak.net/svn/lxml/trunk/src/lxml/etreepublic.pxd > > by this Cython file: > > http://codespeak.net/svn/lxml/trunk/src/lxml/lxml.objectify.pyx > > > I don't think there is a test case for cimporting extension classes > yet. > > I'll have to see when I find the time to look into this, but maybe > not before > the end of the week. > > Stefan From stefan_ml at behnel.de Mon Apr 7 19:35:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Apr 2008 19:35:27 +0200 Subject: [Cython] python callback In-Reply-To: <20080407055726.GA9347@basestar> References: <20080405043416.GA9069@basestar> <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> <20080407055726.GA9347@basestar> Message-ID: <47FA5B5F.6020101@behnel.de> Gabriel Gellner wrote: > Looking through the scipy odeint C wrapper, I realized the simple pattern to > solve this (just having the callback call a global python function, that is > set by the driver.) > > Does anyone think this is worth making a Wiki page for? If so I would be happy > to write up the solution. There already is a (tiny) FAQ entry on callbacks, but it would be good to have a real page on this topic (even if your solution here is less common than passing a void*). http://wiki.cython.org/FAQ#head-0ea9184c404ed73456c5988eaf208d91bf04accb Stefan From robertwb at math.washington.edu Mon Apr 7 19:59:32 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 7 Apr 2008 10:59:32 -0700 Subject: [Cython] Lisp inspired transforms In-Reply-To: <47F8C640.10901@martincmartin.com> References: <47F8C640.10901@martincmartin.com> Message-ID: <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> On Apr 6, 2008, at 5:46 AM, Martin C. Martin wrote: > Hi all, > > I've been doing some thinking and prototyping of a transform system > inspired by Common Lisp macros. You can see the results as the > newest CEP: > > http://wiki.cython.org/enhancements/metaprogramming > > Briefly, it allows you to define a transform in the Cython source > code. The transform runs at compile time, and takes the *parse > trees* of its arguments. In the examples, I define a simple > symbolic differentiator > "deriv" which means you can write: > > def eggs(a, b): > return deriv(5*a+b, a) > > and it is translated at compile time to: > > def eggs(a, b): > return (((5 * 1) + (0 * a)) + 0) > > You can use this, for example, with numerical optimization > techniques like Newton's method. An example which differentiates 5 > * x**3 - 10 and finds it's root is given on the above wiki page, > here's the output: > > >>> TestTrans.myfunct(10.0) > (4990.0, 1500.0) > >>> TestTrans.newtons(TestTrans.myfunct, 10.0) > f( 10.0 ) = 4990.0 , f'( 10.0 ) = 1500.0 > f( 6.67333333333 ) = 1475.93037185 , f'( 6.67333333333 ) = > 668.000666667 > f( 4.46385893383 ) = 434.735082042 , f'( 4.46385893383 ) = > 298.890548717 > f( 3.00936301916 ) = 126.267956666 , f'( 3.00936301916 ) = > 135.843986716 > f( 2.07985587115 ) = 34.9852072625 , f'( 2.07985587115 ) = > 64.8870066716 > f( 1.54068464016 ) = 8.28568621842 , f'( 1.54068464016 ) = > 35.6056374064 > f( 1.3079774954 ) = 1.18847303518 , f'( 1.3079774954 ) = 25.6620769269 > f( 1.26166506953 ) = 0.0415843883547 , f'( 1.26166506953 ) = > 23.8769812152 > f( 1.25992345957 ) = 5.73769235945e-05 , f'( 1.25992345957 ) = > 23.8111068596 > f( 1.2599210499 ) = 1.09734443754e-10 , f'( 1.2599210499 ) = > 23.8110157797 > f( 1.25992104989 ) = 0.0 , f'( 1.25992104989 ) = 23.8110157795 > 1.2599210498948732 > > It's also useful for moving computation from run time to compile > time, and makes it easy for the programmer to specify what should > happen where. > > There are many other use cases as well. > > I'm attaching a patch that implements the examples. It's very > proof-of-concept, which means its really rough. :) I also > implemented the bar minimum to get the examples working, but it > gives you an idea. > > Best, > Martin Thanks. This looks very interesting. Also, I much prefer this type of "macro" to the text-substitution C-style macros that have occasionally been suggested. One thought that kept going through my head as I was reading this, however, is that one of the current defects (in my mind) is its inconsistency with actual Python, and the associated learning curve to read and write Python. Some of this inconsistency (e.g. (minimal) static typing information) is necessary to achieve the kind of speedups we need, but I think the focus should be on features that narrow the gap between Cython and Python, not those that widen it. On the other hand, features like the one above can be very useful, and it'd be a shame to deny their availability to "power users" (though it would take away one thing that I really like about Cython--almost any Python programmer can at least read a Cython function without any additional knowledge). - Robert From robertwb at math.washington.edu Mon Apr 7 20:19:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 7 Apr 2008 11:19:14 -0700 Subject: [Cython] Results of XPathTransform / W3CDOM experiments In-Reply-To: <47F8B762.4010700@student.matnat.uio.no> References: <47F8B762.4010700@student.matnat.uio.no> Message-ID: On Apr 6, 2008, at 4:43 AM, Dag Sverre Seljebotn wrote: > I'm still fooling around with some experiments. The following is > now working (in my local repo) as a way of transforming for-froms: > > class ForInToForFrom(XPathTransform): > @template("pyr:ForInStatNode[iterator/pyr:IteratorNode/sequence/ > pyr:SimpleCallNode/function/pyr:NameNode/@name = 'range']") > def for_in_range_to_for_from_range(self, node): > result = Nodes.ForFromStatNode(... > ... > return result > > Everything happens on the Pyrex tree, there's no translation to XML > or anything like that. Example attached (though you can't run it > outside me repo, it's just for demonstration). > > The question is: Is this a way forward for transforms? Perhaps, but I doubt it. I would echo Fabrizio's call for a dozen or so transformations that one would want to do with examples that the xpath way of going about it is cleaner. Also, I don't see the end user as writing their own transformations much (nor do I think it's a good idea to encourage it--it makes the language much more obscure). It also greatly increases the dependancies to run Cython. > For some more examples, consider that one could for instance select > all equality statements that must have some coercion by > > "pyr:SimpleAssignmentNode[lhs/@type != rhs/@type]" > > But this is contrived, coercion won't work this way. I think coercion (especially the relationship between types) is too complicated to be done this way. (OK, maybe it's possible, but it would almost certainly be horribly inefficient and hard to understand). > But also consider that one can select inner functions by > > "pyr:FuncDefNode//pyr:FuncDefNode" > > and outer functions only by > > "pyr:ModuleNode/body/pyr:FuncDefNode" > > and so on. > > The gains are highest if XPath selections are used for all > transforms written, because then the finite state machines (see > below) can (in principle at least) be combined so that only one > tree traversal per phase is needed regardless of how the code is > modularized into multiple transforms. (If combining, one must use a > subset of XPath where only the descendants axis is available > outside of predicates, I guess this is the same as XSLT match > statements?). I think most of the code processing is done in "phases" rather than a bunch of transforms that can be done all at once. Optimizations are perhaps an exception. > What I've done: > - Put a subset of the W3C DOM API on top of the tree. No > modifications to Cython code tree was necesarry except adding a > base class (and I finally had a legitimate use for a metaclass or > two. Yay!). A "side-effect" is that the tree can be streamed to XML > (see example code). > - Use the webpath XPath 2.0 transform to select nodes (http:// > sourceforge.net/projects/webpath), and act on them on traversal. > > Questions: > - Anyone know of good DOM transformation libraries for Cython? Perhaps lxml does this? > - Does anyone think this would be useful? > - Does anyone think this could be a standard for writing transforms? > - Any other good uses for a W3C DOM on our parse trees? (it is a > seperate component) I'll assume that streaming in and out of XSLT > is not going to be convenient, but something else perhaps? > > Some notes: > - It currently scales horribly with the number of "templates"; one > full tree traversal per match. In order to fix this, one either has > to find a better XPath library (which must be hacked a bit), an > XSLT processor or similar implemented entirely in Python, or a full- > time week is needed to improve webpath by using a Finite State > Automata library (which does the standard non-deterministic > automatas to deterministic automatas, there are several good ones > and this is not too hard to do). Again, there's the question of dependancies. I'd rather not require anything but Python itself. > Does it matter if we do 30 traversals on the tree rather than 2-3? > As long as it can be optimized "in principle"? > > - On the other hand, once that is done, one can "combine" tree > traversals so that multiple transforms work in the same traversal, > meaning that the number of traversals will be reduced compared to > what is in sight now. > > - But, the current less efficient implementation is working. > > I might probably leave it for now at this because the gains seem > less than the effort, but if anyone thinks this is interesting then > speak up and we can see. Perhaps it can be offered as a plugin, so people who want to do things like this can use it. But I'm not convinced that this is the direction we want to take for the Cython core. - Robert From stefan_ml at behnel.de Mon Apr 7 20:58:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Apr 2008 20:58:07 +0200 Subject: [Cython] Lisp inspired transforms In-Reply-To: <47F8C640.10901@martincmartin.com> References: <47F8C640.10901@martincmartin.com> Message-ID: <47FA6EBF.3050702@behnel.de> Hi, Martin C. Martin wrote: > I've been doing some thinking and prototyping of a transform system > inspired by Common Lisp macros. You can see the results as the newest CEP: > > http://wiki.cython.org/enhancements/metaprogramming > > Briefly, it allows you to define a transform in the Cython source code. > The transform runs at compile time, and takes the *parse trees* of its > arguments. I like the way this reads, yes. I'd like to see the "deftrans" functions in a separate source file, maybe a ".pxt"? I wouldn't want to see them mixed with normal Cython code. Stefan From martin at martincmartin.com Mon Apr 7 23:05:08 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Mon, 07 Apr 2008 17:05:08 -0400 Subject: [Cython] Lisp inspired transforms In-Reply-To: <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> Message-ID: <47FA8C84.5070207@martincmartin.com> Robert Bradshaw wrote: > > Thanks. This looks very interesting. Also, I much prefer this type of > "macro" to the text-substitution C-style macros that have occasionally > been suggested. Glad you like it. It's certainly a lot cleaner. > One thought that kept going through my head as I was reading this, > however, is that one of the current defects (in my mind) is its > inconsistency with actual Python, and the associated learning curve to > read and write Python. Some of this inconsistency (e.g. (minimal) static > typing information) is necessary to achieve the kind of speedups we > need, but I think the focus should be on features that narrow the gap > between Cython and Python, not those that widen it. > > On the other hand, features like the one above can be very useful, and > it'd be a shame to deny their availability to "power users" (though it > would take away one thing that I really like about Cython--almost any > Python programmer can at least read a Cython function without any > additional knowledge). Yes, this certainly adds a whole new dimension to Cython that's not in Python. Perhaps this is a time to reflect on the goals that Cython has had up to now, and whether to expand them. Cython and Pyrex started with the goal of making it easy to wrap existing C/C++ code as Python extensions, but if you add the ability to declare that a variable has a Python type, you get a language where you can code Python when you need to, but then when you need speed, sprinkle a few type declarations around and get code and speed similar to C. Is it worth widening the gap a little to provide a useful language in its own right? The gap is still much, much smaller than that between C++ and C, for example. Best, Martin From ggellner at uoguelph.ca Mon Apr 7 23:10:01 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Mon, 7 Apr 2008 17:10:01 -0400 Subject: [Cython] python callback In-Reply-To: <47FA5B5F.6020101@behnel.de> References: <20080405043416.GA9069@basestar> <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> <20080407055726.GA9347@basestar> <47FA5B5F.6020101@behnel.de> Message-ID: <20080407211001.GA11424@basestar> On Mon, Apr 07, 2008 at 07:35:27PM +0200, Stefan Behnel wrote: > > Gabriel Gellner wrote: > > Looking through the scipy odeint C wrapper, I realized the simple pattern to > > solve this (just having the callback call a global python function, that is > > set by the driver.) > > > > Does anyone think this is worth making a Wiki page for? If so I would be happy > > to write up the solution. > > There already is a (tiny) FAQ entry on callbacks, but it would be good to have > a real page on this topic (even if your solution here is less common than > passing a void*). > > http://wiki.cython.org/FAQ#head-0ea9184c404ed73456c5988eaf208d91bf04accb > > Stefan > One thing about this FAQ entry, the Cython releases doesn't seem to include the callback folder in Demos. I had to go into the Mercural repo to get it. Gabriel > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Tue Apr 8 00:34:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 8 Apr 2008 00:34:44 +0200 (CEST) Subject: [Cython] Results of XPathTransform / W3CDOM experiments In-Reply-To: References: <47F8B762.4010700@student.matnat.uio.no> Message-ID: <51334.193.157.243.12.1207607684.squirrel@webmail.uio.no> > xpath way of going about it is cleaner. Also, I don't see the end > user as writing their own transformations much (nor do I think it's a > good idea to encourage it--it makes the language much more obscure). This was definitely considered a feature to help with developing Cython, if transforms made a coding-style convenient where each change could be described in relative isolation in one function. But I'll definitely stick it in the drawer. Fun experiment though. I should have known that finally having a legitimate use for Python metaclasses was too good to be true... ;-) Dag Sverre From martin at martincmartin.com Tue Apr 8 03:45:38 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Mon, 07 Apr 2008 21:45:38 -0400 Subject: [Cython] Lisp inspired transforms In-Reply-To: <47FA6EBF.3050702@behnel.de> References: <47F8C640.10901@martincmartin.com> <47FA6EBF.3050702@behnel.de> Message-ID: <47FACE42.9050702@martincmartin.com> Hi Stefan, Stefan Behnel wrote: > Hi, > > Martin C. Martin wrote: >> I've been doing some thinking and prototyping of a transform system >> inspired by Common Lisp macros. You can see the results as the newest CEP: >> >> http://wiki.cython.org/enhancements/metaprogramming >> >> Briefly, it allows you to define a transform in the Cython source code. >> The transform runs at compile time, and takes the *parse trees* of its >> arguments. > > I like the way this reads, yes. > > I'd like to see the "deftrans" functions in a separate source file, maybe a > ".pxt"? I wouldn't want to see them mixed with normal Cython code. It's true they're an "advanced" feature, but they're often closely tied to other code you write. For example, suppose you'd like to keep track of some statistics on a given expression. It could be CPU time, memory allocated, or a simple count of the number of times you reach a line. And you'd like to put these throughout your code, to measure different sections. (We do this so we know where both time and memory allocation are going.) You could keep a hash table that maps the name of the section to the accumulating value, but if you don't want the hash table overhead, you could assign a numeric id to each name, and use that id to index into an array. In the example below, I'm assuming names_to_ids and num_ids are available at compile time, and that the final value of num_ids makes it into the C code. names_to_ids = {} num_ids = 0 # Get names_to_ids[name] if it exists, otherwise add it to the hash # table. def getid(name): if name in names_to_ids: return names_to_ids[name] else: id = num_ids names_to_ids[name] = id num_ids += 1 return id record_array = [0]*num_ids # Created at runtime, ideally as a C array. # Generating IDs and converting names to IDs happens at compile time. deftrans record(name, *expressions): id = getid(name) return "record_internal(%c, *%c)" % (id, expressions) # Get the start time, then evaluate the expressions, then get the # end time, subtract to get the elapsed time, and accumulate that. def record_internal(id, *expressions): ... record_array[id] += elapsed_time So here there's a lot of mixing of transforms and regular functions. It seems unnatural to me to put record() into a separate file. What do you think? Best, Martin From robertwb at math.washington.edu Tue Apr 8 09:52:11 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 8 Apr 2008 00:52:11 -0700 Subject: [Cython] Lisp inspired transforms In-Reply-To: <47FA8C84.5070207@martincmartin.com> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> Message-ID: <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> On Apr 7, 2008, at 2:05 PM, Martin C. Martin wrote: > Robert Bradshaw wrote: >> Thanks. This looks very interesting. Also, I much prefer this type >> of "macro" to the text-substitution C-style macros that have >> occasionally been suggested. > > Glad you like it. It's certainly a lot cleaner. > >> One thought that kept going through my head as I was reading this, >> however, is that one of the current defects (in my mind) is its >> inconsistency with actual Python, and the associated learning >> curve to read and write Python. Some of this inconsistency (e.g. >> (minimal) static typing information) is necessary to achieve the >> kind of speedups we need, but I think the focus should be on >> features that narrow the gap between Cython and Python, not those >> that widen it. >> >> On the other hand, features like the one above can be very >> useful, and >> it'd be a shame to deny their availability to "power >> users" (though it >> would take away one thing that I really like about Cython--almost >> any >> Python programmer can at least read a Cython function without any >> additional knowledge). > > > Yes, this certainly adds a whole new dimension to Cython that's not > in Python. > > Perhaps this is a time to reflect on the goals that Cython has had > up to now, and whether to expand them. Cython and Pyrex started > with the goal of making it easy to wrap existing C/C++ code as > Python extensions, but if you add the ability to declare that a > variable has a Python type, you get a language where you can code > Python when you need to, but then when you need speed, sprinkle a > few type declarations around and get code and speed similar to C. > > Is it worth widening the gap a little to provide a useful language > in its own right? The gap is still much, much smaller than that > between C++ and C, for example. The goal of Cython is for it to be the best way to write Python extension modules. This has two (interrelated) sub-goals: make it easy to wrap C/C++ libraries, and make it a good Python -> C compiler. The target (and largest) audience is especially Python programmers. Though it is tempting to head down the language development path, adding little (or big) features that make it more powerful than Python itself, I think doing so will actually be counterproductive to the goals stated above. Perhaps there could be a Cython++ that is a proper superset of the Cython language with more powerful features (though I'd hope not near the gap of C vs. C++) but in the near term we should be focusing on things like being able to compile all of Python as it is. - Robert From robertwb at math.washington.edu Tue Apr 8 10:08:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 8 Apr 2008 01:08:14 -0700 Subject: [Cython] New project : Spyke python-to-C compiler In-Reply-To: <20080407175606.d1tlnbczk0wo8sk0@webmail.ualberta.ca> References: <20080407175606.d1tlnbczk0wo8sk0@webmail.ualberta.ca> Message-ID: Forwarding some correspondence with an author of another Python-to-C compiler: On Apr 7, 2008, at 4:56 PM, Rahul Garg wrote: > Quoting Robert Bradshaw : > >> Have you heard of Cython before? Do you have any thoughts on how it >> compares/overlaps/relates to Spyke? >> >> - Robert > > Hi. > 1. Cython and Spyke have certain differences but mostly related to the > surface syntax accepted. From what I understand (and do correct me if > I am wrong) Cython is a Python like language but Cython code isnt 100% > Python and does not run on the Python interpreter. > > Example : > > Lets assume you had the following Python code : > > def f(x): return 2*x > > If you want to convert this to C, then from what I understand in > Pyrex you would write a different file with the extension pyx. The > function would look like : > > cdef int f(int x): return 2*x > > Now this code is very similar to Python but will not run on the Python > interpreter. Spyke on the other hand would want you to declare types > within the Python source itself : > > "int -> int" > def f(x): return 2*x > > This string is discarded by the Python interpreter so its still valid > Python code and will run on the Python interpreter as well is capable > of being compiled. So its mostly a matter of surface syntax currently > and it might even be possible to interconvert the 2 formats. > > Cython is also ahead in terms of language supported and probably give > better performance currently. But Spyke takes the philosophy that > the code > should run on the Python interpreter so has no additional keywords etc > and all additional info is specified through mechanisms ignored by the > Python interpreter. Spyke copies ideas from Python Language Wrapper > project (PLW) project. You can get a paper on PLW here : > http://www.cs.utk.edu/~luszczek/pubs/plw-200605.pdf This is a difference, but just on the surface. We have a goal to be able to support this kind of thing as well (though we don't intend on making the old syntax obsolete, as it is very easy to use when there are a lot of types). > 2. Long term : Spyke is actually for my thesis on compiler > optimizations for dynamic languages. Some experiments with parallel > constructs are also on the roadmap. Also I will focus heavily on > loop optimizations and such for numpy code. Spyke does currently > understand how slicing and indexing work and numpy arrays are sort- > of native datatypes for Spyke. I also intend to implement a whole > program compiler. I have a number of ideas related to targeting a > modified Python interpreter but those probably will be in a > different branch. That's cool. Most of the differences seem to be in this point, though we hope to get better NumPy support soon. > 3. Integration : Spyke is written in Java mostly. On code level > therefore its difficult to integrate Spyke and Cython but ideas can > freely go back and forth. I will follow Cython closely. Many of the > issues remain the same since the semantics of the language > supported is similar. We'll keep our eye on Spyke too. Thanks for answering. Do you mind if I forward this response to the cython mailing list? - Robert From robertwb at math.washington.edu Tue Apr 8 10:11:36 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 8 Apr 2008 01:11:36 -0700 Subject: [Cython] New project : Spyke python-to-C compiler In-Reply-To: <20080407184837.4yn4yzipc888o44s@webmail.ualberta.ca> References: <20080407175606.d1tlnbczk0wo8sk0@webmail.ualberta.ca> <15BA280D-E9EB-4A4E-8604-D2133E3D279A@math.washington.edu> <20080407184837.4yn4yzipc888o44s@webmail.ualberta.ca> Message-ID: <2831E432-CA4B-4F62-94A7-C13D68F2B317@math.washington.edu> On Apr 7, 2008, at 5:48 PM, Rahul Garg wrote: > > Quoting Robert Bradshaw : > >> Thanks for answering. Do you mind if I forward this response to the >> cython mailing list? >> >> - Robert > > 1. Please forward if appropriate. Also include this reply please :) > > 2. I forgot to ask : Do you use some particular tests/benchmarks > etc? I can try and adapt those to Spyke-syntax if some test cases > are available. Not much, but Stefan Behnel did do some tests at http://codespeak.net/ pipermail/cython-dev/2008-March/000107.html . > 3. Looking forward : with function annotations in Python-3k it > might actually be a good idea to work towards a unified syntax. > That way code will be portable b/w the 2 compilers and can also be > adopted by other tools like editors, debuggers and such. Of course > this is just an idea off the top of my head so need to look into > more details. Yes, this would be very good. We certainly plan to make use of function annotations (and decorators) to be able to specify types. - Robert From dagss at student.matnat.uio.no Tue Apr 8 10:29:38 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 08 Apr 2008 10:29:38 +0200 Subject: [Cython] uneval + me and Martin's macro discussion Message-ID: <47FB2CF2.6000907@student.matnat.uio.no> Over the weekend me and Martin discussed (long-term, hypothetically) how macros could be "better integrated" with the Python way of doing things (and macros in general); for anyone interested. It ended up at 40 k, and can be found here: http://wiki.cython.org/enhancements/uneval/emails Anyway, my own private conclusions on this ended up in this pre-CEP: http://wiki.cython.org/enhancements/uneval -- Dag Sverre From robertwb at math.washington.edu Tue Apr 8 10:38:02 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 8 Apr 2008 01:38:02 -0700 Subject: [Cython] python callback In-Reply-To: <20080407211001.GA11424@basestar> References: <20080405043416.GA9069@basestar> <1243.70.107.93.141.1207414701.squirrel@webmail10.pair.com> <20080407055726.GA9347@basestar> <47FA5B5F.6020101@behnel.de> <20080407211001.GA11424@basestar> Message-ID: Thanks. I just added it. On Apr 7, 2008, at 2:10 PM, Gabriel Gellner wrote: > On Mon, Apr 07, 2008 at 07:35:27PM +0200, Stefan Behnel wrote: >> >> Gabriel Gellner wrote: >>> Looking through the scipy odeint C wrapper, I realized the simple >>> pattern to >>> solve this (just having the callback call a global python >>> function, that is >>> set by the driver.) >>> >>> Does anyone think this is worth making a Wiki page for? If so I >>> would be happy >>> to write up the solution. >> >> There already is a (tiny) FAQ entry on callbacks, but it would be >> good to have >> a real page on this topic (even if your solution here is less >> common than >> passing a void*). >> >> http://wiki.cython.org/ >> FAQ#head-0ea9184c404ed73456c5988eaf208d91bf04accb >> >> Stefan >> > One thing about this FAQ entry, the Cython releases doesn't seem to > include > the callback folder in Demos. I had to go into the Mercural repo > to get it. > > Gabriel >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Tue Apr 8 10:41:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 08 Apr 2008 10:41:00 +0200 Subject: [Cython] Numpy and Lisp inspired transforms In-Reply-To: <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> Message-ID: <47FB2F9C.5040608@student.matnat.uio.no> > > Though it is tempting to head down the language development path, > adding little (or big) features that make it more powerful than > Python itself, I think doing so will actually be counterproductive to > the goals stated above. Perhaps there could be a Cython++ that is a > I'm for the moment considering that using Martin's code here, and extending the approach to have support for "member macros" (that can also be used for operator overloading), numpy.ndarray.__getitem__ could perhaps be implemented quicker if Martin's work is included. Which might "up" the priority. (Not saying that approach is taken, but it is a thought). As for Cython as a language, one has to consider that also templates, function overloading, and other stuff that it is "natural" to add because we now have a _typed_ (templates overloading) _compiled_ (macros) language. Though perhaps not macros as powerful as these... (shrug) Priorities are a different matter though, and probably most of what I've proposed (as features in themselves, rather than NunPy requirements) might have less priority than getting Python code to run (which is why I should get more down-to-earth with my thoughts about Cython real soon. Which I will.). -- Dag Sverre From robertwb at math.washington.edu Tue Apr 8 10:40:06 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 8 Apr 2008 01:40:06 -0700 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <47F9E881.7080405@behnel.de> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> <47F9E881.7080405@behnel.de> Message-ID: <99CAFB68-4433-4AFF-A25B-B093EF56D76A@math.washington.edu> On Apr 7, 2008, at 2:25 AM, Stefan Behnel wrote: > Hi Robert, > > thanks for this release. A few comments: > > The file "MANIFEST" shouldn't be in the repository as it's > autogenerated from > the file "MANIFEST.in" by distutils. Yep. Realized that after the fact. It's not in the directory any more. > Also, there seems to be a problem with cimporting public extension > classes > from another Cython module. I was away last week, so I couldn't > test with the > release candidate, but building lxml now gives me: And this is exactly what I was hoping to avoid with an rc...bad timing I guess. Anyways, I've fixed this now and released 0.9.6.13.1. > > ----------------- > Traceback (most recent call last): > File "setup.py", line 106, in > **extra_options > File "distutils/core.py", line 151, in setup > File "distutils/dist.py", line 974, in run_commands > File "distutils/dist.py", line 994, in run_command > File "distutils/command/build_ext.py", line 290, in run > File "/.../Cython/Distutils/build_ext.py", line 81, in > build_extensions > ext.sources = self.cython_sources(ext.sources, ext) > File "/.../Cython/Distutils/build_ext.py", line 193, in > cython_sources > full_module_name=module_name) > File "/.../Cython/Compiler/Main.py", line 304, in compile > return context.compile(source, options, full_module_name) > File "/.../Cython/Compiler/Main.py", line 201, in compile > tree.process_implementation(scope, options, result) > File "/.../Cython/Compiler/ModuleNode.py", line 78, in > process_implementation > self.generate_c_code(env, options, result) > File "/.../Cython/Compiler/ModuleNode.py", line 262, in > generate_c_code > self.generate_declarations_for_modules(env, modules, code.h) > File "/.../Cython/Compiler/ModuleNode.py", line 454, in > generate_declarations_for_modules > vtabslot_list = self.generate_vtabslot_list(vtabslot_dict) > File "/.../Cython/Compiler/ModuleNode.py", line 418, in > generate_vtabslot_list > if(recurse_vtabslot_check_inheritance(vtab_list[j],vtab_list[i], > vtab_dict)==1): > File "/.../Cython/Compiler/ModuleNode.py", line 44, in > recurse_vtabslot_check_inheritance > base = dict[base.type.base_type.objstruct_cname] > KeyError: 'LxmlElementBase' > ----------------- > > This class is cimported from this definition in a .pxd: > > cdef extern from "lxml.etree_api.h": > cdef class lxml.etree.ElementBase(_Element) \ > [ object LxmlElementBase ]: > ... > > http://codespeak.net/svn/lxml/trunk/src/lxml/etreepublic.pxd > > by this Cython file: > > http://codespeak.net/svn/lxml/trunk/src/lxml/lxml.objectify.pyx > > > I don't think there is a test case for cimporting extension classes > yet. > > I'll have to see when I find the time to look into this, but maybe > not before > the end of the week. > > Stefan From stefan_ml at behnel.de Tue Apr 8 14:38:24 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 8 Apr 2008 14:38:24 +0200 (CEST) Subject: [Cython] New project : Spyke python-to-C compiler In-Reply-To: <2831E432-CA4B-4F62-94A7-C13D68F2B317@math.washington.edu> References: <20080407175606.d1tlnbczk0wo8sk0@webmail.ualberta.ca> <15BA280D-E9EB-4A4E-8604-D2133E3D279A@math.washington.edu> <20080407184837.4yn4yzipc888o44s@webmail.ualberta.ca> <2831E432-CA4B-4F62-94A7-C13D68F2B317@math.washington.edu> Message-ID: <58749.194.114.62.67.1207658304.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi, > On Apr 7, 2008, at 5:48 PM, Rahul Garg wrote: >> 2. I forgot to ask : Do you use some particular tests/benchmarks >> etc? I can try and adapt those to Spyke-syntax if some test cases >> are available. Besides the pybench test runs that Robert showed you, we also have a test suite. It's quite simple and based on doctest. http://hg.cython.org/cython-devel/file/e005b58d83b8/tests/run/ The main idea is to let Cython compile a .pyx source file to an extension module and run the doctest strings of that module from Python. This provides a very comfortable way of comparing test results between Cython and Python. Here is an example I like: http://hg.cython.org/cython-devel/file/e005b58d83b8/tests/run/unicodeliterals.pyx Stefan From stefan_ml at behnel.de Tue Apr 8 14:50:04 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 8 Apr 2008 14:50:04 +0200 (CEST) Subject: [Cython] Lisp inspired transforms In-Reply-To: <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> Message-ID: <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Though it is tempting to head down the language development path, > adding little (or big) features that make it more powerful than > Python itself, I think doing so will actually be counterproductive to > the goals stated above. Perhaps there could be a Cython++ that is a > proper superset of the Cython language with more powerful features > (though I'd hope not near the gap of C vs. C++) but in the near term > we should be focusing on things like being able to compile all of > Python as it is. I agree with Robert. As long as Cython does not support closures, for example, it cannot come close enough to being a real option for speeding up existing (non-static) Python code and making it easy to use for non-C-but-Python programmers. For the time being, we should try to a) implement as many Python (3?) language features as possible, keeping in mind that a correct implementation is more important than a fast one, especially for dynamic features that do not have a direct counterpart in the C world. b) get a well-designed and well-integrated compile-time code transformation infrastructure in place, thus allowing to provide pluggable language enhancements *later* and independent of the core compiler, which could then become an advanced Cython++ distribution (or make it back into mainstream). I see the major focus here on adjusting the line between compile-time and runtime code evaluation, and maybe some additional AOP features (as the ones Martin described). I think it would help Cython a *lot* to have a stable core language feature set that is well based in the Python language, *before* we start extending the language with all sorts of 'cool' new features that may already have some sort of (run-time) equivalent in Python. For a programming language, stability is a very valuable feature of its own - and there should (preferrably) be "one way to do it". That's why I like Martin's transformers, for example, they look like plain Python but run at compile time. I think that's the way to do it. Stefan From martin at martincmartin.com Tue Apr 8 15:20:32 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Tue, 08 Apr 2008 09:20:32 -0400 Subject: [Cython] Lisp inspired transforms In-Reply-To: <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <47FB7120.6030104@martincmartin.com> Stefan Behnel wrote: > Robert Bradshaw wrote: >> Though it is tempting to head down the language development path, >> adding little (or big) features that make it more powerful than >> Python itself, I think doing so will actually be counterproductive to >> the goals stated above. Perhaps there could be a Cython++ that is a >> proper superset of the Cython language with more powerful features >> (though I'd hope not near the gap of C vs. C++) but in the near term >> we should be focusing on things like being able to compile all of >> Python as it is. > > I agree with Robert. As long as Cython does not support closures, for > example, it cannot come close enough to being a real option for speeding > up existing (non-static) Python code and making it easy to use for > non-C-but-Python programmers. Both your and Robert's thoughts are very wise. > For the time being, we should try to > > a) implement as many Python (3?) language features as possible, keeping in > mind that a correct implementation is more important than a fast one, > especially for dynamic features that do not have a direct counterpart in > the C world. > > b) get a well-designed and well-integrated compile-time code > transformation infrastructure in place, thus allowing to provide pluggable > language enhancements *later* and independent of the core compiler, which > could then become an advanced Cython++ distribution (or make it back into > mainstream). I see the major focus here on adjusting the line between > compile-time and runtime code evaluation, and maybe some additional AOP > features (as the ones Martin described). > > I think it would help Cython a *lot* to have a stable core language > feature set that is well based in the Python language, *before* we start > extending the language with all sorts of 'cool' new features that may > already have some sort of (run-time) equivalent in Python. For a > programming language, stability is a very valuable feature of its own - > and there should (preferrably) be "one way to do it". That's why I like > Martin's transformers, for example, they look like plain Python but run at > compile time. I think that's the way to do it. Do you see deftrans as the "well-integrated compile-time code transformation infrastructure," and thus a target for (base) Cython, rather than Cython++? Or by infrastructure, do you mean a change to the compiler to support Python-coded transforms, without saying where these transforms come from? In the latter, deftrans (a syntatical convention for defining the transforms) becomes part of Cython++, and if the experiment works out, is incorporated back into Cython. Best, Martin From martin at martincmartin.com Tue Apr 8 15:46:24 2008 From: martin at martincmartin.com (Martin C. Martin) Date: Tue, 08 Apr 2008 09:46:24 -0400 Subject: [Cython] Lisp inspired transforms In-Reply-To: <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <47FB7730.4060503@martincmartin.com> To recap the motivation for having it in base Cython: - C++ template metaprogramming shows the value of compile time metaprogramming, i.e. writing programs that run at compile time and that produce programs as their output. - However, C++ templates are essentially a separate, purely functional programming language embedded in C++, with different syntax and semantics than C+. Of course, they were never really intended or designed for metaprogramming. But now that they're being used that way, it's clear that the substitution model of templates is awkward and unnatural. If we're going to write little functions to run at compile time, it makes sense to use the syntax and semantics of the base language. Compile time metaprogramming doesn't exist in Python, so adding it to Cython means extending Cython beyond what Python has. There are a couple options: 1. Add a way to generate C++ templates, and use that for metaprogramming. It keeps Cython as "writing both Python and C++ with an extended Python syntax." 2. Don't add metaprogramming, even C++ templates. Keeps Cython close to Python and C and C++, including all the good & bad parts. 3. Add a way to specify Python code for the transformation. This recognizes that metaprogramming is a valuable activity that Cython developers will want to do; that the existing way to do it in C++ is more-or-less not up to the task; and that it's better to provide a new, cleaner mechanism using what we've learned in hindsight. Of course, metaprogramming in an imperative, stateful language, opens a can of worms, e.g. it will be valuable to modify Python data at compile time, and have that serialized once all transforming is done, then loaded at the start of runtime. I don't think any of these problems are particularly difficult though. So, what do people this is the best way forward for Cython? Best, Martin From beach at verinet.com Tue Apr 8 16:33:33 2008 From: beach at verinet.com (David J. C. Beach) Date: Tue, 8 Apr 2008 08:33:33 -0600 Subject: [Cython] Fwd: Lisp inspired transforms References: Message-ID: <8AE9A196-EA5C-4C38-9152-7369AAE3790A@verinet.com> I've been experimenting with using Cython using a simple python preprocessor that I wrote. It would seem that everything I've done with my preprocessor could potentially be done with "Lisp inspired transforms". I've been particularly interested in creating a math library for small matrices and vectors which uses explicit loop- unrolling at compile time. I believe that the idea behind Cython calls for this type of metaprogramming more than Python itself. The reason for this is that Cython is capable of running sophisticated algorithms at C-speed, and this means that compile-time inlining, unrolling, and policy-based abstractions (a la Alexandrescu) become somewhat compelling. It also seems possible to implement generator functions in Pyrex using these code transforms. I do, however, understand the hesitance to add a Cython feature that falls completely outside both Python and C heritage. I have 11 years experience with Python and 13 with C++. The kind of hatred I've developed for C++ is the slow-building kind that grows over years of watching superior alternatives foregone because "C++ is an industry standard". I implore you *not* to adopt the C++ template meta-programming syntax! FWIW, I like options 3, 2, and 1, in that order. C++ is an absurdly hard-to-parse language. I fear that #1 could derail the project. Just my $0.02. David On Apr 8, 2008, at 7:46 AM, Martin C. Martin wrote: > To recap the motivation for having it in base Cython: > > - C++ template metaprogramming shows the value of compile time > metaprogramming, i.e. writing programs that run at compile time and > that > produce programs as their output. > > - However, C++ templates are essentially a separate, purely functional > programming language embedded in C++, with different syntax and > semantics than C+. Of course, they were never really intended or > designed for metaprogramming. But now that they're being used that > way, > it's clear that the substitution model of templates is awkward and > unnatural. If we're going to write little functions to run at compile > time, it makes sense to use the syntax and semantics of the base > language. > > Compile time metaprogramming doesn't exist in Python, so adding it to > Cython means extending Cython beyond what Python has. There are a > couple options: > > 1. Add a way to generate C++ templates, and use that for > metaprogramming. It keeps Cython as "writing both Python and C++ with > an extended Python syntax." > > 2. Don't add metaprogramming, even C++ templates. Keeps Cython > close to > Python and C and C++, including all the good & bad parts. > > 3. Add a way to specify Python code for the transformation. This > recognizes that metaprogramming is a valuable activity that Cython > developers will want to do; that the existing way to do it in C++ is > more-or-less not up to the task; and that it's better to provide a > new, > cleaner mechanism using what we've learned in hindsight. > > Of course, metaprogramming in an imperative, stateful language, > opens a > can of worms, e.g. it will be valuable to modify Python data at > compile > time, and have that serialized once all transforming is done, then > loaded at the start of runtime. I don't think any of these problems > are > particularly difficult though. > > So, what do people this is the best way forward for Cython? > > Best, > Martin > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -- David Beach From stefan_ml at behnel.de Tue Apr 8 18:09:37 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 8 Apr 2008 18:09:37 +0200 (CEST) Subject: [Cython] Lisp inspired transforms In-Reply-To: <47FB7730.4060503@martincmartin.com> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <47FB7730.4060503@martincmartin.com> Message-ID: <45162.194.114.62.67.1207670977.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Martin C. Martin wrote: > Compile time metaprogramming doesn't exist in Python, so adding it to > Cython means extending Cython beyond what Python has. Cython has a couple of additional features that make sense because it is a compiled language. I think what you call "metaprogramming" (and generally most things that allow doing things at compile-time instead of run-time) makes sense for Cython. > There are a couple options: > > 1. Add a way to generate C++ templates, and use that for > metaprogramming. It keeps Cython as "writing both Python and C++ with > an extended Python syntax." But that would be C++ specific and can't work with C. > 3. Add a way to specify Python code for the transformation. This > recognizes that metaprogramming is a valuable activity that Cython > developers will want to do; that the existing way to do it in C++ is > more-or-less not up to the task; and that it's better to provide a new, > cleaner mechanism using what we've learned in hindsight. I would say so. I would currently position it as a) an extension mechanism for Cython itself and b) an advanced feature that most people won't use (in the same way most people don't use metaclasses) - but as usual with OSS, you never know what people will use it for. > Of course, metaprogramming in an imperative, stateful language, opens a > can of worms, e.g. it will be valuable to modify Python data at compile > time, and have that serialized once all transforming is done, then > loaded at the start of runtime. I don't think any of these problems are > particularly difficult though. > > So, what do people this is the best way forward for Cython? I'll have to take a closer look at your proposal and compare it a bit more to the other approaches we had so far (especially Dag's work), before I make up my mind about it. Maybe others can already comment a bit deeper on this. Stefan From dagss at student.matnat.uio.no Tue Apr 8 18:47:42 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 08 Apr 2008 18:47:42 +0200 Subject: [Cython] Lisp inspired transforms In-Reply-To: <45162.194.114.62.67.1207670977.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <47FB7730.4060503@martincmartin.com> <45162.194.114.62.67.1207670977.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <47FBA1AE.7020004@student.matnat.uio.no> +1 for polishing it and provide option c) as a plugin for now and see how it goes, and discuss inclusion in main Cython after it has proven itself. > I'll have to take a closer look at your proposal and compare it a bit more > to the other approaches we had so far (especially Dag's work), before I > make up my mind about it. Maybe others can already comment a bit deeper on > this. > Since you bring up my name: a) Clean NumPy integration (that is, with only a pxd file, not a full NumPy plugin) needs some kind of metaprogramming support, but can either work with Martin's explicit approach or my implicit approach, doesn't matter much. (The plan is to not use meta-programming at first, but that will be slow and metaprogramming is key to getting full NumPy speed). b) About my work in relation to this, see the uneval page: http://wiki.cython.org/enhancements/uneval If Martin's work is accepted now, and my own approach for meta-programming is ever done later, then uneval provides a very natural bridge between them. The two seems to be very complementary. Martin's is "explicit" and simple but for advanced users, mine is "easy-to-use" for beginners but more difficult to really understand for advanced users. So doing Martin's first, and then see if my more complicated approach is really needed should be fine as long as uneval provides a natural transition path. uneval() would return the same kind of tree that Martin allows work on, whatever that tree ends up being (as I understand it the exact syntax used is an example, one should add a small API layer on top to isolate it more from Cython core). -- Dag Sverre From stefan_ml at behnel.de Tue Apr 8 18:43:41 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Apr 2008 18:43:41 +0200 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <99CAFB68-4433-4AFF-A25B-B093EF56D76A@math.washington.edu> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> <47F9E881.7080405@behnel.de> <99CAFB68-4433-4AFF-A25B-B093EF56D76A@math.washington.edu> Message-ID: <47FBA0BD.5050207@behnel.de> Hi, Robert Bradshaw wrote: > On Apr 7, 2008, at 2:25 AM, Stefan Behnel wrote: >> there seems to be a problem with cimporting public extension classes >> from another Cython module. > > I've fixed this now and released 0.9.6.13.1. Still broken, but in a different way. Changeset c2a988dd8e6c by Gary Furnish has this diff: ----------------------------------- @@ -356,23 +467,6 @@ class ModuleNode(Nodes.Node, Nodes.Block self.generate_struct_union_definition(entry, code) elif type.is_enum: self.generate_enum_definition(entry, code) - elif type.is_extension_type: - self.generate_obj_struct_definition(type, code) [...] ----------------------------------- removing these two lines keeps the extension type structs from appearing in the API header file. The changeset is pretty huge and contains more than one change, so I'm not sure I understand everything it does. Gary, could you explain it to me? Stefan From gfurnish at gfurnish.net Tue Apr 8 20:17:51 2008 From: gfurnish at gfurnish.net (Gary Furnish) Date: Tue, 8 Apr 2008 12:17:51 -0600 Subject: [Cython] Cython 0.9.6.13 Released In-Reply-To: <47FBA0BD.5050207@behnel.de> References: <4CCD2032-38DC-4334-9AD4-1B21A8EDF897@math.washington.edu> <47F9E881.7080405@behnel.de> <99CAFB68-4433-4AFF-A25B-B093EF56D76A@math.washington.edu> <47FBA0BD.5050207@behnel.de> Message-ID: <8f8f8530804081117q1dee0d66q7bc9188875bd22c8@mail.gmail.com> I'll have to look at the code later tonight. It is possible something got removed that should not have been. On Tue, Apr 8, 2008 at 10:43 AM, Stefan Behnel wrote: > Hi, > > Robert Bradshaw wrote: > > On Apr 7, 2008, at 2:25 AM, Stefan Behnel wrote: > >> there seems to be a problem with cimporting public extension classes > >> from another Cython module. > > > > I've fixed this now and released 0.9.6.13.1. > > Still broken, but in a different way. Changeset c2a988dd8e6c by Gary > Furnish > has this diff: > > ----------------------------------- > @@ -356,23 +467,6 @@ class ModuleNode(Nodes.Node, Nodes.Block > self.generate_struct_union_definition(entry, code) > elif type.is_enum: > self.generate_enum_definition(entry, code) > - elif type.is_extension_type: > - self.generate_obj_struct_definition(type, code) > [...] > ----------------------------------- > > removing these two lines keeps the extension type structs from appearing > in > the API header file. > > The changeset is pretty huge and contains more than one change, so I'm not > sure I understand everything it does. Gary, could you explain it to me? > > Stefan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080408/605036f4/attachment.htm From stefan_ml at behnel.de Tue Apr 8 19:50:31 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Apr 2008 19:50:31 +0200 Subject: [Cython] Cython circular cdef import patch In-Reply-To: <8f8f8530803251117x4efeeafch7243af2b9b302f2e@mail.gmail.com> References: <8f8f8530803251117x4efeeafch7243af2b9b302f2e@mail.gmail.com> Message-ID: <47FBB067.5080001@behnel.de> Hi, Gary Furnish wrote: > This patch adds extra logic to code generation to sort dependencies to > guarantee that C code is generated in the right order for circular cdef > imports. It does NOT solve python circular import issues. > a.pyx: cimport b > b.pyx: cimport a > Is thus legal, but you can not reference a or b in the global namespace of > b.pyx or a.pyx (such as to instantiate a class). > This patch also modifies cython to output the exact line in the C code where > an exception was thrown in addition to the currently displayed pyx file and > line. This enables significantly faster developmental debugging. > Finally it splits module initialization into two phases: one that initiates > types and handles imports, and another that executes python commands at the > global namespace level. This will be more useful as Cython starts to assume > more advanced optimization and code generation features > The patch is available at: > http://trac.sagemath.org/sage_trac/ticket/2655(the third attachment > only) and is based against 0.9.6.12, > although I can rebase if needed. I am hoping this can be merged into > Cython. The patch doesn't work for me. When I fix the problems with the header file generation, I end up with C code that defines the "__pyx_obj_*" class structs in the wrong order. Note that there are no circular dependencies involved in my case and that the two classes where the problem occurs are defined in the same source file (although their highest base class is defined in a separate file, not sure if that matters. Personally, seeing all problems I have with this patch and not really seeing a clear gain, I would just revert the patch for now. Stefan From gfurnish at gfurnish.net Wed Apr 9 00:00:54 2008 From: gfurnish at gfurnish.net (Gary Furnish) Date: Tue, 8 Apr 2008 16:00:54 -0600 Subject: [Cython] Cython circular cdef import patch In-Reply-To: <47FBB067.5080001@behnel.de> References: <8f8f8530803251117x4efeeafch7243af2b9b302f2e@mail.gmail.com> <47FBB067.5080001@behnel.de> Message-ID: <8f8f8530804081500p18b87acesd9e1d9103b67bfc6@mail.gmail.com> Can you produce a testcase for this issue? It is *critical* for fast symbolics in Sage, so I would prefer to fix the bug as opposed to reverting. I see the problem now, but I will need the test cases you are using to produce a fix. Once I have them it should be a relatively easy patch. On Tue, Apr 8, 2008 at 11:50 AM, Stefan Behnel wrote: > Hi, > > Gary Furnish wrote: > > This patch adds extra logic to code generation to sort dependencies to > > guarantee that C code is generated in the right order for circular cdef > > imports. It does NOT solve python circular import issues. > > a.pyx: cimport b > > b.pyx: cimport a > > Is thus legal, but you can not reference a or b in the global namespace > of > > b.pyx or a.pyx (such as to instantiate a class). > > This patch also modifies cython to output the exact line in the C code > where > > an exception was thrown in addition to the currently displayed pyx file > and > > line. This enables significantly faster developmental debugging. > > Finally it splits module initialization into two phases: one that > initiates > > types and handles imports, and another that executes python commands at > the > > global namespace level. This will be more useful as Cython starts to > assume > > more advanced optimization and code generation features > > The patch is available at: > > http://trac.sagemath.org/sage_trac/ticket/2655(thethird attachment > > only) and is based against 0.9.6.12, > > although I can rebase if needed. I am hoping this can be merged into > > Cython. > > The patch doesn't work for me. When I fix the problems with the header > file > generation, I end up with C code that defines the "__pyx_obj_*" class > structs > in the wrong order. Note that there are no circular dependencies involved > in > my case and that the two classes where the problem occurs are defined in > the > same source file (although their highest base class is defined in a > separate > file, not sure if that matters. > > Personally, seeing all problems I have with this patch and not really > seeing a > clear gain, I would just revert the patch for now. > > Stefan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080408/e2d4fe2f/attachment.htm From robertwb at math.washington.edu Wed Apr 9 01:32:03 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 8 Apr 2008 16:32:03 -0700 Subject: [Cython] Lisp inspired transforms In-Reply-To: <47FBA1AE.7020004@student.matnat.uio.no> References: <47F8C640.10901@martincmartin.com> <35BE3266-C5D5-4695-A90A-9B749E91453B@math.washington.edu> <47FA8C84.5070207@martincmartin.com> <489C6F51-72E7-4B39-B1BB-D1EF9C3B5365@math.washington.edu> <8568.194.114.62.67.1207659004.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <47FB7730.4060503@martincmartin.com> <45162.194.114.62.67.1207670977.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <47FBA1AE.7020004@student.matnat.uio.no> Message-ID: <8BF96D07-9478-4E38-87A8-64966FBCCC8B@math.washington.edu> On Apr 8,