From tokidokix at gmail.com Thu Jul 1 12:09:55 2010 From: tokidokix at gmail.com (toki doki) Date: Thu, 1 Jul 2010 19:09:55 +0900 Subject: [Cython] Some c++ bugs Message-ID: Hi, I have been trying to use cython-devel with c++ and ran into the following bugs (that seem to be unreported as far as I can say from a quick glance at Trac). First, and most annoying: in spite of what is written in the docs (http://wiki.cython.org/WrappingCPlusPlus), nested c++ class declarations seem to be unsupported. Thus, the following snippet (taken from the doc) for wrapping std::vector does not work and raise a syntax error: cdef extern from "" namespace "std": cdef cppclass vector[T]: cdef cppclass iterator: T operator*() iterator operator++() vector() void push_back(T&) T& operator[](int) iterator begin() iterator end() Second: ctypedef'ing templated types raises a syntax Error (for example: "ctypedef vector[int] VectOfInt" ). It would be nice if these two problems were solved for the cython0.13 release (or even before that, for those of us that dare to use cython-devel :-) Anyway, congratulations on a great and useful project. It (and before that, good old pyrex too) has been very useful for me during the years of my PhD. Can't wait for full c++ and generators support :-) Toki From stefan_ml at behnel.de Thu Jul 1 13:16:48 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 01 Jul 2010 13:16:48 +0200 Subject: [Cython] Some c++ bugs In-Reply-To: References: Message-ID: <4C2C7920.2070308@behnel.de> toki doki, 01.07.2010 12:09: > I have been trying to use cython-devel with c++ and ran into the > following bugs (that seem to be unreported as far as I can say from a > quick glance at Trac). > > First, and most annoying: in spite of what is written in the docs > (http://wiki.cython.org/WrappingCPlusPlus), nested c++ class > declarations seem to be unsupported. Thus, the following snippet > (taken from the doc) for wrapping std::vector does not work and raise > a syntax error: > > cdef extern from "" namespace "std": > cdef cppclass vector[T]: > cdef cppclass iterator: > T operator*() > iterator operator++() > vector() > void push_back(T&) > T& operator[](int) > iterator begin() > iterator end() > > Second: ctypedef'ing templated types raises a syntax Error (for > example: "ctypedef vector[int] VectOfInt" ). > > It would be nice if these two problems were solved for the cython0.13 > release (or even before that, for those of us that dare to use > cython-devel :-) Hi, since you seem to be interested in getting these implemented, why not help us by providing a test case for these features? See the C++ tests in the tests/ directory, as well as the docs: http://wiki.cython.org/HackerGuide Also, please open trac tickets for these two features. Thanks! Stefan From baihaoyu at gmail.com Thu Jul 1 13:40:06 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Thu, 1 Jul 2010 19:40:06 +0800 Subject: [Cython] Some c++ bugs In-Reply-To: References: Message-ID: On Thu, Jul 1, 2010 at 6:09 PM, toki doki wrote: > Hi, > > I have been trying to use cython-devel with c++ and ran into the > following bugs (that seem to be unreported as far as I can say from a > quick glance at Trac). > > First, and most annoying: in spite of what is written in the docs > (http://wiki.cython.org/WrappingCPlusPlus), nested c++ class > declarations seem to be unsupported. Thus, the following snippet > (taken from the doc) for wrapping std::vector does not work and raise > a syntax error: > > cdef extern from "" namespace "std": > ? ?cdef cppclass vector[T]: > ? ? ? ?cdef cppclass iterator: > ? ? ? ? ? ?T operator*() > ? ? ? ? ? ?iterator operator++() > ? ? ? ?vector() > ? ? ? ?void push_back(T&) > ? ? ? ?T& operator[](int) > ? ? ? ?iterator begin() > ? ? ? ?iterator end() > Refer to Cython/Includes/libcpp/vector.pxd for a example of nested class: cdef extern from "" namespace "std": cdef cppclass vector[T]: cppclass iterator: T& operator*() iterator operator++() iterator operator--() bint operator==(iterator) bint operator!=(iterator) ... > Second: ctypedef'ing templated types raises a syntax Error (for > example: "ctypedef vector[int] VectOfInt" ). > > It would be nice if these two problems were solved for the cython0.13 > release (or even before that, for those of us that dare to use > cython-devel :-) > > Anyway, congratulations on a great ?and useful project. It (and before > that, good old pyrex too) has been very useful for me during the years > of my PhD. Can't wait for full c++ and generators support :-) > > Toki For your second issue, I have a patch that let you do "cdef cppclass VectOfInt(vector[int]): pass" (which is also not supported by cython-devel yet), which would get the same effect as ctypedef. See the attached file. The patch is not thoroughly tested yet. Feel free to take it and improve it if you are interested. -- Haoyu BAI School of Computing, National University of Singapore. -------------- next part -------------- A non-text attachment was scrubbed... Name: templateinherit.diff Type: text/x-patch Size: 4768 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100701/d2236717/attachment.bin From dagss at student.matnat.uio.no Thu Jul 1 14:43:44 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Jul 2010 14:43:44 +0200 Subject: [Cython] Some c++ bugs In-Reply-To: References: Message-ID: <4C2C8D80.70408@student.matnat.uio.no> Haoyu Bai wrote: > On Thu, Jul 1, 2010 at 6:09 PM, toki doki wrote: > >> Hi, >> >> I have been trying to use cython-devel with c++ and ran into the >> following bugs (that seem to be unreported as far as I can say from a >> quick glance at Trac). >> >> First, and most annoying: in spite of what is written in the docs >> (http://wiki.cython.org/WrappingCPlusPlus), nested c++ class >> declarations seem to be unsupported. Thus, the following snippet >> (taken from the doc) for wrapping std::vector does not work and raise >> a syntax error: >> >> cdef extern from "" namespace "std": >> cdef cppclass vector[T]: >> cdef cppclass iterator: >> T operator*() >> iterator operator++() >> vector() >> void push_back(T&) >> T& operator[](int) >> iterator begin() >> iterator end() >> >> > > Refer to Cython/Includes/libcpp/vector.pxd for a example of nested class: > > cdef extern from "" namespace "std": > cdef cppclass vector[T]: > cppclass iterator: > T& operator*() > iterator operator++() > iterator operator--() > bint operator==(iterator) > bint operator!=(iterator) > ... > > >> Second: ctypedef'ing templated types raises a syntax Error (for >> example: "ctypedef vector[int] VectOfInt" ). >> >> It would be nice if these two problems were solved for the cython0.13 >> release (or even before that, for those of us that dare to use >> cython-devel :-) >> >> Anyway, congratulations on a great and useful project. It (and before >> that, good old pyrex too) has been very useful for me during the years >> of my PhD. Can't wait for full c++ and generators support :-) >> >> Toki >> > > For your second issue, I have a patch that let you do "cdef cppclass > VectOfInt(vector[int]): pass" (which is also not supported by > cython-devel yet), which would get the same effect as ctypedef. See > the attached file. The patch is not thoroughly tested yet. Feel free > to take it and improve it if you are interested. > I thought we only supported "cppclass" only in extern blocks, i.e. we don't ever emit code from Cython to create C++ classes? Dag Sverre From baihaoyu at gmail.com Thu Jul 1 15:15:28 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Thu, 1 Jul 2010 21:15:28 +0800 Subject: [Cython] Some c++ bugs In-Reply-To: <4C2C8D80.70408@student.matnat.uio.no> References: <4C2C8D80.70408@student.matnat.uio.no> Message-ID: On Thu, Jul 1, 2010 at 8:43 PM, Dag Sverre Seljebotn wrote: > Haoyu Bai wrote: >> On Thu, Jul 1, 2010 at 6:09 PM, toki doki wrote: >> >>> Hi, >>> >>> I have been trying to use cython-devel with c++ and ran into the >>> following bugs (that seem to be unreported as far as I can say from a >>> quick glance at Trac). >>> >>> First, and most annoying: in spite of what is written in the docs >>> (http://wiki.cython.org/WrappingCPlusPlus), nested c++ class >>> declarations seem to be unsupported. Thus, the following snippet >>> (taken from the doc) for wrapping std::vector does not work and raise >>> a syntax error: >>> >>> cdef extern from "" namespace "std": >>> ? ?cdef cppclass vector[T]: >>> ? ? ? ?cdef cppclass iterator: >>> ? ? ? ? ? ?T operator*() >>> ? ? ? ? ? ?iterator operator++() >>> ? ? ? ?vector() >>> ? ? ? ?void push_back(T&) >>> ? ? ? ?T& operator[](int) >>> ? ? ? ?iterator begin() >>> ? ? ? ?iterator end() >>> >>> >> >> Refer to Cython/Includes/libcpp/vector.pxd for a example of nested class: >> >> cdef extern from "" namespace "std": >> ? ? cdef cppclass vector[T]: >> ? ? ? ? cppclass iterator: >> ? ? ? ? ? ? T& operator*() >> ? ? ? ? ? ? iterator operator++() >> ? ? ? ? ? ? iterator operator--() >> ? ? ? ? ? ? bint operator==(iterator) >> ? ? ? ? ? ? bint operator!=(iterator) >> ? ? ? ? ... >> >> >>> Second: ctypedef'ing templated types raises a syntax Error (for >>> example: "ctypedef vector[int] VectOfInt" ). >>> >>> It would be nice if these two problems were solved for the cython0.13 >>> release (or even before that, for those of us that dare to use >>> cython-devel :-) >>> >>> Anyway, congratulations on a great ?and useful project. It (and before >>> that, good old pyrex too) has been very useful for me during the years >>> of my PhD. Can't wait for full c++ and generators support :-) >>> >>> Toki >>> >> >> For your second issue, I have a patch that let you do "cdef cppclass >> VectOfInt(vector[int]): pass" (which is also not supported by >> cython-devel yet), which would get the same effect as ctypedef. See >> the attached file. The patch is not thoroughly tested yet. Feel free >> to take it and improve it if you are interested. >> > I thought we only supported "cppclass" only in extern blocks, i.e. we > don't ever emit code from Cython to create C++ classes? > > Dag Sverre Oh sorry, there should not be a ": pass", my mistake. :) -- Haoyu BAI School of Computing, National University of Singapore. From baihaoyu at gmail.com Thu Jul 1 16:38:54 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Thu, 1 Jul 2010 22:38:54 +0800 Subject: [Cython] Some c++ bugs In-Reply-To: References: <4C2C8D80.70408@student.matnat.uio.no> Message-ID: On Thu, Jul 1, 2010 at 9:15 PM, Haoyu Bai wrote: > On Thu, Jul 1, 2010 at 8:43 PM, Dag Sverre Seljebotn > wrote: >> Haoyu Bai wrote: >>> On Thu, Jul 1, 2010 at 6:09 PM, toki doki wrote: >>> >>>> Hi, >>>> >>>> I have been trying to use cython-devel with c++ and ran into the >>>> following bugs (that seem to be unreported as far as I can say from a >>>> quick glance at Trac). >>>> >>>> First, and most annoying: in spite of what is written in the docs >>>> (http://wiki.cython.org/WrappingCPlusPlus), nested c++ class >>>> declarations seem to be unsupported. Thus, the following snippet >>>> (taken from the doc) for wrapping std::vector does not work and raise >>>> a syntax error: >>>> >>>> cdef extern from "" namespace "std": >>>> ? ?cdef cppclass vector[T]: >>>> ? ? ? ?cdef cppclass iterator: >>>> ? ? ? ? ? ?T operator*() >>>> ? ? ? ? ? ?iterator operator++() >>>> ? ? ? ?vector() >>>> ? ? ? ?void push_back(T&) >>>> ? ? ? ?T& operator[](int) >>>> ? ? ? ?iterator begin() >>>> ? ? ? ?iterator end() >>>> >>>> >>> >>> Refer to Cython/Includes/libcpp/vector.pxd for a example of nested class: >>> >>> cdef extern from "" namespace "std": >>> ? ? cdef cppclass vector[T]: >>> ? ? ? ? cppclass iterator: >>> ? ? ? ? ? ? T& operator*() >>> ? ? ? ? ? ? iterator operator++() >>> ? ? ? ? ? ? iterator operator--() >>> ? ? ? ? ? ? bint operator==(iterator) >>> ? ? ? ? ? ? bint operator!=(iterator) >>> ? ? ? ? ... >>> >>> >>>> Second: ctypedef'ing templated types raises a syntax Error (for >>>> example: "ctypedef vector[int] VectOfInt" ). >>>> >>>> It would be nice if these two problems were solved for the cython0.13 >>>> release (or even before that, for those of us that dare to use >>>> cython-devel :-) >>>> >>>> Anyway, congratulations on a great ?and useful project. It (and before >>>> that, good old pyrex too) has been very useful for me during the years >>>> of my PhD. Can't wait for full c++ and generators support :-) >>>> >>>> Toki >>>> >>> >>> For your second issue, I have a patch that let you do "cdef cppclass >>> VectOfInt(vector[int]): pass" (which is also not supported by >>> cython-devel yet), which would get the same effect as ctypedef. See >>> the attached file. The patch is not thoroughly tested yet. Feel free >>> to take it and improve it if you are interested. >>> >> I thought we only supported "cppclass" only in extern blocks, i.e. we >> don't ever emit code from Cython to create C++ classes? >> >> Dag Sverre > > Oh sorry, there should not be a ": pass", my mistake. :) > > No, "cdef cppclass VectOfInt(vector[int]): pass" is valid inside a extern block, which declares VectOfInt a class inherits vector[int] but do not have any additional method. And that's not to emit code to create a C++ class. -- Haoyu BAI School of Computing, National University of Singapore. From sturla at molden.no Thu Jul 1 19:17:05 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 01 Jul 2010 19:17:05 +0200 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: <4C2AEA6B.5080808@behnel.de> References: <4C2AEA6B.5080808@behnel.de> Message-ID: <4C2CCD91.6080101@molden.no> Stefan Behnel skrev: > Good luck. I don't mean to discourage you, but this is a pretty ambitious goal. > Python's VM is slow, indeed. It uses a stack-based VM instead of registers; it is implemented in the worst possible way, as code and data form a three of hash tables; it has reference counting instead of garbage collection (produces high cache traffic); it has no JIT compilation. It's doomed to be slow. Many implementations of Common Lisp and Scheme can compete with C (e.g. SBCL, CMUCL, Allegro, Clozure, Ikarus). None of the billion dollar investment that was made into making Lisp fast (during the AI hype 20-30 years ago) has made it into Python. There were even hardware processors (aka 'Lisp machines') that could run Lisp natively. We can sometimes read on Python lists that Lisp's performance comes from optional static typic. That is not the whole story. For example, enourmous efforsts were made for making memory mangement efficient (cons is very effcient, lists are transformed to contiguous arrays internally, tail-recursion is inlined, etc). Lot of this technology is now forgotten, as those who worked on it are retired or has lost interest. Python's memory management (reference counts and malloc/free) is the least efficient ever invented by man. Nothing is stored contiguously (a list of int is not a buffer of int, but an array of pointers). And what data traffic does it take to keep refcounts synchronized in the cache of multiple processors? It's almost a wonder that the computer does not burst into flames. LuaJIT is one or two orders of magnitude faster than Python. It is not just JIT and a proper GC that gives it a performance boost, it also uses a register-based VM (like Parrot and Dalvik). Lua is a dynamic language very similar to Python. There is nothing in the Python language that keeps it from performing like Lua. StrongTalk VM for Smalltalk performs close to C. Smalltalk is also a dynamic language like Python. Sun actually purchased StrongTalk to use the VM for Java -- it is Java's Hotspot JIT compiler. Many claim that Java's performance comes from static typing, but the VM was actually designed for a dynamic language. We all know how Java performes compared to Python, or even compared to C. Why the discrepancy when the VM was designed for Smalltalk? One thing is that much of the experience from boosting Lisp actually made it into StrongTalk, unlike CPython. Reinventing the wheel is not a good thing, as it implies discaring modern tiers. It's not rocket science to make a fast Python. We can make a compiler that runs Python on any of the VMs above (e.g. Android's Dalvik engine or Lua JIT). A Python front-end for LuaJIT would be nearly 100 times faster than the current CPython, with very little investment. It's just that nobody has bothered... However: The main strength of Python is the magnitude of extension libraries available. A completely new Python will be nearly useless on its own. That's the big discouragement. And it is easier to write some C++ or Cython than fixing CPython. Sturla From dalcinl at gmail.com Thu Jul 1 20:52:49 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 1 Jul 2010 15:52:49 -0300 Subject: [Cython] Py 2.7 deprecates PyCObject Message-ID: In Python 2.7, creating a PyCObject is marked with PendingDeprecationWarning... Should we special case 2.7 and use PyCapsule ? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From sturla at molden.no Fri Jul 2 06:03:12 2010 From: sturla at molden.no (Sturla Molden) Date: Fri, 02 Jul 2010 06:03:12 +0200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: References: Message-ID: <4C2D6500.8000206@molden.no> Lisandro Dalcin skrev: > In Python 2.7, creating a PyCObject is marked with > PendingDeprecationWarning... Should we special case 2.7 and use > PyCapsule ? > > All programs that use PyCObject become a security problem. PyCObject makes no "type checks" on the void pointer, and can therefore be used to crash the interpreter or execute exploit code as destructor. "Type checking" void pointers might seem like a misnomer. But what PyCapsule does is to associalte the pointer with a name (a character string), that sort of acts like an access password or run-time type information. Sturla From dalcinl at gmail.com Fri Jul 2 06:46:32 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Jul 2010 01:46:32 -0300 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2D6500.8000206@molden.no> References: <4C2D6500.8000206@molden.no> Message-ID: On 2 July 2010 01:03, Sturla Molden wrote: > Lisandro Dalcin skrev: >> In Python 2.7, creating a PyCObject is marked with >> PendingDeprecationWarning... Should we special case 2.7 and use >> PyCapsule ? >> >> > All programs that use PyCObject become a security problem. PyCObject > makes no "type checks" on the void pointer, and can therefore be used to > crash the interpreter or execute exploit code as destructor. "Type > checking" void pointers might seem like a misnomer. But what PyCapsule > does is to associalte the pointer with a name (a character string), that > sort of acts like an access password or run-time type information. > 1) PyCObject_FromVoidPtrAndDesc() and PyCObject_GetDesc() serve to associate extra info on the pointer. 2) IMHO, the way to better handle the security issue is by providing slots and API's in module and type objects. But at this point, discussing on this is pointless. I'm just asking if Cython should avoid the warning/error when running under "python -W default/error ..." . I think we should fix it, I volunteer to do it. Sturla, I should count your comment as a +1, right?. What other people think? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Jul 2 08:21:10 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 08:21:10 +0200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2D6500.8000206@molden.no> References: <4C2D6500.8000206@molden.no> Message-ID: <4C2D8556.3040809@behnel.de> Sturla Molden, 02.07.2010 06:03: > All programs that use PyCObject become a security problem. PyCObject > makes no "type checks" on the void pointer, and can therefore be used to > crash the interpreter or execute exploit code as destructor. Like that was hard to do from C code. Stefan From tokidokix at gmail.com Fri Jul 2 09:40:31 2010 From: tokidokix at gmail.com (toki doki) Date: Fri, 2 Jul 2010 16:40:31 +0900 Subject: [Cython] Some c++ bugs In-Reply-To: References: Message-ID: > Refer to Cython/Includes/libcpp/vector.pxd for a example of nested class: > > cdef extern from "" namespace "std": > ? ?cdef cppclass vector[T]: > ? ? ? ?cppclass iterator: > ? ? ? ? ? ?T& operator*() > ? ? ? ? ? ?iterator operator++() > ? ? ? ? ? ?iterator operator--() > ? ? ? ? ? ?bint operator==(iterator) > ? ? ? ? ? ?bint operator!=(iterator) > ? ? ? ?... Thanks!! That solve my problem. It was only a "documentation bug". Then, the page "http://wiki.cython.org/WrappingCPlusPlus" needs a couple of updates: 1- Remove the cdef in front of "cppclass iterator" in the example of the section "Templates". 2- A note on how nested class are declared without a cdef should be added. I asked for an access to the wiki and trac. If I am granted access, I can do these modifications. But if somebody who has already access want to do it before, please do so :-) . > > For your second issue, I have a patch that let you do "cdef cppclass > VectOfInt(vector[int]): pass" (which is also not supported by > cython-devel yet), which would get the same effect as ctypedef. See > the attached file. The patch is not thoroughly tested yet. Feel free > to take it and improve it if you are interested. That looks like it would be a good workaround. I will try it and report any bugs I find. Will this patch be included in cython0.13 ? Thanks again, Toki From tokidokix at gmail.com Fri Jul 2 09:42:34 2010 From: tokidokix at gmail.com (toki doki) Date: Fri, 2 Jul 2010 16:42:34 +0900 Subject: [Cython] Some c++ bugs In-Reply-To: <4C2C7920.2070308@behnel.de> References: <4C2C7920.2070308@behnel.de> Message-ID: > > Hi, > > since you seem to be interested in getting these implemented, why not help > us by providing a test case for these features? See the C++ tests in the > tests/ directory, as well as the docs: > > http://wiki.cython.org/HackerGuide > > Also, please open trac tickets for these two features. > > Thanks! Actually, Haoyu Bai already solved my problems (see the other mail). There would still be some small changes needed to the documentation. And maybe a "wishlist bug report" for a ctypedef really working with templated types (instead of Haoyu Bai workaround). I will ask for an access to Trac and the doc wiki. For this and maybe future bugs I encounter. The procedure is to just send a login name and password to Robert Bradshaw, right? Best, Toki From tokidokix at gmail.com Fri Jul 2 12:25:00 2010 From: tokidokix at gmail.com (toki doki) Date: Fri, 2 Jul 2010 19:25:00 +0900 Subject: [Cython] More c++ bugs Message-ID: Well, I just found three more c++ bugs. I will file them when I have Trac access. In the meantime, I post them here before I forget about them this week-end. ******************************************************* * Bug#1: ******************************************************* Compiler crash when a template argument is an undeclared object. Compiling the following code: """"" from libcpp.vector cimport vector cdef vector[x] *vect """"" crash the compiler with the following traceback: """"" Traceback (most recent call last): File "/usr/local/bin/cython", line 8, in main(command_line = 1) File "/home/toki/install/cython-devel/Cython/Compiler/Main.py", line 776, in main result = compile(sources, options) File "/home/toki/install/cython-devel/Cython/Compiler/Main.py", line 751, in compile return compile_multiple(source, options) File "/home/toki/install/cython-devel/Cython/Compiler/Main.py", line 723, in compile_multiple result = run_pipeline(source, options) File "/home/toki/install/cython-devel/Cython/Compiler/Main.py", line 593, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/home/toki/install/cython-devel/Cython/Compiler/Main.py", line 234, in run_pipeline data = phase(data) File "/home/toki/install/cython-devel/Cython/Compiler/ParseTreeTransforms.py", line 965, in __call__ return super(AnalyseDeclarationsTransform, self).__call__(root) File "/home/toki/install/cython-devel/Cython/Compiler/Visitor.py", line 276, in __call__ return super(CythonTransform, self).__call__(node) File "/home/toki/install/cython-devel/Cython/Compiler/Visitor.py", line 259, in __call__ return self.visit(root) File "/home/toki/install/cython-devel/Cython/Compiler/Visitor.py", line 28, in visit return handler_method(obj) File "/home/toki/install/cython-devel/Cython/Compiler/ParseTreeTransforms.py", line 973, in visit_ModuleNode node.analyse_declarations(self.env_stack[-1]) File "/home/toki/install/cython-devel/Cython/Compiler/ModuleNode.py", line 63, in analyse_declarations self.body.analyse_declarations(env) File "/home/toki/install/cython-devel/Cython/Compiler/Nodes.py", line 341, in analyse_declarations stat.analyse_declarations(env) File "/home/toki/install/cython-devel/Cython/Compiler/Nodes.py", line 911, in analyse_declarations base_type = self.base_type.analyse(env) File "/home/toki/install/cython-devel/Cython/Compiler/Nodes.py", line 836, in analyse self.type = base_type.specialize_here(self.pos, template_types) File "/home/toki/install/cython-devel/Cython/Compiler/PyrexTypes.py", line 1924, in specialize_here return self.specialize(dict(zip(self.templates, template_values))) File "/home/toki/install/cython-devel/Cython/Compiler/PyrexTypes.py", line 1940, in specialize specialized.scope = self.scope.specialize(values) File "/home/toki/install/cython-devel/Cython/Compiler/Symtab.py", line 1605, in specialize entry.cname) File "/home/toki/install/cython-devel/Cython/Compiler/Symtab.py", line 350, in declare_type entry = self.declare(name, cname, type, pos, visibility) File "/home/toki/install/cython-devel/Cython/Compiler/Symtab.py", line 306, in declare if type.is_buffer and not isinstance(self, LocalScope): AttributeError: 'NoneType' object has no attribute 'is_buffer' """""" ************************************************* * Bug#2: ************************************************** operator[] doesn't return the right type. If cppclass Foo contain the declaration: "T operator[](int)", and foo is an instance of Foo, foo[4] is thought by cython to be of type Foo (instead of T) . For example: """"" from libcpp.vector cimport vector cdef vector[int] *vect=new vector[int]() vect.push_back(5) vect.push_back(6) print vect[0] """"" gives the following error: """"" print vect[0] ^ /home/toki/progs/cython13tests/bug1.pyx:6:10: Cannot convert 'vector' to Python object """"" **************************************************** * Bug#3: **************************************************** cython.operator.dereference and functions returning a reference are not recognized as l-values. For example, the following code: """"" cimport cython.operator.dereference from libcpp.vector cimport vector cdef vector[int] *vect=new vector[int]() vect.push_back(5) vect.push_back(6) cdef vector[int].iterator iter=vect.begin() cython.operator.dereference(iter)=8 """""" gives the following error: """""" cython.operator.dereference(iter)=5 ^ /home/toki/progs/cython13tests/bug2.pyx:8:27: Cannot assign to or delete this """""" In the same way, replacing the last line by: "vect.at(0)=5" gives: """""" vect.at(0)=5 ^ /home/toki/progs/cython13tests/bug2.pyx:7:7: Cannot assign to or delete this """""" While " *iter=5 " and " vect.at(0)=5 " ares valid C++ code. ************************************************************* That will be all for today :-) From stefan_ml at behnel.de Fri Jul 2 12:39:31 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 12:39:31 +0200 Subject: [Cython] More c++ bugs In-Reply-To: References: Message-ID: <4C2DC1E3.2030307@behnel.de> toki doki, 02.07.2010 12:25: > **************************************************** > * Bug#3: > **************************************************** > cython.operator.dereference and functions returning a reference are > not recognized as l-values. > > For example, the following code: > """"" > cimport cython.operator.dereference > from libcpp.vector cimport vector > > cdef vector[int] *vect=new vector[int]() > vect.push_back(5) > vect.push_back(6) > cdef vector[int].iterator iter=vect.begin() > cython.operator.dereference(iter)=8 > """""" > gives the following error: > > """""" > cython.operator.dereference(iter)=5 > ^ > /home/toki/progs/cython13tests/bug2.pyx:8:27: Cannot assign to or delete this > """""" > > In the same way, replacing the last line by: "vect.at(0)=5" gives: > > """""" > vect.at(0)=5 > ^ > /home/toki/progs/cython13tests/bug2.pyx:7:7: Cannot assign to or delete this > """""" > > While " *iter=5 " and " vect.at(0)=5 " ares valid C++ code. ... bug not valid Cython code. If this is to be supported at all, it needs at least a different spelling. Stefan From sturla at molden.no Fri Jul 2 14:06:16 2010 From: sturla at molden.no (Sturla Molden) Date: Fri, 2 Jul 2010 14:06:16 +0200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2D8556.3040809@behnel.de> References: <4C2D6500.8000206@molden.no> <4C2D8556.3040809@behnel.de> Message-ID: Den 2. juli 2010 kl. 08.21 skrev Stefan Behnel : > Sturla Molden, 02.07.2010 06:03: >> All programs that use PyCObject become a security problem. PyCObject >> makes no "type checks" on the void pointer, and can therefore be >> used to >> crash the interpreter or execute exploit code as destructor. > > Like that was hard to do from C PyCObject opens for exploits from Python code. Sturla From stefan_ml at behnel.de Fri Jul 2 14:16:10 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 14:16:10 +0200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: References: <4C2D6500.8000206@molden.no> <4C2D8556.3040809@behnel.de> Message-ID: <4C2DD88A.3060601@behnel.de> Sturla Molden, 02.07.2010 14:06: > Den 2. juli 2010 kl. 08.21 skrev Stefan Behnel: > >> Sturla Molden, 02.07.2010 06:03: >>> All programs that use PyCObject become a security problem. PyCObject >>> makes no "type checks" on the void pointer, and can therefore be >>> used to >>> crash the interpreter or execute exploit code as destructor. >> >> Like that was hard to do from C > > PyCObject opens for exploits from Python code. Seriously, if I can make you run my Python code on your server, I doubt that PyCObject is your main problem. Stefan From dagss at student.matnat.uio.no Fri Jul 2 14:31:34 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 02 Jul 2010 14:31:34 +0200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2DD88A.3060601@behnel.de> References: <4C2D6500.8000206@molden.no> <4C2D8556.3040809@behnel.de> <4C2DD88A.3060601@behnel.de> Message-ID: <4C2DDC26.40908@student.matnat.uio.no> Stefan Behnel wrote: > Sturla Molden, 02.07.2010 14:06: > >> Den 2. juli 2010 kl. 08.21 skrev Stefan Behnel: >> >> >>> Sturla Molden, 02.07.2010 06:03: >>> >>>> All programs that use PyCObject become a security problem. PyCObject >>>> makes no "type checks" on the void pointer, and can therefore be >>>> used to >>>> crash the interpreter or execute exploit code as destructor. >>>> >>> Like that was hard to do from C >>> >> PyCObject opens for exploits from Python code. >> > > Seriously, if I can make you run my Python code on your server, I doubt > that PyCObject is your main problem. > Google AppEngine, that kind of stuff. The point would be to make it faster to audit the code for such purposes, I think. Dag Sverre From stefan_ml at behnel.de Fri Jul 2 14:46:01 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 14:46:01 +0200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2D6500.8000206@molden.no> References: <4C2D6500.8000206@molden.no> Message-ID: <4C2DDF89.4000504@behnel.de> Sturla Molden, 02.07.2010 06:03: > But what PyCapsule > does is to associalte the pointer with a name (a character string), that > sort of acts like an access password or run-time type information. Note that Pyrex and Cython have been using this mechanism right from the beginning, when PyCapsule wasn't even in the makings. This has nothing to do with "security", just "safety" at most. Stefan From luiji at users.sourceforge.net Fri Jul 2 15:44:59 2010 From: luiji at users.sourceforge.net (Luiji Maryo) Date: Fri, 2 Jul 2010 09:44:59 -0400 Subject: [Cython] (no subject) In-Reply-To: <4C2B2ACD.5050905@student.matnat.uio.no> References: <4C2B2ACD.5050905@student.matnat.uio.no> Message-ID: Yeah, I also realized that the figures make no sense, since Python runs better then 10,000x slower then C, and I find it theoretically impossible for it to be faster then C. I am going to leave some libraries such as Tk alone, since they are fast enough. I'm mostly focusing on optimizing the mathematical libraries. On Wed, Jun 30, 2010 at 7:30 AM, Dag Sverre Seljebotn wrote: > Luiji Maryo wrote: >> Hello, >> >> For a project I am working on I am creating a super-optimized Python >> distrobution of which I hope will reach heights of 10,000 times faster >> code. ?What I am planning to do is bundle Psyco with a refactor of the >> > I hope you'll adjust that figure. Take for instance matrix muliplication > -- that's probably one of the benchmarks you'll find where Python > performs worst. But even comparing that to specifically CPU-tuned > assembly code using SSE instructions, Python is only about 4000-5000 > times slower :-) > > For more "normal" OOP code, we're talking between 2x and 100x I think. > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- - Luiji Maryo (a.k.a. Brain Boy) Visit me at http://brainboyblogger.blogspot.com/. From luiji at users.sourceforge.net Fri Jul 2 15:57:26 2010 From: luiji at users.sourceforge.net (Luiji Maryo) Date: Fri, 2 Jul 2010 09:57:26 -0400 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: <4C2CCD91.6080101@molden.no> References: <4C2AEA6B.5080808@behnel.de> <4C2CCD91.6080101@molden.no> Message-ID: Responses to Stefan Behnel: I do know the goal is ambitious, but less then you probably think. Let me clarify and say that I am only optimizing modules that will benefit, such as the mathematical libraries. Libraries such as Tk can be left alone, since they are simply wrappers for the Tk native library (IIRC). As for project background, this has to do with an argument with colleges of mine. We wanted to make a game, and I brought up the greatness of Python, but they said that it would be too slow for game development. Even though it is sufficiently fast, the game we were working on was very graphics intensive and when comparing a C version to a Python version of basic design philosophy C was noticably faster. However, I brought up the fact that Python is much quicker to code and outputs smaller files, and is much more cross-platformable. They argued that development speed could be sacraficed for game speed, and in the end I suggested to solve the issue by making a faster Python. They were quickly discouraged, and desided to pause the project for awhile so that I can get Python closer to C speed while they work on other games. Unladen Swallow seems to have the same project goal as me: to make a noticably faster Python implementation. I will look more into that project. Thanks for the information! Responses to Sturla Molden: Yes, CPython does seem to avoid very great concepts such as register-based memory management, garbage collection over reference counting, and JIT, which is why I decided as part of the optimized Python to use Pysco, Unladen Swallow, or another optimized Python interpreter. Thank you for your comments (and for adding the title that I moronically forgot), and I will start doing some optimizations. Any further development on this will be found at http://luiji.github.com/, but for now you probably will not see much development. If anybody is interested in the project, then you can subscribe to my GitHub account to a notification when I create a new repository for the optimized Python Standard Library (PSL as I will call it in further e-mails, since stdlib sounds too much like the C Stdlib). Thanks, - Luiji Maryo On Thu, Jul 1, 2010 at 1:17 PM, Sturla Molden wrote: > Stefan Behnel skrev: >> Good luck. I don't mean to discourage you, but this is a pretty ambitious goal. >> > > Python's VM is slow, indeed. It uses a stack-based VM instead of > registers; it is implemented in the worst possible way, as code and data > form a three of hash tables; it has reference counting instead of > garbage collection (produces high cache traffic); it has no JIT > compilation. It's doomed to be slow. > > Many implementations of Common Lisp and Scheme can compete with ?C (e.g. > SBCL, CMUCL, Allegro, Clozure, Ikarus). None of the billion dollar > investment that was made into making Lisp fast (during the AI hype 20-30 > years ago) has made it into Python. There were even hardware processors > (aka 'Lisp machines') that could run Lisp natively. We can sometimes > read on Python lists that Lisp's performance comes from optional static > typic. That is not the whole story. For example, enourmous efforsts were > made for making memory mangement efficient (cons is very effcient, lists > are transformed to contiguous arrays internally, tail-recursion is > inlined, etc). Lot of this technology is now forgotten, as those who > worked on it are retired or has lost interest. > > Python's memory management (reference counts and malloc/free) is the > least efficient ever invented by man. Nothing is stored contiguously (a > list of int is not a buffer of int, but an array of pointers). And what > data traffic does it take to keep refcounts synchronized in the cache of > multiple processors? It's almost a wonder that the computer does not > burst into flames. > > LuaJIT is one or two orders of magnitude faster than Python. It is not > just JIT and a proper GC that gives it a performance boost, it also uses > a register-based VM (like Parrot and Dalvik). Lua is a dynamic language > very similar to Python. There is nothing in the Python language that > keeps it from performing like Lua. > > StrongTalk VM for Smalltalk performs close to C. Smalltalk is also a > dynamic language like Python. Sun actually purchased StrongTalk to use > the VM for Java -- it is Java's Hotspot JIT compiler. Many claim that > Java's performance comes from static typing, but the VM was actually > designed for a dynamic language. We all know how Java performes compared > to Python, or even compared to C. Why the discrepancy when the VM was > designed for Smalltalk? One thing is that much of the experience from > boosting Lisp actually made it into StrongTalk, unlike CPython. > Reinventing the wheel is not a good thing, as it implies discaring > modern tiers. > > It's not rocket science to make a fast Python. We can make a compiler > that runs Python on any of the VMs above (e.g. Android's Dalvik engine > or Lua JIT). A Python front-end for LuaJIT would be nearly 100 times > faster than the current CPython, with very little investment. It's just > that nobody has bothered... > > However: The main strength of Python is the magnitude of extension > libraries available. A completely new Python will be nearly useless on > its own. That's the big discouragement. And it is easier to write some > C++ or Cython than fixing CPython. > > Sturla > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- - Luiji Maryo (a.k.a. Brain Boy) Visit me at http://brainboyblogger.blogspot.com/. From dalcinl at gmail.com Fri Jul 2 16:00:16 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Jul 2010 11:00:16 -0300 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2DDF89.4000504@behnel.de> References: <4C2D6500.8000206@molden.no> <4C2DDF89.4000504@behnel.de> Message-ID: On 2 July 2010 09:46, Stefan Behnel wrote: > Sturla Molden, 02.07.2010 06:03: >> But what PyCapsule >> does is to associalte the pointer with a name (a character string), that >> sort of acts like an access password or run-time type information. > > Note that Pyrex and Cython have been using this mechanism right from the > beginning, when PyCapsule wasn't even in the makings. This has nothing to > do with "security", just "safety" at most. > Please focus :-) and comment on my original questions... Should Cython switch to PyCapsule for Py>=2.7 && Py<3.0 ?? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Jul 2 16:04:29 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 16:04:29 +0200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: References: Message-ID: <4C2DF1ED.6010107@behnel.de> Lisandro Dalcin, 01.07.2010 20:52: > In Python 2.7, creating a PyCObject is marked with > PendingDeprecationWarning... Should we special case 2.7 and use > PyCapsule ? Good question. I think we should not change anything for 2.6, even if PyCapsule is available there, but given the warning, I think it makes sense to switch for 2.7, yes. Stefan From luiji at users.sourceforge.net Fri Jul 2 16:05:00 2010 From: luiji at users.sourceforge.net (Luiji Maryo) Date: Fri, 2 Jul 2010 10:05:00 -0400 Subject: [Cython] Forums vs. Mailing Lists Message-ID: I suggest that Cython switch from a mailing list system to a forum system, as forums have a few very good advantages: 1. The posts can be edited in the case of mistake. 2. The posts may be reported as spam. 3. It is much easier to have a long thread as supposed to individual responses creating separate threads. 4. Giving out your e-mail address to everybody using the forum is not a requirement. If mailing lists are in fact advantageous, please explain why. -- - Luiji Maryo (a.k.a. Brain Boy) Visit me at http://brainboyblogger.blogspot.com/. From stefan_ml at behnel.de Fri Jul 2 16:19:40 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 16:19:40 +0200 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: References: <4C2AEA6B.5080808@behnel.de> <4C2CCD91.6080101@molden.no> Message-ID: <4C2DF57C.8020405@behnel.de> Luiji Maryo, 02.07.2010 15:57: > As for project background, this has to do with an argument with > colleges of mine. We wanted to make a game, and I brought up the > greatness of Python, but they said that it would be too slow for game > development. Even though it is sufficiently fast, the game we were > working on was very graphics intensive and when comparing a C version > to a Python version of basic design philosophy C was noticably faster. > > However, I brought up the fact that Python is much quicker to code and > outputs smaller files, and is much more cross-platformable. They > argued that development speed could be sacraficed for game speed, and > in the end I suggested to solve the issue by making a faster Python. > They were quickly discouraged, and desided to pause the project for > awhile so that I can get Python closer to C speed while they work on > other games. I may be mistaken, but I would expect that in the games business, time-to-market is much more important than raw speed. Isn't performance mostly left to graphics cards these days? If you assume the development of a game to take 3-5 years, there's a lot you can expect the mainstream computers to do for you in terms of performance at the time where you finally hit the market. Implementing everything in C or C++ right from the start sounds like a premature optimisation to me. I assume you were using tools like PyGame, PyOpenGL, CLyther etc.? Stefan From rshepard at appl-ecosys.com Fri Jul 2 16:14:36 2010 From: rshepard at appl-ecosys.com (Rich Shepard) Date: Fri, 2 Jul 2010 07:14:36 -0700 (PDT) Subject: [Cython] Forums vs. Mailing Lists In-Reply-To: References: Message-ID: On Fri, 2 Jul 2010, Luiji Maryo wrote: > I suggest that Cython switch from a mailing list system to a forum > system, as forums have a few very good advantages: > 1. The posts can be edited in the case of mistake. > 2. The posts may be reported as spam. > 3. It is much easier to have a long thread as supposed to individual > responses creating separate threads. > 4. Giving out your e-mail address to everybody using the forum is not > a requirement. I strongly disagree. > If mailing lists are in fact advantageous, please explain why. Mail messages can -- and should be -- edited and spell-checked before being sent. Errors that are missed can be corrected in a follow up messasge. Mail messages can be filtered for spam, and that's what most of us do. Spam that avoids the filters can be reported to the hosting service. It is also easier to set up and maintain spam filters on a MTA than on a Web-based forum. Mailing lists push messages to subscribers. Web-based fora require subscribers to actively go there and pull down messages of interest. MUAs (Mail User Agents; a.k.a. 'mail readers') thread messages; there is no individual thread per message unless the subject line is changed. If you are ashamed of being identified with messages you post you should consider not posting anything. Why would anyone take seriously what you write if you hide your identity? Further, you can subscribe using whatever username you wish, including list-specific aliases if you wish. This topic comes up now and then on most mail lists where those who never used computers or the 'Net before the WWW think that's the only way to go. Many of us prefer typing in terminals and working on the command line and you can always follow a mail list on gmane and similar services without forcing everyone into that mode. Rich From stefan_ml at behnel.de Fri Jul 2 16:21:20 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 16:21:20 +0200 Subject: [Cython] Forums vs. Mailing Lists In-Reply-To: References: Message-ID: <4C2DF5E0.9050803@behnel.de> Luiji Maryo, 02.07.2010 16:05: > I suggest that Cython switch from a mailing list system to a forum > system, as forums have a few very good advantages: > 1. The posts can be edited in the case of mistake. > 2. The posts may be reported as spam. > 3. It is much easier to have a long thread as supposed to individual > responses creating separate threads. > 4. Giving out your e-mail address to everybody using the forum is not > a requirement. > > If mailing lists are in fact advantageous, please explain why. Sorry, what? why? This isn't worth discussing, IMHO. Stefan From rtreagan at gmail.com Fri Jul 2 16:58:31 2010 From: rtreagan at gmail.com (Russell Reagan) Date: Fri, 2 Jul 2010 09:58:31 -0500 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: References: <4C2AEA6B.5080808@behnel.de> <4C2CCD91.6080101@molden.no> Message-ID: <2D9406AA-20F9-437B-B7DD-DC6E11BEF34A@gmail.com> I am curious, in what ways does Cython not meet your needs? I think it would be beneficial to know, in order to identify areas which Cython needs to be improved in order to be more useful in real world projects. I have not done a great deal of work in Cython yet, but my perception is that with Cython, you should be able to write in Python, modify the Python with augmenting .pxd files, and get close to the equivalent speed of C code. So I am curious if: 1. My theoretical view of Cython is flawed. Or... 2. Practice currently has not caught up to theory. Or... 3. Cython is not intended to address your specific situation, and I have misunderstood your intentions and goals. On Jul 2, 2010, at 8:57 AM, Luiji Maryo wrote: > Responses to Stefan Behnel: > > I do know the goal is ambitious, but less then you probably think. > Let me clarify and say that I am only optimizing modules that will > benefit, such as the mathematical libraries. Libraries such as Tk can > be left alone, since they are simply wrappers for the Tk native > library (IIRC). > > As for project background, this has to do with an argument with > colleges of mine. We wanted to make a game, and I brought up the > greatness of Python, but they said that it would be too slow for game > development. Even though it is sufficiently fast, the game we were > working on was very graphics intensive and when comparing a C version > to a Python version of basic design philosophy C was noticably faster. > > However, I brought up the fact that Python is much quicker to code and > outputs smaller files, and is much more cross-platformable. They > argued that development speed could be sacraficed for game speed, and > in the end I suggested to solve the issue by making a faster Python. > They were quickly discouraged, and desided to pause the project for > awhile so that I can get Python closer to C speed while they work on > other games. > > Unladen Swallow seems to have the same project goal as me: to make a > noticably faster Python implementation. I will look more into that > project. Thanks for the information! > > Responses to Sturla Molden: > > Yes, CPython does seem to avoid very great concepts such as > register-based memory management, garbage collection over reference > counting, and JIT, which is why I decided as part of the optimized > Python to use Pysco, Unladen Swallow, or another optimized Python > interpreter. > > Thank you for your comments (and for adding the title that I > moronically forgot), and I will start doing some optimizations. Any > further development on this will be found at http://luiji.github.com/, > but for now you probably will not see much development. If anybody is > interested in the project, then you can subscribe to my GitHub account > to a notification when I create a new repository for the optimized > Python Standard Library (PSL as I will call it in further e-mails, > since stdlib sounds too much like the C Stdlib). > > Thanks, > - Luiji Maryo > > On Thu, Jul 1, 2010 at 1:17 PM, Sturla Molden > wrote: >> Stefan Behnel skrev: >>> Good luck. I don't mean to discourage you, but this is a pretty >>> ambitious goal. >>> >> >> Python's VM is slow, indeed. It uses a stack-based VM instead of >> registers; it is implemented in the worst possible way, as code and >> data >> form a three of hash tables; it has reference counting instead of >> garbage collection (produces high cache traffic); it has no JIT >> compilation. It's doomed to be slow. >> >> Many implementations of Common Lisp and Scheme can compete with C >> (e.g. >> SBCL, CMUCL, Allegro, Clozure, Ikarus). None of the billion dollar >> investment that was made into making Lisp fast (during the AI hype >> 20-30 >> years ago) has made it into Python. There were even hardware >> processors >> (aka 'Lisp machines') that could run Lisp natively. We can sometimes >> read on Python lists that Lisp's performance comes from optional >> static >> typic. That is not the whole story. For example, enourmous efforsts >> were >> made for making memory mangement efficient (cons is very effcient, >> lists >> are transformed to contiguous arrays internally, tail-recursion is >> inlined, etc). Lot of this technology is now forgotten, as those who >> worked on it are retired or has lost interest. >> >> Python's memory management (reference counts and malloc/free) is the >> least efficient ever invented by man. Nothing is stored >> contiguously (a >> list of int is not a buffer of int, but an array of pointers). And >> what >> data traffic does it take to keep refcounts synchronized in the >> cache of >> multiple processors? It's almost a wonder that the computer does not >> burst into flames. >> >> LuaJIT is one or two orders of magnitude faster than Python. It is >> not >> just JIT and a proper GC that gives it a performance boost, it also >> uses >> a register-based VM (like Parrot and Dalvik). Lua is a dynamic >> language >> very similar to Python. There is nothing in the Python language that >> keeps it from performing like Lua. >> >> StrongTalk VM for Smalltalk performs close to C. Smalltalk is also a >> dynamic language like Python. Sun actually purchased StrongTalk to >> use >> the VM for Java -- it is Java's Hotspot JIT compiler. Many claim that >> Java's performance comes from static typing, but the VM was actually >> designed for a dynamic language. We all know how Java performes >> compared >> to Python, or even compared to C. Why the discrepancy when the VM was >> designed for Smalltalk? One thing is that much of the experience from >> boosting Lisp actually made it into StrongTalk, unlike CPython. >> Reinventing the wheel is not a good thing, as it implies discaring >> modern tiers. >> >> It's not rocket science to make a fast Python. We can make a compiler >> that runs Python on any of the VMs above (e.g. Android's Dalvik >> engine >> or Lua JIT). A Python front-end for LuaJIT would be nearly 100 times >> faster than the current CPython, with very little investment. It's >> just >> that nobody has bothered... >> >> However: The main strength of Python is the magnitude of extension >> libraries available. A completely new Python will be nearly useless >> on >> its own. That's the big discouragement. And it is easier to write >> some >> C++ or Cython than fixing CPython. >> >> Sturla >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > - Luiji Maryo (a.k.a. Brain Boy) > Visit me at http://brainboyblogger.blogspot.com/. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From tokidokix at gmail.com Fri Jul 2 17:57:20 2010 From: tokidokix at gmail.com (toki doki) Date: Sat, 3 Jul 2010 00:57:20 +0900 Subject: [Cython] More c++ bugs In-Reply-To: <4C2DC1E3.2030307@behnel.de> References: <4C2DC1E3.2030307@behnel.de> Message-ID: >> >> While ?" *iter=5 " and " vect.at(0)=5 " ares valid C++ code. > > ... bug not valid Cython code. If this is to be supported at all, it needs > at least a different spelling. > > Stefan > I see what you mean. Although "valid cython code" should be a moving target. "new vector[int]()" used to be invalid cython code too. Furthermore, it might also affect the operator[] (but I can't test that due to bug#2). That would mean that "vect[3]=5" would also be refused by cython in spite of being valid cython code. I understand correct handling of c++ references will be difficult to solve, though (or maybe it is solved and has not been documented yet...). This "bug" may belong more to a wishlist than to a bug list. cheers, Toki From stefan_ml at behnel.de Fri Jul 2 18:12:15 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Jul 2010 18:12:15 +0200 Subject: [Cython] More c++ bugs In-Reply-To: References: <4C2DC1E3.2030307@behnel.de> Message-ID: <4C2E0FDF.4040509@behnel.de> toki doki, 02.07.2010 17:57: >>> While " *iter=5 " and " vect.at(0)=5 " ares valid C++ code. >> >> ... bug not valid Cython code. This was supposed to spell "but not valid Python code". >> If this is to be supported at all, it needs >> at least a different spelling. >> > I see what you mean. Although "valid cython code" should be a moving > target. Sort of, but rather towards less special syntax and more Python syntax. "vect.at(0)=5" doesn't fit anything in Python and someone who wants to make that part of the language will have a hard time getting through my veto. > "new vector[int]()" used to be invalid cython code too. Right, and although I don't really like reading it, it only works within Cython code that targets C++ (and not pure Python code), so at least it doesn't hurt Python compatibility. > Furthermore, it might also affect the operator[] (but I can't test > that due to bug#2). That would mean that "vect[3]=5" would also be > refused by cython in spite of being valid cython code. That is perfectly valid Python syntax, though. Overloading that to refer to the C++ [] operator when used on a C++ object sounds fine to me. > I understand correct handling of c++ references will be difficult to > solve Absolutely. Stefan From robertwb at gmail.com Fri Jul 2 20:39:50 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 2 Jul 2010 11:39:50 -0700 Subject: [Cython] Forums vs. Mailing Lists In-Reply-To: References: Message-ID: On Fri, Jul 2, 2010 at 7:05 AM, Luiji Maryo wrote: > I suggest that Cython switch from a mailing list system to a forum > system, as forums have a few very good advantages: > 1. ?The posts can be edited in the case of mistake. Think and proofread before you send. There's value to an immutable archive of what was said as well. > 2. ?The posts may be reported as spam. Same with mailing lists. > 3. ?It is much easier to have a long thread as supposed to individual > responses creating separate threads. If replying to a message creates a new thread, and you can't group messages by thread, you are using a seriously broken email reader. > 4. ?Giving out your e-mail address to everybody using the forum is not > a requirement. But it's very useful in case anyone wants to reply. > If mailing lists are in fact advantageous, please explain why. Email is much more versatile than forums, let alone whatever the latest forum-board-of-the-day web interface is. This is especially true for those of us that subscribe to many lists and do any kind of filtering. Also, if anyone wants a forum-like view, use http://groups.google.com/group/cython-users . In any case, no, we're not moving away from email. - Robert From robertwb at gmail.com Fri Jul 2 20:46:57 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 2 Jul 2010 11:46:57 -0700 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: <2D9406AA-20F9-437B-B7DD-DC6E11BEF34A@gmail.com> References: <4C2AEA6B.5080808@behnel.de> <4C2CCD91.6080101@molden.no> <2D9406AA-20F9-437B-B7DD-DC6E11BEF34A@gmail.com> Message-ID: On Fri, Jul 2, 2010 at 7:58 AM, Russell Reagan wrote: > I am curious, in what ways does Cython not meet your needs? Same here. Seems like for a game, though, much of the heaving lifting would be done by the underlying graphics modules (e.g. OpenGL on a GPU). > I think it > would be beneficial to know, in order to identify areas which Cython > needs to be improved in order to be more useful in real world projects. +1 > I have not done a great deal of work in Cython yet, but my perception > is that with Cython, you should be able to write in Python, modify the > Python with augmenting .pxd files, and get close to the equivalent > speed of C code. > > So I am curious if: > > 1. My theoretical view of Cython is flawed. Or... Your understanding is right on. > 2. Practice currently has not caught up to theory. Or... There's still some work to do, but it's already the case for most numerical things (of course we could always make thing easier) and many other usecases as well. (The speed advantages of Cython for non-numerical things are not usually in the 100-1000x, mostly reflective of the fact that the speed disadvantages of Python vs. C are not as extreem in this case.) > 3. Cython is not intended to address your specific situation, and I > have misunderstood your intentions and goals. Cython is used to make specific pieces of code fast, not speed up the entire interpreter (though one could presumably compile anything one is interested in). - Robert From dalcinl at gmail.com Fri Jul 2 20:58:43 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Jul 2010 15:58:43 -0300 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2DF1ED.6010107@behnel.de> References: <4C2DF1ED.6010107@behnel.de> Message-ID: On 2 July 2010 11:04, Stefan Behnel wrote: > Lisandro Dalcin, 01.07.2010 20:52: >> In Python 2.7, creating a PyCObject is marked with >> PendingDeprecationWarning... Should we special case 2.7 and use >> PyCapsule ? > > Good question. I think we should not change anything for 2.6, even if > PyCapsule is available there, but given the warning, I think it makes sense > to switch for 2.7, yes. > http://hg.cython.org/cython-devel/rev/721769394428 -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at gmail.com Fri Jul 2 20:53:00 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 2 Jul 2010 11:53:00 -0700 Subject: [Cython] Some c++ bugs In-Reply-To: References: <4C2C7920.2070308@behnel.de> Message-ID: On Fri, Jul 2, 2010 at 12:42 AM, toki doki wrote: >> >> Hi, >> >> since you seem to be interested in getting these implemented, why not help >> us by providing a test case for these features? See the C++ tests in the >> tests/ directory, as well as the docs: >> >> http://wiki.cython.org/HackerGuide >> >> Also, please open trac tickets for these two features. >> >> Thanks! > > Actually, Haoyu Bai already solved my problems (see the other mail). > > There would still be some small changes needed to the documentation. Yep. You should be able to just edit the wiki, and feel free to pull from http://hg.cython.org/cython-docs (and send us patches/host a personal repo we can pull from) for documentation fixes. > And maybe a "wishlist bug report" for a ctypedef really working with > templated types (instead of ?Haoyu Bai workaround). > > I will ask for an access to Trac and the doc wiki. For this and maybe > future bugs ?I encounter. The procedure is to just send a login name > and password to Robert Bradshaw, right? Exactly. Or an htpasswd entry will do if you don't want to reveal your password (though I wouldn't advise using a valuable one--the reason we have any kind of access restriction is just to combat spammers). - Robert From sturla at molden.no Fri Jul 2 21:11:57 2010 From: sturla at molden.no (Sturla Molden) Date: Fri, 02 Jul 2010 21:11:57 +0200 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: References: <4C2AEA6B.5080808@behnel.de> <4C2CCD91.6080101@molden.no> Message-ID: <4C2E39FD.9010106@molden.no> Luiji Maryo skrev: > As for project background, this has to do with an argument with > colleges of mine. We wanted to make a game, and I brought up the > greatness of Python, but they said that it would be too slow for game > development. Even though it is sufficiently fast, the game we were > working on was very graphics intensive and when comparing a C version > to a Python version of basic design philosophy C was noticably faster. > > Several games use Python for game logics, including Battlefield 2 and Civilization IV. But you would obviously not write your graphics engine in Python (OpenCL might change that in the future). There are good 3D engines callable from Python though, such as PyOgre and Disney's Panda3d (which they have used for several games, e.g. Pirates of the Caribbean). You should also take a look at PyGame. It is a pythonic wrapper around SDL. It allows you to write cross-platform game and multimedia apps in Python. SDL/PyGame uses DirectX (sound, input, 2D) and OpenGL (3D) on Windows. I am working on a small program that use OpenGL for visualizing large amounts of scientific data. I have written some of the code that calls OpenGL in C, and simply call that from Python. In my experience, if I have a loop that calls glVertex* many times, it makes sence to write that loop in C. Iterating in Python and making thousands of calls to glVertex* (using ctypes or PyOpenGL) obviously gives me slow graphics. But the complexity between writing these loops in C (or Cython) and Python is really minue, if any. It is all the other stuff that matters. There are other tricks that work as well, and allows fast OpenGL graphics in Python without using C: - Display lists. They are slower to make, but execute equally fast. - Vertex arrays with NumPy arrays as buffer. - Vertex buffers in video RAM. - Python is fantastic for working with vertex shaders and OpenCL. Vertex arrays usng NumPy (and to a lesser extent display lists) can be a drop-in replacement for multiple calls to glVertex*. For graphics using vertex shaders and OpenCL, Python is actually better than C or C++. It might seems surprising, until you realize it is just text manipulation on your part. We can manipulate text faster and more easily in Python than C++. Thus, Python is a better choise for generating vertex shaders and OpenCL code, and the OpenGL/OpenCL driver does the rest. In my experience, tight loops with calls to glVertex* is the only place where Python can be proven "too slow". A graphics intensive program will need to store data contiguous buffers. I prefer NumPy, but you can also use the array module from Python's standard library (or tuples, but not lists). Regards, Sturla From robertwb at gmail.com Sat Jul 3 03:42:52 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 2 Jul 2010 18:42:52 -0700 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: References: <4C2DF1ED.6010107@behnel.de> Message-ID: On Fri, Jul 2, 2010 at 11:58 AM, Lisandro Dalcin wrote: > On 2 July 2010 11:04, Stefan Behnel wrote: >> Lisandro Dalcin, 01.07.2010 20:52: >>> In Python 2.7, creating a PyCObject is marked with >>> PendingDeprecationWarning... Should we special case 2.7 and use >>> PyCapsule ? >> >> Good question. I think we should not change anything for 2.6, even if >> PyCapsule is available there, but given the warning, I think it makes sense >> to switch for 2.7, yes. Fine by me. Less spurious warnings is always a good thing. > > http://hg.cython.org/cython-devel/rev/721769394428 Thanks. - Robert From greg.ewing at canterbury.ac.nz Sat Jul 3 04:23:40 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Jul 2010 14:23:40 +1200 Subject: [Cython] Forums vs. Mailing Lists In-Reply-To: References: Message-ID: <4C2E9F2C.1040107@canterbury.ac.nz> Luiji Maryo wrote: > I suggest that Cython switch from a mailing list system to a forum > system Please, no. If you do that, I won't be involved in any Cython discussions any more, because I don't follow web forums as a rule, only newsgroups and mailing lists. > If mailing lists are in fact advantageous, please explain why. Simple, fast, uniform user interface, with content that comes to me instead of me having to chase around multiple web sites to keep up with the things I'm interested in. -- Greg From luiji at users.sourceforge.net Sat Jul 3 06:05:16 2010 From: luiji at users.sourceforge.net (Luiji Maryo) Date: Sat, 3 Jul 2010 00:05:16 -0400 Subject: [Cython] Forums vs. Mailing Lists In-Reply-To: <4C2E9F2C.1040107@canterbury.ac.nz> References: <4C2E9F2C.1040107@canterbury.ac.nz> Message-ID: So I do see the advantages of a mail system, and Google Groups cought my attention. Along with that, I remembered how big the mailing list is and how strenuous conversion would be. Sorry, I should have thought more before posting. On Friday, July 2, 2010, Greg Ewing wrote: > Luiji Maryo wrote: >> I suggest that Cython switch from a mailing list system to a forum >> system > > Please, no. If you do that, I won't be involved in any > Cython discussions any more, because I don't follow > web forums as a rule, only newsgroups and mailing lists. > >> If mailing lists are in fact advantageous, please explain why. > > Simple, fast, uniform user interface, with content that > comes to me instead of me having to chase around multiple > web sites to keep up with the things I'm interested in. > > -- > Greg > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- - Luiji Maryo (a.k.a. Brain Boy) Visit me at http://brainboyblogger.blogspot.com/. From luiji at users.sourceforge.net Sat Jul 3 06:07:10 2010 From: luiji at users.sourceforge.net (Luiji Maryo) Date: Sat, 3 Jul 2010 00:07:10 -0400 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: <2D9406AA-20F9-437B-B7DD-DC6E11BEF34A@gmail.com> References: <4C2AEA6B.5080808@behnel.de> <4C2CCD91.6080101@molden.no> <2D9406AA-20F9-437B-B7DD-DC6E11BEF34A@gmail.com> Message-ID: I never said Cython doesn't meet my needs, I said it met my needs so much that I wanted to optimized the PSL with it. On Friday, July 2, 2010, Russell Reagan wrote: > I am curious, in what ways does Cython not meet your needs? I think it > would be beneficial to know, in order to identify areas which Cython > needs to be improved in order to be more useful in real world projects. > > I have not done a great deal of work in Cython yet, but my perception > is that with Cython, you should be able to write in Python, modify the > Python with augmenting .pxd files, and get close to the equivalent > speed of C code. > > So I am curious if: > > 1. My theoretical view of Cython is flawed. Or... > > 2. Practice currently has not caught up to theory. Or... > > 3. Cython is not intended to address your specific situation, and I > have misunderstood your intentions and goals. > > On Jul 2, 2010, at 8:57 AM, Luiji Maryo > wrote: > >> Responses to Stefan Behnel: >> >> I do know the goal is ambitious, but less then you probably think. >> Let me clarify and say that I am only optimizing modules that will >> benefit, such as the mathematical libraries. ?Libraries such as Tk can >> be left alone, since they are simply wrappers for the Tk native >> library (IIRC). >> >> As for project background, this has to do with an argument with >> colleges of mine. ?We wanted to make a game, and I brought up the >> greatness of Python, but they said that it would be too slow for game >> development. ?Even though it is sufficiently fast, the game we were >> working on was very graphics intensive and when comparing a C version >> to a Python version of basic design philosophy C was noticably faster. >> >> However, I brought up the fact that Python is much quicker to code and >> outputs smaller files, and is much more cross-platformable. ?They >> argued that development speed could be sacraficed for game speed, and >> in the end I suggested to solve the issue by making a faster Python. >> They were quickly discouraged, and desided to pause the project for >> awhile so that I can get Python closer to C speed while they work on >> other games. >> >> Unladen Swallow seems to have the same project goal as me: to make a >> noticably faster Python implementation. ?I will look more into that >> project. ?Thanks for the information! >> >> Responses to Sturla Molden: >> >> Yes, CPython does seem to avoid very great concepts such as >> register-based memory management, garbage collection over reference >> counting, and JIT, which is why I decided as part of the optimized >> Python to use Pysco, Unladen Swallow, or another optimized Python >> interpreter. >> >> Thank you for your comments (and for adding the title that I >> moronically forgot), and I will start doing some optimizations. ?Any >> further development on this will be found at http://luiji.github.com/, >> but for now you probably will not see much development. ?If anybody is >> interested in the project, then you can subscribe to my GitHub account >> to a notification when I create a new repository for the optimized >> Python Standard Library (PSL as I will call it in further e-mails, >> since stdlib sounds too much like the C Stdlib). >> >> Thanks, >> - Luiji Maryo >> >> On Thu, Jul 1, 2010 at 1:17 PM, Sturla Molden >> wrote: >>> Stefan Behnel skrev: >>>> Good luck. I don't mean to discourage you, but this is a pretty >>>> ambitious goal. >>>> >>> >>> Python's VM is slow, indeed. It uses a stack-based VM instead of >>> registers; it is implemented in the worst possible way, as code and >>> data >>> form a three of hash tables; it has reference counting instead of >>> garbage collection (produces high cache traffic); it has no JIT >>> compilation. It's doomed to be slow. >>> >>> Many implementations of Common Lisp and Scheme can compete with ?C >>> (e.g. >>> SBCL, CMUCL, Allegro, Clozure, Ikarus). None of the billion dollar >>> investment that was made into making Lisp fast (during the AI hype >>> 20-30 >>> years ago) has made it into Python. There were even hardware >>> processors >>> (aka 'Lisp machines') that could run Lisp natively. We can sometimes >>> read on Python lists that Lisp's performance comes from optional >>> static >>> typic. That is not the whole story. For example, enourmous efforsts >>> were >>> made for making memory mangement efficient (cons is very effcient, >>> lists >>> are transformed to contiguous arrays internally, tail-recursion is >>> inlined, etc). Lot of this technology is now forgotten, as those who >>> worked on it are retired or has lost interest. >>> >>> Python's memory management (reference counts and malloc/free) is the >>> least efficient ever invented by man. Nothing is stored >>> contiguously (a >>> list of int is not a buffer of int, but an array of pointers). And >>> what >>> data traffic does it take to keep refcounts synchronized in the >>> cache of >>> multiple processors? It's almost a wonder that the computer does not >>> burst into flames. >>> > -- - Luiji Maryo (a.k.a. Brain Boy) Visit me at http://brainboyblogger.blogspot.com/. From greg.ewing at canterbury.ac.nz Sat Jul 3 04:02:05 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Jul 2010 14:02:05 +1200 Subject: [Cython] Py 2.7 deprecates PyCObject In-Reply-To: <4C2D6500.8000206@molden.no> References: <4C2D6500.8000206@molden.no> Message-ID: <4C2E9A1D.5070409@canterbury.ac.nz> Sturla Molden wrote: > All programs that use PyCObject become a security problem. It's not quite true to say "all" here, because it's possible to use the descr slot of a PyCObject to hold type information. However, that's not the purpose that the docs advertise it as being for, and there's probably a lot of PyCObject-using code around that doesn't use it that way. So replacing PyCObject with something more explicitly safe seems like a good idea. -- Greg From dagss at student.matnat.uio.no Sat Jul 3 09:40:00 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 03 Jul 2010 09:40:00 +0200 Subject: [Cython] More c++ bugs Message-ID: <3360994823.392079@smtp.netcom.no> (Sorry for top-posting, on phone.) NOTE that this problem is equally present on function arguments: In Python, f(x) doesn't ever change the value of x! While in C++, f could take a reference. I'd like Cython to not follow C++ here. There's been long threads about this in the past. My suggestion was to treat references as pointers in Cython (they are, after all, C++ syntax csandy for pointers):: f(&x) # output as f(*(&x)) in C++ vec.at(idx)[0] = 0 I still think this is the lesser of all evils. I suppose a special case should be made for operator[] (C++ only has a getitem operator while Python distinguish getitem/setitem, so some kind of special rule is needed here anyway). Dag Sverre Seljebotn -----Original Message----- From: Stefan Behnel Date: Friday, Jul 2, 2010 6:12 pm Subject: Re: [Cython] More c++ bugs To: toki doki CC: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net toki doki, 02.07.2010 17:57: >>> While " *iter=5 " and " vect.at(0)=5 " ares valid C++ code. >> >> ... bug not valid Cython code. > >This was supposed to spell "but not valid Python code". > > >>> If this is to be supported at all, it needs >> at least a different spelling. >> > I see what you mean. Although "valid cython code" should be a moving > target. > >Sort of, but rather towards less special syntax and more Python syntax. >"vect.at(0)=5" doesn't fit anything in Python and someone who wants to make >that part of the language will have a hard time getting through my veto. > > >> "new vector[int]()" used to be invalid cython code too. > >Right, and although I don't really like reading it, it only works within >Cython code that targets C++ (and not pure Python code), so at least it >doesn't hurt Python compatibility. > > >> Furthermore, it might also affect the operator[] (but I can't test > that due to bug#2). That would mean that "vect[3]=5" would also be > refused by cython in spite of being valid cython code. > >That is perfectly valid Python syntax, though. Overloading that to refer to >the C++ [] operator when used on a C++ object sounds fine to me. > > >> I understand correct handling of c++ references will be difficult to > solve > >Absolutely. > >Stefan >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Sat Jul 3 10:10:17 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 03 Jul 2010 10:10:17 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <3360994823.392079@smtp.netcom.no> References: <3360994823.392079@smtp.netcom.no> Message-ID: <4C2EF069.9070109@behnel.de> Dag Sverre Seljebotn, 03.07.2010 09:40: > From: Stefan Behnel >> toki doki, 02.07.2010 17:57: >>> I understand correct handling of c++ references will be difficult >>> to solve >> >> Absolutely. > > NOTE that this problem is equally present on function arguments: In > Python, > > f(x) > > doesn't ever change the value of x! While in C++, f could take a > reference. I'd like Cython to not follow C++ here. Same from here. > There's been long threads about this in the past. My suggestion was to > treat references as pointers in Cython (they are, after all, C++ syntax > csandy for pointers):: > > f(&x) # output as f(*(&x)) in C++ > vec.at(idx)[0] = 0 > > I still think this is the lesser of all evils. I suppose a special case > should be made for operator[] (C++ only has a getitem operator while > Python distinguish getitem/setitem, so some kind of special rule is > needed here anyway). This doesn't exactly make the code beautiful or well readable, but it does make the 'feature' usable without breaking any language constraints, and I think once you get used to it, it'll be clear enough. We shouldn't forget that this can only ever be used on explicitly (or implicitly but obvious) typed names anyway, so the information that is required to interpret the operation in the code will be somewhat close for the reader. Stefan From stefan_ml at behnel.de Sat Jul 3 18:25:07 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 03 Jul 2010 18:25:07 +0200 Subject: [Cython] website and hg repos down Message-ID: <4C2F6463.1080802@behnel.de> Hi, just a quick note that the website and the hg repos are currently down. This is being worked on. Stefan From sturla at molden.no Sat Jul 3 21:37:06 2010 From: sturla at molden.no (Sturla Molden) Date: Sat, 03 Jul 2010 21:37:06 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <3360994823.392079@smtp.netcom.no> References: <3360994823.392079@smtp.netcom.no> Message-ID: <4C2F9162.7030103@molden.no> Dag Sverre Seljebotn skrev: > f(x) > > doesn't ever change the value of x! Only if x is immutable. > While in C++, f could take a reference. In Python, x is always a reference. But there are immutable types. So while a Python int is immutable, a C or C++ int is mutable. If you want to enforce immutability even in C++, then make local copies before making the function call. Sturla From sturla at molden.no Sat Jul 3 22:01:00 2010 From: sturla at molden.no (Sturla Molden) Date: Sat, 03 Jul 2010 22:01:00 +0200 Subject: [Cython] More c++ bugs In-Reply-To: References: Message-ID: <4C2F96FC.7010903@molden.no> toki doki skrev: > > """"" > from libcpp.vector cimport vector > > cdef vector[int] *vect=new vector[int]() > vect.push_back(5) > vect.push_back(6) > print vect[0] > """"" > > gives the following error: > > """"" > print vect[0] > ^ > /home/toki/progs/cython13tests/bug1.pyx:6:10: Cannot convert > 'vector' to Python object > """"" > > That's not a bug. That is you not knowing C++. vect[0] dereferences a pointer in your code. What you get is the an object of type vector as Cython says, not an int contained in the vector. You want to use the [] operator on an C++ object, not a C++ pointer, which means: cdef vector[int] vect() # if Cython accept this vect.push_back(5) vect.push_back(6) print vect[0] Second, you should be flogged for calling new outside a class contructor. That will give you memory or resource leaks due to C++ exceptions skipping the call to delete. Therefore, new only belongs in constructors and delete only belongs in destructors. If you want dynamic allocation anywhere else, you use std::vector<>. An no, you never allocate a std::vector with new. STL objects are to be put on the stack. They allocate memory from the heap on their own. If you want to control how they do their allocation, you specify a custom allocator as the last template argument. And finally, when you use a std::vector as function argument, you pass references -- not pointers. That way operators like [] will continue to work as expected: void foobar(std::vector a); // ok but slow void foobar(std::vector& a); // ok void foobar(std::vector *a); // bad C++ has more gotchas than you can believe. It takes at least 10 years to master. C, C#, Python and Java are a lot simpler, even if it may not seem so superficially. Sturla From sturla at molden.no Sat Jul 3 23:06:28 2010 From: sturla at molden.no (Sturla Molden) Date: Sat, 03 Jul 2010 23:06:28 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <4C2F96FC.7010903@molden.no> References: <4C2F96FC.7010903@molden.no> Message-ID: <4C2FA654.1030507@molden.no> Sturla Molden skrev: > Second, you should be flogged for calling new outside a class > contructor. That will give you memory or resource leaks due to C++ > exceptions skipping the call to delete. This is by the way one of the biggest differences between C and C++. In C you can use and place malloc and free wherever you want, as no exceptions can mess up your pairing of malloc and free. When programmers "graduate" from C to C++, they think new and delete can be used as drop-in replacements for malloc and delete. Even most C++ textbooks will tell you this, only to demonstrate the author in incompetent. In C++, the drop in replacement for a pair of malloc and free is like this: { // int *a = (int*)malloc(n); std::vector a(n); // code between malloc and free goes here } // free(a); The curly brackets determine the life of the std::vector object, and has similar function to the placement of malloc and free in C. This is not entirely true though: In C, setjmp and longjmp can mess up pairing of free and malloc like C++ exceptions. Programs that use setjmp and longjmp (like Apace) must therefore use design techniques like "memory pools" to safeguard against this. This also affects use of other C resources in C++. They must only be allocated and deallocated in constructors and destructors. If you use cstdio instead of C++ iostream, it is only safe to put std::fopen in a constructor and std::fclose in a destructor. If you have C code like this: FILE *myfile = fopen(name); fclose(myfile); You must write a wrapper, #include class file { std::FILE* fid; file(const char *name) {fid = std::fopen(name);}; ~file() {if (fid != NULL) std::fclose(fid);}; }; And then you have this translation from C to C++: { // FILE *myfile = fopen(name); file myfile(); } // fclose(myfile); I know this is the Cython list, but it applies when we use Cython to generate C++ or interface with C++ as well. C resource allocation in constructors and destructors also applies to Cython. We must use __cinit__ and __dealloc__ just like C++ constructors and destructors. Again, this is due to exceptions producing resource leaks if we don't do this. So for example when we use Cython to interface with Windows API, calls to CloseHandle() always belongs in a __dealloc__ method. Any other placement means it can be skipped by an exception, and we have a resource leak. fopen and malloc are only safe in __cinit__, and free and fclose are obly safe in __dealloc__. C is very different from Cython and C++ because C has no exceptions (ignoring setjmp and longjmp -- don't even consider using those with Cython). How many users of C++ or Cython knows or even understands this? I guess very few, and that is why C++ generates so many software bugs. In Cython, we should be aware that we have the same problem, due to direct access to C. So Cython can be a bug generator like C++. Java and Python do not have this issue, as there are no direct access to C. (Well actually, Python using ctypes has this issue too, but that is a special case.) Therefore I do not expect average Python or Java developers to know this as well. I think a warning in the docs is in place. Better still, known cases of C resource allocation (stdio.fopen or stdlib.malloc) should be disallowed outside __cinit__. Ditto for known cases of C resource deallocation and __dealloc__. Code that don't match allocation and deallocation should also we disallowed. There should be some sort of tag to use in the declaration to tell the Cython compiler this. This is just a suggestion: cres void *malloc(int), void free(void*) # cres for "C resource" cres void *fopen(FILE*), fclose(FILE*) cres HANDLE CreateEvent(LPSECURITY_ATTRIBUTES, BOOL, BOOL, LPCTSTR), BOOL CloseHandle(HANDLE) Sturla From dagss at student.matnat.uio.no Sun Jul 4 00:59:00 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 04 Jul 2010 00:59:00 +0200 Subject: [Cython] More c++ bugs Message-ID: <3361049998.505334@smtp.netcom.no> Sturla Molden skrev: Dag Sverre Seljebotn skrev: > f(x) > >> doesn't ever change the value of x! >Only if x is immutable. This is just wrong -- there's a fundamental difference of changing which reference is stored in a variable in the calling scope, and modifying objects. You can change x, but you can't reassing x. Perhaps my wording could have been clearer. >> While in C++, f could take a reference. >In Python, x is always a reference. But there are immutable types. So >while a Python int is immutable, a C or C++ int is mutable. > >If you want to enforce immutability even in C++, then make local copies >before making the function call. Yes, of course... Dag From sturla at molden.no Sun Jul 4 02:00:26 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 04 Jul 2010 02:00:26 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <3361049998.505334@smtp.netcom.no> References: <3361049998.505334@smtp.netcom.no> Message-ID: <4C2FCF1A.2060806@molden.no> Dag Sverre Seljebotn skrev: > This is just wrong -- there's a fundamental difference of changing which reference is stored in a variable in the calling scope, and modifying objects. You can change x, but you can't reassing x. > > Perhaps my wording could have been clearer. > C++ references per definition cannot be changed, as they operate like const pointers. You can only modify the value of the variable they alias. References are logical aliases of other variables, which must be initialized to alias to an object when declared. The value of a reference can never change. int& a; // illegal int b; int& a = b; // ok This means the same as: int *const _a = &b; #define a _a[0] So if you write int c; a = c; this change the value of b to that of c. It does not rebind a to alias c! This is a big difference between Python and C++ sematics. Assignment rebinds names in Python; assignment copies values (or whatever you overload) in C++. We have the same sematical difference when calling functions: void foobar(int& c); the reference c is intialized to alias the argument with which foobar is called. But you can never rebind c to alias another object in the local scope of foobar. If you assign to c you modify the variable aliased by c. Sturla Molden From sturla at molden.no Sun Jul 4 02:34:54 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 04 Jul 2010 02:34:54 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <4C2FCF1A.2060806@molden.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> Message-ID: <4C2FD72E.6090100@molden.no> Sturla Molden skrev: > We have the same sematical difference when calling functions: > > void foobar(int& c); > > the reference c is intialized to alias the argument with which foobar is > called. But you can never rebind c to alias another object in the local > scope of foobar. If you assign to c you modify the variable aliased by c. > And as for Python, if we have c = 5 foobar(c) the value of c cannot be changed as c is immutable. But if we have from ctypes import c_int c = c_int(5) foobar(c) then the call to foobar can change the value of c by assigning to c.value in its local scope. In either case, if foobar assigns to c in its local scope (i.e. rebinds the name), nothing happens to c in the calling scope. This is obviously very different from C++, but in neither case can foobar rebind the argument in the calling scope. We should not confuse the issues of name binding vs. assignment and mutability vs. immutability. It's not the same. But Python and C++ differs in both cases. Only in pedantic functional languages are you a guaranteed that foobar(x) does not have side effects on x. I am not sure why you are worried about side effects on x from a function call foobar(x) in C++. Both languages can produce side effects on x here. But C++ does not have immutable types. Sturla From tokidokix at gmail.com Sun Jul 4 19:31:28 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 02:31:28 +0900 Subject: [Cython] More c++ bugs In-Reply-To: <4C2F96FC.7010903@molden.no> References: <4C2F96FC.7010903@molden.no> Message-ID: > That's not a bug. That is you not knowing C++. ?vect[0] dereferences a > pointer in your code. What you get is the an object of type vector > as Cython says, not an int contained in the vector. Ooooppss. Shame on me! This is indeed a mistake from me, not a bug. I should have re-read the code more carefully. I think I was slightly disturbed here by the fact that cython use the dot operator for both the "." and "->" of C. Seeing "vect.push_back(6)" above "print vect[0]" made me forget that vect was not a vector but a pointer. Replacing vect[0] by vect[0][0], everything is OK. Sorry, sorry for that and thanks for noticing. As for me not understanding c++, however, please note that: 1- this was meant as a minimal bug code stripped out of any context 2- cython doesn't allow (AFAIU) stack-allocated c++ object as members of a python extension class. Therefore, if one wants to create a Python class wrapping a vector, I think one has to write things like "self.pVect=new vector" (and of course add the appropriate "del self.pVect" in the "__dealloc__" method of the class. Of course, I am still trying to understand how the c++ mode of cython works, therefore I might be wrong. From sturla at molden.no Sun Jul 4 21:39:36 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 04 Jul 2010 21:39:36 +0200 Subject: [Cython] More c++ bugs In-Reply-To: References: <4C2F96FC.7010903@molden.no> Message-ID: <4C30E378.4040506@molden.no> toki doki skrev: > As for me not understanding c++, however, please note that: > 1- this was meant as a minimal bug code stripped out of any context > 2- cython doesn't allow (AFAIU) stack-allocated c++ object as members > of a python extension class. Therefore, if one wants to create a > Python class wrapping a vector, I think one has to write things like > "self.pVect=new vector" (and of course add the appropriate "del > self.pVect" in the "__dealloc__" method of the class. Of course, I am > still trying to understand how the c++ mode of cython works, therefore > I might be wrong. > Out of curiosity, what happens if allocation fails, and C++ raises std::bad_alloc? In C++ if you have used new and delete outside constructors, it might give you memory leaks. In Cython I suppose the C++ exception goes uncaught, so the program just crashes. I would use Python lists or NumPy arrays instead. Sturla From tokidokix at gmail.com Mon Jul 5 03:01:14 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 10:01:14 +0900 Subject: [Cython] More c++ bugs In-Reply-To: <4C30E378.4040506@molden.no> References: <4C2F96FC.7010903@molden.no> <4C30E378.4040506@molden.no> Message-ID: > Out of curiosity, what happens if allocation fails, and C++ raises > std::bad_alloc? In C++ if you have used new and delete outside > constructors, it might give you memory leaks. In Cython I suppose the > C++ exception goes uncaught, so the program just crashes. I would use > Python lists or NumPy arrays instead. Python Lists are way too inefficient (memory-wise and computation-wise) for my use case. NumPy array could have been an option. But using a std::vector wrapped in a python extension allows me to easily re-use a lot of code pre-written for c++. It's probably possible to adapt a STL interface to a NumPy array structure to achieve the same effect, but I haven't looked into that yet. However, some of my uses involve vectors of vectors of different size, and vectors of complex objects, for which Numpy arrays are not a very good replacements. As for catching std::bad_alloc: this is mostly research code, so I have not been trying too hard to bulletproof my code against all possible exceptions for now. However, wouldn't a try/catch block be enough to handle this? Also, I believe there is a mechanism in cython0.13 to automatically convert c++ exceptions to python ones (by declaring "except +"). From tokidokix at gmail.com Mon Jul 5 08:13:07 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 15:13:07 +0900 Subject: [Cython] C++: Tickets #549, #550, #551 and more Message-ID: I just created tickets for some of the bugs I reported (without the one that was a mistake from me, as pointed by Sturla Molden, of course). I did not report previously Bug #550; see the track ticket for details. Ticket #549 (http://trac.cython.org/cython_trac/ticket/549) : compiler crash with non-declared template argument Test case: crash_non_declared_template_argument_T549.pyx Ticket #550 (http://trac.cython.org/cython_trac/ticket/550) : Cython can generate incorrect c++ code when a function uses a reference argument Test case: incorrect_code_with_reference_argument_T550.pyx Ticket #551 (http://trac.cython.org/cython_trac/ticket/551) : Cython does not allow ctypedef'ing templated types Test case: ctypedef_of_templated_type_T551.pyx I attach a diff adding the test cases to tests/run and modifying tests/bugs.txt accordingly. I did not submit an enhancement ticket for the "left-value assignment problems" yet, because I wanted to make some things clear first. To recap, there are actually two problems, I think: 1- If foo_iterator is an iterator, operator.dereference(foo_iterator) is not an l-value for cython. Therefore it is not possible to do the equivalent of " *foo_iterator=bar " in cython (except if foo_iterator is a pointer and not a general iterator) 2- functions returning references are not l-values for cython. Therefore, for example, it is not possible to use the ".at" method of std::vectors to assign a value. It looks like 1 and 2 could be solved by having a cython operator like operator.set_lvalue(foo,bar) that would emit " foo=bar" provided foo is either a function returning a reference or an operator.dereference call or a standard l-value. Therefore, just in case, I wanted to ask is such an operator had been already coded but not yet documented (there seems to be hardly any doc about the operators in cython.operator, actually). Secondly, it seems that (1) is both a bit more annoying and a bit easier (AFAICT without knowing cython internals) to solve than (2). Therefore, would it be OK to make two different tickets, in the hope that (1) can be solved earlier? Maybe a possible solution is to overload operator.dereference so that it can take a second argument? Thus operator.dereference(foo_iterator ,bar) would produce the c++ code: " *foo_iterator=bar ". A related "missing feature" is that it is not possible to take the address of a dereferenced iterator: " &operator.dereference(foo_iterator) " is invalid. cheers, Toki -------------- next part -------------- A non-text attachment was scrubbed... Name: testscases_549_550_551.diff Type: text/x-patch Size: 1717 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100705/229d3693/attachment.bin From tokidokix at gmail.com Mon Jul 5 08:36:01 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 15:36:01 +0900 Subject: [Cython] About the cython/C++ doc Message-ID: Right now, there are two files containing information about the c++ mode of cython. One in the user guide (http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html), and another one in the wiki (http://wiki.cython.org/WrappingCPlusPlus). Note that googling for "cython c++" brings you to the wiki page. I think that, historically, the wiki file contained the tips and tricks to write c++ code with cython/pyrex versions <=0.12. It was then modified to include infos about the new c++ mode of cython0.13 . The userguide file seems to have started as a copy of the wiki file but has been edited separately, so that the two files are out of sync. For example, the user guide file miss infos about nested classes syntax and cython.operator.dereference. The wiki file show an example of nested class declaration, but with the wrong syntax. Considering that : (1) - It seems a lot of duplicate effort to maintain two doc files about the same features. (2) - The old version of the wiki file would still be useful for people who can't upgrade to cython0.13, or have to maintain older code. , I propose to do the following: 1- Reverse http://wiki.cython.org/WrappingCPlusPlus to the version for cython<=0.12 2- Maybe rename it to WrappingCPlusPlus_ForCython012AndLower 3- Add a big warning on top of WrappingCPlusPlus_ForCython012AndLower saying that "This information is deprecated. Starting with version 0.13, cython has full native support for c++. See section xxx of the user guide for details" 4- Add all the missing information that was contained in the wiki file to the userguide file. This include (at least): nested class declaration, cython.operator.dereference, cython.operator.preincrement, (any other cython.operator I should know? I suppose there is at least a operator.postincrement ...) What do you think? From tokidokix at gmail.com Mon Jul 5 08:37:59 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 15:37:59 +0900 Subject: [Cython] More c++ bugs In-Reply-To: References: <4C2F96FC.7010903@molden.no> <4C30E378.4040506@molden.no> Message-ID: > However, wouldn't a try/catch block be > enough to handle this? On the other hand, it is true there might not be a way to include c++ try/catch blocks in cython code... From stefan_ml at behnel.de Mon Jul 5 08:52:26 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 08:52:26 +0200 Subject: [Cython] About the cython/C++ doc In-Reply-To: References: Message-ID: <4C31812A.3010701@behnel.de> toki doki, 05.07.2010 08:36: > I propose to do the following: > 1- Reverse http://wiki.cython.org/WrappingCPlusPlus to the version for > cython<=0.12 > 2- Maybe rename it to WrappingCPlusPlus_ForCython012AndLower > 3- Add a big warning on top of WrappingCPlusPlus_ForCython012AndLower > saying that "This information is deprecated. Starting with version > 0.13, cython has full native support for c++. See section xxx of the > user guide for details" > 4- Add all the missing information that was contained in the wiki file > to the userguide file. This include (at least): nested class > declaration, cython.operator.dereference, > cython.operator.preincrement, (any other cython.operator I should > know? I suppose there is at least a operator.postincrement ...) Are you volunteering? Stefan From tokidokix at gmail.com Mon Jul 5 08:56:23 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 15:56:23 +0900 Subject: [Cython] About the cython/C++ doc In-Reply-To: <4C31812A.3010701@behnel.de> References: <4C31812A.3010701@behnel.de> Message-ID: > Are you volunteering? > > Stefan > yes :-) But first tell me if you are OK with the proposition. From stefan_ml at behnel.de Mon Jul 5 09:12:08 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 09:12:08 +0200 Subject: [Cython] About the cython/C++ doc In-Reply-To: References: Message-ID: <4C3185C8.7090800@behnel.de> toki doki, 05.07.2010 08:36: > Considering that : > (1) - It seems a lot of duplicate effort to maintain two doc files > about the same features. Actually, there was some effort to get them merged already, but it turned out to be more than what was open for investment at the time. > (2) - The old version of the wiki file would still be useful for > people who can't upgrade to cython0.13, or have to maintain older > code. Sure. Most of that is equivalent to what Pyrex provides, but I'm fine with keeping that Wiki page alive (and somewhat hidden). > I propose to do the following: > 1- Reverse http://wiki.cython.org/WrappingCPlusPlus to the version for > cython<=0.12 > 2- Maybe rename it to WrappingCPlusPlus_ForCython012AndLower > 3- Add a big warning on top of WrappingCPlusPlus_ForCython012AndLower > saying that "This information is deprecated. Starting with version > 0.13, cython has full native support for c++. See section xxx of the > user guide for details" +1 > 4- Add all the missing information that was contained in the wiki file > to the userguide file. This include (at least): nested class > declaration, cython.operator.dereference, > cython.operator.preincrement, (any other cython.operator I should > know? I suppose there is at least a operator.postincrement ...) +1 There is a cython-docs project on the hg site that holds all the documentation. Please grab a copy and edit away as you see fit. Thanks! Stefan From tokidokix at gmail.com Mon Jul 5 09:26:45 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 16:26:45 +0900 Subject: [Cython] Struct-to-Dictionnary conversion Message-ID: After checking the mercurial repository, I noticed that in changeset 3358:47c1ba8093e3, David Barnett had already checked in a test case for the "ctypedef templated type" problem (but he had not created a track ticket). But I guess having two different test cases for the same problem is not a problem. However, this changeset contains another test case that surprised me because it concerns a feature I did not even know about. Apparently, cython can automatically convert C struct to python dictionnaries. Furthermore, it provides a "C++ constructor-like" constructor function for the struct, even in C-mode (and even in cython<0.13). Thus, the following code is valid. And, for example, foo(5,9) will return {first:5, second:9}. """""" ctypedef struct myStruct: int first int second def foo(int first,int second): cdef myStruct ms ms=myStruct(first,second) return ms """"""" This is nice, but appears a bit tricky to use. For example, the following alternate definition of foo """"""" def foo(int first,int second): return myStruct(first,second) """"""" brings the following error: """"" return myStruct(first,second) ^ /home/fabien/progs/cython13tests/bug7.pyx:9:19: Cannot interpret dict as type 'Python object' """"" Furthermore, the feature is broken when the language is set to C++. Then, the generated code contains a line like : """"" __pyx_t_4bug7_myStruct; """"" and gcc complains with: "error: declaration does not declare anything" Therefore, I want to know: Is this feature officially supported? Is it already in the doc? Should it be added to the doc?. If it is an official feature, then I should file a new ticket and test case (the one from David Barnett seems to concern a different problem), since it is broken in c++ mode. Toki From dagss at student.matnat.uio.no Mon Jul 5 09:33:39 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 05 Jul 2010 09:33:39 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <4C2FD72E.6090100@molden.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> Message-ID: <4C318AD3.8070204@student.matnat.uio.no> Sturla Molden wrote: > Sturla Molden skrev: > >> We have the same sematical difference when calling functions: >> >> void foobar(int& c); >> >> the reference c is intialized to alias the argument with which foobar is >> called. But you can never rebind c to alias another object in the local >> scope of foobar. If you assign to c you modify the variable aliased by c. >> Sorry, this could have been worded better -- references here refers to Python references = C pointers (more or less), not C++ references. > And as for Python, if we have > > c = 5 > foobar(c) > > the value of c cannot be changed as c is immutable. > > But if we have > > from ctypes import c_int > c = c_int(5) > foobar(c) > > then the call to foobar can change the value of c by assigning to > c.value in its local scope. > > In either case, if foobar assigns to c in its local scope (i.e. rebinds > the name), nothing happens to c in the calling scope. > > This is obviously very different from C++, but in neither case can > foobar rebind the argument in the calling scope. > > We should not confuse the issues of name binding vs. assignment and > mutability vs. immutability. It's not the same. But Python and C++ > differs in both cases. > Yes, it was you who brought mutability into the mix; I claimed it was irrelevant. Anyway...you keep stating things that we all know very well. Are you agreeing or disagreeing with my and Stefan's position? Is there a point to this discussion at all? Specifically: cdef int x = ... cdef MyCppClass *y = ... f(x, y) Do you want f to be allowed, if the arguments are declared as references, to change the value of x and which memory location y points to? There's no need to go into explaining what C++ does (I'm well aware of it), it's a simple yes or no question, and the question is simply "do we allow this feature of C++ into the Cython language, or do we take another approach instead that looks less strange with Python semantics?". (Anyway, I think the question is pretty much settled.) Dag Sverre From tokidokix at gmail.com Mon Jul 5 10:33:24 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 17:33:24 +0900 Subject: [Cython] C++: Tickets #549, #550, #551 and more In-Reply-To: References: Message-ID: > I did not submit an enhancement ticket for the "left-value assignment > problems" yet, because I wanted to make some things clear first. To > recap, there are actually two problems, I think: > 1- ?If foo_iterator is an iterator, operator.dereference(foo_iterator) > is not an l-value for cython. Therefore it is not possible to do the > equivalent of " *foo_iterator=bar " ?in cython (except if foo_iterator > is a pointer and not a general iterator) > 2- ?functions returning references are not l-values for cython. > Therefore, for example, it is not possible to use the ".at" method of > std::vectors to assign a value. I had not read the parallel thread between Stefan Behnel , Dag Sverre Seljebotn and Sturla Molden. If I understand correctly, they advocate that the return value of functions declared as returning a reference should be automatically converted to a pointer (to the returned referenced object). It makes sense. All things being equal, I would have prefered a "set_lvalue" operator, but I am not the one doing the implementation (nor do I understand the possible implementation difficulties). This solve however only a part of the problem. More precisely, if we want to apply the same idea to operator.dereference, we need to change its definition so that dereference(foo) produce the c++ code " &(*foo) ". Then, it is indeed possible to do all the necessary operations in cython: *foo ---> "dereference(foo)[0]" *foo=bar ---> "dereference(foo)[0]=bar" bar=&(*foo) ---> "bar=dereference(foo)" But this might lead to many errors by the users: when doing "bar=dereference(foo)", many people will expect that bar will be equal to *foo, and not &(*foo). Unfortunately, the problem is the same for functions returning references: if a user write: foo=vect.at(4)+10, he might be expecting to add 10 to the value at index 4 of vect, but in reality he will add 10 to the address of this value. Or is there something I do not understand? Toki From dagss at student.matnat.uio.no Mon Jul 5 10:43:31 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 05 Jul 2010 10:43:31 +0200 Subject: [Cython] C++: Tickets #549, #550, #551 and more In-Reply-To: References: Message-ID: <4C319B33.1060509@student.matnat.uio.no> toki doki wrote: >> I did not submit an enhancement ticket for the "left-value assignment >> problems" yet, because I wanted to make some things clear first. To >> recap, there are actually two problems, I think: >> 1- If foo_iterator is an iterator, operator.dereference(foo_iterator) >> is not an l-value for cython. Therefore it is not possible to do the >> equivalent of " *foo_iterator=bar " in cython (except if foo_iterator >> is a pointer and not a general iterator) >> 2- functions returning references are not l-values for cython. >> Therefore, for example, it is not possible to use the ".at" method of >> std::vectors to assign a value. >> > > > I had not read the parallel thread between Stefan Behnel , Dag Sverre > Seljebotn and Sturla Molden. If I understand correctly, they advocate > that the return value of functions declared as returning a reference > should be automatically converted to a pointer (to the returned > referenced object). It makes sense. All things being equal, I would > have prefered a "set_lvalue" operator, but I am not the one doing the > implementation (nor do I understand the possible implementation > difficulties). > Which you should have posted to, instead of bringing the topic into this other thread. > Unfortunately, the problem is the same for functions returning > references: if a user write: foo=vect.at(4)+10, he might be expecting > to add 10 to the value at index 4 of vect, but in reality he will add > 10 to the address of this value. > Yes, the proposal is to make you have to write foo = vect.at(4)[0] + 10 to get the equivalent of the C++ "foo = vect.at(4) + 10". I'll add another alternative to the other thread though. Dag Sverre From dagss at student.matnat.uio.no Mon Jul 5 10:56:50 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 05 Jul 2010 10:56:50 +0200 Subject: [Cython] C++ and references In-Reply-To: <4C318AD3.8070204@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> Message-ID: <4C319E52.1050301@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Sturla Molden wrote: > >> Sturla Molden skrev: >> >> >>> We have the same sematical difference when calling functions: >>> >>> void foobar(int& c); >>> >>> the reference c is intialized to alias the argument with which foobar is >>> called. But you can never rebind c to alias another object in the local >>> scope of foobar. If you assign to c you modify the variable aliased by c. >>> >>> > Sorry, this could have been worded better -- references here refers to > Python references = C pointers (more or less), not C++ references. > > >> And as for Python, if we have >> >> c = 5 >> foobar(c) >> >> the value of c cannot be changed as c is immutable. >> >> But if we have >> >> from ctypes import c_int >> c = c_int(5) >> foobar(c) >> >> then the call to foobar can change the value of c by assigning to >> c.value in its local scope. >> >> In either case, if foobar assigns to c in its local scope (i.e. rebinds >> the name), nothing happens to c in the calling scope. >> >> This is obviously very different from C++, but in neither case can >> foobar rebind the argument in the calling scope. >> >> We should not confuse the issues of name binding vs. assignment and >> mutability vs. immutability. It's not the same. But Python and C++ >> differs in both cases. >> >> > Yes, it was you who brought mutability into the mix; I claimed it was > irrelevant. > > Anyway...you keep stating things that we all know very well. Are you > agreeing or disagreeing with my and Stefan's position? Is there a point > to this discussion at all? > > Specifically: > > cdef int x = ... > cdef MyCppClass *y = ... > f(x, y) > > Do you want f to be allowed, if the arguments are declared as > references, to change the value of x and which memory location y points to? > > There's no need to go into explaining what C++ does (I'm well aware of > it), it's a simple yes or no question, and the question is simply "do we > allow this feature of C++ into the Cython language, or do we take > another approach instead that looks less strange with Python semantics?". > > (Anyway, I think the question is pretty much settled.) > Or perhaps not, the example toki doki posted gets ugly, as one needs to do foo = vect.at(4)[0] + 10 to get the equivalent of the C++ "foo = vect.at(4) + 10". So here's some additional proposals: I) What we could do is to add a rule that if you have cdef cppclass MyClass: int& at(int idx) Then Cython will automatically add an "assign_*" method: cdef MyClass *x = new MyClass(10) x.assign_at(0, 2) # Cython generates: x.at(0) = 2 print x.at(0) + 10 # at(0) fetches value I.e.: If the return value is a reference, create a psuedo-method named "assign_*" which takes the value to assign as the last argument. (If such a method is already declared manually, assume the psuedo-method isn't needed?) II) A common pattern is "void f(const MyStruct& foo)" (i.e. pass a pointer just to speed things up, but the contents won't be changed). In that case, note that it can be declared in Cython just as cdef void f(MyStruct foo) which should work well, Cython will just emit the right code. III) The case Sturla talked about I think boils down to this: cdef cppclass MyClass: void f(SomeStruct& foo) In this case, it's not entirely unnatural for Python code to do this myinstance.f(foostruct) and have the members of foostruct modified. It does contrast current Cython code, but not Python code, I think. But, it seems that one should be consistent with the next item: IV) There's still a problem with "void f(int& foo)" as well as "void f(SomeStruct*& foo), in which case we should probably demand myinstance(&foo) And then it seems prudent to require it in III as well. Dag Sverre From stefan_ml at behnel.de Mon Jul 5 11:15:12 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 11:15:12 +0200 Subject: [Cython] C++ and references In-Reply-To: <4C319E52.1050301@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C319E52.1050301@student.matnat.uio.no> Message-ID: <4C31A2A0.9040007@behnel.de> Dag Sverre Seljebotn, 05.07.2010 10:56: > the example toki doki posted gets ugly, as one needs to do > > foo = vect.at(4)[0] + 10 > > to get the equivalent of the C++ "foo = vect.at(4) + 10". What's so ugly about that? Stefan From dagss at student.matnat.uio.no Mon Jul 5 11:34:56 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 05 Jul 2010 11:34:56 +0200 Subject: [Cython] C++ and references In-Reply-To: <4C31A2A0.9040007@behnel.de> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C319E52.1050301@student.matnat.uio.no> <4C31A2A0.9040007@behnel.de> Message-ID: <4C31A740.5010701@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn, 05.07.2010 10:56: > >> the example toki doki posted gets ugly, as one needs to do >> >> foo = vect.at(4)[0] + 10 >> >> to get the equivalent of the C++ "foo = vect.at(4) + 10". >> > > What's so ugly about that? > Just that it looks so different from corresponding C++ code. But yes, it is something one would quickly learn, and just having a single rule might be easier to learn than lots of candy. Dag Sverre From tokidokix at gmail.com Mon Jul 5 12:13:34 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 19:13:34 +0900 Subject: [Cython] C++ and references In-Reply-To: <4C31A2A0.9040007@behnel.de> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C319E52.1050301@student.matnat.uio.no> <4C31A2A0.9040007@behnel.de> Message-ID: On Mon, Jul 5, 2010 at 6:15 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 05.07.2010 10:56: >> the example toki doki posted gets ugly, as one needs to do >> >> foo = vect.at(4)[0] + 10 >> >> to get the equivalent of the C++ "foo = vect.at(4) + 10". > > What's so ugly about that? > > Stefan > _______________________________________________ Well, I was just concerned that many users might unintentionnally write " foo=vect.at(4)+10 " when they meant " foo=vect.at(4)[0]+10 ". If foo is declared to be an unsigned int or a pointer, there might not be any warning from the compiler. And that could result in a bug very difficult to understand. An (ugly) example: """""""" cdef vector[void **] vect cdef void *foo # Assume that vect has been populated with meaningful pointers foo= vect.at(2)[3] # foo is equal to vect[5] instead of vect[2][3] """""""" In the above code, no compiler warnings will tell the user that he did something wrong. It will be very difficult for him to understand what is the problem. One could say that somebody using vectors of void** beg to be punished. Still, it is possible there exist more valid examples. Also, the issue of how to write " *foo=bar " when foo is a general iterator remains. (one cannot write " operator.dereference(foo)=bar "). Anyway, you are the one to do the implementation, so that's for you to decide :-) Toki From tokidokix at gmail.com Mon Jul 5 12:17:21 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 19:17:21 +0900 Subject: [Cython] C++: Tickets #549, #550, #551 and more In-Reply-To: <4C319B33.1060509@student.matnat.uio.no> References: <4C319B33.1060509@student.matnat.uio.no> Message-ID: >> > Which you should have posted to, instead of bringing the topic into this > other thread. Sorry. I did not really want to enter this debate actually. Here, I was just interested in the question of whether the issues of the assignment to dereference(foo) and the assignment to (e.g) vector.at(foo) should be solved simultaneously or independently. From dagss at student.matnat.uio.no Mon Jul 5 12:24:42 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 05 Jul 2010 12:24:42 +0200 Subject: [Cython] C++ and references In-Reply-To: References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C319E52.1050301@student.matnat.uio.no> <4C31A2A0.9040007@behnel.de> Message-ID: <4C31B2EA.9040104@student.matnat.uio.no> toki doki wrote: > On Mon, Jul 5, 2010 at 6:15 PM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn, 05.07.2010 10:56: >> >>> the example toki doki posted gets ugly, as one needs to do >>> >>> foo = vect.at(4)[0] + 10 >>> >>> to get the equivalent of the C++ "foo = vect.at(4) + 10". >>> >> What's so ugly about that? >> >> Stefan >> _______________________________________________ >> > > Well, I was just concerned that many users might unintentionnally > write " foo=vect.at(4)+10 " when they meant " foo=vect.at(4)[0]+10 ". > If foo is declared to be an unsigned int or a pointer, there might not > be any warning from the compiler. And that could result in a bug very > difficult to understand. > > An (ugly) example: > > """""""" > cdef vector[void **] vect > cdef void *foo > # Assume that vect has been populated with meaningful pointers > foo= vect.at(2)[3] # foo is equal to vect[5] instead of vect[2][3] > """""""" > In the above code, no compiler warnings will tell the user that he did > something wrong. It will be very difficult for him to understand what > is the problem. One could say that somebody using vectors of void** > beg to be punished. Still, it is possible there exist more valid > examples. > > Also, the issue of how to write " *foo=bar " when foo is a general > iterator remains. (one cannot write " operator.dereference(foo)=bar > "). > Argh, yes that is a real problem. The easiest way of assigning to items in a C++ list would be this? (&(cython.dereference(my_iterator)))[0] = value Hmmm... Dag Sverre From tokidokix at gmail.com Mon Jul 5 12:39:44 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 19:39:44 +0900 Subject: [Cython] About the cython/C++ doc In-Reply-To: <4C3185C8.7090800@behnel.de> References: <4C3185C8.7090800@behnel.de> Message-ID: I have started to update the user guide. And contrary to what I thought, the user guide page actually only contains info about the old method for writing c++ extensions. I was pretty sure I had seen a page mixing the old method with infos about the new C++ support earlier today. But it's very possible that I mistook a window for another and that at one point I was reading the wiki while thinking I was reading the userguide. Still, just in case, can you confirm me that the page cython-doc/src/userguide/wrapping_CPlusPlus.rst is the latest version that was produced? (you did mention there had been some earlier attempt at merging the wiki with the user guide... is the unfinished merge somewhere in the doc repository?) thanks, Toki From tokidokix at gmail.com Mon Jul 5 12:50:58 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 19:50:58 +0900 Subject: [Cython] C++ and references In-Reply-To: <4C31B2EA.9040104@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C319E52.1050301@student.matnat.uio.no> <4C31A2A0.9040007@behnel.de> <4C31B2EA.9040104@student.matnat.uio.no> Message-ID: On Mon, Jul 5, 2010 at 7:24 PM, Dag Sverre Seljebotn wrote: > toki doki wrote: >> On Mon, Jul 5, 2010 at 6:15 PM, Stefan Behnel wrote: >> >>> Dag Sverre Seljebotn, 05.07.2010 10:56: >>> >>>> the example toki doki posted gets ugly, as one needs to do >>>> >>>> foo = vect.at(4)[0] + 10 >>>> >>>> to get the equivalent of the C++ "foo = vect.at(4) + 10". >>>> >>> What's so ugly about that? >>> >>> Stefan >>> _______________________________________________ >>> >> >> Well, I was just concerned that many users might unintentionnally >> write " foo=vect.at(4)+10 " when they meant " foo=vect.at(4)[0]+10 ". >> If foo is declared to be an unsigned int or a pointer, there might not >> be any warning from the compiler. And that could result in a bug very >> difficult to understand. >> >> An (ugly) example: >> >> """""""" >> cdef vector[void **] vect >> cdef void *foo >> # Assume that vect has been populated with meaningful pointers >> foo= vect.at(2)[3] ? # foo is equal to ?vect[5] instead of vect[2][3] >> """""""" >> In the above code, no compiler warnings will tell the user that he did >> something wrong. It will be very difficult for him to understand what >> is the problem. One could say that somebody using vectors of void** >> beg to be punished. Still, it is possible there exist more valid >> examples. >> >> Also, the issue of how to write " *foo=bar " when foo is a general >> iterator remains. (one cannot write " operator.dereference(foo)=bar >> "). >> > Argh, yes that is a real problem. The easiest way of assigning to items > in a C++ list would be this? > > (&(cython.dereference(my_iterator)))[0] = value > > Hmmm... > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Actually, that won't even work. Current cython-devel refuses to take the address of a dereference call. It complains with a "Taking address of non-lvalue" error. Toki From stefan_ml at behnel.de Mon Jul 5 12:59:52 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 12:59:52 +0200 Subject: [Cython] About the cython/C++ doc In-Reply-To: References: <4C3185C8.7090800@behnel.de> Message-ID: <4C31BB28.2020105@behnel.de> toki doki, 05.07.2010 12:39: > I have started to update the user guide. And contrary to what I > thought, the user guide page actually only contains info about the old > method for writing c++ extensions. > > I was pretty sure I had seen a page mixing the old method with infos > about the new C++ support earlier today. But it's very possible that I > mistook a window for another and that at one point I was reading the > wiki while thinking I was reading the userguide. > > Still, just in case, can you confirm me that the page > cython-doc/src/userguide/wrapping_CPlusPlus.rst is the latest version > that was produced? (you did mention there had been some earlier > attempt at merging the wiki with the user guide... is the unfinished > merge somewhere in the doc repository?) I only know of these pages: http://wiki.cython.org/enhancements/cpp http://wiki.cython.org/gsoc09/daniloaf/progress http://wiki.cython.org/WrappingCPlusPlus http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html The last one is the result of the .rst file in cython-docs (likely an older version since the web site is for the current 0.12.1 release). Stefan From greg.ewing at canterbury.ac.nz Mon Jul 5 13:25:24 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 05 Jul 2010 23:25:24 +1200 Subject: [Cython] C++: Tickets #549, #550, #551 and more In-Reply-To: References: Message-ID: <4C31C124.1040204@canterbury.ac.nz> toki doki wrote: > we need to change > its definition so that dereference(foo) produce the c++ code " > &(*foo) ". Isn't &(*foo) always equivalent to just foo? -- Greg From dagss at student.matnat.uio.no Mon Jul 5 13:18:34 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 05 Jul 2010 13:18:34 +0200 Subject: [Cython] C++: Tickets #549, #550, #551 and more In-Reply-To: <4C31C124.1040204@canterbury.ac.nz> References: <4C31C124.1040204@canterbury.ac.nz> Message-ID: <4C31BF8A.8060600@student.matnat.uio.no> Greg Ewing wrote: > toki doki wrote: > >> we need to change >> its definition so that dereference(foo) produce the c++ code " >> &(*foo) ". >> > > Isn't &(*foo) always equivalent to just foo? > Not at all (in C++). * can invoke arbitrary code (operator*), which is used in the iterators in the standard C++ containers to denote accessing the value of the iterator (and an iterator is not a pointer). So &(*it) converts from std::list::iterator to an int*. Dag Sverre From stefan_ml at behnel.de Mon Jul 5 13:57:35 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 13:57:35 +0200 Subject: [Cython] C++ and references In-Reply-To: References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C319E52.1050301@student.matnat.uio.no> <4C31A2A0.9040007@behnel.de> Message-ID: <4C31C8AF.5020900@behnel.de> toki doki, 05.07.2010 12:13: > On Mon, Jul 5, 2010 at 6:15 PM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 05.07.2010 10:56: >>> the example toki doki posted gets ugly, as one needs to do >>> >>> foo = vect.at(4)[0] + 10 >>> >>> to get the equivalent of the C++ "foo = vect.at(4) + 10". >> >> What's so ugly about that? >> >> Stefan >> _______________________________________________ > > Well, I was just concerned that many users might unintentionnally > write " foo=vect.at(4)+10 " when they meant " foo=vect.at(4)[0]+10 ". > If foo is declared to be an unsigned int or a pointer, there might not > be any warning from the compiler. And that could result in a bug very > difficult to understand. No. You need a cast to convert a pointer to an int. So the above will fail if foo is declared as an int. It may not fail if foo is undeclared and inferred as a pointer, though, but in that case, there will likely be other things that fail later on. Stefan From stefan_ml at behnel.de Mon Jul 5 14:22:35 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 14:22:35 +0200 Subject: [Cython] C++: Tickets #549, #550, #551 and more In-Reply-To: <4C31BF8A.8060600@student.matnat.uio.no> References: <4C31C124.1040204@canterbury.ac.nz> <4C31BF8A.8060600@student.matnat.uio.no> Message-ID: <4C31CE8B.5040602@behnel.de> Dag Sverre Seljebotn, 05.07.2010 13:18: > Greg Ewing wrote: >> toki doki wrote: >> >>> we need to change >>> its definition so that dereference(foo) produce the c++ code " >>> &(*foo) ". >> >> Isn't&(*foo) always equivalent to just foo? >> > Not at all (in C++). * can invoke arbitrary code (operator*), which is > used in the iterators in the standard C++ containers to denote accessing > the value of the iterator (and an iterator is not a pointer). > > So&(*it) converts from std::list::iterator to an int*. So, why not have a "C++ reference type" instead of a pointer? It would essentially behave like a pointer and support indexing for dereferencing, but would not be compatible with pointers and would lead to the right code being generated for dereferencing and assigning. Stefan From tokidokix at gmail.com Mon Jul 5 15:57:06 2010 From: tokidokix at gmail.com (toki doki) Date: Mon, 5 Jul 2010 22:57:06 +0900 Subject: [Cython] About the cython/C++ doc In-Reply-To: References: <4C3185C8.7090800@behnel.de> <4C31BB28.2020105@behnel.de> Message-ID: OK, I have merged the content of the Wrapping_C++ wiki into the user guide Wrapping_C++. I did only minimal changes to the content of the wiki. Therefore, the style could do with some improving, but at least all the infos should be included. I slightly changed the structure, so that there is now only 4 main chapters: - Overview - A simple Tutorial (covering the basics of wrapping a C++ class) - Advanced C++ features (covering everything that was not discussed in the tutorial) - Caveats and limitations ( extern "C" functions, references, ...) Compared to the wiki version, I added the following: - nested class declaration syntax - operators in cython.operator (only dereference and preincrement; if someone know more of them, please tell me) - pre-declared pxds for the standard library Now, to do the "documentation switch", it should just be enough to: -- update the user guide with the attached file -- update the wiki files Do you want to do the switch now or wait for the cython0.13 release? Of course, anybody, feel free to first check the new file. It can certainly do with some revisions. ******* I have also a couple of questions that would deserved to be answered in the doc. 1- The original user guide recommend the following for initializing an extension class wrapping the C++ class Rectangle: def __cinit__(self, int x0, int y0, int x1, int y1): self.thisptr = new_Rectangle(x0, y0, x1, y1) Should we recommend "def __cinit__(self, int x0, int y0, int x1, int y1) except +MemoryError" instead? 2- Am I correct that it is not possible to use stack-allocated C++ objects as Python extension class members? 3- The whole C++ reference thing still seems to be a bit messy, so I did not discuss it much in the doc. I am curious about one small detail, though. In the STL pxds, the push_back function for vector is, for example, declared like this: void push_back(&T) Why not just "void push_back(T)" ? More precisely: is there any point to tell python which arguments of a declared function are references?. It seems the code that cython must generate is the same in any case... 4- I think there is no support for templated functions. Am I right? That's all for now. Toki -------------- next part -------------- A non-text attachment was scrubbed... Name: updateWrapping_CPlusPlus_for013.diff Type: text/x-patch Size: 19933 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100705/34a1627a/attachment.bin From stefan_ml at behnel.de Mon Jul 5 16:32:33 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 16:32:33 +0200 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 Message-ID: <4C31ED01.4080807@behnel.de> Hi, Py2.7 final gives me this: Python 2.7 (r27:82500, Jul 5 2010, 13:37:06) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> {i for i in range(10)}; i set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) Traceback (most recent call last): File "", line 1, in NameError: name 'i' is not defined >>> {i:i for i in range(10)}; i {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} Traceback (most recent call last): File "", line 1, in NameError: name 'i' is not defined List comprehensions have not changed: >>> [i for i in range(10)]; i [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 9 I know, I have asked this more than once already, but I think this is finally enough of a reason to fix the current behaviour also in Cython. Objections? Stefan From dalcinl at gmail.com Mon Jul 5 17:15:40 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 5 Jul 2010 12:15:40 -0300 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 In-Reply-To: <4C31ED01.4080807@behnel.de> References: <4C31ED01.4080807@behnel.de> Message-ID: On 5 July 2010 11:32, Stefan Behnel wrote: > Hi, > > Py2.7 final gives me this: > > ? ? Python 2.7 (r27:82500, Jul ?5 2010, 13:37:06) > ? ? [GCC 4.4.3] on linux2 > ? ? Type "help", "copyright", "credits" or "license" for more information. > > ? ? >>> {i for i in range(10)}; i > ? ? set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > ? ? Traceback (most recent call last): > ? ? ?File "", line 1, in > ? ? NameError: name 'i' is not defined > > ? ? >>> {i:i for i in range(10)}; i > ? ? {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} > ? ? Traceback (most recent call last): > ? ? ? File "", line 1, in > ? ? NameError: name 'i' is not defined > > List comprehensions have not changed: > > ? ? >>> [i for i in range(10)]; i > ? ? [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > ? ? 9 > > I know, I have asked this more than once already, but I think this is > finally enough of a reason to fix the current behaviour also in Cython. > No objections from my side. However, perhaps you would like to consider a compiler directive to enable this annoying thing? Ideally, by default, it should be enabled on *.py files and disabled on *.pyx files. PS: We have to start in making Cython target Python 3, and provide support for Python 2 features on-demand, let say by using a compiler directive #python2: enable-something=True,use-this-semantics=True -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Mon Jul 5 17:34:24 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 17:34:24 +0200 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 In-Reply-To: References: <4C31ED01.4080807@behnel.de> Message-ID: <4C31FB80.2030602@behnel.de> Lisandro Dalcin, 05.07.2010 17:15: > On 5 July 2010 11:32, Stefan Behnel wrote: >> Py2.7 final gives me this: >> >> Python 2.7 (r27:82500, Jul 5 2010, 13:37:06) >> [GCC 4.4.3] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >> >>> {i for i in range(10)}; i >> set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >> Traceback (most recent call last): >> File "", line 1, in >> NameError: name 'i' is not defined >> >> >>> {i:i for i in range(10)}; i >> {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} >> Traceback (most recent call last): >> File "", line 1, in >> NameError: name 'i' is not defined >> >> List comprehensions have not changed: >> >> >>> [i for i in range(10)]; i >> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >> 9 >> >> I know, I have asked this more than once already, but I think this is >> finally enough of a reason to fix the current behaviour also in Cython. > > No objections from my side. However, perhaps you would like to > consider a compiler directive to enable this annoying thing? Ideally, > by default, it should be enabled on *.py files and disabled on *.pyx > files. What purpose would that serve? I think it's one way or the other. (And my preference should be clear by now.) > PS: We have to start in making Cython target Python 3, and provide > support for Python 2 features on-demand, let say by using a compiler > directive #python2: enable-something=True,use-this-semantics=True There are "-2" and "-3" options at the command line and a "language_level" directive at the source level now. Stefan From dalcinl at gmail.com Mon Jul 5 18:11:10 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 5 Jul 2010 13:11:10 -0300 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 In-Reply-To: <4C31FB80.2030602@behnel.de> References: <4C31ED01.4080807@behnel.de> <4C31FB80.2030602@behnel.de> Message-ID: On 5 July 2010 12:34, Stefan Behnel wrote: > Lisandro Dalcin, 05.07.2010 17:15: >> On 5 July 2010 11:32, Stefan Behnel wrote: >>> Py2.7 final gives me this: >>> >>> ? ? ?Python 2.7 (r27:82500, Jul ?5 2010, 13:37:06) >>> ? ? ?[GCC 4.4.3] on linux2 >>> ? ? ?Type "help", "copyright", "credits" or "license" for more information. >>> >>> ? ? ?>>> ?{i for i in range(10)}; i >>> ? ? ?set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> ? ? ?Traceback (most recent call last): >>> ? ? ? File "", line 1, in >>> ? ? ?NameError: name 'i' is not defined >>> >>> ? ? ?>>> ?{i:i for i in range(10)}; i >>> ? ? ?{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} >>> ? ? ?Traceback (most recent call last): >>> ? ? ? ?File "", line 1, in >>> ? ? ?NameError: name 'i' is not defined >>> >>> List comprehensions have not changed: >>> >>> ? ? ?>>> ?[i for i in range(10)]; i >>> ? ? ?[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> ? ? ?9 >>> >>> I know, I have asked this more than once already, but I think this is >>> finally enough of a reason to fix the current behaviour also in Cython. >> >> No objections from my side. However, perhaps you would like to >> consider a compiler directive to enable this annoying thing? Ideally, >> by default, it should be enabled on *.py files and disabled on *.pyx >> files. > > What purpose would that serve? I think it's one way or the other. (And my > preference should be clear by now.) > Just backward compatibility, in case Cython is asked to handle *.py files. > >> PS: We have to start in making Cython target Python 3, and provide >> support for Python 2 features on-demand, let say by using a compiler >> directive #python2: enable-something=True,use-this-semantics=True > > There are "-2" and "-3" options at the command line and a "language_level" > directive at the source level now. > OK. Good to know we have -2/-3. Then, what are you going to do under -2 for list comprehensions? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Mon Jul 5 18:19:35 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Jul 2010 18:19:35 +0200 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 In-Reply-To: References: <4C31ED01.4080807@behnel.de> <4C31FB80.2030602@behnel.de> Message-ID: <4C320617.5090708@behnel.de> Lisandro Dalcin, 05.07.2010 18:11: > On 5 July 2010 12:34, Stefan Behnel wrote: >> Lisandro Dalcin, 05.07.2010 17:15: >>> On 5 July 2010 11:32, Stefan Behnel wrote: >>>> Py2.7 final gives me this: >>>> >>>> Python 2.7 (r27:82500, Jul 5 2010, 13:37:06) >>>> [GCC 4.4.3] on linux2 >>>> Type "help", "copyright", "credits" or "license" for more information. >>>> >>>> >>> {i for i in range(10)}; i >>>> set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> NameError: name 'i' is not defined >>>> >>>> >>> {i:i for i in range(10)}; i >>>> {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> NameError: name 'i' is not defined >>>> >>>> List comprehensions have not changed: >>>> >>>> >>> [i for i in range(10)]; i >>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>>> 9 >>>> >>>> I know, I have asked this more than once already, but I think this is >>>> finally enough of a reason to fix the current behaviour also in Cython. >>> >>> No objections from my side. However, perhaps you would like to >>> consider a compiler directive to enable this annoying thing? Ideally, >>> by default, it should be enabled on *.py files and disabled on *.pyx >>> files. >> >> What purpose would that serve? I think it's one way or the other. (And my >> preference should be clear by now.) > > Just backward compatibility, in case Cython is asked to handle *.py files. > >> >>> PS: We have to start in making Cython target Python 3, and provide >>> support for Python 2 features on-demand, let say by using a compiler >>> directive #python2: enable-something=True,use-this-semantics=True >> >> There are "-2" and "-3" options at the command line and a "language_level" >> directive at the source level now. >> > > OK. Good to know we have -2/-3. Then, what are you going to do under > -2 for list comprehensions? Oh, list comprehensions won't change (unless compiling with -3), thus keeping the same quirky state that they have in Py2.x. Only set/dict comprehensions will change, following Py2.7, so that list comprehensions are the *only* remaining quirk here when compiling Py2 code. The -3 behaviour is already implemented anyway. Stefan From dagss at student.matnat.uio.no Mon Jul 5 19:38:18 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 05 Jul 2010 19:38:18 +0200 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 In-Reply-To: <4C320617.5090708@behnel.de> References: <4C31ED01.4080807@behnel.de> <4C31FB80.2030602@behnel.de> <4C320617.5090708@behnel.de> Message-ID: <4C32188A.6030307@student.matnat.uio.no> Stefan Behnel wrote: > Lisandro Dalcin, 05.07.2010 18:11: > >> On 5 July 2010 12:34, Stefan Behnel wrote: >> >>> Lisandro Dalcin, 05.07.2010 17:15: >>> >>>> On 5 July 2010 11:32, Stefan Behnel wrote: >>>> >>>>> Py2.7 final gives me this: >>>>> >>>>> Python 2.7 (r27:82500, Jul 5 2010, 13:37:06) >>>>> [GCC 4.4.3] on linux2 >>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>> >>>>> >>> {i for i in range(10)}; i >>>>> set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>>> Traceback (most recent call last): >>>>> File "", line 1, in >>>>> NameError: name 'i' is not defined >>>>> >>>>> >>> {i:i for i in range(10)}; i >>>>> {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} >>>>> Traceback (most recent call last): >>>>> File "", line 1, in >>>>> NameError: name 'i' is not defined >>>>> >>>>> List comprehensions have not changed: >>>>> >>>>> >>> [i for i in range(10)]; i >>>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>>>> 9 >>>>> >>>>> I know, I have asked this more than once already, but I think this is >>>>> finally enough of a reason to fix the current behaviour also in Cython. >>>>> >>>> No objections from my side. However, perhaps you would like to >>>> consider a compiler directive to enable this annoying thing? Ideally, >>>> by default, it should be enabled on *.py files and disabled on *.pyx >>>> files. >>>> >>> What purpose would that serve? I think it's one way or the other. (And my >>> preference should be clear by now.) >>> >> Just backward compatibility, in case Cython is asked to handle *.py files. >> >> >>>> PS: We have to start in making Cython target Python 3, and provide >>>> support for Python 2 features on-demand, let say by using a compiler >>>> directive #python2: enable-something=True,use-this-semantics=True >>>> >>> There are "-2" and "-3" options at the command line and a "language_level" >>> directive at the source level now. >>> >>> >> OK. Good to know we have -2/-3. Then, what are you going to do under >> -2 for list comprehensions? >> > > Oh, list comprehensions won't change (unless compiling with -3), thus > keeping the same quirky state that they have in Py2.x. Only set/dict > comprehensions will change, following Py2.7, so that list comprehensions > are the *only* remaining quirk here when compiling Py2 code. > Oh. Then I really can't imagine how anyone could ever disagree with you on this one. Dag Sverre From stefan_ml at behnel.de Tue Jul 6 08:21:21 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 06 Jul 2010 08:21:21 +0200 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 In-Reply-To: <4C32188A.6030307@student.matnat.uio.no> References: <4C31ED01.4080807@behnel.de> <4C31FB80.2030602@behnel.de> <4C320617.5090708@behnel.de> <4C32188A.6030307@student.matnat.uio.no> Message-ID: <4C32CB61.50106@behnel.de> Dag Sverre Seljebotn, 05.07.2010 19:38: > Stefan Behnel wrote: >> Lisandro Dalcin, 05.07.2010 18:11: >>> On 5 July 2010 12:34, Stefan Behnel wrote: >>>> Lisandro Dalcin, 05.07.2010 17:15: >>>>> On 5 July 2010 11:32, Stefan Behnel wrote: >>>>>> Py2.7 final gives me this: >>>>>> >>>>>> Python 2.7 (r27:82500, Jul 5 2010, 13:37:06) >>>>>> [GCC 4.4.3] on linux2 >>>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>>> >>>>>> >>> {i for i in range(10)}; i >>>>>> set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>>>> Traceback (most recent call last): >>>>>> File "", line 1, in >>>>>> NameError: name 'i' is not defined >>>>>> >>>>>> >>> {i:i for i in range(10)}; i >>>>>> {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} >>>>>> Traceback (most recent call last): >>>>>> File "", line 1, in >>>>>> NameError: name 'i' is not defined >>>>>> >>>>>> List comprehensions have not changed: >>>>>> >>>>>> >>> [i for i in range(10)]; i >>>>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>>>>> 9 >>>>>> >>>>>> I know, I have asked this more than once already, but I think this is >>>>>> finally enough of a reason to fix the current behaviour also in Cython. >>>>> [...] >>>>> PS: We have to start in making Cython target Python 3, and provide >>>>> support for Python 2 features on-demand, let say by using a compiler >>>>> directive #python2: enable-something=True,use-this-semantics=True >>>>> >>>> There are "-2" and "-3" options at the command line and a "language_level" >>>> directive at the source level now. >>>> >>> OK. Good to know we have -2/-3. Then, what are you going to do under >>> -2 for list comprehensions? >> >> Oh, list comprehensions won't change (unless compiling with -3), thus >> keeping the same quirky state that they have in Py2.x. Only set/dict >> comprehensions will change, following Py2.7, so that list comprehensions >> are the *only* remaining quirk here when compiling Py2 code. >> > Oh. Then I really can't imagine how anyone could ever disagree with you > on this one. Haven't heard from Robert yet, but let's assume that Py2.7 taking the lead won't be a problem for him, either. http://hg.cython.org/cython-closures/rev/cfbc219602b6 Stefan From stefan_ml at behnel.de Tue Jul 6 08:37:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 06 Jul 2010 08:37:56 +0200 Subject: [Cython] Cython 0.13 status: coming along nicely. In-Reply-To: References: Message-ID: <4C32CF44.7050101@behnel.de> Craig Citro, 12.06.2010 09:21: > Here's my current plan: > > - Saturday: add tests, push some patches. (I'm flying to CA tomorrow, > which is a perfect time for this.) > - Sunday: cut cython-0.13.alpha0, *finally*. After that, I'm going to > pull the new patches into cython-closures, and test Sage with that. > Here are some famous last words: I don't expect to run into too much > trouble, because nothing is *using* the new features. If that goes > well, I'll cut alpha1 shortly thereafter. So, what's the status here? Stefan From robertwb at gmail.com Wed Jul 7 05:53:30 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 6 Jul 2010 20:53:30 -0700 Subject: [Cython] Cython 0.13 status: coming along nicely. In-Reply-To: <4C32CF44.7050101@behnel.de> References: <4C32CF44.7050101@behnel.de> Message-ID: On Mon, Jul 5, 2010 at 11:37 PM, Stefan Behnel wrote: > Craig Citro, 12.06.2010 09:21: >> Here's my current plan: >> >> - Saturday: add tests, push some patches. (I'm flying to CA tomorrow, >> which is a perfect time for this.) >> - Sunday: cut cython-0.13.alpha0, *finally*. After that, I'm going to >> pull the new patches into cython-closures, and test Sage with that. >> Here are some famous last words: I don't expect to run into too much >> trouble, because nothing is *using* the new features. If that goes >> well, I'll cut alpha1 shortly thereafter. > > So, what's the status here? I'd love to know too. I modified the Sage compile process to get https://sage.math.washington.edu:8091/hudson/job/cython-devel-sage-build/95/testReport/ Craig, you have fixes to many of these, right? (It looks like they're mostly inferred str and undeclared type imports.) I don't want to duplicate effort, but would like to dive into this. - Robert From robertwb at gmail.com Wed Jul 7 06:07:34 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 6 Jul 2010 21:07:34 -0700 Subject: [Cython] set/dict comprehensions don't leak in Py2.7 In-Reply-To: <4C32CB61.50106@behnel.de> References: <4C31ED01.4080807@behnel.de> <4C31FB80.2030602@behnel.de> <4C320617.5090708@behnel.de> <4C32188A.6030307@student.matnat.uio.no> <4C32CB61.50106@behnel.de> Message-ID: On Mon, Jul 5, 2010 at 11:21 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 05.07.2010 19:38: >> Stefan Behnel wrote: >>> Oh, list comprehensions won't change (unless compiling with -3), thus >>> keeping the same quirky state that they have in Py2.x. Only set/dict >>> comprehensions will change, following Py2.7, so that list comprehensions >>> are the *only* remaining quirk here when compiling Py2 code. >>> >> Oh. Then I really can't imagine how anyone could ever disagree with you >> on this one. > > Haven't heard from Robert yet, but let's assume that Py2.7 taking the lead > won't be a problem for him, either. Makes sense to me--For instance, it won't break any existing Python code. - Robert From robertwb at gmail.com Wed Jul 7 06:21:12 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 6 Jul 2010 21:21:12 -0700 Subject: [Cython] About the cython/C++ doc In-Reply-To: References: <4C3185C8.7090800@behnel.de> <4C31BB28.2020105@behnel.de> Message-ID: On Mon, Jul 5, 2010 at 6:57 AM, toki doki wrote: > OK, I have merged the content of the Wrapping_C++ wiki into the user > guide Wrapping_C++. I did only minimal changes to the content of the > wiki. Therefore, the style could do with some improving, but at least > all the infos should be included. Thanks for jumping in and doing this. The old documentation was in quite a sorry state, in part because no one wanted to clean up the old way of doing things (as it was "soon" to be supplanted) and then when the new stuff finally got merged, the documentation wasn't updated. > I slightly changed the structure, so that there is now only 4 main chapters: > - ?Overview > - A simple Tutorial ?(covering the basics of wrapping a C++ class) > - Advanced C++ features (covering everything that was not discussed in > the tutorial) > - Caveats and limitations ( extern "C" functions, references, ...) > > Compared to the wiki version, I added the following: > - nested class declaration syntax > - operators in cython.operator (only dereference and preincrement; if > someone know more of them, please tell me) > - pre-declared pxds for the standard library > > Now, to do the "documentation switch", it should just be enough to: > -- update the user guide with the attached file > -- update the wiki files > > Do you want to do the switch now or wait for the cython0.13 release? > Of course, anybody, feel free to first check the new file. It can > certainly do with some revisions. I've pushed it to the docs repo, which will replace what's at docs.cython.org when we release. http://hg.cython.org/cython-docs/rev/b4f6d17816d7 We can probably change the wiki right now, making a <= 0.12 page with the old instructions and link to the new stuff, which will get replaced by a link to the docs once those go live. This is one of those things where, down the road, I would love automatic testing on. > ******* > I have also a couple of questions that would deserved to be answered in the doc. > > 1- The original user guide recommend the following for initializing an > extension class wrapping the C++ class Rectangle: > ? ?def __cinit__(self, int x0, int y0, int x1, int y1): > ? ? ? ?self.thisptr = new_Rectangle(x0, y0, x1, y1) > Should we recommend "def __cinit__(self, int x0, int y0, int x1, int > y1) except +MemoryError" ?instead? Yes, great catch. > 2- Am I correct that it is not possible to use stack-allocated C++ > objects as Python extension class members? Yes, at least as it stands now. Python extension classes are themselves allocated on the heap, and so we'd need to handle that in some special way, and it just gets really messy. (The details of stack-allocated classes are thorny in C++ itself, and things just get worse when trying to reason with them from another language...) > 3- The whole C++ reference thing still seems to be a bit messy, It is, and probably still has bugs, but was essential for some stuff. > so I > did not discuss it much in the doc. I am curious about one small > detail, though. In the STL pxds, the push_back function for vector is, > for example, declared like this: > > void push_back(&T) > > Why not just "void push_back(T)" ? ?More precisely: is there any point > to tell python which arguments of a declared function are references?. > It seems the code that cython must generate is the same in any case... Hmm... good point. I don't think it matters there (code-generation wise, semantically it makes a difference, but one would get that reading the C++ docs). > 4- ?I think there is no support for templated functions. Am I right? Correct (for now). > That's all for now. Thanks again! - Robert From robertwb at gmail.com Wed Jul 7 06:33:13 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 6 Jul 2010 21:33:13 -0700 Subject: [Cython] More c++ bugs In-Reply-To: <4C318AD3.8070204@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> Message-ID: On Mon, Jul 5, 2010 at 12:33 AM, Dag Sverre Seljebotn wrote: > Sturla Molden wrote: >> Sturla Molden skrev: >> >>> We have the same sematical difference when calling functions: >>> >>> ? ?void foobar(int& c); >>> >>> the reference c is intialized to alias the argument with which foobar is >>> called. But you can never rebind c to alias another object in the local >>> scope of foobar. If you assign to c you modify the variable aliased by c. >>> > Sorry, this could have been worded better -- references here refers to > Python references = C pointers (more or less), not C++ references. > >> And as for Python, if we have >> >> ? ?c = 5 >> ? ?foobar(c) >> >> the value of c cannot be changed as c is immutable. >> >> But if we have >> >> ? ?from ctypes import c_int >> ? ?c = c_int(5) >> ? ?foobar(c) >> >> then the call to foobar can change the value of c by assigning to >> c.value in its local scope. >> >> In either case, if foobar assigns to c in its local scope (i.e. rebinds >> the name), nothing happens to c in the calling scope. >> >> This is obviously very different from C++, but in neither case can >> foobar rebind the argument in the calling scope. >> >> We should not confuse the issues of name binding vs. assignment and >> mutability vs. immutability. It's not the same. But Python and C++ >> differs in both cases. >> > Yes, it was you who brought mutability into the mix; I claimed it was > irrelevant. > > Anyway...you keep stating things that we all know very well. Are you > agreeing or disagreeing with my and Stefan's position? Is there a point > to this discussion at all? > > Specifically: > > cdef int x = ... > cdef MyCppClass *y = ... > f(x, y) > > Do you want f to be allowed, if the arguments are declared as > references, to change the value of x and which memory location y points to? > > There's no need to go into explaining what C++ does (I'm well aware of > it), it's a simple yes or no question, and the question is simply "do we > allow this feature of C++ into the Cython language, or do we take > another approach instead that looks less strange with Python semantics?". > > (Anyway, I think the question is pretty much settled.) The primary goal with C++ support is to provide just enough C++isms to easily use any of the many C++ libraries out there, but not so much that the user can easily hang themselves. The goal is not to facilitate writing C++ in Cython, but rather using C++ from Cython (though the two overlap quite a bit on the technical plane). If we want to disallow, say, references in function arguments, now is the time to do so. Is there any reason this is necessary to allow? As for the messiness of the rest of C++, we use the new operator because stack allocated objects are a mess (what with mixing scopes and all). Eventually we should be able to catch C++ exceptions with the try/except block, and not let C++ exceptions escape from try/finally which would make things a bit safer. - Robert From craigcitro at gmail.com Wed Jul 7 06:47:16 2010 From: craigcitro at gmail.com (Craig Citro) Date: Tue, 6 Jul 2010 21:47:16 -0700 Subject: [Cython] Cython 0.13 status: coming along nicely. In-Reply-To: References: <4C32CF44.7050101@behnel.de> Message-ID: >> So, what's the status here? > > I'd love to know too. Yeah, I'm sorry -- I've totally been terrible about actually replying to email, which is bad form. Here's the status: I decided to try seeing what the state of affairs was with just moving my current stack of patches over to cython-closures. It turns out it's not so bad, but there's still some trouble with type analysis/inference, not related to closures, but instead to the other code in the closures branch. I'm going to sit down right now and clean up these patches and the patches against sage and push, so that it's no longer a mystery where we're at. > Craig, you have fixes to many of these, right? (It looks like they're > mostly inferred str and undeclared type imports.) I don't want to > duplicate effort, but would like to dive into this. > Yep, I'm going to work on getting everything pushed right now, so that I'm not the only one who can look at this stuff. -cc From dagss at student.matnat.uio.no Wed Jul 7 08:49:25 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 07 Jul 2010 08:49:25 +0200 Subject: [Cython] More c++ bugs In-Reply-To: References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> Message-ID: <4C342375.2060409@student.matnat.uio.no> Robert Bradshaw wrote: > The primary goal with C++ support is to provide just enough C++isms to > easily use any of the many C++ libraries out there, but not so much > that the user can easily hang themselves. The goal is not to > facilitate writing C++ in Cython, but rather using C++ from Cython > (though the two overlap quite a bit on the technical plane). If we > want to disallow, say, references in function arguments, now is the > time to do so. Is there any reason this is necessary to allow? > I don't think anything of this has been on writing C++, just about talking to C++ libraries. The reason to allow references in function arguments in "cdef extern" C++ functions is that, well, libraries do come with such constructs, and it is not clear how one would communicate with them if references in function arguments isn't supported somehow. They can be used either as out arguments, or for passing around some global context, etc. Dag Sverre From stefan_ml at behnel.de Wed Jul 7 09:04:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 07 Jul 2010 09:04:56 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <4C342375.2060409@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C342375.2060409@student.matnat.uio.no> Message-ID: <4C342718.20701@behnel.de> Dag Sverre Seljebotn, 07.07.2010 08:49: > Robert Bradshaw wrote: >> The primary goal with C++ support is to provide just enough C++isms to >> easily use any of the many C++ libraries out there, but not so much >> that the user can easily hang themselves. The goal is not to >> facilitate writing C++ in Cython, but rather using C++ from Cython >> (though the two overlap quite a bit on the technical plane). If we >> want to disallow, say, references in function arguments, now is the >> time to do so. Is there any reason this is necessary to allow? >> > I don't think anything of this has been on writing C++, just about > talking to C++ libraries. > > The reason to allow references in function arguments in "cdef extern" > C++ functions is that, well, libraries do come with such constructs, and > it is not clear how one would communicate with them if references in > function arguments isn't supported somehow. They can be used either as > out arguments, or for passing around some global context, etc. With my limited knowledge of C++, I don't see a problem here. After all, Cython ignores the exact values stored in those references, and as longs as you can't pass a ref-counted Python object as a C++ reference, it doesn't have to know anything about them. This will not work, though: function_taking_a_reference(some_object) but it should not be too hard to catch by the compiler, in case it's currently valid. Stefan From craigcitro at gmail.com Wed Jul 7 09:18:50 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 7 Jul 2010 00:18:50 -0700 Subject: [Cython] Testing import/cimport and tests spanning nested directories Message-ID: Hi all, So I'm currently trying to write some tests for a subtle issue involving resolving imports and cimports. (I think I emailed about this a while back.) The particular issue isn't so important right this minute -- the only thing that matters is that to test it, I need a test with a few layers of directories, with __init__.py files sprinkled in several places, and potentially *not* at the root. Unfortunately, this doesn't work so well with the current testing setup -- at least, as I understand, there's no way to tell it "when you try to run this test, also copy along this particular chunk of the current directory." So there are two obvious choices: 1) Give a test a way of specifying what other files should come along as part of it. 2) Create a directory called nested (or something else) somewhere in the tests/ tree (maybe one for each of the current subdirectories?), and add code to runtests.py to handle that whole tree at once. Each of these has its ups and downs -- but I'd love it if someone told me that the testing code already did what I'm looking for, and I just didn't know it. ;) I think (1) feels like the "classier" solution, but (2) would make it easy for someone to *add* a nested test without having to create a whole bunch of files/directories themselves. That said, we'd have to be careful that no one accidentally broke a test by modifying it -- tests really should be nearly immutable. Any thoughts? -cc From dagss at student.matnat.uio.no Wed Jul 7 09:36:37 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 07 Jul 2010 09:36:37 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <4C342718.20701@behnel.de> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C342375.2060409@student.matnat.uio.no> <4C342718.20701@behnel.de> Message-ID: <4C342E85.2030908@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn, 07.07.2010 08:49: > >> Robert Bradshaw wrote: >> >>> The primary goal with C++ support is to provide just enough C++isms to >>> easily use any of the many C++ libraries out there, but not so much >>> that the user can easily hang themselves. The goal is not to >>> facilitate writing C++ in Cython, but rather using C++ from Cython >>> (though the two overlap quite a bit on the technical plane). If we >>> want to disallow, say, references in function arguments, now is the >>> time to do so. Is there any reason this is necessary to allow? >>> >>> >> I don't think anything of this has been on writing C++, just about >> talking to C++ libraries. >> >> The reason to allow references in function arguments in "cdef extern" >> C++ functions is that, well, libraries do come with such constructs, and >> it is not clear how one would communicate with them if references in >> function arguments isn't supported somehow. They can be used either as >> out arguments, or for passing around some global context, etc. >> > > With my limited knowledge of C++, I don't see a problem here. After all, > Cython ignores the exact values stored in those references, and as longs as > you can't pass a ref-counted Python object as a C++ reference, it doesn't > have to know anything about them. > > This will not work, though: > > function_taking_a_reference(some_object) > > but it should not be too hard to catch by the compiler, in case it's > currently valid. > Hmm. Not sure if I follow you. Just to make sure everybody's on the same page, I'll put down what C++ references is, just in case. They're syntax candy around const pointers. A reference is a) a pointer, b) always points to something, c) never change to point to something else, d) is always automatically dereferenced (so assignment etc. acts on the pointed-to object). int a = 4; int &b = a; b = 3; // changes a to 3. There's no way to change the pointer b itself. The current state in Cython leads to the following problem: C++: void f(int& x) { x = 10; } Cython: cdef extern from "foo.h": void f(int x) # int* will cause syntax errors, it must be plain int cdef int x = 3 y = 3 # object f(x) # x changes to 10 f(y) # the temporary result of conversion is changed -- y is not changed! So the current state is both inobvious and would lead to hard to find bugs IMO. So that's why I argue explicit support is needed in some form or another. Dag Sverre From dagss at student.matnat.uio.no Wed Jul 7 09:40:05 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 07 Jul 2010 09:40:05 +0200 Subject: [Cython] Testing import/cimport and tests spanning nested directories In-Reply-To: References: Message-ID: <4C342F55.2020500@student.matnat.uio.no> Craig Citro wrote: > Hi all, > > So I'm currently trying to write some tests for a subtle issue > involving resolving imports and cimports. (I think I emailed about > this a while back.) The particular issue isn't so important right this > minute -- the only thing that matters is that to test it, I need a > test with a few layers of directories, with __init__.py files > sprinkled in several places, and potentially *not* at the root. > Unfortunately, this doesn't work so well with the current testing > setup -- at least, as I understand, there's no way to tell it "when > you try to run this test, also copy along this particular chunk of the > current directory." So there are two obvious choices: > > 1) Give a test a way of specifying what other files should come along > as part of it. > 2) Create a directory called nested (or something else) somewhere in > the tests/ tree (maybe one for each of the current subdirectories?), > and add code to runtests.py to handle that whole tree at once. > > Each of these has its ups and downs -- but I'd love it if someone told > me that the testing code already did what I'm looking for, and I just > didn't know it. ;) I think (1) feels like the "classier" solution, but > (2) would make it easy for someone to *add* a nested test without > having to create a whole bunch of files/directories themselves. That > said, we'd have to be careful that no one accidentally broke a test by > modifying it -- tests really should be nearly immutable. > > Any thoughts? > If this ends up being coded from scratch, I think it would make testing easier if one could actually write down all the files in a single testcase bundle file (e.g. with a "test" suffix). Something like the following, which is simply split up into seperate files by runtests.py: == mydir/__init__.py # This file is empty == mydir/foo.pyx cdef int x = 3 == mydir/foo.pxd cdef int x This is because such tests would tend to have a lot of very small files testing specific functionality, and it would be very tiring to browse around such trees and open all the files to see what is going on. Dag Sverre From craigcitro at gmail.com Wed Jul 7 09:46:40 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 7 Jul 2010 00:46:40 -0700 Subject: [Cython] Testing import/cimport and tests spanning nested directories In-Reply-To: <4C342F55.2020500@student.matnat.uio.no> References: <4C342F55.2020500@student.matnat.uio.no> Message-ID: > If this ends up being coded from scratch, I think it would make testing > easier if one could actually write down all the files in a single > testcase bundle file (e.g. with a "test" suffix). Something like the > following, which is simply split up into seperate files by runtests.py: > I *really* like this idea -- I've already gotten tired of browsing through the few files in the tests I've got. ;) Then we wouldn't need to worry about having an extra tree floating around, and it's very natural for someone to copy such a test to use as a base for what they want to do ... +1 -cc From stefan_ml at behnel.de Wed Jul 7 09:49:03 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 07 Jul 2010 09:49:03 +0200 Subject: [Cython] Testing import/cimport and tests spanning nested directories In-Reply-To: References: Message-ID: <4C34316F.70000@behnel.de> Craig Citro, 07.07.2010 09:18: > So I'm currently trying to write some tests for a subtle issue > involving resolving imports and cimports. (I think I emailed about > this a while back.) The particular issue isn't so important right this > minute -- the only thing that matters is that to test it, I need a > test with a few layers of directories, with __init__.py files > sprinkled in several places, and potentially *not* at the root. > Unfortunately, this doesn't work so well with the current testing > setup -- at least, as I understand, there's no way to tell it "when > you try to run this test, also copy along this particular chunk of the > current directory." Well, it works for Cython translation time dependencies, as those are searched relative to the directory of the main .pyx file. So include tests work, as do .pxd file dependencies. Wrapper tests are supported by a naming convention, i.e. you can write a wrapper code file wrapper.pyx and a C file wrapper_whatever.c and the C file will be copied to the C compiler working directory automatically. I'm not quite sure I understand what else you need. Even subdirectories for translation time imports should work in the current setup. Could you give an example? Stefan From craigcitro at gmail.com Wed Jul 7 10:04:33 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 7 Jul 2010 01:04:33 -0700 Subject: [Cython] Testing import/cimport and tests spanning nested directories In-Reply-To: <4C34316F.70000@behnel.de> References: <4C34316F.70000@behnel.de> Message-ID: > I'm not quite sure I understand what else you need. Even subdirectories for > translation time imports should work in the current setup. Could you give > an example? > I'm trying to make sure that code gets generated correctly when you both import and cimport something, and I'm trying to be exhaustive about checking it when it's in the same package, in a different package, nested further into the package, etc. So I need to have a file test.pyx, and then two files nested/target.pyx and nested/target.pxd, so that test.pyx sees target.pxd at runtime, but target.so gets built and is available for testing at runtime. In fact, I should even test that in the case that target.* is in the same directory as test.pyx -- so I guess the nesting issue is orthogonal ... -cc From stefan_ml at behnel.de Wed Jul 7 10:21:52 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 07 Jul 2010 10:21:52 +0200 Subject: [Cython] Testing import/cimport and tests spanning nested directories In-Reply-To: References: <4C342F55.2020500@student.matnat.uio.no> Message-ID: <4C343920.5030001@behnel.de> Craig Citro, 07.07.2010 09:46: >> If this ends up being coded from scratch, I think it would make testing >> easier if one could actually write down all the files in a single >> testcase bundle file (e.g. with a "test" suffix). Something like the >> following, which is simply split up into seperate files by runtests.py: > > I *really* like this idea -- I've already gotten tired of browsing > through the few files in the tests I've got. ;) Then we wouldn't need > to worry about having an extra tree floating around, and it's very > natural for someone to copy such a test to use as a base for what they > want to do ... Ok, given your explanation in your other mail, this makes sense to me, too. That makes it easy enough to write tests with simple files in different places, be it .py files or .pyx files or .pxd files. The import mechanism is clearly lacking a lot of tests, so I'm happy to see this getting solved. It could also be an alternative to (or replace?) the naming convention for wrapper files, assuming that a short(!) wrapper test is much easier to read when everything is in one file. For longer files, this can get harder to read and edit, though, depending on the editor support (Emacs will work well, I guess). I guess both ways have their raison d'?tre... Stefan From craigcitro at gmail.com Wed Jul 7 10:28:14 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 7 Jul 2010 01:28:14 -0700 Subject: [Cython] Cython 0.13 status: coming along nicely. In-Reply-To: References: <4C32CF44.7050101@behnel.de> Message-ID: > Yep, I'm going to work on getting everything pushed right now, so that > I'm not the only one who can look at this stuff. > So I didn't get to add tests to everything, but I'm about to head to bed. Instead of just disappearing into the night, I thought I'd put the patches somewhere people can pull and play with them -- I've got a repo up here: http://sage.math.washington.edu/home/craigcitro/cython-0.13-working/ There are also several changes to the Sage library (which Robert asked about above). These apply cleanly in order against sage-4.4.2. You can find them here: http://sage.math.washington.edu/home/craigcitro/cython-0.13-working/sage-patches/ As you'd expect, the patches are subsumed by the .hg bundle. (Robert, I'll do the work of making tickets for these and cleaning up commit messages tomorrow, once other stuff is ready to go.) So as soon as I've got tests written for all of these, I think everything can get pushed -- and cut the cython-0.13 alpha I promised previously. :P After that, I think it's worth trying to make a quick push to sort out the last issues with sage + the closures code -- I don't think it's in bad shape, and with several of us finally having some time to work again, I'm hopeful that we can get everything fixed up fairly quickly ... but we've seen where my optimism has gotten us in the past. ;) -cc From greg.ewing at canterbury.ac.nz Wed Jul 7 12:32:15 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Jul 2010 22:32:15 +1200 Subject: [Cython] More c++ bugs In-Reply-To: <4C342375.2060409@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C342375.2060409@student.matnat.uio.no> Message-ID: <4C3457AF.9000207@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > The reason to allow references in function arguments in "cdef extern" > C++ functions is that, well, libraries do come with such constructs, and > it is not clear how one would communicate with them if references in > function arguments isn't supported somehow. Yep, I think that if you want to get serious about interfacing with C++, you're going to have to incorporate C++ references as a native Cython data type. Also change the syntax to allow a function call to be an lvalue. -- Greg From stefan_ml at behnel.de Wed Jul 7 13:03:20 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 07 Jul 2010 13:03:20 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <4C342E85.2030908@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C342375.2060409@student.matnat.uio.no> <4C342718.20701@behnel.de> <4C342E85.2030908@student.matnat.uio.no> Message-ID: <4C345EF8.1070208@behnel.de> Dag Sverre Seljebotn, 07.07.2010 09:36: > Stefan Behnel wrote: >> With my limited knowledge of C++, I don't see a problem here. After all, >> Cython ignores the exact values stored in those references, and as longs as >> you can't pass a ref-counted Python object as a C++ reference, it doesn't >> have to know anything about them. >> >> This will not work, though: >> >> function_taking_a_reference(some_object) >> >> but it should not be too hard to catch by the compiler, in case it's >> currently valid. > [...] > The current state in Cython leads to the following problem: > > C++: > void f(int& x) { x = 10; } > > Cython: > cdef extern from "foo.h": > void f(int x) # int* will cause syntax errors, it must be plain int > > cdef int x = 3 > y = 3 # object > f(x) # x changes to 10 > f(y) # the temporary result of conversion is changed -- y is not changed! Right, that's ugly. However, what I actually meant above is this: C++: (not sure if this syntax works) void f(PyObject*& x) { x = PyInt_FromLong(10); } Cython: cdef extern from "foo.h": void f(object x) f(5) But I'd also be fine with laughing at the hole in the foot of a user who does that. > So the current state is both inobvious and would lead to hard to find > bugs IMO. So that's why I argue explicit support is needed in some form > or another. Agreed. I think a specific type for C++ references would solve this, though. If you could write something like cdef extern from "foo.h": void f(&int x) # syntax improvements pending :) then Cython could take care about required coercions, re-assignments and compile time errors. Stefan From tokidokix at gmail.com Wed Jul 7 15:22:07 2010 From: tokidokix at gmail.com (toki doki) Date: Wed, 7 Jul 2010 22:22:07 +0900 Subject: [Cython] More c++ bugs In-Reply-To: <4C342375.2060409@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C342375.2060409@student.matnat.uio.no> Message-ID: On Wed, Jul 7, 2010 at 3:49 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> The primary goal with C++ support is to provide just enough C++isms to >> easily use any of the many C++ libraries out there, but not so much >> that the user can easily hang themselves. The goal is not to >> facilitate writing C++ in Cython, but rather using C++ from Cython >> (though the two overlap quite a bit on the technical plane). If we >> want to disallow, say, references in function arguments, now is the >> time to do so. Is there any reason this is necessary to allow? >> > I don't think anything of this has been on writing C++, just about > talking to C++ libraries. > > The reason to allow references in function arguments in "cdef extern" > C++ functions is that, well, libraries do come with such constructs, and > it is not clear how one would communicate with them if references in > function arguments isn't supported somehow. They can be used either as > out arguments, or for passing around some global context, etc. > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Actually, I don't really get template assign(T &target, const V &value){ target=value;} cdef extern from "": cdef cppclass vector[T]: assign(vect.at(4),56) From tokidokix at gmail.com Wed Jul 7 16:17:38 2010 From: tokidokix at gmail.com (toki doki) Date: Wed, 7 Jul 2010 23:17:38 +0900 Subject: [Cython] More c++ bugs In-Reply-To: References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C342375.2060409@student.matnat.uio.no> Message-ID: >> I don't think anything of this has been on writing C++, just about >> talking to C++ libraries. >> >> The reason to allow references in function arguments in "cdef extern" >> C++ functions is that, well, libraries do come with such constructs, and >> it is not clear how one would communicate with them if references in >> function arguments isn't supported somehow. They can be used either as >> out arguments, or for passing around some global context, etc. >> >> Dag Sverre Sorry for the partially written mail above. My fingers ripped... I actually wanted to say I don't get why it is important for Cython to know anything about C++ references. But then I saw the post of Dag below and saw his point. I will still comment on why I don't think specific support for C++ references is that important in my (short) experience (just my 2 cents, please do not flame me if you think otherwise or that I did not see some obvious cases :-) . I am currently wrapping several small C++ libraries/programs, and I found that I can basically do everything without declaring references in Cython: If a C++ function use references as arguments, I just use the plain type in the Cython declaration. (for example, "void push_back(T)" instead of "void push_back(&T)" in a wrapping of std::vector). This also protects me from bug #550. I do the same for functions returning an reference (i.e the declared return value is the plain type, not a reference). And when a function returning a reference needs to be a l-value, I use the following templated function declared in "my_cpp_utils.h": """"""" template void assign(T &target, const V &value){ target=value;} """"""" Thus, a cython code using std::vector would be like this: """""""" cdef extern from "": cdef cppclass vector[T]: void push_back(T) T at(size_t) cdef extern from "my_cpp_utils.h": void assign(int,int) vector[int] vect(10) assign(vect.at(4),56) vect.push_back(30) """""""" No reference ever declared... The only problem is that I have to add a declaration for "assign" for every type combination I need to use, since templated functions are not supported by cython. Dag made a good point about functions having automatically converted python objects as arguments. I would suggest an easy solution is to just forbid to use python objects as reference argument. To get back to Dag's example, as a user, I would not consider having to write the following a big burden: f(y) #rejected by cython with a helpful error message, so I know there is a subtlety here # after understanding the issue, I can just write: int z=y f(z) # OK y=z # Also, note that the majority (I think) of C++ functions having reference argument do so just to avoid the cost of a copy of the argument (when this cost is high) and use "const references". These functions require no special handling, and trying to synchronize the value of the python object with its automatically created C equivalent would add useless overhead (there is no hard guaranty that a function declaring a "const reference" argument will not modify it though, but that would be very bad behaviour). cheers, Toki From cb at pdos.csail.mit.edu Wed Jul 7 17:22:35 2010 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Wed, 7 Jul 2010 11:22:35 -0400 Subject: [Cython] Closure Issue Message-ID: <20100707152235.GA19282@pdos.lcs.mit.edu> The code generated by the following compiles without error but then tries to INCREF a null pointer for x (all for closures branch tip): cdef f(x): # def here => works fine def g(y): return y*x # cdef here => compile error return g(x) # faults@ INCREF(.*cur_scope->.*v_x print f(2) If the intent is not to support the above scenario then this is a compiler-should-fail-but-doesn't bug, else a code generation bug. It may be a known issue, but I couldn't find a test case or a Trac issue. So, I thought that I should mention it just in case. From robertwb at gmail.com Wed Jul 7 17:25:26 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 7 Jul 2010 08:25:26 -0700 Subject: [Cython] Cython 0.13 status: coming along nicely. In-Reply-To: References: <4C32CF44.7050101@behnel.de> Message-ID: On Wed, Jul 7, 2010 at 1:28 AM, Craig Citro wrote: >> Yep, I'm going to work on getting everything pushed right now, so that >> I'm not the only one who can look at this stuff. >> > > So I didn't get to add tests to everything, but I'm about to head to > bed. Instead of just disappearing into the night, I thought I'd put > the patches somewhere people can pull and play with them -- I've got a > repo up here: > > ?http://sage.math.washington.edu/home/craigcitro/cython-0.13-working/ > > There are also several changes to the Sage library (which Robert asked > about above). These apply cleanly in order against sage-4.4.2. You can > find them here: > > ?http://sage.math.washington.edu/home/craigcitro/cython-0.13-working/sage-patches/ > > As you'd expect, the patches are subsumed by the .hg bundle. (Robert, > I'll do the work of making tickets for these and cleaning up commit > messages tomorrow, once other stuff is ready to go.) Excellent. I've added them to the queue for https://sage.math.washington.edu:8091/hudson/job/cython-devel-sage-build/ (we can pop them off and refine them if you have further changes). I haven't pushed your Cython changes, you can do that when you're confortable with them. > So as soon as I've got tests written for all of these, I think > everything can get pushed -- and cut the cython-0.13 alpha I promised > previously. :P After that, I think it's worth trying to make a quick > push to sort out the last issues with sage + the closures code -- I > don't think it's in bad shape, and with several of us finally having > some time to work again, I'm hopeful that we can get everything fixed > up fairly quickly ... but we've seen where my optimism has gotten us > in the past. ;) Well, now we've got double the optimism going on :) - Robert From robertwb at gmail.com Wed Jul 7 17:37:59 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 7 Jul 2010 08:37:59 -0700 Subject: [Cython] Closure Issue In-Reply-To: <20100707152235.GA19282@pdos.lcs.mit.edu> References: <20100707152235.GA19282@pdos.lcs.mit.edu> Message-ID: On Wed, Jul 7, 2010 at 8:22 AM, Chuck Blake wrote: > The code generated by the following compiles without error but then > tries to INCREF a null pointer for x (all for closures branch tip): > > ? ?cdef f(x): ? ? ? ? ? ? ? ?# def ?here => works fine > ? ? ? ?def g(y): return y*x ?# cdef here => compile error > ? ? ? ?return g(x) ? ? ? ? ? # faults@ INCREF(.*cur_scope->.*v_x > ? ?print f(2) > > If the intent is not to support the above scenario then this is a > compiler-should-fail-but-doesn't bug, else a code generation bug. > > It may be a known issue, but I couldn't find a test case or a Trac > issue. ?So, I thought that I should mention it just in case. Yep, looks like a bug to me: http://trac.cython.org/cython_trac/ticket/554 - Robert From stefan_ml at behnel.de Wed Jul 7 19:05:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 07 Jul 2010 19:05:56 +0200 Subject: [Cython] cpdef (unbound) method inheritance broken In-Reply-To: References: Message-ID: <4C34B3F4.9080204@behnel.de> Lisandro Dalcin, 10.06.2010 16:50: > Please try the attached test case. If you switch the comment in the > first two lines, you will get the failure. So far, I could not figure > out what's going on. Your code is basically this: --------------------- cdef class Foo2: cpdef hello(self): print 'Foo.hello()' cdef class Bar2(Foo2): cpdef hello(self): print 'Bar.hello()' Foo.hello(b) --------------------- The problem is that this prints "Bar.hello()", not "Foo.hello()" as one would expect (and which is how CPython behaves). gdb gives me this for C function call in the "Foo2.hallo" method wrapper: 601 __pyx_t_1 = ((struct __pyx_vtabstruct_11methinherit_Foo2 *)((struct __pyx_obj_11methinherit_Foo2 *)__pyx_v_self)->__pyx_vtab)->hello(((struct __pyx_obj_11methinherit_Foo2 *)__pyx_v_self), 1); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 35; __pyx_clineno = __LINE__; goto __pyx_L1_error;} (gdb) print __pyx_v_self $1 = (gdb) pyo __pyx_v_self object : type : methinherit.Bar2 refcount: 3 address : 0x7ffff7ee40f0 $2 = void (gdb) print ((struct __pyx_obj_11methinherit_Foo2 *)__pyx_v_self)->__pyx_vtab $3 = (struct __pyx_vtabstruct_11methinherit_Foo2 *) 0x7ffff69ba2f0 (gdb) print ((struct __pyx_obj_11methinherit_Foo2 *)__pyx_v_self)->__pyx_vtab->hello $4 = (PyObject *(*)(struct __pyx_obj_11methinherit_Foo2 *, int)) 0x7ffff67b6492 <__pyx_f_11methinherit_4Bar2_hello> Note the "__pyx_f_11methinherit_4Bar2_hello" function in the last line. So, the problem is that the cpdef wrapper effectively calls "self.hello", which is "Bar.hello" in this case. This is the wrong thing to do for an unbound method. However, I'm not sure how to fix this, either. Any ideas? Stefan From sturla at molden.no Thu Jul 8 00:22:15 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 08 Jul 2010 00:22:15 +0200 Subject: [Cython] More c++ bugs In-Reply-To: <4C342E85.2030908@student.matnat.uio.no> References: <3361049998.505334@smtp.netcom.no> <4C2FCF1A.2060806@molden.no> <4C2FD72E.6090100@molden.no> <4C318AD3.8070204@student.matnat.uio.no> <4C342375.2060409@student.matnat.uio.no> <4C342718.20701@behnel.de> <4C342E85.2030908@student.matnat.uio.no> Message-ID: <4C34FE17.7080302@molden.no> Dag Sverre Seljebotn skrev: > They're syntax candy around const pointers. A reference is a) a pointer, > b) always points to something, c) never change to point to something > else, d) is always automatically dereferenced (so assignment etc. acts > on the pointed-to object). > > int a = 4; > int &b = a; > b = 3; // changes a to 3. There's no way to change the pointer b itself. > > A reference is not a pointer but a logical alias. So in your code above, a and b are aliases (i.e. different names) for the same variable. That means: &b == &a when b is a reference to a. Pointers are variables. So if references were pointers, a and b would be stored in different addresses: &b != &a This is the important difference between const pointers and references. Sturla > The current state in Cython leads to the following problem: > > C++: > void f(int& x) { x = 10; } > > Cython: > cdef extern from "foo.h": > void f(int x) # int* will cause syntax errors, it must be plain int > > cdef int x = 3 > y = 3 # object > f(x) # x changes to 10 > f(y) # the temporary result of conversion is changed -- y is not changed! > > So the current state is both inobvious and would lead to hard to find > bugs IMO. So that's why I argue explicit support is needed in some form > or another. > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at gmail.com Thu Jul 8 02:35:22 2010 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 7 Jul 2010 17:35:22 -0700 Subject: [Cython] Testing import/cimport and tests spanning nested directories In-Reply-To: <4C342F55.2020500@student.matnat.uio.no> References: <4C342F55.2020500@student.matnat.uio.no> Message-ID: On Wed, Jul 7, 2010 at 12:40 AM, Dag Sverre Seljebotn wrote: > Craig Citro wrote: >> Hi all, >> >> So I'm currently trying to write some tests for a subtle issue >> involving resolving imports and cimports. (I think I emailed about >> this a while back.) The particular issue isn't so important right this >> minute -- the only thing that matters is that to test it, I need a >> test with a few layers of directories, with __init__.py files >> sprinkled in several places, and potentially *not* at the root. >> Unfortunately, this doesn't work so well with the current testing >> setup -- at least, as I understand, there's no way to tell it "when >> you try to run this test, also copy along this particular chunk of the >> current directory." So there are two obvious choices: >> >> 1) Give a test a way of specifying what other files should come along >> as part of it. >> 2) Create a directory called nested (or something else) somewhere in >> the tests/ tree (maybe one for each of the current subdirectories?), >> and add code to runtests.py to handle that whole tree at once. >> >> Each of these has its ups and downs -- but I'd love it if someone told >> me that the testing code already did what I'm looking for, and I just >> didn't know it. ;) I think (1) feels like the "classier" solution, but >> (2) would make it easy for someone to *add* a nested test without >> having to create a whole bunch of files/directories themselves. That >> said, we'd have to be careful that no one accidentally broke a test by >> modifying it -- tests really should be nearly immutable. >> >> Any thoughts? >> > If this ends up being coded from scratch, I think it would make testing > easier if one could actually write down all the files in a single > testcase bundle file (e.g. with a "test" suffix). Something like the > following, which is simply split up into seperate files by runtests.py: > > == mydir/__init__.py > # This file is empty > == mydir/foo.pyx > cdef int x = 3 > == mydir/foo.pxd > cdef int x > > This is because such tests would tend to have a lot of very small files > testing specific functionality, and it would be very tiring to browse > around such trees and open all the files to see what is going on. I think that's an excellent idea. - Robert From baihaoyu at gmail.com Thu Jul 8 16:40:17 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Thu, 8 Jul 2010 22:40:17 +0800 Subject: [Cython] cpdef (unbound) method inheritance broken In-Reply-To: <4C34B3F4.9080204@behnel.de> References: <4C34B3F4.9080204@behnel.de> Message-ID: On Thu, Jul 8, 2010 at 1:05 AM, Stefan Behnel wrote: > Lisandro Dalcin, 10.06.2010 16:50: >> Please try the attached test case. If you switch the comment in the >> first two lines, you will get the failure. So far, I could not figure >> out what's going on. > > Your code is basically this: > > --------------------- > cdef class Foo2: > ? ? cpdef hello(self): > ? ? ? ? print 'Foo.hello()' > > cdef class Bar2(Foo2): > ? ? cpdef hello(self): > ? ? ? ? print 'Bar.hello()' > > Foo.hello(b) > --------------------- > > The problem is that this prints "Bar.hello()", not "Foo.hello()" as one > would expect (and which is how CPython behaves). > > gdb gives me this for C function call in the "Foo2.hallo" method wrapper: > > 601 ? ? ? __pyx_t_1 = ((struct __pyx_vtabstruct_11methinherit_Foo2 > *)((struct __pyx_obj_11methinherit_Foo2 > *)__pyx_v_self)->__pyx_vtab)->hello(((struct __pyx_obj_11methinherit_Foo2 > *)__pyx_v_self), 1); if (unlikely(!__pyx_t_1)) {__pyx_filename = > __pyx_f[0]; __pyx_lineno = 35; __pyx_clineno = __LINE__; goto __pyx_L1_error;} > (gdb) print __pyx_v_self > $1 = > (gdb) pyo __pyx_v_self > object ?: > type ? ?: methinherit.Bar2 > refcount: 3 > address : 0x7ffff7ee40f0 > $2 = void > (gdb) print ((struct __pyx_obj_11methinherit_Foo2 *)__pyx_v_self)->__pyx_vtab > $3 = (struct __pyx_vtabstruct_11methinherit_Foo2 *) 0x7ffff69ba2f0 > (gdb) print ((struct __pyx_obj_11methinherit_Foo2 > *)__pyx_v_self)->__pyx_vtab->hello > $4 = (PyObject *(*)(struct __pyx_obj_11methinherit_Foo2 *, int)) > 0x7ffff67b6492 <__pyx_f_11methinherit_4Bar2_hello> > > Note the "__pyx_f_11methinherit_4Bar2_hello" function in the last line. > > So, the problem is that the cpdef wrapper effectively calls "self.hello", > which is "Bar.hello" in this case. This is the wrong thing to do for an > unbound method. However, I'm not sure how to fix this, either. > > Any ideas? > > Stefan How about implement a tp_descr_get to dispatch the vtable instead of dispatching it inside the method? Then Bar.hello will return the correct unbounded method and self.hello will call the descriptor and return the bounded method. -- Haoyu BAI School of Computing, National University of Singapore. From tokidokix at gmail.com Fri Jul 9 15:09:14 2010 From: tokidokix at gmail.com (toki doki) Date: Fri, 9 Jul 2010 22:09:14 +0900 Subject: [Cython] About the cython/C++ doc In-Reply-To: References: <4C3185C8.7090800@behnel.de> <4C31BB28.2020105@behnel.de> Message-ID: > We can probably change the wiki right now, making a <= 0.12 page with > the old instructions and link to the new stuff, which will get > replaced by a link to the docs once those go live. I would have done this, but I cannot login on the wiki. I seems I was added on Trac, but not on the wiki... From tokidokix at gmail.com Fri Jul 9 15:42:59 2010 From: tokidokix at gmail.com (toki doki) Date: Fri, 9 Jul 2010 22:42:59 +0900 Subject: [Cython] Python bool/C++ bool : Confusing behaviour Message-ID: I have been bitten by this twice this week, so I thought I would tell about it: Contrary to C, C++ has a boolean type, using the same keyword as Python: "bool". When declaring a C++ function with a (C++) bool argument, like for example " int myFunc(int val, bool exact)", it seems logical (although it is not, in insight) to declare it that way: cdef extern from "xxx.hpp": int myFunc(int,bool) However, cython will understand that "bool" declare a python object of (python) type bool. Thus, when doing "myFunc(5,False)", myFunc receive as argument a pointer to the python object "False". And here is the catch: gcc will implicitly convert this pointer to a C++ "True" value. After checking, this implicit conversion seems to be part of the C++ standard. Therefore, the problem is as follow: Cython and gcc will emit no warning when a C++ bool is declared by "bool" in cython. But when a python "False" is given as an argument of a C++ function typed by "bool", the C++ function will receive a C++ "true". Of course, the right way to do it is to declare the C++ bool as a "bint". But since the other C/C++ types can be declared by their normal C names, I think many people will assume that "bool" is the right keyword here. And since Cython and g++ do not give any warning about it, this might lead to some slightly puzzling bugs in users code. I am not sure you can do something about it (maybe handle it the same way that you handle C int and Python int ?). But if you cannot, it deserves at least a warning in the docs. (and for the record, although I have been posting quite a lot this week about bugs and limitations of the c++ mode of cython, I am overall delighted with it; it works much better and has much more features than I was expecting when I started using it last week :-) cheers, Toki From robertwb at math.washington.edu Fri Jul 9 17:30:29 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 9 Jul 2010 08:30:29 -0700 Subject: [Cython] About the cython/C++ doc In-Reply-To: References: <4C3185C8.7090800@behnel.de> <4C31BB28.2020105@behnel.de> Message-ID: On Fri, Jul 9, 2010 at 6:09 AM, toki doki wrote: >> We can probably change the wiki right now, making a <= 0.12 page with >> the old instructions and link to the new stuff, which will get >> replaced by a link to the docs once those go live. > > I would have done this, but I cannot login on the wiki. I seems I was > added on Trac, but not on the wiki... The wiki accounts are self-service, as we've got a capacha. - Robert From robertwb at math.washington.edu Fri Jul 9 17:29:22 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 9 Jul 2010 08:29:22 -0700 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: References: Message-ID: On Fri, Jul 9, 2010 at 6:42 AM, toki doki wrote: > I have been bitten by this twice this week, so I thought I would tell about it: > > Contrary to C, C++ has a boolean type, using the same keyword as > Python: "bool". When declaring a C++ function with a (C++) bool > argument, like for example " int myFunc(int val, bool exact)", it > seems logical (although it is not, in insight) to declare it that way: > > cdef extern from "xxx.hpp": > ? ? int myFunc(int,bool) > > However, cython will understand that "bool" declare a python object of > (python) type bool. Thus, when doing "myFunc(5,False)", myFunc receive > as argument a pointer to the python object "False". And here is the > catch: gcc will implicitly convert this pointer to a C++ ?"True" > value. After checking, this implicit conversion seems to be part of > the C++ standard. > > Therefore, the problem is as follow: > Cython and gcc will emit no warning when a C++ bool is declared by > "bool" in cython. But ?when a python "False" is given as an argument > of a C++ function typed by "bool", the C++ function will receive a C++ > "true". > > Of course, the right way to do it is to declare the C++ bool ?as a > "bint". But since the other C/C++ types can be declared by their > normal C names, I think many people will assume that "bool" is the > right keyword here. And since Cython and g++ do not give any warning > about it, this might lead to some slightly puzzling bugs in users > code. Hmm... yes, this is particularly bad, as it takes the pointer which should never be NULL (= False) > I am not sure you can do something about it ?(maybe handle it the same > way that you handle C int and Python int ?). But if you cannot, it > deserves at least a warning in the docs. bint and bool are semantically different types, and to introduce a bool type we would have to either disallow or emulate it in non-C++ code. (This is why I called it bint rather than bool in the first case.) For the moment, though, I'm wondering why we even bother letting people declare the type as bool--it's not a very useful (Python) type to statically declare, but can cause a lot of confusion. > (and for the record, although I have been posting quite a lot this > week about bugs and limitations of the c++ mode of cython, I am > overall delighted with it; it works much better and has much more > features than I was expecting when I started using it last week :-) Glad to hear. And we love bug reports, it helps us make the product better. - Robert From wstein at gmail.com Sat Jul 10 09:26:59 2010 From: wstein at gmail.com (William Stein) Date: Sat, 10 Jul 2010 09:26:59 +0200 Subject: [Cython] cython Message-ID: Hi, Langtangen is saying very, very nice things about Cython during his Eyroscipy keynote right now... William -- William Stein Professor of Mathematics University of Washington http://wstein.org From wstein at gmail.com Sat Jul 10 15:25:18 2010 From: wstein at gmail.com (William Stein) Date: Sat, 10 Jul 2010 15:25:18 +0200 Subject: [Cython] yet another talk... which surprisingly is about cython (!) Message-ID: Hi, Here is yet another talk at EuroScipy, which turns out to be about Cython. Check out the abstract: http://www.euroscipy.org/talk/2294 -- William Stein Professor of Mathematics University of Washington http://wstein.org From stefan_ml at behnel.de Sat Jul 10 21:11:31 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 10 Jul 2010 21:11:31 +0200 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: References: Message-ID: <4C38C5E3.1030605@behnel.de> Robert Bradshaw, 09.07.2010 17:29: > For the moment, though, I'm wondering why we even bother letting > people declare the type as bool--it's not a very useful (Python) type > to statically declare, but can cause a lot of confusion. Agreed, use cases are hard to come up with. However, given that it's there, I'd prefer keeping the C++ type in a separate namespace for now and maybe(!) improve Cython warnings to find common traps when compiling in C++ mode. Stefan From dagss at student.matnat.uio.no Sat Jul 10 23:45:52 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 10 Jul 2010 23:45:52 +0200 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C38C5E3.1030605@behnel.de> References: <4C38C5E3.1030605@behnel.de> Message-ID: <4C38EA10.4030009@student.matnat.uio.no> On 07/10/2010 09:11 PM, Stefan Behnel wrote: > Robert Bradshaw, 09.07.2010 17:29: > >> For the moment, though, I'm wondering why we even bother letting >> people declare the type as bool--it's not a very useful (Python) type >> to statically declare, but can cause a lot of confusion. >> > Agreed, use cases are hard to come up with. However, given that it's there, > I'd prefer keeping the C++ type in a separate namespace for now and > maybe(!) improve Cython warnings to find common traps when compiling in C++ > mode. > But int and float has the exact same problem! And there we inherited C semantics from Pyrex... Therefore C++ bool makes sense IMO, to be consistent with int and float (but I'd like to see a new solution that doesn't confuse Python programmers at some point, both for bool, int and float). Dag Sverre From robertwb at math.washington.edu Sun Jul 11 00:00:38 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 10 Jul 2010 15:00:38 -0700 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C38EA10.4030009@student.matnat.uio.no> References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> Message-ID: On Sat, Jul 10, 2010 at 2:45 PM, Dag Sverre Seljebotn wrote: > On 07/10/2010 09:11 PM, Stefan Behnel wrote: >> Robert Bradshaw, 09.07.2010 17:29: >> >>> For the moment, though, I'm wondering why we even bother letting >>> people declare the type as bool--it's not a very useful (Python) type >>> to statically declare, but can cause a lot of confusion. >>> >> Agreed, use cases are hard to come up with. However, given that it's there, >> I'd prefer keeping the C++ type in a separate namespace for now and >> maybe(!) improve Cython warnings to find common traps when compiling in C++ >> mode. >> > > But int and float has the exact same problem! And there we inherited C > semantics from Pyrex... And, for Py2, long has a double meaning as well. There is very little use (currently) for declaring things to be Python ints, longs, floats, bools. > Therefore C++ bool makes sense IMO, to be consistent with int and float Yep. The main problem that I see is that most of the usecases for bool will be wrong, resulting in compiling, running code that does the wrong thing. I'd rather not have the type at all. > (but I'd like to see a new solution that doesn't confuse Python > programmers at some point, both for bool, int and float). Sure. - Robert From stefan_ml at behnel.de Sun Jul 11 08:18:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 Jul 2010 08:18:56 +0200 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> Message-ID: <4C396250.1020403@behnel.de> Robert Bradshaw, 11.07.2010 00:00: > On Sat, Jul 10, 2010 at 2:45 PM, Dag Sverre Seljebotn wrote: >> On 07/10/2010 09:11 PM, Stefan Behnel wrote: >>> Robert Bradshaw, 09.07.2010 17:29: >>> >>>> For the moment, though, I'm wondering why we even bother letting >>>> people declare the type as bool--it's not a very useful (Python) type >>>> to statically declare, but can cause a lot of confusion. >>>> >>> Agreed, use cases are hard to come up with. However, given that it's there, >>> I'd prefer keeping the C++ type in a separate namespace for now and >>> maybe(!) improve Cython warnings to find common traps when compiling in C++ >>> mode. >>> >> >> But int and float has the exact same problem! And there we inherited C >> semantics from Pyrex... > > And, for Py2, long has a double meaning as well. There is very little > use (currently) for declaring things to be Python ints, longs, floats, > bools. Well, 'long' is exactly the reason: Since there are 'int' and 'long' in Py2, which are completely interchangeable, it's impossible to type a variable as Python 'int' or 'long' without restricting your code too much. If you type it as 'int', a C int will behave much better as it accepts both Python integer types. If there was only Py3 (with only one Python 'int' type), this problem would have occurred much more often (and I predict that it will become more apparent when more people start developing Py3 code). 'float' is different here as there is no redundant type in Python, but since 'float' is equivalent to C double, there's no reason not to type your variable as a C double directly. Also, it'll actually work to declare your variable as 'float' if you only pass Python floats. But it won't necessarily do exactly the right thing. In general, numeric types are much less of a problem as they have well defined coercion semantics. The boolean types of C, C++ and Python do not. >> Therefore C++ bool makes sense IMO, to be consistent with int and float > > Yep. The main problem that I see is that most of the usecases for bool > will be wrong, resulting in compiling, running code that does the > wrong thing. I'd rather not have the type at all. Actually, now that I think some more about it - Cython 0.13 moves many Python types into C space automatically, so we may consider *always* mapping 'bool' to the native boolean type in the target language. Python's 'bool' type is compatible with integers anyway, so we could map 'bool' to a 0/1 restricted bint in C and bool in C++ (assuming appropriate coercion semantics for C++/Python). Ok, the bool type in C++ isn't compatible with integers, but I think that's only a minor issue. The only case where you'd really see this is when you try to add a 'bool' to an integer. This has no impact on Python compilation or pure Python mode, and it's not hard to explain to someone who wraps C++ that 'bool' is really C++ bool. We could still make all three types available through separate cimport namespaces. Stefan From stefan_ml at behnel.de Sun Jul 11 10:57:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 Jul 2010 10:57:16 +0200 Subject: [Cython] cpdef (unbound) method inheritance broken In-Reply-To: References: <4C34B3F4.9080204@behnel.de> Message-ID: <4C39876C.8090909@behnel.de> Haoyu Bai, 08.07.2010 16:40: > On Thu, Jul 8, 2010 at 1:05 AM, Stefan Behnel wrote: >> --------------------- >> cdef class Foo2: >> cpdef hello(self): >> print 'Foo.hello()' >> >> cdef class Bar2(Foo2): >> cpdef hello(self): >> print 'Bar.hello()' >> >> Foo.hello(b) >> --------------------- >> >> The problem is that this prints "Bar.hello()", not "Foo.hello()" as one >> would expect (and which is how CPython behaves). >> [...] >> So, the problem is that the cpdef wrapper effectively calls "self.hello", >> which is "Bar.hello" in this case. This is the wrong thing to do for an >> unbound method. However, I'm not sure how to fix this, either. > > How about implement a tp_descr_get to dispatch the vtable instead of > dispatching it inside the method? Then Bar.hello will return the > correct unbounded method and self.hello will call the descriptor and > return the bounded method. Interesting. So we'd basically implement our own method lookup descriptor? You already have to return a separate instance for each lookup to inject the reference to 'self', so you could move the vtable lookup overhead into the method lookup rather than into the call. Might be worth a try. It also sounds like a lot of overhead, though... Stefan From greg.ewing at canterbury.ac.nz Mon Jul 12 02:30:00 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Jul 2010 12:30:00 +1200 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C396250.1020403@behnel.de> References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> <4C396250.1020403@behnel.de> Message-ID: <4C3A6208.9000105@canterbury.ac.nz> Stefan Behnel wrote: > Well, 'long' is exactly the reason: Since there are 'int' and 'long' in > Py2, which are completely interchangeable, it's impossible to type a > variable as Python 'int' or 'long' without restricting your code too much. > ... If there was only Py3 (with only one Python 'int' > type), this problem would have occurred much more often (and I predict that > it will become more apparent when more people start developing Py3 code). What use cases do you think there will be for typing something as a Python int, even in Py3? I'm having trouble thinking of any. -- Greg From stefan_ml at behnel.de Mon Jul 12 08:01:10 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 12 Jul 2010 08:01:10 +0200 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C3A6208.9000105@canterbury.ac.nz> References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> <4C396250.1020403@behnel.de> <4C3A6208.9000105@canterbury.ac.nz> Message-ID: <4C3AAFA6.2080702@behnel.de> Greg Ewing, 12.07.2010 02:30: > Stefan Behnel wrote: >> Well, 'long' is exactly the reason: Since there are 'int' and 'long' in >> Py2, which are completely interchangeable, it's impossible to type a >> variable as Python 'int' or 'long' without restricting your code too much. > > ... If there was only Py3 (with only one Python 'int' >> type), this problem would have occurred much more often (and I predict that >> it will become more apparent when more people start developing Py3 code). > > What use cases do you think there will be for typing something > as a Python int, even in Py3? I'm having trouble thinking of > any. My point was more why people currently don't run into it and why I think there is a difference to a boolean type. I didn't mean to say that not being able to type something as Python int is a real world problem that we need to work on, just that people would run into it more easily in a Py3 world than in code targeting Py2, where it's clearly counterproductive. Stefan From craigcitro at gmail.com Mon Jul 12 08:50:11 2010 From: craigcitro at gmail.com (Craig Citro) Date: Sun, 11 Jul 2010 23:50:11 -0700 Subject: [Cython] Type inference and memory management Message-ID: Hi all, I ran into an interesting question while getting Sage to build and pass tests against the current cython-devel tip (side note: it does on both my laptop and sage.math!), and I thought I'd get some opinions on the "right" way to fix it. Consider the following bit of code: def __repr__(self): cdef char* ss = fmpz_poly_to_string(self.poly) s = ss free(ss) return s You don't need to know what an fmpz_poly is -- the relevant fact is that it's a call into some C library, and it returns a string representation in the form of a char *. This code then does what you expect: gets the value from the C library and returns it. The key is that it's using the fact that variables implicitly default to Python objects here -- the "s = ss" line will cause the underlying char * to be copied into a variable whose memory is managed by Python. I think this is considered the "Cythonic" way of doing things -- at least, there are entries in the FAQ suggesting that this is a good idea. Now enter type inference. It looks at this block and says, "hey, s is only ever assigned to a char * -- let's call it a char *, too." Of course, this is a disaster -- it changes the semantics of the all-important "s = ss" line. As a result, the return value is (a Python copy of) some random junk. This is easy enough to fix -- we can be more explicit about our intentions, and declare s to be an object, which works great. However, this is likely to break at least some user code in the wild -- especially since we've been recommending this as the "right" way to do things. I can see at least a few options: 1) Break the code above, tell people to explicitly declare things to be objects. 2) Decide that if a variable gets returned by a function which is either a def'd function or returns a Python object, and we don't have an explicit type declaration already, then we only infer something which is a subtype of Python object. (Right now, we almost never infer anything more specific than Python object anyway.) 3) Something else in between, i.e. make some decisions based on the type that gets inferred. Or, better yet, do some control-flow analysis/alias analysis and start shadowing the variable with a Python object only once you need to. (Clearly this is a pipe dream right now.) I'm personally leaning towards (2) with the hopes of one day getting closer to (3). I've got a patch written that does (2), which I'm happy to push if people agree on that one. I think the tradeoff between (1) and (2) is fairly standard -- (1) is going to generate (potentially) faster code, but (2) is going to be much friendlier for someone migrating Python code. Thoughts? -cc From dagss at student.matnat.uio.no Mon Jul 12 09:08:19 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 12 Jul 2010 09:08:19 +0200 Subject: [Cython] Type inference and memory management In-Reply-To: References: Message-ID: <4C3ABF63.3030208@student.matnat.uio.no> On 07/12/2010 08:50 AM, Craig Citro wrote: > Hi all, > > I ran into an interesting question while getting Sage to build and > pass tests against the current cython-devel tip (side note: it does on > both my laptop and sage.math!), and I thought I'd get some opinions on > the "right" way to fix it. > > Consider the following bit of code: > > def __repr__(self): > cdef char* ss = fmpz_poly_to_string(self.poly) > s = ss > free(ss) > return s > > You don't need to know what an fmpz_poly is -- the relevant fact is > that it's a call into some C library, and it returns a string > representation in the form of a char *. This code then does what you > expect: gets the value from the C library and returns it. The key is > that it's using the fact that variables implicitly default to Python > objects here -- the "s = ss" line will cause the underlying char * to > be copied into a variable whose memory is managed by Python. I think > this is considered the "Cythonic" way of doing things -- at least, > there are entries in the FAQ suggesting that this is a good idea. > > Now enter type inference. It looks at this block and says, "hey, s is > only ever assigned to a char * -- let's call it a char *, too." Of > course, this is a disaster -- it changes the semantics of the > all-important "s = ss" line. As a result, the return value is (a > Python copy of) some random junk. This is easy enough to fix -- we can > be more explicit about our intentions, and declare s to be an object, > which works great. However, this is likely to break at least some user > code in the wild -- especially since we've been recommending this as > the "right" way to do things. > > I can see at least a few options: > > 1) Break the code above, tell people to explicitly declare things to be objects. > > 2) Decide that if a variable gets returned by a function which is > either a def'd function or returns a Python object, and we don't have > an explicit type declaration already, then we only infer something > which is a subtype of Python object. (Right now, we almost never infer > anything more specific than Python object anyway.) > +1 if it is sufficient. Does this cover all cases you can think of? What happens with s = ss return s + 'hello' and so on? Dag Sverre From stefan_ml at behnel.de Mon Jul 12 10:10:07 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 12 Jul 2010 10:10:07 +0200 Subject: [Cython] Type inference and memory management In-Reply-To: References: Message-ID: <4C3ACDDF.2030506@behnel.de> Craig Citro, 12.07.2010 08:50: > I ran into an interesting question while getting Sage to build and > pass tests against the current cython-devel tip (side note: it does on > both my laptop and sage.math!) Ah, finally! Nice! > def __repr__(self): > cdef char* ss = fmpz_poly_to_string(self.poly) > s = ss > free(ss) > return s > Now enter type inference. It looks at this block and says, "hey, s is > only ever assigned to a char * -- let's call it a char *, too." Of > course, this is a disaster -- it changes the semantics of the > all-important "s = ss" line. As a result, the return value is (a > Python copy of) some random junk. This is easy enough to fix -- we can > be more explicit about our intentions, and declare s to be an object, > which works great. However, this is likely to break at least some user > code in the wild -- especially since we've been recommending this as > the "right" way to do things. Yes, Robert and I were aware of this at the time. The problem is that the above code really *is* broken because it relies on Cython not being smart enough to understand what's going on. Code like this needs fixing as early as possible, because Cython is only ever going to get smarter and this *will* have to break at some point. So we thought it would be better to break it early, rather than carrying along some kind of backwards compatible quirk to keep this 'working' (which can't be done in most cases anyway, without disabling most of the current type inference). Any kind of rule that we add here would make things less predictable and less understandable. > 1) Break the code above, tell people to explicitly declare things to be objects. Or use a cast, yes. That's my favourite. > 2) Decide that if a variable gets returned by a function which is > either a def'd function or returns a Python object, and we don't have > an explicit type declaration already, then we only infer something > which is a subtype of Python object. That would be an insufficient and rather hard to explain rule. It basically says: if Cython happens to be smart enough to figure out that the value will end up being returned from the function, the code will behave different than in the case that it fails to trace the value propagation. IMHO, much worse than the current behaviour. > (Right now, we almost never infer > anything more specific than Python object anyway.) Well, yes, "almost". It's already false in some important cases and those cases are very likely to grow in the future. > (1) is going to generate (potentially) > faster code, but (2) is going to be much friendlier for someone > migrating Python code. No. Someone who migrates Python code will not start from code that relies on the above anyway. Remember that this only happens with pointers, so pure Python code is not affected. Stefan From greg.ewing at canterbury.ac.nz Mon Jul 12 10:34:15 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Jul 2010 20:34:15 +1200 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C3AAFA6.2080702@behnel.de> References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> <4C396250.1020403@behnel.de> <4C3A6208.9000105@canterbury.ac.nz> <4C3AAFA6.2080702@behnel.de> Message-ID: <4C3AD387.3060002@canterbury.ac.nz> Stefan Behnel wrote: > My point was more why people currently don't run into it and why I think > there is a difference to a boolean type. Well, it seems to me the reason people don't run into it is just as likely to be simply that it never occurs to anyone to try to type something as a Python int in the first place. And, for the same reasons, they're not likely to want to type anything as a Python bool either. -- Greg From stefan_ml at behnel.de Mon Jul 12 10:47:02 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 12 Jul 2010 10:47:02 +0200 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C3AD387.3060002@canterbury.ac.nz> References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> <4C396250.1020403@behnel.de> <4C3A6208.9000105@canterbury.ac.nz> <4C3AAFA6.2080702@behnel.de> <4C3AD387.3060002@canterbury.ac.nz> Message-ID: <4C3AD686.1050905@behnel.de> Greg Ewing, 12.07.2010 10:34: > Stefan Behnel wrote: > >> My point was more why people currently don't run into it and why I think >> there is a difference to a boolean type. > > Well, it seems to me the reason people don't run into it > is just as likely to be simply that it never occurs to > anyone to try to type something as a Python int in the > first place. And, for the same reasons, they're not > likely to want to type anything as a Python bool either. Ok, then we're back to this: """ we may consider *always* mapping 'bool' to the native boolean type in the target language. Python's 'bool' type is compatible with integers anyway, so we could map 'bool' to a 0/1 restricted bint in C and bool in C++ (assuming appropriate coercion semantics for C++/Python). """ Stefan From dalcinl at gmail.com Mon Jul 12 16:32:42 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 12 Jul 2010 11:32:42 -0300 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C3AD686.1050905@behnel.de> References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> <4C396250.1020403@behnel.de> <4C3A6208.9000105@canterbury.ac.nz> <4C3AAFA6.2080702@behnel.de> <4C3AD387.3060002@canterbury.ac.nz> <4C3AD686.1050905@behnel.de> Message-ID: On 12 July 2010 05:47, Stefan Behnel wrote: > Greg Ewing, 12.07.2010 10:34: >> Stefan Behnel wrote: >> >>> My point was more why people currently don't run into it and why I think >>> there is a difference to a boolean type. >> >> Well, it seems to me the reason people don't run into it >> is just as likely to be simply that it never occurs to >> anyone to try to type something as a Python int in the >> first place. And, for the same reasons, they're not >> likely to want to type anything as a Python bool either. > > Ok, then we're back to this: > > """ > we may consider *always* > mapping 'bool' to the native boolean type in the target language. Python's > 'bool' type is compatible with integers anyway, so we could map 'bool' to > a 0/1 restricted bint in C and bool in C++ (assuming appropriate coercion > semantics for C++/Python). > """ > What about C99 _Bool ? I think we are going to need a #define for Cython's C-side bool. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Mon Jul 12 17:49:31 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 12 Jul 2010 08:49:31 -0700 Subject: [Cython] Python bool/C++ bool : Confusing behaviour In-Reply-To: <4C396250.1020403@behnel.de> References: <4C38C5E3.1030605@behnel.de> <4C38EA10.4030009@student.matnat.uio.no> <4C396250.1020403@behnel.de> Message-ID: On Sat, Jul 10, 2010 at 11:18 PM, Stefan Behnel wrote: > Robert Bradshaw, 11.07.2010 00:00: >> On Sat, Jul 10, 2010 at 2:45 PM, Dag Sverre Seljebotn wrote: >>> On 07/10/2010 09:11 PM, Stefan Behnel wrote: >>>> Robert Bradshaw, 09.07.2010 17:29: >>>> >>>>> For the moment, though, I'm wondering why we even bother letting >>>>> people declare the type as bool--it's not a very useful (Python) type >>>>> to statically declare, but can cause a lot of confusion. >>>>> >>>> Agreed, use cases are hard to come up with. However, given that it's there, >>>> I'd prefer keeping the C++ type in a separate namespace for now and >>>> maybe(!) improve Cython warnings to find common traps when compiling in C++ >>>> mode. >>>> >>> >>> But int and float has the exact same problem! And there we inherited C >>> semantics from Pyrex... >> >> And, for Py2, long has a double meaning as well. There is very little >> use (currently) for declaring things to be Python ints, longs, floats, >> bools. > > Well, 'long' is exactly the reason: Since there are 'int' and 'long' in > Py2, which are completely interchangeable, it's impossible to type a > variable as Python 'int' or 'long' without restricting your code too much. > If you type it as 'int', a C int will behave much better as it accepts both > Python integer types. If there was only Py3 (with only one Python 'int' > type), this problem would have occurred much more often (and I predict that > it will become more apparent when more people start developing Py3 code). > > 'float' is different here as there is no redundant type in Python, but > since 'float' is equivalent to C double, there's no reason not to type your > variable as a C double directly. Also, it'll actually work to declare your > variable as 'float' if you only pass Python floats. But it won't > necessarily do exactly the right thing. > > In general, numeric types are much less of a problem as they have well > defined coercion semantics. The boolean types of C, C++ and Python do not. > > >>> Therefore C++ bool makes sense IMO, to be consistent with int and float >> >> Yep. The main problem that I see is that most of the usecases for bool >> will be wrong, resulting in compiling, running code that does the >> wrong thing. I'd rather not have the type at all. > > Actually, now that I think some more about it - Cython 0.13 moves many > Python types into C space automatically, so we may consider *always* > mapping 'bool' to the native boolean type in the target language. Python's > 'bool' type is compatible with integers anyway, so we could map 'bool' to a > 0/1 restricted bint in C and bool in C++ (assuming appropriate coercion > semantics for C++/Python). > > Ok, the bool type in C++ isn't compatible with integers, but I think that's > only a minor issue. The only case where you'd really see this is when you > try to add a 'bool' to an integer. This has no impact on Python compilation > or pure Python mode, and it's not hard to explain to someone who wraps C++ > that 'bool' is really C++ bool. Yes, that's what I'm thinking. In the immediate future, should we remove/deprecate the bool as a Python object type? - Robert From robertwb at math.washington.edu Mon Jul 12 17:57:47 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 12 Jul 2010 08:57:47 -0700 Subject: [Cython] Type inference and memory management In-Reply-To: <4C3ACDDF.2030506@behnel.de> References: <4C3ACDDF.2030506@behnel.de> Message-ID: On Mon, Jul 12, 2010 at 1:10 AM, Stefan Behnel wrote: > Craig Citro, 12.07.2010 08:50: >> I ran into an interesting question while getting Sage to build and >> pass tests against the current cython-devel tip (side note: it does on >> both my laptop and sage.math!) > > Ah, finally! Nice! > > >> ? ? ?def __repr__(self): >> ? ? ? ? ?cdef char* ss = fmpz_poly_to_string(self.poly) >> ? ? ? ? ?s = ss >> ? ? ? ? ?free(ss) >> ? ? ? ? ?return s > >> Now enter type inference. It looks at this block and says, "hey, s is >> only ever assigned to a char * -- let's call it a char *, too." Of >> course, this is a disaster -- it changes the semantics of the >> all-important "s = ss" line. As a result, the return value is (a >> Python copy of) some random junk. This is easy enough to fix -- we can >> be more explicit about our intentions, and declare s to be an object, >> which works great. However, this is likely to break at least some user >> code in the wild -- especially since we've been recommending this as >> the "right" way to do things. > > Yes, Robert and I were aware of this at the time. The problem is that the > above code really *is* broken because it relies on Cython not being smart > enough to understand what's going on. Code like this needs fixing as early > as possible, because Cython is only ever going to get smarter and this > *will* have to break at some point. So we thought it would be better to > break it early, rather than carrying along some kind of backwards > compatible quirk to keep this 'working' (which can't be done in most cases > anyway, without disabling most of the current type inference). Any kind of > rule that we add here would make things less predictable and less > understandable. > > >> 1) Break the code above, tell people to explicitly declare things to be objects. > > Or use a cast, yes. That's my favourite. +1 Either cdef object s = ss or s = ss >> 2) Decide that if a variable gets returned by a function which is >> either a def'd function or returns a Python object, and we don't have >> an explicit type declaration already, then we only infer something >> which is a subtype of Python object. > > That would be an insufficient and rather hard to explain rule. It basically > says: if Cython happens to be smart enough to figure out that the value > will end up being returned from the function, the code will behave > different than in the case that it fails to trace the value propagation. > IMHO, much worse than the current behaviour. I fully agree, this is way to messy and we would have to keep on adding cases or deal with the fact that it doesn't cover near what we'd want it too. It makes type inference way less local as well. >> (Right now, we almost never infer >> anything more specific than Python object anyway.) > > Well, yes, "almost". It's already false in some important cases and those > cases are very likely to grow in the future. > > >> (1) is going to generate (potentially) >> faster code, but (2) is going to be much friendlier for someone >> migrating Python code. > > No. Someone who migrates Python code will not start from code that relies > on the above anyway. Remember that this only happens with pointers, so pure > Python code is not affected. Yep. The only people that write cdef char* ss = ... s = ss to implicitly convert to a Python object are people who already really know Cython well (which, admittedly, does cover a lot of people who've had a hand in writing Sage). - Robert From dagss at student.matnat.uio.no Mon Jul 12 19:52:54 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 12 Jul 2010 19:52:54 +0200 Subject: [Cython] Impressions from EuroSciPy 2010 Message-ID: <4C3B5676.3060101@student.matnat.uio.no> Right now I'm in Charles de Gaulle, about to fly home after EuroScipy 2010. I just wanted to post the main impression from the conference: Cython appears now to be a very mainstream tool in scientific Python computing. At some point it felt like every other talk mentioned Cython favourably, and Cython was prominent in both keynotes. (It was also nice that Konrad Hinsen made sure to attribute Pyrex as well.) I don't mean to gloat, but...well...it's hard not to share this: At one point a happy boost::python user got up and asked if nobody had heard about it, since he heard nothing about it. He asked everybody who had heard about it to raise their hand (very many) and then everybody who used it and where happy with it to raise their hand (and that was only a few). (Of course, he didn't ask about how many were using Cython. Lots of people have only heard about Cython and are not using it. Still interesting.) Apart from that, there's was lots of interesting people and talks :-) Dag Sverre From jsalvati at u.washington.edu Mon Jul 12 23:03:18 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Mon, 12 Jul 2010 14:03:18 -0700 Subject: [Cython] Generic Loops for numpy UFuncs in Cython Message-ID: Hello, I'd like to write some UFuncs in Cython (using http://wiki.cython.org/MarkLodato/CreatingUfuncs) , but I have struggled with writing a Generic Loop so I can write UFuncs with types other than those supplied by numpy. Does anyone have an example they could post? Or give me a hint at what I am doing incorrectly? The numpy docs give the following as an example of a valid loop, and I believe that for Generic Loops, the actual function is passed in as value in *extra: static void double_add(char *args, npy_intp *dimensions, npy_intp *steps, void *extra) { npy_intp i; npy_intp is1=steps[0], is2=steps[1]; npy_intp os=steps[2], n=dimensions[0]; char *i1=args[0], *i2=args[1], *op=args[2]; for (i=0; iop) = *( extra[0]) ( *( i1) ) i1 += is1 op += os -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100712/f5ac9476/attachment.htm From dalcinl at gmail.com Tue Jul 13 01:02:33 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 12 Jul 2010 20:02:33 -0300 Subject: [Cython] Py_buffer: setting view.obj=self in __getbuffer__ Message-ID: Disclaimer: For PEP 3118, the differences in docs vs. implementation plus the differences in 2.6/2.7/3.x are a big mess. That being said, perhaps my following comments are wrong. Additionally, I guess Dag is the only one that can make some comment on this without wasting a lot of valuable time diving in CPython's source code. After an enjoyable time debugging crashes and reference leaks, I think Cython should automatically set "view.obj = self" in __getbuffer__ special method. If this is not done, references can be easily leaked by creating a memoryview object (from Python land, our using PyMemoryView_FromObject), just look at Py2.7 Objects/memoryobject.c, function memory_dealloc, you will notice that iff view->obj==NULL, them self->base is never decrefed! Perhaps I'm missing something, but this really smells like a CPython bug. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From Dave.Cross at cdl.co.uk Tue Jul 13 12:26:05 2010 From: Dave.Cross at cdl.co.uk (Dave Cross) Date: Tue, 13 Jul 2010 11:26:05 +0100 Subject: [Cython] bug #520: scanner broken for DOS line endings Message-ID: Robert Bradshaw writes: > > ... it seems it > isn't a big issue (though should still be fixed). > I assure you that it is a big issue for me. I'm a relatively new pythonista and I've sold my company on a big development using it. I want us to adopt cython as well and my early trials on older versions went well. Now we're approaching production use of Python code and I resumed work on Cython only to be stumped at Helloworld! I'm still looking for a simple fix - changing the line ending style is *NOT* an option - and I'm not impressed by this or by the *nix-centric flavour of the comments. Help please.

**********************************************************************
Please consider the environment - do you really need to print this email?

This email is intended only for the person(s) named above and may contain private and confidential information. If it has come to you in error, please destroy and permanently delete any copy in your possession and contact us on +44 (0) 161 480 4420. The information in this email is copyright © CDL Group Holdings Limited. We cannot accept any liability for any loss or damage sustained as a result of software viruses. It is your responsibility to carry out such virus checking as is necessary before opening any attachment.
Cheshire Datasystems Limited uses software which automatically screens incoming emails for inappropriate content and attachments. If the software identifies such content or attachment, the email will be forwarded to our Technology Department for checking. You should be aware that any email which you send to Cheshire Datasystems Limited is subject to this procedure.
Cheshire Datasystems Limited, Strata House, Kings Reach Road, Stockport SK4 2HD
Registered in England and Wales with Company Number 3991057
VAT registration: 727 1188 33

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100713/a7c4447f/attachment-0001.htm 

From stefan_ml at behnel.de  Tue Jul 13 13:59:42 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 13 Jul 2010 13:59:42 +0200
Subject: [Cython] bug #520: scanner broken for DOS line endings
In-Reply-To: 
References: 
Message-ID: <4C3C552E.70309@behnel.de>

Dave Cross, 13.07.2010 12:26:
> This email is intended only for the person(s) named above and may
> contain private and confidential information. If it has come to you in
> error, please destroy and permanently delete any copy in your possession
> and contact us on +44 (0) 161 480 4420. The information in this email is
> copyright© CDL Group Holdings Limited.

Since I wasn't named in the list of recipients, I guess I'll just have to 
destroy this message. Won't call you, though. Sorry.

Note that this message may also have been archived at several places, so be 
prepared to get a lot of phone calls in the future.

Stefan

From jonovik at gmail.com  Tue Jul 13 14:57:41 2010
From: jonovik at gmail.com (Jon Olav Vik)
Date: Tue, 13 Jul 2010 12:57:41 +0000 (UTC)
Subject: [Cython] bug #520: scanner broken for DOS line endings
References: 
	<4C3C552E.70309@behnel.de>
Message-ID: 

Stefan Behnel  writes:

> Dave Cross, 13.07.2010 12:26:
> > This email is intended only for the person(s) named above and may
> > contain private and confidential information. If it has come to you in
> > error, please destroy and permanently delete any copy in your possession
> > and contact us on +44 (0) 161 480 4420. The information in this email is
> > copyright? CDL Group Holdings Limited.
> 
> Since I wasn't named in the list of recipients, I guess I'll just have to 
> destroy this message. Won't call you, though. Sorry.
> 
> Note that this message may also have been archived at several places, so be 
> prepared to get a lot of phone calls in the future.

Heck, I'll dare the retribution of CDL Group Holdings Limited to point out that 
this is fixed in 0.13:

http://trac.cython.org/cython_trac/ticket/520



From Dave.Cross at cdl.co.uk  Tue Jul 13 15:10:22 2010
From: Dave.Cross at cdl.co.uk (Dave Cross)
Date: Tue, 13 Jul 2010 14:10:22 +0100
Subject: [Cython] bug #520: scanner broken for DOS line endings
In-Reply-To: 
References: <4C3C552E.70309@behnel.de>
	
Message-ID: 

Thank you for that.

.. and I'm pleased that our lawyers have provided some amusement. It's probably the only benefit they've ever been to anybody!

Dave.

> -----Original Message-----
> From: cython-dev-bounces at codespeak.net [mailto:cython-dev-
> bounces at codespeak.net] On Behalf Of Jon Olav Vik
> Sent: 13 July 2010 13:58
> To: cython-dev at codespeak.net
> Subject: Re: [Cython] bug #520: scanner broken for DOS line endings
> 
> Stefan Behnel  writes:
> 
> > Dave Cross, 13.07.2010 12:26:
> > > This email is intended only for the person(s) named above and may
> > > contain private and confidential information. If it has come to you
> > > in error, please destroy and permanently delete any copy in your
> > > possession and contact us on +44 (0) 161 480 4420. The information
> > > in this email is copyright? CDL Group Holdings Limited.
> >
> > Since I wasn't named in the list of recipients, I guess I'll just have
> > to destroy this message. Won't call you, though. Sorry.
> >
> > Note that this message may also have been archived at several places,
> > so be prepared to get a lot of phone calls in the future.
> 
> Heck, I'll dare the retribution of CDL Group Holdings Limited to point out that
> this is fixed in 0.13:
> 
> http://trac.cython.org/cython_trac/ticket/520
> 
> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
**********************************************************************
Please consider the environment - do you really need to print this email?

This email is intended only for the person(s) named above and may contain private and confidential information. If it has come to you in error, please destroy and permanently delete any copy in your possession and contact us on +44 (0) 161 480 4420. The information in this email is copyright ? CDL Group Holdings Limited. We cannot accept any liability for any loss or damage sustained as a result of software viruses. It is your responsibility to carry out such virus checking as is necessary before opening any attachment.

Cheshire Datasystems Limited uses software which automatically screens incoming emails for inappropriate content and attachments. If the software identifies such content or attachment, the email will be forwarded to our Technology Department for checking. You should be aware that any email which you send to Cheshire Datasystems Limited is subject to this procedure.

Cheshire Datasystems Limited, Strata House, Kings Reach Road, Stockport SK4 2HD
Registered in England and Wales with Company Number 3991057
VAT registration: 727 1188 33

From robertwb at math.washington.edu  Tue Jul 13 17:57:18 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 13 Jul 2010 08:57:18 -0700
Subject: [Cython] bug #520: scanner broken for DOS line endings
In-Reply-To: 
References: 
	
Message-ID: 

On Tue, Jul 13, 2010 at 3:26 AM, Dave Cross  wrote:
> Robert Bradshaw  writes:

> I assure you that it is a big issue for me. I'm a relatively new pythonista
> and I've sold my company on a big development using it. I want us to adopt
> cython as well and my early trials on older versions went well. Now we're
> approaching production use of Python code and I resumed work on Cython only
> to be stumped at Helloworld!
>
> I'm still looking for a simple fix - changing the line ending style is *NOT*
> an option - and I'm not impressed by this or by the *nix-centric flavour of
> the comments.

The project is as *nix-centric as its developer base--though we try to
do our best, I don't even have a Windows box. Being an open source
project, the real complaint for lack of Window support should go to
all the people who use Windows but don't contribute. (There are
several people who have done a lot in this direction, which I'm very
grateful for, and fortunately the system is mostly platform
independent.)

> Help please.

As mentioned, it has been resolved.

- Robert

From robertwb at math.washington.edu  Tue Jul 13 19:02:00 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 13 Jul 2010 10:02:00 -0700
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: <4C3B5676.3060101@student.matnat.uio.no>
References: <4C3B5676.3060101@student.matnat.uio.no>
Message-ID: 

On Mon, Jul 12, 2010 at 10:52 AM, Dag Sverre Seljebotn
 wrote:
> Right now I'm in Charles de Gaulle, about to fly home after EuroScipy 2010.
>
> I just wanted to post the main impression from the conference: Cython
> appears now to be a very mainstream tool in scientific Python computing.
> At some point it felt like every other talk mentioned Cython favourably,
> and Cython was prominent in both keynotes. (It was also nice that Konrad
> Hinsen made sure to attribute Pyrex as well.)
>
> I don't mean to gloat, but...well...it's hard not to share this: At one
> point a happy boost::python user got up and asked if nobody had heard
> about it, since he heard nothing about it. He asked everybody who had
> heard about it to raise their hand (very many) and then everybody who
> used it and where happy with it to raise their hand (and that was only a
> few).
>
> (Of course, he didn't ask about how many were using Cython. Lots of
> people have only heard about Cython and are not using it. Still
> interesting.)
>
> Apart from that, there's was lots of interesting people and talks :-)

Cool, thanks for the report. I heard second hand that Cython was a
common topic at SciPy in Texas last week too.

- Robert

From fperez.net at gmail.com  Tue Jul 13 22:04:52 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 13 Jul 2010 13:04:52 -0700
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	
Message-ID: 

On Tue, Jul 13, 2010 at 10:02 AM, Robert Bradshaw
 wrote:
>
> Cool, thanks for the report. I heard second hand that Cython was a
> common topic at SciPy in Texas last week too.

There is no question whatsoever that Cython is a *major* piece of the
puzzle for scientific/numerical work in Python, and I think every
single one of us is both grateful for the great work you guys have
done, and rooting for you to be able to continue successfully.

The discussions I had at Scipy2010 with Kurt on weave (following up on
the conversations with Robert) were  a small example of that, as we
all see Cython replacing the disparate set of tools we've had for
low-level language access with a well integrated machinery via Cython
that exposes C, C++ and Fortran in the most consistent way possible.

Cheers,

f

From kwmsmith at gmail.com  Wed Jul 14 02:37:21 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 13 Jul 2010 19:37:21 -0500
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
Message-ID: 

On Tue, Jul 13, 2010 at 3:04 PM, Fernando Perez  wrote:
> On Tue, Jul 13, 2010 at 10:02 AM, Robert Bradshaw
>  wrote:
>>
>> Cool, thanks for the report. I heard second hand that Cython was a
>> common topic at SciPy in Texas last week too.
>
> There is no question whatsoever that Cython is a *major* piece of the
> puzzle for scientific/numerical work in Python, and I think every
> single one of us is both grateful for the great work you guys have
> done, and rooting for you to be able to continue successfully.
>
> The discussions I had at Scipy2010 with Kurt on weave (following up on
> the conversations with Robert) were ?a small example of that, as we
> all see Cython replacing the disparate set of tools we've had for
> low-level language access with a well integrated machinery via Cython
> that exposes C, C++ and Fortran in the most consistent way possible.

Thanks for mentioning it -- I've been wanting to get that moving along
and see what Robert & everyone else's thoughts are.  Fwrap is taking
all my disposable time right now, but I'll invest some time on a
mock-up soon.  I'll be able to leverage much of what's in fwrap
currently for the project, I think.

It will be possible to do something like this with fwrap at some future date:

in a regular Python file, run with the Python interpreter:

def array_func_to_speedup(array1, array2):
    # array1 & array2 are 2D numpy arrays
    # still figuring out the exact syntax, inspired by scipy.weave
    from fwrap import weave
    fsrc = '''\
    integer i,j
    do j = 1, array1_d2
        do i = 1, array1_d1
            array1(i, j) = array2(i, j)**2
        enddo
    enddo
    '''
    weave.inline(fsrc, ['array1', 'array2'], )
    assert np.all(array1 == array2**2)

The 1st-class array support in Fortran 90/95 is a really nice fit
here, I think, and the syntax is fairly natural.

It won't be too difficult to take the infrastructure in fwrap &
scipy.weave and do the above but with C code rather than Fortran 90.

But this belongs in another thread.

Kurt

From sturla at molden.no  Wed Jul 14 05:59:42 2010
From: sturla at molden.no (Sturla Molden)
Date: Wed, 14 Jul 2010 05:59:42 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>		
	
Message-ID: <4C3D362E.80901@molden.no>

Kurt Smith skrev:
> def array_func_to_speedup(array1, array2):
>     # array1 & array2 are 2D numpy arrays
>     # still figuring out the exact syntax, inspired by scipy.weave
>     from fwrap import weave
>     fsrc = '''\
>     integer i,j
>     do j = 1, array1_d2
>         do i = 1, array1_d1
>             array1(i, j) = array2(i, j)**2
>         enddo
>     enddo
>     '''
>     weave.inline(fsrc, ['array1', 'array2'], )
>     assert np.all(array1 == array2**2)
>
> The 1st-class array support in Fortran 90/95 is a really nice fit
> here, I think, and the syntax is fairly natural.
>
>   
Actually,

   fsrc = "array1 = array2**2"

would suffice :-)

That just shows why Fortran 90 is less painful than C for the work 
scientists often do.


Sturla













From kwmsmith at gmail.com  Wed Jul 14 07:30:30 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 14 Jul 2010 00:30:30 -0500
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: <4C3D362E.80901@molden.no>
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
	
	<4C3D362E.80901@molden.no>
Message-ID: 

On Tue, Jul 13, 2010 at 10:59 PM, Sturla Molden  wrote:
> Kurt Smith skrev:
>> def array_func_to_speedup(array1, array2):
>> ? ? # array1 & array2 are 2D numpy arrays
>> ? ? # still figuring out the exact syntax, inspired by scipy.weave
>> ? ? from fwrap import weave
>> ? ? fsrc = '''\
>> ? ? integer i,j
>> ? ? do j = 1, array1_d2
>> ? ? ? ? do i = 1, array1_d1
>> ? ? ? ? ? ? array1(i, j) = array2(i, j)**2
>> ? ? ? ? enddo
>> ? ? enddo
>> ? ? '''
>> ? ? weave.inline(fsrc, ['array1', 'array2'], )
>> ? ? assert np.all(array1 == array2**2)
>>
>> The 1st-class array support in Fortran 90/95 is a really nice fit
>> here, I think, and the syntax is fairly natural.
>>
>>
> Actually,
>
> ? fsrc = "array1 = array2**2"
>
> would suffice :-)
>
> That just shows why Fortran 90 is less painful than C for the work
> scientists often do.

I should have picked a less trivial example :)  But you're right --
Fortran's 1st class arrays have some very nice features, kind of like
a simpler-and-faster version of numpy arrays.

Unfortunately, for some compilers I've used, array expressions like:

array1 = array2 + array3**2

Are slower than the 'do' version with explicit indexing -- that is to
say, slower than they should be.  Apparently the compilers create
temporary arrays, exactly analogous to how numpy does it (although
many times faster and with more opportunities for optimization).

Perhaps I was using an old compiler when I tested this, or didn't turn
on the right compile flags.  Perhaps you can correct me on this.

Kurt

>
>
> Sturla

From sturla at molden.no  Wed Jul 14 11:57:00 2010
From: sturla at molden.no (Sturla Molden)
Date: Wed, 14 Jul 2010 11:57:00 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
Message-ID: <2e0e566346c14cd47bd3a735156504b3.squirrel@webmail.uio.no>

> On Tue, Jul 13, 2010 at 10:59 PM, Sturla Molden  wrote:

> Are slower than the 'do' version with explicit indexing -- that is to
> say, slower than they should be.  Apparently the compilers create
> temporary arrays, exactly analogous to how numpy does it (although
> many times faster and with more opportunities for optimization).
>
> Perhaps I was using an old compiler when I tested this, or didn't turn
> on the right compile flags.  Perhaps you can correct me on this.

I think you are right, at least for x86 with gfortran. But it's not only a
matter of speed. Fortran 95 arrays have all the niceness of NumPy arrays,
including broadcasting and slicing. Fortran also knows about complex
numbers, and have a richer standard library. I am willing to suffer some
performance loss to avoid the pain of writing C or Fortran 77.

Regarding performance of Fortran 90 array expressions:

Fortran 90/95 arrays expressions can give poorer poorer or better
performance than similar do-loops, depending on hardware and compiler. You
can see the same with "forall" loops and "where" constructs. Fortran 90
was designed for parallel computing. Therefore array expressions, forall
loops, and where constructs have therefore some inherent "parallelism"
required by the standard. This will sometimes require a temporary array to
be made, but memory allocation and access can be slow on x86. An example
would e.g. be this:

a(2:n) = a(1:n-1) + a(2:n)

A  do loop could use a couple of temporary variables kept in registers
instead:

r1 = a(1)
do i = 2,n
    r2 = a(i)
    a(i) = r1 + a(i)
    r1 = r2
end do

Obviously just putting r1 and r2 in registers would be faster than
allocating a temporary array when using x86, without any other
optimization.

Some f95 compilers (Absoft and Intel) are good at auto-vectorizing array
statements to SIMD or multithreaded code. Sometimes an array expression
can be easier to auto-vectorize. Using a couple of temporary variables
means we must run the code sequentially. So maximum performance on x86
could require manual partial loop unrolling, sort of making the code fit
the hardware.

On older vector computers like the Cray, a temporary array would probably
be much faster than a do loop, as it could multiply two arrays in one
operation. They were SIMD monsters.

On modern GPUs a temporary array solution could be the faster, due to the
massive multi-threading they have inside, and because memory is faster.

The number of temporary variables matters too. If you have a RISC
processor like PPC, there are a lot of registers available. So a solution
with many temporary variables and a do-loop could be very efficient. While
on an x86 there are few registers available, which means a temporary array
could be faster if temporaries get too plentiful, but still slower than
using a do loop with few temporaries.

So there is an interaction between code, hardware and compiler. But at
least with gfortran on x86, I expect do-loops to generally perform better
than array expressions, forall and where.

And regarding speed of Fortran vs. C and compilers:

It used to be the case that Fortran was "faster than C". But that was
before C and C++ compilers became alias-analysis champions and Fortran 95
was adopted. I often see gcc produce code that runs twice as fast as
gfortran, whereas g77 used to beat gcc without problems 10 years ago. With
Intel or Absoft compilers, Fortran still gives the better results. When
using Fortran 95 and GNU compilers, I am not surprised if C code would run
twice as fast on x86. gfortran is fairly good, but not the most aggressive
compiler on the market. Intel ifort is probably the best, but very
expensive. Absoft can be a good compromise between quality and price.

Sturla



From fperez.net at gmail.com  Wed Jul 14 12:15:31 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 14 Jul 2010 03:15:31 -0700
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	 
	 
	 
	<4C3D362E.80901@molden.no>
	
Message-ID: 

On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith  wrote:
>
> Unfortunately, for some compilers I've used, array expressions like:
>
> array1 = array2 + array3**2
>
> Are slower than the 'do' version with explicit indexing -- that is to
> say, slower than they should be. ?Apparently the compilers create
> temporary arrays, exactly analogous to how numpy does it (although
> many times faster and with more opportunities for optimization).

as a minor note, blitz++ handles this with expression templates, and
that's what weave uses for array manipulations.  So one gets very
numpy-like arrays in c++ that have also efficient behavior in many
expressions.

Not saying that we don't also want 'inline fortran', just that weave's
inline/blitz support is actually quite powerful.  Except that right
now it seems to be broken, as I just ran into with some of my simple
examples:

http://projects.scipy.org/scipy/ticket/739

Bummer...

Cheers,

f

From sturla at molden.no  Wed Jul 14 13:03:00 2010
From: sturla at molden.no (Sturla Molden)
Date: Wed, 14 Jul 2010 13:03:00 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
	
	<4C3D362E.80901@molden.no>
	
	
Message-ID: 

> On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith  wrote:

> as a minor note, blitz++ handles this with expression templates,

Blitz++ do this just for arrays with fixed size at compile time.

The gain from template meta-programming is almost amortized from better
C++ compilers. Today C++ compiler tend to produce better machine code than
anything else (C and Fortran included) on x86 and amd64. While when
blitz++ was written, they were slow bloatware generators that took a
5x-10x performance hit compared with Fortran77. They don't do that
anymore.

If you have Langtangen's book on "Python Scripting for Computational
Science", chapter 10.3 explains how to wrap NumPy arrays in a C++ class.

C++ also support complex numbers in the class std::complex, and
std::vector works almost like a Python list. std::map can be used almost
like a dictionary. So yes, C++ should be preferred over C. But preferred
over Fortran 95?

There are two issues I think:

* C++ is very hard to use correctly, and can take 10 years to fully
master. I does not take that long to learn Fortran 95.

* We have Python (or Cython) to do the non-numerical work. If we just want
to crunch numbers, there is nothing like Fortran.

So what about Cython?

In my experience, I always have to resort to Fortran or C++. Cython cannot
use NumPy arrays efficiently as function arguments. That is a big show
stopper. Cython therefore degenerate to C (working with pointers instead
of arrays), and we could just as well have written C, which is what we
want to avoid. So that leaves us with C++ (wrapped NumPy arrays) or
Fortran 95.

Sturla


From stefan_ml at behnel.de  Wed Jul 14 13:10:34 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 14 Jul 2010 13:10:34 +0200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: 
References: 
Message-ID: <4C3D9B2A.8080308@behnel.de>

Craig Citro, 14.07.2010 10:14:
> I'm working on getting cython-closures to compile and pass doctests
> with Sage. I have everything building and starting up okay, but I ran
> into a problem actually running doctests -- everything's segfaulting
> like crazy. I tracked down the problem, which is with the min/max
> optimization you added to Optimize.py. Robert and I glanced at it this
> afternoon, and spotted two issues: (1) there's some bad typing that
> gets generated in the C code (a Py_ssize_t gets treated as a PyObject
> *), and (2) the arguments get evaluated multiple times.

I looked into it and trying to fix it made me run into a temp handling bug. 
LetRefNodes do not properly do their ref-counting when they work as a 
pass-through from temp nodes to other temp nodes. I'll see if I can fix 
that first.

Stefan

From kwmsmith at gmail.com  Wed Jul 14 18:01:38 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 14 Jul 2010 11:01:38 -0500
Subject: [Cython] Cython weave thread, was: "Impressions from EuroScipy 2010"
Message-ID: 

On Wed, Jul 14, 2010 at 5:15 AM, Fernando Perez  wrote:
> On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith  wrote:
>>
>> Unfortunately, for some compilers I've used, array expressions like:
>>
>> array1 = array2 + array3**2
>>
>> Are slower than the 'do' version with explicit indexing -- that is to
>> say, slower than they should be.  Apparently the compilers create
>> temporary arrays, exactly analogous to how numpy does it (although
>> many times faster and with more opportunities for optimization).
>
> as a minor note, blitz++ handles this with expression templates, and
> that's what weave uses for array manipulations.  So one gets very
> numpy-like arrays in c++ that have also efficient behavior in many
> expressions.
>
> Not saying that we don't also want 'inline fortran', just that weave's
> inline/blitz support is actually quite powerful.

Yes, it is quite powerful.  It will take some doing to get equivalent
support for all that weave has to offer.  I'm hoping to allow
something like this, modulo syntactic stuff:

def cy_weave_example(arr):
    src = \
'''
for(npy_intp i=0; i
References: 
Message-ID: 

On Wed, Jul 14, 2010 at 6:03 AM, Sturla Molden  wrote:
>> On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith  wrote:
>
>> as a minor note, blitz++ handles this with expression templates,
>
> Blitz++ do this just for arrays with fixed size at compile time.
>
> The gain from template meta-programming is almost amortized from better
> C++ compilers. Today C++ compiler tend to produce better machine code than
> anything else (C and Fortran included) on x86 and amd64. While when
> blitz++ was written, they were slow bloatware generators that took a
> 5x-10x performance hit compared with Fortran77. They don't do that
> anymore.
>
> If you have Langtangen's book on "Python Scripting for Computational
> Science", chapter 10.3 explains how to wrap NumPy arrays in a C++ class.
>
> C++ also support complex numbers in the class std::complex, and
> std::vector works almost like a Python list. std::map can be used almost
> like a dictionary. So yes, C++ should be preferred over C. But preferred
> over Fortran 95?
>
> There are two issues I think:
>
> * C++ is very hard to use correctly, and can take 10 years to fully
> master. I does not take that long to learn Fortran 95.
>
> * We have Python (or Cython) to do the non-numerical work. If we just want
> to crunch numbers, there is nothing like Fortran.

In case I haven't said it: I'm intending to have weave.inline-like
support for both inlined fortran (which will come first, since the
functionality is already there in fwrap) and for inlined C. The
inlined C will possibly have some sugar for array indexing syntax
e.g., "arr[i,j]" in the inlined C code would be transformed to valid
C.

>From a developer & maintainer's perspective, having 'fortran inline'
gives us 1st-class arrays for free in the inlined code which is a huge
plus. There is no need for index overloading & the requisite
maintenance of the support code. The expressiveness in the inlined
fortran code for array expressions is pretty good, and you can stay in
the flow of your numpy array code.  It delegates to Fortran what it's
good at doing, and reserves everything else for the Python level, like
Sturla has pointed out.  Many fortran compilers give us threaded array
operations for free, too, which is very nice, as Sturla mentioned.

I'm planning to leverage the infrastructure for 'fortran inline' to
make the 'C inline' easier.

>
> So what about Cython?
>
> In my experience, I always have to resort to Fortran or C++. Cython cannot
> use NumPy arrays efficiently as function arguments. That is a big show
> stopper. Cython therefore degenerate to C (working with pointers instead
> of arrays), and we could just as well have written C, which is what we
> want to avoid. So that leaves us with C++ (wrapped NumPy arrays) or
> Fortran 95.

The other part of my GSoC last summer was improving buffer support in
Cython as envisioned by Dag Sverre; the basic functionality is there
in the kurt-gsoc branch and it is the first steps towards 1st-class
arrays in the Cython language.  I hope Dag & I can find some time to
get it merged, but outside constraints have prevented it.

Kurt

From dagss at student.matnat.uio.no  Wed Jul 14 20:23:39 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 14 Jul 2010 20:23:39 +0200
Subject: [Cython] Cython weave thread,
 was: "Impressions from EuroScipy 2010"
In-Reply-To: 
References: 
Message-ID: <4C3E00AB.5000704@student.matnat.uio.no>

On 07/14/2010 06:01 PM, Kurt Smith wrote:
> On Wed, Jul 14, 2010 at 5:15 AM, Fernando Perez  wrote:
>    
>> On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith  wrote:
>>      
>>> Unfortunately, for some compilers I've used, array expressions like:
>>>
>>> array1 = array2 + array3**2
>>>
>>> Are slower than the 'do' version with explicit indexing -- that is to
>>> say, slower than they should be.  Apparently the compilers create
>>> temporary arrays, exactly analogous to how numpy does it (although
>>> many times faster and with more opportunities for optimization).
>>>        
>> as a minor note, blitz++ handles this with expression templates, and
>> that's what weave uses for array manipulations.  So one gets very
>> numpy-like arrays in c++ that have also efficient behavior in many
>> expressions.
>>
>> Not saying that we don't also want 'inline fortran', just that weave's
>> inline/blitz support is actually quite powerful.
>>      
> Yes, it is quite powerful.  It will take some doing to get equivalent
> support for all that weave has to offer.  I'm hoping to allow
> something like this, modulo syntactic stuff:
>
> def cy_weave_example(arr):
>      src = \
> '''
> for(npy_intp i=0; i      for(npy_intp j=0; j          arr[i,j] = i * j;
>      }
> }
> '''
>      cyweave.inline(src, ['arr'])
>
> The "arr[i,j]" syntax will take some work, and I'm still thinking
> about it.  It's not valid C, obviously.  One possibility would be to
> do some simple preprocessing to transform into something like
> "arr[__2D_idx(i, j, arr_shape[1])]" where "__2D_idx(...)" is a
> generated macro which is something like:
>
> #define __2D_idx(i, j, s1) ((i) * (s1) + (j))
>
> If we were to use the C syntax "arr[i][j]", it would require a weave
> recompile whenever the shape of "arr" changes, which would be
> annoying.  The "arr[i,j]" would preserve the array-like feel and would
> be doable.  Perhaps there are pitfalls in doing a poor-man's bracket
> overloading -- I'm certainly open to suggestions.
>    

Isn't there's enough tools going in this direction already? This would 
very closely duplicate what is aimed for (or already exists) in Cython, 
and not be backwards compatible with Weave...

As long as it is not something people are already using anyway, why not 
allow Cython code within the string? I think you can do pretty much the 
same as what you propose for C currently, and more future possibilities 
then seem open (such as array expressions/first class arrays).

+1 to make Weave with C++ and Blitz run on top of Cython or fwrap, it's 
always good to consolidate efforts.

Dag Sverre

From robertwb at math.washington.edu  Thu Jul 15 03:59:24 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 14 Jul 2010 18:59:24 -0700
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
	
	<4C3D362E.80901@molden.no>
	
	
	
Message-ID: 

On Wed, Jul 14, 2010 at 4:03 AM, Sturla Molden  wrote:
>> On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith  wrote:
>
>> as a minor note, blitz++ handles this with expression templates,
>
> Blitz++ do this just for arrays with fixed size at compile time.
>
> The gain from template meta-programming is almost amortized from better
> C++ compilers. Today C++ compiler tend to produce better machine code than
> anything else (C and Fortran included) on x86 and amd64. While when
> blitz++ was written, they were slow bloatware generators that took a
> 5x-10x performance hit compared with Fortran77. They don't do that
> anymore.
>
> If you have Langtangen's book on "Python Scripting for Computational
> Science", chapter 10.3 explains how to wrap NumPy arrays in a C++ class.
>
> C++ also support complex numbers in the class std::complex, and
> std::vector works almost like a Python list. std::map can be used almost
> like a dictionary. So yes, C++ should be preferred over C. But preferred
> over Fortran 95?
>
> There are two issues I think:
>
> * C++ is very hard to use correctly, and can take 10 years to fully
> master. I does not take that long to learn Fortran 95.
>
> * We have Python (or Cython) to do the non-numerical work. If we just want
> to crunch numbers, there is nothing like Fortran.
>
> So what about Cython?
>
> In my experience, I always have to resort to Fortran or C++. Cython cannot
> use NumPy arrays efficiently as function arguments. That is a big show stopper.

Could you clarify? I suppose for very small arrays, there's the extra
O(1) type-check/stride extraction overhead. Is there anything else? Of
course there's always more room for improvement for the array type.

> Cython therefore degenerate to C (working with pointers instead
> of arrays), and we could just as well have written C, which is what we
> want to avoid. So that leaves us with C++ (wrapped NumPy arrays) or
> Fortran 95.

Personally, I'd rather work with C than C++ or Fortan (and prefer
Python/Cython to both), but that's a matter of preference.

- Robert

From robertwb at math.washington.edu  Thu Jul 15 04:09:01 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 14 Jul 2010 19:09:01 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: <4C3E00AB.5000704@student.matnat.uio.no>
References: 
	<4C3E00AB.5000704@student.matnat.uio.no>
Message-ID: 

On Wed, Jul 14, 2010 at 11:23 AM, Dag Sverre Seljebotn
 wrote:
> On 07/14/2010 06:01 PM, Kurt Smith wrote:
>> On Wed, Jul 14, 2010 at 5:15 AM, Fernando Perez ?wrote:
>>
>>> On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith ?wrote:
>>>
>>>> Unfortunately, for some compilers I've used, array expressions like:
>>>>
>>>> array1 = array2 + array3**2
>>>>
>>>> Are slower than the 'do' version with explicit indexing -- that is to
>>>> say, slower than they should be. ?Apparently the compilers create
>>>> temporary arrays, exactly analogous to how numpy does it (although
>>>> many times faster and with more opportunities for optimization).
>>>>
>>> as a minor note, blitz++ handles this with expression templates, and
>>> that's what weave uses for array manipulations. ?So one gets very
>>> numpy-like arrays in c++ that have also efficient behavior in many
>>> expressions.
>>>
>>> Not saying that we don't also want 'inline fortran', just that weave's
>>> inline/blitz support is actually quite powerful.
>>>
>> Yes, it is quite powerful. ?It will take some doing to get equivalent
>> support for all that weave has to offer. ?I'm hoping to allow
>> something like this, modulo syntactic stuff:
>>
>> def cy_weave_example(arr):
>> ? ? ?src = \
>> '''
>> for(npy_intp i=0; i> ? ? ?for(npy_intp j=0; j> ? ? ? ? ?arr[i,j] = i * j;
>> ? ? ?}
>> }
>> '''
>> ? ? ?cyweave.inline(src, ['arr'])
>>
>> The "arr[i,j]" syntax will take some work, and I'm still thinking
>> about it. ?It's not valid C, obviously. ?One possibility would be to
>> do some simple preprocessing to transform into something like
>> "arr[__2D_idx(i, j, arr_shape[1])]" where "__2D_idx(...)" is a
>> generated macro which is something like:
>>
>> #define __2D_idx(i, j, s1) ((i) * (s1) + (j))
>>
>> If we were to use the C syntax "arr[i][j]", it would require a weave
>> recompile whenever the shape of "arr" changes, which would be
>> annoying. ?The "arr[i,j]" would preserve the array-like feel and would
>> be doable. ?Perhaps there are pitfalls in doing a poor-man's bracket
>> overloading -- I'm certainly open to suggestions.
>>
>
> Isn't there's enough tools going in this direction already? This would
> very closely duplicate what is aimed for (or already exists) in Cython,
> and not be backwards compatible with Weave...
>
> As long as it is not something people are already using anyway, why not
> allow Cython code within the string? I think you can do pretty much the
> same as what you propose for C currently, and more future possibilities
> then seem open (such as array expressions/first class arrays).

That was my first thought too--what advantage is there to inlining C
into the mix? Fortran I can see more of a usecase for, though even
then I hope that Cython will be able to handle most of those usecases
as well.

Though I can see the advantages of weave, I think it'd be even better
if one didin't have to switch back and forth between completely
unrelated syntaxes/types/etc. Do people use weave because there are
certain things that are more natural to express in Fortran (and/or C),
or is it primarily a convenient way to inline compiled code? Still
waiting for someone to put up a wiki page on what they like from weave
and what should go into Cython.

I think an

@cython.compile
def foo(...):
    ...

could go a long way here, and even

...
cython.eval("""
[valid cython code]
""")

could have its merits too (though it's not near as clean.

- Robert

From carl.witty at gmail.com  Thu Jul 15 04:47:52 2010
From: carl.witty at gmail.com (Carl Witty)
Date: Wed, 14 Jul 2010 19:47:52 -0700
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	 
	 
	 
	<4C3D362E.80901@molden.no>
	 
	 
	 
	
Message-ID: 

On Wed, Jul 14, 2010 at 6:59 PM, Robert Bradshaw
 wrote:
>> In my experience, I always have to resort to Fortran or C++. Cython cannot
>> use NumPy arrays efficiently as function arguments. That is a big show stopper.
>
> Could you clarify? I suppose for very small arrays, there's the extra
> O(1) type-check/stride extraction overhead. Is there anything else? Of
> course there's always more room for improvement for the array type.

Sometimes I write functions that don't iterate over every element of
their argument arrays :)

In particular, I remember that when I was writing
sage/plot/plot3d/implicit_surface.pyx, I wanted to make several chunks
of code into methods but decided not to for efficiency reasons.  (I'm
not sure if I actually measured, or if I was just worried.)

If I was doing it again, I think I'd give up on the Cython numpy
support and just use C pointers in that code...

Carl

From robertwb at math.washington.edu  Thu Jul 15 05:47:00 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 14 Jul 2010 20:47:00 -0700
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
	
	<4C3D362E.80901@molden.no>
	
	
	
	
	
Message-ID: 

On Wed, Jul 14, 2010 at 7:47 PM, Carl Witty  wrote:
> On Wed, Jul 14, 2010 at 6:59 PM, Robert Bradshaw
>  wrote:
>>> In my experience, I always have to resort to Fortran or C++. Cython cannot
>>> use NumPy arrays efficiently as function arguments. That is a big show stopper.
>>
>> Could you clarify? I suppose for very small arrays, there's the extra
>> O(1) type-check/stride extraction overhead. Is there anything else? Of
>> course there's always more room for improvement for the array type.
>
> Sometimes I write functions that don't iterate over every element of
> their argument arrays :)

Good point. How does Fortran handle this better? Does it just know
that the strides are not aliased? (Or, does it know dimensions/other
data at compile time.)

> In particular, I remember that when I was writing
> sage/plot/plot3d/implicit_surface.pyx, I wanted to make several chunks
> of code into methods but decided not to for efficiency reasons. ?(I'm
> not sure if I actually measured, or if I was just worried.)

For reference, http://hg.sagemath.org/sage-main/file/8dec8b43ccca/sage/plot/plot3d/implicit_surface.pyx#l1

> If I was doing it again, I think I'd give up on the Cython numpy
> support and just use C pointers in that code...

Clearly we need to be doing thing better :).

- Robert

From fperez.net at gmail.com  Thu Jul 15 06:16:52 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 14 Jul 2010 21:16:52 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: 
References:  
	<4C3E00AB.5000704@student.matnat.uio.no>
	
Message-ID: 

On Wed, Jul 14, 2010 at 7:09 PM, Robert Bradshaw
 wrote:
> Still
> waiting for someone to put up a wiki page on what they like from weave
> and what should go into Cython.

Ahem :)

A simple example of the blitz expressions which are nice:

In [2]: import numpy as np

In [3]: from scipy import weave

In [4]:

In [5]: a = np.arange(10)

In [6]: x = np.empty_like(a)

In [7]: weave.blitz('x=a+2*a*a')
creating /tmp/fperez/python26_intermediate/compiler_8dc7dab30c62f7787b818e8ffdc19f8c

In [8]: print x
[  0   3  10  21  36  55  78 105 136 171]

In [9]: print x-(a+2*a*a)
[0 0 0 0 0 0 0 0 0 0]


Interactively, weave gives you a feel of 'interactive C++' thanks to
its tying of the module name to the code hash:

In [17]: src = 'std::cout << "a is " << a << std::endl;'

In [18]: a = 10

In [19]: inline(src, ['a'])  # compilation happens here
a is 10

In [20]: a = 4.99

In [21]: inline(src, ['a'])  #compiling again, for a float
a is 4.99

In [22]: a = 12

In [23]: inline(src, ['a']) # no compilation, code for a int is reused
a is 12


Some examples that use the blitz indexing support:

# Returning a scalar quantity computed from an array.
def trace(mat):
    """Return the trace of a matrix.
    """
    nrow,ncol = mat.shape
    code = \
"""
double tr=0.0;

for(int i=0;i
References: 
	<4C3E00AB.5000704@student.matnat.uio.no>
	
Message-ID: 

On Wed, Jul 14, 2010 at 9:09 PM, Robert Bradshaw
 wrote:
> On Wed, Jul 14, 2010 at 11:23 AM, Dag Sverre Seljebotn
>  wrote:
>> On 07/14/2010 06:01 PM, Kurt Smith wrote:
>>> On Wed, Jul 14, 2010 at 5:15 AM, Fernando Perez ?wrote:
>>>
>>>> On Tue, Jul 13, 2010 at 10:30 PM, Kurt Smith ?wrote:
>>>>
>>>>> Unfortunately, for some compilers I've used, array expressions like:
>>>>>
>>>>> array1 = array2 + array3**2
>>>>>
>>>>> Are slower than the 'do' version with explicit indexing -- that is to
>>>>> say, slower than they should be. ?Apparently the compilers create
>>>>> temporary arrays, exactly analogous to how numpy does it (although
>>>>> many times faster and with more opportunities for optimization).
>>>>>
>>>> as a minor note, blitz++ handles this with expression templates, and
>>>> that's what weave uses for array manipulations. ?So one gets very
>>>> numpy-like arrays in c++ that have also efficient behavior in many
>>>> expressions.
>>>>
>>>> Not saying that we don't also want 'inline fortran', just that weave's
>>>> inline/blitz support is actually quite powerful.
>>>>
>>> Yes, it is quite powerful. ?It will take some doing to get equivalent
>>> support for all that weave has to offer. ?I'm hoping to allow
>>> something like this, modulo syntactic stuff:
>>>
>>> def cy_weave_example(arr):
>>> ? ? ?src = \
>>> '''
>>> for(npy_intp i=0; i>> ? ? ?for(npy_intp j=0; j>> ? ? ? ? ?arr[i,j] = i * j;
>>> ? ? ?}
>>> }
>>> '''
>>> ? ? ?cyweave.inline(src, ['arr'])
>>>
>>> The "arr[i,j]" syntax will take some work, and I'm still thinking
>>> about it. ?It's not valid C, obviously. ?One possibility would be to
>>> do some simple preprocessing to transform into something like
>>> "arr[__2D_idx(i, j, arr_shape[1])]" where "__2D_idx(...)" is a
>>> generated macro which is something like:
>>>
>>> #define __2D_idx(i, j, s1) ((i) * (s1) + (j))
>>>
>>> If we were to use the C syntax "arr[i][j]", it would require a weave
>>> recompile whenever the shape of "arr" changes, which would be
>>> annoying. ?The "arr[i,j]" would preserve the array-like feel and would
>>> be doable. ?Perhaps there are pitfalls in doing a poor-man's bracket
>>> overloading -- I'm certainly open to suggestions.
>>>
>>
>> Isn't there's enough tools going in this direction already? This would
>> very closely duplicate what is aimed for (or already exists) in Cython,
>> and not be backwards compatible with Weave...
>>
>> As long as it is not something people are already using anyway, why not
>> allow Cython code within the string? I think you can do pretty much the
>> same as what you propose for C currently, and more future possibilities
>> then seem open (such as array expressions/first class arrays).
>
> That was my first thought too--what advantage is there to inlining C
> into the mix? Fortran I can see more of a usecase for, though even
> then I hope that Cython will be able to handle most of those usecases
> as well.
>
> Though I can see the advantages of weave, I think it'd be even better
> if one didin't have to switch back and forth between completely
> unrelated syntaxes/types/etc. Do people use weave because there are
> certain things that are more natural to express in Fortran (and/or C),
> or is it primarily a convenient way to inline compiled code? Still
> waiting for someone to put up a wiki page on what they like from weave
> and what should go into Cython.

I'll get to it soon; but this thread serves to get the ideas out there
in raw form.

My impressions are from using weave a bit in the past and from those
who have mentioned interest in keeping it alive with a Cython
transplant (primarily Fernando).  I welcome Fernando (& others) to
chime in with the other killer features that weave offers.  And
critics of the weave 'approach' are more than weclome, too :-)

(Note: I've only used weave, but much of what I say applies to blitz, too.)

1) For an existing function that has to be sped up, weave has a lower
barrier to usage than Cython, even with pyximport.  (pyximport doesn't
enable an equivalent functionality as weave, so it isn't exactly
germane).  It is very nice to just drop into C/C++ in the middle of a
function and have things compiled & run automatically.  This is in
large part because weave allows incremental usage wherever needed on a
function-by-function basis, whereas Cython's smallest compilation unit
is the module.

2) I liked weave because you know you're getting to the metal with no
more than 5-10 lines of setup code, and with absolutely no fuss with
external tools or having to create a setup.py.  This may sound lazy,
but it really is nice not to have to deal with these things.
Sometimes numpy array expressions aren't the most natural way to
express something, and you just want to cut through it all and get a
simple, fast C 'for' loop in there and move on, but cythonizing would
be too big of a step, or impossible if you have an inner function
somewhere else in the module, for example.  Weave scratches the itch
very nicely.

3) The C-source-in-a-string is pretty ugly, I admit.  Weave is very
far on the "practicality" side of the "practicality vs. purity" scale.
 It is a small benefit, though, to have the C/C++ code quarantined to
one place in a function, and to always know exactly what to expect
from each part of your code -- python remains python, C remains C.
The fluidity of Cython is a tremendous advantage, but it does require
some care to be sure you're typing everything that needs to be typed,
turning off boundschecking when it's safe, etc.  I tend to use 'cython
-a' or inspecting the generated source to be sure I'm "totally C"
fairly often.  Weave gives this to you automatically.  In short, with
weave, I don't have to pay attention as much to get the same degree of
performance.  The tradeoff is ugly code.

4) Dag mentioned allowing Cython code in the source string. This
certainly has merit and I'm somewhat in favor of it.  It would avoid
reinventing the wheel for the array indexing stuff, and would have a
more pythonic feel.  But it would negate some of the benefits of
having "just C" in the source string.

6) Up until now I've been assuming that this functionality will be
implemented as an external tool.  Perhaps you (Robert, Dag, others)
have thoughts on getting some subset of weave's features in Cython
itself, namely the function-level compilation of a chunk of code.
Robert mentions this below with a @cython.compile decorator.  And the
'cython.eval()' has all the essential features of
weave/blitz right there, including the ugliness ;-)

So, remaining issues:

-) Essential features of weave/blitz that I haven't mentioned.

-) Is this thing better off as an external tool, or integrated in cython itself?
    If the source string is cython code, then it's better as an
internal-to-cython tool.  If C, then an external tool would suffice.

-) I'll summarize this, make it less verbose, and put it on the wiki.

Kurt

>
> I think an
>
> @cython.compile
> def foo(...):
> ? ?...
>
> could go a long way here, and even
>
> ...
> cython.eval("""
> [valid cython code]
> """)
>
> could have its merits too (though it's not near as clean.
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Thu Jul 15 07:14:13 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 14 Jul 2010 22:14:13 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: 
References: 
	<4C3E00AB.5000704@student.matnat.uio.no>
	
	
Message-ID: 

On Wed, Jul 14, 2010 at 9:16 PM, Fernando Perez  wrote:
> On Wed, Jul 14, 2010 at 7:09 PM, Robert Bradshaw
>  wrote:
>> Still
>> waiting for someone to put up a wiki page on what they like from weave
>> and what should go into Cython.
>
> Ahem :)
>
> A simple example of the blitz expressions which are nice:
>
> In [2]: import numpy as np
>
> In [3]: from scipy import weave
>
> In [4]:
>
> In [5]: a = np.arange(10)
>
> In [6]: x = np.empty_like(a)
>
> In [7]: weave.blitz('x=a+2*a*a')
> creating /tmp/fperez/python26_intermediate/compiler_8dc7dab30c62f7787b818e8ffdc19f8c
>
> In [8]: print x
> [ ?0 ? 3 ?10 ?21 ?36 ?55 ?78 105 136 171]
>
> In [9]: print x-(a+2*a*a)
> [0 0 0 0 0 0 0 0 0 0]
>
>
> Interactively, weave gives you a feel of 'interactive C++' thanks to
> its tying of the module name to the code hash:
>
> In [17]: src = 'std::cout << "a is " << a << std::endl;'
>
> In [18]: a = 10
>
> In [19]: inline(src, ['a']) ?# compilation happens here
> a is 10
>
> In [20]: a = 4.99
>
> In [21]: inline(src, ['a']) ?#compiling again, for a float
> a is 4.99
>
> In [22]: a = 12
>
> In [23]: inline(src, ['a']) # no compilation, code for a int is reused
> a is 12
>
>
> Some examples that use the blitz indexing support:
>
> # Returning a scalar quantity computed from an array.
> def trace(mat):
> ? ?"""Return the trace of a matrix.
> ? ?"""
> ? ?nrow,ncol = mat.shape
> ? ?code = \
> """
> double tr=0.0;
>
> for(int i=0;i ? ?tr += mat(i,i);
> return_val = tr;
> """
> ? ?return inline(code,['mat','nrow','ncol'],
> ? ? ? ? ? ? ? ? ?type_converters = converters.blitz)
>
> # In-place operations on arrays in general work without any problems
> def in_place_mult(num,mat):
> ? ?"""In-place multiplication of a matrix by a scalar.
> ? ?"""
> ? ?nrow,ncol = mat.shape
> ? ?code = \
> """
> for(int i=0;i ? ?for(int j=0;j ? ? ? ?mat(i,j) *= num;
>
> """
> ? ?inline(code,['num','mat','nrow','ncol'],
> ? ? ? ? ? type_converters = converters.blitz)
>
>
> But at least I don't care if this *specific syntax* survives, I only
> care about the ability to produce fast code to manipulate numpy arrays
> with clean syntax. ?Weave with inline/blitz helps really well in
> practice on this front, and if we can get these abilities with Cython,
> that would be great.

I think we can.

> One more use case I have had in the past: I've auto-generated complex
> C++ codes with dimension-specific numbers of loops and other
> parameters of a problem, to produce objects which, once instantiated,
> had a few key methods run very fast. ?This proved quite easy to do
> with weave, because I could compute the code as a string with nice
> python code at object initialization time (when the
> dimensions/parameters became known), and then ?during the object's
> life it would always use the compiled version. ?So while 'coding in
> strings' is in most situations awful, in this case it actually was
> very handy to have the flexibility to use string-generated code and
> weave handle all the gory details for me.

:) I know all about auto-generated code.

> I hope this is useful, sorry for not having provided these details before.

Yes very. I got a bit of this in our conversation a couple of weeks
back, but thanks for putting this down in a concrete form. (Also,
there's nothing like actually seeing code.)

- Robert

From robertwb at math.washington.edu  Thu Jul 15 07:27:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 14 Jul 2010 22:27:57 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: 
References: 
	<4C3E00AB.5000704@student.matnat.uio.no>
	
	
Message-ID: 

On Wed, Jul 14, 2010 at 9:49 PM, Kurt Smith  wrote:
> On Wed, Jul 14, 2010 at 9:09 PM, Robert Bradshaw
>  wrote:
>>
>> Though I can see the advantages of weave, I think it'd be even better
>> if one didin't have to switch back and forth between completely
>> unrelated syntaxes/types/etc. Do people use weave because there are
>> certain things that are more natural to express in Fortran (and/or C),
>> or is it primarily a convenient way to inline compiled code? Still
>> waiting for someone to put up a wiki page on what they like from weave
>> and what should go into Cython.
>
> I'll get to it soon; but this thread serves to get the ideas out there
> in raw form.
>
> My impressions are from using weave a bit in the past and from those
> who have mentioned interest in keeping it alive with a Cython
> transplant (primarily Fernando). ?I welcome Fernando (& others) to
> chime in with the other killer features that weave offers. ?And
> critics of the weave 'approach' are more than weclome, too :-)
>
> (Note: I've only used weave, but much of what I say applies to blitz, too.)
>
> 1) For an existing function that has to be sped up, weave has a lower
> barrier to usage than Cython, even with pyximport. ?(pyximport doesn't
> enable an equivalent functionality as weave, so it isn't exactly
> germane). ?It is very nice to just drop into C/C++ in the middle of a
> function and have things compiled & run automatically. ?This is in
> large part because weave allows incremental usage wherever needed on a
> function-by-function basis, whereas Cython's smallest compilation unit
> is the module.

We'd certainly like to support function-level compilation (though
there are technical questions about how to get the source) and perhaps
even line-by-line via strings. Right now the Sage notebook gives this
feel for Cython, which is probably why I haven't felt the pinch (I
rarely write and invoke a setup.py manually).

> 2) I liked weave because you know you're getting to the metal with no
> more than 5-10 lines of setup code, and with absolutely no fuss with
> external tools or having to create a setup.py. ?This may sound lazy,
> but it really is nice not to have to deal with these things.
> Sometimes numpy array expressions aren't the most natural way to
> express something, and you just want to cut through it all and get a
> simple, fast C 'for' loop in there and move on, but cythonizing would
> be too big of a step, or impossible if you have an inner function
> somewhere else in the module, for example. ?Weave scratches the itch
> very nicely.

Yep. If I didn't use the notebook, I would also share your pain.

> 3) The C-source-in-a-string is pretty ugly, I admit. ?Weave is very
> far on the "practicality" side of the "practicality vs. purity" scale.
> ?It is a small benefit, though, to have the C/C++ code quarantined to
> one place in a function, and to always know exactly what to expect
> from each part of your code -- python remains python, C remains C.
> The fluidity of Cython is a tremendous advantage, but it does require
> some care to be sure you're typing everything that needs to be typed,
> turning off boundschecking when it's safe, etc. ?I tend to use 'cython
> -a' or inspecting the generated source to be sure I'm "totally C"
> fairly often. ?Weave gives this to you automatically. ?In short, with
> weave, I don't have to pay attention as much to get the same degree of
> performance. ?The tradeoff is ugly code.

That's a good point--we should be a lot closer with type inference
(and especially if we do the trick of compiling/specializing the code
at runtime to the input types) but the fact that untyped code is just
slower rather than broken is an interesting one. Almost makes me want
to have a "C only" flag or something like that.

> 4) Dag mentioned allowing Cython code in the source string. This
> certainly has merit and I'm somewhat in favor of it. ?It would avoid
> reinventing the wheel for the array indexing stuff, and would have a
> more pythonic feel. ?But it would negate some of the benefits of
> having "just C" in the source string.

For those who like C. (Well, and the above point.)

> 6) Up until now I've been assuming that this functionality will be
> implemented as an external tool. ?Perhaps you (Robert, Dag, others)
> have thoughts on getting some subset of weave's features in Cython
> itself, namely the function-level compilation of a chunk of code.
> Robert mentions this below with a @cython.compile decorator. ?And the
> 'cython.eval()' has all the essential features of
> weave/blitz right there, including the ugliness ;-)

The whole build system question is a mess--one that I'm hoping fwrap
can solve nicely :). It'd be nice if whatever comes out of it the
semantics of inline C/C++/Fortran/Cython all had somewhat the same
semantics, and possibly even shared some of the same code.

> So, remaining issues:
>
> -) Essential features of weave/blitz that I haven't mentioned.
>
> -) Is this thing better off as an external tool, or integrated in cython itself?
> ? ?If the source string is cython code, then it's better as an
> internal-to-cython tool. ?If C, then an external tool would suffice.

I could see Cython as a "plugin" to a system that does this, aside
C/C++/Fortran/??? It would take source and context, and return a
callable/run the code. OTOH, some of the simpler features may be
easier to put right into Cython itself. Certainly we want to make it
easier to use Cython without having to fudge with all the build system
stuff (which is, quite honestly, probably the hardest part of learning
to use Cython).

> -) I'll summarize this, make it less verbose, and put it on the wiki.

That would be great. One we have something, others can add their
killer/pet feature.

- Robert

From kwmsmith at gmail.com  Thu Jul 15 08:02:32 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 15 Jul 2010 01:02:32 -0500
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
	
	<4C3D362E.80901@molden.no>
	
	
	
	
	
	
Message-ID: 

On Wed, Jul 14, 2010 at 10:47 PM, Robert Bradshaw
 wrote:
> On Wed, Jul 14, 2010 at 7:47 PM, Carl Witty  wrote:
>> On Wed, Jul 14, 2010 at 6:59 PM, Robert Bradshaw
>>  wrote:
>>>> In my experience, I always have to resort to Fortran or C++. Cython cannot
>>>> use NumPy arrays efficiently as function arguments. That is a big show stopper.
>>>
>>> Could you clarify? I suppose for very small arrays, there's the extra
>>> O(1) type-check/stride extraction overhead. Is there anything else? Of
>>> course there's always more room for improvement for the array type.
>>
>> Sometimes I write functions that don't iterate over every element of
>> their argument arrays :)
>
> Good point. How does Fortran handle this better? Does it just know
> that the strides are not aliased? (Or, does it know dimensions/other
> data at compile time.)

You can do the following in Fortran 90 (if this is what Carl was referring to):

do i = 1, n
    lharr(i, :) = m * rharr(i, :) + b
enddo

Where 'm' and 'b' can be scalars or 1D arrays of the proper length.
Think numpy-arrays-lite, including simple broadcasting rules and array
expressions, and much faster.

Kurt

From dagss at student.matnat.uio.no  Thu Jul 15 10:24:08 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 15 Jul 2010 10:24:08 +0200
Subject: [Cython] Cython weave thread,
 was: "Impressions from EuroScipy 	2010"
In-Reply-To: 
References: 	<4C3E00AB.5000704@student.matnat.uio.no>		
	
Message-ID: <4C3EC5A8.3040202@student.matnat.uio.no>

On 07/15/2010 07:27 AM, Robert Bradshaw wrote:
> On Wed, Jul 14, 2010 at 9:49 PM, Kurt Smith  wrote:
>    
>> On Wed, Jul 14, 2010 at 9:09 PM, Robert Bradshaw
>>   wrote:
>>      
>>> Though I can see the advantages of weave, I think it'd be even better
>>> if one didin't have to switch back and forth between completely
>>> unrelated syntaxes/types/etc. Do people use weave because there are
>>> certain things that are more natural to express in Fortran (and/or C),
>>> or is it primarily a convenient way to inline compiled code? Still
>>> waiting for someone to put up a wiki page on what they like from weave
>>> and what should go into Cython.
>>>        
>> I'll get to it soon; but this thread serves to get the ideas out there
>> in raw form.
>>
>> My impressions are from using weave a bit in the past and from those
>> who have mentioned interest in keeping it alive with a Cython
>> transplant (primarily Fernando).  I welcome Fernando (&  others) to
>> chime in with the other killer features that weave offers.  And
>> critics of the weave 'approach' are more than weclome, too :-)
>>
>> (Note: I've only used weave, but much of what I say applies to blitz, too.)
>>
>> 1) For an existing function that has to be sped up, weave has a lower
>> barrier to usage than Cython, even with pyximport.  (pyximport doesn't
>> enable an equivalent functionality as weave, so it isn't exactly
>> germane).  It is very nice to just drop into C/C++ in the middle of a
>> function and have things compiled&  run automatically.  This is in
>> large part because weave allows incremental usage wherever needed on a
>> function-by-function basis, whereas Cython's smallest compilation unit
>> is the module.
>>      
> We'd certainly like to support function-level compilation (though
> there are technical questions about how to get the source) and perhaps
> even line-by-line via strings. Right now the Sage notebook gives this
> feel for Cython, which is probably why I haven't felt the pinch (I
> rarely write and invoke a setup.py manually).
>    

Here's some work which gets the source from decorated functions (in pure 
Python...the decorator could extract source, compile with Cython, and 
replace the decorated function with one imported from the compiled one):

http://github.com/GaelVaroquaux/joblib/blob/master/joblib/func_inspect.py

Of course this requires a stronger pure Python mode.
>> 3) The C-source-in-a-string is pretty ugly, I admit.  Weave is very
>> far on the "practicality" side of the "practicality vs. purity" scale.
>>   It is a small benefit, though, to have the C/C++ code quarantined to
>> one place in a function, and to always know exactly what to expect
>> from each part of your code -- python remains python, C remains C.
>> The fluidity of Cython is a tremendous advantage, but it does require
>> some care to be sure you're typing everything that needs to be typed,
>> turning off boundschecking when it's safe, etc.  I tend to use 'cython
>> -a' or inspecting the generated source to be sure I'm "totally C"
>> fairly often.  Weave gives this to you automatically.  In short, with
>> weave, I don't have to pay attention as much to get the same degree of
>> performance.  The tradeoff is ugly code.
>>      
> That's a good point--we should be a lot closer with type inference
> (and especially if we do the trick of compiling/specializing the code
> at runtime to the input types) but the fact that untyped code is just
> slower rather than broken is an interesting one. Almost makes me want
> to have a "C only" flag or something like that.
>    
We already discussed having a decorator for disallowing untyped 
variables. Basically you would need to type every variable in a 
function, but typing as "object" would be OK. I think this is more 
useful than "C only" in this context.

Dag Sverre

From dagss at student.matnat.uio.no  Thu Jul 15 13:26:46 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 15 Jul 2010 13:26:46 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>					
	
Message-ID: <4C3EF076.5060708@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Wed, Jul 14, 2010 at 7:47 PM, Carl Witty  wrote:
>   
>> On Wed, Jul 14, 2010 at 6:59 PM, Robert Bradshaw
>>  wrote:
>>     
>>>> In my experience, I always have to resort to Fortran or C++. Cython cannot
>>>> use NumPy arrays efficiently as function arguments. That is a big show stopper.
>>>>         
>>> Could you clarify? I suppose for very small arrays, there's the extra
>>> O(1) type-check/stride extraction overhead. Is there anything else? Of
>>> course there's always more room for improvement for the array type.
>>>       
>> Sometimes I write functions that don't iterate over every element of
>> their argument arrays :)
>>     
>
> Good point. How does Fortran handle this better? Does it just know
> that the strides are not aliased? (Or, does it know dimensions/other
> data at compile time.)
>   
The problem in Cython is that one must reacquire the buffer if it is 
passed as a Python object, which has some overhead. Fortran just passes 
a pointer and associated shape.
>   
>> In particular, I remember that when I was writing
>> sage/plot/plot3d/implicit_surface.pyx, I wanted to make several chunks
>> of code into methods but decided not to for efficiency reasons.  (I'm
>> not sure if I actually measured, or if I was just worried.)
>>     
>
> For reference, http://hg.sagemath.org/sage-main/file/8dec8b43ccca/sage/plot/plot3d/implicit_surface.pyx#l1
>
>   
>> If I was doing it again, I think I'd give up on the Cython numpy
>> support and just use C pointers in that code...
>>     
>
> Clearly we need to be doing thing better :).
>   
Definitely, this is the biggest long-standing complaint. I find myself 
working around it many times. This is what the kurt-gsoc branch is 
about. I just need time for it that I don't have (and nobody else has 
stepped up).

Dag Sverre

From ndbecker2 at gmail.com  Thu Jul 15 16:24:07 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 15 Jul 2010 10:24:07 -0400
Subject: [Cython] status of exposing numpy array to c++
Message-ID: 

What is the current state of using cython to expose numpy arrays to c++ 
code?

I have the (typical) setup of python for high-level code, c++ for 
specialized algorithms.  I'm using numpy as the containers.

My c++ algorithms are usually written to a generic c++ (boost::range-based) 
interface.

Currently, the best solution I know of is to use boost::python and pyublas 
to interface.

Are there any cython examples?  Has any of the recent cython work made 
cython more attractive for this use?


From robertwb at math.washington.edu  Thu Jul 15 18:48:25 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 09:48:25 -0700
Subject: [Cython] status of exposing numpy array to c++
In-Reply-To: 
References: 
Message-ID: 

On Thu, Jul 15, 2010 at 7:24 AM, Neal Becker  wrote:
> What is the current state of using cython to expose numpy arrays to c++
> code?
>
> I have the (typical) setup of python for high-level code, c++ for
> specialized algorithms. ?I'm using numpy as the containers.
>
> My c++ algorithms are usually written to a generic c++ (boost::range-based)
> interface.
>
> Currently, the best solution I know of is to use boost::python and pyublas
> to interface.
>
> Are there any cython examples? ?Has any of the recent cython work made
> cython more attractive for this use?

Are you trying to create C++ wrappers for NumPy arrays? In this case
Cython is almost certainly the wrong tool. The new C++ features allow
you to more easily use C++ libraries,  not write arbitrary C++ code.

- Robert

From sturla at molden.no  Thu Jul 15 18:54:40 2010
From: sturla at molden.no (Sturla Molden)
Date: Thu, 15 Jul 2010 18:54:40 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>			
	
Message-ID: <4C3F3D50.30708@molden.no>

Den 15.07.2010 03:59, skrev Robert Bradshaw:
>
>> In my experience, I always have to resort to Fortran or C++. Cython cannot
>> use NumPy arrays efficiently as function arguments. That is a big show stopper.
>>      
>    

> Could you clarify? I suppose for very small arrays, there's the extra
> O(1) type-check/stride extraction overhead. Is there anything else? Of
> course there's always more room for improvement for the array type.
>
>    

First there is always the overhead of refcounting and unboxing the 
buffer, even if we pass the array unmodified. Second, if we want to pass 
a slice/subarray (a very common case), we get:

- creation of a Python slice object
- more refcounting
- a Python call to the (unoptimized) [] operator
- recounting
- creation of a new Python numpy.ndarray object
- more refcoutning
- passing the new view array to the function
- more refcounting
- unboxing the buffer
- etc.

Imagine this happening in a tight loop. It's close to useless. (Yes, and 
we cannot release the GIL, but that's a minor point.)

To get the performance we need, we have to use C pointers instead, e.g. 
pass &(array[n,0,0]) instead of array[n,:,:]. But while array[n,:,:] has 
two dimensions, the C pointer &(array[n,0,0]) only has only one. A C 
pointer also carries no information about the subarray shape. So 
numerical Cython code quickly degenerates to C.

In Fortran 95, we have arrays that works very similarly to NumPy. 
array[n,:,:] in NumPy is array(:,:,n) in Fortran 95 (assuming C order in 
NumPy.) Fortran 95 arrays boradcast like NumPy: a*b in NumPy is a*b in 
Fortran 95. Arrays can be repeated, transposed, and reshaped; they can 
be queried for size and shape; indexed with integer of boolean arrays; 
there are numerical functions in the RTL than work like ufuncs (sin, 
cos, etc.). We can create ufunc-like functions using the "elemental" 
keyword. Fortran has all the niceness of NumPy, but with the speed of C 
(often better).  Also, array expressions in Fortran 95 are easy to 
auto-vectorize to use simd extensions or multithreading. We can also use 
OpenMP directly in Fortran. There is a "where" construct that 
corresponds to masked "fancy indexing" in NumPy (or numpy.where), and a 
"forall" keyword that lets the compiler implement nested loops more 
efficiently (i.e. no specific order of loop counters are requested, so 
it can easily be vectorized to simd or multithreading.)

Fortran 95 is easy to read and just as powerful as C. I is nothing like 
Fortran 77, which is commonly understood by "Fortran" among C/C++ 
developers. Unlike Fortran 77, there are dynamic allocation, and 
pointers which are more safe than C pointers. Fortran 95 pointers can 
even reference a strided subarray (just like a slice returns a view in 
NumPy), and carry shape information about the subarray. Fortran 95 code 
can be organized into modules (that are compiled separately), and one 
module can import another, just like Python. (Fortran 2003 will have OOP 
with inheritance and polymorphism as well, but compiler vendors are 
still catching up.) So there is a reason why scientists are keeping 
Fortran alive. Computer scientists don't understand it, but they usually 
have no idea how important array processing is to scientific computing. 
And they generally believe Fortran today is still FORTRAN 77 and g77, 
which is a ccompletely false assumption.

Just consider what this would do in Cython

    for i in range(N):
       res[i] = foobar(b0*array0[i::2,:,:] + b1*array1[i,:,::2] + 
b2*array2[i,:,:])

and compare with Fortran 95:

    do i = 1,n
       res(i) = foobar(b0*array0(:,:,i::2) + b1*array1(::2,:,i) + 
b2*array2(:,:,i))
    end do

The differences are () instead of [], do instead of for, and indexes are 
usually tranposed (due to column-major ordering). But the Fortran 95 
code would run as fast as hand-written C, while the Cython would hardly 
be any better than Python.

Sturla




From sturla at molden.no  Thu Jul 15 19:05:28 2010
From: sturla at molden.no (Sturla Molden)
Date: Thu, 15 Jul 2010 19:05:28 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>					
	
Message-ID: <4C3F3FD8.3000203@molden.no>

Den 15.07.2010 05:47, skrev Robert Bradshaw:
> Good point. How does Fortran handle this better? Does it just know
> that the strides are not aliased? (Or, does it know dimensions/other
> data at compile time.)
>
>    

Fortran generally disallow aliasing, though the compilers don't enforce 
it, they just assume it never happens. However, if an array is declared 
"pointer" or "target" (i.e. target for a pointer), the Fortran compiler 
will know that arguments can be aliased and be more careful (e.g. it can 
make temporary copies).

Fortran arrays and pointers are structs that have information about 
strides and shapes (aka "dope vectors"), just like NumPy arrays.

Sturla


From robertwb at math.washington.edu  Thu Jul 15 19:06:47 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 10:06:47 -0700
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: <4C3F3D50.30708@molden.no>
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
	
	<4C3D362E.80901@molden.no>
	
	
	
	
	<4C3F3D50.30708@molden.no>
Message-ID: 

On Thu, Jul 15, 2010 at 9:54 AM, Sturla Molden  wrote:
> Den 15.07.2010 03:59, skrev Robert Bradshaw:
>>
>>> In my experience, I always have to resort to Fortran or C++. Cython cannot
>>> use NumPy arrays efficiently as function arguments. That is a big show stopper.
>>>
>>
>
>> Could you clarify? I suppose for very small arrays, there's the extra
>> O(1) type-check/stride extraction overhead. Is there anything else? Of
>> course there's always more room for improvement for the array type.
>>
>>
>
> First there is always the overhead of refcounting and unboxing the
> buffer, even if we pass the array unmodified. Second, if we want to pass
> a slice/subarray (a very common case), we get:
>
> - creation of a Python slice object
> - more refcounting
> - a Python call to the (unoptimized) [] operator
> - recounting
> - creation of a new Python numpy.ndarray object
> - more refcoutning
> - passing the new view array to the function
> - more refcounting
> - unboxing the buffer
> - etc.
>
> Imagine this happening in a tight loop. It's close to useless. (Yes, and
> we cannot release the GIL, but that's a minor point.)
>
> To get the performance we need, we have to use C pointers instead, e.g.
> pass &(array[n,0,0]) instead of array[n,:,:]. But while array[n,:,:] has
> two dimensions, the C pointer &(array[n,0,0]) only has only one. A C
> pointer also carries no information about the subarray shape. So
> numerical Cython code quickly degenerates to C.
>
> In Fortran 95, we have arrays that works very similarly to NumPy.
> array[n,:,:] in NumPy is array(:,:,n) in Fortran 95 (assuming C order in
> NumPy.) Fortran 95 arrays boradcast like NumPy: a*b in NumPy is a*b in
> Fortran 95. Arrays can be repeated, transposed, and reshaped; they can
> be queried for size and shape; indexed with integer of boolean arrays;
> there are numerical functions in the RTL than work like ufuncs (sin,
> cos, etc.). We can create ufunc-like functions using the "elemental"
> keyword. Fortran has all the niceness of NumPy, but with the speed of C
> (often better). ?Also, array expressions in Fortran 95 are easy to
> auto-vectorize to use simd extensions or multithreading. We can also use
> OpenMP directly in Fortran. There is a "where" construct that
> corresponds to masked "fancy indexing" in NumPy (or numpy.where), and a
> "forall" keyword that lets the compiler implement nested loops more
> efficiently (i.e. no specific order of loop counters are requested, so
> it can easily be vectorized to simd or multithreading.)
>
> Fortran 95 is easy to read and just as powerful as C. I is nothing like
> Fortran 77, which is commonly understood by "Fortran" among C/C++
> developers. Unlike Fortran 77, there are dynamic allocation, and
> pointers which are more safe than C pointers. Fortran 95 pointers can
> even reference a strided subarray (just like a slice returns a view in
> NumPy), and carry shape information about the subarray. Fortran 95 code
> can be organized into modules (that are compiled separately), and one
> module can import another, just like Python. (Fortran 2003 will have OOP
> with inheritance and polymorphism as well, but compiler vendors are
> still catching up.) So there is a reason why scientists are keeping
> Fortran alive. Computer scientists don't understand it, but they usually
> have no idea how important array processing is to scientific computing.
> And they generally believe Fortran today is still FORTRAN 77 and g77,
> which is a ccompletely false assumption.

Well, many projects are still using Fortran 77 (just like many C
projects still are based on on c89 rather than moving to c99). It does
make me happy to hear that the language is (will be?) catching up to
modern times, but the lag time of old projects moving to new standards
is often measured in decades (!).

Totally agree with you about array processing being undervalued, IMHO
this is one of the primary long-term advantages of Fortran.

> Just consider what this would do in Cython
>
> ? ?for i in range(N):
> ? ? ? res[i] = foobar(b0*array0[i::2,:,:] + b1*array1[i,:,::2] +
> b2*array2[i,:,:])
>
> and compare with Fortran 95:
>
> ? ?do i = 1,n
> ? ? ? res(i) = foobar(b0*array0(:,:,i::2) + b1*array1(::2,:,i) +
> b2*array2(:,:,i))
> ? ?end do
>
> The differences are () instead of [], do instead of for, and indexes are
> usually tranposed (due to column-major ordering). But the Fortran 95
> code would run as fast as hand-written C, while the Cython would hardly
> be any better than Python.

Yep. If the arrays are large, the speed difference should be
negligible, but there's a huge penalty in object allocation for small
ones.

It sounds like what we need is a non-object-backed array (e.g. a raw
NumPy array struct) where some of the metadata (e.g. type) is carried
around by the compiler rather than at runtime. The question here is
how to handle memory allocation--is this done manually in Fortran? I
suppose slices could just be structs passed around by value.

- Robert

From ndbecker2 at gmail.com  Thu Jul 15 19:22:01 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 15 Jul 2010 13:22:01 -0400
Subject: [Cython] status of exposing numpy array to c++
References: 
	
Message-ID: 

Robert Bradshaw wrote:

> On Thu, Jul 15, 2010 at 7:24 AM, Neal Becker
>  wrote:
>> What is the current state of using cython to expose numpy arrays to c++
>> code?
>>
>> I have the (typical) setup of python for high-level code, c++ for
>> specialized algorithms.  I'm using numpy as the containers.
>>
>> My c++ algorithms are usually written to a generic c++
>> (boost::range-based) interface.
>>
>> Currently, the best solution I know of is to use boost::python and
>> pyublas to interface.
>>
>> Are there any cython examples?  Has any of the recent cython work made
>> cython more attractive for this use?
> 
> Are you trying to create C++ wrappers for NumPy arrays? In this case
> Cython is almost certainly the wrong tool. The new C++ features allow
> you to more easily use C++ libraries,  not write arbitrary C++ code.
> 
> - Robert

No, I already have lots of algorithms written in c++ for a generic range-
based interface (templated).  These can be instantiated to use with any 
container types that fit the range concept.

Right now I'm instantiating them using pyublas::numpy_vector as the 
containers, which allows me to use them from python with numpy.

For example, consider this c++ algorithm:

template
inline out_t repeat (in_t const& in, int cnt) {
  out_t out (boost::size (in)*cnt);
  typename boost::range_const_iterator::type i = boost::begin (in);
  typename boost::range_iterator::type o = boost::begin (out);

  for (; i != boost::end (in); ++i, o+=cnt) {
    std::fill (o, (o+cnt), *i);
  }

  return out;
}

This could be used on a numpy vector, instantiated as:

repeat,pyublas::numpy_vector >

Could cython be used to instantiate and call this function, which takes
a numpy vector as input/output?


From njs at pobox.com  Thu Jul 15 19:23:14 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 15 Jul 2010 10:23:14 -0700
Subject: [Cython] status of exposing numpy array to c++
In-Reply-To: 
References: 
	
Message-ID: 

On Thu, Jul 15, 2010 at 9:48 AM, Robert Bradshaw
 wrote:
> Are you trying to create C++ wrappers for NumPy arrays? In this case
> Cython is almost certainly the wrong tool. The new C++ features allow
> you to more easily use C++ libraries, ?not write arbitrary C++ code.

But there are some reasonable C++ numerics libraries -- e.g., Eigen --
and a bit of glue code to map ndarray's into a C++ library's array
types and then working with those directly in Cython seems like it
would be a sweet spot for some of the things people are trying to do
in these threads.

-- Nathaniel

From stefan_ml at behnel.de  Thu Jul 15 19:34:08 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 15 Jul 2010 19:34:08 +0200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: 
References: 
	<4C3D80AB.2070003@behnel.de>
	
Message-ID: <4C3F4690.3030300@behnel.de>

Craig, I hope you don't mind if I send my reply to cython-dev.

Craig Citro, 14.07.2010 22:45:
> So Robert and I chatted about this over lunch -- we came up with a
> very different fix, which was along these lines: we want to do a
> source->source transformation (actually implemented with
> ParseTreeFragments) early on in the pipeline, and then let the type
> analysis/temp allocation/etc get handled automagically.

This can be done in some but not many cases, and it's not easy. One of the 
problems is that some information that has an impact on the optimisation is 
context specific and is just not known before type analysis. In most cases, 
nodes and their results have to be reused, and there isn't currently a good 
way to do that, especially not before type inference and analysis, which 
may end up wanting to assign different types to the same node result in the 
different contexts.

I had started thinking about this more than once already. There's a 
transform called EarlyReplaceBuiltinCalls, for example, which tries to do 
some of the tree restructuring before type analysis, but all(?) of the 
later optimisations in OptimizeBuiltinCalls just don't fit in there, as 
they need too many details about the types they work on.

The min/max issue at hand is also a serious one, as it shows that the reuse 
of node results can be truly hard to get right w.r.t. ref-counting across 
multiple reassignments from different types.

That being said, any additional transformation that we can move before type 
analysis is certainly worth it, and any infrastructure that we can build up 
for that is a major step forward.

Also note that we decided to modify the type analysis step into a 
transform-like operation that is allowed to replace the current node based 
on the type that was determined for the original AST nodes. This would 
actually combine the best of both worlds: enough type information to take a 
decision, but at a point where the rest of the type analysis can still 
react on it.


> Robert also
> noticed a clever way to thread the type information back through to
> avoid extra conversions, which would be another bonus.

Happy to hear more.

Stefan

From robert.kern at gmail.com  Thu Jul 15 19:38:54 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 15 Jul 2010 12:38:54 -0500
Subject: [Cython] On the proper use of object
Message-ID: 

What are the recommendations for the use of the "object" type versus "PyObject 
*"? Ondrej pointed out a potential bug in my line_profiler package:

 
http://github.com/certik/line_profiler/commit/dc3ad198f8a810d34da5da71ffccf01c5e05a161

I have a cdef function that serves as the callback for the PyEval_SetTrace() 
functionality. It has a PyObject* argument that is sometimes NULL. This argument 
happens to be unused. In the version of Cython I originally developed 
line_profile under , no code referenced this argument. In the development 
version of Cython that Ondrej used to compile line_profiler, a Py_INCREF is 
apparently generated. When the argument is NULL, this obviously fails.

Is this intentional? Should "object" be avoided when the argument could possibly 
be NULL? Or should the generated code use Py_XINCREF/Py_XDECREF instead to avoid 
failure?

Thanks.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From stefan_ml at behnel.de  Thu Jul 15 19:43:09 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 15 Jul 2010 19:43:09 +0200
Subject: [Cython] On the proper use of object
In-Reply-To: 
References: 
Message-ID: <4C3F48AD.8070501@behnel.de>

Hi, note that this is a question for the cython-users mailing list.

Robert Kern, 15.07.2010 19:38:
> What are the recommendations for the use of the "object" type versus "PyObject
> *"? Ondrej pointed out a potential bug in my line_profiler package:
>
> http://github.com/certik/line_profiler/commit/dc3ad198f8a810d34da5da71ffccf01c5e05a161
>
> I have a cdef function that serves as the callback for the PyEval_SetTrace()
> functionality. It has a PyObject* argument that is sometimes NULL. This argument
> happens to be unused. In the version of Cython I originally developed
> line_profile under , no code referenced this argument. In the development
> version of Cython that Ondrej used to compile line_profiler, a Py_INCREF is
> apparently generated. When the argument is NULL, this obviously fails.
>
> Is this intentional? Should "object" be avoided when the argument could possibly
> be NULL? Or should the generated code use Py_XINCREF/Py_XDECREF instead to avoid
> failure?

In Cython, object and PyObject* are not compatible. The first is an 
automatically ref-counted object, whereas the second is a plain pointer. 
For an argument that can be NULL, you cannot use object.

Stefan

From robertwb at math.washington.edu  Thu Jul 15 19:56:24 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 10:56:24 -0700
Subject: [Cython] On the proper use of object
In-Reply-To: 
References: 
Message-ID: 

On Thu, Jul 15, 2010 at 10:38 AM, Robert Kern  wrote:
> What are the recommendations for the use of the "object" type versus "PyObject
> *"? Ondrej pointed out a potential bug in my line_profiler package:
>
>
> http://github.com/certik/line_profiler/commit/dc3ad198f8a810d34da5da71ffccf01c5e05a161
>
> I have a cdef function that serves as the callback for the PyEval_SetTrace()
> functionality. It has a PyObject* argument that is sometimes NULL. This argument
> happens to be unused. In the version of Cython I originally developed
> line_profile under , no code referenced this argument. In the development
> version of Cython that Ondrej used to compile line_profiler, a Py_INCREF is
> apparently generated. When the argument is NULL, this obviously fails.
>
> Is this intentional? Should "object" be avoided when the argument could possibly
> be NULL? Or should the generated code use Py_XINCREF/Py_XDECREF instead to avoid
> failure?

Typically PyObject* is used for borrowed references (i.e there's no
need for you do decref it when you're done with it) whereas objects is
used when you have ownership of a reference and need to handle the
decref at the end. As for NULL, it is fine in the former (though be
careful to use the Py_X... functions if necessary) but not in the
latter.

In this callback, the argument should be declared to take a PyObject*,
and if you need to use it, cast it to an object if it's not NULL
(maybe using None if it is).

- Robert

From robertwb at math.washington.edu  Thu Jul 15 20:01:53 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 11:01:53 -0700
Subject: [Cython] status of exposing numpy array to c++
In-Reply-To: 
References: 
	
	
Message-ID: 

On Thu, Jul 15, 2010 at 10:22 AM, Neal Becker  wrote:
> Robert Bradshaw wrote:
>
>> On Thu, Jul 15, 2010 at 7:24 AM, Neal Becker
>>  wrote:
>>> What is the current state of using cython to expose numpy arrays to c++
>>> code?
>>>
>>> I have the (typical) setup of python for high-level code, c++ for
>>> specialized algorithms. ?I'm using numpy as the containers.
>>>
>>> My c++ algorithms are usually written to a generic c++
>>> (boost::range-based) interface.
>>>
>>> Currently, the best solution I know of is to use boost::python and
>>> pyublas to interface.
>>>
>>> Are there any cython examples? ?Has any of the recent cython work made
>>> cython more attractive for this use?
>>
>> Are you trying to create C++ wrappers for NumPy arrays? In this case
>> Cython is almost certainly the wrong tool. The new C++ features allow
>> you to more easily use C++ libraries, ?not write arbitrary C++ code.
>>
>> - Robert
>
> No, I already have lots of algorithms written in c++ for a generic range-
> based interface (templated). ?These can be instantiated to use with any
> container types that fit the range concept.
>
> Right now I'm instantiating them using pyublas::numpy_vector as the
> containers, which allows me to use them from python with numpy.
>
> For example, consider this c++ algorithm:
>
> template
> inline out_t repeat (in_t const& in, int cnt) {
> ?out_t out (boost::size (in)*cnt);
> ?typename boost::range_const_iterator::type i = boost::begin (in);
> ?typename boost::range_iterator::type o = boost::begin (out);
>
> ?for (; i != boost::end (in); ++i, o+=cnt) {
> ? ?std::fill (o, (o+cnt), *i);
> ?}
>
> ?return out;
> }
>
> This could be used on a numpy vector, instantiated as:
>
> repeat,pyublas::numpy_vector >
>
> Could cython be used to instantiate and call this function, which takes
> a numpy vector as input/output?

Sorry I completely misunderstood your question. Yes, this is something
you could do.

- Robert

From ondrej at certik.cz  Thu Jul 15 20:03:53 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 15 Jul 2010 11:03:53 -0700
Subject: [Cython] On the proper use of object
In-Reply-To: 
References: 
	
Message-ID: 

On Thu, Jul 15, 2010 at 10:56 AM, Robert Bradshaw
 wrote:
> On Thu, Jul 15, 2010 at 10:38 AM, Robert Kern  wrote:
>> What are the recommendations for the use of the "object" type versus "PyObject
>> *"? Ondrej pointed out a potential bug in my line_profiler package:
>>
>>
>> http://github.com/certik/line_profiler/commit/dc3ad198f8a810d34da5da71ffccf01c5e05a161
>>
>> I have a cdef function that serves as the callback for the PyEval_SetTrace()
>> functionality. It has a PyObject* argument that is sometimes NULL. This argument
>> happens to be unused. In the version of Cython I originally developed
>> line_profile under , no code referenced this argument. In the development
>> version of Cython that Ondrej used to compile line_profiler, a Py_INCREF is
>> apparently generated. When the argument is NULL, this obviously fails.
>>
>> Is this intentional? Should "object" be avoided when the argument could possibly
>> be NULL? Or should the generated code use Py_XINCREF/Py_XDECREF instead to avoid
>> failure?
>
> Typically PyObject* is used for borrowed references (i.e there's no
> need for you do decref it when you're done with it) whereas objects is
> used when you have ownership of a reference and need to handle the
> decref at the end. As for NULL, it is fine in the former (though be
> careful to use the Py_X... functions if necessary) but not in the
> latter.
>
> In this callback, the argument should be declared to take a PyObject*,
> and if you need to use it, cast it to an object if it's not NULL
> (maybe using None if it is).


Thanks for the replies. I just happened to be running the latest
cython-devel and I was horrified that the profiler just segfaulted for
me. Fortunately it was trivial to find the problem using gdb.

Ondrej

From robert.kern at gmail.com  Thu Jul 15 20:07:29 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 15 Jul 2010 13:07:29 -0500
Subject: [Cython] On the proper use of object
In-Reply-To: <4C3F48AD.8070501@behnel.de>
References:  <4C3F48AD.8070501@behnel.de>
Message-ID: 

On 7/15/10 12:43 PM, Stefan Behnel wrote:
> Hi, note that this is a question for the cython-users mailing list.

Sorry. Since I am also questioning the implementation, this seemed like a 
reasonable place. :-)

> Robert Kern, 15.07.2010 19:38:
>> What are the recommendations for the use of the "object" type versus "PyObject
>> *"? Ondrej pointed out a potential bug in my line_profiler package:
>>
>> http://github.com/certik/line_profiler/commit/dc3ad198f8a810d34da5da71ffccf01c5e05a161
>>
>> I have a cdef function that serves as the callback for the PyEval_SetTrace()
>> functionality. It has a PyObject* argument that is sometimes NULL. This argument
>> happens to be unused. In the version of Cython I originally developed
>> line_profile under , no code referenced this argument. In the development
>> version of Cython that Ondrej used to compile line_profiler, a Py_INCREF is
>> apparently generated. When the argument is NULL, this obviously fails.
>>
>> Is this intentional? Should "object" be avoided when the argument could possibly
>> be NULL? Or should the generated code use Py_XINCREF/Py_XDECREF instead to avoid
>> failure?
>
> In Cython, object and PyObject* are not compatible. The first is an
> automatically ref-counted object, whereas the second is a plain pointer.
> For an argument that can be NULL, you cannot use object.

Would using Py_XINCREF/Py_XDECREF allow the automatic refcounting to simply 
ignore NULL pointers but still work for the usual case? Or would there be other 
issues?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From dagss at student.matnat.uio.no  Thu Jul 15 20:12:57 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 15 Jul 2010 20:12:57 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>					<4C3F3D50.30708@molden.no>
	
Message-ID: <4C3F4FA9.5090908@student.matnat.uio.no>

On 07/15/2010 07:06 PM, Robert Bradshaw wrote:
>
> It sounds like what we need is a non-object-backed array (e.g. a raw
> NumPy array struct) where some of the metadata (e.g. type) is carried
> around by the compiler rather than at runtime. The question here is
> how to handle memory allocation--is this done manually in Fortran? I
> suppose slices could just be structs passed around by value.
>    

Which, BTW, is EXACTLY what the int[:,:]-type and the kurt-gsoc branch 
is all about.

Dag Sverre

From robertwb at math.washington.edu  Thu Jul 15 20:10:12 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 11:10:12 -0700
Subject: [Cython] On the proper use of object
In-Reply-To: 
References:  <4C3F48AD.8070501@behnel.de>
	
Message-ID: 

On Thu, Jul 15, 2010 at 11:07 AM, Robert Kern  wrote:
> On 7/15/10 12:43 PM, Stefan Behnel wrote:
>> Hi, note that this is a question for the cython-users mailing list.
>
> Sorry. Since I am also questioning the implementation, this seemed like a
> reasonable place. :-)
>
>> Robert Kern, 15.07.2010 19:38:
>>> What are the recommendations for the use of the "object" type versus "PyObject
>>> *"? Ondrej pointed out a potential bug in my line_profiler package:
>>>
>>> http://github.com/certik/line_profiler/commit/dc3ad198f8a810d34da5da71ffccf01c5e05a161
>>>
>>> I have a cdef function that serves as the callback for the PyEval_SetTrace()
>>> functionality. It has a PyObject* argument that is sometimes NULL. This argument
>>> happens to be unused. In the version of Cython I originally developed
>>> line_profile under , no code referenced this argument. In the development
>>> version of Cython that Ondrej used to compile line_profiler, a Py_INCREF is
>>> apparently generated. When the argument is NULL, this obviously fails.
>>>
>>> Is this intentional? Should "object" be avoided when the argument could possibly
>>> be NULL? Or should the generated code use Py_XINCREF/Py_XDECREF instead to avoid
>>> failure?
>>
>> In Cython, object and PyObject* are not compatible. The first is an
>> automatically ref-counted object, whereas the second is a plain pointer.
>> For an argument that can be NULL, you cannot use object.
>
> Would using Py_XINCREF/Py_XDECREF allow the automatic refcounting to simply
> ignore NULL pointers but still work for the usual case? Or would there be other
> issues?

There would be lots of other issues. An object is assumed to be an
valid Python reference, ready for passing on to other functions,
insertion into a list, assigning to variables, etc.

- Robert

From robert.kern at gmail.com  Thu Jul 15 20:18:43 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 15 Jul 2010 13:18:43 -0500
Subject: [Cython] On the proper use of object
In-Reply-To: 
References: 
	<4C3F48AD.8070501@behnel.de>	
	
Message-ID: 

On 7/15/10 1:10 PM, Robert Bradshaw wrote:
> On Thu, Jul 15, 2010 at 11:07 AM, Robert Kern  wrote:

>> Would using Py_XINCREF/Py_XDECREF allow the automatic refcounting to simply
>> ignore NULL pointers but still work for the usual case? Or would there be other
>> issues?
>
> There would be lots of other issues. An object is assumed to be an
> valid Python reference, ready for passing on to other functions,
> insertion into a list, assigning to variables, etc.

Works for me.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From kwmsmith at gmail.com  Thu Jul 15 20:22:57 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 15 Jul 2010 13:22:57 -0500
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: <4C3EF076.5060708@student.matnat.uio.no>
References: <4C3B5676.3060101@student.matnat.uio.no>
	
	
	
	<4C3D362E.80901@molden.no>
	
	
	
	
	
	
	<4C3EF076.5060708@student.matnat.uio.no>
Message-ID: 

On Thu, Jul 15, 2010 at 6:26 AM, Dag Sverre Seljebotn
 wrote:

>>> On Wed, Jul 14, 2010 at 6:59 PM, Robert Bradshaw
>>>  wrote:
>>>
[snip]

> Definitely, this is the biggest long-standing complaint. I find myself
> working around it many times. This is what the kurt-gsoc branch is
> about. I just need time for it that I don't have (and nobody else has
> stepped up).

It's on my list, but as you know, I need to get fwrap out the door
with numpy support, using the existing Cython buffers.

That, and finish my PhD.  Damn. :-)

From dagss at student.matnat.uio.no  Thu Jul 15 20:26:35 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 15 Jul 2010 20:26:35 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>							<4C3EF076.5060708@student.matnat.uio.no>
	
Message-ID: <4C3F52DB.5050200@student.matnat.uio.no>

On 07/15/2010 08:22 PM, Kurt Smith wrote:
> On Thu, Jul 15, 2010 at 6:26 AM, Dag Sverre Seljebotn
>   wrote:
>
>    
>>>> On Wed, Jul 14, 2010 at 6:59 PM, Robert Bradshaw
>>>>   wrote:
>>>>
>>>>          
> [snip]
>
>    
>> Definitely, this is the biggest long-standing complaint. I find myself
>> working around it many times. This is what the kurt-gsoc branch is
>> about. I just need time for it that I don't have (and nobody else has
>> stepped up).
>>      
> It's on my list, but as you know, I need to get fwrap out the door
> with numpy support, using the existing Cython buffers.
>
> That, and finish my PhD.  Damn. :-)
>    
I should have written "nobody else but me and Kurt". I wouldn't want you 
to get distracted from fwrap by less useful Cython stuff :-)

Dag Sverre

From sturla at molden.no  Thu Jul 15 21:04:23 2010
From: sturla at molden.no (Sturla Molden)
Date: Thu, 15 Jul 2010 21:04:23 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>					<4C3F3D50.30708@molden.no>
	
Message-ID: <4C3F5BB7.9000106@molden.no>

Den 15.07.2010 19:06, skrev Robert Bradshaw:
> Yep. If the arrays are large, the speed difference should be
> negligible, but there's a huge penalty in object allocation for small
> ones.
>    

Yes, it gets amortized for large arrays. I've seen this when working 
with PyOpenGL as well. For example looping over glVertex* can suck 
tremendously, while glDrawArray has the ctypes overhead amortized for 
large vertex arrays. Tight loops on glVertex* is better done in Cython 
for that reason.

> It sounds like what we need is a non-object-backed array (e.g. a raw
> NumPy array struct) where some of the metadata (e.g. type) is carried
> around by the compiler rather than at runtime.

Yes, but it will be a major task. E.g. Cython has to implement 
overloaded operators (similar to NumPy), slicing, etc., to get close to 
what Fortran 95 already does.

>   The question here is
> how to handle memory allocation--is this done manually in Fortran? I
> suppose slices could just be structs passed around by value.
>    

Allocatable arrays are garbage collected in Fortan 95 (no specific 
method is required by the standard).

If we allocate to Fortran 95 pointers instead, we also have to manually 
deallocate.

We can also allocate to multiple arrays in one allocate statement; i.e. 
we only have to check for memory error once, and can amortize the cost 
of heap allocation.

There are also automatic arrays (like C99), and the Fortran RTL can 
decide to put them on the stack or the heap, depending on size.

It is really like using NumPy without the overhead. While far less 
painful than C, the other niceness of Python is not there. :(
So it's common to just use Fortran 95 to crunch numbers, and use 
scripting from R, S-Plus, Matlab or Python to control it.

Sturla

From stefan_ml at behnel.de  Thu Jul 15 21:04:36 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 15 Jul 2010 21:04:36 +0200
Subject: [Cython] On the proper use of object
In-Reply-To: 
References:  <4C3F48AD.8070501@behnel.de>
	
Message-ID: <4C3F5BC4.1020807@behnel.de>

Robert Kern, 15.07.2010 20:07:
> On 7/15/10 12:43 PM, Stefan Behnel wrote:
>> Hi, note that this is a question for the cython-users mailing list.
>
> Sorry. Since I am also questioning the implementation, this seemed like a
> reasonable place. :-)
>
>> Robert Kern, 15.07.2010 19:38:
>>> What are the recommendations for the use of the "object" type versus "PyObject
>>> *"? Ondrej pointed out a potential bug in my line_profiler package:
>>>
>>> http://github.com/certik/line_profiler/commit/dc3ad198f8a810d34da5da71ffccf01c5e05a161
>>>
>>> I have a cdef function that serves as the callback for the PyEval_SetTrace()
>>> functionality. It has a PyObject* argument that is sometimes NULL. This argument
>>> happens to be unused. In the version of Cython I originally developed
>>> line_profile under , no code referenced this argument. In the development
>>> version of Cython that Ondrej used to compile line_profiler, a Py_INCREF is
>>> apparently generated. When the argument is NULL, this obviously fails.
>>>
>>> Is this intentional? Should "object" be avoided when the argument could possibly
>>> be NULL? Or should the generated code use Py_XINCREF/Py_XDECREF instead to avoid
>>> failure?
>>
>> In Cython, object and PyObject* are not compatible. The first is an
>> automatically ref-counted object, whereas the second is a plain pointer.
>> For an argument that can be NULL, you cannot use object.
>
> Would using Py_XINCREF/Py_XDECREF allow the automatic refcounting to simply
> ignore NULL pointers but still work for the usual case? Or would there be other
> issues?

You mean, inside of Cython generated code? Yes, tons of issues. The 
assumption is that an object reference is *never* NULL, as NULL is the 
error notification value in CPython's C-API. So, for example, if you 
returned such a value from a function, the calling code would assume that 
an exception was raised.

Stefan

From sturla at molden.no  Thu Jul 15 21:16:06 2010
From: sturla at molden.no (Sturla Molden)
Date: Thu, 15 Jul 2010 21:16:06 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: 
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>					<4C3F3D50.30708@molden.no>
	
Message-ID: <4C3F5E76.3090407@molden.no>

Den 15.07.2010 19:06, skrev Robert Bradshaw:
> Well, many projects are still using Fortran 77 (just like many C
> projects still are based on on c89 rather than moving to c99).

The difference between Fortran 77 and Fortran 95 is much bigger than C89 
vs. C99. And yes, I too prefer C to Fortran 77 :-)

There are things in Fortran 95 that are still really bad. E.g. I/O is 
modelled on reading magnetic tapes rather than memory mapping arrays. 
For practical use, one can skip ~50% of Fortran textbooks just by 
ignoring I/O chapters. "Eveyone" prefer to do I/O in C or Python instead.

Sturla


From stefan_ml at behnel.de  Thu Jul 15 21:26:15 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 15 Jul 2010 21:26:15 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: <4C3EF076.5060708@student.matnat.uio.no>
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>						
	<4C3EF076.5060708@student.matnat.uio.no>
Message-ID: <4C3F60D7.4040708@behnel.de>

Dag Sverre Seljebotn, 15.07.2010 13:26:
> This is what the kurt-gsoc branch is
> about. I just need time for it that I don't have (and nobody else has
> stepped up).

Well, I won't do it because I'm not interested, but a major problem might 
be that it's not obvious to most developers what has happened in this 
branch, what features are there and which are missing, and what needs to be 
done to make it sufficiently usable to merge (part of) it back into mainline.

If you or Kurt could give an outline of all that, others *might* step up 
and do something.

Stefan

From ndbecker2 at gmail.com  Thu Jul 15 21:33:14 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 15 Jul 2010 15:33:14 -0400
Subject: [Cython] status of exposing numpy array to c++
References: 
	
	
	
Message-ID: 

Robert Bradshaw wrote:

> On Thu, Jul 15, 2010 at 10:22 AM, Neal Becker
>  wrote:
>> Robert Bradshaw wrote:
>>
>>> On Thu, Jul 15, 2010 at 7:24 AM, Neal Becker
>>>  wrote:
>>>> What is the current state of using cython to expose numpy arrays to c++
>>>> code?
>>>>
>>>> I have the (typical) setup of python for high-level code, c++ for
>>>> specialized algorithms.  I'm using numpy as the containers.
>>>>
>>>> My c++ algorithms are usually written to a generic c++
>>>> (boost::range-based) interface.
>>>>
>>>> Currently, the best solution I know of is to use boost::python and
>>>> pyublas to interface.
>>>>
>>>> Are there any cython examples?  Has any of the recent cython work made
>>>> cython more attractive for this use?
>>>
>>> Are you trying to create C++ wrappers for NumPy arrays? In this case
>>> Cython is almost certainly the wrong tool. The new C++ features allow
>>> you to more easily use C++ libraries,  not write arbitrary C++ code.
>>>
>>> - Robert
>>
>> No, I already have lots of algorithms written in c++ for a generic range-
>> based interface (templated).  These can be instantiated to use with any
>> container types that fit the range concept.
>>
>> Right now I'm instantiating them using pyublas::numpy_vector as the
>> containers, which allows me to use them from python with numpy.
>>
>> For example, consider this c++ algorithm:
>>
>> template
>> inline out_t repeat (in_t const& in, int cnt) {
>> out_t out (boost::size (in)*cnt);
>> typename boost::range_const_iterator::type i = boost::begin (in);
>> typename boost::range_iterator::type o = boost::begin (out);
>>
>> for (; i != boost::end (in); ++i, o+=cnt) {
>> std::fill (o, (o+cnt), *i);
>> }
>>
>> return out;
>> }
>>
>> This could be used on a numpy vector, instantiated as:
>>
>> repeat,pyublas::numpy_vector >
>>
>> Could cython be used to instantiate and call this function, which takes
>> a numpy vector as input/output?
> 
> Sorry I completely misunderstood your question. Yes, this is something
> you could do.
> 
> - Robert

Any pointers to (current) examples and/or docs?


From dagss at student.matnat.uio.no  Thu Jul 15 21:45:32 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 15 Jul 2010 21:45:32 +0200
Subject: [Cython] Impressions from EuroSciPy 2010
In-Reply-To: <4C3F60D7.4040708@behnel.de>
References: <4C3B5676.3060101@student.matnat.uio.no>				<4C3D362E.80901@molden.no>							<4C3EF076.5060708@student.matnat.uio.no>
	<4C3F60D7.4040708@behnel.de>
Message-ID: <4C3F655C.6090703@student.matnat.uio.no>

On 07/15/2010 09:26 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 15.07.2010 13:26:
>    
>> This is what the kurt-gsoc branch is
>> about. I just need time for it that I don't have (and nobody else has
>> stepped up).
>>      
> Well, I won't do it because I'm not interested, but a major problem might
> be that it's not obvious to most developers what has happened in this
> branch, what features are there and which are missing, and what needs to be
> done to make it sufficiently usable to merge (part of) it back into mainline.
>
> If you or Kurt could give an outline of all that, others *might* step up
> and do something.
>    

That is quite true. It is not good to have a branch lying around for a 
year like that.

The fault is entirely mine; I gaped over quite a bit more than I could 
chew, and also told Kurt that I'd put it on me to merge it so that he 
could focus on fwrap -- and then subsequently failed to set aside time 
to do it.

But as it is, I don't even have time to write the kind of outline you 
suggest. If anyone is interested, I could perhaps find time to Skype 
about it. I do hope to get around to it myself at some point after my 
thesis is done though, which is probably the most efficient way of 
getting this done.

(It would take quite a bit more than what is in the branch for the full 
functionality that Sturla asks about, but just being able to pass 
buffers around without reacquisition would be a good start.)

I'm sorry,
Dag Sverre

From paustin at eos.ubc.ca  Thu Jul 15 21:48:14 2010
From: paustin at eos.ubc.ca (Phil Austin)
Date: Thu, 15 Jul 2010 12:48:14 -0700
Subject: [Cython] status of exposing numpy array to c++
In-Reply-To: 
References: 
	
	
Message-ID: <4C3F65FE.1020809@eos.ubc.ca>

On 07/15/2010 10:23 AM, Nathaniel Smith wrote:
>
>  But there are some reasonable C++ numerics libraries -- e.g., Eigen --
>  and a bit of glue code to map ndarray's into a C++ library's array
>  types and then working with those directly in Cython seems like it
>  would be a sweet spot for some of the things people are trying to do
>  in these threads.

and Runar Tenfjord has posted this eigen-cython bridge
to cython-users:

http://bit.ly/cython_eigen

   -- Phil

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100715/10bf7048/attachment.htm 

From stefan_ml at behnel.de  Thu Jul 15 22:34:31 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 15 Jul 2010 22:34:31 +0200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: <4C3F4690.3030300@behnel.de>
References: 	<4C3D80AB.2070003@behnel.de>	
	<4C3F4690.3030300@behnel.de>
Message-ID: <4C3F70D7.1070705@behnel.de>

Stefan Behnel, 15.07.2010 19:34:
> In most cases,
> nodes and their results have to be reused, and there isn't currently a good
> way to do that, especially not before type inference and analysis, which
> may end up wanting to assign different types to the same node result in the
> different contexts.

Let's exercise this a bit. The min/max transformation is a perfect example 
here. It expands this code

     x = min(a,b,c)

into this code

     x = a
     if x > b: x = b
     if x > c: x = c

Now, you have to evaluate each input parameter exactly once, and each 
evaluation can potentially throw an exception. How would you do this in a 
source-level transformation? The only way I see is to introduce temp nodes 
already at this point, i.e. you'd make it

     x, _b, _c = a, b, c
     if x > _b: x = _b
     if x > _c: x = _c

Next, to make the comparisons efficient if x has a Python type, _b and _c 
must have the same type as x. Otherwise, both the comparison and the 
assignment would lead to redundant coercions. How would you assure that? 
Note that type inference will not work here, as it only impacts the type of 
x. All other type analysis steps are local.

I can see us making it less local by injecting back-references to those 
nodes that depend on a node's type. That might allow us to minimise the 
number of neccessary coercions for a value. However, to do this right, we 
need flow analysis to even see which code paths actually lead to a coercion.

Stefan

From stefan_ml at behnel.de  Thu Jul 15 23:15:57 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 15 Jul 2010 23:15:57 +0200
Subject: [Cython] Cython weave thread,
 was: "Impressions from EuroScipy 2010"
In-Reply-To: <4C3EC5A8.3040202@student.matnat.uio.no>
References: 	<4C3E00AB.5000704@student.matnat.uio.no>			
	<4C3EC5A8.3040202@student.matnat.uio.no>
Message-ID: <4C3F7A8D.3010304@behnel.de>

Dag Sverre Seljebotn, 15.07.2010 10:24:
> On 07/15/2010 07:27 AM, Robert Bradshaw wrote:
>> We'd certainly like to support function-level compilation (though
>> there are technical questions about how to get the source) and perhaps
>> even line-by-line via strings.

+1 for function level compilation through a decorator. Shouldn't be hard to 
do at all, as long as we have the code readily available (i.e. the .py 
file, not just a .pyc file or something).

It would be a nice additional feature to compile the code from a string 
instead of a file in that case, and to report the line number offset correctly.


>> Right now the Sage notebook gives this
>> feel for Cython, which is probably why I haven't felt the pinch (I
>> rarely write and invoke a setup.py manually).

Yes, the Sage notebook is a really nice little front-end for Cython 
development. Almost like an interactive shell. The sick thing is that you 
have something in the GB range on your hard drive just to run a little web 
app that compiles and runs code...

Now that I say that, what about IPython support for Cython compilation? 
ICython, anyone?


> Here's some work which gets the source from decorated functions (in pure
> Python...the decorator could extract source, compile with Cython, and
> replace the decorated function with one imported from the compiled one):
>
> http://github.com/GaelVaroquaux/joblib/blob/master/joblib/func_inspect.py

Looks a bit funny. Wouldn't inspect.getsource() be enough for the 
beginning? The comments in the above code seem to suggest that the only 
reason not to use inspect is that the source file can change on the fly. I 
would expect that users would normally restart Python when they modify the 
sources and want them to get recompiled  ...


> Of course this requires a stronger pure Python mode.

Agreed, although the current state of the pure Python mode is certainly not 
a reason to wait with providing this feature.


>>> 3) The C-source-in-a-string is pretty ugly, I admit.  Weave is very
>>> far on the "practicality" side of the "practicality vs. purity" scale.
>>>    It is a small benefit, though, to have the C/C++ code quarantined to
>>> one place in a function, and to always know exactly what to expect
>>> from each part of your code -- python remains python, C remains C.
>>> The fluidity of Cython is a tremendous advantage, but it does require
>>> some care to be sure you're typing everything that needs to be typed,
>>> turning off boundschecking when it's safe, etc.  I tend to use 'cython
>>> -a' or inspecting the generated source to be sure I'm "totally C"
>>> fairly often.  Weave gives this to you automatically.  In short, with
>>> weave, I don't have to pay attention as much to get the same degree of
>>> performance.  The tradeoff is ugly code.
>>>
>> That's a good point--we should be a lot closer with type inference
>> (and especially if we do the trick of compiling/specializing the code
>> at runtime to the input types) but the fact that untyped code is just
>> slower rather than broken is an interesting one. Almost makes me want
>> to have a "C only" flag or something like that.
>>
> We already discussed having a decorator for disallowing untyped
> variables. Basically you would need to type every variable in a
> function, but typing as "object" would be OK. I think this is more
> useful than "C only" in this context.

Yes, makes sense to me. I remember that it was Lisandro who pushed for this 
at the time, and I had it in the back of my head that this had been 
implemented at some point. It seems not.

Stefan

From fperez.net at gmail.com  Thu Jul 15 23:29:06 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 15 Jul 2010 14:29:06 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: <4C3F7A8D.3010304@behnel.de>
References:  
	<4C3E00AB.5000704@student.matnat.uio.no>
	 
	 
	 
	<4C3EC5A8.3040202@student.matnat.uio.no> <4C3F7A8D.3010304@behnel.de>
Message-ID: 

On Thu, Jul 15, 2010 at 2:15 PM, Stefan Behnel  wrote:
>
> Now that I say that, what about IPython support for Cython compilation?
> ICython, anyone?

Absolutely, totally possible and welcome.  I should add that this
summer we're in the middle of a major refactoring of ipython, and we
should also get lightweight 'mini-notebook' environments for IPython
out of this work (early, pre-pre-alpha prototype work at
http://github.com/muzgash/ipython/tree/ipythonqt and
http://github.com/epatters/ipython/tree/qtfrontend with a more
terminal-like feel but in Qt).

Once that lands, I'd like to change IPython's magics to work more like
Sage: sage took the %syntax but defined it to call the .eval() method
of %whatever; I think we can easily adopt that approach as well, so
that at least api-wise ipython and sage would be compatible.  This
would make it easier to have a magic that works in both (possibly
isolating sage/ipython-specific parts).

In the meantime, a plain python magic that waits for input until some
marker or EOF arrives and passes it to the cython machinery would be
very easy to write, and would work *today* in ipython.

Cheers,

f

From robertwb at math.washington.edu  Fri Jul 16 02:07:58 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 17:07:58 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: <4C3F7A8D.3010304@behnel.de>
References: 
	<4C3E00AB.5000704@student.matnat.uio.no>
	
	
	
	<4C3EC5A8.3040202@student.matnat.uio.no>
	<4C3F7A8D.3010304@behnel.de>
Message-ID: 

On Thu, Jul 15, 2010 at 2:15 PM, Stefan Behnel  wrote:
> Dag Sverre Seljebotn, 15.07.2010 10:24:
>> On 07/15/2010 07:27 AM, Robert Bradshaw wrote:
>>> We'd certainly like to support function-level compilation (though
>>> there are technical questions about how to get the source) and perhaps
>>> even line-by-line via strings.
>
> +1 for function level compilation through a decorator. Shouldn't be hard to
> do at all, as long as we have the code readily available (i.e. the .py
> file, not just a .pyc file or something).
>
> It would be a nice additional feature to compile the code from a string
> instead of a file in that case, and to report the line number offset correctly.

Yep.

>>> Right now the Sage notebook gives this
>>> feel for Cython, which is probably why I haven't felt the pinch (I
>>> rarely write and invoke a setup.py manually).
>
> Yes, the Sage notebook is a really nice little front-end for Cython
> development. Almost like an interactive shell. The sick thing is that you
> have something in the GB range on your hard drive just to run a little web
> app that compiles and runs code...

Here's a nickel. (OK, that won't quite get you a GB, but close...)

> Now that I say that, what about IPython support for Cython compilation?
> ICython, anyone?
>
>
>> Here's some work which gets the source from decorated functions (in pure
>> Python...the decorator could extract source, compile with Cython, and
>> replace the decorated function with one imported from the compiled one):
>>
>> http://github.com/GaelVaroquaux/joblib/blob/master/joblib/func_inspect.py
>
> Looks a bit funny. Wouldn't inspect.getsource() be enough for the
> beginning? The comments in the above code seem to suggest that the only
> reason not to use inspect is that the source file can change on the fly. I
> would expect that users would normally restart Python when they modify the
> sources and want them to get recompiled ?...

I think the trickier part is getting it from am interactive prompt,
though IPython has done most of the hard work in this direction.

>> Of course this requires a stronger pure Python mode.
>
> Agreed, although the current state of the pure Python mode is certainly not
> a reason to wait with providing this feature.

Pure python mode is enough to do quite a bit here, and the stuff like
cdef classes/methods are harder to take advantage of locally as well.
(Future direction--letting the second decorated function be aware of
everything that came before it and able to invoke it via c semantics,
kind of an auto-cimport?)

>>>> 3) The C-source-in-a-string is pretty ugly, I admit. ?Weave is very
>>>> far on the "practicality" side of the "practicality vs. purity" scale.
>>>> ? ?It is a small benefit, though, to have the C/C++ code quarantined to
>>>> one place in a function, and to always know exactly what to expect
>>>> from each part of your code -- python remains python, C remains C.
>>>> The fluidity of Cython is a tremendous advantage, but it does require
>>>> some care to be sure you're typing everything that needs to be typed,
>>>> turning off boundschecking when it's safe, etc. ?I tend to use 'cython
>>>> -a' or inspecting the generated source to be sure I'm "totally C"
>>>> fairly often. ?Weave gives this to you automatically. ?In short, with
>>>> weave, I don't have to pay attention as much to get the same degree of
>>>> performance. ?The tradeoff is ugly code.
>>>>
>>> That's a good point--we should be a lot closer with type inference
>>> (and especially if we do the trick of compiling/specializing the code
>>> at runtime to the input types) but the fact that untyped code is just
>>> slower rather than broken is an interesting one. Almost makes me want
>>> to have a "C only" flag or something like that.
>>>
>> We already discussed having a decorator for disallowing untyped
>> variables. Basically you would need to type every variable in a
>> function, but typing as "object" would be OK. I think this is more
>> useful than "C only" in this context.
>
> Yes, makes sense to me. I remember that it was Lisandro who pushed for this
> at the time, and I had it in the back of my head that this had been
> implemented at some point. It seems not.

Nowadays that would be trivial to do. Any ideas for a directive name?

- Robert

From fperez.net at gmail.com  Fri Jul 16 03:36:23 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 15 Jul 2010 18:36:23 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: 
References:  
	<4C3E00AB.5000704@student.matnat.uio.no>
	 
	 
	 
	<4C3EC5A8.3040202@student.matnat.uio.no> <4C3F7A8D.3010304@behnel.de> 
	
Message-ID: 

On Thu, Jul 15, 2010 at 5:07 PM, Robert Bradshaw
 wrote:
>
> I think the trickier part is getting it from am interactive prompt,
> though IPython has done most of the hard work in this direction.
>

Ugly but functional:

http://github.com/ipython/ipython/blob/0.10.1/IPython/kernel/contexts.py#L110

So yes, obviously just like the sage notebook, since we have the
user's input we can get the source for interactively defined
functions.  This is something you normally can't access because python
itself does not store that info.  A while ago that was proposed, as a
__source__ attribute, but it was ultimately rejected:

http://www.gossamer-threads.com/lists/python/dev/367506?do=post_view_threaded

Which means for functions defined in files, these tricks are *very*
brittle.  Given a function's code object, there are many ways in which
one can fail to get back its source: the __file__ attribute may be
mis-set (I've seen it from compilations in chroot environments, where
the path to the pyc is bogus), the import may come from a zip archive
without sources, etc.

In a sense, sage/ipython are more robust here than python itself,
because they have full knowledge of the user's input history and can
access the source of all functions directly.  But the analysis can get
tricky to get right for things that have been otherwise decorated,
dynamically generated, callable objects, etc.

In summary, this can be made to  work, but it will carry some restrictions.

Cheers,

f

From robertwb at math.washington.edu  Fri Jul 16 04:00:01 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 19:00:01 -0700
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 	2010"
In-Reply-To: 
References: 
	<4C3E00AB.5000704@student.matnat.uio.no>
	
	
	
	<4C3EC5A8.3040202@student.matnat.uio.no>
	<4C3F7A8D.3010304@behnel.de>
	
	
Message-ID: 

On Thu, Jul 15, 2010 at 6:36 PM, Fernando Perez  wrote:
> On Thu, Jul 15, 2010 at 5:07 PM, Robert Bradshaw
>  wrote:
>>
>> I think the trickier part is getting it from am interactive prompt,
>> though IPython has done most of the hard work in this direction.
>>
>
> Ugly but functional:
>
> http://github.com/ipython/ipython/blob/0.10.1/IPython/kernel/contexts.py#L110
>
> So yes, obviously just like the sage notebook, since we have the
> user's input we can get the source for interactively defined
> functions. ?This is something you normally can't access because python
> itself does not store that info. ?A while ago that was proposed, as a
> __source__ attribute, but it was ultimately rejected:
>
> http://www.gossamer-threads.com/lists/python/dev/367506?do=post_view_threaded
>
> Which means for functions defined in files, these tricks are *very*
> brittle. ?Given a function's code object, there are many ways in which
> one can fail to get back its source: the __file__ attribute may be
> mis-set (I've seen it from compilations in chroot environments, where
> the path to the pyc is bogus), the import may come from a zip archive
> without sources, etc.
>
> In a sense, sage/ipython are more robust here than python itself,
> because they have full knowledge of the user's input history and can
> access the source of all functions directly. ?But the analysis can get
> tricky to get right for things that have been otherwise decorated,
> dynamically generated, callable objects, etc.
>
> In summary, this can be made to ?work, but it will carry some restrictions.

Of course, as a fallback, we could try throwing some decompiler at the
code object :).

From robertwb at math.washington.edu  Fri Jul 16 04:59:04 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 19:59:04 -0700
Subject: [Cython] bug in optimize_min_max
In-Reply-To: <4C3F70D7.1070705@behnel.de>
References: 
	<4C3D80AB.2070003@behnel.de>
	
	<4C3F4690.3030300@behnel.de> <4C3F70D7.1070705@behnel.de>
Message-ID: 

On Thu, Jul 15, 2010 at 1:34 PM, Stefan Behnel  wrote:
> Stefan Behnel, 15.07.2010 19:34:
>> In most cases,
>> nodes and their results have to be reused, and there isn't currently a good
>> way to do that, especially not before type inference and analysis, which
>> may end up wanting to assign different types to the same node result in the
>> different contexts.
>
> Let's exercise this a bit. The min/max transformation is a perfect example
> here. It expands this code
>
> ? ? x = min(a,b,c)
>
> into this code
>
> ? ? x = a
> ? ? if x > b: x = b
> ? ? if x > c: x = c
>
> Now, you have to evaluate each input parameter exactly once, and each
> evaluation can potentially throw an exception. How would you do this in a
> source-level transformation? The only way I see is to introduce temp nodes
> already at this point,

Yep, you have to anyway to preserve the semantics.

> i.e. you'd make it
>
> ? ? x, _b, _c = a, b, c
> ? ? if x > _b: x = _b
> ? ? if x > _c: x = _c
>
> Next, to make the comparisons efficient if x has a Python type, _b and _c
> must have the same type as x.

The expanded code would be

res, _b, _c = a, b, c
if res > _b: res = _b
if res > _c: res = _c
# cleanup _b, _c
x = res

so the assignment (and type of the user-visible x) is orthogonal. Of
course, a could be an object too...One could even do

r1, _b, _c = a, b, c
r2 = r1 if r1 < _b else _b
r3 = r2 if r2 < _c else _c

so it stays in C land as long as possible, but that may not be worth it.

> Otherwise, both the comparison and the
> assignment would lead to redundant coercions. How would you assure that?
> Note that type inference will not work here, as it only impacts the type of
> x. All other type analysis steps are local.
>
> I can see us making it less local by injecting back-references to those
> nodes that depend on a node's type. That might allow us to minimise the
> number of neccessary coercions for a value. However, to do this right, we
> need flow analysis to even see which code paths actually lead to a coercion.

Hopefully a simple kind of common subexpression elimination could
remove the redundant coercions (as well as help out in a lot of other
places).


Also, the 1-value min/max could be optimized away, and the 2-value one could be

    x = a if a < b else b

in conjunction with another possible optimization

    (a if X else b).coerce_to(T)

becomes

    a.coerce_to(T) if X else b.coerce_to(T)

- Robert

From stefan_ml at behnel.de  Fri Jul 16 05:52:38 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 16 Jul 2010 05:52:38 +0200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: 
References: 	<4C3D80AB.2070003@behnel.de>		<4C3F4690.3030300@behnel.de>
	<4C3F70D7.1070705@behnel.de>
	
Message-ID: <4C3FD786.1080102@behnel.de>

Robert Bradshaw, 16.07.2010 04:59:
> On Thu, Jul 15, 2010 at 1:34 PM, Stefan Behnel wrote:
>> The min/max transformation is a perfect example here. It expands this code
>>
>>      x = min(a,b,c)
>>
>> into this code
>>
>>      x = a
>>      if x>  b: x = b
>>      if x>  c: x = c
>>
>> Now, you have to evaluate each input parameter exactly once, and each
>> evaluation can potentially throw an exception. How would you do this in a
>> source-level transformation? The only way I see is to introduce temp nodes
>> already at this point,
>
> The expanded code would be
>
> res, _b, _c = a, b, c
> if res>  _b: res = _b
> if res>  _c: res = _c
> # cleanup _b, _c
> x = res
>
> so the assignment (and type of the user-visible x) is orthogonal. Of
> course, a could be an object too...One could even do
>
> r1, _b, _c = a, b, c
> r2 = r1 if r1<  _b else _b
> r3 = r2 if r2<  _c else _c
>
> so it stays in C land as long as possible, but that may not be worth it.

Hmmm, I didn't think of that. That might even fix the problem at hand. But 
it also means that we may (currently) end up with a lot more ref-counting 
than strictly necessary for Python values. I'll have to look at the details.


>> Otherwise, both the comparison and the
>> assignment would lead to redundant coercions. How would you assure that?
>> Note that type inference will not work here, as it only impacts the type of
>> x. All other type analysis steps are local.
>>
>> I can see us making it less local by injecting back-references to those
>> nodes that depend on a node's type. That might allow us to minimise the
>> number of neccessary coercions for a value. However, to do this right, we
>> need flow analysis to even see which code paths actually lead to a coercion.
>
> Hopefully a simple kind of common subexpression elimination could
> remove the redundant coercions (as well as help out in a lot of other
> places).

You can't really do common subexpression elimination on Python code as even 
operators may have side effects. You can't even know if there may be *no* 
be side effects before the type analysis tells you that you are only 
dealing with plain C values.

No, I think the only way to deal with this early enough is to introduce a 
ReusedExpression node, and to handle that basically everywhere, also during 
type analysis. We currently do a lot of node type checks in the optimiser, 
so we'll need a better way to deal with that, too.


> Also, the 1-value min/max could be optimized away

Nope:

   l = [1,2,3]
   assert min(l) == 1

can't be optimised away at all, whereas

   min(1)

raises a TypeError. The third use case, generator expressions, are not 
currently supported in Cython, but would also be nice to have, even if 
they'd only deal with Python values in most cases. The reduced looping 
overhead and the inlined generator iteration is always worth it.

Note that min() is different from sum() here, as

    cdef int x = min(i*2 for i in xrange(2, 1000000000000000000000))

requires Python long values to evaluate correctly before it can assign the 
C int 4 to x, whereas

    cdef int x = sum(i*2 for i in xrange(2, 1000000000000000000000))

only requires i to be a Python variable but can otherwise safely overflow 
several times and thus use a plain C int for the intermediate sums.


>, and the 2-value one could be
>
>      x = a if a<  b else b

Right, that works. Absolutely worth doing as it's a pretty common case and 
the resulting code avoids an unnecessary initial assignment.


> in conjunction with another possible optimization
>
>      (a if X else b).coerce_to(T)
>
> becomes
>
>      a.coerce_to(T) if X else b.coerce_to(T)

I thought we did that already.

Stefan

From robertwb at math.washington.edu  Fri Jul 16 06:06:10 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 15 Jul 2010 21:06:10 -0700
Subject: [Cython] bug in optimize_min_max
In-Reply-To: <4C3FD786.1080102@behnel.de>
References: 
	<4C3D80AB.2070003@behnel.de>
	
	<4C3F4690.3030300@behnel.de> <4C3F70D7.1070705@behnel.de>
	
	<4C3FD786.1080102@behnel.de>
Message-ID: 

On Thu, Jul 15, 2010 at 8:52 PM, Stefan Behnel  wrote:
> Robert Bradshaw, 16.07.2010 04:59:
>> On Thu, Jul 15, 2010 at 1:34 PM, Stefan Behnel wrote:
>>> The min/max transformation is a perfect example here. It expands this code
>>>
>>> ? ? ?x = min(a,b,c)
>>>
>>> into this code
>>>
>>> ? ? ?x = a
>>> ? ? ?if x> ?b: x = b
>>> ? ? ?if x> ?c: x = c
>>>
>>> Now, you have to evaluate each input parameter exactly once, and each
>>> evaluation can potentially throw an exception. How would you do this in a
>>> source-level transformation? The only way I see is to introduce temp nodes
>>> already at this point,
>>
>> The expanded code would be
>>
>> res, _b, _c = a, b, c
>> if res> ?_b: res = _b
>> if res> ?_c: res = _c
>> # cleanup _b, _c
>> x = res
>>
>> so the assignment (and type of the user-visible x) is orthogonal. Of
>> course, a could be an object too...One could even do
>>
>> r1, _b, _c = a, b, c
>> r2 = r1 if r1< ?_b else _b
>> r3 = r2 if r2< ?_c else _c
>>
>> so it stays in C land as long as possible, but that may not be worth it.
>
> Hmmm, I didn't think of that. That might even fix the problem at hand. But
> it also means that we may (currently) end up with a lot more ref-counting
> than strictly necessary for Python values. I'll have to look at the details.

In any case, it's still less refcounting than stuffing then all into
tuples. Hopefully we can get rid of a lot of redundant refcounting
once we have control flow analysis.

>>> Otherwise, both the comparison and the
>>> assignment would lead to redundant coercions. How would you assure that?
>>> Note that type inference will not work here, as it only impacts the type of
>>> x. All other type analysis steps are local.
>>>
>>> I can see us making it less local by injecting back-references to those
>>> nodes that depend on a node's type. That might allow us to minimise the
>>> number of neccessary coercions for a value. However, to do this right, we
>>> need flow analysis to even see which code paths actually lead to a coercion.
>>
>> Hopefully a simple kind of common subexpression elimination could
>> remove the redundant coercions (as well as help out in a lot of other
>> places).
>
> You can't really do common subexpression elimination on Python code as even
> operators may have side effects. You can't even know if there may be *no*
> be side effects before the type analysis tells you that you are only
> dealing with plain C values.

The common subexpression elimination would happen at a much later
stage in the pipeline. I trust gcc to do an OK job for pure C values,
but we should be able to assert C value -> Python value is side-effect
free at least.

> No, I think the only way to deal with this early enough is to introduce a
> ReusedExpression node, and to handle that basically everywhere, also during
> type analysis. We currently do a lot of node type checks in the optimiser,
> so we'll need a better way to deal with that, too.

We had CloneNode, how would ReusedExpressionNode be better? (Also, I
don't see how it would help with types.)

>> Also, the 1-value min/max could be optimized away
>
> Nope:
>
> ? l = [1,2,3]
> ? assert min(l) == 1
>
> can't be optimised away at all, whereas
>
> ? min(1)
>
> raises a TypeError.

Oh, I forgot that min(args) == min(*args)

> The third use case, generator expressions, are not
> currently supported in Cython, but would also be nice to have, even if
> they'd only deal with Python values in most cases. The reduced looping
> overhead and the inlined generator iteration is always worth it.
>
> Note that min() is different from sum() here, as
>
> ? ?cdef int x = min(i*2 for i in xrange(2, 1000000000000000000000))
>
> requires Python long values to evaluate correctly before it can assign the
> C int 4 to x, whereas
>
> ? ?cdef int x = sum(i*2 for i in xrange(2, 1000000000000000000000))
>
> only requires i to be a Python variable but can otherwise safely overflow
> several times and thus use a plain C int for the intermediate sums.

Yep.

>>, and the 2-value one could be
>>
>> ? ? ?x = a if a< ?b else b
>
> Right, that works. Absolutely worth doing as it's a pretty common case and
> the resulting code avoids an unnecessary initial assignment.
>
>
>> in conjunction with another possible optimization
>>
>> ? ? ?(a if X else b).coerce_to(T)
>>
>> becomes
>>
>> ? ? ?a.coerce_to(T) if X else b.coerce_to(T)
>
> I thought we did that already.

Maybe we do :)

- Robert

From stefan_ml at behnel.de  Fri Jul 16 06:49:15 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 16 Jul 2010 06:49:15 +0200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: 
References: 	<4C3D80AB.2070003@behnel.de>		<4C3F4690.3030300@behnel.de>
	<4C3F70D7.1070705@behnel.de>		<4C3FD786.1080102@behnel.de>
	
Message-ID: <4C3FE4CB.1050700@behnel.de>

Robert Bradshaw, 16.07.2010 06:06:
> On Thu, Jul 15, 2010 at 8:52 PM, Stefan Behnel wrote:
>> Robert Bradshaw, 16.07.2010 04:59:
>>> Hopefully a simple kind of common subexpression elimination could
>>> remove the redundant coercions (as well as help out in a lot of other
>>> places).
>>
>> You can't really do common subexpression elimination on Python code as even
>> operators may have side effects. You can't even know if there may be *no*
>> be side effects before the type analysis tells you that you are only
>> dealing with plain C values.
>
> The common subexpression elimination would happen at a much later
> stage in the pipeline.

Ah, right, you were really only talking about reducing coercions by doing 
them once and reusing the result. That might work, yes. So it's not true 
CSE in the sense of really working on expressions but rather something at a 
NameNode/temps level.


> I trust gcc to do an OK job for pure C values,
> but we should be able to assert C value ->  Python value is side-effect
> free at least.

That way, yes. The other way, no. And that's a problem already if you don't 
know the types at the time where you transform the code. Some of the 
required information might be available after the analyse_declarations 
phase, though.


>> No, I think the only way to deal with this early enough is to introduce a
>> ReusedExpression node, and to handle that basically everywhere, also during
>> type analysis. We currently do a lot of node type checks in the optimiser,
>> so we'll need a better way to deal with that, too.
>
> We had CloneNode, how would ReusedExpressionNode be better?

Well, we need some kind of specific node that lets us know that an 
expression is reused, so that temp handling, ref-counting and your proposed 
coercion removal can do the right thing. Either that, or a larger reworking 
in ExprNodes.py to support reusing arbitrary expression results in 
different places, which currently is impossible due to the way temp 
handling works.


 > (Also, I don't see how it would help with types.)

You'd special case it during type analysis in that its type is only 
determined after evaluating all occurrences, and then is set to the most 
efficient type for all use patterns that minimises the number of coercions.

However, I think this can't easily be done with the current evaluate-once 
flow of type analysis.

Stefan


From binet at cern.ch  Fri Jul 16 11:55:50 2010
From: binet at cern.ch (Sebastien Binet)
Date: Fri, 16 Jul 2010 11:55:50 +0200
Subject: [Cython] Cython weave thread,
	was: "Impressions from EuroScipy 2010"
In-Reply-To: <4C3F7A8D.3010304@behnel.de>
References: 
	<4C3E00AB.5000704@student.matnat.uio.no>
	
	
	
	<4C3EC5A8.3040202@student.matnat.uio.no>
	<4C3F7A8D.3010304@behnel.de>
Message-ID: <87aapriwd5.fsf@cern.ch>


Stefan,

On Thu, 15 Jul 2010 23:15:57 +0200, Stefan Behnel  wrote:
> Dag Sverre Seljebotn, 15.07.2010 10:24:
> > On 07/15/2010 07:27 AM, Robert Bradshaw wrote:
> >> We'd certainly like to support function-level compilation (though
> >> there are technical questions about how to get the source) and perhaps
> >> even line-by-line via strings.
> 
> +1 for function level compilation through a decorator. Shouldn't be hard to 
> do at all, as long as we have the code readily available (i.e. the .py 
> file, not just a .pyc file or something).
> 
> It would be a nice additional feature to compile the code from a string 
> instead of a file in that case, and to report the line number offset correctly.
> 
> 
> >> Right now the Sage notebook gives this
> >> feel for Cython, which is probably why I haven't felt the pinch (I
> >> rarely write and invoke a setup.py manually).
> 
> Yes, the Sage notebook is a really nice little front-end for Cython 
> development. Almost like an interactive shell. The sick thing is that you 
> have something in the GB range on your hard drive just to run a little web 
> app that compiles and runs code...
> 
> Now that I say that, what about IPython support for Cython compilation? 
> ICython, anyone?

I already provided something along the lines on this list (I can't find
the url anymore)
and I "packaged" it up for my experiment:
http://alxr.usatlas.bnl.gov/lxr/source/atlas/Tools/PyUtils/bin/icython.py

being built atop the python interpreter, it has some limitations (like
being only able to see 'cpdef' functions, creating one .so per little
cython snippet and probably other limitations I never ran into)

cheers,
sebastien.

-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

From ndbecker2 at gmail.com  Fri Jul 16 12:39:46 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 16 Jul 2010 06:39:46 -0400
Subject: [Cython] status of exposing numpy array to c++
References: 
	
	
	<4C3F65FE.1020809@eos.ubc.ca>
Message-ID: 

Phil Austin wrote:

> On 07/15/2010 10:23 AM, Nathaniel Smith wrote:
>>
>>  But there are some reasonable C++ numerics libraries -- e.g., Eigen --
>>  and a bit of glue code to map ndarray's into a C++ library's array
>>  types and then working with those directly in Cython seems like it
>>  would be a sweet spot for some of the things people are trying to do
>>  in these threads.
> 
> and Runar Tenfjord has posted this eigen-cython bridge
> to cython-users:
> 
> http://bit.ly/cython_eigen
> 
>    -- Phil

eigen-cython bridge looks very interesting.  Looking at the code briefly, it 
wasn't clear what capabilities it has.  It looked to me that it could expose 
native eigen vectors+matrixes to python.  Can it operate on numpy arrays?  
That would be required for me.


From stefan_ml at behnel.de  Fri Jul 16 13:24:54 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 16 Jul 2010 13:24:54 +0200
Subject: [Cython] status of exposing numpy array to c++
In-Reply-To: 
References: 			<4C3F65FE.1020809@eos.ubc.ca>
	
Message-ID: <4C404186.2030408@behnel.de>

Neal Becker, 16.07.2010 12:39:
> Phil Austin wrote:
>
>> On 07/15/2010 10:23 AM, Nathaniel Smith wrote:
>>>
>>>   But there are some reasonable C++ numerics libraries -- e.g., Eigen --
>>>   and a bit of glue code to map ndarray's into a C++ library's array
>>>   types and then working with those directly in Cython seems like it
>>>   would be a sweet spot for some of the things people are trying to do
>>>   in these threads.
>>
>> and Runar Tenfjord has posted this eigen-cython bridge
>> to cython-users:
>>
>> http://bit.ly/cython_eigen
>>
>>     -- Phil
>
> eigen-cython bridge looks very interesting.  Looking at the code briefly, it
> wasn't clear what capabilities it has.  It looked to me that it could expose
> native eigen vectors+matrixes to python.  Can it operate on numpy arrays?
> That would be required for me.

Sounds like the right place to ask is the cython-users mailing list then.

Stefan

From greg.ewing at canterbury.ac.nz  Sat Jul 17 03:09:13 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 17 Jul 2010 13:09:13 +1200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: <4C3FD786.1080102@behnel.de>
References: 
	<4C3D80AB.2070003@behnel.de>
	
	<4C3F4690.3030300@behnel.de> <4C3F70D7.1070705@behnel.de>
	
	<4C3FD786.1080102@behnel.de>
Message-ID: <4C4102B9.4050104@canterbury.ac.nz>

Stefan Behnel wrote:

> No, I think the only way to deal with this early enough is to introduce a 
> ReusedExpression node, and to handle that basically everywhere

There was discussion a while back about some kind of
LetNode -- did you implement that? Wouldn't it give you
what you need here?

-- 
Greg

From craigcitro at gmail.com  Sat Jul 17 09:49:48 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Sat, 17 Jul 2010 00:49:48 -0700
Subject: [Cython] cython-closures tip passing all Sage doctests!
Message-ID: 

Hi all,

I think the subject line pretty much says it all -- I just
successfully built sage-4.4.4 against the tip of cython-closures, and
it builds, runs, and passes all doctests(*). As far as I'm concerned,
that means it's time to roll ourselves an alpha. (Robert, do you want
to tell me what steps are involved in that? Is there a list
somewhere?) Also, what's the general rule for avoiding someone
clobbering the "working" state with a push? Should I just add a tag
for "0.13-alpha0" and we could always recover from there?

I just ran the Cython tests in the closures branch itself -- it looks
like there are a few doctest failures with the recent code there. I'm
happy to start helping with these tomorrow, but can we designate
cython-closures as "bugfix only" until the actual release? For that
matter, with closures becoming devel, soon we won't need a closures
branch anymore ... finally. ;)

(*) Actually, there's one doctest failure going on -- but it's a bug
in Sage, which I've reported.
(http://trac.sagemath.org/sage_trac/ticket/9524) It could be that
there's an associated underlying Cython bug, but I don't think so in
this case -- either way, it shouldn't stop us from rolling an alpha,
and a release if we still think it's not an underlying Cython bug.

-cc

From stefan_ml at behnel.de  Sat Jul 17 12:46:17 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 17 Jul 2010 12:46:17 +0200
Subject: [Cython] cython-closures tip passing all Sage doctests!
In-Reply-To: 
References: 
Message-ID: <4C4189F9.6000509@behnel.de>

Craig Citro, 17.07.2010 09:49:
> I think the subject line pretty much says it all -- I just
> successfully built sage-4.4.4 against the tip of cython-closures, and
> it builds, runs, and passes all doctests(*). As far as I'm concerned,
> that means it's time to roll ourselves an alpha. (Robert, do you want
> to tell me what steps are involved in that? Is there a list
> somewhere?) Also, what's the general rule for avoiding someone
> clobbering the "working" state with a push? Should I just add a tag
> for "0.13-alpha0" and we could always recover from there?

I only fixed a couple of things in the genexpr inlining code which 
shouldn't have any impact on existing Cython code bases, so the latest tip 
will work just as well. However, we're not there yet. I would like to have 
Cython's own test suite green on all Python versions even before the alpha.


> I just ran the Cython tests in the closures branch itself -- it looks
> like there are a few doctest failures with the recent code there.

Yes, a handful of tests still fail in Py3.

http://trac.cython.org/cython_trac/ticket/543

Looks like Haoyu has a fix for at least one of them.


> I'm happy to start helping with these tomorrow, but can we designate
> cython-closures as "bugfix only" until the actual release? For that
> matter, with closures becoming devel, soon we won't need a closures
> branch anymore ... finally. ;)

Note that we don't currently run the Hudson tests on all Python platforms 
for cython-closures but only for cython-devel. I'd suggest merging 
cython-closures into cython-devel before releasing the alpha, and giving 
Hudson a couple of minutes to run all tests to see if we missed anything 
there (which we likely did).

I'll take a look at the current cython-devel failures in Py<=2.5 as I broke 
at least one of them while fixing Unicode problems on narrow platforms, but 
I'd be happy to get some help in fixing the others. Some tests have been 
broken for ages. See e.g. the end of

https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py24-c/138/console

Stefan

From robertwb at math.washington.edu  Sat Jul 17 18:23:31 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 17 Jul 2010 09:23:31 -0700
Subject: [Cython] cython-closures tip passing all Sage doctests!
In-Reply-To: <4C4189F9.6000509@behnel.de>
References: 
	<4C4189F9.6000509@behnel.de>
Message-ID: 

On Sat, Jul 17, 2010 at 3:46 AM, Stefan Behnel  wrote:
> Craig Citro, 17.07.2010 09:49:
>> I think the subject line pretty much says it all -- I just
>> successfully built sage-4.4.4 against the tip of cython-closures, and
>> it builds, runs, and passes all doctests(*).

Excellent news!

>> As far as I'm concerned,
>> that means it's time to roll ourselves an alpha. (Robert, do you want
>> to tell me what steps are involved in that? Is there a list
>> somewhere?) Also, what's the general rule for avoiding someone
>> clobbering the "working" state with a push? Should I just add a tag
>> for "0.13-alpha0" and we could always recover from there?

I think it's worth rolling an alpha, noting that there are still some
failures, and get it out there. When we get everything actually
passing, we're at beta stage. I don't feel too strongly about this.

> I only fixed a couple of things in the genexpr inlining code which
> shouldn't have any impact on existing Cython code bases, so the latest tip
> will work just as well. However, we're not there yet. I would like to have
> Cython's own test suite green on all Python versions even before the alpha.
>
>
>> I just ran the Cython tests in the closures branch itself -- it looks
>> like there are a few doctest failures with the recent code there.
>
> Yes, a handful of tests still fail in Py3.
>
> http://trac.cython.org/cython_trac/ticket/543
>
> Looks like Haoyu has a fix for at least one of them.
>
>
>> I'm happy to start helping with these tomorrow, but can we designate
>> cython-closures as "bugfix only" until the actual release? For that
>> matter, with closures becoming devel, soon we won't need a closures
>> branch anymore ... finally. ;)
>
> Note that we don't currently run the Hudson tests on all Python platforms
> for cython-closures but only for cython-devel. I'd suggest merging
> cython-closures into cython-devel before releasing the alpha, and giving
> Hudson a couple of minutes to run all tests to see if we missed anything
> there (which we likely did).

Good idea.

> I'll take a look at the current cython-devel failures in Py<=2.5 as I broke
> at least one of them while fixing Unicode problems on narrow platforms, but
> I'd be happy to get some help in fixing the others. Some tests have been
> broken for ages. See e.g. the end of
>
> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py24-c/138/console

What's the ETA for this? I thought that once Sage built, that was the
last major problem. In any case, +1 to merging into -devel and making
that bugfix-only so we can actually release.

Does anyone want to go to
http://trac.cython.org/cython_trac/milestone/0.13 and bump stuff to
0.13.1 (or later) if it's not trivial or a blocker? (I'll do this if
no one beats me too it, but I've got to run now.)

- Robert

From stefan_ml at behnel.de  Sat Jul 17 18:57:12 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 17 Jul 2010 18:57:12 +0200
Subject: [Cython] cython-closures tip passing all Sage doctests!
In-Reply-To: 
References: 	<4C4189F9.6000509@behnel.de>
	
Message-ID: <4C41E0E8.3010305@behnel.de>

Robert Bradshaw, 17.07.2010 18:23:
> On Sat, Jul 17, 2010 at 3:46 AM, Stefan Behnel wrote:
>> I'll take a look at the current cython-devel failures in Py<=2.5 as I broke
>> at least one of them while fixing Unicode problems on narrow platforms, but
>> I'd be happy to get some help in fixing the others. Some tests have been
>> broken for ages. See e.g. the end of
>>
>> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py24-c/138/console
>
> What's the ETA for this? I thought that once Sage built, that was the
> last major problem.

I think it was. Most test failures looked like it was just the tests that 
were broken. I fixed most of them. Here's the last one that I'm not sure 
about. In any case, given that it's a pure pre-Py2.5 issue, it's nothing 
that would be worth delaying a release.

https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py24-c/148/testReport/doctest/DocTestCase/Doctest__ssize_t_T399/


> In any case, +1 to merging into -devel and making
> that bugfix-only so we can actually release.

+1


> Does anyone want to go to
> http://trac.cython.org/cython_trac/milestone/0.13 and bump stuff to
> 0.13.1 (or later) if it's not trivial or a blocker? (I'll do this if
> no one beats me too it, but I've got to run now.)

I'll take another look, and I think all core developers should do that. 
Adding a short comment when changing the target milestone or making 
something a blocker would be nice.

Stefan

From stefan_ml at behnel.de  Mon Jul 19 07:21:11 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 19 Jul 2010 07:21:11 +0200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: <4C4102B9.4050104@canterbury.ac.nz>
References: 	<4C3D80AB.2070003@behnel.de>		<4C3F4690.3030300@behnel.de>
	<4C3F70D7.1070705@behnel.de>		<4C3FD786.1080102@behnel.de>
	<4C4102B9.4050104@canterbury.ac.nz>
Message-ID: <4C43E0C7.20107@behnel.de>

Greg Ewing, 17.07.2010 03:09:
> Stefan Behnel wrote:
>> No, I think the only way to deal with this early enough is to introduce a
>> ReusedExpression node, and to handle that basically everywhere
>
> There was discussion a while back about some kind of
> LetNode -- did you implement that? Wouldn't it give you
> what you need here?

Yes, we have that, although the corresponding node here is the LetRefNode 
that holds the result itself.

When I initially wrote the above, I was looking into issues related to the 
ref-counting done by the LetRefNode. We currently use it for a couple of 
different use patterns, and some of them don't work correctly. For example, 
it's a difference if you just use it to reference a value that it owns, or 
if you keep reassigning other values to its temp variable. The latter 
doesn't currently work. It may be possible to make it work locally within 
the current nodes, but I'm not sure it is. Reassigning to temp variables 
isn't really something that is supported by the current infrastructure. 
That's the main issue here. There is currently no way to say that a temp 
variable already holds an owned reference that needs decref-ing.

Stefan

From mistobaan at gmail.com  Mon Jul 19 08:46:36 2010
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Mon, 19 Jul 2010 08:46:36 +0200
Subject: [Cython] OpenCL
Message-ID: 

Hi everyone

I want to explore a bit more the project of bringing support for
OpenCL to Cython.
Have been already any discussion about it ? ideas?

As far as I know, starting with a pxi of the standard OpenCL features
would be a first step.
And maybe later add support to kernel compilation.
I the last year I developed mainly in CUDA, but the principles apply
to OpenCL as well.

Thank you all for the awesome work done for Cython.

Looking forward for feedback

Fabrizio
------------------------------------------------------------------------------
Luck favors the prepared mind. (Pasteur)

From sturla at molden.no  Mon Jul 19 13:39:22 2010
From: sturla at molden.no (Sturla Molden)
Date: Mon, 19 Jul 2010 13:39:22 +0200
Subject: [Cython] OpenCL
In-Reply-To: 
References: 
Message-ID: <4C44396A.5050405@molden.no>

Fabrizio Milo aka misto skrev:
> Hi everyone
>
> I want to explore a bit more the project of bringing support for
> OpenCL to Cython.
>   
OpenCL's C API is just a few C functions, none of which are speed 
critical. The speed critical part is the kernel, which is just a text 
string in the program. There is already a PyOpenCL project, which nicely 
maps OpenCL C++ classes to Python objects and NumPy. It is useful from 
Python and Cython alike. You will probably not gain a lot from moving to 
Cython, as the Python overhead is easily amortized.

Sturla



From mistobaan at gmail.com  Mon Jul 19 13:54:27 2010
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Mon, 19 Jul 2010 13:54:27 +0200
Subject: [Cython] OpenCL
In-Reply-To: <4C44396A.5050405@molden.no>
References: 
	<4C44396A.5050405@molden.no>
Message-ID: 

> There is already a PyOpenCL project, which nicely
> maps OpenCL C++ classes to Python objects and NumPy.

The integration would allow extension to use OpenCL from inside the
C-extension without
going into python-space.

Translating a cython-like source code to opencl would be  a second cool step.
I now there is already something like that going on

http://clyther.sourceforge.net/

Fabrizio
--------------------------
Luck favors the prepared mind. (Pasteur)

From sturla at molden.no  Mon Jul 19 16:35:18 2010
From: sturla at molden.no (Sturla Molden)
Date: Mon, 19 Jul 2010 16:35:18 +0200
Subject: [Cython] OpenCL
In-Reply-To: 
References: 	<4C44396A.5050405@molden.no>
	
Message-ID: <4C4462A6.2000705@molden.no>

Fabrizio Milo aka misto skrev:
> The integration would allow extension to use OpenCL from inside the
> C-extension without
> going into python-space.
>   
Yes, but why? What do you hope to gain? The Python overhead is always 
amortized.

You have much bigger overhead here: the PCI bus trafficing data to and 
from video ram! That is slow. It is so slow that even a pure Python 
implementation will be faster for simple computations. You have to do a 
lot of computation on the GPU to amortize away this cost. I've seen some 
measurements indicating that a CPU can do hundreds of multiplications in 
the time needed just to tranfer just one byte. This is why GPUs are not 
used on massive supercomputers. So then a couple of extra Python objects 
will not matter for sure.

Anyway, the OpenCL API is just a few C functions you can easily define 
to Cython using

cdef extern from "opencl.h":
    whatever

But PyOpenCL will save you some headache setting it up correctly.

Getting rid of PyOpenCL's Python overhead belongs to the realm of 
immature optimization. Start by using PyOpenCL, if it proves to slow, 
and this is due to Python, then get rid of it. The real work is in 
writing the kernel anyway. The rest is just boilerplate.


Sturla

From mistobaan at gmail.com  Mon Jul 19 16:46:44 2010
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Mon, 19 Jul 2010 16:46:44 +0200
Subject: [Cython] OpenCL
In-Reply-To: <4C4462A6.2000705@molden.no>
References: 
	<4C44396A.5050405@molden.no>
	
	<4C4462A6.2000705@molden.no>
Message-ID: 

>this is why GPUs are not used on massive supercomputers.
I guess you don't work on massive supercomputer industry.

The ram transfer is right. the other assertions are not.
Have you ever developed in OpenCL or CUDA ?

Writing the kernel is the most important part.
Integration is also crucial.
Having one language to rule them all is nice.

I am sure a lot of folks of Sage would use OpenCl for their
computations for dataset bigger
than a defined threshold.

Thanks for the interest :)

Fabrizio
--------------------------
Luck favors the prepared mind. (Pasteur)

From gregor.thalhammer at gmail.com  Mon Jul 19 22:41:20 2010
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Mon, 19 Jul 2010 22:41:20 +0200
Subject: [Cython] OpenCL
In-Reply-To: 
References: 
Message-ID: 

Last week I thought about writing a OpenCL wrapper with Cython. I
spent nearly two days trying to compile PyOpenCL on my 64bit Win 7
system. On the other hand, compiling extensions with cython was always
smooth. I see clear advantages of using cython on maintability (e.g.,
porting to Python 3.1) and easy support for numpy arrays. I whish
existing wrappers would have used cython. I believe it's quite
straightforward, ideally replicating the C++ interface to OpenCL. Now
I managed to compile PyOpenCl, so my efforts are fading away.
I like the idea of clyther very much, to write OpenCL kernels in
python. Clyther is a young project, perhaps cython will rule it out.
On the cython wiki enhancements page there is the proposal to also
support OpenCl as a target language.
Gregor

2010/7/19 Fabrizio Milo aka misto :
> Hi everyone
>
> I want to explore a bit more the project of bringing support for
> OpenCL to Cython.
> Have been already any discussion about it ? ideas?
>
> As far as I know, starting with a pxi of the standard OpenCL features
> would be a first step.
> And maybe later add support to kernel compilation.
> I the last year I developed mainly in CUDA, but the principles apply
> to OpenCL as well.
>
> Thank you all for the awesome work done for Cython.
>
> Looking forward for feedback
>
> Fabrizio
> ------------------------------------------------------------------------------
> Luck favors the prepared mind. (Pasteur)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From greg.ewing at canterbury.ac.nz  Tue Jul 20 01:19:08 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 20 Jul 2010 11:19:08 +1200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: <4C43E0C7.20107@behnel.de>
References: 
	<4C3D80AB.2070003@behnel.de>
	
	<4C3F4690.3030300@behnel.de> <4C3F70D7.1070705@behnel.de>
	
	<4C3FD786.1080102@behnel.de> <4C4102B9.4050104@canterbury.ac.nz>
	<4C43E0C7.20107@behnel.de>
Message-ID: <4C44DD6C.9020407@canterbury.ac.nz>

Stefan Behnel wrote:
> For example, 
> it's a difference if you just use it to reference a value that it owns, or 
> if you keep reassigning other values to its temp variable. ... Reassigning
> to temp variables isn't really something that is supported by the current
 > infrastructure.

Hmmm... During the LetNode discussion, I wasn't really thinking
of the variables as being temp variables in the same sense as
those used for intermediate results. Rather, they would behave
more like real local variables, just as if the user had written
out the corresponding code.

Does thinking about it from that perspective help at all?

-- 
Greg

From dwf at cs.toronto.edu  Tue Jul 20 01:20:02 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Mon, 19 Jul 2010 19:20:02 -0400 (EDT)
Subject: [Cython] OpenCL
In-Reply-To: 
References: 
	
Message-ID: 



On Mon, 19 Jul 2010, Gregor Thalhammer wrote:

> Last week I thought about writing a OpenCL wrapper with Cython. I
> spent nearly two days trying to compile PyOpenCL on my 64bit Win 7
> system. On the other hand, compiling extensions with cython was always
> smooth.

The boost::python dependency was annoying, yes; you'll be happy to know 
that Andreas (author of PyOpenCL) has started shipping the minimal subset 
of boost::python that PyOpenCL needs instead of relying on it as an 
external dependency.

We encouraged him at the SciPy10 sprints to port to Cython but he realized 
that this would require a lot of effort and become far less maintainable
(I think C++ templating really helps keep PyOpenCL fairly compact). I 
think bundling a subset of boost is a reasonable compromise.

> I see clear advantages of using cython on maintability (e.g.,
> porting to Python 3.1) and easy support for numpy arrays.

NumPy arrays are actually extremely easy to support using boost. Not as 
nice as Cython, I don't think, but close.

> I believe it's quite straightforward, ideally replicating the
> C++ interface to OpenCL. Now I managed to compile PyOpenCl, so my 
> efforts are fading away.

I think one can do significantly better than the C++ interface; PyOpenCL 
is most of the way there.

> I like the idea of clyther very much, to write OpenCL kernels in
> python. Clyther is a young project, perhaps cython will rule it out.

For a slightly different tack, you should take a look at Theano:
http://www.deeplearning.net/software/theano -- which aims to do code 
generation from symbolic computation graphs. Currently they can generate 
C code, compile and dynamically load Python extension modules, as well
as do something similar with CUDA code, though James (one of the principal 
developers) is very interested in OpenCL, especially in its potential to 
target both CPU and GPU with only very minor modifications and thus 
reduce the complexity of Theano (though CPU implementations of OpenCL 
seem empirically more sluggish than what their C code generator + gcc can 
pull off currently, hopefully this will improve).

David

From stefan_ml at behnel.de  Wed Jul 21 10:35:24 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 21 Jul 2010 10:35:24 +0200
Subject: [Cython] bug in optimize_min_max
In-Reply-To: <4C44DD6C.9020407@canterbury.ac.nz>
References: 	<4C3D80AB.2070003@behnel.de>		<4C3F4690.3030300@behnel.de>
	<4C3F70D7.1070705@behnel.de>		<4C3FD786.1080102@behnel.de>
	<4C4102B9.4050104@canterbury.ac.nz>	<4C43E0C7.20107@behnel.de>
	<4C44DD6C.9020407@canterbury.ac.nz>
Message-ID: <4C46B14C.8010104@behnel.de>

Greg Ewing, 20.07.2010 01:19:
> Stefan Behnel wrote:
>> For example,
>> it's a difference if you just use it to reference a value that it owns, or
>> if you keep reassigning other values to its temp variable. ... Reassigning
>> to temp variables isn't really something that is supported by the current
>   >  infrastructure.
>
> Hmmm... During the LetNode discussion, I wasn't really thinking
> of the variables as being temp variables in the same sense as
> those used for intermediate results. Rather, they would behave
> more like real local variables, just as if the user had written
> out the corresponding code.
>
> Does thinking about it from that perspective help at all?

It may. So you'd basically introduce a third kind of variable (+ name 
prefix), similar to a NameNode but with limited scope. That might actually 
work better than what we have now. Worth a try.

Stefan

From kayhayen at gmx.de  Wed Jul 21 17:57:00 2010
From: kayhayen at gmx.de (Kay Hayen)
Date: Wed, 21 Jul 2010 17:57:00 +0200
Subject: [Cython] Pure Python
Message-ID: <4C4718CC.7050006@gmx.de>


Hello Cython-World,

some time ago, I found Cython and wanted to use it. I even came up with 
a couple of patches, which I believe didn't get merged, mostly because 
it soon became apparent that we have not the same goals completely.

My interest and goal was to use Cython for my pure Python code. I 
believe that Python is the language and that using language extensions 
that turn it to another language is not what I want.

Obviously, this is not exactly what Cython is about. So I thought of 
having my own "toy compiler" which translates pure Python into C or 
another language which is then compiled.

Time is short for me, I have a day job and everything, but this is to 
inform you that I believe I have succeeded with it.

That is, I can take pure Python 2.5 syntax and compile it to C++ and it 
does pass most of the CPython tests. It is capable of generator 
expressions and list comprehensions, selection or/and, etc. in all its 
beauty. It does lambda expressions, decorators for classes and 
functions. It does exception handling, args with "**" and "*", 
conditional expressions, and most interestingly, it is capable of 
generator functions (yield).

Well, just everything. You can imagine, I learned a lot about Python in 
the process.

My initial approach was to start from CPython bytecode. (Stefan, you 
likely recall that discussion.) Turned out that bytecode got me very 
far, also in understanding how CPython works, but for some cases, I just 
wasn't happy. I was just almost everything. I wanted everything though.

Then I used the parser module of Python itself, and that went much 
better. Of course the semantics was often new to me, and there is always 
somebody who uses a syntax in just one test.

But I managed to pull through that. BTW: Did you know [ x,y for x in 
range(4) for y in range(5) ] and could you predict its value? How often 
is each iterated being created? Deeply confusing stuff. I also 
absolutely enjoy these strangenesses.

Currently, I have for the first release 4 goals.

a) Pass the CPython tests even more completely. Some decorators e.g. use 
the function decorated also as a dictionary to remember things.

My "compiled function" type doesn't allow that yet.

I do handle eval, exec, locals, globals, execfile, etc. correctly, but 
there is stuff I will never have, e.g. a "func_code" object, or an 
execution frame. I find it very convenient to have the CPython tests, I 
also have some of my own, but being able use "unittest" was a real break 
through, you can surely imagine.

b) I never compiled the compiler with the compiler. I surely want to do 
that.

c) Whole program support. I currently compile module by module from 
".py" to ".so" or ".exe". I can recursively compile into os.path, but 
currently don't have the code to make it actually run.

d) Detect dead code. Really, the compiler is a result of solo extreme 
programming. I at least want the compiler to tell me about unused code 
and test coverage.

That would be milestone 1 and address primarily correctness. How fast it 
is, I didn't care much yet. In some cases faster than CPython and 
Cython, but that is not the point yet. Currently libpython does almost 
all the hard work, and what I mostly do, is to eliminate the bytecode 
generation and interpretation, and replace it with C++. Depending on the 
program, that is close to nothing.

The idea though, is clearly to get to the point, where I have it 
optimize the C type cases.

I expect that a release achieving these 3 goals will happen at the end 
of August. I hope to be able to build on your experiences as well, and I 
also hope to convince you to accept my idea of a "hints" module with 
decorators that should at least optionally replace the cdef syntax.

My core idea there, is that if I say "@hints.returns( int )" or 
something like this, it would decorate the function into something that 
asserts that this is really true, where for the compiler or Cython, the 
use of hints.returns() could be more in place.

Probably also interesting will be an exchange of optimization methods, 
although I have not ventured there much already. I did e.g. compile 
pystone only to find how much faster CPython was. Then I stole the 
specialization from them. And I guess, I can learn a lot more from 
Cython too. I am sure, I have a couple of tricks now, that I will gladly 
share after my release.

You clearly invested a lot of effort to see what's fast and what is not, 
if only because to me the C-API looks alike and you have tested those 
already.

Regarding newer CPython, I already use Python 2.6 syntax, but not the 
"with" statement, and I didn't look at 2.7 at all yet. I will probably 
only make the jump to there after the release. I do not care much about 
Python 3.x at this point.

Best regards,
Kay Hayen

From jared at jaredforsyth.com  Wed Jul 21 18:00:43 2010
From: jared at jaredforsyth.com (Jared Forsyth)
Date: Wed, 21 Jul 2010 12:00:43 -0400
Subject: [Cython] Pure Python
In-Reply-To: <4C4718CC.7050006@gmx.de>
References: <4C4718CC.7050006@gmx.de>
Message-ID: 

Feel like posting any links to source code?

Thanks,
Jared

On Wed, Jul 21, 2010 at 11:57 AM, Kay Hayen  wrote:

>
> Hello Cython-World,
>
> some time ago, I found Cython and wanted to use it. I even came up with
> a couple of patches, which I believe didn't get merged, mostly because
> it soon became apparent that we have not the same goals completely.
>
> My interest and goal was to use Cython for my pure Python code. I
> believe that Python is the language and that using language extensions
> that turn it to another language is not what I want.
>
> Obviously, this is not exactly what Cython is about. So I thought of
> having my own "toy compiler" which translates pure Python into C or
> another language which is then compiled.
>
> Time is short for me, I have a day job and everything, but this is to
> inform you that I believe I have succeeded with it.
>
> That is, I can take pure Python 2.5 syntax and compile it to C++ and it
> does pass most of the CPython tests. It is capable of generator
> expressions and list comprehensions, selection or/and, etc. in all its
> beauty. It does lambda expressions, decorators for classes and
> functions. It does exception handling, args with "**" and "*",
> conditional expressions, and most interestingly, it is capable of
> generator functions (yield).
>
> Well, just everything. You can imagine, I learned a lot about Python in
> the process.
>
> My initial approach was to start from CPython bytecode. (Stefan, you
> likely recall that discussion.) Turned out that bytecode got me very
> far, also in understanding how CPython works, but for some cases, I just
> wasn't happy. I was just almost everything. I wanted everything though.
>
> Then I used the parser module of Python itself, and that went much
> better. Of course the semantics was often new to me, and there is always
> somebody who uses a syntax in just one test.
>
> But I managed to pull through that. BTW: Did you know [ x,y for x in
> range(4) for y in range(5) ] and could you predict its value? How often
> is each iterated being created? Deeply confusing stuff. I also
> absolutely enjoy these strangenesses.
>
> Currently, I have for the first release 4 goals.
>
> a) Pass the CPython tests even more completely. Some decorators e.g. use
> the function decorated also as a dictionary to remember things.
>
> My "compiled function" type doesn't allow that yet.
>
> I do handle eval, exec, locals, globals, execfile, etc. correctly, but
> there is stuff I will never have, e.g. a "func_code" object, or an
> execution frame. I find it very convenient to have the CPython tests, I
> also have some of my own, but being able use "unittest" was a real break
> through, you can surely imagine.
>
> b) I never compiled the compiler with the compiler. I surely want to do
> that.
>
> c) Whole program support. I currently compile module by module from
> ".py" to ".so" or ".exe". I can recursively compile into os.path, but
> currently don't have the code to make it actually run.
>
> d) Detect dead code. Really, the compiler is a result of solo extreme
> programming. I at least want the compiler to tell me about unused code
> and test coverage.
>
> That would be milestone 1 and address primarily correctness. How fast it
> is, I didn't care much yet. In some cases faster than CPython and
> Cython, but that is not the point yet. Currently libpython does almost
> all the hard work, and what I mostly do, is to eliminate the bytecode
> generation and interpretation, and replace it with C++. Depending on the
> program, that is close to nothing.
>
> The idea though, is clearly to get to the point, where I have it
> optimize the C type cases.
>
> I expect that a release achieving these 3 goals will happen at the end
> of August. I hope to be able to build on your experiences as well, and I
> also hope to convince you to accept my idea of a "hints" module with
> decorators that should at least optionally replace the cdef syntax.
>
> My core idea there, is that if I say "@hints.returns( int )" or
> something like this, it would decorate the function into something that
> asserts that this is really true, where for the compiler or Cython, the
> use of hints.returns() could be more in place.
>
> Probably also interesting will be an exchange of optimization methods,
> although I have not ventured there much already. I did e.g. compile
> pystone only to find how much faster CPython was. Then I stole the
> specialization from them. And I guess, I can learn a lot more from
> Cython too. I am sure, I have a couple of tricks now, that I will gladly
> share after my release.
>
> You clearly invested a lot of effort to see what's fast and what is not,
> if only because to me the C-API looks alike and you have tested those
> already.
>
> Regarding newer CPython, I already use Python 2.6 syntax, but not the
> "with" statement, and I didn't look at 2.7 at all yet. I will probably
> only make the jump to there after the release. I do not care much about
> Python 3.x at this point.
>
> Best regards,
> Kay Hayen
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100721/437a8d45/attachment.htm 

From stefan_ml at behnel.de  Wed Jul 21 18:03:55 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 21 Jul 2010 18:03:55 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C4718CC.7050006@gmx.de>
References: <4C4718CC.7050006@gmx.de>
Message-ID: <4C471A6B.4000202@behnel.de>

Kay Hayen, 21.07.2010 17:57:
> I can take pure Python 2.5 syntax and compile it to C++ and it
> does pass most of the CPython tests.

You didn't provide a link to the sources. But why did you start from 
scratch instead of using ShedSkin?

Stefan

From dagss at student.matnat.uio.no  Wed Jul 21 21:54:00 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 21 Jul 2010 21:54:00 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C4718CC.7050006@gmx.de>
References: <4C4718CC.7050006@gmx.de>
Message-ID: <4C475058.9020703@student.matnat.uio.no>

On 07/21/2010 05:57 PM, Kay Hayen wrote:
> Hello Cython-World,
>
> some time ago, I found Cython and wanted to use it. I even came up with
> a couple of patches, which I believe didn't get merged, mostly because
> it soon became apparent that we have not the same goals completely.
>
> My interest and goal was to use Cython for my pure Python code. I
> believe that Python is the language and that using language extensions
> that turn it to another language is not what I want.
>
> Obviously, this is not exactly what Cython is about. So I thought of
> having my own "toy compiler" which translates pure Python into C or
> another language which is then compiled.
>    

I believe Cython is about this as well -- the extensions to the language 
is only because we have to (because Cython isn't smart enough in places 
where we need speed), not because we want to (except for communicating 
with native C libraries, that is).

Compiling as much pure Python code as possible is definitely a goal for 
Cython. Of course, within the context of egtensions loadable into the 
CPython interpreter. We are happy to accept patches that make pure 
Python code without any Cython-specific syntax run faster. We also have 
a mode using Python decorators to provide types ("pure Python" mode of 
Cython).

As Stefan noted, ShedSkin compiles Python to C without CPython. Of 
course, doing something from scratch can be fun and a learning experience.

Dag Sverre

From kayhayen at gmx.de  Wed Jul 21 21:51:53 2010
From: kayhayen at gmx.de (Kay Hayen)
Date: Wed, 21 Jul 2010 21:51:53 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C471A6B.4000202@behnel.de>
References: <4C4718CC.7050006@gmx.de> <4C471A6B.4000202@behnel.de>
Message-ID: <4C474FD9.1000904@gmx.de>

Hello Stefan,

you wrote:

> Kay Hayen, 21.07.2010 17:57:
>> I can take pure Python 2.5 syntax and compile it to C++ and it
>> does pass most of the CPython tests.
>
> You didn't provide a link to the sources. But why did you start from
> scratch instead of using ShedSkin?

I am not yet ready to release the sources to the world. I want it to be 
actually useful before people note that it's not. And before people see 
relicts of previous attempts and are just confused by it.

Lets say it's not cleaned up well enough, for me to feel comfortable 
with a release right now. Only on a functional level, I have become 
confident.

I think one month is a good time frame. I promise to release it no 
matter what the state will be at the end of August. For private reasons, 
I will have 3 weeks in August, where I can invest significant time, to 
make it a good code drop. Plus, I have not even decided on a name yet.

I evaluated ShedSkin too, and it seemed at the time to have different 
goals and not as much driven by a community either. It seemed to try and 
use C++ standard containers over Python containers to achieve speedups.

Actually I am not at all interested in the C++ part. I am still somewhat 
competent in it as a programming language, but also had to research 
quite a bit on it, to get some things represented. But the use of C++ 
was not at clear to me.

I have also considered to generate Vala or D, or even Ada (which I truly 
love as much as Python for different tasks), but the plan was always 
that this language is not supposed to be important at all, and newer 
languages are evolving themselves too much.

I do not consider that integration of C++ code with Python code is 
useful to me. I want a tool that lets me write in Python and still not 
be "slow". Emphasis on that last part, "to me". I just want something 
faster, that falls back to CPython. I imagine people run the compiler 
over any program, and it's faster, but nothing changed in its behaviour.

For example, I wrote a bit level decoding tool in Python, even in nice 
to read Python, and I want the compiler to make it efficient, based on 
the knowledge the input was a str and what ord(), len(). [] typically do 
to it. Obviously writing it in C would be much more efficient. But also 
much less fun.

In the end C++0x support in recent gcc releases convinced me that C++ is 
a good and useful target language. With some variadic template functions 
e.g., I was able to push a couple of things into the C++ part, where 
previously I would have generated code for it.

I noticed from discussions here that many people do care about C++ and I 
am amazed at some of the things possible with Cython now. It clearly has 
become ever more powerful is this domain. Impressive work you did there 
guys.

Best regards,
Kay Hayen

From kayhayen at gmx.de  Thu Jul 22 10:09:26 2010
From: kayhayen at gmx.de (Kay Hayen)
Date: Thu, 22 Jul 2010 10:09:26 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C475058.9020703@student.matnat.uio.no>
References: <4C4718CC.7050006@gmx.de> <4C475058.9020703@student.matnat.uio.no>
Message-ID: <4C47FCB6.8070308@gmx.de>

Hello Dag,

 > [...] We also have
> a mode using Python decorators to provide types ("pure Python" mode of
> Cython).

Ah, that's very interesting. At the time I used Cython (I still have the 
0.10.3 release somewhere), this was not present. So clearly, I will 
study the Cython docs and try to use that syntax instead.

Being Python is so important for pylint, pychecker and most importantly 
flymake mode to work well. When I first encountered and used PyRex many 
years ago, that was immediately the problem with it.

Also at the time the Cython project was very hesitant to make bigger 
changes, because it was following PyRex where possible. I wanted to e.g. 
change nodes and reference counting, so that generated code would be 
more local, and had written (to be generated) C code to back it up.

Potentially PyRex similarity is no longer as important as it once was to 
the project as well.

My main motivating factor for rolling my own stuff was to have a toy 
where one could more easily demonstrate the usefulness or not of things, 
in the first place. Then I can try and integrate it into Cython, which 
will have the actual community behind it.

But as I said, it will be merely "correct" code generated. It won't be 
actually better code yet. But it will e.g. demonstrate a way to easily 
handle generator expressions and functions, something I believe Cython 
still doesn't do?

It will be a good tool for proof of concepts and overall it will help 
the research for the best path by trying out others. And who knows, 
potentially it will become useful by itself too.

> As Stefan noted, ShedSkin compiles Python to C without CPython. Of
> course, doing something from scratch can be fun and a learning experience.

I admire this project and also what Google does with aiming at LLVM 
instead, but I'm not out to replace CPython at all. I just want to use 
readable Python and have a compiler to make up for it.

As to the learning experience, you are correct, it's fun and useful, 
very much. Just learning how wild slicing in CPython can become, was 
really amazing.

Best regards,
Kay Hayen

From dagss at student.matnat.uio.no  Thu Jul 22 10:28:07 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Jul 2010 10:28:07 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C47FCB6.8070308@gmx.de>
References: <4C4718CC.7050006@gmx.de> <4C475058.9020703@student.matnat.uio.no>
	<4C47FCB6.8070308@gmx.de>
Message-ID: <4C480117.1030704@student.matnat.uio.no>

Kay Hayen wrote:
> Hello Dag,
>
>  > [...] We also have
>   
>> a mode using Python decorators to provide types ("pure Python" mode of
>> Cython).
>>     
>
> Ah, that's very interesting. At the time I used Cython (I still have the 
> 0.10.3 release somewhere), this was not present. So clearly, I will 
> study the Cython docs and try to use that syntax instead.
>   
See http://wiki.cython.org/pure

It's not as developed as we'd like it to be, but at least it is a proof 
that we have the same aims that you have :-) And any help in this area 
in general is very much appreciated; it wouldn't need to be using the 
current syntax if something else is much better.

Haoyu Bai is currently doing a GSoC which may introduce Python 3 
annotations as well:

def f(x: cython.int) -> cython.int:
    ...

> Being Python is so important for pylint, pychecker and most importantly 
> flymake mode to work well. When I first encountered and used PyRex many 
> years ago, that was immediately the problem with it.
>
> Also at the time the Cython project was very hesitant to make bigger 
> changes, because it was following PyRex where possible. I wanted to e.g. 
> change nodes and reference counting, so that generated code would be 
> more local, and had written (to be generated) C code to back it up.
>
> Potentially PyRex similarity is no longer as important as it once was to 
> the project as well.
>   
Pyrex compatibility is much less important than it was, though we of 
course want as much backwards compatibility as possible with older 
Cython versions, which automatically means not breaking too much of the 
Pyrex syntax. But, as you see we're happy for *additional* syntax in a 
pure Python mode, and then we don't need to care about Pyrex at all.

The source code bases has diverged a lot, one cannot take patches from 
Pyrex and apply on the Cython tree any longer.
> My main motivating factor for rolling my own stuff was to have a toy 
> where one could more easily demonstrate the usefulness or not of things, 
> in the first place. Then I can try and integrate it into Cython, which 
> will have the actual community behind it.
>
> But as I said, it will be merely "correct" code generated. It won't be 
> actually better code yet. But it will e.g. demonstrate a way to easily 
> handle generator expressions and functions, something I believe Cython 
> still doesn't do?
>   
Closures (inner functions) are present in the next release (0.13). 
Generator expressions are expected to land sometimes in the next half 
year to a year, I think, though always take such estimates with a grain 
of salt.

Dag Sverre

From stefan_ml at behnel.de  Thu Jul 22 10:45:50 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Jul 2010 10:45:50 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C47FCB6.8070308@gmx.de>
References: <4C4718CC.7050006@gmx.de> <4C475058.9020703@student.matnat.uio.no>
	<4C47FCB6.8070308@gmx.de>
Message-ID: <4C48053E.9040000@behnel.de>

Kay Hayen, 22.07.2010 10:09:
> Being Python is so important for pylint, pychecker and most importantly
> flymake mode to work well. When I first encountered and used PyRex many
> years ago, that was immediately the problem with it.

Absolutely.


> My main motivating factor for rolling my own stuff was to have a toy
> where one could more easily demonstrate the usefulness or not of things,
> in the first place. Then I can try and integrate it into Cython, which
> will have the actual community behind it.

Your elaborations keep reminding me of this article:

http://www.joelonsoftware.com/articles/fog0000000069.html


> But as I said, it will be merely "correct" code generated. It won't be
> actually better code yet. But it will e.g. demonstrate a way to easily
> handle generator expressions and functions, something I believe Cython
> still doesn't do?

Well, no, not really. We have /inlined/ generator expressions for some 
special use cases, and I have written a CEP about implementing generators 
for Cython:

http://wiki.cython.org/enhancements/generators

But no-one has found the time to do it yet. I think it's easier if you can 
start from scratch (as ShedSkin and you did) and don't have to integrate 
with CPython's infrastructure like Cython. Generators aren't complex at 
all, but making them look and work like CPython generators isn't trivial.


>> As Stefan noted, ShedSkin compiles Python to C without CPython. Of
>> course, doing something from scratch can be fun and a learning experience.
>
> I admire this project and also what Google does with aiming at LLVM
> instead, but I'm not out to replace CPython at all. I just want to use
> readable Python and have a compiler to make up for it.

Hmm, then I still don't get what the purpose of your project is. I mean, 
it's ok to do it because "why not", but you seem to have specifically aimed 
for a CPython replacement (as opposed to extending CPython, as Cython 
does). So you're more in line with ShedSkin and PyPy than with Cython.

Stefan

From dagss at student.matnat.uio.no  Thu Jul 22 10:53:17 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 22 Jul 2010 10:53:17 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C48053E.9040000@behnel.de>
References: <4C4718CC.7050006@gmx.de>
	<4C475058.9020703@student.matnat.uio.no>	<4C47FCB6.8070308@gmx.de>
	<4C48053E.9040000@behnel.de>
Message-ID: <4C4806FD.8000300@student.matnat.uio.no>

Stefan Behnel wrote:
> Kay Hayen, 22.07.2010 10:09:
>   
>> Being Python is so important for pylint, pychecker and most importantly
>> flymake mode to work well. When I first encountered and used PyRex many
>> years ago, that was immediately the problem with it.
>>     
>
> Absolutely.
>
>
>   
>> My main motivating factor for rolling my own stuff was to have a toy
>> where one could more easily demonstrate the usefulness or not of things,
>> in the first place. Then I can try and integrate it into Cython, which
>> will have the actual community behind it.
>>     
>
> Your elaborations keep reminding me of this article:
>
> http://www.joelonsoftware.com/articles/fog0000000069.html
>   
Yes, though that ignores the "fun" aspect of it all :-)
>
>   
>>> As Stefan noted, ShedSkin compiles Python to C without CPython. Of
>>> course, doing something from scratch can be fun and a learning experience.
>>>       
>> I admire this project and also what Google does with aiming at LLVM
>> instead, but I'm not out to replace CPython at all. I just want to use
>> readable Python and have a compiler to make up for it.
>>     
>
> Hmm, then I still don't get what the purpose of your project is. I mean, 
> it's ok to do it because "why not", but you seem to have specifically aimed 
> for a CPython replacement (as opposed to extending CPython, as Cython 
> does). So you're more in line with ShedSkin and PyPy than with Cython.
>   
Curiously, I don't get this impression at all. This paragraph in the OP 
was perhaps confusing?

"""

 I 
believe that Python is the language and that using language extensions 
that turn it to another language is not what I want.

"""

Note that it says "language extensions", not "Python extensions" :-) 
Also note that Kay is "not out to replace CPython at all". To me it 
seems that Cython w/ pure Python mode is pretty much exactly what the 
project is about?

Dag Sverre

From stefan_ml at behnel.de  Thu Jul 22 11:00:43 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Jul 2010 11:00:43 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C4806FD.8000300@student.matnat.uio.no>
References: <4C4718CC.7050006@gmx.de>	<4C475058.9020703@student.matnat.uio.no>	<4C47FCB6.8070308@gmx.de>	<4C48053E.9040000@behnel.de>
	<4C4806FD.8000300@student.matnat.uio.no>
Message-ID: <4C4808BB.6000902@behnel.de>

Dag Sverre Seljebotn, 22.07.2010 10:53:
> Stefan Behnel wrote:
>> Kay Hayen, 22.07.2010 10:09:
 >>> Dag wrote:
>>>> As Stefan noted, ShedSkin compiles Python to C without CPython. Of
>>>> course, doing something from scratch can be fun and a learning experience.
>>>>
>>> I admire this project and also what Google does with aiming at LLVM
>>> instead, but I'm not out to replace CPython at all. I just want to use
>>> readable Python and have a compiler to make up for it.
>>
>> Hmm, then I still don't get what the purpose of your project is. I mean,
>> it's ok to do it because "why not", but you seem to have specifically aimed
>> for a CPython replacement (as opposed to extending CPython, as Cython
>> does). So you're more in line with ShedSkin and PyPy than with Cython.
>
> Curiously, I don't get this impression at all.

Hmmm, looks like I missed this paragraph in the original mail:

"""
Currently libpython does almost
all the hard work, and what I mostly do, is to eliminate the bytecode
generation and interpretation, and replace it with C++. Depending on the
program, that is close to nothing.
"""

This may indicate that the generated C++ code actually interfaces with 
CPython. Maybe Kay can clear this up.

Stefan

From kayhayen at gmx.de  Thu Jul 22 11:02:27 2010
From: kayhayen at gmx.de (Kay Hayen)
Date: Thu, 22 Jul 2010 11:02:27 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C480117.1030704@student.matnat.uio.no>
References: <4C4718CC.7050006@gmx.de>
	<4C475058.9020703@student.matnat.uio.no>	<4C47FCB6.8070308@gmx.de>
	<4C480117.1030704@student.matnat.uio.no>
Message-ID: <4C480923.1090503@gmx.de>

> See http://wiki.cython.org/pure
>
> It's not as developed as we'd like it to be, but at least it is a proof
> that we have the same aims that you have :-) And any help in this area
> in general is very much appreciated; it wouldn't need to be using the
> current syntax if something else is much better.

it looks quite nice, except that it hardcodes a name. Is there a Python 
cython module with Python source code that keeps code working with CPython?

>> But as I said, it will be merely "correct" code generated. It won't be
>> actually better code yet. But it will e.g. demonstrate a way to easily
>> handle generator expressions and functions, something I believe Cython
>> still doesn't do?
>>
> Closures (inner functions) are present in the next release (0.13).
> Generator expressions are expected to land sometimes in the next half
> year to a year, I think, though always take such estimates with a grain
> of salt.

Well yes, I am adequately proud that my compiler does not only do 
closures, but generator expressions and generator functions as closures, 
classes with closures, and meta classes.

All of that. And I find Python without to be not the same language nearly.

Yours,
Kay

From stefan_ml at behnel.de  Thu Jul 22 11:08:41 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Jul 2010 11:08:41 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C480923.1090503@gmx.de>
References: <4C4718CC.7050006@gmx.de>	<4C475058.9020703@student.matnat.uio.no>	<4C47FCB6.8070308@gmx.de>	<4C480117.1030704@student.matnat.uio.no>
	<4C480923.1090503@gmx.de>
Message-ID: <4C480A99.1070101@behnel.de>

Kay Hayen, 22.07.2010 11:02:
>> See http://wiki.cython.org/pure
>>
>> It's not as developed as we'd like it to be, but at least it is a proof
>> that we have the same aims that you have :-) And any help in this area
>> in general is very much appreciated; it wouldn't need to be using the
>> current syntax if something else is much better.
>
> it looks quite nice, except that it hardcodes a name. Is there a Python
> cython module with Python source code that keeps code working with CPython?

Sure.


>>> But as I said, it will be merely "correct" code generated. It won't be
>>> actually better code yet. But it will e.g. demonstrate a way to easily
>>> handle generator expressions and functions, something I believe Cython
>>> still doesn't do?
>>>
>> Closures (inner functions) are present in the next release (0.13).
>> Generator expressions are expected to land sometimes in the next half
>> year to a year, I think, though always take such estimates with a grain
>> of salt.
>
> Well yes, I am adequately proud that my compiler does not only do
> closures, but generator expressions and generator functions as closures,
> classes with closures, and meta classes.

Yep, class closures are another long-standing open issue in Cython, and 
meta-classes aren't there, either.

Stefan

From stefan_ml at behnel.de  Thu Jul 22 11:14:07 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Jul 2010 11:14:07 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C480A99.1070101@behnel.de>
References: <4C4718CC.7050006@gmx.de>	<4C475058.9020703@student.matnat.uio.no>	<4C47FCB6.8070308@gmx.de>	<4C480117.1030704@student.matnat.uio.no>	<4C480923.1090503@gmx.de>
	<4C480A99.1070101@behnel.de>
Message-ID: <4C480BDF.3080407@behnel.de>

Stefan Behnel, 22.07.2010 11:08:
> Kay Hayen, 22.07.2010 11:02:
>> Well yes, I am adequately proud that my compiler does not only do
>> closures, but generator expressions and generator functions as closures,
>> classes with closures, and meta classes.
>
> Yep, class closures are another long-standing open issue in Cython, and
> meta-classes aren't there, either.

BTW, I think the main reason why these are missing is that Cython 
development is largely driven by personal needs, and these features are 
rarely used in real-world Python code. Also, as a descendent of Pyrex, 
Cython was originally not intended to compile regular Python code, and most 
code that it compiles even today was written either for Pyrex or Cython 
specifically. That reduces the urge to get unimplemented language features 
done. I think generators are the most frequently requested one, though.

Stefan

From kayhayen at gmx.de  Thu Jul 22 11:21:49 2010
From: kayhayen at gmx.de (Kay Hayen)
Date: Thu, 22 Jul 2010 11:21:49 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C4808BB.6000902@behnel.de>
References: <4C4718CC.7050006@gmx.de>	<4C475058.9020703@student.matnat.uio.no>	<4C47FCB6.8070308@gmx.de>	<4C48053E.9040000@behnel.de>	<4C4806FD.8000300@student.matnat.uio.no>
	<4C4808BB.6000902@behnel.de>
Message-ID: <4C480DAD.80305@gmx.de>

hello Stefan,

> This may indicate that the generated C++ code actually interfaces with
> CPython. Maybe Kay can clear this up.

Well, the core of building a class, is e.g. this code:

    PyObject *result = PyObject_CallFunctionObjArgs( metaclass, 
class_name, bases, dict, NULL );

So, I let CPython call the metaclass, and do not yet even attempt to get 
in its way with it. So support for metaclasses sounds big, but actually 
was small enough.

The only difficulty is selecting the metaclass from the given bases if 
not specified. Then set a couple of attributes, much like the code in 
CPython does, that builds a class.

So well, yes, behind meta classes is nearly nothing. The hard work was 
functions/generators and closures. Once these are like normal functions, 
they can be used in classes, and it works without any optimization. I 
just need to have code some places, where the C-API does not expose 
details the CPython core takes care of.

Rest assured, the project was simple enough for one person. :-)

Yours,
Kay


From robertwb at math.washington.edu  Thu Jul 22 11:25:02 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 22 Jul 2010 02:25:02 -0700
Subject: [Cython] Pure Python
In-Reply-To: <4C480BDF.3080407@behnel.de>
References: <4C4718CC.7050006@gmx.de> <4C475058.9020703@student.matnat.uio.no>
	<4C47FCB6.8070308@gmx.de> <4C480117.1030704@student.matnat.uio.no>
	<4C480923.1090503@gmx.de> <4C480A99.1070101@behnel.de>
	<4C480BDF.3080407@behnel.de>
Message-ID: 

On Thu, Jul 22, 2010 at 2:14 AM, Stefan Behnel  wrote:
> Stefan Behnel, 22.07.2010 11:08:
>> Kay Hayen, 22.07.2010 11:02:
>>> Well yes, I am adequately proud that my compiler does not only do
>>> closures, but generator expressions and generator functions as closures,
>>> classes with closures, and meta classes.
>>
>> Yep, class closures are another long-standing open issue in Cython, and
>> meta-classes aren't there, either.
>
> BTW, I think the main reason why these are missing is that Cython
> development is largely driven by personal needs, and these features are
> rarely used in real-world Python code. Also, as a descendent of Pyrex,
> Cython was originally not intended to compile regular Python code, and most
> code that it compiles even today was written either for Pyrex or Cython
> specifically. That reduces the urge to get unimplemented language features
> done. I think generators are the most frequently requested one, though.

This is surely the reason. I would go so far as to say the majority of
the development (and complexity) is due to the fact that we want to
operate will with C types and libraries (and cdef classes), which is
where most of our users see the most value. Of course we are not as
tied to this as we used to be, and look forward to the day when most
of this happens transparently under the hood.

Generators are hot on our list, and that's the last big item I can
think of (though there's countless little ones.)

- Robert

From stefan_ml at behnel.de  Thu Jul 22 11:32:52 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 22 Jul 2010 11:32:52 +0200
Subject: [Cython] Pure Python
In-Reply-To: <4C480DAD.80305@gmx.de>
References: <4C4718CC.7050006@gmx.de>	<4C475058.9020703@student.matnat.uio.no>	<4C47FCB6.8070308@gmx.de>	<4C48053E.9040000@behnel.de>	<4C4806FD.8000300@student.matnat.uio.no>	<4C4808BB.6000902@behnel.de>
	<4C480DAD.80305@gmx.de>
Message-ID: <4C481044.6030701@behnel.de>

Kay Hayen, 22.07.2010 11:21:
>> This may indicate that the generated C++ code actually interfaces with
>> CPython. Maybe Kay can clear this up.

I take it from your response that your implementation is actually closer to 
Cython than I thought, except that it is currently focused purely on being 
a Python compiler.


> Well, the core of building a class, is e.g. this code:
>
>      PyObject *result = PyObject_CallFunctionObjArgs( metaclass,
> class_name, bases, dict, NULL );
>
> So, I let CPython call the metaclass, and do not yet even attempt to get
> in its way with it.

Ah, ok, so this is only meant for Python classes, right? That's simple 
enough, sure. The tricky thing in Cython is the 'expected' interaction with 
extension types, i.e. a language design question. That's currently not 
clear at all.

Stefan

From erik.tollerud at gmail.com  Thu Jul 22 19:04:19 2010
From: erik.tollerud at gmail.com (Erik Tollerud)
Date: Thu, 22 Jul 2010 10:04:19 -0700
Subject: [Cython] Apache License link wrong on Cython web page
Message-ID: 

While strictly speaking not a "development" issue, this is probably
the best place to post this...?

The link on the front page to the Apache License is no longer leading
to the Apache License, instead it points to some page for bad domain
names... I think you want to change it to
http://www.apache.org/licenses/LICENSE-2.0.html

-- 
Erik Tollerud

From robertwb at math.washington.edu  Thu Jul 22 19:49:36 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 22 Jul 2010 10:49:36 -0700
Subject: [Cython] Apache License link wrong on Cython web page
In-Reply-To: 
References: 
Message-ID: 

On Thu, Jul 22, 2010 at 10:04 AM, Erik Tollerud  wrote:
> While strictly speaking not a "development" issue, this is probably
> the best place to post this...?
>
> The link on the front page to the Apache License is no longer leading
> to the Apache License, instead it points to some page for bad domain
> names... I think you want to change it to
> http://www.apache.org/licenses/LICENSE-2.0.html

Thanks. Yes, this is fine place to fix this. Done.

- Robert

From cb at pdos.csail.mit.edu  Fri Jul 23 00:27:14 2010
From: cb at pdos.csail.mit.edu (Chuck Blake)
Date: Thu, 22 Jul 2010 18:27:14 -0400
Subject: [Cython] Cython-dev Digest, Vol 31, Issue 15
Message-ID: <20100722222714.GA1026@pdos.lcs.mit.edu>

It would be nice to be able to support the new command line
compiler directives in distutils-based setup.py build scripts.
E.g., something like this

    extens = [ Extension(...), Extension(...), ... ]
    for x in extens:
        x.pyrex_directives = { "infer_types.verbose": True,
                               "embedsignature": True }

The attached hg export format patch (against the current
cython-closures tip) accomplishes this.  I've run this
and tested it on my own setup.py's and it has worked fine.
Thoughts?
-------------- next part --------------
# HG changeset patch
# User Chuck Blake 
# Date 1279837066 14400
# Node ID f3ce2586682a874544a1ceb2318ab388fbc81f3e
# Parent  f04cdb0edf515ebe1baa348959ad17d7dbf40a73
Add pyrex_directives dictionary optional attribute of Extension objects to
support distutils/setup.py-based directives setting.

diff -r f04cdb0edf51 -r f3ce2586682a Cython/Distutils/build_ext.py
--- a/Cython/Distutils/build_ext.py	Wed Jul 21 00:50:00 2010 +0200
+++ b/Cython/Distutils/build_ext.py	Thu Jul 22 18:17:46 2010 -0400
@@ -44,6 +44,8 @@
          "put generated C files in temp directory"),
         ('pyrex-gen-pxi', None,
             "generate .pxi file for public declarations"),
+        ('pyrex-directives=', None,
+            "compiler directive overrides"),
         ])
 
     boolean_options.extend([
@@ -56,6 +58,7 @@
         self.pyrex_create_listing = 0
         self.pyrex_line_directives = 0
         self.pyrex_include_dirs = None
+        self.pyrex_directives = None
         self.pyrex_c_in_temp = 0
         self.pyrex_gen_pxi = 0
 
@@ -66,6 +69,8 @@
         elif type(self.pyrex_include_dirs) is StringType:
             self.pyrex_include_dirs = \
                 self.pyrex_include_dirs.split(os.pathsep)
+        if self.pyrex_directives is None:
+            self.pyrex_directives = {}
     # finalize_options ()
 
     def build_extensions(self):
@@ -139,6 +144,13 @@
             if not i in includes:
                 includes.append(i)
 
+        # Set up Cython compiler directives:
+        #    1. Start with the command line option.
+        #    2. Add in any (unique) entries from the extension
+        #         pyrex_directives (if Cython.Distutils.extension is used).
+        directives = self.pyrex_directives
+        directives.update(extension.pyrex_directives)
+
         # Set the target_ext to '.c'.  Cython will change this to '.cpp' if
         # needed.
         if cplus:
@@ -189,6 +201,7 @@
                 options = CompilationOptions(pyrex_default_options, 
                     use_listing_file = create_listing,
                     include_path = includes,
+                    compiler_directives = directives,
                     output_file = target,
                     cplus = cplus,
                     emit_linenums = line_directives,
diff -r f04cdb0edf51 -r f3ce2586682a Cython/Distutils/extension.py
--- a/Cython/Distutils/extension.py	Wed Jul 21 00:50:00 2010 +0200
+++ b/Cython/Distutils/extension.py	Thu Jul 22 18:17:46 2010 -0400
@@ -19,6 +19,8 @@
     """pyrex_include_dirs : [string]
         list of directories to search for Pyrex header files (.pxd) (in
         Unix form for portability)
+    pyrex_directives : {string:value}
+        dict of compiler directives
     pyrex_create_listing_file : boolean
         write pyrex error messages to a listing (.lis) file.
     pyrex_line_directivess : boolean
@@ -48,6 +50,7 @@
             depends = None,
             language = None,
             pyrex_include_dirs = None,
+            pyrex_directives = None,
             pyrex_create_listing = 0,
             pyrex_line_directives = 0,
             pyrex_cplus = 0,
@@ -72,6 +75,7 @@
             **kw)
 
         self.pyrex_include_dirs = pyrex_include_dirs or []
+        self.pyrex_directives = pyrex_directives or {}
         self.pyrex_create_listing = pyrex_create_listing
         self.pyrex_line_directives = pyrex_line_directives
         self.pyrex_cplus = pyrex_cplus

From robertwb at math.washington.edu  Fri Jul 23 01:02:48 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 22 Jul 2010 16:02:48 -0700
Subject: [Cython] Cython-dev Digest, Vol 31, Issue 15
In-Reply-To: <20100722222714.GA1026@pdos.lcs.mit.edu>
References: <20100722222714.GA1026@pdos.lcs.mit.edu>
Message-ID: 

On Thu, Jul 22, 2010 at 3:27 PM, Chuck Blake  wrote:
> It would be nice to be able to support the new command line
> compiler directives in distutils-based setup.py build scripts.
> E.g., something like this
>
> ? ?extens = [ Extension(...), Extension(...), ... ]
> ? ?for x in extens:
> ? ? ? ?x.pyrex_directives = { "infer_types.verbose": True,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "embedsignature": True }
>
> The attached hg export format patch (against the current
> cython-closures tip) accomplishes this. ?I've run this
> and tested it on my own setup.py's and it has worked fine.
> Thoughts?

Looks good to me, thanks! I'm not sure pyrex_directive is the right
attribute to use here, as pyrex itself doesn't have (as far as I know)
the notion of directives, but that's not a big deal and I'm sure it'll
be very useful. I've pushed it.

- Robert

From cb at pdos.csail.mit.edu  Fri Jul 23 04:28:32 2010
From: cb at pdos.csail.mit.edu (Chuck Blake)
Date: Thu, 22 Jul 2010 22:28:32 -0400
Subject: [Cython] distutils compiler directives
In-Reply-To: 
References: <20100722222714.GA1026@pdos.lcs.mit.edu>
	
Message-ID: <20100723022832.GA3691@pdos.lcs.mit.edu>

>Looks good to me, thanks! I'm not sure pyrex_directive is the right
>attribute to use here, as pyrex itself doesn't have (as far as I know)
>the notion of directives, but that's not a big deal and I'm sure it'll
>be very useful. I've pushed it.

Everything else is the distutils stuff is prefixed with pyrex_, and
I believe some of those are cython-only as well.  I'm already loving
infertypes.verbose enabled on all my packages.

There was a problem with the Hudson autobuild.  I probably too closely
imitated the logic for pyrex_include_dirs...The attached fixes it and
I don't think will ever be wrong but someone might want to look at
this more carefully.  (Also, sorry about the original subject line.).
-------------- next part --------------
# HG changeset patch
# User Chuck Blake 
# Date 1279851105 14400
# Node ID 30e0e8a9169ea14821336257fe71733cd58f582e
# Parent  f3ce2586682a874544a1ceb2318ab388fbc81f3e
Easy enough to wrap in a hasattr.  On my system only py31 complains, but on
the Hudson autobuild, most versions complain about a missing attribute.

diff -r f3ce2586682a -r 30e0e8a9169e Cython/Distutils/build_ext.py
--- a/Cython/Distutils/build_ext.py	Thu Jul 22 18:17:46 2010 -0400
+++ b/Cython/Distutils/build_ext.py	Thu Jul 22 22:11:45 2010 -0400
@@ -149,7 +149,8 @@
         #    2. Add in any (unique) entries from the extension
         #         pyrex_directives (if Cython.Distutils.extension is used).
         directives = self.pyrex_directives
-        directives.update(extension.pyrex_directives)
+        if hasattr(extension, "pyrex_directives"):
+            directives.update(extension.pyrex_directives)
 
         # Set the target_ext to '.c'.  Cython will change this to '.cpp' if
         # needed.

From robertwb at math.washington.edu  Fri Jul 23 05:38:28 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 22 Jul 2010 20:38:28 -0700
Subject: [Cython] distutils compiler directives
In-Reply-To: <20100723022832.GA3691@pdos.lcs.mit.edu>
References: <20100722222714.GA1026@pdos.lcs.mit.edu>
	
	<20100723022832.GA3691@pdos.lcs.mit.edu>
Message-ID: 

On Thu, Jul 22, 2010 at 7:28 PM, Chuck Blake  wrote:
>>Looks good to me, thanks! I'm not sure pyrex_directive is the right
>>attribute to use here, as pyrex itself doesn't have (as far as I know)
>>the notion of directives, but that's not a big deal and I'm sure it'll
>>be very useful. I've pushed it.
>
> Everything else is the distutils stuff is prefixed with pyrex_, and
> I believe some of those are cython-only as well. ?I'm already loving
> infertypes.verbose enabled on all my packages.
>
> There was a problem with the Hudson autobuild. ?I probably too closely
> imitated the logic for pyrex_include_dirs...The attached fixes it and
> I don't think will ever be wrong but someone might want to look at
> this more carefully. ?(Also, sorry about the original subject line.).

Thanks. Looking much better :)

- Robert

From stefan_ml at behnel.de  Fri Jul 23 08:10:58 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Jul 2010 08:10:58 +0200
Subject: [Cython] Inline caching for method calls
Message-ID: <4C493272.3000506@behnel.de>

Hi,

Stefan Brunthaler started a thread on python-dev about a couple of patches 
he has written for CPython as part of his PhD. It seems that he has 
implemented inline caching for CPython.

http://comments.gmane.org/gmane.comp.python.devel/115362

http://en.wikipedia.org/wiki/Inline_caching

Something like this might be interesting for Cython, too. He hasn't 
published the patches yet, but they might well be worth a look when he does.

Stefan

From robertwb at math.washington.edu  Fri Jul 23 08:52:02 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 22 Jul 2010 23:52:02 -0700
Subject: [Cython] Inline caching for method calls
In-Reply-To: <4C493272.3000506@behnel.de>
References: <4C493272.3000506@behnel.de>
Message-ID: 

On Thu, Jul 22, 2010 at 11:10 PM, Stefan Behnel  wrote:
> Hi,
>
> Stefan Brunthaler started a thread on python-dev about a couple of patches
> he has written for CPython as part of his PhD. It seems that he has
> implemented inline caching for CPython.
>
> http://comments.gmane.org/gmane.comp.python.devel/115362
>
> http://en.wikipedia.org/wiki/Inline_caching
>
> Something like this might be interesting for Cython, too. He hasn't
> published the patches yet, but they might well be worth a look when he does.

Very cool. I have actually thought a lot about this, and I know
unladen swallow did some stuff in this direction for module/builtin
lookups. Ideally, there could be some kind of a hook called, or even a
dirty bit, that gets set when an objects attribute set gets modified,
so we wouldn't have to re-look things up most of the time.

- Robert

From stefan_ml at behnel.de  Fri Jul 23 10:18:41 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Jul 2010 10:18:41 +0200
Subject: [Cython] Inline caching for method calls
In-Reply-To: 
References: <4C493272.3000506@behnel.de>
	
Message-ID: <4C495061.1080004@behnel.de>

Robert Bradshaw, 23.07.2010 08:52:
> On Thu, Jul 22, 2010 at 11:10 PM, Stefan Behnel wrote:
>> Stefan Brunthaler started a thread on python-dev about a couple of patches
>> he has written for CPython as part of his PhD. It seems that he has
>> implemented inline caching for CPython.
>>
>> http://comments.gmane.org/gmane.comp.python.devel/115362
>>
>> http://en.wikipedia.org/wiki/Inline_caching
>>
>> Something like this might be interesting for Cython, too. He hasn't
>> published the patches yet, but they might well be worth a look when he does.
>
> Very cool. I have actually thought a lot about this, and I know
> unladen swallow did some stuff in this direction for module/builtin
> lookups. Ideally, there could be some kind of a hook called, or even a
> dirty bit, that gets set when an objects attribute set gets modified,
> so we wouldn't have to re-look things up most of the time.

I just had a funny idea. We could simply embed each call to a Python method 
in a snippet of code that does a bit of caching. It would use a fixed size 
static C array (say, size 4) of structs for each call node:

     {Python type, underlying C function of method}

and add all types to that array that it finds as call target at runtime 
that do not have a __dict__ (most builtin types but (sadly) no modules). 
Then, before doing the CPython call dance, it would just quickly run 
through the array to check if it finds the current type, and on success, 
call the function for that method directly. The normal Python attribute 
lookup and call would then become a fallback that would only be required 
for dynamic Python objects. We could also add types to it that we deem 
unoptimisable (e.g. because the method is not efficiently callable) and set 
their method pointer to NULL, so that we can skip over any further 
optimisation attempts in the future. We always hold the GIL when 
manipulating the arrays, so static caching will work perfectly.

What do you think?

I'm not so sure yet about the ref-counting implications of keeping such a 
static pointer to a type, that may also need some more consideration. A 
borrowed reference will not work if the type is allowed to get garbage 
collected, as the type's address may then end up being reused for another 
type, leaving the method pointer invalid.

Stefan

From robertwb at math.washington.edu  Fri Jul 23 18:14:45 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 23 Jul 2010 09:14:45 -0700
Subject: [Cython] Inline caching for method calls
In-Reply-To: <4C495061.1080004@behnel.de>
References: <4C493272.3000506@behnel.de>
	
	<4C495061.1080004@behnel.de>
Message-ID: 

On Fri, Jul 23, 2010 at 1:18 AM, Stefan Behnel  wrote:
> Robert Bradshaw, 23.07.2010 08:52:
>> On Thu, Jul 22, 2010 at 11:10 PM, Stefan Behnel wrote:
>>> Stefan Brunthaler started a thread on python-dev about a couple of patches
>>> he has written for CPython as part of his PhD. It seems that he has
>>> implemented inline caching for CPython.
>>>
>>> http://comments.gmane.org/gmane.comp.python.devel/115362
>>>
>>> http://en.wikipedia.org/wiki/Inline_caching
>>>
>>> Something like this might be interesting for Cython, too. He hasn't
>>> published the patches yet, but they might well be worth a look when he does.
>>
>> Very cool. I have actually thought a lot about this, and I know
>> unladen swallow did some stuff in this direction for module/builtin
>> lookups. Ideally, there could be some kind of a hook called, or even a
>> dirty bit, that gets set when an objects attribute set gets modified,
>> so we wouldn't have to re-look things up most of the time.
>
> I just had a funny idea. We could simply embed each call to a Python method
> in a snippet of code that does a bit of caching. It would use a fixed size
> static C array (say, size 4) of structs for each call node:
>
> ? ? {Python type, underlying C function of method}
>
> and add all types to that array that it finds as call target at runtime
> that do not have a __dict__ (most builtin types but (sadly) no modules).
> Then, before doing the CPython call dance, it would just quickly run
> through the array to check if it finds the current type, and on success,
> call the function for that method directly. The normal Python attribute
> lookup and call would then become a fallback that would only be required
> for dynamic Python objects. We could also add types to it that we deem
> unoptimisable (e.g. because the method is not efficiently callable) and set
> their method pointer to NULL, so that we can skip over any further
> optimisation attempts in the future. We always hold the GIL when
> manipulating the arrays, so static caching will work perfectly.
>
> What do you think?

There is a nontrivial amount of boilerplate that the CPython
CallObject methods do w.r.t. dispatching to different kinds of
methods, so we may want to, e.g. specialize to methods that have a
single calling convention (that matches the way they are used?)

I think the real savings is in larger class hierarchies, and there we
really need some CPython hook to verify that the dictionaries have not
changed.

> I'm not so sure yet about the ref-counting implications of keeping such a
> static pointer to a type, that may also need some more consideration. A
> borrowed reference will not work if the type is allowed to get garbage
> collected, as the type's address may then end up being reused for another
> type, leaving the method pointer invalid.

That could be an issue, as types can get garbage collected.

- Robert

From dalcinl at gmail.com  Fri Jul 23 18:26:01 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 23 Jul 2010 13:26:01 -0300
Subject: [Cython] Inline caching for method calls
In-Reply-To: 
References: <4C493272.3000506@behnel.de>
	
	<4C495061.1080004@behnel.de>
	
Message-ID: 

On 23 July 2010 13:14, Robert Bradshaw  wrote:
>
> I think the real savings is in larger class hierarchies, and there we
> really need some CPython hook to verify that the dictionaries have not
> changed.
>

Python2.6+ already have a method cache for types. Is this what you are
talking about?


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Fri Jul 23 19:38:58 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 23 Jul 2010 10:38:58 -0700
Subject: [Cython] Inline caching for method calls
In-Reply-To: 
References: <4C493272.3000506@behnel.de>
	
	<4C495061.1080004@behnel.de>
	
	
Message-ID: 

On Fri, Jul 23, 2010 at 9:26 AM, Lisandro Dalcin  wrote:
> On 23 July 2010 13:14, Robert Bradshaw  wrote:
>>
>> I think the real savings is in larger class hierarchies, and there we
>> really need some CPython hook to verify that the dictionaries have not
>> changed.
>>
>
> Python2.6+ already have a method cache for types. Is this what you are
> talking about?

Ah, yes, exactly. I guess I should follow python-dev more closely.

- Robert

From stefan_ml at behnel.de  Fri Jul 23 19:46:41 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 23 Jul 2010 19:46:41 +0200
Subject: [Cython] Inline caching for method calls
In-Reply-To: 
References: <4C493272.3000506@behnel.de>		<4C495061.1080004@behnel.de>		
	
Message-ID: <4C49D581.2010100@behnel.de>

Robert Bradshaw, 23.07.2010 19:38:
> On Fri, Jul 23, 2010 at 9:26 AM, Lisandro Dalcin wrote:
>> On 23 July 2010 13:14, Robert Bradshaw wrote:
>>>
>>> I think the real savings is in larger class hierarchies, and there we
>>> really need some CPython hook to verify that the dictionaries have not
>>> changed.
>>
>> Python2.6+ already have a method cache for types. Is this what you are
>> talking about?
>
> Ah, yes, exactly. I guess I should follow python-dev more closely.

Right, I had forgotten about it, too. Guess it's a dead idea then.

Stefan

From luiji at users.sourceforge.net  Sat Jul 24 05:14:48 2010
From: luiji at users.sourceforge.net (Luiji Maryo)
Date: Fri, 23 Jul 2010 23:14:48 -0400
Subject: [Cython] Optimised Python runtime using Cython (added subject
	to match content)
In-Reply-To: 
References: 
	<4C2AEA6B.5080808@behnel.de> <4C2CCD91.6080101@molden.no>
	
	<2D9406AA-20F9-437B-B7DD-DC6E11BEF34A@gmail.com>
	
Message-ID: 

Firstly, this is a graphics intensive *2-D* game, so no Ogre.

I now see that me and my collegues are complete morons.  We all just
used SDL and completely forgot about OpenGL.  Do'h!

I've brought this to their attention, we all punched ourselves, and we
decided to write the full game in Python, however we plan on
recommending that users start using Psyco and other JIT Pythons.

Anyway, this project has gone to rest.  I may still do some
experimentation later on, but for now I have to focus on game
development.

Thanks for all of the feedback, it really helped!

On Saturday, July 3, 2010, Luiji Maryo  wrote:
> I never said Cython doesn't meet my needs, I said it met my needs so
> much that I wanted to optimized the PSL with it.
>
> On Friday, July 2, 2010, Russell Reagan  wrote:
>> I am curious, in what ways does Cython not meet your needs? I think it
>> would be beneficial to know, in order to identify areas which Cython
>> needs to be improved in order to be more useful in real world projects.
>>
>> I have not done a great deal of work in Cython yet, but my perception
>> is that with Cython, you should be able to write in Python, modify the
>> Python with augmenting .pxd files, and get close to the equivalent
>> speed of C code.
>>
>> So I am curious if:
>>
>> 1. My theoretical view of Cython is flawed. Or...
>>
>> 2. Practice currently has not caught up to theory. Or...
>>
>> 3. Cython is not intended to address your specific situation, and I
>> have misunderstood your intentions and goals.
>>
>> On Jul 2, 2010, at 8:57 AM, Luiji Maryo 
>> wrote:
>>
>>> Responses to Stefan Behnel:
>>>
>>> I do know the goal is ambitious, but less then you probably think.
>>> Let me clarify and say that I am only optimizing modules that will
>>> benefit, such as the mathematical libraries. ?Libraries such as Tk can
>>> be left alone, since they are simply wrappers for the Tk native
>>> library (IIRC).
>>>
>>> As for project background, this has to do with an argument with
>>> colleges of mine. ?We wanted to make a game, and I brought up the
>>> greatness of Python, but they said that it would be too slow for game
>>> development. ?Even though it is sufficiently fast, the game we were
>>> working on was very graphics intensive and when comparing a C version
>>> to a Python version of basic design philosophy C was noticably faster.
>>>
>>> However, I brought up the fact that Python is much quicker to code and
>>> outputs smaller files, and is much more cross-platformable. ?They
>>> argued that development speed could be sacraficed for game speed, and
>>> in the end I suggested to solve the issue by making a faster Python.
>>> They were quickly discouraged, and desided to pause the project for
>>> awhile so that I can get Python closer to C speed while they work on
>>> other games.
>>>
>>> Unladen Swallow seems to have the same project goal as me: to make a
>>> noticably faster Python implementation. ?I will look more into that
>>> project. ?Thanks for the information!
>>>
>>> Responses to Sturla Molden:
>>>
>>> Yes, CPython does seem to avoid very great concepts such as
>>> register-based memory management, garbage collection over reference
>>> counting, and JIT, which is why I decided as part of the optimized
>>> Python to use Pysco, Unladen Swallow, or another optimized Python
>>> interpreter.
>>>
>>> Thank you for your comments (and for adding the title that I
>>> moronically forgot), and I will start doing some optimizations. ?Any
>>> further development on this will be found at http://luiji.github.com/,
>>> but for now you probably will not see much development. ?If anybody is
>>> interested in the project, then you can subscribe to my GitHub account
>>> to a notification when I create a new repository for the optimized
>>> Python Standard Library (PSL as I will call it in further e-mails,
>>> since stdlib sounds too much like the C Stdlib).
>>>
>>> Thanks,
>>> - Luiji Maryo
>>>
>>> On Thu, Jul 1, 2010 at 1:17 PM, Sturla Molden 
>>> wrote:
>>>> Stefan Behnel skrev:
>>>>> Good luck. I don't mean to discourage you, but this is a pretty
>>>>> ambitious goal.
>>>>>
>>>>
>>>> Python's VM is slow, indeed. It uses a stack-based VM instead of
>>>> registers; it is implemented in the worst possible way, as code and
>>>> data
>>>> form a three of hash tables; it has reference counting instead of
>>>> garbage collection (produces high cache traffic); it has no JIT
>>>> compilation. It's doomed to be slow.
>>>>
>>>> Many implementations of Common Lisp and Scheme can compete with--
> - Luiji Maryo (a.k.a. Brain Boy)
> Visit me at http://brainboyblogger.blogspot.com/.
>

-- 
- Luiji Maryo (a.k.a. Brain Boy)
Visit me at http://brainboyblogger.blogspot.com/.

From robertwb at math.washington.edu  Sat Jul 24 12:46:54 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 24 Jul 2010 03:46:54 -0700
Subject: [Cython] Cython 0.13 beta.
Message-ID: 

I've packaged up a beta for 0.13. Try it out, and let us know if there
are any blockers. We are now officially in a feature freeze.

http://cython.org/release/Cython-0.13.beta0.tar.gz
http://cython.org/release/Cython-0.13.beta0.zip

- Robert

From stefan_ml at behnel.de  Sat Jul 24 16:16:15 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 24 Jul 2010 16:16:15 +0200
Subject: [Cython] Python bool/C++ bool : Confusing behaviour
In-Reply-To: 
References: 
	
Message-ID: <4C4AF5AF.1090904@behnel.de>

Robert Bradshaw, 09.07.2010 17:29:
> For the moment, though, I'm wondering why we even bother letting
> people declare the type as bool--it's not a very useful (Python) type
> to statically declare, but can cause a lot of confusion.

We have to take care not to break the 'bool' builtin, though, i.e. code 
like this must continue to work:

     x = bool(obj)
     assert isinstance(x, bool)

Stefan

From robertwb at math.washington.edu  Sat Jul 24 17:49:55 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 24 Jul 2010 08:49:55 -0700
Subject: [Cython] Python bool/C++ bool : Confusing behaviour
In-Reply-To: <4C4AF5AF.1090904@behnel.de>
References: 
	
	<4C4AF5AF.1090904@behnel.de>
Message-ID: 

On Sat, Jul 24, 2010 at 7:16 AM, Stefan Behnel  wrote:
> Robert Bradshaw, 09.07.2010 17:29:
>> For the moment, though, I'm wondering why we even bother letting
>> people declare the type as bool--it's not a very useful (Python) type
>> to statically declare, but can cause a lot of confusion.
>
> We have to take care not to break the 'bool' builtin, though, i.e. code
> like this must continue to work:
>
> ? ? x = bool(obj)
> ? ? assert isinstance(x, bool)

That would work just fine if bool is not a builtin type.

- Robert

From arfrever.fta at gmail.com  Sat Jul 24 21:10:23 2010
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Sat, 24 Jul 2010 21:10:23 +0200
Subject: [Cython] Cython 0.13 beta.
In-Reply-To: 
References: 
Message-ID: <201007242110.24359.Arfrever.FTA@gmail.com>

2 tests fail:

======================================================================
ERROR: compiling (cpp) and running cpp_overload_wrapper
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 427, in run_doctests
    doctest.DocTestSuite(module_name).run(result)
  File "/usr/lib64/python2.6/doctest.py", line 2291, in DocTestSuite
    module = _normalize_module(module)
  File "/usr/lib64/python2.6/doctest.py", line 203, in _normalize_module
    return __import__(module, globals(), locals(), ["*"])
ImportError: /var/tmp/portage/dev-python/cython-0.13_beta0/work/Cython-0.13.beta0/BUILD/wrappers/cpp/cpp_overload_wrapper.so: undefined symbol: _Z10doublefuncddd

======================================================================
ERROR: compiling (cpp) and running cppwrap
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 427, in run_doctests
    doctest.DocTestSuite(module_name).run(result)
  File "/usr/lib64/python2.6/doctest.py", line 2291, in DocTestSuite
    module = _normalize_module(module)
  File "/usr/lib64/python2.6/doctest.py", line 203, in _normalize_module
    return __import__(module, globals(), locals(), ["*"])
ImportError: /var/tmp/portage/dev-python/cython-0.13_beta0/work/Cython-0.13.beta0/BUILD/wrappers/cpp/cppwrap.so: undefined symbol: _Z10doublefuncddd

----------------------------------------------------------------------
Ran 4218 tests in 1054.062s

FAILED (errors=2)

-- 
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100724/ba768f21/attachment.pgp 

From stefan_ml at behnel.de  Sat Jul 24 21:24:45 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 24 Jul 2010 21:24:45 +0200
Subject: [Cython] Cython 0.13 beta.
In-Reply-To: <201007242110.24359.Arfrever.FTA@gmail.com>
References: 
	<201007242110.24359.Arfrever.FTA@gmail.com>
Message-ID: <4C4B3DFD.5070905@behnel.de>

Hi,

thanks for the report.

Arfrever Frehtes Taifersar Arahesis, 24.07.2010 21:10:
 > ERROR: compiling (cpp) and running cpp_overload_wrapper
> ERROR: compiling (cpp) and running cppwrap
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "runtests.py", line 416, in run
>      self.run_doctests(self.module, result)
>    File "runtests.py", line 427, in run_doctests
>      doctest.DocTestSuite(module_name).run(result)
>    File "/usr/lib64/python2.6/doctest.py", line 2291, in DocTestSuite
>      module = _normalize_module(module)
>    File "/usr/lib64/python2.6/doctest.py", line 203, in _normalize_module
>      return __import__(module, globals(), locals(), ["*"])
> ImportError: /var/tmp/portage/dev-python/cython-0.13_beta0/work/Cython-0.13.beta0/BUILD/wrappers/cpp/cppwrap.so: undefined symbol: _Z10doublefuncddd

Both tests pass in Hudson.

What kind of system is this? Gentoo-Linux? Using what C compiler version?


> Ran 4218 tests

That's a huge number. I didn't pay much attention to the size of our test 
suite lately, but I'm happy to see that it's been growing.

Stefan

From robertwb at math.washington.edu  Sun Jul 25 00:38:07 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 24 Jul 2010 15:38:07 -0700
Subject: [Cython] Cython 0.13 beta.
In-Reply-To: <4C4B3DFD.5070905@behnel.de>
References: 
	<201007242110.24359.Arfrever.FTA@gmail.com>
	<4C4B3DFD.5070905@behnel.de>
Message-ID: 

On Sat, Jul 24, 2010 at 12:24 PM, Stefan Behnel  wrote:
> Hi,
>
> thanks for the report.
>
> Arfrever Frehtes Taifersar Arahesis, 24.07.2010 21:10:
> ?> ERROR: compiling (cpp) and running cpp_overload_wrapper
>> ERROR: compiling (cpp) and running cppwrap
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? ?File "runtests.py", line 416, in run
>> ? ? ?self.run_doctests(self.module, result)
>> ? ?File "runtests.py", line 427, in run_doctests
>> ? ? ?doctest.DocTestSuite(module_name).run(result)
>> ? ?File "/usr/lib64/python2.6/doctest.py", line 2291, in DocTestSuite
>> ? ? ?module = _normalize_module(module)
>> ? ?File "/usr/lib64/python2.6/doctest.py", line 203, in _normalize_module
>> ? ? ?return __import__(module, globals(), locals(), ["*"])
>> ImportError: /var/tmp/portage/dev-python/cython-0.13_beta0/work/Cython-0.13.beta0/BUILD/wrappers/cpp/cppwrap.so: undefined symbol: _Z10doublefuncddd
>
> Both tests pass in Hudson.
>
> What kind of system is this? Gentoo-Linux? Using what C compiler version?
>
>
>> Ran 4218 tests
>
> That's a huge number. I didn't pay much attention to the size of our test
> suite lately, but I'm happy to see that it's been growing.
>

Yeah. It's certainly gotten to the point that running them manually
takes quite a while (which is a good thing.)

- Robert

From arfrever.fta at gmail.com  Sun Jul 25 02:53:44 2010
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Sun, 25 Jul 2010 02:53:44 +0200
Subject: [Cython] Cython 0.13 beta.
In-Reply-To: <4C4B3DFD.5070905@behnel.de>
References: 
	<201007242110.24359.Arfrever.FTA@gmail.com>
	<4C4B3DFD.5070905@behnel.de>
Message-ID: <201007250253.45449.Arfrever.FTA@gmail.com>

2010-07-24 21:24:45 Stefan Behnel napisa?(a):
> Hi,
> 
> thanks for the report.
> 
> Arfrever Frehtes Taifersar Arahesis, 24.07.2010 21:10:
>  > ERROR: compiling (cpp) and running cpp_overload_wrapper
> > ERROR: compiling (cpp) and running cppwrap
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >    File "runtests.py", line 416, in run
> >      self.run_doctests(self.module, result)
> >    File "runtests.py", line 427, in run_doctests
> >      doctest.DocTestSuite(module_name).run(result)
> >    File "/usr/lib64/python2.6/doctest.py", line 2291, in DocTestSuite
> >      module = _normalize_module(module)
> >    File "/usr/lib64/python2.6/doctest.py", line 203, in _normalize_module
> >      return __import__(module, globals(), locals(), ["*"])
> > ImportError: /var/tmp/portage/dev-python/cython-0.13_beta0/work/Cython-0.13.beta0/BUILD/wrappers/cpp/cppwrap.so: undefined symbol: _Z10doublefuncddd
> 
> Both tests pass in Hudson.
> 
> What kind of system is this? Gentoo-Linux? Using what C compiler version?

Gentoo Linux, amd64, GCC 4.4.4.

-- 
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100725/c2a0019e/attachment.pgp 

From kwmsmith at gmail.com  Sun Jul 25 05:41:31 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sat, 24 Jul 2010 22:41:31 -0500
Subject: [Cython] [cython-users] Cython 0.13 beta.
In-Reply-To: 
References: 
Message-ID: 

Some cryptic errors here.  I just installed Debian 'lenny' on my box,
so some numpy tests don't run yet.

$ gcc -v
gcc version 4.3.2 (Debian 4.3.2-1.1)

======================================================================
ERROR: compiling (c) and running cimport_cython_T505
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 471, in run_doctests
    (module_name, result_code & 255))
Exception: Tests in module 'cimport_cython_T505' were unexpectedly
killed by signal 11

======================================================================
ERROR: compiling (cpp) and running int_float_builtins_as_casts_T400
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 471, in run_doctests
    (module_name, result_code & 255))
Exception: Tests in module 'int_float_builtins_as_casts_T400' were
unexpectedly killed by signal 11

======================================================================
ERROR: compiling (cpp) and running unicodeencode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 471, in run_doctests
    (module_name, result_code & 255))
Exception: Tests in module 'unicodeencode' were unexpectedly killed by signal 11

======================================================================
ERROR: compiling (cpp) and running unicodemethods
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 471, in run_doctests
    (module_name, result_code & 255))
Exception: Tests in module 'unicodemethods' were unexpectedly killed
by signal 11

======================================================================
ERROR: compiling (cpp) and running cpp_overload_wrapper
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 442, in run_doctests
    tests = doctest.DocTestSuite(module_name)
  File "/usr/lib/python2.5/doctest.py", line 2266, in DocTestSuite
    module = _normalize_module(module)
  File "/usr/lib/python2.5/doctest.py", line 200, in _normalize_module
    return __import__(module, globals(), locals(), ["*"])
ImportError: /home/ksmith/Downloads/Cython-0.13.beta0/BUILD/wrappers/cpp/cpp_overload_wrapper.so:
undefined symbol: _Z10doublefuncddd

======================================================================
ERROR: compiling (cpp) and running cpp_overload_wrapper
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 481, in run_doctests
    (module_name, result_code))
Exception: Tests in module 'cpp_overload_wrapper' exited with status 1

======================================================================
ERROR: compiling (cpp) and running cppwrap
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 442, in run_doctests
    tests = doctest.DocTestSuite(module_name)
  File "/usr/lib/python2.5/doctest.py", line 2266, in DocTestSuite
    module = _normalize_module(module)
  File "/usr/lib/python2.5/doctest.py", line 200, in _normalize_module
    return __import__(module, globals(), locals(), ["*"])
ImportError: /home/ksmith/Downloads/Cython-0.13.beta0/BUILD/wrappers/cpp/cppwrap.so:
undefined symbol: _Z10doublefuncddd

======================================================================
ERROR: compiling (cpp) and running cppwrap
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 416, in run
    self.run_doctests(self.module, result)
  File "runtests.py", line 481, in run_doctests
    (module_name, result_code))
Exception: Tests in module 'cppwrap' exited with status 1

----------------------------------------------------------------------
Ran 4146 tests in 437.094s

FAILED (errors=8)
Following tests excluded because of missing dependencies on your system:
   run.numpy_ValueError_T172
   run.numpy_bufacc_T155
   run.numpy_cimport
   run.numpy_test
   run.pstats_profile_test

From stefan_ml at behnel.de  Sun Jul 25 09:18:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 25 Jul 2010 09:18:06 +0200
Subject: [Cython] [cython-users] Cython 0.13 beta.
In-Reply-To: 
References: 
	
Message-ID: <4C4BE52E.2030801@behnel.de>

Kurt Smith, 25.07.2010 05:41:
> Some cryptic errors here.  I just installed Debian 'lenny' on my box,
> so some numpy tests don't run yet.
>[...]
> Exception: Tests in module 'cimport_cython_T505' were unexpectedly
> killed by signal 11

Could you send in the parts of the test log where the failing tests are 
being run? There may have been GCC error output that is not repeated at the 
end.

Stefan

From stefan_ml at behnel.de  Sun Jul 25 15:36:55 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 25 Jul 2010 15:36:55 +0200
Subject: [Cython] Cython 0.13 beta.
In-Reply-To: 
References: 
Message-ID: <4C4C3DF7.7080604@behnel.de>

Robert Bradshaw, 24.07.2010 12:46:
> I've packaged up a beta for 0.13. Try it out, and let us know if there
> are any blockers. We are now officially in a feature freeze.

Given that this isn't an RC yet - I have a patch that adds some more cases 
to the None check elimination, e.g. for type coercions and C-API calls. 
Does that qualify as a feature?

Stefan

From carl.witty at gmail.com  Tue Jul 27 08:36:01 2010
From: carl.witty at gmail.com (Carl Witty)
Date: Mon, 26 Jul 2010 23:36:01 -0700
Subject: [Cython] Cython performance bug for special functions
Message-ID: 

I just posted a patch on #561 that greatly speeds up the case when you
define a special function in a cdef class and then inherit in a
non-cdef class.  Consider the following code:

cdef class NullAdder:
    def __add__(self, other):
        return self

class NullAdder2(NullAdder):
    pass

Before the patch, I got the following timings:

sage: timeit('na + na', number=10^5, repeat=30)
100000 loops, best of 30: 94.3 ns per loop
sage: na2 = NullAdder2()
sage: timeit('na2 + na2', number=10^5, repeat=30)
100000 loops, best of 30: 261 ns per loop

so being in a subclass had 170ns of overhead.  (The overhead depends
on the method; the overhead for __getattr__ is more like 700ns on the
same computer.)

After the patch, there is no penalty for the subclass:

sage: timeit('na + na', number=10^5, repeat=30)
100000 loops, best of 30: 92 ns per loop
sage: na2 = NullAdder2()
sage: timeit('na2 + na2', number=10^5, repeat=30)
100000 loops, best of 30: 92.9 ns per loop

More details are on the ticket.

I tested the patch with the Cython test suite (against 0.12.1,
0.13.beta0, and current mercurial cython-devel), and by rebuilding
Sage 4.5.1 with the modified version and running its test suite.  All
tests passed.

One not-quite-a-microbenchmark got 20% faster in Sage with this patch
(adding QQbar(0) to itself went from about 26.5 microseconds to about
21 microseconds).

Given the potentially large performance benefit, I think you should
consider this patch (or something like it that's perhaps a little
cleaner) for 0.13.

Carl

From dagss at student.matnat.uio.no  Tue Jul 27 08:55:17 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 27 Jul 2010 08:55:17 +0200
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References: 
Message-ID: <4C4E82D5.1020909@student.matnat.uio.no>

Carl Witty wrote:
> I just posted a patch on #561 that greatly speeds up the case when you
> define a special function in a cdef class and then inherit in a
> non-cdef class.  Consider the following code:
>   
Thanks, this is really cool. (I'm not doing the release so I won't 
comment on that.)

I know that you tested it well, but for the benefit for future testing, 
did you identify a test in the Cython test suite which exercises this 
aspect alone? (It would be good to leave a note of which one the ticket.)

Dag Sverre

From stefan_ml at behnel.de  Tue Jul 27 09:10:17 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Jul 2010 09:10:17 +0200
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: <4C4E82D5.1020909@student.matnat.uio.no>
References: 
	<4C4E82D5.1020909@student.matnat.uio.no>
Message-ID: <4C4E8659.4040904@behnel.de>

Dag Sverre Seljebotn, 27.07.2010 08:55:
> Carl Witty wrote:
>> I just posted a patch on #561 that greatly speeds up the case when you
>> define a special function in a cdef class and then inherit in a
>> non-cdef class.  Consider the following code:
>
> I know that you tested it well, but for the benefit for future testing,
> did you identify a test in the Cython test suite which exercises this
> aspect alone?

+1, I commented on the bug tracker.

Stefan

From robertwb at math.washington.edu  Tue Jul 27 11:07:45 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 27 Jul 2010 02:07:45 -0700
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References: 
Message-ID: 

On Mon, Jul 26, 2010 at 11:36 PM, Carl Witty  wrote:
> I just posted a patch on #561 that greatly speeds up the case when you
> define a special function in a cdef class and then inherit in a
> non-cdef class. ?Consider the following code:
>
> cdef class NullAdder:
> ? ?def __add__(self, other):
> ? ? ? ?return self
>
> class NullAdder2(NullAdder):
> ? ?pass
>
> Before the patch, I got the following timings:
>
> sage: timeit('na + na', number=10^5, repeat=30)
> 100000 loops, best of 30: 94.3 ns per loop
> sage: na2 = NullAdder2()
> sage: timeit('na2 + na2', number=10^5, repeat=30)
> 100000 loops, best of 30: 261 ns per loop
>
> so being in a subclass had 170ns of overhead. ?(The overhead depends
> on the method; the overhead for __getattr__ is more like 700ns on the
> same computer.)
>
> After the patch, there is no penalty for the subclass:
>
> sage: timeit('na + na', number=10^5, repeat=30)
> 100000 loops, best of 30: 92 ns per loop
> sage: na2 = NullAdder2()
> sage: timeit('na2 + na2', number=10^5, repeat=30)
> 100000 loops, best of 30: 92.9 ns per loop

Very, very cool. I've actually been thinking a lot about this myself
(given the recent thread in sage-devel). I think one reason we did our
own wrappers was for introspection/docstrings--does that break? Did
you fix the most egregious __getattr__?

> More details are on the ticket.
>
> I tested the patch with the Cython test suite (against 0.12.1,
> 0.13.beta0, and current mercurial cython-devel), and by rebuilding
> Sage 4.5.1 with the modified version and running its test suite. ?All
> tests passed.
>
> One not-quite-a-microbenchmark got 20% faster in Sage with this patch
> (adding QQbar(0) to itself went from about 26.5 microseconds to about
> 21 microseconds).
>
> Given the potentially large performance benefit, I think you should
> consider this patch (or something like it that's perhaps a little
> cleaner) for 0.13.

It's certainly worth considering!

- Robert

From bob at bobcowdery.plus.com  Tue Jul 27 12:16:37 2010
From: bob at bobcowdery.plus.com (Bob Cowdery)
Date: Tue, 27 Jul 2010 11:16:37 +0100
Subject: [Cython] xml.dom will not import
Message-ID: <4C4EB205.4020900@bobcowdery.plus.com>

 Hi all

I have something strange happening with a plain python module compiled
with Cython.

The import statement:

import xml.dom as dom

is causing the following error:

TypeError: Item in ''from list'' not a string.  (Note these are single
quote marks and the first set are the slanting variety)

This works fine on Windows 7 and one of my Linux boxes where Python and
extensions are compiled as USC2. It gives the above error on Linux boxes
where Python and extensions are compiled as UCS4. I can however import
fine if not compiled so whatever extension xml.dom uses (pyexpat.pyd I
think) it must be compatible with the Python build. Everything else (and
there is a lot of it) compiles and works fine on all boxes. I can
replace just that one module with a .py version and the non-working
Linux boxes then work ok so nothing is fundamentally amiss.

Can anyone shed any light on what's going on here please.

Thanks
Bob

From stefan_ml at behnel.de  Tue Jul 27 12:33:12 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Jul 2010 12:33:12 +0200
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4EB205.4020900@bobcowdery.plus.com>
References: <4C4EB205.4020900@bobcowdery.plus.com>
Message-ID: <4C4EB5E8.7060809@behnel.de>

Bob Cowdery, 27.07.2010 12:16:
> I have something strange happening with a plain python module compiled
> with Cython.

Which version of Cython?

Stefan

From bob at bobcowdery.plus.com  Tue Jul 27 12:39:32 2010
From: bob at bobcowdery.plus.com (Bob Cowdery)
Date: Tue, 27 Jul 2010 11:39:32 +0100
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4EB5E8.7060809@behnel.de>
References: <4C4EB205.4020900@bobcowdery.plus.com> <4C4EB5E8.7060809@behnel.de>
Message-ID: <4C4EB764.1050004@bobcowdery.plus.com>

 Hi Stefan

I'm building for the failing machine on Ubuntu 9.04 and it says the
Cython version is 0.10.3-1build1 installed from the Ubuntu repository.

Bob

On 27/07/2010 11:33, Stefan Behnel wrote:
> Bob Cowdery, 27.07.2010 12:16:
>> I have something strange happening with a plain python module compiled
>> with Cython.
> Which version of Cython?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Tue Jul 27 12:46:18 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Jul 2010 12:46:18 +0200
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4EB764.1050004@bobcowdery.plus.com>
References: <4C4EB205.4020900@bobcowdery.plus.com> <4C4EB5E8.7060809@behnel.de>
	<4C4EB764.1050004@bobcowdery.plus.com>
Message-ID: <4C4EB8FA.9050603@behnel.de>

Bob Cowdery, 27.07.2010 12:39:
> On 27/07/2010 11:33, Stefan Behnel wrote:
>> Bob Cowdery, 27.07.2010 12:16:
>>> I have something strange happening with a plain python module compiled
>>> with Cython.
>> Which version of Cython?
>
> I'm building for the failing machine on Ubuntu 9.04 and it says the
> Cython version is 0.10.3-1build1 installed from the Ubuntu repository.

A lot has happened since then, so I would doubt that this is still a 
problem in the latest Cython release (0.12.1) and particularly not in the 
upcoming Cython 0.13.

Stefan

From stefan_ml at behnel.de  Tue Jul 27 12:55:52 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Jul 2010 12:55:52 +0200
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References: 
	
Message-ID: <4C4EBB38.2040505@behnel.de>

Robert Bradshaw, 27.07.2010 11:07:
> On Mon, Jul 26, 2010 at 11:36 PM, Carl Witty wrote:
>> Given the potentially large performance benefit, I think you should
>> consider this patch (or something like it that's perhaps a little
>> cleaner) for 0.13.
>
> It's certainly worth considering!

Given that this is a change that may potentially break stuff and we don't 
even have tests for it, I think we should not let this delay the release of 
0.13 any further than we already did anyway. Most people don't test 
development releases, so I expect a lot of bug reports after the release of 
0.13 that will make 0.13.1 (and maybe even 0.13.2) a close follow-up anyway.

However, if someone writes suitable tests for all affected special methods 
in a timely fashion, I'd be +0 on inclusion, just for the same reason of 
getting a 0.13.1 out soon anyway, in which we may still rip it back out.

Stefan

From bob at bobcowdery.plus.com  Tue Jul 27 13:08:52 2010
From: bob at bobcowdery.plus.com (Bob Cowdery)
Date: Tue, 27 Jul 2010 12:08:52 +0100
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4EB8FA.9050603@behnel.de>
References: <4C4EB205.4020900@bobcowdery.plus.com> <4C4EB5E8.7060809@behnel.de>
	<4C4EB764.1050004@bobcowdery.plus.com> <4C4EB8FA.9050603@behnel.de>
Message-ID: <4C4EBE44.6090809@bobcowdery.plus.com>

 Ok, thanks. I will install a newer version and let you know if it works.

Bob

On 27/07/2010 11:46, Stefan Behnel wrote:
> Bob Cowdery, 27.07.2010 12:39:
>> On 27/07/2010 11:33, Stefan Behnel wrote:
>>> Bob Cowdery, 27.07.2010 12:16:
>>>> I have something strange happening with a plain python module compiled
>>>> with Cython.
>>> Which version of Cython?
>>
>> I'm building for the failing machine on Ubuntu 9.04 and it says the
>> Cython version is 0.10.3-1build1 installed from the Ubuntu repository.
>
> A lot has happened since then, so I would doubt that this is still a
> problem in the latest Cython release (0.12.1) and particularly not in
> the upcoming Cython 0.13.
>
> Stefan


From bob at bobcowdery.plus.com  Tue Jul 27 13:48:12 2010
From: bob at bobcowdery.plus.com (Bob Cowdery)
Date: Tue, 27 Jul 2010 12:48:12 +0100
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4EB8FA.9050603@behnel.de>
References: <4C4EB205.4020900@bobcowdery.plus.com> <4C4EB5E8.7060809@behnel.de>
	<4C4EB764.1050004@bobcowdery.plus.com> <4C4EB8FA.9050603@behnel.de>
Message-ID: <4C4EC77C.9000901@bobcowdery.plus.com>

 I built and installed 0.12.1. First attempt it was still wrong then I
realised it was still using 0.10.3. I changed the path and if I do
'cython -V' it reports 0.12.3 but it still doesn't work. How can I be
sure that Python is pulling in the correct version when it builds?

Bob

On 27/07/2010 11:46, Stefan Behnel wrote:
> Bob Cowdery, 27.07.2010 12:39:
>> On 27/07/2010 11:33, Stefan Behnel wrote:
>>> Bob Cowdery, 27.07.2010 12:16:
>>>> I have something strange happening with a plain python module compiled
>>>> with Cython.
>>> Which version of Cython?
>>
>> I'm building for the failing machine on Ubuntu 9.04 and it says the
>> Cython version is 0.10.3-1build1 installed from the Ubuntu repository.
>
> A lot has happened since then, so I would doubt that this is still a
> problem in the latest Cython release (0.12.1) and particularly not in
> the upcoming Cython 0.13.
>
> Stefan


From stefan_ml at behnel.de  Tue Jul 27 14:12:27 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Jul 2010 14:12:27 +0200
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4EB205.4020900@bobcowdery.plus.com>
References: <4C4EB205.4020900@bobcowdery.plus.com>
Message-ID: <4C4ECD2B.6020309@behnel.de>

Bob Cowdery, 27.07.2010 12:16:
> I have something strange happening with a plain python module compiled
> with Cython.
>
> The import statement:
>
> import xml.dom as dom
>
> is causing the following error:
>
> TypeError: Item in ''from list'' not a string.  (Note these are single
> quote marks and the first set are the slanting variety)

This works for me in a wide build of Python 2.5, 2.7 and 3.2 as well as a 
narrow build of Python 2.6, using the latest cython-devel branch (i.e. 
Cython 0.13). We also have a similar test case in our regression test suite.

http://hg.cython.org/cython-devel/archive/tip.tar.gz

Please try that version and provide the Python version that you are using 
and the exact error output, including traceback.

Stefan

From stefan_ml at behnel.de  Tue Jul 27 14:15:38 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Jul 2010 14:15:38 +0200
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4ECD2B.6020309@behnel.de>
References: <4C4EB205.4020900@bobcowdery.plus.com> <4C4ECD2B.6020309@behnel.de>
Message-ID: <4C4ECDEA.8040600@behnel.de>

Also note that this thread would be better suited for the cython-users 
mailing list.

Stefan

From bob at bobcowdery.plus.com  Tue Jul 27 16:21:03 2010
From: bob at bobcowdery.plus.com (Bob Cowdery)
Date: Tue, 27 Jul 2010 15:21:03 +0100
Subject: [Cython] xml.dom will not import
In-Reply-To: <4C4ECD2B.6020309@behnel.de>
References: <4C4EB205.4020900@bobcowdery.plus.com> <4C4ECD2B.6020309@behnel.de>
Message-ID: <4C4EEB4F.5010201@bobcowdery.plus.com>

 Stefan

Thanks. I have it working now. It was my fault as it was still running 
0.10 as I was spawning Python to do the build and it adds all its stuff
in front of what I was setting. I just explicitly added to the beginning
of sys.path in the setup file if its a Linux build and that seems to do
the trick. Probably a much better way to do that.

Apologies for posting to the wrong list. I didn't look carefully enough
at the subscribe page and just scrolled to the bottom.

Regards
Bob

On 27/07/2010 13:12, Stefan Behnel wrote:
> Bob Cowdery, 27.07.2010 12:16:
>> I have something strange happening with a plain python module compiled
>> with Cython.
>>
>> The import statement:
>>
>> import xml.dom as dom
>>
>> is causing the following error:
>>
>> TypeError: Item in ''from list'' not a string.  (Note these are single
>> quote marks and the first set are the slanting variety)
>
> This works for me in a wide build of Python 2.5, 2.7 and 3.2 as well
> as a narrow build of Python 2.6, using the latest cython-devel branch
> (i.e. Cython 0.13). We also have a similar test case in our regression
> test suite.
>
> http://hg.cython.org/cython-devel/archive/tip.tar.gz
>
> Please try that version and provide the Python version that you are
> using and the exact error output, including traceback.
>
> Stefan


From carl.witty at gmail.com  Tue Jul 27 21:33:04 2010
From: carl.witty at gmail.com (Carl Witty)
Date: Tue, 27 Jul 2010 12:33:04 -0700
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References:  
	
Message-ID: 

On Tue, Jul 27, 2010 at 2:07 AM, Robert Bradshaw
 wrote:
> Very, very cool. I've actually been thinking a lot about this myself
> (given the recent thread in sage-devel). I think one reason we did our
> own wrappers was for introspection/docstrings--does that break? Did
> you fix the most egregious __getattr__?

Yes, this does break docstrings on these functions.  I think this
could be fixed; for instance, by creating our own statically-declared
wrapperbase/slotdef structs that have the right docstring, and
creating our own wrapper objects using these structs.  I could try to
work on this, but probably not soon enough for 0.13.

Yes, this fixes the performance regression William found when defining
__getattr__.

Carl

From carl.witty at gmail.com  Tue Jul 27 21:40:25 2010
From: carl.witty at gmail.com (Carl Witty)
Date: Tue, 27 Jul 2010 12:40:25 -0700
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: <4C4EBB38.2040505@behnel.de>
References:  
	 
	<4C4EBB38.2040505@behnel.de>
Message-ID: 

On Tue, Jul 27, 2010 at 3:55 AM, Stefan Behnel  wrote:
> Given that this is a change that may potentially break stuff and we don't
> even have tests for it, I think we should not let this delay the release of
> 0.13 any further than we already did anyway. Most people don't test
> development releases, so I expect a lot of bug reports after the release of
> 0.13 that will make 0.13.1 (and maybe even 0.13.2) a close follow-up anyway.
>
> However, if someone writes suitable tests for all affected special methods
> in a timely fashion, I'd be +0 on inclusion, just for the same reason of
> getting a 0.13.1 out soon anyway, in which we may still rip it back out.

What would you consider suitable tests?  All I can think of is to
write Cython classes that define all the special methods, then write
Python code that extracts each special method as a bound method and
then calls it.  Can you think of more tests?

I'll try to write these tests tonight.  I'm sorry I didn't include
tests in the patch; it was getting late and I wanted to post what I
had before I went to bed.

What about the two known behavioral changes (__getattr__ turns into
__getattribute__ (mentioned on the ticket), and docstrings are lost
(as Robert suspected and I just verified)).  Are these enough that you
don't think the patch should get in to 0.13?  (In which case I would
start working on the next version of the patch, instead of writing
tests for this version.)

Carl

From robertwb at math.washington.edu  Tue Jul 27 21:43:36 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 27 Jul 2010 12:43:36 -0700
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References: 
	
	
Message-ID: 

On Tue, Jul 27, 2010 at 12:33 PM, Carl Witty  wrote:
> On Tue, Jul 27, 2010 at 2:07 AM, Robert Bradshaw
>  wrote:
>> Very, very cool. I've actually been thinking a lot about this myself
>> (given the recent thread in sage-devel). I think one reason we did our
>> own wrappers was for introspection/docstrings--does that break? Did
>> you fix the most egregious __getattr__?
>
> Yes, this does break docstrings on these functions. ?I think this
> could be fixed; for instance, by creating our own statically-declared
> wrapperbase/slotdef structs that have the right docstring, and
> creating our own wrapper objects using these structs. ?I could try to
> work on this, but probably not soon enough for 0.13.

Well, for special methods, I consider this kind of performance
degradation to be a more important issue than introspection. It's
really amazing no one has caught this before.

> Yes, this fixes the performance regression William found when defining
> __getattr__.

Excellent!

- Robert

From robertwb at math.washington.edu  Tue Jul 27 21:47:33 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 27 Jul 2010 12:47:33 -0700
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References: 
	
	<4C4EBB38.2040505@behnel.de>
	
Message-ID: 

On Tue, Jul 27, 2010 at 12:40 PM, Carl Witty  wrote:
> On Tue, Jul 27, 2010 at 3:55 AM, Stefan Behnel  wrote:
>> Given that this is a change that may potentially break stuff and we don't
>> even have tests for it, I think we should not let this delay the release of
>> 0.13 any further than we already did anyway. Most people don't test
>> development releases, so I expect a lot of bug reports after the release of
>> 0.13 that will make 0.13.1 (and maybe even 0.13.2) a close follow-up anyway.
>>
>> However, if someone writes suitable tests for all affected special methods
>> in a timely fashion, I'd be +0 on inclusion, just for the same reason of
>> getting a 0.13.1 out soon anyway, in which we may still rip it back out.
>
> What would you consider suitable tests? ?All I can think of is to
> write Cython classes that define all the special methods, then write
> Python code that extracts each special method as a bound method and
> then calls it. ?Can you think of more tests?

That would be sufficient, and a very good test to have.

> I'll try to write these tests tonight. ?I'm sorry I didn't include
> tests in the patch; it was getting late and I wanted to post what I
> had before I went to bed.

Totally understandable.

> What about the two known behavioral changes (__getattr__ turns into
> __getattribute__ (mentioned on the ticket), and docstrings are lost
> (as Robert suspected and I just verified)). ?Are these enough that you
> don't think the patch should get in to 0.13? ?(In which case I would
> start working on the next version of the patch, instead of writing
> tests for this version.)

I'm fine with the above two changes. Go ahead and write another patch
that builds on the one you posted.

- Robert

From stefan_ml at behnel.de  Tue Jul 27 21:50:19 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 27 Jul 2010 21:50:19 +0200
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References: 
	
	<4C4EBB38.2040505@behnel.de>
	
Message-ID: <4C4F387B.7040101@behnel.de>

Carl Witty, 27.07.2010 21:40:
> On Tue, Jul 27, 2010 at 3:55 AM, Stefan Behnel  wrote:
>> Given that this is a change that may potentially break stuff and we don't
>> even have tests for it, I think we should not let this delay the release of
>> 0.13 any further than we already did anyway. Most people don't test
>> development releases, so I expect a lot of bug reports after the release of
>> 0.13 that will make 0.13.1 (and maybe even 0.13.2) a close follow-up anyway.
>>
>> However, if someone writes suitable tests for all affected special methods
>> in a timely fashion, I'd be +0 on inclusion, just for the same reason of
>> getting a 0.13.1 out soon anyway, in which we may still rip it back out.
>
> What would you consider suitable tests?  All I can think of is to
> write Cython classes that define all the special methods, then write
> Python code that extracts each special method as a bound method and
> then calls it.  Can you think of more tests?
>
> I'll try to write these tests tonight.  I'm sorry I didn't include
> tests in the patch; it was getting late and I wanted to post what I
> had before I went to bed.

That's fine, we're just too late in the release cycle (IMHO) to include 
potentially unsafe patches, especially when they lack dedicated tests.


> What about the two known behavioral changes (__getattr__ turns into
> __getattribute__ (mentioned on the ticket), and docstrings are lost
> (as Robert suspected and I just verified)).  Are these enough that you
> don't think the patch should get in to 0.13?

I'd say so. It breaks Python compatibility only for the sake of performance.


> (In which case I would
> start working on the next version of the patch, instead of writing
> tests for this version.)

Please do both. ;)

Stefan

From robertwb at math.washington.edu  Tue Jul 27 21:58:44 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 27 Jul 2010 12:58:44 -0700
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: <4C4F387B.7040101@behnel.de>
References: 
	
	<4C4EBB38.2040505@behnel.de>
	
	<4C4F387B.7040101@behnel.de>
Message-ID: 

On Tue, Jul 27, 2010 at 12:50 PM, Stefan Behnel  wrote:
> Carl Witty, 27.07.2010 21:40:
>> On Tue, Jul 27, 2010 at 3:55 AM, Stefan Behnel ?wrote:
>>> Given that this is a change that may potentially break stuff and we don't
>>> even have tests for it, I think we should not let this delay the release of
>>> 0.13 any further than we already did anyway. Most people don't test
>>> development releases, so I expect a lot of bug reports after the release of
>>> 0.13 that will make 0.13.1 (and maybe even 0.13.2) a close follow-up anyway.
>>>
>>> However, if someone writes suitable tests for all affected special methods
>>> in a timely fashion, I'd be +0 on inclusion, just for the same reason of
>>> getting a 0.13.1 out soon anyway, in which we may still rip it back out.
>>
>> What would you consider suitable tests? ?All I can think of is to
>> write Cython classes that define all the special methods, then write
>> Python code that extracts each special method as a bound method and
>> then calls it. ?Can you think of more tests?
>>
>> I'll try to write these tests tonight. ?I'm sorry I didn't include
>> tests in the patch; it was getting late and I wanted to post what I
>> had before I went to bed.
>
> That's fine, we're just too late in the release cycle (IMHO) to include
> potentially unsafe patches, especially when they lack dedicated tests.
>
>
>> What about the two known behavioral changes (__getattr__ turns into
>> __getattribute__ (mentioned on the ticket), and docstrings are lost
>> (as Robert suspected and I just verified)). ?Are these enough that you
>> don't think the patch should get in to 0.13?
>
> I'd say so. It breaks Python compatibility only for the sake of performance.

On that note, cdef classes are already non-Python compatible in
several ways, so I don't think this is as issue as potentially
breaking pure Python code (though of course I'd like them to be more
like Python classes). If we ever auto-cdef classes, we'll have to make
sure to preserve behavior like this.

- Robert

From carl.witty at gmail.com  Wed Jul 28 03:11:24 2010
From: carl.witty at gmail.com (Carl Witty)
Date: Tue, 27 Jul 2010 18:11:24 -0700
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References:  
	 
	<4C4EBB38.2040505@behnel.de>
	 
	
Message-ID: 

On Tue, Jul 27, 2010 at 12:47 PM, Robert Bradshaw
 wrote:
>> What would you consider suitable tests? ?All I can think of is to
>> write Cython classes that define all the special methods, then write
>> Python code that extracts each special method as a bound method and
>> then calls it. ?Can you think of more tests?
>
> That would be sufficient, and a very good test to have.

The test patch is up at #561.

I tested it under Python 2.5 and 2.6.

By the way, how do you test under Python 3?  I tried "make test3"
(after changing python3.0 to python3.1 in the Makefile) but got this
error:

python3.1 runtests.py -vv --no-cython
Running tests without Cython.
Python 3.1.2 (release31-maint, Jul  8 2010, 09:18:08)
[GCC 4.4.4]

Traceback (most recent call last):
  File "runtests.py", line 856, in 
    pyxbuild_dir=os.path.join(WORKDIR, "support"))
  File "/home/cwitty/hg-cython/cython-devel/pyximport/pyxbuild.py",
line 85, in pyx_to_dll
    dist.run_commands()
  File "/usr/lib/python3.1/distutils/dist.py", line 919, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.1/distutils/dist.py", line 938, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.1/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/home/cwitty/hg-cython/cython-devel/Cython/Distutils/build_ext.py",
line 80, in build_extensions
    ext.sources = self.cython_sources(ext.sources, ext)
  File "/home/cwitty/hg-cython/cython-devel/Cython/Distutils/build_ext.py",
line 91, in cython_sources
    from Cython.Compiler.Main \
  File "/home/cwitty/hg-cython/cython-devel/Cython/Compiler/Main.py", line 35
    print t.dump()
          ^
SyntaxError: invalid syntax

Carl

From stefan_ml at behnel.de  Wed Jul 28 07:23:22 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 28 Jul 2010 07:23:22 +0200
Subject: [Cython] Cython performance bug for special functions
In-Reply-To: 
References: 
	
	<4C4EBB38.2040505@behnel.de>	
	
	
Message-ID: <4C4FBECA.30703@behnel.de>

Carl Witty, 28.07.2010 03:11:
> On Tue, Jul 27, 2010 at 12:47 PM, Robert Bradshaw wrote:
>>> What would you consider suitable tests?  All I can think of is to
>>> write Cython classes that define all the special methods, then write
>>> Python code that extracts each special method as a bound method and
>>> then calls it.  Can you think of more tests?
>>
>> That would be sufficient, and a very good test to have.
>
> The test patch is up at #561.
>
> I tested it under Python 2.5 and 2.6.

Thanks. When we apply it, Hudson will test the others.


> By the way, how do you test under Python 3?  I tried "make test3"
> (after changing python3.0 to python3.1 in the Makefile) but got this
> error:
>
> python3.1 runtests.py -vv --no-cython

Ah, yes, looks like the make target is broken. Remove the "--no-cython" 
switch, it'll just work out of the box.

Stefan

From stefan_ml at behnel.de  Wed Jul 28 11:06:21 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 28 Jul 2010 11:06:21 +0200
Subject: [Cython] sage-tests in Hudson
Message-ID: <4C4FF30D.5010806@behnel.de>

Hi,

the sage-tests job keeps failing on Hudson, but the failing tests basically 
change on each build. I don't know how the sage sources are updated on the 
Hudson machine - would there be a way to let Hudson check them out via 
Mercurial (or git or whatever Sage uses) so that it can present the list of 
changes since the last build? That would make it easier to see if the newly 
failing and back-to-working tests have related changes in Sage itself, or 
if the failure is more likely due to a change in Cython.

Stefan

From robertwb at math.washington.edu  Wed Jul 28 18:17:48 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 28 Jul 2010 09:17:48 -0700
Subject: [Cython] sage-tests in Hudson
In-Reply-To: <4C4FF30D.5010806@behnel.de>
References: <4C4FF30D.5010806@behnel.de>
Message-ID: 

On Wed, Jul 28, 2010 at 2:06 AM, Stefan Behnel  wrote:
> Hi,
>
> the sage-tests job keeps failing on Hudson, but the failing tests basically
> change on each build. I don't know how the sage sources are updated on the
> Hudson machine - would there be a way to let Hudson check them out via
> Mercurial (or git or whatever Sage uses) so that it can present the list of
> changes since the last build? That would make it easier to see if the newly
> failing and back-to-working tests have related changes in Sage itself, or
> if the failure is more likely due to a change in Cython.

Sage hasn't changed (other than the patches to get it to build) since
I installed it into hudson. I think this is the same heisenbug we saw
in our own test suite :(. We need to run them under gdb or something
to figure out where to look.

- Robert

From chemejosh at gmail.com  Wed Jul 28 19:03:19 2010
From: chemejosh at gmail.com (Josh Allen)
Date: Wed, 28 Jul 2010 13:03:19 -0400
Subject: [Cython] Package compiling in pure Python mode
Message-ID: 

Hello again,

I'm working on a Python/Cython project with a large number of modules,
several of which I have Cythonized. I have collected all of the modules into
a single package. Attempting to compile these modules has revealed a couple
of potential issues, which I will outline below for a small test case.

First, the directory layout for the test case, which contains two modules
"foo" and "bar" in a package "bug":

/bug/__init__.py
/bug/bar.pxd
/bug/bar.py
/bug/foo.pxd
/bug/foo.py
/setup.py
/test.py

Next, the contents of the files:

# /bug/foo.py:
class Foo:
    def hello(self):
        print 'Hello from foo.Foo.hello()'

# /bug/foo.pxd:
cdef class Foo:
    cpdef hello(self)

# /bug/bar.pxd:
from foo cimport Foo
cdef class Bar:
    cdef public Foo foo
    cpdef hello(self)

# /bug/bar.py:
import cython
from foo import Foo
class Bar:
    def hello(self):
        cython.declare(f=Foo)
        f = Foo()
        print 'Hello from bar.Bar.hello()'

# setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
    cmdclass={'build_ext': build_ext},
    ext_modules=[Extension('bug.foo', ['bug/foo.py']), Extension('bug.bar',
['bug/bar.py'])]
)

# test.py
from bug.foo import Foo
from bug.bar import Bar
f = Foo()
f.hello() # Should print 'Hello from Foo.hello()'
b = Bar()
b.hello() # Should print 'Hello from Bar.hello()'

If I try to compile as is -- by running "python setup.py build_ext
--inplace" from the top-level directory --, Cython fails at the line "from
foo import Foo" in bar.py with the error "Assignment to non-lvalue 'Foo'".
If I remove this line, then everything compiles and runs properly. However,
I need this line to exist in order to run the file in pure Python mode.

I also tried moving the setup.py file within the bug package, adjusting its
content accordingly, and running from within the bug folder. (This is
probably not a good location for the setup.py file, but I was trying to see
if anything would work.) This also causes Cython to fail, but this time at
the line "cython.declare(f=Foo)" in bar.py with the error "Unknown type".
However, running cython from the console on bar.py while within the bug
directory does produce a valid C file.

Both of these cases are for Python 2.6.5 and Cython 0.12.1 on Ubuntu 10.04.
I also attempted to try it for the Cython 0.13 beta, but the problem seems
to persist. (Incidentally, is running "cython -V" with the 0.13 beta
supposed to print "Cython version 0.12.1"?)

Let me know if there's any other info you need.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100728/a97e2e9a/attachment.htm 

From robertwb at math.washington.edu  Wed Jul 28 19:55:13 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 28 Jul 2010 10:55:13 -0700
Subject: [Cython] Package compiling in pure Python mode
In-Reply-To: 
References: 
Message-ID: 

On Wed, Jul 28, 2010 at 10:03 AM, Josh Allen  wrote:
> Hello again,
>
> I'm working on a Python/Cython project with a large number of modules,
> several of which I have Cythonized. I have collected all of the modules into
> a single package. Attempting to compile these modules has revealed a couple
> of potential issues, which I will outline below for a small test case.
>
> First, the directory layout for the test case, which contains two modules
> "foo" and "bar" in a package "bug":
>
> /bug/__init__.py
> /bug/bar.pxd
> /bug/bar.py
> /bug/foo.pxd
> /bug/foo.py
> /setup.py
> /test.py
>
> Next, the contents of the files:
>
> # /bug/foo.py:
> class Foo:
> ??? def hello(self):
> ??????? print 'Hello from foo.Foo.hello()'
>
> # /bug/foo.pxd:
> cdef class Foo:
> ??? cpdef hello(self)
>
> # /bug/bar.pxd:
> from foo cimport Foo
> cdef class Bar:
> ??? cdef public Foo foo
> ??? cpdef hello(self)
>
> # /bug/bar.py:
> import cython
> from foo import Foo
> class Bar:
> ??? def hello(self):
> ??????? cython.declare(f=Foo)
> ??????? f = Foo()
> ??????? print 'Hello from bar.Bar.hello()'
>
> # setup.py
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
> setup(
> ??? cmdclass={'build_ext': build_ext},
> ??? ext_modules=[Extension('bug.foo', ['bug/foo.py']), Extension('bug.bar',
> ['bug/bar.py'])]
> )
>
> # test.py
> from bug.foo import Foo
> from bug.bar import Bar
> f = Foo()
> f.hello() # Should print 'Hello from Foo.hello()'
> b = Bar()
> b.hello() # Should print 'Hello from Bar.hello()'
>
> If I try to compile as is -- by running "python setup.py build_ext
> --inplace" from the top-level directory --, Cython fails at the line "from
> foo import Foo" in bar.py with the error "Assignment to non-lvalue 'Foo'".
> If I remove this line, then everything compiles and runs properly. However,
> I need this line to exist in order to run the file in pure Python mode.
>
> I also tried moving the setup.py file within the bug package, adjusting its
> content accordingly, and running from within the bug folder. (This is
> probably not a good location for the setup.py file, but I was trying to see
> if anything would work.) This also causes Cython to fail, but this time at
> the line "cython.declare(f=Foo)" in bar.py with the error "Unknown type".
> However, running cython from the console on bar.py while within the bug
> directory does produce a valid C file.
>
> Both of these cases are for Python 2.6.5 and Cython 0.12.1 on Ubuntu 10.04.
> I also attempted to try it for the Cython 0.13 beta, but the problem seems
> to persist. (Incidentally, is running "cython -V" with the 0.13 beta
> supposed to print "Cython version 0.12.1"?)

Hmm... no. Sounds like you're still picking up the old one.

> Let me know if there's any other info you need.

To help us diagnose the problem, does it persist if you change the
extension to .pyx?

- Robert

From stefan_ml at behnel.de  Wed Jul 28 21:50:16 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 28 Jul 2010 21:50:16 +0200
Subject: [Cython] sage-tests in Hudson
In-Reply-To: 
References: <4C4FF30D.5010806@behnel.de>
	
Message-ID: <4C5089F8.6000702@behnel.de>

Robert Bradshaw, 28.07.2010 18:17:
> Sage hasn't changed (other than the patches to get it to build) since
> I installed it into hudson.

Ok, then it clearly smells like a Heisenbug.


> I think this is the same heisenbug we saw
> in our own test suite :(. We need to run them under gdb or something
> to figure out where to look.

Yes, gdb or maybe valgrind. I don't think there's a way to get any closer 
without that.

Stefan

From david.n.mashburn at gmail.com  Thu Jul 29 02:46:23 2010
From: david.n.mashburn at gmail.com (David Mashburn)
Date: Wed, 28 Jul 2010 19:46:23 -0500
Subject: [Cython] Unrecognized Character issue for Windows 7
Message-ID: <4C50CF5F.7030004@gmail.com>

I've seen the bug about the unrecognized characters on Windows, but I
had another, seemingly related issue...

I'm currently using cython v. 0.12.1 on Windows 7, and I can't get it to
recognize ANY newlines...
Not '\n'
Not '\r'
Not '\r\n'
Files without newlines run fine, but with any type of newline, the
unrecognized character error raises...

Did I mention this only happens on Windows 7?
I tried the exact same thing on XP and it runs without issue... and I am
pretty sure I had no trouble on Vista either.

This was the command I invoked cython with in both cases:
C:\Python26\python.exe "C:\Python26\Scripts\cython-script.py"
".pyx" -o ".c"

Any help would be appreciated!  I need Windows 7 support for some of the
people in my group.

Is the problem already fixed in the hg version of cython?

Thanks for your help!
-David


From robertwb at math.washington.edu  Thu Jul 29 02:51:26 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 28 Jul 2010 17:51:26 -0700
Subject: [Cython] Unrecognized Character issue for Windows 7
In-Reply-To: <4C50CF5F.7030004@gmail.com>
References: <4C50CF5F.7030004@gmail.com>
Message-ID: 

On Wed, Jul 28, 2010 at 5:46 PM, David Mashburn
 wrote:
> I've seen the bug about the unrecognized characters on Windows, but I
> had another, seemingly related issue...
>
> I'm currently using cython v. 0.12.1 on Windows 7, and I can't get it to
> recognize ANY newlines...
> Not '\n'
> Not '\r'
> Not '\r\n'
> Files without newlines run fine, but with any type of newline, the
> unrecognized character error raises...
>
> Did I mention this only happens on Windows 7?
> I tried the exact same thing on XP and it runs without issue... and I am
> pretty sure I had no trouble on Vista either.
>
> This was the command I invoked cython with in both cases:
> C:\Python26\python.exe "C:\Python26\Scripts\cython-script.py"
> ".pyx" -o ".c"
>
> Any help would be appreciated! ?I need Windows 7 support for some of the
> people in my group.
>
> Is the problem already fixed in the hg version of cython?

It might be--do you want to try it out? You can download the latest
beta from http://cython.org/release/Cython-0.13.beta0.zip

- Robert

From dagss at student.matnat.uio.no  Thu Jul 29 10:09:20 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 29 Jul 2010 10:09:20 +0200
Subject: [Cython] template types
In-Reply-To: 
References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com>	<4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com>		<49FFEFBE.6010800@behnel.de>	<03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no>	<4A00924D.9070202@behnel.de>	<4A009A97.50608@student.matnat.uio.no>		<4A00AD43.1050709@behnel.de>	<4A00AF71.7010103@behnel.de>		<4A011CFE.7070006@behnel.de>	<3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no>	<8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu>	<4A014AEF.7030901@student.matnat.uio.no>
	
Message-ID: <4C513730.5040503@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 6, 2009, at 1:31 AM, Dag Sverre Seljebotn wrote:
>
>   
>> BTW will this discussion also be used for C++ templates, so that the
>> consensus seem to be
>>
>> some_cpp_class[int]
>>
>> for template instantiation?
>>     
>
> I think so. Speak now or forever hold your peace... :)
>   
Following up this *really* old thread, I just wanted to note that I just 
discovered that IronPython apparently uses the [] syntax as well to 
instantiate C# generics (=templates). So seems like we made a good 
choice for that reason as well.

Dag Sverre

From david.n.mashburn at gmail.com  Thu Jul 29 19:01:29 2010
From: david.n.mashburn at gmail.com (David Mashburn)
Date: Thu, 29 Jul 2010 12:01:29 -0500
Subject: [Cython] Unrecognized Character issue for Windows 7
Message-ID: <4C51B3E9.6070405@gmail.com>

Thanks Robert!
Upgrading did it!

I did have to change my run command to:

C:\Python26\python.exe "C:\Python26\Lib\site-packages\cython.py" ".pyx" -o ".c"

rather than

C:\Python26\python.exe "C:\Python26\Scripts\cython-script.py" ".pyx" -o ".c"
or
C:\Python26\python.exe "C:\Python26\Scripts\cython.py" ".pyx" -o ".c"

The latter seemed to somehow still be connected with 0.12 after upgrade.

Thanks,
-David


On Wed, Jul 28, 2010 at 5:46 PM, David Mashburn
> wrote:
>/ I've seen the bug about the unrecognized characters on Windows, but I
/>/ had another, seemingly related issue...
/>/
/>/ I'm currently using cython v. 0.12.1 on Windows 7, and I can't get it to
/>/ recognize ANY newlines...
/>/ Not '\n'
/>/ Not '\r'
/>/ Not '\r\n'
/>/ Files without newlines run fine, but with any type of newline, the
/>/ unrecognized character error raises...
/>/
/>/ Did I mention this only happens on Windows 7?
/>/ I tried the exact same thing on XP and it runs without issue... and I am
/>/ pretty sure I had no trouble on Vista either.
/>/
/>/ This was the command I invoked cython with in both cases:
/>/ C:\Python26\python.exe "C:\Python26\Scripts\cython-script.py"
/>/ ".pyx" -o ".c"
/>/
/>/ Any help would be appreciated!  I need Windows 7 support for some of the
/>/ people in my group.
/>/
/>/ Is the problem already fixed in the hg version of cython?
/
It might be--do you want to try it out? You can download the latest
beta from http://cython.org/release/Cython-0.13.beta0.zip

- Robert

------------------------------------------------------------------------


From robertwb at math.washington.edu  Thu Jul 29 19:46:23 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 29 Jul 2010 10:46:23 -0700
Subject: [Cython] Unrecognized Character issue for Windows 7
In-Reply-To: <4C51B3E9.6070405@gmail.com>
References: <4C51B3E9.6070405@gmail.com>
Message-ID: 

On Thu, Jul 29, 2010 at 10:01 AM, David Mashburn
 wrote:
> Thanks Robert!
> Upgrading did it!

Great!

> I did have to change my run command to:
>
> C:\Python26\python.exe "C:\Python26\Lib\site-packages\cython.py" ".pyx" -o ".c"
>
> rather than
>
> C:\Python26\python.exe "C:\Python26\Scripts\cython-script.py" ".pyx" -o ".c"
> or
> C:\Python26\python.exe "C:\Python26\Scripts\cython.py" ".pyx" -o ".c"
>
> The latter seemed to somehow still be connected with 0.12 after upgrade.

Just out of curiosity, where did you get your original copy of Cython?

-Robert

> On Wed, Jul 28, 2010 at 5:46 PM, David Mashburn
> > wrote:
>>/ I've seen the bug about the unrecognized characters on Windows, but I
> />/ had another, seemingly related issue...
> />/
> />/ I'm currently using cython v. 0.12.1 on Windows 7, and I can't get it to
> />/ recognize ANY newlines...
> />/ Not '\n'
> />/ Not '\r'
> />/ Not '\r\n'
> />/ Files without newlines run fine, but with any type of newline, the
> />/ unrecognized character error raises...
> />/
> />/ Did I mention this only happens on Windows 7?
> />/ I tried the exact same thing on XP and it runs without issue... and I am
> />/ pretty sure I had no trouble on Vista either.
> />/
> />/ This was the command I invoked cython with in both cases:
> />/ C:\Python26\python.exe "C:\Python26\Scripts\cython-script.py"
> />/ ".pyx" -o ".c"
> />/
> />/ Any help would be appreciated! ?I need Windows 7 support for some of the
> />/ people in my group.
> />/
> />/ Is the problem already fixed in the hg version of cython?
> /
> It might be--do you want to try it out? You can download the latest
> beta from http://cython.org/release/Cython-0.13.beta0.zip
>
> - Robert
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>