From stavdev at gmail.com Thu Aug 5 05:26:43 2010 From: stavdev at gmail.com (Michael Kamensky) Date: Thu, 5 Aug 2010 07:26:43 +0400 Subject: [Cython] A couple bug reports + possibly suboptimal solutions for them Message-ID: Hello, I'd like to report a bug with Cython 0.12.1 which is also present in the latest SVN version: embedding an interpreter with the --embed option (to create the .exe file) fails on MS Windows with the error "undefined reference to WinMain at 16". I was able to overcome that error by changing this line in the output .c file: int wmain(int argc, wchar_t **argv) { into this: int main(int argc, wchar_t **argv) { The second problem (also on MS Windows) is that sys.argv do not get parsed correctly due to the fact that it seems like the command line is parsed as "char**", but Python expects "wchar_t**". Here's what I did to overcome the issue: I changed this: PySys_SetArgv(argc, argv); __pyx_module_is_main_test1 = 1; Into this: //PySys_SetArgv(argc, argv); __pyx_module_is_main_test1 = 1; PyObject *py_argv= PyList_New(0); int i; for (i=0; i I'm working with Enthought on a branch of Cython to target .NET for IronPython. I'm maintaining a repository at http://bitbucket.org/cwitty/cython-for-ironpython (I just started writing code, so there's not much interesting there yet). I'm hopeful that eventually this code will be merged back into Cython; I don't want to fork Cython. I'm starting by essentially duplicating all of the generate_* methods into generate_dotnet_* methods, and modifying the duplicated methods to generate C++/CLI code. My hope is that once I have a working compiler, I can refactor to merge most of the methods back together (eliminating the duplicate code) while still not being very intrusive to the original codebase. Carl From dagss at student.matnat.uio.no Thu Aug 5 09:08:25 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 05 Aug 2010 09:08:25 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: Message-ID: <4C5A6369.1000606@student.matnat.uio.no> On 08/05/2010 06:01 AM, Carl Witty wrote: > I'm working with Enthought on a branch of Cython to target .NET for > IronPython. I'm maintaining a repository at > http://bitbucket.org/cwitty/cython-for-ironpython (I just started > writing code, so there's not much interesting there yet). > Great, I think this is fantastic news. > I'm hopeful that eventually this code will be merged back into Cython; > I don't want to fork Cython. > I hope this stays in Cython as well. > I'm starting by essentially duplicating all of the generate_* methods > into generate_dotnet_* methods, and modifying the duplicated methods > to generate C++/CLI code. My hope is that once I have a working > Ah, this is very interesting -- when I've spoken with Enthought on the matter we've largely ignored C++ for this and were talking about a C# backend. > compiler, I can refactor to merge most of the methods back together > (eliminating the duplicate code) while still not being very intrusive > to the original codebase. > Have you considered writing a transform instead of adding more methods to the nodes? E.g. something like Cython/CodeWriter.py for the .NET backend? That would be even less intrusive...ideally, one would just configure the pipeline differently depending on what the backend is. (I once suggested that the CPython generation was also moved into a seperate generator class like this as well; of course, that's a lot of unnecessary work) I'm just not too sure about having "if dotnet:" all over the place (eventually). Of course, you should just do whatever gives results quickest at this point (and I very much trust your judgement), I'm just making sure the option is mentioned. Dag Sverre From robertwb at math.washington.edu Thu Aug 5 10:27:27 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Aug 2010 01:27:27 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5A6369.1000606@student.matnat.uio.no> References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn wrote: > On 08/05/2010 06:01 AM, Carl Witty wrote: >> I'm working with Enthought on a branch of Cython to target .NET for >> IronPython. ?I'm maintaining a repository at >> http://bitbucket.org/cwitty/cython-for-ironpython (I just started >> writing code, so there's not much interesting there yet). >> > Great, I think this is fantastic news. I'm excited about this too. >> I'm hopeful that eventually this code will be merged back into Cython; >> I don't want to fork Cython. >> > I hope this stays in Cython as well. >> I'm starting by essentially duplicating all of the generate_* methods >> into generate_dotnet_* methods, and modifying the duplicated methods >> to generate C++/CLI code. ?My hope is that once I have a working >> > Ah, this is very interesting -- when I've spoken with Enthought on the > matter we've largely ignored C++ for this and were talking about a C# > backend. >> compiler, I can refactor to merge most of the methods back together >> (eliminating the duplicate code) while still not being very intrusive >> to the original codebase. >> > > Have you considered writing a transform instead of adding more methods > to the nodes? E.g. something like Cython/CodeWriter.py for the .NET > backend? That would be even less intrusive...ideally, one would just > configure the pipeline differently depending on what the backend is. +1 The transform could go through and replace any nodes that need special casing with special dot_net versions (that only need to know how to generate their code). This would keep things nice and modular. (The real question is how much code could be shared between the CPython bindings and the IronPython ones--the raw C code generation would probably be virtually identical.) > (I once suggested that the CPython generation was also moved into a > seperate generator class like this as well; of course, that's a lot of > unnecessary work) > > I'm just not too sure about having "if dotnet:" all over the place > (eventually). Of course, you should just do whatever gives results > quickest at this point (and I very much trust your judgement), I'm just > making sure the option is mentioned. On this note, it would be cool if, eventually, Jython could be supported through similar mechanisms. - Robert From dagss at student.matnat.uio.no Thu Aug 5 11:39:36 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 05 Aug 2010 11:39:36 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: <4C5A86D8.8020705@student.matnat.uio.no> Robert Bradshaw wrote: > On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn > wrote: > >> On 08/05/2010 06:01 AM, Carl Witty wrote: >> >>> I'm working with Enthought on a branch of Cython to target .NET for >>> IronPython. I'm maintaining a repository at >>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started >>> writing code, so there's not much interesting there yet). >>> >>> >> Great, I think this is fantastic news. >> > > I'm excited about this too. > > >>> I'm hopeful that eventually this code will be merged back into Cython; >>> I don't want to fork Cython. >>> >>> >> I hope this stays in Cython as well. >> >>> I'm starting by essentially duplicating all of the generate_* methods >>> into generate_dotnet_* methods, and modifying the duplicated methods >>> to generate C++/CLI code. My hope is that once I have a working >>> >>> >> Ah, this is very interesting -- when I've spoken with Enthought on the >> matter we've largely ignored C++ for this and were talking about a C# >> backend. >> >>> compiler, I can refactor to merge most of the methods back together >>> (eliminating the duplicate code) while still not being very intrusive >>> to the original codebase. >>> >>> >> Have you considered writing a transform instead of adding more methods >> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET >> backend? That would be even less intrusive...ideally, one would just >> configure the pipeline differently depending on what the backend is. >> > > +1 The transform could go through and replace any nodes that need > special casing with special dot_net versions (that only need to know > how to generate their code). This would keep things nice and modular. > (The real question is how much code could be shared between the > CPython bindings and the IronPython ones--the raw C code generation > would probably be virtually identical.) > I was actually thinking that one had the "transform" (or rather "sink") just emit/serialize code directly when visiting a node. The "return value" could either be string fragments, or if that is too inefficient, nothing at all. def visit_BinopNode(self, node): # Very naive, I know things are more complicated... return '%s %s %s' % (self.visit(node.operand1), node.operator, self.visit(node.operand2)) def visit_FuncNode(self, node): # Here we would return a StringIOTree instead (see Cython/StringIOTree.py), # for efficiency. or, perhaps just: def visit_BinopNode(self, node): self.visit(node.operand1) self.put(node.operator) self.visit(node.operand2) So generate_... would never be called, on *any* node, when doing .NET output. One could deal with shared CPython/.NET code by creating a superclass: class CLikeCodeGenerator: # pull out stuff that is common between CPython and .NET here class DotNetCodeGenerator(CLikeCodeGenerator): # .NET-specific nodes handled here class CPythonCodeGenerator(CLikeCodeGenerator): def visit_Node(self, node): # for now, just call generate_... if the node wasn't handled # by CLikeCodeGenerator This makes it easy to plug in Java as well without making Nodes.py and ExprNodes.py even larger and more tangled and more full of conditionals... Dag Sverre From dagss at student.matnat.uio.no Thu Aug 5 11:43:26 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 05 Aug 2010 11:43:26 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5A86D8.8020705@student.matnat.uio.no> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5A86D8.8020705@student.matnat.uio.no> Message-ID: <4C5A87BE.80207@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: > >> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn >> wrote: >> >> >>> On 08/05/2010 06:01 AM, Carl Witty wrote: >>> >>> >>>> I'm working with Enthought on a branch of Cython to target .NET for >>>> IronPython. I'm maintaining a repository at >>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started >>>> writing code, so there's not much interesting there yet). >>>> >>>> >>>> >>> Great, I think this is fantastic news. >>> >>> >> I'm excited about this too. >> >> >> >>>> I'm hopeful that eventually this code will be merged back into Cython; >>>> I don't want to fork Cython. >>>> >>>> >>>> >>> I hope this stays in Cython as well. >>> >>> >>>> I'm starting by essentially duplicating all of the generate_* methods >>>> into generate_dotnet_* methods, and modifying the duplicated methods >>>> to generate C++/CLI code. My hope is that once I have a working >>>> >>>> >>>> >>> Ah, this is very interesting -- when I've spoken with Enthought on the >>> matter we've largely ignored C++ for this and were talking about a C# >>> backend. >>> >>> >>>> compiler, I can refactor to merge most of the methods back together >>>> (eliminating the duplicate code) while still not being very intrusive >>>> to the original codebase. >>>> >>>> >>>> >>> Have you considered writing a transform instead of adding more methods >>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET >>> backend? That would be even less intrusive...ideally, one would just >>> configure the pipeline differently depending on what the backend is. >>> >>> >> +1 The transform could go through and replace any nodes that need >> special casing with special dot_net versions (that only need to know >> how to generate their code). This would keep things nice and modular. >> (The real question is how much code could be shared between the >> CPython bindings and the IronPython ones--the raw C code generation >> would probably be virtually identical.) >> >> > I was actually thinking that one had the "transform" (or rather "sink") > just emit/serialize code directly when visiting a node. The "return > value" could either be string fragments, or if that is too inefficient, > nothing at all. > > def visit_BinopNode(self, node): > # Very naive, I know things are more complicated... > return '%s %s %s' % (self.visit(node.operand1), node.operator, > self.visit(node.operand2)) > > def visit_FuncNode(self, node): > # Here we would return a StringIOTree instead (see > Cython/StringIOTree.py), > # for efficiency. > > or, perhaps just: > > def visit_BinopNode(self, node): > self.visit(node.operand1) > self.put(node.operator) > self.visit(node.operand2) > > So generate_... would never be called, on *any* node, when doing .NET > output. > > One could deal with shared CPython/.NET code by creating a superclass: > Actually, in the first pass, one could implement DotNetCodeGenerator like this: class DotNetCodeGenerator(CLikeCodeGenerator): # Default fallback to CPython generation def visit_Node(self, node): node.generate_...(self.code) def visit_ExprNode(self, node): # Slightly different logic I believe... node.generate_...(self.code) # Any .NET specific stuff def visit_FuncNode(self, node): ... Dag Sverre > class CLikeCodeGenerator: > # pull out stuff that is common between CPython and .NET here > > class DotNetCodeGenerator(CLikeCodeGenerator): > # .NET-specific nodes handled here > > class CPythonCodeGenerator(CLikeCodeGenerator): > def visit_Node(self, node): > # for now, just call generate_... if the node wasn't handled > # by CLikeCodeGenerator > > > This makes it easy to plug in Java as well without making Nodes.py and > ExprNodes.py even larger and more tangled and more full of conditionals... > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Thu Aug 5 12:56:59 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Aug 2010 03:56:59 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5A86D8.8020705@student.matnat.uio.no> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5A86D8.8020705@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 2:39 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn >> wrote: >> >>> On 08/05/2010 06:01 AM, Carl Witty wrote: >>> >>>> I'm working with Enthought on a branch of Cython to target .NET for >>>> IronPython. ?I'm maintaining a repository at >>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started >>>> writing code, so there's not much interesting there yet). >>>> >>>> >>> Great, I think this is fantastic news. >>> >> >> I'm excited about this too. >> >> >>>> I'm hopeful that eventually this code will be merged back into Cython; >>>> I don't want to fork Cython. >>>> >>>> >>> I hope this stays in Cython as well. >>> >>>> I'm starting by essentially duplicating all of the generate_* methods >>>> into generate_dotnet_* methods, and modifying the duplicated methods >>>> to generate C++/CLI code. ?My hope is that once I have a working >>>> >>>> >>> Ah, this is very interesting -- when I've spoken with Enthought on the >>> matter we've largely ignored C++ for this and were talking about a C# >>> backend. >>> >>>> compiler, I can refactor to merge most of the methods back together >>>> (eliminating the duplicate code) while still not being very intrusive >>>> to the original codebase. >>>> >>>> >>> Have you considered writing a transform instead of adding more methods >>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET >>> backend? That would be even less intrusive...ideally, one would just >>> configure the pipeline differently depending on what the backend is. >>> >> >> +1 The transform could go through and replace any nodes that need >> special casing with special dot_net versions (that only need to know >> how to generate their code). This would keep things nice and modular. >> (The real question is how much code could be shared between the >> CPython bindings and the IronPython ones--the raw C code generation >> would probably be virtually identical.) >> > I was actually thinking that one had the "transform" (or rather "sink") > just emit/serialize code directly when visiting a node. The "return > value" could either be string fragments, or if that is too inefficient, > nothing at all. > > def visit_BinopNode(self, node): > ? ?# Very naive, I know things are more complicated... > ? ?return '%s %s %s' % (self.visit(node.operand1), node.operator, > self.visit(node.operand2)) > > def visit_FuncNode(self, node): > ? ?# Here we would return a StringIOTree instead (see > Cython/StringIOTree.py), > ? ?# for efficiency. > > or, perhaps just: > > def visit_BinopNode(self, node): > ? ?self.visit(node.operand1) > ? ?self.put(node.operator) > ? ?self.visit(node.operand2) > > So generate_... would never be called, on *any* node, when doing .NET > output. > > One could deal with shared CPython/.NET code by creating a superclass: > > class CLikeCodeGenerator: > ? ?# pull out stuff that is common between CPython and .NET here > > class DotNetCodeGenerator(CLikeCodeGenerator): > ? ?# .NET-specific nodes handled here > > class CPythonCodeGenerator(CLikeCodeGenerator): > ? ?def visit_Node(self, node): > ? ? ? ?# for now, just call generate_... if the node wasn't handled > ? ? ? ?# by CLikeCodeGenerator > > > This makes it easy to plug in Java as well without making Nodes.py and > ExprNodes.py even larger and more tangled and more full of conditionals... Ah, yes, I remember discussing this before. If that could be pulled off, it would be nice and clean. The fact that memory management is handled for you (I think) makes things much simpler. My one concerns is if (taken to the extreem) having a visitor that knows how to generate code for every type (and the requisite internals) is preferable to encapsulating this knowledge on a node-by-node basis. Perhaps. - Robert From chemejosh at gmail.com Thu Aug 5 15:07:59 2010 From: chemejosh at gmail.com (Josh Allen) Date: Thu, 5 Aug 2010 09:07:59 -0400 Subject: [Cython] Package compiling in pure Python mode In-Reply-To: References: Message-ID: Sorry for the delayed response; it seems that my GMail ate your reply. I was trying to set up the 0.13 beta to install alongside the stable 0.12.1. However, I was not able to get it to work this way, so I decided to go ahead and install the 0.13 beta in place of 0.12.1. When I did this, I found that this bug seems to have been addressed. For the record, when I switched from .py to .pyx (and added cdefs and cpdefs as appropriate) 0.12.1 still failed with the same error. Thanks for your help! Josh On Wed, Jul 28, 2010 at 1:03 PM, Josh Allen wrote: > Hello again, > > I'm working on a Python/Cython project with a large number of modules, > several of which I have Cythonized. I have collected all of the modules into > a single package. Attempting to compile these modules has revealed a couple > of potential issues, which I will outline below for a small test case. > > First, the directory layout for the test case, which contains two modules > "foo" and "bar" in a package "bug": > > /bug/__init__.py > /bug/bar.pxd > /bug/bar.py > /bug/foo.pxd > /bug/foo.py > /setup.py > /test.py > > Next, the contents of the files: > > # /bug/foo.py: > class Foo: > def hello(self): > print 'Hello from foo.Foo.hello()' > > # /bug/foo.pxd: > cdef class Foo: > cpdef hello(self) > > # /bug/bar.pxd: > from foo cimport Foo > cdef class Bar: > cdef public Foo foo > cpdef hello(self) > > # /bug/bar.py: > import cython > from foo import Foo > class Bar: > def hello(self): > cython.declare(f=Foo) > f = Foo() > print 'Hello from bar.Bar.hello()' > > # setup.py > from distutils.core import setup > from distutils.extension import Extension > from Cython.Distutils import build_ext > setup( > cmdclass={'build_ext': build_ext}, > ext_modules=[Extension('bug.foo', ['bug/foo.py']), Extension('bug.bar', > ['bug/bar.py'])] > ) > > # test.py > from bug.foo import Foo > from bug.bar import Bar > f = Foo() > f.hello() # Should print 'Hello from Foo.hello()' > b = Bar() > b.hello() # Should print 'Hello from Bar.hello()' > > If I try to compile as is -- by running "python setup.py build_ext > --inplace" from the top-level directory --, Cython fails at the line "from > foo import Foo" in bar.py with the error "Assignment to non-lvalue 'Foo'". > If I remove this line, then everything compiles and runs properly. However, > I need this line to exist in order to run the file in pure Python mode. > > I also tried moving the setup.py file within the bug package, adjusting its > content accordingly, and running from within the bug folder. (This is > probably not a good location for the setup.py file, but I was trying to see > if anything would work.) This also causes Cython to fail, but this time at > the line "cython.declare(f=Foo)" in bar.py with the error "Unknown type". > However, running cython from the console on bar.py while within the bug > directory does produce a valid C file. > > Both of these cases are for Python 2.6.5 and Cython 0.12.1 on Ubuntu 10.04. > I also attempted to try it for the Cython 0.13 beta, but the problem seems > to persist. (Incidentally, is running "cython -V" with the 0.13 beta > supposed to print "Cython version 0.12.1"?) > > Let me know if there's any other info you need. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100805/54ee2801/attachment.htm From carl.witty at gmail.com Thu Aug 5 18:58:58 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 5 Aug 2010 09:58:58 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5A6369.1000606@student.matnat.uio.no> References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn wrote: > On 08/05/2010 06:01 AM, Carl Witty wrote: >> I'm starting by essentially duplicating all of the generate_* methods >> into generate_dotnet_* methods, and modifying the duplicated methods >> to generate C++/CLI code. ?My hope is that once I have a working >> > Ah, this is very interesting -- when I've spoken with Enthought on the > matter we've largely ignored C++ for this and were talking about a C# > backend. I convinced the Enthought people that a C++/CLI backend was the best first step. For a variety of reasons (mostly the fact that Cython's "extern from" describes the API, not the ABI, but also catching C++ exceptions and turning them into Python exceptions) a C# backend would have to generate C or C++ wrappers for every operation on "extern from" values/types (function call, global variable read, global variable write, struct member read, struct member write, ...). On the other hand, a C++/CLI backend can handle "extern from" in exactly the same way that the current C/C++ backend does. Once the C++/CLI backend works, a C# backend is a possible next step. All of the code generation for C# to interface with IronPython would be exactly the same as C++/CLI (modulo minor syntactic issues that can easily be localized). Even if the ultimate goal is a C# backend, I think that very little of the work on a C++/CLI backend would be wasted. >> compiler, I can refactor to merge most of the methods back together >> (eliminating the duplicate code) while still not being very intrusive >> to the original codebase. >> > > Have you considered writing a transform instead of adding more methods > to the nodes? E.g. something like Cython/CodeWriter.py for the .NET > backend? That would be even less intrusive...ideally, one would just > configure the pipeline differently depending on what the backend is. I did consider this briefly. Given the choice between writing a backend basically from scratch, or being able to copy a working backend and incrementally modify it, the latter seemed a lot easier. :) Once I have a working compiler, and it's time to clean up the code for submission, I'll definitely consider this approach. Based on the tiny amount of work I've done so far, though, I suspect that I may be able to share a lot of code with the existing backend that would end up duplicated if I made my backend into a transform. > (I once suggested that the CPython generation was also moved into a > seperate generator class like this as well; of course, that's a lot of > unnecessary work) Sounds like a good idea to me. Also, if the existing backend and the C++/CLI backend were both transforms, that would make it a lot easier for them to share code (they could both inherit from a generic "code generation" transform, for instance). But I doubt if I'll have time to work on this... > I'm just not too sure about having "if dotnet:" all over the place > (eventually). Of course, you should just do whatever gives results > quickest at this point (and I very much trust your judgement), I'm just > making sure the option is mentioned. Keeping the number of "if dotnet:" down is definitely an important long-term goal. (For now, I'm adding lots of them, but I expect to remove them and handle the cases some other way before I submit the code.) > Dag Sverre Carl From carl.witty at gmail.com Thu Aug 5 19:13:22 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 5 Aug 2010 10:13:22 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 1:27 AM, Robert Bradshaw wrote: > +1 The transform could go through and replace any nodes that need > special casing with special dot_net versions (that only need to know > how to generate their code). This would keep things nice and modular. > (The real question is how much code could be shared between the > CPython bindings and the IronPython ones--the raw C code generation > would probably be virtually identical.) I do like this idea. My tentative plan for merging the existing backend and the C++/CLI backend back together is to move most of the C code fragments from Nodes.py/ExprNodes.py into more methods on Code.py's CCodeWriter. Then I would split the generic parts of CCodeWriter into a new base class BaseCodeWriter, and add a new subclass CPPCLICodeWriter. Then (hopefully) a single conditional in ModuleNode.generate_c_code, where it creates the rootwriter object, could handle most of the differences between the backends. For nodes where the code generation difference is too large to be reasonably hidden in Code.py, your transform sounds like a reasonable way to avoid "if dotnet:". > On this note, it would be cool if, eventually, Jython could be > supported through similar mechanisms. That would be very cool. (I don't know anything about Jython internals or the JVM foreign function interface, so I don't know how hard it would be.) > - Robert Carl From carl.witty at gmail.com Thu Aug 5 19:24:19 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 5 Aug 2010 10:24:19 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5A87BE.80207@student.matnat.uio.no> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5A86D8.8020705@student.matnat.uio.no> <4C5A87BE.80207@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 2:43 AM, Dag Sverre Seljebotn wrote: > Actually, in the first pass, one could implement DotNetCodeGenerator > like this: > > class DotNetCodeGenerator(CLikeCodeGenerator): > ? ?# Default fallback to CPython generation > ? ?def visit_Node(self, node): > ? ? ? ?node.generate_...(self.code) > > ? ?def visit_ExprNode(self, node): > ? ? ? ?# Slightly different logic I believe... > ? ? ? ?node.generate_...(self.code) > > ? ?# Any .NET specific stuff > ? ?def visit_FuncNode(self, node): ... It looks like this would work if you're calling generate_result_code, say. Other generate_ methods, like generate_function_definitions, are recursive, so if you ever called the CPython version that would be used for all of the children of the node as well as the node itself -- there's no mechanism for the transform to get control back. > Dag Sverre Carl From carl.witty at gmail.com Thu Aug 5 19:32:13 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 5 Aug 2010 10:32:13 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> <4C5A86D8.8020705@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 3:56 AM, Robert Bradshaw wrote: > My one concerns is if (taken to the extreem) having a visitor that > knows how to generate code for every type (and the requisite > internals) is preferable to encapsulating this knowledge on a > node-by-node basis. Perhaps. Ah yes... the perennial style debate: if you have a bunch of data types, and a bunch of operations on those types, is it better to group the code by type, or by operation? (And of course there's no "right" answer to such a generic question.) AFAIK, other compilers (like gcc and LLVM) group code by operation -- I think they would typically have a source file for each compiler pass, for example. On the other hand, Cython probably has more node types (and semantically richer node types), and fewer operations, than those compilers, so the right answer for them isn't necessarily right for Cython... > - Robert Carl From robertwb at math.washington.edu Fri Aug 6 05:26:38 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Aug 2010 20:26:38 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty wrote: > On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn > wrote: >> On 08/05/2010 06:01 AM, Carl Witty wrote: >>> I'm starting by essentially duplicating all of the generate_* methods >>> into generate_dotnet_* methods, and modifying the duplicated methods >>> to generate C++/CLI code. ?My hope is that once I have a working >>> >> Ah, this is very interesting -- when I've spoken with Enthought on the >> matter we've largely ignored C++ for this and were talking about a C# >> backend. > > I convinced the Enthought people that a C++/CLI backend was the best > first step. ?For a variety of reasons (mostly the fact that Cython's > "extern from" describes the API, not the ABI, but also catching C++ > exceptions and turning them into Python exceptions) a C# backend would > have to generate C or C++ wrappers for every operation on "extern > from" values/types (function call, global variable read, global > variable write, struct member read, struct member write, ...). ?On the > other hand, a C++/CLI backend can handle "extern from" in exactly the > same way that the current C/C++ backend does. > > Once the C++/CLI backend works, a C# backend is a possible next step. > All of the code generation for C# to interface with IronPython would > be exactly the same as C++/CLI (modulo minor syntactic issues that can > easily be localized). ?Even if the ultimate goal is a C# backend, I > think that very little of the work on a C++/CLI backend would be > wasted. The ultimate goal is really just Cython working (well) for IronPython, right? How essential is C# generation (vs. C++/CLI) to that goal? >>> compiler, I can refactor to merge most of the methods back together >>> (eliminating the duplicate code) while still not being very intrusive >>> to the original codebase. >>> >> >> Have you considered writing a transform instead of adding more methods >> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET >> backend? That would be even less intrusive...ideally, one would just >> configure the pipeline differently depending on what the backend is. > > I did consider this briefly. ?Given the choice between writing a > backend basically from scratch, or being able to copy a working > backend and incrementally modify it, the latter seemed a lot easier. > :) > > Once I have a working compiler, and it's time to clean up the code for > submission, I'll definitely consider this approach. ?Based on the tiny > amount of work I've done so far, though, I suspect that I may be able > to share a lot of code with the existing backend that would end up > duplicated if I made my backend into a transform. I think for a task like this, once you've gotten something working it'll be a lot clearer what the right form is as well. >> (I once suggested that the CPython generation was also moved into a >> seperate generator class like this as well; of course, that's a lot of >> unnecessary work) > > Sounds like a good idea to me. ?Also, if the existing backend and the > C++/CLI backend were both transforms, that would make it a lot easier > for them to share code (they could both inherit from a generic "code > generation" transform, for instance). ?But I doubt if I'll have time > to work on this... > >> I'm just not too sure about having "if dotnet:" all over the place >> (eventually). Of course, you should just do whatever gives results >> quickest at this point (and I very much trust your judgement), I'm just >> making sure the option is mentioned. > > Keeping the number of "if dotnet:" down is definitely an important > long-term goal. ?(For now, I'm adding lots of them, but I expect to > remove them and handle the cases some other way before I submit the > code.) Excellent. - Robert From robertwb at math.washington.edu Fri Aug 6 05:30:51 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Aug 2010 20:30:51 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 10:13 AM, Carl Witty wrote: > On Thu, Aug 5, 2010 at 1:27 AM, Robert Bradshaw > wrote: >> +1 The transform could go through and replace any nodes that need >> special casing with special dot_net versions (that only need to know >> how to generate their code). This would keep things nice and modular. >> (The real question is how much code could be shared between the >> CPython bindings and the IronPython ones--the raw C code generation >> would probably be virtually identical.) > > I do like this idea. > > My tentative plan for merging the existing backend and the C++/CLI > backend back together is to move most of the C code fragments from > Nodes.py/ExprNodes.py into more methods on Code.py's CCodeWriter. > Then I would split the generic parts of CCodeWriter into a new base > class BaseCodeWriter, and add a new subclass CPPCLICodeWriter. ?Then > (hopefully) a single conditional in ModuleNode.generate_c_code, where > it creates the rootwriter object, could handle most of the differences > between the backends. That sounds like a very good move in the long run. > For nodes where the code generation difference > is too large to be reasonably hidden in Code.py, your transform sounds > like a reasonable way to avoid "if dotnet:". > >> On this note, it would be cool if, eventually, Jython could be >> supported through similar mechanisms. > > That would be very cool. ?(I don't know anything about Jython > internals or the JVM foreign function interface, so I don't know how > hard it would be.) I don't know either, but the very act of supporting any other backend should shed light on this, even if the technical details are different. On Thu, Aug 5, 2010 at 10:32 AM, Carl Witty wrote: > On Thu, Aug 5, 2010 at 3:56 AM, Robert Bradshaw > wrote: >> My one concerns is if (taken to the extreem) having a visitor that >> knows how to generate code for every type (and the requisite >> internals) is preferable to encapsulating this knowledge on a >> node-by-node basis. Perhaps. > > Ah yes... the perennial style debate: if you have a bunch of data > types, and a bunch of operations on those types, is it better to group > the code by type, or by operation? ?(And of course there's no "right" > answer to such a generic question.) > > AFAIK, other compilers (like gcc and LLVM) group code by operation -- > I think they would typically have a source file for each compiler > pass, for example. ?On the other hand, Cython probably has more node > types (and semantically richer node types), and fewer operations, than > those compilers, so the right answer for them isn't necessarily right > for Cython... Yes, it's far from clear. If we're moving this direction long-term, it may be possible to use transforms to reduce things to a smaller subset of simpler node types earlier on. - Robert From robertwb at math.washington.edu Fri Aug 6 05:37:21 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Aug 2010 20:37:21 -0700 Subject: [Cython] A couple bug reports + possibly suboptimal solutions for them In-Reply-To: References: Message-ID: On Wed, Aug 4, 2010 at 8:26 PM, Michael Kamensky wrote: > Hello, > > I'd like to report a bug with Cython 0.12.1 which is also present in the > latest SVN version: > embedding an interpreter with the --embed option (to create the .exe file) > fails on MS Windows > with the error "undefined reference to WinMain at 16". I was able to overcome > that error by > changing this line in the output .c file: > > int wmain(int argc, wchar_t **argv) { > > into this: > > int main(int argc, wchar_t **argv) { > > The second problem (also on MS Windows) is that sys.argv do not get parsed > correctly due to > the fact that it seems like the command line is parsed as "char**", but > Python expects "wchar_t**". > Here's what I did to overcome the issue: > > I changed this: > > ? PySys_SetArgv(argc, argv);??? __pyx_module_is_main_test1 = 1; > > Into this: > > ? //PySys_SetArgv(argc, argv);??? __pyx_module_is_main_test1 = 1; > ? PyObject *py_argv= PyList_New(0); > ? int i; > ????? for (i=0; i ????????? PyObject *item= PyUnicode_Decode(argv[i], strlen(argv[i]), > Py_FileSystemDefaultEncoding, NULL); > ????? if(item==NULL) { // should never happen > ????????? PyErr_Print(); > ????????? PyErr_Clear(); > ????? } > ????? else { > ????????? PyList_Append(py_argv, item); > ????????? Py_DECREF(item); > ????? } > ????? } > ????? PySys_SetObject("argv", py_argv); > ????? Py_DECREF(py_argv); > > > I'm not sure if it's optimal, I do get warnings during compilation, but at > least it works (it might not be cross-platform > or whatever, so it's just a proposed bug report with a demo as to how I > solved the issue). Hope it helps. :) I think these are one and the same issue--wmain is supposed to be a main that takes "wide" char arguments. According to http://msdn.microsoft.com/en-us/library/aa299386(VS.60).aspx one can use either (for MS tools I guess, what complier are you using?) > I love your project, thank you so much for Cython! :D Thanks for the complements. - Robert From carl.witty at gmail.com Fri Aug 6 06:55:47 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 5 Aug 2010 21:55:47 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 8:26 PM, Robert Bradshaw wrote: > On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty wrote: >> Once the C++/CLI backend works, a C# backend is a possible next step. >> All of the code generation for C# to interface with IronPython would >> be exactly the same as C++/CLI (modulo minor syntactic issues that can >> easily be localized). ?Even if the ultimate goal is a C# backend, I >> think that very little of the work on a C++/CLI backend would be >> wasted. > > The ultimate goal is really just Cython working (well) for IronPython, > right? How essential is C# generation (vs. C++/CLI) to that goal? For IronPython running under Microsoft's .NET implementation for Windows, as far as I know there are no problems with C++/CLI. For IronPython running under Mono on non-Windows platforms, C++/CLI doesn't help at all; there are no implementations of C++/CLI on non-Windows platforms and no effort toward making an implementation. So the purpose of a C# backend would be to support Mono. Carl From robertwb at math.washington.edu Fri Aug 6 07:14:14 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Aug 2010 22:14:14 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Thu, Aug 5, 2010 at 9:55 PM, Carl Witty wrote: > On Thu, Aug 5, 2010 at 8:26 PM, Robert Bradshaw > wrote: >> On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty wrote: >>> Once the C++/CLI backend works, a C# backend is a possible next step. >>> All of the code generation for C# to interface with IronPython would >>> be exactly the same as C++/CLI (modulo minor syntactic issues that can >>> easily be localized). ?Even if the ultimate goal is a C# backend, I >>> think that very little of the work on a C++/CLI backend would be >>> wasted. >> >> The ultimate goal is really just Cython working (well) for IronPython, >> right? How essential is C# generation (vs. C++/CLI) to that goal? > > For IronPython running under Microsoft's .NET implementation for > Windows, as far as I know there are no problems with C++/CLI. ?For > IronPython running under Mono on non-Windows platforms, C++/CLI > doesn't help at all; there are no implementations of C++/CLI on > non-Windows platforms and no effort toward making an implementation. > So the purpose of a C# backend would be to support Mono. Ah. As much as I support the Mono project, I would imagine that most Mono users who wanted to use Python would just use CPython directly (though I wouldn't rule out any usecase of IronPython on Mono). Of course this is just pure speculation, I don't follow the Mono project very closely. If it comes out naturally once the C++/CLI is going well that would be a huge plus. - Robert From dagss at student.matnat.uio.no Fri Aug 6 11:12:59 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 06 Aug 2010 11:12:59 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: <4C5BD21B.1080902@student.matnat.uio.no> Carl Witty wrote: > On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn > wrote: > >> On 08/05/2010 06:01 AM, Carl Witty wrote: >> >>> I'm starting by essentially duplicating all of the generate_* methods >>> into generate_dotnet_* methods, and modifying the duplicated methods >>> to generate C++/CLI code. My hope is that once I have a working >>> >>> >> Ah, this is very interesting -- when I've spoken with Enthought on the >> matter we've largely ignored C++ for this and were talking about a C# >> backend. >> > > I convinced the Enthought people that a C++/CLI backend was the best > first step. For a variety of reasons (mostly the fact that Cython's > "extern from" describes the API, not the ABI, but also catching C++ > exceptions and turning them into Python exceptions) a C# backend would > have to generate C or C++ wrappers for every operation on "extern > from" values/types (function call, global variable read, global > variable write, struct member read, struct member write, ...). On the > other hand, a C++/CLI backend can handle "extern from" in exactly the > same way that the current C/C++ backend does. > Ahh. Yes, that makes everything much nicer indeed. (I know very little about .NET, but I did manage to identify the "extern from" as the critical non-obvious piece to get Cython running...) My reason for asking is that I looked forward to all the goodies of C#: The "dynamic" type with call-site caching when calling methods on pure Python objects, and (down the line) builtin support for closures. (I don't know how much of that is available in C++/CLI?) Speaking, hypothetically, about a C# backend, my two preferred options for _C_ code are either a) require that an ABI is specified in the Cython code (which is backwards-compatible with specifying API), or b) a two-stage build to generate C source that generates C# source (using offsetof and sizeof etc. to get hold of the ABI). Interfacing with C++ would need run-time wrappers, anyway, though... Anyway, I'm really glad that you have such a good handle on these things; I'm really happy Enthought asked you about this, they couldn't have got someone better for the task. Dag Sverre From dagss at student.matnat.uio.no Fri Aug 6 11:31:00 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 06 Aug 2010 11:31:00 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: <4C5BD654.8080205@student.matnat.uio.no> Robert Bradshaw wrote: > On Thu, Aug 5, 2010 at 9:55 PM, Carl Witty wrote: > >> On Thu, Aug 5, 2010 at 8:26 PM, Robert Bradshaw >> wrote: >> >>> On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty wrote: >>> >>>> Once the C++/CLI backend works, a C# backend is a possible next step. >>>> All of the code generation for C# to interface with IronPython would >>>> be exactly the same as C++/CLI (modulo minor syntactic issues that can >>>> easily be localized). Even if the ultimate goal is a C# backend, I >>>> think that very little of the work on a C++/CLI backend would be >>>> wasted. >>>> >>> The ultimate goal is really just Cython working (well) for IronPython, >>> right? How essential is C# generation (vs. C++/CLI) to that goal? >>> >> For IronPython running under Microsoft's .NET implementation for >> Windows, as far as I know there are no problems with C++/CLI. For >> IronPython running under Mono on non-Windows platforms, C++/CLI >> doesn't help at all; there are no implementations of C++/CLI on >> non-Windows platforms and no effort toward making an implementation. >> So the purpose of a C# backend would be to support Mono. >> > > Ah. As much as I support the Mono project, I would imagine that most > Mono users who wanted to use Python would just use CPython directly > (though I wouldn't rule out any usecase of IronPython on Mono). Of > course this is just pure speculation, I don't follow the Mono project > very closely. If it comes out naturally once the C++/CLI is going well > that would be a huge plus. > Mono support could be important *long-term*, because it allows all the non-Windows users to not break things on the .NET side when developing and merging patches. One can include Mono builds in build farms, whether it is Cython, NumPy, SciPy etc. For that purpose Wine could work as well though if .NET is well supported in Wine. Well, given some common scientific Python build farm infrastructure (where Sage and Enthought pools together?), even VirtualBox would be workable. But at the end of the day, Mono will always be more convenient to set up for Linux devs. Ignoring it short-term shouldn't hurt though. Also, though I don't really know, it might be that Cython+C# would beat Cython+CPython performance wise for non-typed Python stuff. And no GIL! CPython could get its act together with a LLVM backend etc., but it would be nice strategically to have Mono available as a competing platform, in case CPython doesn't manage to improve matters. Dag Sverre From carl.witty at gmail.com Fri Aug 6 18:52:57 2010 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 6 Aug 2010 09:52:57 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5BD21B.1080902@student.matnat.uio.no> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5BD21B.1080902@student.matnat.uio.no> Message-ID: On Fri, Aug 6, 2010 at 2:12 AM, Dag Sverre Seljebotn wrote: > My reason for asking is that I looked forward to all the goodies of C#: > The "dynamic" type with call-site caching when calling methods on pure > Python objects, and (down the line) builtin support for closures. (I > don't know how much of that is available in C++/CLI?) Unfortunately, as far as I can tell "dynamic" is useless for a Cython backend, because it will use .NET semantics rather than Python semantics for .NET types. For example, IronPython represents a Python string as a .NET string -- not a wrapper or a subclass, just a plain .NET string. Then there's a bit of magic in the IronPython method lookup code, so that it will notice if you're looking up a Python method on a .NET string and do the right thing. But the magic won't be there in the code generated for C# "dynamic", so this would fail: dynamic s = "a random string"; dynamic h = s.__hash__(); Cython-generated code will have the same call-site caching as "dynamic"; we'll just have to generate the code ourselves. In fact, there is some of this already in the code I posted; for an untyped addition, it will generate the following: A declaration in a "module global variables" class: CallSite< Func< CallSite^, Object^, Object^, Object^ >^ >^ __site_op_add_16_14; This executes once at module load time: __site_op_add_16_14 = CallSite< Func< CallSite^, Object^, Object^, Object^ >^ >::Create(context->CreateBinaryOperationBinder(ExpressionType::Add)); And then to actually do the addition: __pyx_t_1 = __pyx_globals->__site_op_add_16_14->Target(__pyx_globals->__site_op_add_16_14, __pyx_v_v1, __pyx_v_v2); (The 16 and 14 are the line and column number of the addition operation; I thought this would be nicer than some sort of counter. I encoded the operation type in the call site name so you can see what the call is doing.) I think it's true that C# supports closures and C++/CLI does not. I haven't thought much about the issues here, but I was assuming that the IronPython backend would implement closures in the same way as the CPython backend, rather than relying on closure support in the target language. > Speaking, hypothetically, about a C# backend, my two preferred options > for _C_ code are either a) require that an ABI is specified in the > Cython code (which is backwards-compatible with specifying API), or b) a > two-stage build to generate C source that generates C# source (using > offsetof and sizeof etc. to get hold of the ABI). Option (a) is really annoying for libraries like libc, with a portable API but wildly varying ABIs. For example, to wrap the stat() system call, you would need to deal with the fact that "struct stat" has different members with different names on different OSes (and OS versions), and the fact that errno might or might not be provided by #define errno (*_errno()) But maybe there are no other libraries like libc; I can't think of other examples off the top of my head. I guess that usually if you want to portably wrap a library, it's because the library implementation is portable and will have a similar ABI on different platforms. And wrapping libc is not so useful, since so much of it is already provided by Python. Option (b) is intriguing, and not something I'd considered. It doesn't help for "#define errno (*_errno())", but should make it possible to not write wrappers for struct member access, which would be a big efficiency advantage. I'll definitely consider this if I do end up working on a C# backend. > Interfacing with C++ would need run-time wrappers, anyway, though... > > Anyway, I'm really glad that you have such a good handle on these > things; I'm really happy Enthought asked you about this, they couldn't > have got someone better for the task. Thanks! Carl From dagss at student.matnat.uio.no Fri Aug 6 19:05:46 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 06 Aug 2010 19:05:46 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> <4C5BD21B.1080902@student.matnat.uio.no> Message-ID: <4C5C40EA.5020808@student.matnat.uio.no> Carl Witty wrote: > A declaration in a "module global variables" class: > > CallSite< Func< CallSite^, Object^, Object^, Object^ >^ >^ > __site_op_add_16_14; > > This executes once at module load time: > > __site_op_add_16_14 = CallSite< Func< CallSite^, Object^, Object^, > Object^ >^ >::Create(context->CreateBinaryOperationBinder(ExpressionType::Add)); > > And then to actually do the addition: > > __pyx_t_1 = __pyx_globals->__site_op_add_16_14->Target(__pyx_globals->__site_op_add_16_14, > __pyx_v_v1, __pyx_v_v2); > > (The 16 and 14 are the line and column number of the addition > operation; I thought this would be nicer than some sort of counter. I > encoded the operation type in the call site name so you can see what > the call is doing.) > Great, that looks really cool! Dag Sverre From stefan_ml at behnel.de Sun Aug 8 11:15:54 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Aug 2010 11:15:54 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5BD654.8080205@student.matnat.uio.no> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5BD654.8080205@student.matnat.uio.no> Message-ID: <4C5E75CA.7080203@behnel.de> Dag Sverre Seljebotn, 06.08.2010 11:31: > Mono support could be important *long-term*, because it allows all the > non-Windows users to not break things on the .NET side when developing > and merging patches. One can include Mono builds in build farms, whether > it is Cython, NumPy, SciPy etc. > > For that purpose Wine could work as well though if .NET is well > supported in Wine. Well, given some common scientific Python build farm > infrastructure (where Sage and Enthought pools together?), even > VirtualBox would be workable. But at the end of the day, Mono will > always be more convenient to set up for Linux devs. Actually, .NET not being portable pretty much rules out any use case I might have for it. Mono is the only reason for me to consider IronPython a viable alternative to CPython. > Also, though I don't really know, it might be that Cython+C# would beat > Cython+CPython performance wise for non-typed Python stuff. And no GIL! Remember that the GIL makes your life easier. Cython is about writing extension modules, and very, very many of those only work *because* of the GIL. Here's an example: http://code.activestate.com/recipes/577336-fast-re-entrant-optimistic-lock-implemented-in-cyt/ In the Cython wrappers I wrote so far, a *lot* of critical code truly relies on the GIL. So, one part of porting Cython to .NET would be to add a global lock for the code (at least when threading is used, which may be tricky to determine). Obviously, this could become a compiler option for the .NET backend, but the lock would still have to be turned off explicitly in order to prevent breaking Cython code by default. > CPython could get its act together with a LLVM backend etc., but it > would be nice strategically to have Mono available as a competing > platform, in case CPython doesn't manage to improve matters. That would be one heck of a dependency, though. I don't know how big LLVM is, but I would expect it to be a lot smaller than Mono. Stefan From dagss at student.matnat.uio.no Sun Aug 8 21:53:59 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 08 Aug 2010 21:53:59 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5E75CA.7080203@behnel.de> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5BD654.8080205@student.matnat.uio.no> <4C5E75CA.7080203@behnel.de> Message-ID: <4C5F0B57.10209@student.matnat.uio.no> On 08/08/2010 11:15 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 06.08.2010 11:31: > >> Mono support could be important *long-term*, because it allows all the >> non-Windows users to not break things on the .NET side when developing >> and merging patches. One can include Mono builds in build farms, whether >> it is Cython, NumPy, SciPy etc. >> >> For that purpose Wine could work as well though if .NET is well >> supported in Wine. Well, given some common scientific Python build farm >> infrastructure (where Sage and Enthought pools together?), even >> VirtualBox would be workable. But at the end of the day, Mono will >> always be more convenient to set up for Linux devs. >> > Actually, .NET not being portable pretty much rules out any use case I > might have for it. Mono is the only reason for me to consider IronPython a > viable alternative to CPython. > I think the main point of the .NET Cython port is to allow libraries to also target .NET. I.e. I don't think many of the *current* CPython users will really use it themselves (Mono or not), it is all about reaching out to *new* users who are currently using .NET . The usefulness of this depends, of course, very much on the library in question. For SciPy and NumPy there is definitely a demand (IronClad was developed to run NumPy IIRC). Dag Sverre From carl.witty at gmail.com Sun Aug 8 22:11:03 2010 From: carl.witty at gmail.com (Carl Witty) Date: Sun, 8 Aug 2010 13:11:03 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5E75CA.7080203@behnel.de> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5BD654.8080205@student.matnat.uio.no> <4C5E75CA.7080203@behnel.de> Message-ID: On Sun, Aug 8, 2010 at 2:15 AM, Stefan Behnel wrote: > Remember that the GIL makes your life easier. Cython is about writing > extension modules, and very, very many of those only work *because* of the > GIL. Here's an example: > > http://code.activestate.com/recipes/577336-fast-re-entrant-optimistic-lock-implemented-in-cyt/ > > In the Cython wrappers I wrote so far, a *lot* of critical code truly > relies on the GIL. > > So, one part of porting Cython to .NET would be to add a global lock for > the code (at least when threading is used, which may be tricky to > determine). Obviously, this could become a compiler option for the .NET > backend, but the lock would still have to be turned off explicitly in order > to prevent breaking Cython code by default. OK, in GIL-compatibility mode I can add a global lock for all Cython code (which would then be a global lock for all cdef data). This still won't be exactly equivalent to CPython, though; since IronPython has no GIL, I can't block pure-Python code from running simultaneously with Cython code. So Cython code that assumes (for example) that it holds a lock on all Python dictionaries (including module dictionaries), by virtue of holding the GIL, could break. This would be good enough for the "Fast, re-entrant, optimistic lock" you linked to, since it's manipulating cdef data (although it wouldn't be as fast any more, since it's adding an extra lock/unlock on every method call). Does your code assume that the Cython GIL locks Python data structures, or only cdef data structures? >> CPython could get its act together with a LLVM backend etc., but it >> would be nice strategically to have Mono available as a competing >> platform, in case CPython doesn't manage to improve matters. > > That would be one heck of a dependency, though. I don't know how big LLVM > is, but I would expect it to be a lot smaller than Mono. I was curious, so I did some checking. According to PEP 3146, Unladen Swallow adds a little over 10 megabytes to the size of the Python binary. When I add together the Installed-Size of the recursive mono dependencies for IronPython (not including IronPython itself) on my Debian system, I get 23 megabytes; but that includes implementations of the Windows Forms and a bunch of web stuff. If it were repackaged to split out the parts of IronPython that depend on those libraries, it looks like mono (including the JIT and the core libraries) would be more like 15 megabytes. Carl From stefan_ml at behnel.de Sun Aug 8 22:31:41 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Aug 2010 22:31:41 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> <4C5BD654.8080205@student.matnat.uio.no> <4C5E75CA.7080203@behnel.de> Message-ID: <4C5F142D.6030308@behnel.de> Carl Witty, 08.08.2010 22:11: > On Sun, Aug 8, 2010 at 2:15 AM, Stefan Behnel wrote: >> Remember that the GIL makes your life easier. Cython is about writing >> extension modules, and very, very many of those only work *because* of the >> GIL. Here's an example: >> >> http://code.activestate.com/recipes/577336-fast-re-entrant-optimistic-lock-implemented-in-cyt/ >> >> In the Cython wrappers I wrote so far, a *lot* of critical code truly >> relies on the GIL. >> >> So, one part of porting Cython to .NET would be to add a global lock for >> the code (at least when threading is used, which may be tricky to >> determine). Obviously, this could become a compiler option for the .NET >> backend, but the lock would still have to be turned off explicitly in order >> to prevent breaking Cython code by default. > > OK, in GIL-compatibility mode I can add a global lock for all Cython > code (which would then be a global lock for all cdef data). This > still won't be exactly equivalent to CPython, though; since IronPython > has no GIL, I can't block pure-Python code from running simultaneously > with Cython code. So Cython code that assumes (for example) that it > holds a lock on all Python dictionaries (including module > dictionaries), by virtue of holding the GIL, could break. Right. I think the requirement here is more like a lock for Cython code, but global to the system. While it is true that the GIL prevents the execution of /any/ Python code while an extension holds it, I would expect that the most important property is that it prevents other threads from executing parts of the extension itself. > This would be good enough for the "Fast, re-entrant, optimistic lock" > you linked to, since it's manipulating cdef data (although it wouldn't > be as fast any more, since it's adding an extra lock/unlock on every > method call). It would basically degrade into the normal RLock implementation, as a lock would have to be required on each request, independent of the low-level lock that the FastRLock class wraps internally. > Does your code assume that the Cython GIL locks Python data > structures, or only cdef data structures? It only assumes that no-one can enter the lock/unlock methods while they are being executed (unless they call into Python code). That's the main property that I keep using in my own code, and I think the main property of the GIL from the POV of most extensions. So it basically protects all C level structures from nondeterministic concurrency. >>> CPython could get its act together with a LLVM backend etc., but it >>> would be nice strategically to have Mono available as a competing >>> platform, in case CPython doesn't manage to improve matters. >> >> That would be one heck of a dependency, though. I don't know how big LLVM >> is, but I would expect it to be a lot smaller than Mono. > > I was curious, so I did some checking. According to PEP 3146, Unladen > Swallow adds a little over 10 megabytes to the size of the Python > binary. That's a lot more than I expected. Pretty huge in absolute numbers. > When I add together the Installed-Size of the recursive mono > dependencies for IronPython (not including IronPython itself) on my > Debian system, I get 23 megabytes; but that includes implementations > of the Windows Forms and a bunch of web stuff. If it were repackaged > to split out the parts of IronPython that depend on those libraries, > it looks like mono (including the JIT and the core libraries) would be > more like 15 megabytes. So it's basically in the same ball-park as LLVM. Stefan From sturla at molden.no Mon Aug 9 19:42:18 2010 From: sturla at molden.no (Sturla Molden) Date: Mon, 9 Aug 2010 19:42:18 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: <4C5E75CA.7080203@behnel.de> References: <4C5A6369.1000606@student.matnat.uio.no> <4C5BD654.8080205@student.matnat.uio.no> <4C5E75CA.7080203@behnel.de> Message-ID: <0c4a0f5d602750c068943a3942d34036.squirrel@webmail.uio.no> > Dag Sverre Seljebotn, 06.08.2010 11:31: > Remember that the GIL makes your life easier. Cython is about writing > extension modules, and very, very many of those only work *because* of the > GIL. One of the main virtues of CPython is that extension code (compiled C, C++, Fortran) is atomic with respect to the interpreter, unless the GIL is released. Sturla From cournape at gmail.com Wed Aug 11 06:56:54 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 11 Aug 2010 13:56:54 +0900 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Fri, Aug 6, 2010 at 1:55 PM, Carl Witty wrote: > For IronPython running under Microsoft's .NET implementation for > Windows, as far as I know there are no problems with C++/CLI. ?For > IronPython running under Mono on non-Windows platforms, C++/CLI > doesn't help at all; there are no implementations of C++/CLI on > non-Windows platforms and no effort toward making an implementation. > So the purpose of a C# backend would be to support Mono. I would think that it would also help for a jython backend, as there is no equivalent to C++/CLI in the java land AFAIK, and changing from C# to java back in cython should be much easier. For people who are not so familiar with C++/Cli, could you expand a bit more on the choice of C++/CLI vs C# as a first step ? The "extern" part sounds logical (a VM like the CLR or the JVM has strong strong ABI constraints), but the exception part not so much. Also, what does C++/Cli buys you instead of pure C or C++ (through p/invoke) ? CLI is gc aware, right ? I am also curious about the differences between C++/CLI vs p/invoke (the latter having an equivalent in the java), cheers, Davi From robertwb at math.washington.edu Wed Aug 11 07:58:49 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Aug 2010 22:58:49 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Tue, Aug 10, 2010 at 9:56 PM, David Cournapeau wrote: > On Fri, Aug 6, 2010 at 1:55 PM, Carl Witty wrote: > >> For IronPython running under Microsoft's .NET implementation for >> Windows, as far as I know there are no problems with C++/CLI. ?For >> IronPython running under Mono on non-Windows platforms, C++/CLI >> doesn't help at all; there are no implementations of C++/CLI on >> non-Windows platforms and no effort toward making an implementation. >> So the purpose of a C# backend would be to support Mono. > > I would think that it would also help for a jython backend, as there > is no equivalent to C++/CLI in the java land AFAIK, and changing from > C# to java back in cython should be much easier. > > For people who are not so familiar with C++/Cli, could you expand a > bit more on the choice of C++/CLI vs C# as a first step ? The "extern" > part sounds logical (a VM like the CLR or the JVM has strong strong > ABI constraints), but the exception part not so much. Also, what does > C++/Cli buys you instead of pure C or C++ (through p/invoke) ? CLI is > gc aware, right ? I am also curious about the differences between > C++/CLI vs p/invoke (the latter having an equivalent in the java), I'd imagine that the primary goal of using CLI is that IronPython objects live in the CLI environment, and the goal is to play nice with that. I'm not very familiar with all of this though, so would be curious to hear more details myself. - Robert From carl.witty at gmail.com Wed Aug 11 08:42:06 2010 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 10 Aug 2010 23:42:06 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Tue, Aug 10, 2010 at 9:56 PM, David Cournapeau wrote: > On Fri, Aug 6, 2010 at 1:55 PM, Carl Witty wrote: > >> For IronPython running under Microsoft's .NET implementation for >> Windows, as far as I know there are no problems with C++/CLI. ?For >> IronPython running under Mono on non-Windows platforms, C++/CLI >> doesn't help at all; there are no implementations of C++/CLI on >> non-Windows platforms and no effort toward making an implementation. >> So the purpose of a C# backend would be to support Mono. > > I would think that it would also help for a jython backend, as there > is no equivalent to C++/CLI in the java land AFAIK, and changing from > C# to java back in cython should be much easier. It certainly seems likely that a C# backend would be a much better starting point for a Jython port than a C++/CLI backend. (I'm not familiar enough with Jython internals or with the JVM->C interface to be more precise, though.) > For people who are not so familiar with C++/Cli, could you expand a > bit more on the choice of C++/CLI vs C# as a first step ? The "extern" > part sounds logical (a VM like the CLR or the JVM has strong strong > ABI constraints), but the exception part not so much. Sure. The issue with exceptions is almost trivial, but still important. According to the Mono P/Invoke documentation, P/Invoke is for talking to C functions. If a function throws a C++ exception, it wasn't a C function. And in particular, under Mono, if you P/Invoke a function that throws a C++ exception, very bad things happen; the exception can't be caught by managed code, and probably your application crashes. So the Cython construct: cdef extern from "some_file.h": cdef int foo() except + (which tells Cython to catch any C++ exceptions thrown by foo() and turn them into Python exceptions) can't be implemented only in managed code and P/Invoke; you need at least a little C++ wrapper for each such function that catches all exceptions and returns them another way. > Also, what does > C++/Cli buys you instead of pure C or C++ (through p/invoke) ? In a way, C++/CLI is like Cython. Cython is a sort of mixture of C and Python, and you can interoperate (almost) seamlessly with either language from Cython code. Similarly, C++/CLI is a mixture of C++ and the CLI, and you can interoperate (almost) seamlessly with either from C++/CLI code. This makes it a very nice target language for Cython: the C-like code compiles straight to C++, and the Python-like code compiles fairly nicely to the CLI side (with a lot of help from the IronPython implementation). For example, IronPython objects are garbage collected; you don't have to do anything special to support garbage collection (on CLI objects) in C++/CLI. The "raise" statement translates to "throw"; "try:/except:" translates to "try/catch". (Of course, having try:/except: translate to try/catch has a downside as well; it increases the differences between the CPython and IronPython backends of Cython, which makes it harder to share code between them.) > CLI is > gc aware, right ? I am also curious about the differences between > C++/CLI vs p/invoke (the latter having an equivalent in the java), I explained above why C++/CLI might be a particularly nice language if you needed to interface to (say) both C and C#. That's not really so important for a compiler, though... you can tolerate some ugliness in the output code, since that's all hidden inside the compiler. (Cython does fine generating C, even though C has no built-in Python support.) In my opinion, the major advantage of C++/CLI over C# with P/Invoke is that C++/CLI has a working C++ header file parser. This means that we can keep the current Cython semantics, where users describe to Cython the API (rather than the ABI) of the library they're trying to call; then Cython generates C, C++, or C++/CLI code that conforms to this API, and the compiler handles the rest. If the user did specify the ABI, then we could make Cython generate C# code with P/Invoke to wrap C functions. (Wrapping C++ with P/Invoke has multiple issues; the exception thing I already described, non-portable name mangling, non-portable object layout across compilers... I don't think that's ever likely to work.) That would be a lot of new code, though. With C++/CLI, we can generate the exact same code from the C-like portions of Cython that we do for C or C++. In another sense, C++/CLI adds no capabilities over C# with P/Invoke, because by default it compiles to the same bytecode language. In fact, you can use the free Reflector decompiler to decompile C++/CLI-generated bytecode into C# source. (I haven't tried recompiling the C#, but it looks approximately legal.) It's sort of interesting; you get a strange mixture of low-level pointer arithmetic code and normal high-level code. If you compile "errno", it decompiles to "*(_errno())". Accessing the .st_size member of a struct stat decompiles back to "*((int*) (&_stati + 20))". (I mentioned before that you couldn't deal with C++ exceptions using P/Invoke. That's true for Mono, but it's not an issue for C++/CLI, because Microsoft's C++ uses the same exception mechanism as the CLI.) Carl From stefan_ml at behnel.de Wed Aug 11 14:32:42 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Aug 2010 14:32:42 +0200 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 Message-ID: <4C62986A.4050709@behnel.de> Hi, Hudson reports two test failures since the patch for #561 was pushed: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py31-c/269/testReport/ Stefan From dalcinl at gmail.com Wed Aug 11 17:43:25 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Aug 2010 12:43:25 -0300 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: <4C62986A.4050709@behnel.de> References: <4C62986A.4050709@behnel.de> Message-ID: On 11 August 2010 09:32, Stefan Behnel wrote: > Hi, > > Hudson reports two test failures since the patch for #561 was pushed: > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py31-c/269/testReport/ > Well, I wrote that test, and I would be in favor of changing the test, unless there is an obvious fix in order to support such code (after all it is valid Python, right?) ... Anyway, special methods are, well, special :-) and I'm not sure it is worth to should support things like "__int__ = something" in a cdef class -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From carl.witty at gmail.com Wed Aug 11 23:49:31 2010 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Aug 2010 14:49:31 -0700 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On Wed, Aug 11, 2010 at 8:43 AM, Lisandro Dalcin wrote: > On 11 August 2010 09:32, Stefan Behnel wrote: >> Hi, >> >> Hudson reports two test failures since the patch for #561 was pushed: >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py31-c/269/testReport/ >> > > Well, I wrote that test, and I would be in favor of changing the test, > unless there is an obvious fix in order to support such code (after > all it is valid Python, right?) ... Anyway, special methods are, well, > special :-) and I'm not sure it is worth to should support things like > "__int__ = something" in a cdef class I'll write a patch that fixes these problems. For T287, should I fix the doctest or the #561 patch? Before #561, if a cdef class defined __long__ and not __int__, then they would get both __int__ and __long__ methods available from Python (where the __int__ wrapper was generated by Python and the __long__ wrapper was generated by Cython). After #561, this is still true for Python 2 (except that the __long__ wrapper is generated by Python), but for Python 3 only an __int__ method is available from Python. In favor of the current post-561 behavior: who cares if there's a __long__ method under Python 3 (that's not backed by a type slot)? It's very unlikely that anybody would try to call it anyway, since most values won't have __long__ methods. (Also, reverting this part of the patch goes back to the bad performance case on __long__ -- probably not too important, but I thought I'd mention it.) In favor of reverting: The user wrote a __long__ method; they should get a __long__ method. Opinions? Carl From dalcinl at gmail.com Thu Aug 12 00:08:05 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Aug 2010 19:08:05 -0300 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On 11 August 2010 18:49, Carl Witty wrote: > > In favor of reverting: The user wrote a __long__ method; they should > get a __long__ method. > Remember that __xxx__ symbols are reserved in Python, then I would say they are reserved in Cython. If a user writes a method called __xxx__ and Cython does something different, please do not complain. In short, I'm -1 in reverting. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From cournape at gmail.com Thu Aug 12 05:08:03 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 12 Aug 2010 12:08:03 +0900 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Wed, Aug 11, 2010 at 3:42 PM, Carl Witty wrote: > > Sure. ?The issue with exceptions is almost trivial, but still important. > > According to the Mono P/Invoke documentation, P/Invoke is for talking > to C functions. ?If a function throws a C++ exception, it wasn't a C > function. ?And in particular, under Mono, if you P/Invoke a function > that throws a C++ exception, very bad things happen; the exception > can't be caught by managed code, and probably your application > crashes. Ok. This issue is likely the same in the java world - running a C++ function without C++ runtime support cannot work, hence the need for a wrapper, like it is usually done when you want to dlopen a C++ library. > > In my opinion, the major advantage of C++/CLI over C# with P/Invoke is > that C++/CLI has a working C++ header file parser. ?This means that we > can keep the current Cython semantics, where users describe to Cython > the API (rather than the ABI) of the library they're trying to call; > then Cython generates C, C++, or C++/CLI code that conforms to this > API, and the compiler handles the rest. So just to be sure I understand, using C++/CLI gives you a way to generate the C++ wrapper needed to expose a C-like ABI to cython (and python runtime) ? A bit like python boost does ? It seems to me that having a "toolchain" to do this would also be useful for auto-generating .pxd from header files directly. May be enough to motivate me to continue the work I started there using clang It seems that fixing this would solve the issue for both java and C# output at the same time, which is a good point. Thanks a lot for the info, that's useful, and this is exciting development. David From carl.witty at gmail.com Thu Aug 12 07:11:36 2010 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Aug 2010 22:11:36 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Wed, Aug 11, 2010 at 8:08 PM, David Cournapeau wrote: > On Wed, Aug 11, 2010 at 3:42 PM, Carl Witty wrote: >> In my opinion, the major advantage of C++/CLI over C# with P/Invoke is >> that C++/CLI has a working C++ header file parser. ?This means that we >> can keep the current Cython semantics, where users describe to Cython >> the API (rather than the ABI) of the library they're trying to call; >> then Cython generates C, C++, or C++/CLI code that conforms to this >> API, and the compiler handles the rest. > > So just to be sure I understand, using C++/CLI gives you a way to > generate the C++ wrapper needed to expose a C-like ABI to cython (and > python runtime) ? A bit like python boost does ? Well, not exactly. Cython can already deal with a C++ API (better in the not-yet-released 0.13 than in previous versions); I would say that Cython itself is the tool that takes a C++ API and exposes a C ABI to the Python runtime. The advantage of C++/CLI is that it can directly consume C and C++ APIs (as expected, since it's basically a superset of C++). With C++/CLI there isn't really a wrapper. I identified two reasons for a wrapper; one is to translate C or C++ APIs into known ABIs, and the other is to catch and deal with C++ exceptions. With C++/CLI, neither is needed (just like we don't need a wrapper when generating C++ to target CPython). > It seems to me that having a "toolchain" to do this would also be > useful for auto-generating .pxd from header files directly. May be > enough to motivate me to continue the work I started there using clang > > It seems that fixing this would solve the issue for both java and C# > output at the same time, which is a good point. You're talking about a toolchain to generate C-ABI wrappers for arbitrary C++ code? That would indeed be a good place to start for JVM or C# Cython backends. (Although there would still be a lot of work involved; like I said, Cython currently targets C++ APIs directly, so there would be a lot of changes to make it target your wrappers instead.) Carl From strombrg at gmail.com Thu Aug 12 07:57:35 2010 From: strombrg at gmail.com (Dan Stromberg) Date: Wed, 11 Aug 2010 22:57:35 -0700 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: On Wed, Aug 11, 2010 at 10:11 PM, Carl Witty wrote: > On Wed, Aug 11, 2010 at 8:08 PM, David Cournapeau > wrote: > > On Wed, Aug 11, 2010 at 3:42 PM, Carl Witty > wrote: > >> In my opinion, the major advantage of C++/CLI over C# with P/Invoke is > >> that C++/CLI has a working C++ header file parser. This means that we > >> can keep the current Cython semantics, where users describe to Cython > >> the API (rather than the ABI) of the library they're trying to call; > >> then Cython generates C, C++, or C++/CLI code that conforms to this > >> API, and the compiler handles the rest. > > > > So just to be sure I understand, using C++/CLI gives you a way to > > generate the C++ wrapper needed to expose a C-like ABI to cython (and > > python runtime) ? A bit like python boost does ? > > Well, not exactly. Cython can already deal with a C++ API (better in > the not-yet-released 0.13 than in previous versions); I would say that > Cython itself is the tool that takes a C++ API and exposes a C ABI to > the Python runtime. The advantage of C++/CLI is that it can directly > consume C and C++ APIs (as expected, since it's basically a superset > of C++). With C++/CLI there isn't really a wrapper. I identified two > reasons for a wrapper; one is to translate C or C++ APIs into known > ABIs, and the other is to catch and deal with C++ exceptions. With > C++/CLI, neither is needed (just like we don't need a wrapper when > generating C++ to target CPython). > > > It seems to me that having a "toolchain" to do this would also be > > useful for auto-generating .pxd from header files directly. May be > > enough to motivate me to continue the work I started there using clang > > > > It seems that fixing this would solve the issue for both java and C# > > output at the same time, which is a good point. > > You're talking about a toolchain to generate C-ABI wrappers for > arbitrary C++ code? That would indeed be a good place to start for > JVM or C# Cython backends. (Although there would still be a lot of > work involved; like I said, Cython currently targets C++ APIs > directly, so there would be a lot of changes to make it target your > wrappers instead.) > > Carl > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > I don't know much about Cython internals, but if something could be done that would facilitate Cython interop with Jython, IronPython, Pypy, clpython, hotpy and maybe even pyjamas (perhaps not pyjamas because pyjamas is going to mostly be used for running things in a browser without a plugin), that would probably be a very good thing for Python in general. Pypy is probably the most noteworthy Python implementation that seems to be getting left out of the discussion so far (but I admit, I've not read the whole thread). One quick note about C++/CLI: IIRC, it's a bit like Cython itself. ISTR that C++/CLI can call C/C++ ABI and .Net ABI callables, much as Cython can call CPython, C and C++ callables. -- Dan Stromberg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100811/30acc893/attachment.htm From stefan_ml at behnel.de Thu Aug 12 08:06:26 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Aug 2010 08:06:26 +0200 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: <4C638F62.6030705@behnel.de> Carl Witty, 11.08.2010 23:49: > Before #561, if a cdef class defined __long__ and not __int__, then > they would get both __int__ and __long__ methods available from Python > (where the __int__ wrapper was generated by Python and the __long__ > wrapper was generated by Cython). > > After #561, this is still true for Python 2 (except that the __long__ > wrapper is generated by Python), but for Python 3 only an __int__ > method is available from Python. > > In favor of the current post-561 behavior: who cares if there's a > __long__ method under Python 3 (that's not backed by a type slot)? Py2's __long__ is the closest equivalent to __int__ in Py3, so I think the case that a user defines a __long__ method but no __int__ method should let Cython generate an __int__ method in Py3. BTW, this means we have to check the inheritance hierarchy to make sure there really isn't an __int__ method around that should be used instead, but AFAIR, this is done anyway. Stefan From stefan_ml at behnel.de Thu Aug 12 08:16:55 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Aug 2010 08:16:55 +0200 Subject: [Cython] working on Cython for IronPython In-Reply-To: References: <4C5A6369.1000606@student.matnat.uio.no> Message-ID: <4C6391D7.9090306@behnel.de> Dan Stromberg, 12.08.2010 07:57: > I don't know much about Cython internals, but if something could be done > that would facilitate Cython interop with Jython, IronPython, Pypy, > clpython, hotpy and maybe even pyjamas (perhaps not pyjamas because pyjamas > is going to mostly be used for running things in a browser without a > plugin), that would probably be a very good thing for Python in general. Yes, I think Cython would be the right tool to enable this. At least the platforms that have some kind of CPython C-API compatible layer should be relatively easy to target in a first run, and Cython could apply some adaptations in the C code to make them run better (e.g. avoid some special casing of builtins that proves to be hard to port or inefficient to emulate). > Pypy is probably the most noteworthy Python implementation that seems to be > getting left out of the discussion so far True. Although they do have a compatibility layer by now, I bet there is lots of stuff that could be done in the code that Cython generates to make extensions run faster/better/safer with PyPy. I'll take the heresy to ask on their mailing list if they think there's anything Cython can do. Stefan From robertwb at math.washington.edu Thu Aug 12 08:25:47 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Aug 2010 23:25:47 -0700 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On Wed, Aug 11, 2010 at 2:49 PM, Carl Witty wrote: > On Wed, Aug 11, 2010 at 8:43 AM, Lisandro Dalcin wrote: >> On 11 August 2010 09:32, Stefan Behnel wrote: >>> Hi, >>> >>> Hudson reports two test failures since the patch for #561 was pushed: >>> >>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py31-c/269/testReport/ >>> >> >> Well, I wrote that test, and I would be in favor of changing the test, >> unless there is an obvious fix in order to support such code (after >> all it is valid Python, right?) ... Anyway, special methods are, well, >> special :-) and I'm not sure it is worth to should support things like >> "__int__ = something" in a cdef class > > I'll write a patch that fixes these problems. ?For T287, should I fix > the doctest or the #561 patch? > > Before #561, if a cdef class defined __long__ and not __int__, then > they would get both __int__ and __long__ methods available from Python > (where the __int__ wrapper was generated by Python and the __long__ > wrapper was generated by Cython). > > After #561, this is still true for Python 2 (except that the __long__ > wrapper is generated by Python), but for Python 3 only an __int__ > method is available from Python. > > In favor of the current post-561 behavior: who cares if there's a > __long__ method under Python 3 (that's not backed by a type slot)? > It's very unlikely that anybody would try to call it anyway, since > most values won't have __long__ methods. ?(Also, reverting this part > of the patch goes back to the bad performance case on __long__ -- > probably not too important, but I thought I'd mention it.) > > In favor of reverting: The user wrote a __long__ method; they should > get a __long__ method. > > Opinions? I'm fine with the post-561 behavior as well, for cdef classes at least (and Python classes are handled by Python itself...) - Robert From dalcinl at gmail.com Thu Aug 12 16:00:49 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 12 Aug 2010 11:00:49 -0300 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: <4C638F62.6030705@behnel.de> References: <4C62986A.4050709@behnel.de> <4C638F62.6030705@behnel.de> Message-ID: On 12 August 2010 03:06, Stefan Behnel wrote: > > Py2's __long__ is the closest equivalent to __int__ in Py3, so I think the > case that a user defines a __long__ method but no __int__ method should let > Cython generate an __int__ method in Py3. > That's how things work. I've implemented such behavior long ago. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Thu Aug 12 16:10:53 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 12 Aug 2010 11:10:53 -0300 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On 11 August 2010 18:49, Carl Witty wrote: > > I'll write a patch that fixes these problems. > The testcase you added for #561 fails like this with Python 3: $ python3 runtests.py -T561 special_methods_T561.c:5000:2: error: #error __setslice__ and __delslice__ not supported in Python 3. special_methods_T561.c:5182:2: error: #error __setslice__ and __delslice__ not supported in Python 3. special_methods_T561.c:5364:2: error: #error __setslice__ and __delslice__ not supported in Python 3. I would say that you should just comment out the tests involving these methods. We could try to support them at some point in Python 3, but I'm not particularly interested in contributing the patch ;-) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Thu Aug 12 16:13:02 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Aug 2010 16:13:02 +0200 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> <4C638F62.6030705@behnel.de> Message-ID: <4C64016E.1040005@behnel.de> Lisandro Dalcin, 12.08.2010 16:00: > On 12 August 2010 03:06, Stefan Behnel wrote: >> >> Py2's __long__ is the closest equivalent to __int__ in Py3, so I think the >> case that a user defines a __long__ method but no __int__ method should let >> Cython generate an __int__ method in Py3. > > That's how things work. I've implemented such behavior long ago. Well, that's how it used to be, at least, and I think that's how it should work in the future. Stefan From stefan_ml at behnel.de Thu Aug 12 16:16:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Aug 2010 16:16:56 +0200 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: <4C640258.1060909@behnel.de> Lisandro Dalcin, 12.08.2010 16:10: > On 11 August 2010 18:49, Carl Witty wrote: >> >> I'll write a patch that fixes these problems. >> > > The testcase you added for #561 fails like this with Python 3: > > $ python3 runtests.py -T561 > > special_methods_T561.c:5000:2: error: #error __setslice__ and > __delslice__ not supported in Python 3. > special_methods_T561.c:5182:2: error: #error __setslice__ and > __delslice__ not supported in Python 3. > special_methods_T561.c:5364:2: error: #error __setslice__ and > __delslice__ not supported in Python 3. > > > I would say that you should just comment out the tests involving these > methods. We could try to support them at some point in Python 3, but > I'm not particularly interested in contributing the patch ;-) IIRC, the decision was to not do anything about these methods. That's what the above error comes from. Their use has been discouraged since long before Python 3 was on the horizon, so if users implement them, it's their fault. Stefan From carl.witty at gmail.com Thu Aug 12 17:51:23 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 12 Aug 2010 08:51:23 -0700 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On Thu, Aug 12, 2010 at 7:10 AM, Lisandro Dalcin wrote: > On 11 August 2010 18:49, Carl Witty wrote: >> >> I'll write a patch that fixes these problems. >> > > The testcase you added for #561 fails like this with Python 3: > > $ python3 runtests.py -T561 > > special_methods_T561.c:5000:2: error: #error __setslice__ and > __delslice__ not supported in Python 3. > special_methods_T561.c:5182:2: error: #error __setslice__ and > __delslice__ not supported in Python 3. > special_methods_T561.c:5364:2: error: #error __setslice__ and > __delslice__ not supported in Python 3. > > > I would say that you should just comment out the tests involving these > methods. We could try to support them at some point in Python 3, but > I'm not particularly interested in contributing the patch ;-) Yes, that's what I started to do; but then I hit #566 and decided to fix it first. Carl From dalcinl at gmail.com Thu Aug 12 18:14:32 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 12 Aug 2010 13:14:32 -0300 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: <4C640258.1060909@behnel.de> References: <4C62986A.4050709@behnel.de> <4C640258.1060909@behnel.de> Message-ID: On 12 August 2010 11:16, Stefan Behnel wrote: > Lisandro Dalcin, 12.08.2010 16:10: >> On 11 August 2010 18:49, Carl Witty wrote: >>> >>> I'll write a patch that fixes these problems. >>> >> >> The testcase you added for #561 fails like this with Python 3: >> >> $ python3 runtests.py -T561 >> >> special_methods_T561.c:5000:2: error: #error __setslice__ and >> __delslice__ not supported in Python 3. >> special_methods_T561.c:5182:2: error: #error __setslice__ and >> __delslice__ not supported in Python 3. >> special_methods_T561.c:5364:2: error: #error __setslice__ and >> __delslice__ not supported in Python 3. >> >> >> I would say that you should just comment out the tests involving these >> methods. We could try to support them at some point in Python 3, but >> I'm not particularly interested in contributing the patch ;-) > > IIRC, the decision was to not do anything about these methods. That's what > the above error comes from. Their use has been discouraged since long > before Python 3 was on the horizon, so if users implement them, it's their > fault. > Perhaps I was not clear enough. I'm just asking Carl to comment out the test involing these special methods... I do not want the testsuite failing under Python 3. Perhaps these tests could be moved to other test file, that only get tested under a Python 2 runtime. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Thu Aug 12 18:42:15 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Aug 2010 09:42:15 -0700 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On Thu, Aug 12, 2010 at 8:51 AM, Carl Witty wrote: > On Thu, Aug 12, 2010 at 7:10 AM, Lisandro Dalcin wrote: >> On 11 August 2010 18:49, Carl Witty wrote: >>> >>> I'll write a patch that fixes these problems. >>> >> >> The testcase you added for #561 fails like this with Python 3: >> >> $ python3 runtests.py -T561 >> >> special_methods_T561.c:5000:2: error: #error __setslice__ and >> __delslice__ not supported in Python 3. >> special_methods_T561.c:5182:2: error: #error __setslice__ and >> __delslice__ not supported in Python 3. >> special_methods_T561.c:5364:2: error: #error __setslice__ and >> __delslice__ not supported in Python 3. >> >> >> I would say that you should just comment out the tests involving these >> methods. We could try to support them at some point in Python 3, but >> I'm not particularly interested in contributing the patch ;-) > > Yes, that's what I started to do; but then I hit #566 and decided to > fix it first. Thanks. I merged your fix for #566 and commented out the __XXXslice__ methods (until we have a way for testing them only on Py2). - Robert From dalcinl at gmail.com Thu Aug 12 20:08:55 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 12 Aug 2010 15:08:55 -0300 Subject: [Cython] unicodeliterals failures Message-ID: I've mailed Stefan about this some time ago. I can reproduce this problem with - system Python 2.6.2 from Fedora 12 in Linux 32 bits - system Python 2.6.2 from Fedora 12 in Linux 64 bits - system Python 2.6.4 from Fedora 13 in Linux 32 bits but not with: - system Python 3.1.2 from Fedora 13 in Linux 32 bits In all cases, these system Pythons are wide unicode builds. Additionally, I DO NOT get the failure with custom, debug, short unicode Python builds from 2.3 to 3.2 on Linux 32. Any clue? Could this issue be very specific to Py2.6 + wide-unicode? ====================================================================== FAIL: Doctest: unicodeliterals ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/doctest.py", line 2145, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for unicodeliterals File "/u/dalcinl/Devel/Cython/cython-devel/BUILD/run/c/unicodeliterals.so", line 339, in unicodeliterals ---------------------------------------------------------------------- File "/u/dalcinl/Devel/Cython/cython-devel/BUILD/run/c/unicodeliterals.so", line 408, in unicodeliterals Failed example: if wide_literal == expected: print(True) else: print(repr(wide_literal), repr(expected), sys.maxunicode) Expected: True Got: ("u'\\U00101234'", "u'\\udbc4\\ude34'", 1114111) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From carl.witty at gmail.com Thu Aug 12 20:10:51 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 12 Aug 2010 11:10:51 -0700 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On Thu, Aug 12, 2010 at 9:42 AM, Robert Bradshaw wrote: > Thanks. I merged your fix for #566 and commented out the __XXXslice__ > methods (until we have a way for testing them only on Py2). I posted a new patch on #561. You'll have to undo your commenting-out of __XXXslice__ to apply the patch. This patch does not change behavior, only doctests. In particular, after #561, under Python 3, any __cmp__, __div__, __idiv__, __oct__, or __hex__ methods will be discarded, and defining a __nonzero__ method will give you a __bool__ method instead; the new patch doctests all these behaviors. Also, I couldn't figure out how to modify the IntLongC test in type_slots_int_long_T287.pyx to work under Python 3, so I just removed it. Lisandro, is that OK? (All the other tests are still there.) Carl From carl.witty at gmail.com Thu Aug 12 20:15:42 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 12 Aug 2010 11:15:42 -0700 Subject: [Cython] unicodeliterals failures In-Reply-To: References: Message-ID: On Thu, Aug 12, 2010 at 11:08 AM, Lisandro Dalcin wrote: > I've mailed Stefan about this some time ago. > > I can reproduce this problem with > > - system Python 2.6.2 from Fedora 12 in Linux 32 bits > - system Python 2.6.2 from Fedora 12 in Linux 64 bits > - system Python 2.6.4 from Fedora 13 in Linux 32 bits > > but not with: > > - system Python 3.1.2 from Fedora 13 in Linux 32 bits > > In all cases, these system Pythons are wide unicode builds. > > Additionally, I DO NOT get the failure with custom, debug, short > unicode Python builds from 2.3 to 3.2 on Linux 32. > > Any clue? Could this issue be very specific to Py2.6 + wide-unicode? > > > ====================================================================== > FAIL: Doctest: unicodeliterals I just ran the full test suite with system Python 2.6.5 (which is wide unicode) from Debian testing in Linux 64 bits with no failures. Carl From carl.witty at gmail.com Thu Aug 12 20:19:37 2010 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 12 Aug 2010 11:19:37 -0700 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On Thu, Aug 12, 2010 at 11:10 AM, Carl Witty wrote: > On Thu, Aug 12, 2010 at 9:42 AM, Robert Bradshaw > wrote: >> Thanks. I merged your fix for #566 and commented out the __XXXslice__ >> methods (until we have a way for testing them only on Py2). > > I posted a new patch on #561. ?You'll have to undo your commenting-out > of __XXXslice__ to apply the patch. I should have mentioned: I ran the full test suite after this patch on Python 2.6 and Python 3.1. There were no failures with 2.6; under 3.1 there were 6 gcc compilation failures because it couldn't find PyBoolObject. I don't think this has anything to do with #561; I'm guessing it's due to Robert's recent changes. For the record, the 6 failures are: bufaccess cdef_bool_T227 cython_includes isinstance r_pythonapi type_inference Carl From dalcinl at gmail.com Thu Aug 12 20:29:55 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 12 Aug 2010 15:29:55 -0300 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On 12 August 2010 15:10, Carl Witty wrote: > > Also, I couldn't figure out how to modify the IntLongC test in > type_slots_int_long_T287.pyx to work under Python 3, so I just removed > it. ?Lisandro, is that OK? ?(All the other tests are still there.) > It's OK, remove it. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Thu Aug 12 21:12:36 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Aug 2010 12:12:36 -0700 Subject: [Cython] patch for ticket #561 broke the __long__ slot in Py3 In-Reply-To: References: <4C62986A.4050709@behnel.de> Message-ID: On Thu, Aug 12, 2010 at 11:19 AM, Carl Witty wrote: > On Thu, Aug 12, 2010 at 11:10 AM, Carl Witty wrote: >> On Thu, Aug 12, 2010 at 9:42 AM, Robert Bradshaw >> wrote: >>> Thanks. I merged your fix for #566 and commented out the __XXXslice__ >>> methods (until we have a way for testing them only on Py2). >> >> I posted a new patch on #561. ?You'll have to undo your commenting-out >> of __XXXslice__ to apply the patch. > > I should have mentioned: I ran the full test suite after this patch on > Python 2.6 and Python 3.1. ?There were no failures with 2.6; under 3.1 > there were 6 gcc compilation failures because it couldn't find > PyBoolObject. ?I don't think this has anything to do with #561; I'm > guessing it's due to Robert's recent changes. Yep, I recently pushed a fix for that. - Robert From craigcitro at gmail.com Fri Aug 13 10:04:40 2010 From: craigcitro at gmail.com (Craig Citro) Date: Fri, 13 Aug 2010 01:04:40 -0700 Subject: [Cython] Cython 0.13.beta1 Message-ID: Hi all, So here's a second go at Cython 0.13 beta. I'm posting it to the usual place, namely: http://www.cython.org/release/Cython-0.13.beta1.tar.gz This definitely still isn't ready to go out the door, hence the "beta" instead of "rc." A few notes: * We've been running into an interesting problem with rare (~0.05% of the time) segfaults on Sage startup. Interestingly, this has apparently been around a while -- Robert and I have run a handful of tests, and this behavior was already there with Cython-0.12.1. (We didn't try to bisect at all, we just wanted to confirm that it wasn't completely new.) Our not-completely-scientific analysis suggests that the rate of segfaults hasn't changed since the last Cython release, so our current plan is to get this release (finally!) out the door, and then start trying to diagnose this further. There are some questions about whether or not this might be hardware-related, but given how rare the failures are, I don't think we've really run the tests anywhere enough to say we're *not* seeing the same problem elsewhere. * Since it seems like every third email to cython-users has confusion about bool being a C++ vs. Python type as its root cause, Robert's gone ahead and made bool an invalid type identifier. (This is already pushed, so this might not be news.) The plan is to simply make the types explicit for now (which Robert has already done, see cimport cpython.bool and libcpp.bool) and decide later whether or not bool should be a valid identifier by itself. I'm going to add a FAQ entry about this, but I'm going to do it when I'm more coherent (because writing FAQ entries takes more concentration than cutting a beta?). Things we need to fix before release: * There are a handful of unicode-related doctest failures that Lisandro ran into in another thread. I'm guessing Stefan will know what's going on here faster than anyone else? ;) * We need to clean up the last of the fallout from #561 -- if I'm reading the corresponding thread correctly, Carl's got a fix for that we need to push. If so, Carl, feel free to push that. (My machine is currently rebuilding Sage, so I can't test this myself.) Is there anything else that anyone can think of that's blocking us on 0.13? If not, my plan is to cut an rc as soon as we get all the above failures passing, and plan on releasing a few days after that. If anyone has any other thoughts, now's the time. -cc From carl.witty at gmail.com Fri Aug 13 17:33:51 2010 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Aug 2010 08:33:51 -0700 Subject: [Cython] Cython 0.13.beta1 In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 1:04 AM, Craig Citro wrote: > ?* We need to clean up the last of the fallout from #561 -- if I'm > reading the corresponding thread correctly, Carl's got a fix for that > we need to push. If so, Carl, feel free to push that. (My machine is > currently rebuilding Sage, so I can't test this myself.) As far as I know, I don't have write access to the Cython repositories. Carl From robertwb at math.washington.edu Fri Aug 13 17:38:27 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Aug 2010 08:38:27 -0700 Subject: [Cython] Cython 0.13.beta1 In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 8:33 AM, Carl Witty wrote: > On Fri, Aug 13, 2010 at 1:04 AM, Craig Citro wrote: >> ?* We need to clean up the last of the fallout from #561 -- if I'm >> reading the corresponding thread correctly, Carl's got a fix for that >> we need to push. If so, Carl, feel free to push that. (My machine is >> currently rebuilding Sage, so I can't test this myself.) > > As far as I know, I don't have write access to the Cython repositories. You do now--just use your trac username/password. - Robert From craigcitro at gmail.com Fri Aug 13 17:45:58 2010 From: craigcitro at gmail.com (Craig Citro) Date: Fri, 13 Aug 2010 08:45:58 -0700 Subject: [Cython] Cython 0.13.beta1 In-Reply-To: References: Message-ID: > As far as I know, I don't have write access to the Cython repositories. > E-mail me an .htpasswd file and I'll fix that. ;) -cc From carl.witty at gmail.com Fri Aug 13 23:03:35 2010 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Aug 2010 14:03:35 -0700 Subject: [Cython] Cython 0.13.beta1 In-Reply-To: References: Message-ID: On Fri, Aug 13, 2010 at 8:38 AM, Robert Bradshaw wrote: > On Fri, Aug 13, 2010 at 8:33 AM, Carl Witty wrote: >> On Fri, Aug 13, 2010 at 1:04 AM, Craig Citro wrote: >>> ?* We need to clean up the last of the fallout from #561 -- if I'm >>> reading the corresponding thread correctly, Carl's got a fix for that >>> we need to push. If so, Carl, feel free to push that. (My machine is >>> currently rebuilding Sage, so I can't test this myself.) >> >> As far as I know, I don't have write access to the Cython repositories. > > You do now--just use your trac username/password. Thanks! OK, I pushed my patch. Carl From dalcinl at gmail.com Sat Aug 14 21:52:48 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 14 Aug 2010 16:52:48 -0300 Subject: [Cython] 2to3 verbosity Message-ID: Any way the reduce the 2to3 verbosity when running the testsuite with Python 3? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Sat Aug 14 22:30:53 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 14 Aug 2010 17:30:53 -0300 Subject: [Cython] unicode testcase failure in Windows Message-ID: I guess this is a bug in the test code... ====================================================================== FAIL: Doctest: inop.__test__.m_wide_unicode_literal (line 206) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\Python31\lib\doctest.py", line 2104, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for inop.__test__.m_wide_unicode_literal (li ne 206) File "C:\Users\dalcinl\DEVEL\cython-devel\BUILD\run\c\inop.pyd", line unknown line number, in m_wide_unicode_literal (line 206) ---------------------------------------------------------------------- File "C:\Users\dalcinl\DEVEL\cython-devel\BUILD\run\c\inop.pyd", line ?, in inop .__test__.m_wide_unicode_literal (line 206) Failed example: if sys.maxunicode == 65535: m_wide_unicode_literal(py_wide_unicode_character_surrogate1) m_wide_unicode_literal(py_wide_unicode_character_surrogate2) else: m_wide_unicode_literal(ord(py_wide_unicode_character)) 1 Exception raised: Traceback (most recent call last): File "c:\Python31\lib\doctest.py", line 1244, in __run compileflags, 1), test.globs) File "", line 2, in NameError: name 'py_wide_unicode_character_surrogate1' is not defined -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Aug 13 15:19:38 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Aug 2010 15:19:38 +0200 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: Message-ID: <4C65466A.8050602@behnel.de> Robert Bradshaw, 27.07.2010 21:43: > On Tue, Jul 27, 2010 at 12:33 PM, Carl Witty wrote: >> On Tue, Jul 27, 2010 at 2:07 AM, Robert Bradshaw wrote: >>> Very, very cool. I've actually been thinking a lot about this myself >>> (given the recent thread in sage-devel). I think one reason we did our >>> own wrappers was for introspection/docstrings--does that break? Did >>> you fix the most egregious __getattr__? >> >> Yes, this does break docstrings on these functions. I think this >> could be fixed; for instance, by creating our own statically-declared >> wrapperbase/slotdef structs that have the right docstring, and >> creating our own wrapper objects using these structs. I could try to >> work on this, but probably not soon enough for 0.13. > > Well, for special methods, I consider this kind of performance > degradation to be a more important issue than introspection. I just compiled lxml with the current cython-devel and the resulting module failed during initialisation when it tried to retrieve the docstring for a "__getattr__" method when building the automatically generated __test__ dict. Seeing that I used this, I'm pretty sure others did, too. So we will clearly break user code with this change. So, I'm -1 on applying the change for 0.13 without keeping the functionality compatible with what we had before. This is a pure optimisation, so it must not break code. Stefan From stefan_ml at behnel.de Sun Aug 15 14:06:00 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Aug 2010 14:06:00 +0200 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C65466A.8050602@behnel.de> References: <4C65466A.8050602@behnel.de> Message-ID: <4C67D828.7090206@behnel.de> Stefan Behnel, 13.08.2010 15:19: > Robert Bradshaw, 27.07.2010 21:43: >> On Tue, Jul 27, 2010 at 12:33 PM, Carl Witty wrote: >>> On Tue, Jul 27, 2010 at 2:07 AM, Robert Bradshaw wrote: >>>> Very, very cool. I've actually been thinking a lot about this myself >>>> (given the recent thread in sage-devel). I think one reason we did our >>>> own wrappers was for introspection/docstrings--does that break? Did >>>> you fix the most egregious __getattr__? >>> >>> Yes, this does break docstrings on these functions. I think this >>> could be fixed; for instance, by creating our own statically-declared >>> wrapperbase/slotdef structs that have the right docstring, and >>> creating our own wrapper objects using these structs. I could try to >>> work on this, but probably not soon enough for 0.13. >> >> Well, for special methods, I consider this kind of performance >> degradation to be a more important issue than introspection. > > I just compiled lxml with the current cython-devel and the resulting module > failed during initialisation when it tried to retrieve the docstring for a > "__getattr__" method when building the automatically generated __test__ > dict. Seeing that I used this, I'm pretty sure others did, too. So we will > clearly break user code with this change. > > So, I'm -1 on applying the change for 0.13 without keeping the > functionality compatible with what we had before. This is a pure > optimisation, so it must not break code. I saw that Robert worked on this but ISTM that __getattr__ is more special than other special methods in some way. I still get the above error. I pushed an update to the related test to make it fail. Stefan From stefan_ml at behnel.de Sun Aug 15 14:54:25 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Aug 2010 14:54:25 +0200 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C67D828.7090206@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> Message-ID: <4C67E381.8060309@behnel.de> Stefan Behnel, 15.08.2010 14:06: > Stefan Behnel, 13.08.2010 15:19: >> Robert Bradshaw, 27.07.2010 21:43: >>> On Tue, Jul 27, 2010 at 12:33 PM, Carl Witty wrote: >>>> On Tue, Jul 27, 2010 at 2:07 AM, Robert Bradshaw wrote: >>>>> Very, very cool. I've actually been thinking a lot about this myself >>>>> (given the recent thread in sage-devel). I think one reason we did our >>>>> own wrappers was for introspection/docstrings--does that break? Did >>>>> you fix the most egregious __getattr__? >>>> >>>> Yes, this does break docstrings on these functions. I think this >>>> could be fixed; for instance, by creating our own statically-declared >>>> wrapperbase/slotdef structs that have the right docstring, and >>>> creating our own wrapper objects using these structs. I could try to >>>> work on this, but probably not soon enough for 0.13. >>> >>> Well, for special methods, I consider this kind of performance >>> degradation to be a more important issue than introspection. >> >> I just compiled lxml with the current cython-devel and the resulting module >> failed during initialisation when it tried to retrieve the docstring for a >> "__getattr__" method when building the automatically generated __test__ >> dict. Seeing that I used this, I'm pretty sure others did, too. So we will >> clearly break user code with this change. >> >> So, I'm -1 on applying the change for 0.13 without keeping the >> functionality compatible with what we had before. This is a pure >> optimisation, so it must not break code. > > I saw that Robert worked on this but ISTM that __getattr__ is more special > than other special methods in some way. I still get the above error. I > pushed an update to the related test to make it fail. Ah, got it. This is just what Carl wrote in the initial bug report. The new setup fails to provide a __getattr__ method and implements a __getattribute__ method instead. But the problem is not only that the introspection says it's not there, the problem is that the method really is not there. So we break user code that tries to use __getattr__ as an unbound method *and* the code generated by Cython for setting and retrieving docstrings of this method, thus breaking the module initialisation. So it's a double regression that is clearly user visible. Sounds like a release blocker to me. Stefan From stefan_ml at behnel.de Sun Aug 15 20:38:29 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Aug 2010 20:38:29 +0200 Subject: [Cython] 2to3 verbosity In-Reply-To: References: Message-ID: <4C683425.5040405@behnel.de> Lisandro Dalcin, 14.08.2010 21:52: > Any way the reduce the 2to3 verbosity when running the testsuite with Python 3? http://hg.cython.org/cython-devel/rev/61f777ed9ae9 Stefan From stefan_ml at behnel.de Sun Aug 15 20:51:38 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Aug 2010 20:51:38 +0200 Subject: [Cython] unicodeliterals failures In-Reply-To: References: Message-ID: <4C68373A.1010805@behnel.de> Lisandro Dalcin, 12.08.2010 20:08: > I've mailed Stefan about this some time ago. > > I can reproduce this problem with > > - system Python 2.6.2 from Fedora 12 in Linux 32 bits > - system Python 2.6.2 from Fedora 12 in Linux 64 bits > - system Python 2.6.4 from Fedora 13 in Linux 32 bits > > but not with: > > - system Python 3.1.2 from Fedora 13 in Linux 32 bits > > In all cases, these system Pythons are wide unicode builds. > > Additionally, I DO NOT get the failure with custom, debug, short > unicode Python builds from 2.3 to 3.2 on Linux 32. > > Any clue? Could this issue be very specific to Py2.6 + wide-unicode? > > > ====================================================================== > FAIL: Doctest: unicodeliterals > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.6/doctest.py", line 2145, in runTest > raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for unicodeliterals > File "/u/dalcinl/Devel/Cython/cython-devel/BUILD/run/c/unicodeliterals.so", > line 339, in unicodeliterals > > ---------------------------------------------------------------------- > File "/u/dalcinl/Devel/Cython/cython-devel/BUILD/run/c/unicodeliterals.so", > line 408, in unicodeliterals > Failed example: > if wide_literal == expected: print(True) > else: print(repr(wide_literal), repr(expected), sys.maxunicode) > Expected: > True > Got: > ("u'\\U00101234'", "u'\\udbc4\\ude34'", 1114111) It shouldn't expect the surrogate pair but the single character string that it actually gets, so it's the "expected" string that's broken here. Could you check if the generated C code is the same on all systems? What does the respective docstring part look like in the C code? Note that it is appended to the string in the test code, I only need the short snippet above. Stefan From stefan_ml at behnel.de Sun Aug 15 21:21:50 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Aug 2010 21:21:50 +0200 Subject: [Cython] unicodeliterals failures In-Reply-To: References: Message-ID: <4C683E4E.7080503@behnel.de> Lisandro Dalcin, 12.08.2010 20:08: > I've mailed Stefan about this some time ago. > > I can reproduce this problem with > > - system Python 2.6.2 from Fedora 12 in Linux 32 bits > - system Python 2.6.2 from Fedora 12 in Linux 64 bits > - system Python 2.6.4 from Fedora 13 in Linux 32 bits > > but not with: > > - system Python 3.1.2 from Fedora 13 in Linux 32 bits > > In all cases, these system Pythons are wide unicode builds. > ---------------------------------------------------------------------- > File "/u/dalcinl/Devel/Cython/cython-devel/BUILD/run/c/unicodeliterals.so", > line 408, in unicodeliterals > Failed example: > if wide_literal == expected: print(True) > else: print(repr(wide_literal), repr(expected), sys.maxunicode) > Expected: > True > Got: > ("u'\\U00101234'", "u'\\udbc4\\ude34'", 1114111) I can actually reproduce this with Python 2.6.4 on my system but not with Python 2.6.5. Both are wide unicode builds. So it looks like something has changed between those two versions. Given that the test already special cases releases before 2.6, I think the easiest thing to do is to simply move on to 2.6.5 and not run the test in versions older than that. Wide literals are already known to work at this point, as the "wide_literal" value shows. http://hg.cython.org/cython-devel/rev/0faea530ce40 Stefan From dalcinl at gmail.com Mon Aug 16 19:20:01 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 16 Aug 2010 14:20:01 -0300 Subject: [Cython] unicodeliterals failures In-Reply-To: <4C683E4E.7080503@behnel.de> References: <4C683E4E.7080503@behnel.de> Message-ID: On 15 August 2010 16:21, Stefan Behnel wrote: > Lisandro Dalcin, 12.08.2010 20:08: >> I've mailed Stefan about this some time ago. >> >> I can reproduce this problem with >> >> - system Python 2.6.2 from Fedora 12 in Linux 32 bits >> - system Python 2.6.2 from Fedora 12 in Linux 64 bits >> - system Python 2.6.4 from Fedora 13 in Linux 32 bits >> >> but not with: >> >> - system Python 3.1.2 from Fedora 13 in Linux 32 bits >> >> In all cases, these system Pythons are wide unicode builds. >> ---------------------------------------------------------------------- >> File "/u/dalcinl/Devel/Cython/cython-devel/BUILD/run/c/unicodeliterals.so", >> line 408, in unicodeliterals >> Failed example: >> ? ? ?if wide_literal == expected: print(True) >> ? ? ?else: print(repr(wide_literal), repr(expected), sys.maxunicode) >> Expected: >> ? ? ?True >> Got: >> ? ? ?("u'\\U00101234'", "u'\\udbc4\\ude34'", 1114111) > > I can actually reproduce this with Python 2.6.4 on my system but not with > Python 2.6.5. Both are wide unicode builds. So it looks like something has > changed between those two versions. > > Given that the test already special cases releases before 2.6, I think the > easiest thing to do is to simply move on to 2.6.5 and not run the test in > versions older than that. Wide literals are already known to work at this > point, as the "wide_literal" value shows. > > http://hg.cython.org/cython-devel/rev/0faea530ce40 > +1. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Mon Aug 16 23:20:04 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 16 Aug 2010 18:20:04 -0300 Subject: [Cython] cython-release or cython-devel? Message-ID: I pushed a couple of (trivial) fixes in cython-devel that should go for the next release... However, just noticed the cython-release repo... What should I do? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Mon Aug 16 23:27:42 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Aug 2010 14:27:42 -0700 Subject: [Cython] cython-release or cython-devel? In-Reply-To: References: Message-ID: On Mon, Aug 16, 2010 at 2:20 PM, Lisandro Dalcin wrote: > I pushed a couple of (trivial) fixes in cython-devel that should go > for the next release... However, just noticed the cython-release > repo... What should I do? Go ahead and push them to cython-release. Mostly I created that branch because I didn't know how long it was going to take to sort out the Sage issues, but given that nothing big went in in the meantime it didn't turn out to be necessary. - Robert From dalcinl at gmail.com Mon Aug 16 23:53:19 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 16 Aug 2010 18:53:19 -0300 Subject: [Cython] cython-release or cython-devel? In-Reply-To: References: Message-ID: On 16 August 2010 18:27, Robert Bradshaw wrote: > On Mon, Aug 16, 2010 at 2:20 PM, Lisandro Dalcin wrote: >> I pushed a couple of (trivial) fixes in cython-devel that should go >> for the next release... However, just noticed the cython-release >> repo... What should I do? > > Go ahead and push them to cython-release. Now cython-devel and cython-release should be in sync... > Mostly I created that branch > because I didn't know how long it was going to take to sort out the > Sage issues, but given that nothing big went in in the meantime it > didn't turn out to be necessary. > Perhaps you should remove cython-release? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From craigcitro at gmail.com Tue Aug 17 01:38:11 2010 From: craigcitro at gmail.com (Craig Citro) Date: Mon, 16 Aug 2010 16:38:11 -0700 Subject: [Cython] cython-release or cython-devel? In-Reply-To: References: Message-ID: > Perhaps you should remove cython-release? > Actually, the plan is to use it tonight to generate an rc. At that point, we'll actually want the separation -- new stuff should go in devel, and release should stay nice and working. ;) I think it makes sense to have a branch that basically only gets used around release-time ... -cc From stefan_ml at behnel.de Tue Aug 17 08:02:17 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 Aug 2010 08:02:17 +0200 Subject: [Cython] cython-release or cython-devel? In-Reply-To: References: Message-ID: <4C6A25E9.8030200@behnel.de> Craig Citro, 17.08.2010 01:38: >> Perhaps you should remove cython-release? > > Actually, the plan is to use it tonight to generate an rc. At that > point, we'll actually want the separation -- new stuff should go in > devel, and release should stay nice and working. ;) I think it makes > sense to have a branch that basically only gets used around > release-time ... In that case, cython-prerelease would be a better name. Stefan From robertwb at math.washington.edu Tue Aug 17 12:44:54 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Aug 2010 03:44:54 -0700 Subject: [Cython] cython-release or cython-devel? In-Reply-To: <4C6A25E9.8030200@behnel.de> References: <4C6A25E9.8030200@behnel.de> Message-ID: On Mon, Aug 16, 2010 at 11:02 PM, Stefan Behnel wrote: > Craig Citro, 17.08.2010 01:38: >>> Perhaps you should remove cython-release? >> >> Actually, the plan is to use it tonight to generate an rc. At that >> point, we'll actually want the separation -- new stuff should go in >> devel, and release should stay nice and working. ;) I think it makes >> sense to have a branch that basically only gets used around >> release-time ... > > In that case, cython-prerelease would be a better name. Yes, I agree. Fixed. - Robert From craigcitro at gmail.com Wed Aug 18 02:12:34 2010 From: craigcitro at gmail.com (Craig Citro) Date: Tue, 17 Aug 2010 17:12:34 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C67E381.8060309@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: Hi Stefan, > Ah, got it. This is just what Carl wrote in the initial bug report. The new > setup fails to provide a __getattr__ method and implements a > __getattribute__ method instead. But the problem is not only that the > introspection says it's not there, the problem is that the method really is > not there. So we break user code that tries to use __getattr__ as an > unbound method *and* the code generated by Cython for setting and > retrieving docstrings of this method, thus breaking the module > initialisation. So it's a double regression that is clearly user visible. > > Sounds like a release blocker to me. > Robert and I were chatting about this earlier -- did the changes you pushed completely fix this problem, or just cover it up? We couldn't come up with a case that was still a problem off the cuff, but the commit message said something like "for now," which made us think that there might be more to the story. ;) -cc From carl.witty at gmail.com Wed Aug 18 02:35:45 2010 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 17 Aug 2010 17:35:45 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 5:12 PM, Craig Citro wrote: > Hi Stefan, > >> Ah, got it. This is just what Carl wrote in the initial bug report. The new >> setup fails to provide a __getattr__ method and implements a >> __getattribute__ method instead. But the problem is not only that the >> introspection says it's not there, the problem is that the method really is >> not there. So we break user code that tries to use __getattr__ as an >> unbound method *and* the code generated by Cython for setting and >> retrieving docstrings of this method, thus breaking the module >> initialisation. So it's a double regression that is clearly user visible. >> >> Sounds like a release blocker to me. >> > > Robert and I were chatting about this earlier -- did the changes you > pushed completely fix this problem, or just cover it up? We couldn't > come up with a case that was still a problem off the cuff, but the > commit message said something like "for now," which made us think that > there might be more to the story. ;) > > -cc Well, Stefan's patch essentially reverts the part of my patch dealing with __getattr__. So this definitely fixes any __getattr__-related regressions, at the cost of not fixing the particular performance bug that was the reason for my patch (the one that William was so unhappy about on sage-devel). I have an idea for another patch that might make everyone happy. After calling PyType_Ready, copy __getattribute__ (the wrapper object provided by Python) to __getattr__; then the type has a __getattr__, so Stefan's code should work, but it's a Python wrapper object, which will hopefully avoid the performance problem on subclasses. (I'm not sure what this would do with respect to the docstring issue Stefan mentioned.) I could try this out this weekend, but if somebody else wants to give it a try sooner that would be great. Carl From robertwb at math.washington.edu Wed Aug 18 02:48:50 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Aug 2010 17:48:50 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 5:35 PM, Carl Witty wrote: > On Tue, Aug 17, 2010 at 5:12 PM, Craig Citro wrote: >> Hi Stefan, >> >>> Ah, got it. This is just what Carl wrote in the initial bug report. The new >>> setup fails to provide a __getattr__ method and implements a >>> __getattribute__ method instead. But the problem is not only that the >>> introspection says it's not there, the problem is that the method really is >>> not there. So we break user code that tries to use __getattr__ as an >>> unbound method *and* the code generated by Cython for setting and >>> retrieving docstrings of this method, thus breaking the module >>> initialisation. So it's a double regression that is clearly user visible. >>> >>> Sounds like a release blocker to me. >>> >> >> Robert and I were chatting about this earlier -- did the changes you >> pushed completely fix this problem, or just cover it up? We couldn't >> come up with a case that was still a problem off the cuff, but the >> commit message said something like "for now," which made us think that >> there might be more to the story. ;) >> >> -cc > > Well, Stefan's patch essentially reverts the part of my patch dealing > with __getattr__. ?So this definitely fixes any __getattr__-related > regressions, at the cost of not fixing the particular performance bug > that was the reason for my patch (the one that William was so unhappy > about on sage-devel). > > I have an idea for another patch that might make everyone happy. > After calling PyType_Ready, copy __getattribute__ (the wrapper object > provided by Python) to __getattr__; then the type has a __getattr__, > so Stefan's code should work, but it's a Python wrapper object, which > will hopefully avoid the performance problem on subclasses. ?(I'm not > sure what this would do with respect to the docstring issue Stefan > mentioned.) Copying it over isn't the right solution as __getattr_ is different than __getattribute__, and only the latter is created for a tp_getattr slot. I think that providing our own __getattr__ wrapper will not impact the performance of __getattribute__ (the important case), but this needs to be tested. - Robert From carl.witty at gmail.com Wed Aug 18 03:25:39 2010 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 17 Aug 2010 18:25:39 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 5:48 PM, Robert Bradshaw wrote: > On Tue, Aug 17, 2010 at 5:35 PM, Carl Witty wrote: >> Well, Stefan's patch essentially reverts the part of my patch dealing >> with __getattr__. ?So this definitely fixes any __getattr__-related >> regressions, at the cost of not fixing the particular performance bug >> that was the reason for my patch (the one that William was so unhappy >> about on sage-devel). >> >> I have an idea for another patch that might make everyone happy. >> After calling PyType_Ready, copy __getattribute__ (the wrapper object >> provided by Python) to __getattr__; then the type has a __getattr__, >> so Stefan's code should work, but it's a Python wrapper object, which >> will hopefully avoid the performance problem on subclasses. ?(I'm not >> sure what this would do with respect to the docstring issue Stefan >> mentioned.) > > Copying it over isn't the right solution as __getattr_ is different > than __getattribute__, and only the latter is created for a tp_getattr > slot. I think that providing our own __getattr__ wrapper will not > impact the performance of __getattribute__ (the important case), but > this needs to be tested. I'm really quite sure that the performance bug was due to a Cython-created __getattr__ wrapper. In detail: when you create a non-cdef subclass of a cdef class with __getattr__, Python has to decide what to stick in the tp_getattro slot. It notices that there's a __getattr__ defined (that's not a Python-created wrapper object), and so it puts in a generic function that looks up __getattribute__ and __getattr__ in the type dictionary. I guess copying __getattribute__ to __getattr__ would slightly change the behavior when somebody accessed __getattr__ directly. (It wouldn't change the behavior of normal attribute lookup on the object.) With the current cython-devel we could probably fix the Sage __getattr__ performance issue by changing Parent and Element to define __getattribute__ instead of __getattr__. (Maybe this is what you were saying?) Carl From robertwb at math.washington.edu Wed Aug 18 03:40:49 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Aug 2010 18:40:49 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 6:25 PM, Carl Witty wrote: > I'm really quite sure that the performance bug was due to a > Cython-created __getattr__ wrapper. > > In detail: when you create a non-cdef subclass of a cdef class with > __getattr__, Python has to decide what to stick in the tp_getattro > slot. ?It notices that there's a __getattr__ defined (that's not a > Python-created wrapper object), and so it puts in a generic function > that looks up __getattribute__ and __getattr__ in the type dictionary. > > I guess copying __getattribute__ to __getattr__ would slightly change > the behavior when somebody accessed __getattr__ directly. ?(It > wouldn't change the behavior of normal attribute lookup on the > object.) > > With the current cython-devel we could probably fix the Sage > __getattr__ performance issue by changing Parent and Element to define > __getattribute__ instead of __getattr__. ?(Maybe this is what you were > saying?) No, defining __getattribute__ is much worse than defining __getattr__ because then it needs to call __getattribute__ as an opaque Python function before invoking the standard Python lookup mechanisms. Really, I just need to read some code and do some benchmarks. - Robert From carl.witty at gmail.com Wed Aug 18 04:34:23 2010 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 17 Aug 2010 19:34:23 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 6:40 PM, Robert Bradshaw wrote: > On Tue, Aug 17, 2010 at 6:25 PM, Carl Witty wrote: > >> I'm really quite sure that the performance bug was due to a >> Cython-created __getattr__ wrapper. >> >> In detail: when you create a non-cdef subclass of a cdef class with >> __getattr__, Python has to decide what to stick in the tp_getattro >> slot. ?It notices that there's a __getattr__ defined (that's not a >> Python-created wrapper object), and so it puts in a generic function >> that looks up __getattribute__ and __getattr__ in the type dictionary. >> >> I guess copying __getattribute__ to __getattr__ would slightly change >> the behavior when somebody accessed __getattr__ directly. ?(It >> wouldn't change the behavior of normal attribute lookup on the >> object.) >> >> With the current cython-devel we could probably fix the Sage >> __getattr__ performance issue by changing Parent and Element to define >> __getattribute__ instead of __getattr__. ?(Maybe this is what you were >> saying?) > > No, defining __getattribute__ is much worse than defining __getattr__ > because then it needs to call __getattribute__ as an opaque Python > function before invoking the standard Python lookup mechanisms. > > Really, I just need to read some code and do some benchmarks. In Python, __getattribute__ is much worse than __getattr__. But in Cython, __getattribute__ and __getattr__ both set the tp_getattro slot; so you can replace a __getattr__ with a __getattribute__ that's just about as efficient. In either case, we must avoid having Python call __getattribute__ or __getattr__ as an opaque Python function; that's the source of the performance problem. It's vital that Python calls the tp_getattro slot directly. The way to make that happen is to make sure that any __getattribute__ or __getattr__ definitions are Python-generated wrappers (or, at least, that the wrapper's Py_TYPE is PyWrapperDescr_Type). Carl From dalcinl at gmail.com Wed Aug 18 04:43:03 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 17 Aug 2010 23:43:03 -0300 Subject: [Cython] Advice on a patch I'm working on Message-ID: I'm working on a patch trying to (you know me!) silent GCC unused warnings for special methods like __idiv__/__div__ that get generated but not used under a Py 3 runtime... In the process, I would like to make a little cleanup in TypeSlots.py... One of these cleanups, is related to the 'flag' argument to SlotDescriptor.__init__() .. Currently, it's only usage is to conditionally fill the nb_index slot. In short, instead of defining the slot as: MethodSlot(unaryfunc, "nb_index", "__index__", flag = "Py_TPFLAGS_HAVE_INDEX") I'll do it like this: MethodSlot(unaryfunc, "nb_index", "__index__", ifdef = ""PY_VERSION_HEX >= 0x02050000"") Comments ? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Wed Aug 18 06:27:02 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Aug 2010 21:27:02 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 7:34 PM, Carl Witty wrote: > On Tue, Aug 17, 2010 at 6:40 PM, Robert Bradshaw > wrote: >> On Tue, Aug 17, 2010 at 6:25 PM, Carl Witty wrote: >> >>> I'm really quite sure that the performance bug was due to a >>> Cython-created __getattr__ wrapper. >>> >>> In detail: when you create a non-cdef subclass of a cdef class with >>> __getattr__, Python has to decide what to stick in the tp_getattro >>> slot. ?It notices that there's a __getattr__ defined (that's not a >>> Python-created wrapper object), and so it puts in a generic function >>> that looks up __getattribute__ and __getattr__ in the type dictionary. >>> >>> I guess copying __getattribute__ to __getattr__ would slightly change >>> the behavior when somebody accessed __getattr__ directly. ?(It >>> wouldn't change the behavior of normal attribute lookup on the >>> object.) >>> >>> With the current cython-devel we could probably fix the Sage >>> __getattr__ performance issue by changing Parent and Element to define >>> __getattribute__ instead of __getattr__. ?(Maybe this is what you were >>> saying?) >> >> No, defining __getattribute__ is much worse than defining __getattr__ >> because then it needs to call __getattribute__ as an opaque Python >> function before invoking the standard Python lookup mechanisms. >> >> Really, I just need to read some code and do some benchmarks. > > In Python, __getattribute__ is much worse than __getattr__. ?But in > Cython, __getattribute__ and __getattr__ both set the tp_getattro > slot; so you can replace a __getattr__ with a __getattribute__ that's > just about as efficient. > > In either case, we must avoid having Python call __getattribute__ or > __getattr__ as an opaque Python function; that's the source of the > performance problem. ?It's vital that Python calls the tp_getattro > slot directly. ?The way to make that happen is to make sure that any > __getattribute__ or __getattr__ definitions are Python-generated > wrappers (or, at least, that the wrapper's Py_TYPE is > PyWrapperDescr_Type). Yep. I confirmed that just setting creating __getattr__ does incur the performance loss, even if __getattribute__ was good. It looks like there's no way to have a __getattr__ python-visible attribute without a performance penalty. Given that this only impacts cdef classes, what contracts are we making about the availability of unbound special methods? For example, we're OK with not exposing __long__. I'm leaning towards a cdef class being as fast as possible, while a normal class is 100% python semantics. Perhaps we should control this with a flag. For Sage I know we don't care about having __getattr__ visible as much as the speed penalty for every single attribute lookup of derived classes. - Robert From robertwb at math.washington.edu Wed Aug 18 06:30:56 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Aug 2010 21:30:56 -0700 Subject: [Cython] Advice on a patch I'm working on In-Reply-To: References: Message-ID: On Tue, Aug 17, 2010 at 7:43 PM, Lisandro Dalcin wrote: > I'm working on a patch trying to (you know me!) ?silent GCC unused > warnings for special methods like __idiv__/__div__ that get generated > but not used under a Py 3 runtime... Yay. > In the process, I would like to > make a little cleanup in TypeSlots.py... One of these cleanups, is > related to the 'flag' argument to SlotDescriptor.__init__() .. > Currently, it's only usage is to conditionally fill the nb_index slot. > > In short, instead of defining the slot as: > > ? ?MethodSlot(unaryfunc, "nb_index", "__index__", flag = > "Py_TPFLAGS_HAVE_INDEX") > > I'll do it like this: > > ? ?MethodSlot(unaryfunc, "nb_index", "__index__", ifdef = > ""PY_VERSION_HEX >= 0x02050000"") > > > Comments ? I think Py_TPFLAGS_HAVE_INDEX is more explicit, so I prefer that, but can't you use the latter form when there isn't already a flag? That's orthogonal to the name, which I could go either way on. - Robert From carl.witty at gmail.com Wed Aug 18 06:41:07 2010 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 17 Aug 2010 21:41:07 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 9:27 PM, Robert Bradshaw wrote: > Yep. I confirmed that just setting creating __getattr__ does incur the > performance loss, even if __getattribute__ was good. It looks like > there's no way to have a __getattr__ python-visible attribute without > a performance penalty. Well, I think (but I'm not sure) that if the python-visible __getattr__ is an instance of PyWrapperDescr_Type, then the problem doesn't arise. And like I said, there's a somewhat simple workaround if you actually know about the problem, which is to define __getattribute__ instead of __getattr__. Carl From robertwb at math.washington.edu Wed Aug 18 06:54:01 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Aug 2010 21:54:01 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 9:41 PM, Carl Witty wrote: > On Tue, Aug 17, 2010 at 9:27 PM, Robert Bradshaw > wrote: >> Yep. I confirmed that just setting creating __getattr__ does incur the >> performance loss, even if __getattribute__ was good. It looks like >> there's no way to have a __getattr__ python-visible attribute without >> a performance penalty. > > Well, I think (but I'm not sure) that if the python-visible > __getattr__ is an instance of PyWrapperDescr_Type, then the problem > doesn't arise. Looking at typeobjet.c:slot_tp_getattr_hook, I'm not sure even this would help, but I haven't fully unraveled the logic of update_one_slot. Worth further investigation and/or a try. > And like I said, there's a somewhat simple workaround > if you actually know about the problem, which is to define > __getattribute__ instead of __getattr__. Which is essentially what we do in Cython already. (It's a pain to do on the user's side, and almost always not what you want.) - Robert From carl.witty at gmail.com Wed Aug 18 07:05:04 2010 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 17 Aug 2010 22:05:04 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 9:54 PM, Robert Bradshaw wrote: > On Tue, Aug 17, 2010 at 9:41 PM, Carl Witty wrote: >> On Tue, Aug 17, 2010 at 9:27 PM, Robert Bradshaw >> wrote: >>> Yep. I confirmed that just setting creating __getattr__ does incur the >>> performance loss, even if __getattribute__ was good. It looks like >>> there's no way to have a __getattr__ python-visible attribute without >>> a performance penalty. >> >> Well, I think (but I'm not sure) that if the python-visible >> __getattr__ is an instance of PyWrapperDescr_Type, then the problem >> doesn't arise. > > Looking at typeobjet.c:slot_tp_getattr_hook, I'm not sure even this > would help, but I haven't fully unraveled the logic of > update_one_slot. Worth further investigation and/or a try. The goal is to keep our slot method in tp_getattro, in which case slot_tp_getattr_hook isn't involved at all. Carl From robertwb at math.washington.edu Wed Aug 18 08:02:29 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Aug 2010 23:02:29 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 10:05 PM, Carl Witty wrote: > On Tue, Aug 17, 2010 at 9:54 PM, Robert Bradshaw > wrote: >> On Tue, Aug 17, 2010 at 9:41 PM, Carl Witty wrote: >>> On Tue, Aug 17, 2010 at 9:27 PM, Robert Bradshaw >>> wrote: >>>> Yep. I confirmed that just setting creating __getattr__ does incur the >>>> performance loss, even if __getattribute__ was good. It looks like >>>> there's no way to have a __getattr__ python-visible attribute without >>>> a performance penalty. >>> >>> Well, I think (but I'm not sure) that if the python-visible >>> __getattr__ is an instance of PyWrapperDescr_Type, then the problem >>> doesn't arise. >> >> Looking at typeobjet.c:slot_tp_getattr_hook, I'm not sure even this >> would help, but I haven't fully unraveled the logic of >> update_one_slot. Worth further investigation and/or a try. > > The goal is to keep our slot method in tp_getattro, in which case > slot_tp_getattr_hook isn't involved at all. Yep. If we can do that, that'd be great. Perhaps just making __getattr__ a PyWrapperDescr_Type would be enough. I'm running on way too few hours of sleep to attempt that now... - Robert From stefan_ml at behnel.de Wed Aug 18 08:11:21 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Aug 2010 08:11:21 +0200 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> Message-ID: <4C6B7989.1010205@behnel.de> Robert Bradshaw, 18.08.2010 06:27: > Given that this only impacts cdef classes, what contracts are we > making about the availability of unbound special methods? For example, > we're OK with not exposing __long__. I'm leaning towards a cdef class > being as fast as possible, while a normal class is 100% python > semantics. This contradicts our often stated goal that it should be easy to move code from Python to Cython and that Cython code should mimic Python semantics as closely as possible. I find this a very valuable goal. It also poses problems for classes that must be cdef classes because they wrap C values, as in pretty much all wrapper code. The simple fact that it's a cdef class doesn't imply that a user wants Python visible semantics that diverge from Python and thus impact the users of the class. If we can't help it, well, then that's how it must work. But we can clearly avoid the problem in this case. > Perhaps we should control this with a flag. I'm not a big fan at all of changing behaviour with flags, but I'm strictly against doing it in cases where the resulting behaviour is not obvious. It's absolutely not obvious to me that a flag to make a call to a regular Python special method faster makes that method unavailable from Python code. Such an impact is too easy to get missed by tests. Stefan From dalcinl at gmail.com Wed Aug 18 16:14:57 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 11:14:57 -0300 Subject: [Cython] Advice on a patch I'm working on In-Reply-To: References: Message-ID: On 18 August 2010 01:30, Robert Bradshaw wrote: > On Tue, Aug 17, 2010 at 7:43 PM, Lisandro Dalcin wrote: >> I'm working on a patch trying to (you know me!) ?silent GCC unused >> warnings for special methods like __idiv__/__div__ that get generated >> but not used under a Py 3 runtime... > > Yay. > >> In the process, I would like to >> make a little cleanup in TypeSlots.py... One of these cleanups, is >> related to the 'flag' argument to SlotDescriptor.__init__() .. >> Currently, it's only usage is to conditionally fill the nb_index slot. >> >> In short, instead of defining the slot as: >> >> ? ?MethodSlot(unaryfunc, "nb_index", "__index__", flag = >> "Py_TPFLAGS_HAVE_INDEX") >> >> I'll do it like this: >> >> ? ?MethodSlot(unaryfunc, "nb_index", "__index__", ifdef = >> ""PY_VERSION_HEX >= 0x02050000"") >> >> >> Comments ? > > I think Py_TPFLAGS_HAVE_INDEX is more explicit, > This macro is not defined in Py<2.5, Cython defines it to 0 (zero) in Py3 ... This is contrary to the way Py_TPFLAGS_HAVE_NEWBUFFER is defined (by Cython) to 0 in both Py<2.6 and Py3. I really like to review these inconsistencies. But a necesary previous step is to get rid of unnecesary usages of Py_TPFLAGS_HAVE_INDEX. > so I prefer that, but > can't you use the latter form when there isn't already a flag? That's > orthogonal to the name, which I could go either way on. > Yes, basically I'm trying to conditionally fill the slot based on Python version, and not in the definition/value of Py_TPFLAGS_HAVE_INDEX. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed Aug 18 16:38:16 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 11:38:16 -0300 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C6B7989.1010205@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On 18 August 2010 03:11, Stefan Behnel wrote: > > It also poses problems for classes that must be cdef classes because they > wrap C values, as in pretty much all wrapper code. > cdef classes already are quite different. For example, instances do not get a __dict__, nor Cython provides any kind of support for these instances getting one (though I'm working on that). > > ?> Perhaps we should control this with a flag. > > I'm not a big fan at all of changing behaviour with flags, but I'm strictly > against doing it in cases where the resulting behaviour is not obvious. > It's absolutely not obvious to me that a flag to make a call to a regular > Python special method faster makes that method unavailable from Python > code. Such an impact is too easy to get missed by tests. > I agree. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From carl.witty at gmail.com Wed Aug 18 17:53:06 2010 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 18 Aug 2010 08:53:06 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C6B7989.1010205@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 11:11 PM, Stefan Behnel wrote: > Robert Bradshaw, 18.08.2010 06:27: > ?> Perhaps we should control this with a flag. > > I'm not a big fan at all of changing behaviour with flags, but I'm strictly > against doing it in cases where the resulting behaviour is not obvious. > It's absolutely not obvious to me that a flag to make a call to a regular > Python special method faster makes that method unavailable from Python > code. Such an impact is too easy to get missed by tests. More possibilities: control __getattr__ by a special declaration, that is either required (is this possible?) or strongly encouraged to be placed directly on the __getattr__ method rather than scoped over a larger area. (Surely we could come up with a name that addresses Stefan's concerns? @cython.make_fast_getattr_that_is_unavailable_to_python ?) Another option: add a decorator so that instead of: def __getattr__(self, attr): # your code here you can equivalently write: @cython.getattr_semantics_for_getattribute def __getattribute__(self, attr): # your code here and have it mean the same thing. For Cython, this would mean generating essentially the same C code we already put in tp_getattro: PyObject *v = PyObject_GenericGetAttr(o, n); if (!v && PyErr_ExceptionMatches(PyExc_AttributeError)) { PyErr_Clear(); v = # your code here } but for Python this could be done as a regular decorator, although the result would be slow. Although I'm actually in favor of just leaving the post-#561 status, where __getattr__ remains unavailable to Python, using the "cdef classes are different" rationale. Carl From robertwb at math.washington.edu Wed Aug 18 18:30:12 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 09:30:12 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C6B7989.1010205@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On Tue, Aug 17, 2010 at 11:11 PM, Stefan Behnel wrote: > Robert Bradshaw, 18.08.2010 06:27: >> Given that this only impacts cdef classes, what contracts are we >> making about the availability of unbound special methods? For example, >> we're OK with not exposing __long__. I'm leaning towards a cdef class >> being as fast as possible, while a normal class is 100% python >> semantics. > > This contradicts our often stated goal that it should be easy to move code > from Python to Cython and that Cython code should mimic Python semantics as > closely as possible. I find this a very valuable goal. Long term, I think it would be valuable to compile, where possible, users classes into "cpdef" classes that have exactly the same semantics of classes with the speed of cdef classses. > It also poses problems for classes that must be cdef classes because they > wrap C values, as in pretty much all wrapper code. The simple fact that > it's a cdef class doesn't imply that a user wants Python visible semantics > that diverge from Python and thus impact the users of the class. If we > can't help it, well, then that's how it must work. But we can clearly avoid > the problem in this case. This isn't a migration case. > ?> Perhaps we should control this with a flag. > > I'm not a big fan at all of changing behaviour with flags, but I'm strictly > against doing it in cases where the resulting behaviour is not obvious. > It's absolutely not obvious to me that a flag to make a call to a regular > Python special method faster makes that method unavailable from Python > code. Such an impact is too easy to get missed by tests. I think this is analogous to the always_allow_keywords flag. We disallow certain rare usecases for the sake of a widespread performance increase. - Robert From robertwb at math.washington.edu Wed Aug 18 18:38:34 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 09:38:34 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 7:38 AM, Lisandro Dalcin wrote: > On 18 August 2010 03:11, Stefan Behnel wrote: >> >> It also poses problems for classes that must be cdef classes because they >> wrap C values, as in pretty much all wrapper code. >> > > cdef classes already are quite different. For example, instances do > not get a __dict__, nor Cython provides any kind of support for these > instances getting one (though I'm working on that). > While this is a good thing, it should come with a warning that it disables many optimizations (e.g. cpdef methods without attribute lookup). >> ?> Perhaps we should control this with a flag. >> >> I'm not a big fan at all of changing behaviour with flags, but I'm strictly >> against doing it in cases where the resulting behaviour is not obvious. >> It's absolutely not obvious to me that a flag to make a call to a regular >> Python special method faster makes that method unavailable from Python >> code. Such an impact is too easy to get missed by tests. > > I agree. I wouldn't call it a "regular" special method :). No, it's not obvious. But having in not there by default, and an entry in the FAQ about why, with the flag (or even better, a visibility decorator as Carl suggested) to make it visible, should be understandable enough. I still think that fast by default is better, as it's what most users will want, and obvious when you need make it visible, but not obvious when you need to make it fast. - Robert From robertwb at math.washington.edu Wed Aug 18 18:42:17 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 09:42:17 -0700 Subject: [Cython] Advice on a patch I'm working on In-Reply-To: References: Message-ID: On Wed, Aug 18, 2010 at 7:14 AM, Lisandro Dalcin wrote: > On 18 August 2010 01:30, Robert Bradshaw wrote: >> On Tue, Aug 17, 2010 at 7:43 PM, Lisandro Dalcin wrote: >>> I'm working on a patch trying to (you know me!) ?silent GCC unused >>> warnings for special methods like __idiv__/__div__ that get generated >>> but not used under a Py 3 runtime... >> >> Yay. >> >>> In the process, I would like to >>> make a little cleanup in TypeSlots.py... One of these cleanups, is >>> related to the 'flag' argument to SlotDescriptor.__init__() .. >>> Currently, it's only usage is to conditionally fill the nb_index slot. >>> >>> In short, instead of defining the slot as: >>> >>> ? ?MethodSlot(unaryfunc, "nb_index", "__index__", flag = >>> "Py_TPFLAGS_HAVE_INDEX") >>> >>> I'll do it like this: >>> >>> ? ?MethodSlot(unaryfunc, "nb_index", "__index__", ifdef = >>> ""PY_VERSION_HEX >= 0x02050000"") >>> >>> >>> Comments ? >> >> I think Py_TPFLAGS_HAVE_INDEX is more explicit, >> > > This macro is not defined in Py<2.5, Cython defines it to 0 (zero) in > Py3 ... Yeah, I guess that should have been obvious. Given that the semantics are more completed than a single version comparison (I'm guessing 2.5 <= version < 3.0) it's probably worth making it a macro we define ourselves, like CYTHON_TPFLAGS_HAVE_INDEX. > This is contrary to the way Py_TPFLAGS_HAVE_NEWBUFFER is > defined (by Cython) to 0 in both Py<2.6 and Py3. I really like to > review these inconsistencies. But a necesary previous step is to get > rid of unnecesary usages of Py_TPFLAGS_HAVE_INDEX. > >> so I prefer that, but >> can't you use the latter form when there isn't already a flag? That's >> orthogonal to the name, which I could go either way on. >> > > Yes, basically I'm trying to conditionally fill the slot based on > Python version, and not in the definition/value of > Py_TPFLAGS_HAVE_INDEX. I had missed the obvious point that Py_TPFLAGS_HAVE_INDEX isn't going to be defined in pre-__index__ versions, which makes Py_TPFLAGS_HAVE_INDEX not the best to use. - Robert From dalcinl at gmail.com Wed Aug 18 18:50:05 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 13:50:05 -0300 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On 18 August 2010 12:53, Carl Witty wrote: > > Although I'm actually in favor of just leaving the post-#561 status, > where __getattr__ remains unavailable to Python, using the "cdef > classes are different" rationale. > What about cheating and generating special code in Cython's __getattribute__ to special case the "__getattr__" attribute and return something useful? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Wed Aug 18 18:52:42 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 09:52:42 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 9:50 AM, Lisandro Dalcin wrote: > On 18 August 2010 12:53, Carl Witty wrote: >> >> Although I'm actually in favor of just leaving the post-#561 status, >> where __getattr__ remains unavailable to Python, using the "cdef >> classes are different" rationale. >> > > What about cheating and generating special code in Cython's > __getattribute__ to special case the "__getattr__" attribute and > return something useful? I'm not following you here. We currently create a __getattribute__ using the __getattr__ method, this is just a question of exposing the unbound method to Python space. - Robert From dalcinl at gmail.com Wed Aug 18 19:01:34 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 14:01:34 -0300 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On 18 August 2010 13:52, Robert Bradshaw wrote: > On Wed, Aug 18, 2010 at 9:50 AM, Lisandro Dalcin wrote: >> On 18 August 2010 12:53, Carl Witty wrote: >>> >>> Although I'm actually in favor of just leaving the post-#561 status, >>> where __getattr__ remains unavailable to Python, using the "cdef >>> classes are different" rationale. >>> >> >> What about cheating and generating special code in Cython's >> __getattribute__ to special case the "__getattr__" attribute and >> return something useful? > > I'm not following you here. We currently create a __getattribute__ > using the __getattr__ method, this is just a question of exposing the > unbound method to Python space. > cdef Foo: def __getattr__(self, name): ... def Foo_getattr_hook(obj, name): try: attr = Foo_getattr(obj, name) # this is the user defined code in Foo.__getattr__() except AttributeError: if name == '__getattr__': return bound_getattr_method(obj) else: raise .... Foo.tp_getattro = Foo_getattr_hook Too easy... I'm likely missing something... From robertwb at math.washington.edu Wed Aug 18 18:57:30 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 09:57:30 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 9:52 AM, Robert Bradshaw wrote: > On Wed, Aug 18, 2010 at 9:50 AM, Lisandro Dalcin wrote: >> On 18 August 2010 12:53, Carl Witty wrote: >>> >>> Although I'm actually in favor of just leaving the post-#561 status, >>> where __getattr__ remains unavailable to Python, using the "cdef >>> classes are different" rationale. >>> >> >> What about cheating and generating special code in Cython's >> __getattribute__ to special case the "__getattr__" attribute and >> return something useful? > > I'm not following you here. We currently create a __getattribute__ > using the __getattr__ method, this is just a question of exposing the > unbound method to Python space. Oh, I think I see what you mean now. Given that it uses _PyType_Lookup we might be able to get away with this. (Of course that would be trickier work for the unbound method than the bound one, changing the __getattr__ of the type object.) - Robert From stefan_ml at behnel.de Wed Aug 18 19:21:53 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Aug 2010 19:21:53 +0200 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: <4C6C16B1.2090006@behnel.de> Robert Bradshaw, 18.08.2010 18:30: > On Tue, Aug 17, 2010 at 11:11 PM, Stefan Behnel wrote: >> It also poses problems for classes that must be cdef classes because they >> wrap C values, as in pretty much all wrapper code. The simple fact that >> it's a cdef class doesn't imply that a user wants Python visible semantics >> that diverge from Python and thus impact the users of the class. If we >> can't help it, well, then that's how it must work. But we can clearly avoid >> the problem in this case. > > This isn't a migration case. Right, the case at hand is about a possible optimisation. I still do not understand why this is considered worth delaying the release of 0.13. It is not a regression at all, it has been like this for ages. Instead of just taking the time to find a suitable solution that we can apply under the hood, everyone is rushing to force users to change their code to deploy flags and decorators, either one way or another. Why can't we just get 0.13 out of the door with a safe default (like the one I applied to the code base), that breaks no code and doesn't provide any degradation in terms of performance, and then take our time to look for a real solution that does not impact user code? Stefan From craigcitro at gmail.com Wed Aug 18 19:46:06 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 18 Aug 2010 10:46:06 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C6C16B1.2090006@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> <4C6C16B1.2090006@behnel.de> Message-ID: > Right, the case at hand is about a possible optimisation. I still do not > understand why this is considered worth delaying the release of 0.13. It is > not a regression at all, it has been like this for ages. Instead of just > taking the time to find a suitable solution that we can apply under the > hood, everyone is rushing to force users to change their code to deploy > flags and decorators, either one way or another. > At least for me, I wanted this in because I know Sage developers are chomping at the bit to get this fixed. Carl originally ran into this because William discovered a huge performance degradation in Sage (the degradation wasn't Cython's fault directly), and Carl traced it back to this. If we don't get this into 0.13, they'll immediately need a patched Cython with this fix included. I agree that it's unfortunate that it was discovered so far into the (nigh-infinitely-long) release cycle ... -cc From robertwb at math.washington.edu Wed Aug 18 19:53:46 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 10:53:46 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 10:01 AM, Lisandro Dalcin wrote: > On 18 August 2010 13:52, Robert Bradshaw wrote: >> On Wed, Aug 18, 2010 at 9:50 AM, Lisandro Dalcin wrote: >>> On 18 August 2010 12:53, Carl Witty wrote: >>>> >>>> Although I'm actually in favor of just leaving the post-#561 status, >>>> where __getattr__ remains unavailable to Python, using the "cdef >>>> classes are different" rationale. >>>> >>> >>> What about cheating and generating special code in Cython's >>> __getattribute__ to special case the "__getattr__" attribute and >>> return something useful? >> >> I'm not following you here. We currently create a __getattribute__ >> using the __getattr__ method, this is just a question of exposing the >> unbound method to Python space. >> > > cdef Foo: > ? def __getattr__(self, name): > ? ? ? ... > > def Foo_getattr_hook(obj, name): > ? ? try: > ? ? ? ? attr = Foo_getattr(obj, name) # this is the user defined code > in Foo.__getattr__() > ? ?except AttributeError: > ? ? ? ?if name == '__getattr__': > ? ? ? ? ? ? return bound_getattr_method(obj) > ? ? ? ?else: > ? ? ? ? ? ?raise > > .... > > Foo.tp_getattro = Foo_getattr_hook > > > Too easy... I'm likely missing something... This doesn't provide the unbound method. - Robert From robertwb at math.washington.edu Wed Aug 18 20:16:51 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 11:16:51 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C6C16B1.2090006@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> <4C6C16B1.2090006@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 10:21 AM, Stefan Behnel wrote: > Robert Bradshaw, 18.08.2010 18:30: >> On Tue, Aug 17, 2010 at 11:11 PM, Stefan Behnel wrote: >>> It also poses problems for classes that must be cdef classes because they >>> wrap C values, as in pretty much all wrapper code. The simple fact that >>> it's a cdef class doesn't imply that a user wants Python visible semantics >>> that diverge from Python and thus impact the users of the class. If we >>> can't help it, well, then that's how it must work. But we can clearly avoid >>> the problem in this case. >> >> This isn't a migration case. > > Right, the case at hand is about a possible optimisation. I still do not > understand why this is considered worth delaying the release of 0.13. It is > not a regression at all, it has been like this for ages. Instead of just > taking the time to find a suitable solution that we can apply under the > hood, everyone is rushing to force users to change their code to deploy > flags and decorators, either one way or another. > > Why can't we just get 0.13 out of the door with a safe default (like the > one I applied to the code base), that breaks no code and doesn't provide > any degradation in terms of performance, and then take our time to look for > a real solution that does not impact user code? If the release is ready to go out, let's not delay it (though for us Sage folks really want this, so we'll follow up with a 0.13.1 soon, or just use a patched Cython in Sage). In terms of impacting user code, the only case it impacts is if people access __getattr__ by name. Is anyone doing that? - Robert From stefan_ml at behnel.de Wed Aug 18 20:29:42 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Aug 2010 20:29:42 +0200 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> <4C6C16B1.2090006@behnel.de> Message-ID: <4C6C2696.3060803@behnel.de> Robert Bradshaw, 18.08.2010 20:16: > In terms of impacting user code, the only case it impacts is if people > access __getattr__ by name. Is anyone doing that? Well, yes, look at the test case I added. It's quite common to map __getitem__ or __getattr__ on the other method respectively to provide different interfaces for a class, e.g. to provide a __getitem__ fallback for names that do not work as a Python identifier. You can also use the unbound __getattr__ method to read a sequence of attributes. Maybe not the most common but still a valid use case. To me, it's enough to know that this breaks code at all, so it's no longer just a simple optimisation but a change that breaks a standard Python feature that currently works. Stefan From robertwb at math.washington.edu Wed Aug 18 20:47:35 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 11:47:35 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C6C2696.3060803@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> <4C6C16B1.2090006@behnel.de> <4C6C2696.3060803@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 11:29 AM, Stefan Behnel wrote: > Robert Bradshaw, 18.08.2010 20:16: >> In terms of impacting user code, the only case it impacts is if people >> access __getattr__ by name. Is anyone doing that? > > Well, yes, look at the test case I added. It was a test in response to this regression :). > It's quite common to map > __getitem__ or __getattr__ on the other method respectively to provide > different interfaces for a class, e.g. to provide a __getitem__ fallback > for names that do not work as a Python identifier. I've never seen this, but could see that being justified. > You can also use the > unbound __getattr__ method to read a sequence of attributes. Would using the getattr builtin be a more natural way of doing this? Or is the point to skip the normal attribute lookup steps? > Maybe not the > most common but still a valid use case. To me, it's enough to know that > this breaks code at all, so it's no longer just a simple optimisation but a > change that breaks a standard Python feature that currently works. Yes, it's a backwards incompatible step backwards for cdef classes, but I think we have a little liberty for cdef class special methods--they already violate the python conventions in several ways (e.g. for arithmetic). Lets release 0.13 without this lookup regression/performance fix (depending on your point of view) and keep the discussion open for what to do for 0.13.1 (which, if it isn't out soon, may require a patched Cython for Sage.) - Robert From robertwb at math.washington.edu Wed Aug 18 20:57:45 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 11:57:45 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 8:53 AM, Carl Witty wrote: > On Tue, Aug 17, 2010 at 11:11 PM, Stefan Behnel wrote: >> Robert Bradshaw, 18.08.2010 06:27: >> ?> Perhaps we should control this with a flag. >> >> I'm not a big fan at all of changing behaviour with flags, but I'm strictly >> against doing it in cases where the resulting behaviour is not obvious. >> It's absolutely not obvious to me that a flag to make a call to a regular >> Python special method faster makes that method unavailable from Python >> code. Such an impact is too easy to get missed by tests. > > More possibilities: control __getattr__ by a special declaration, that > is either required (is this possible?) or strongly encouraged to be > placed directly on the __getattr__ method rather than scoped over a > larger area. ?(Surely we could come up with a name that addresses > Stefan's concerns? I was just thinking, I really like the idea of a decorator, but that does make it harder to apply on a large scale compared to a flag (e.g. for all of Sage). Of course, maybe that's a good thing. - Robert From stefan_ml at behnel.de Wed Aug 18 21:03:09 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Aug 2010 21:03:09 +0200 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> <4C6C16B1.2090006@behnel.de> <4C6C2696.3060803@behnel.de> Message-ID: <4C6C2E6D.6050104@behnel.de> Robert Bradshaw, 18.08.2010 20:47: > On Wed, Aug 18, 2010 at 11:29 AM, Stefan Behnel wrote: >> Robert Bradshaw, 18.08.2010 20:16: >>> In terms of impacting user code, the only case it impacts is if people >>> access __getattr__ by name. Is anyone doing that? >> >> Well, yes, look at the test case I added. > > It was a test in response to this regression :). > >> It's quite common to map >> __getitem__ or __getattr__ on the other method respectively to provide >> different interfaces for a class, e.g. to provide a __getitem__ fallback >> for names that do not work as a Python identifier. > > I've never seen this, but could see that being justified. > >> You can also use the >> unbound __getattr__ method to read a sequence of attributes. > > Would using the getattr builtin be a more natural way of doing this? > Or is the point to skip the normal attribute lookup steps? To be honest, I don't know. As I said, likely not the most common use case. Another thing I forgot to mention is that Cython uses the lookup internally in two cases: when generating the __test__ dict and in the now modified module init code that sets up the docstrings. Both fail when __getattr__ cannot be found, thus preventing the module from being used at all if users have a docstring in their __getattr__ methods. > Lets release 0.13 without this lookup regression/performance fix > (depending on your point of view) and keep the discussion open for > what to do for 0.13.1 (which, if it isn't out soon, may require a > patched Cython for Sage.) Based on previous experience, I think it'll be a lot easier to release a 0.13.1 once 0.13 is out. ;) Stefan From dalcinl at gmail.com Wed Aug 18 21:04:27 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 16:04:27 -0300 Subject: [Cython] Cython performance bug for special functions In-Reply-To: References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> Message-ID: On 18 August 2010 15:57, Robert Bradshaw wrote: > On Wed, Aug 18, 2010 at 8:53 AM, Carl Witty wrote: >> On Tue, Aug 17, 2010 at 11:11 PM, Stefan Behnel wrote: >>> Robert Bradshaw, 18.08.2010 06:27: >>> ?> Perhaps we should control this with a flag. >>> >>> I'm not a big fan at all of changing behaviour with flags, but I'm strictly >>> against doing it in cases where the resulting behaviour is not obvious. >>> It's absolutely not obvious to me that a flag to make a call to a regular >>> Python special method faster makes that method unavailable from Python >>> code. Such an impact is too easy to get missed by tests. >> >> More possibilities: control __getattr__ by a special declaration, that >> is either required (is this possible?) or strongly encouraged to be >> placed directly on the __getattr__ method rather than scoped over a >> larger area. ?(Surely we could come up with a name that addresses >> Stefan's concerns? > > I was just thinking, I really like the idea of a decorator, but that > does make it harder to apply on a large scale compared to a flag (e.g. > for all of Sage). Of course, maybe that's a good thing. > I would add the decorator/directive, but not document it in any way, and perhaps add a big warning in the source code that this is experimental. Then Sage could enable it, so Sage is happy. We have plenty of time to think about better ways to fix this... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Wed Aug 18 21:37:39 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 12:37:39 -0700 Subject: [Cython] Cython performance bug for special functions In-Reply-To: <4C6C2E6D.6050104@behnel.de> References: <4C65466A.8050602@behnel.de> <4C67D828.7090206@behnel.de> <4C67E381.8060309@behnel.de> <4C6B7989.1010205@behnel.de> <4C6C16B1.2090006@behnel.de> <4C6C2696.3060803@behnel.de> <4C6C2E6D.6050104@behnel.de> Message-ID: On Wed, Aug 18, 2010 at 12:03 PM, Stefan Behnel wrote: > Robert Bradshaw, 18.08.2010 20:47: >> On Wed, Aug 18, 2010 at 11:29 AM, Stefan Behnel wrote: >>> Robert Bradshaw, 18.08.2010 20:16: >>>> In terms of impacting user code, the only case it impacts is if people >>>> access __getattr__ by name. Is anyone doing that? >>> >>> Well, yes, look at the test case I added. >> >> It was a test in response to this regression :). >> >>> It's quite common to map >>> __getitem__ or __getattr__ on the other method respectively to provide >>> different interfaces for a class, e.g. to provide a __getitem__ fallback >>> for names that do not work as a Python identifier. >> >> I've never seen this, but could see that being justified. >> >>> You can also use the >>> unbound __getattr__ method to read a sequence of attributes. >> >> Would using the getattr builtin be a more natural way of doing this? >> Or is the point to skip the normal attribute lookup steps? > > To be honest, I don't know. As I said, likely not the most common use case. > > Another thing I forgot to mention is that Cython uses the lookup internally > in two cases: when generating the __test__ dict and in the now modified > module init code that sets up the docstrings. Both fail when __getattr__ > cannot be found, thus preventing the module from being used at all if users > have a docstring in their __getattr__ methods. This is easily accounted for in our code, the same way we handle __cinit__, etc. >> Lets release 0.13 without this lookup regression/performance fix >> (depending on your point of view) and keep the discussion open for >> what to do for 0.13.1 (which, if it isn't out soon, may require a >> patched Cython for Sage.) > > Based on previous experience, I think it'll be a lot easier to release a > 0.13.1 once 0.13 is out. ;) +1 Release candidate will be out today (with the current tip) and pending no major breakage, that will be the release very shortly. - Robert From dalcinl at gmail.com Wed Aug 18 22:12:30 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 17:12:30 -0300 Subject: [Cython] Py_TPFLAGS_HAVE_VERSION_TAG Message-ID: Take a look at the definition of Py_TPFLAGS_DEFAULT for Python 2.6 and 2.7, you will notice that Py_TPFLAGS_HAVE_VERSION_TAG is only added when building core Python, but not for extension types. This is not the case of Python 3.x. So it seems that extension types will not benefit from the 2.6/2.7 type cache. Am I missing something? Comments? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Thu Aug 19 01:56:11 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Aug 2010 16:56:11 -0700 Subject: [Cython] Py_TPFLAGS_HAVE_VERSION_TAG In-Reply-To: References: Message-ID: On Wed, Aug 18, 2010 at 1:12 PM, Lisandro Dalcin wrote: > Take a look at the definition of Py_TPFLAGS_DEFAULT for Python 2.6 and > 2.7, you will notice that Py_TPFLAGS_HAVE_VERSION_TAG is only added > when building core Python, but not for extension types. This is not > the case of Python 3.x. > > So it seems that extension types will not benefit from the 2.6/2.7 > type cache. Am I missing something? Comments? Is there some kind of a contract we could obey to enable it for our extension types? - Robert From dalcinl at gmail.com Thu Aug 19 02:51:12 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 21:51:12 -0300 Subject: [Cython] Py_TPFLAGS_HAVE_VERSION_TAG In-Reply-To: References: Message-ID: On 18 August 2010 20:56, Robert Bradshaw wrote: > On Wed, Aug 18, 2010 at 1:12 PM, Lisandro Dalcin wrote: >> Take a look at the definition of Py_TPFLAGS_DEFAULT for Python 2.6 and >> 2.7, you will notice that Py_TPFLAGS_HAVE_VERSION_TAG is only added >> when building core Python, but not for extension types. This is not >> the case of Python 3.x. >> >> So it seems that extension types will not benefit from the 2.6/2.7 >> type cache. Am I missing something? Comments? > > Is there some kind of a contract we could obey to enable it for our > extension types? > I do not think so... perhaps we should ask in python-dev ... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Thu Aug 19 03:06:45 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 18 Aug 2010 22:06:45 -0300 Subject: [Cython] Py_TPFLAGS_HAVE_VERSION_TAG In-Reply-To: References: Message-ID: On 18 August 2010 21:51, Lisandro Dalcin wrote: > On 18 August 2010 20:56, Robert Bradshaw wrote: >> On Wed, Aug 18, 2010 at 1:12 PM, Lisandro Dalcin wrote: >>> Take a look at the definition of Py_TPFLAGS_DEFAULT for Python 2.6 and >>> 2.7, you will notice that Py_TPFLAGS_HAVE_VERSION_TAG is only added >>> when building core Python, but not for extension types. This is not >>> the case of Python 3.x. >>> >>> So it seems that extension types will not benefit from the 2.6/2.7 >>> type cache. Am I missing something? Comments? >> >> Is there some kind of a contract we could obey to enable it for our >> extension types? >> > > I do not think so... perhaps we should ask in python-dev ... > http://bugs.python.org/issue1878#msg61381 In sort, Guido decided to remove the flag from TPFLAGS_DEFAULT just in case Py2 extension modules are playing with tp_dict (like Cython (&Pyrex?) did some time ago). I would say it's safe to enable it for Cython-generated extension types. Stefan, have you something to add? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From kayhayen at gmx.de Thu Aug 19 07:02:10 2010 From: kayhayen at gmx.de (Kay Hayen) Date: Thu, 19 Aug 2010 07:02:10 +0200 Subject: [Cython] Releasing Nuitka 0.1 Message-ID: <4C6CBAD2.6080003@gmx.de> Hello, Obviously this is very exciting step for me. I am releasing Nuitka today. Finally. For a long time I knew I would, but actually doing it, is a different beast. Reaching my goals for release turned out to be less far away than I hope, so instead of end of August, I can already release it now. Currently it?s not more than 4% faster than CPython. No surprise there, if all you did, is removing the bytecode interpretation so far. It?s not impressive at all. It?s not even a reason to use it. But it?s also only a start. Clearly, once I get into optimizing the code generation of Nuitka, it will only get better, and then probably in sometimes dramatic steps. But I see this as a long term goal. http://kayhayen24x7.homelinux.org/blog/2010/08/releasing-nuitka-to-the-world/ I hope you will enjoy it as much as I did enjoy writing it. I will appreciate your feedback. My ability to respond will be limited due to my day job which keeps me busy, but I will try. If some minor things come up, there likely will be a 0.1.1 at the end of August. After that I am moving to a new house and you know how time consuming that can be. Yours, Kay Hayen PS: I'm hosting my blog and the home NAS machine, which should run 24x7, so don't be surprised if its slow or temporarily unavailable. From robertwb at math.washington.edu Thu Aug 19 10:48:57 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 19 Aug 2010 01:48:57 -0700 Subject: [Cython] Cython 0.13 release candidate Message-ID: The Cython 0.13 release candidate is out at http://cython.org/release/Cython-0.13.rc0.tar.gz . Unless there any last-minute catastrophic failures, this will be the official release. - Robert From robertwb at math.washington.edu Thu Aug 19 11:05:27 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 19 Aug 2010 02:05:27 -0700 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: References: Message-ID: Ignore that last one, the release candidate is http://cython.org/release/Cython-0.13.rc1.tar.gz On Thu, Aug 19, 2010 at 1:48 AM, Robert Bradshaw wrote: > The Cython 0.13 release candidate is out at > http://cython.org/release/Cython-0.13.rc0.tar.gz . Unless there any > last-minute catastrophic failures, this will be the official release. > > - Robert > From stefan_ml at behnel.de Fri Aug 20 14:34:02 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Aug 2010 14:34:02 +0200 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? Message-ID: <4C6E763A.80602@behnel.de> Hi Haoyu, your GSoC branch has been around for a while now. Given that 0.13 will be out soon (soon-isher than ever before), is there anything that you consider safe enough to get merged into mainline after the release? IMHO, any feature patch is a candidate, especially when it depends on currently illegal syntax (i.e. not impacting existing code). Everything that changes current behaviour is worth considering if it fixes a bug or Python compatibility issue but may have to wait for 0.14 if it impacts code that currently works with 0.13. BTW, it looks like only two trac tickets are currently assigned to you. I know you've been working on a lot more tickets, so it would be good to have your name on all of those that you provided a patch for. http://trac.cython.org/cython_trac/query?owner=haoyu From a quick look, I think #488 (ellipsis) is safe, but I'd like to see some more syntax tests - I bet there are some in Python's own test suite. Ticket #487 (multiple 'with' statements) looks nice, but clearly lacks test. I can't even see a test that makes sure the context managers are executed in the correct nesting order, neither can I see anything that tests the chaining of cleanup actions during exception propagation, let alone partial propagation in cases where one of the context managers swallows an exception. Again, Python's test suite will provide hints on better tests here. Ticket #490 (nonlocal), while I'd be happy to get it in, seems too big a change to go into 0.13.1. It seems you also have a fix for #477. That would be another candidate for 0.13.1. Note that the related test case doesn't actually test that the argument typing has the expected effect. This could be done using a tree assertion based on coercion nodes. Stefan From dalcinl at gmail.com Fri Aug 20 16:24:50 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Aug 2010 11:24:50 -0300 Subject: [Cython] runtests.py --no-cleanup broken Message-ID: Can any of you look at this? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Aug 20 17:26:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Aug 2010 17:26:05 +0200 Subject: [Cython] runtests.py --no-cleanup broken In-Reply-To: References: Message-ID: <4C6E9E8D.2090908@behnel.de> Lisandro Dalcin, 20.08.2010 16:24: > Can any of you look at this? Works for me with cython-devel. What's "broken" on your side? Stefan From dalcinl at gmail.com Fri Aug 20 17:43:47 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Aug 2010 12:43:47 -0300 Subject: [Cython] runtests.py --no-cleanup broken In-Reply-To: <4C6E9E8D.2090908@behnel.de> References: <4C6E9E8D.2090908@behnel.de> Message-ID: On 20 August 2010 12:26, Stefan Behnel wrote: > Lisandro Dalcin, 20.08.2010 16:24: >> Can any of you look at this? > > Works for me with cython-devel. What's "broken" on your side? > I'm very sorry.. I'm doing last minute testing in Linux 64 with Intel compiler and OpenSolaris with the Sun Pro compiler. I got confused. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Fri Aug 20 19:20:32 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Aug 2010 14:20:32 -0300 Subject: [Cython] testsuite in very good shape Message-ID: In the following OS's/compilers combinations with Python 2.6, the testsuite runs cleanly. - Windows XP, MSVC - Linux 64, Intel Compiler (only C, no C++) - OpenSolaris 2009, SunPro compilers (only C, no C++) A test in OS X would be really nice before release... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Aug 20 19:27:17 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Aug 2010 19:27:17 +0200 Subject: [Cython] testsuite in very good shape In-Reply-To: References: Message-ID: <4C6EBAF5.7080908@behnel.de> Lisandro Dalcin, 20.08.2010 19:20: > In the following OS's/compilers combinations with Python 2.6, the > testsuite runs cleanly. > > - Windows XP, MSVC > - Linux 64, Intel Compiler (only C, no C++) > - OpenSolaris 2009, SunPro compilers (only C, no C++) Cool. Thanks for testing! > A test in OS X would be really nice before release... I would assume that Robert did that already. Stefan From kwmsmith at gmail.com Fri Aug 20 20:02:39 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 20 Aug 2010 13:02:39 -0500 Subject: [Cython] testsuite in very good shape In-Reply-To: <4C6EBAF5.7080908@behnel.de> References: <4C6EBAF5.7080908@behnel.de> Message-ID: On Fri, Aug 20, 2010 at 12:27 PM, Stefan Behnel wrote: > Lisandro Dalcin, 20.08.2010 19:20: >> In the following OS's/compilers combinations with Python 2.6, the >> testsuite runs cleanly. >> >> - Windows XP, MSVC >> - Linux 64, Intel Compiler (only C, no C++) >> - OpenSolaris 2009, SunPro compilers (only C, no C++) > > Cool. Thanks for testing! > > > ?> A test in OS X would be really nice before release... > > I would assume that Robert did that already. In case he didn't :-) The testsuite passes on OS X, 10.6.4, except for one. I'm fairly certain it's an issue on my box with multiple virtualenv environments. [178]$ ../bin/python runtests.py -vv --no-cleanup embed Running tests against Cython 0.13.rc1 Python 2.6.4 |CUSTOM| (r264:75706, Dec 11 2009, 10:58:54) [GCC 4.0.1 (Apple Inc. build 5465)] compiling (c) and running embedsignatures ... Doctest: embedsignatures ... ok compiling (cpp) and running embedsignatures ... Doctest: embedsignatures ... ok test_embed (__main__.EmbedTest) ... i686-apple-darwin10-gcc-4.2.1: Python.framework/Versions/6.0.1/Python: No such file or directory make: *** [embedded] Error 1 FAIL ====================================================================== FAIL: test_embed (__main__.EmbedTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 649, in test_embed "make PYTHON='%s' test > make.output" % sys.executable) == 0) AssertionError ---------------------------------------------------------------------- Ran 5 tests in 6.400s FAILED (failures=1) From robert.kern at gmail.com Fri Aug 20 20:53:45 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 20 Aug 2010 13:53:45 -0500 Subject: [Cython] testsuite in very good shape In-Reply-To: References: <4C6EBAF5.7080908@behnel.de> Message-ID: On 8/20/10 1:02 PM, Kurt Smith wrote: > On Fri, Aug 20, 2010 at 12:27 PM, Stefan Behnel wrote: >> Lisandro Dalcin, 20.08.2010 19:20: >>> In the following OS's/compilers combinations with Python 2.6, the >>> testsuite runs cleanly. >>> >>> - Windows XP, MSVC >>> - Linux 64, Intel Compiler (only C, no C++) >>> - OpenSolaris 2009, SunPro compilers (only C, no C++) >> >> Cool. Thanks for testing! >> >> >> > A test in OS X would be really nice before release... >> >> I would assume that Robert did that already. > > In case he didn't :-) > > The testsuite passes on OS X, 10.6.4, except for one. I'm fairly > certain it's an issue on my box with multiple virtualenv environments. > > [178]$ ../bin/python runtests.py -vv --no-cleanup embed > Running tests against Cython 0.13.rc1 > Python 2.6.4 |CUSTOM| (r264:75706, Dec 11 2009, 10:58:54) > [GCC 4.0.1 (Apple Inc. build 5465)] > > compiling (c) and running embedsignatures ... Doctest: embedsignatures ... ok > compiling (cpp) and running embedsignatures ... Doctest: embedsignatures ... ok > test_embed (__main__.EmbedTest) ... i686-apple-darwin10-gcc-4.2.1: > Python.framework/Versions/6.0.1/Python: No such file or directory Are you using EPD? It's possible that the configuration for the embedding test is not compatible with the way EPD lays out things. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robertwb at math.washington.edu Sat Aug 21 10:20:58 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 21 Aug 2010 01:20:58 -0700 Subject: [Cython] testsuite in very good shape In-Reply-To: <4C6EBAF5.7080908@behnel.de> References: <4C6EBAF5.7080908@behnel.de> Message-ID: On Fri, Aug 20, 2010 at 10:27 AM, Stefan Behnel wrote: > Lisandro Dalcin, 20.08.2010 19:20: >> In the following OS's/compilers combinations with Python 2.6, the >> testsuite runs cleanly. >> >> - Windows XP, MSVC >> - Linux 64, Intel Compiler (only C, no C++) >> - OpenSolaris 2009, SunPro compilers (only C, no C++) > > Cool. Thanks for testing! > > > ?> A test in OS X would be really nice before release... > > I would assume that Robert did that already. That would be correct. (OS X 10.6.3, Python 2.6) - Robert From stefan_ml at behnel.de Sat Aug 21 10:32:20 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Aug 2010 10:32:20 +0200 Subject: [Cython] testsuite in very good shape In-Reply-To: References: <4C6EBAF5.7080908@behnel.de> Message-ID: <4C6F8F14.3090101@behnel.de> Kurt Smith, 20.08.2010 20:02: > test_embed (__main__.EmbedTest) ... i686-apple-darwin10-gcc-4.2.1: > Python.framework/Versions/6.0.1/Python: No such file or directory > make: *** [embedded] Error 1 > FAIL > > ====================================================================== > FAIL: test_embed (__main__.EmbedTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "runtests.py", line 649, in test_embed > "make PYTHON='%s' test> make.output" % sys.executable) == 0) > AssertionError The embedding test is a bit fragile. It keeps failing in Python 2.3/4/5 in Hudson as well. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py25-c/lastCompletedBuild/testReport/%28root%29/EmbedTest/test_embed/ test_embed (__main__.EmbedTest) ... /usr/bin/ld: cannot find -lpython2.5 collect2: ld returned 1 exit status make: *** [embedded] Error 1 It seems to depend on a shared Python build, whereas the older Python builds do not use a shared library (which I think is the best way of building CPython for testing purposes). Any volunteer to make the test more robust? Stefan From dalcinl at gmail.com Mon Aug 23 15:33:33 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 23 Aug 2010 10:33:33 -0300 Subject: [Cython] testsuite in very good shape In-Reply-To: <4C6F8F14.3090101@behnel.de> References: <4C6EBAF5.7080908@behnel.de> <4C6F8F14.3090101@behnel.de> Message-ID: On 21 August 2010 05:32, Stefan Behnel wrote: > Kurt Smith, 20.08.2010 20:02: >> test_embed (__main__.EmbedTest) ... i686-apple-darwin10-gcc-4.2.1: >> Python.framework/Versions/6.0.1/Python: No such file or directory >> make: *** [embedded] Error 1 >> FAIL >> >> ====================================================================== >> FAIL: test_embed (__main__.EmbedTest) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ? ?File "runtests.py", line 649, in test_embed >> ? ? ?"make PYTHON='%s' test> ?make.output" % sys.executable) == 0) >> AssertionError > > The embedding test is a bit fragile. It keeps failing in Python 2.3/4/5 in > Hudson as well. > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py25-c/lastCompletedBuild/testReport/%28root%29/EmbedTest/test_embed/ > > ? test_embed (__main__.EmbedTest) ... > ? /usr/bin/ld: cannot find -lpython2.5 > ? collect2: ld returned 1 exit status > ? make: *** [embedded] Error 1 > > It seems to depend on a shared Python build, whereas the older Python > builds do not use a shared library (which I think is the best way of > building CPython for testing purposes). > Actually, the Makefile is really broken, it only works for system Python installs. > > Any volunteer to make the test more robust? > http://hg.cython.org/cython-devel/rev/c9a7095ef019 PS: I'm not sure about OS X. I do not expect framework Python builds to work. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From koepsell at gmail.com Mon Aug 23 20:02:57 2010 From: koepsell at gmail.com (killian koepsell) Date: Mon, 23 Aug 2010 11:02:57 -0700 Subject: [Cython] Preliminary SCons support In-Reply-To: <4ADECCA6.4000104@student.matnat.uio.no> References: <4ADECCA6.4000104@student.matnat.uio.no> Message-ID: Hi, I just found this thread which is a couple of months old. I would like to use scons with cython and C++ and the email below indicates that this is possible. Has there been any progress on this? Where can I find an example how to specify C++ as the language in scons? Thanks, Kilian On Wed, Oct 21, 2009 at 1:56 AM, Dag Sverre Seljebotn wrote: > OK, as requested I've uploaded the scons scripts I use. They're mostly > taken from David Cournapeau's numscons, but works with normal scons. > I'll start with a small user guide and then go to the TODO-list: > > 1) Copy Tools/site_scons to your project dir (sits in cython-devel hg trunk) > 2) Make SConstruct (not the nicest example, but...): > > env = Environment(PYEXT_USE_DISTUTILS=True) > # The above flag makes Python extensions look up compiler flags and > include paths etc. from distutils) > env.Tool("pyext") > env.Tool("cython") > env.Append(PYEXTINCPATH=[numpy.get_include()]) > # Override location of Cython and set some flags > env.Replace(CYTHON="python /uio/arkimedes/s07/dagss/cython/devel/cython.py", > ? ? ? ? ? ?CYTHONFLAGS=['-a', '-I/uio/arkimedes/s07/dagss/cmb/cmblib']) > > # Specify an extension to be compiled > env.PythonExtension('mapdatautils', ['mapdatautils.pyx']) > > # A more complicated example linking in Fortran code and libraries using > ifort. > # ifort must then be used to finally link the extension... > env2 = env.Environment(F90="ifort") > # Also omit -O2 and add Cython refnanny. This is hacky, see TODO. > env2.Replace(PYEXTCFLAGS=['-fno-strict-aliasing', '-DNDEBUG', '-Wall', > ? ? ? ? ? ? ? ? ? ? ? ? '-fwrapv', '-g', '-Wstrict-prototypes', > '-DCYTHON_REFNANNY'], > objs += env2.SharedObject(['comp_normalised_Plm.f90']) > objs += env2.PythonObject(['healpix.pyx']) # note the use of PythonObject > > env2.PythonExtension('healpix', objs, > ? ? ? ? ? ? ? ? ? ?LIBS=['healpix', 'cfitsio'], > ? ? ? ? ? ? ? ? ? ?PYEXTLINKFLAGS=['-shared', '-openmp'], > ? ? ? ? ? ? ? ? ? ?PYEXTLINK='$F90') > > > OK, on to the TODO-list: > ?- Currently the choice for extension module compilation is between > PYEXT_USE_DISTUTILS and a set of sensible defaults. Instead I think one > should be given the choice of a PYTHON to use, and then launch that > PYTHON executable to fetch distutils information. > ?- It should be possible to avoid the replacement of PYEXTCFLAGS above, > perhaps by seperating between optional (-O2) and non-optional > (-fno-strict-aliasing) flags and put them in seperate vars? > ?- The support for C++-generated Cython code is too weak (one has to > specify it in two seperate vars...not shown above) > ?- Dependency tracking (none at all for pyx files currently) > ?- I'm uncertain about having to specify PYEXT_USE_DISTUTILS (or, in > time, PYTHON) prior to loading the pyext tool...is this normal? Or > should the distutils settings be looked up at compilation time rather > than configuration time? (But then one wouldn't be able to dynamically > modify e.g. PYEXTCFLAGS, only "leave it unset to be filled in"...). > > Help and input appreciated, I know way too little scons to pull this off > the right way! But it does what I needed to for, and is already a lot > better than jumping between my shell scripts, makefiles and setup.py's... > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From ndbecker2 at gmail.com Mon Aug 23 13:50:33 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 23 Aug 2010 07:50:33 -0400 Subject: [Cython] Releasing Nuitka 0.1 References: <4C6CBAD2.6080003@gmx.de> Message-ID: What is it? From kayhayen at gmx.de Tue Aug 24 13:08:00 2010 From: kayhayen at gmx.de (Kay Hayen) Date: Tue, 24 Aug 2010 13:08:00 +0200 Subject: [Cython] Releasing Nuitka 0.1 In-Reply-To: References: <4C6CBAD2.6080003@gmx.de> Message-ID: <4C73A810.6070406@gmx.de> Hello Neal, Am 23.08.2010 13:50, schrieb Neal Becker: > What is it? It is a Python compiler. It takes Python source code of a module or a whole program, and translates it to C++ that is then using libpython to execute the code. It's similar or even an offspring to Cython, but with a different focus, namely on using pure Python and supporting all the language constructs, however dynamic they are. Currently it is quite new, but it passes the largest part of the CPython test suite, so it should be very compatible. Best regards, Kay Hayen From stefan_ml at behnel.de Tue Aug 24 13:17:34 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Aug 2010 13:17:34 +0200 Subject: [Cython] Releasing Nuitka 0.1 In-Reply-To: References: <4C6CBAD2.6080003@gmx.de> Message-ID: <4C73AA4E.1000207@behnel.de> Neal Becker, 23.08.2010 13:50: > What is it? It's a Python compiler that Kay has presented on this list in the past. Stefan From kayhayen at gmx.de Mon Aug 23 08:57:32 2010 From: kayhayen at gmx.de (Kay Hayen) Date: Mon, 23 Aug 2010 08:57:32 +0200 Subject: [Cython] Nuitka 0.1.1 (was Releasing Nuitka 0.1) In-Reply-To: <4C6CBAD2.6080003@gmx.de> References: <4C6CBAD2.6080003@gmx.de> Message-ID: <4C721BDC.3050300@gmx.de> Hello everybody, > If some minor things come up, there likely will be a 0.1.1 at the end of > August. After that I am moving to a new house and you know how time > consuming that can be. There is now a 0.1.1 release, which has some quirks fixed that showed up in 0.1, and some more work inside. Esp. the handling of exceptions and assertions has seen much improvement towards compatability. Now it has sys.exc_info() working as expected, so modules like traceback, do the correct thing too. I also merged the CPython tests as far as possible, and added support for a few smaller things too. The diff now seems almost exclusively caused by unicode repr troubles (I saw that Cython users encounter these too), having no function.func_code and otherwise very small things. Managing all these tests appears to be a very good test coverage. http://kayhayen24x7.homelinux.org/blog/2010/08/nuitka-minor-update/ I have started though to migrate it from "compiler" module which was the only thing available and usable at the time I started in the 2.5 time frame, to the "ast" module, which seems good starting 2.6. Once this is completed, likely this week, it's not a huge thing, the differences to "compiler" appear to be mostly improvements, then I will have a 0.2 with which I will do some serious benchmarking. Can you point me to existing benchmarks for Cython vs. CPython comparisons? I would like to start from there and extend what you have to this. But looking at http://hg.cython.org/cython-devel/file/979c14fc453e/tests didn't immediately put anything into my face. I noticed recently that Cython has had a performance regression (was it in Sage) that went unnoticed. So does that mean, you don't have run time speed performance regression tests? I currently only have "pystone" as a benchmark, which is a bit difficult to look at, because it's not good at telling you why it has that speed, aggregating everything into one. If you don't have anything, I will glady contribute something to call the performance regression tester. Ever since I starting to optimize with Valgrind, I wondered if it wouldn't be perfectly well suited to the task of noting a change in the total number of cycles to run test programs. What do you think? Yours, Kay Hayen From baihaoyu at gmail.com Tue Aug 24 16:11:23 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Tue, 24 Aug 2010 22:11:23 +0800 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: <4C6E763A.80602@behnel.de> References: <4C6E763A.80602@behnel.de> Message-ID: Hi, On Fri, Aug 20, 2010 at 8:34 PM, Stefan Behnel wrote: > Hi Haoyu, > > your GSoC branch has been around for a while now. Given that 0.13 will be > out soon (soon-isher than ever before), is there anything that you consider > safe enough to get merged into mainline after the release? > > IMHO, any feature patch is a candidate, especially when it depends on > currently illegal syntax (i.e. not impacting existing code). Everything that > changes current behaviour is worth considering if it fixes a bug or Python > compatibility issue but may have to wait for 0.14 if it impacts code that > currently works with 0.13. > > BTW, it looks like only two trac tickets are currently assigned to you. I > know you've been working on a lot more tickets, so it would be good to have > your name on all of those that you provided a patch for. > > http://trac.cython.org/cython_trac/query?owner=haoyu > I'm not able to assign tickets to me on the trac. Could you (or Robert?) give me the proper right to do that? Thanks! On the Rietveld code review, there's a list of patches I've worked on: http://codereview.appspot.com/user/Haoyu%20Bai > From a quick look, I think #488 (ellipsis) is safe, but I'd like to see some > more syntax tests - I bet there are some in Python's own test suite. > Looking into Python's test suite, one potential issue is Python recognize ". .." as syntax error while my implementation will still accept it as Ellipsis. Could we tolerate this kind of divergent? > Ticket #487 (multiple 'with' statements) looks nice, but clearly lacks test. > I can't even see a test that makes sure the context managers are executed in > the correct nesting order, neither can I see anything that tests the > chaining of cleanup actions during exception propagation, let alone partial > propagation in cases where one of the context managers swallows an > exception. Again, Python's test suite will provide hints on better tests > here. > Thanks. I just didn't aware there are so many aspects should be tested. I'll learn from Python's tests. Craig also pointed out the lacking of tests in the code review. I'll get the problem fixed. > Ticket #490 (nonlocal), while I'd be happy to get it in, seems too big a > change to go into 0.13.1. > Yes I agree. This might be too big a change. > It seems you also have a fix for #477. That would be another candidate for > 0.13.1. Note that the related test case doesn't actually test that the > argument typing has the expected effect. This could be done using a tree > assertion based on coercion nodes. > The fix for #477 is part of the big pure Python enhancement patch. Would it be worth to extract it out as a smaller patch for just this issue? And yes, to test argument typing what I did is to test whether the coercion is done by, eg. feed in the function an integer and then see if it returns a float. Checking for the coercion node is a better idea. I guess use the typeof() should also work? > Stefan > The other patches that could be considered to be included in the 0.13.1 release: #422 - bug in setting __module__: This is just bug-fixing. So should be safe to be included. #423 - Support explicit exception chaining syntax (PEP 3134): This may also lack of sufficient tests. #542 - Support for relative import: I'm not sure about this. Should be able to be included since the syntax is not valid before. Thanks! -- Haoyu BAI School of Computing, National University of Singapore. From stefan_ml at behnel.de Tue Aug 24 16:52:41 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Aug 2010 16:52:41 +0200 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: References: <4C6E763A.80602@behnel.de> Message-ID: <4C73DCB9.1050601@behnel.de> Haoyu Bai, 24.08.2010 16:11: > I'm not able to assign tickets to me on the trac. Could you (or > Robert?) give me the proper right to do that? Thanks! I'm not sure which rights you need for that, so I'll leave that to Robert. > On the Rietveld code review, there's a list of patches I've worked on: > > http://codereview.appspot.com/user/Haoyu%20Bai Ah, nice. >> From a quick look, I think #488 (ellipsis) is safe, but I'd like to see some >> more syntax tests - I bet there are some in Python's own test suite. > > Looking into Python's test suite, one potential issue is Python > recognize ". .." as syntax error while my implementation will still > accept it as Ellipsis. Could we tolerate this kind of divergent? IMHO, no. If Python rejects it and it's not Cython specific syntax, we should reject it as well. (also, when compiling .py files, any non-Python syntax should be rejected) >> Ticket #487 (multiple 'with' statements) looks nice, but clearly lacks test. >> I can't even see a test that makes sure the context managers are executed in >> the correct nesting order, neither can I see anything that tests the >> chaining of cleanup actions during exception propagation, let alone partial >> propagation in cases where one of the context managers swallows an >> exception. Again, Python's test suite will provide hints on better tests >> here. > > Thanks. I just didn't aware there are so many aspects should be > tested. I'll learn from Python's tests. > > Craig also pointed out the lacking of tests in the code review. I'll > get the problem fixed. Great. Many of those tests would have been nice to have before as they also apply to nested with statements, but now that we have a proper nesting syntax, this absolutely needs tests. >> It seems you also have a fix for #477. That would be another candidate for >> 0.13.1. Note that the related test case doesn't actually test that the >> argument typing has the expected effect. This could be done using a tree >> assertion based on coercion nodes. > > The fix for #477 is part of the big pure Python enhancement patch. > Would it be worth to extract it out as a smaller patch for just this > issue? I'm not a big fan of large cumulative patches. Smaller feature patches are easier to review and apply. That allows them to go in earlier, so they can start to receive user side testing. > And yes, to test argument typing what I did is to test whether > the coercion is done by, eg. feed in the function an integer and then > see if it returns a float. Checking for the coercion node is a better > idea. I guess use the typeof() should also work? I think so. > The other patches that could be considered to be included in the 0.13.1 release: > > #422 - bug in setting __module__: This is just bug-fixing. So should > be safe to be included. I didn't give it a review yet but it looks like a suitable candidate. > #423 - Support explicit exception chaining syntax (PEP 3134): This may > also lack of sufficient tests. > > #542 - Support for relative import: I'm not sure about this. Should be > able to be included since the syntax is not valid before. Both are pure feature additions, so, yes, if the code is ok, merging them into 0.13.1 should be fine. Stefan From robertwb at math.washington.edu Tue Aug 24 17:14:01 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Aug 2010 08:14:01 -0700 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: <4C73DCB9.1050601@behnel.de> References: <4C6E763A.80602@behnel.de> <4C73DCB9.1050601@behnel.de> Message-ID: On Tue, Aug 24, 2010 at 7:52 AM, Stefan Behnel wrote: > Haoyu Bai, 24.08.2010 16:11: >> I'm not able to assign tickets to me on the trac. Could you (or >> Robert?) give me the proper right to do that? Thanks! > > I'm not sure which rights you need for that, so I'll leave that to Robert. Done. >> On the Rietveld code review, there's a list of patches I've worked on: >> >> http://codereview.appspot.com/user/Haoyu%20Bai > > Ah, nice. Great work! >>> From a quick look, I think #488 (ellipsis) is safe, but I'd like to see some >>> more syntax tests - I bet there are some in Python's own test suite. >> >> Looking into Python's test suite, one potential issue is Python >> recognize ". .." as syntax error while my implementation will still >> accept it as Ellipsis. Could we tolerate this kind of divergent? > > IMHO, no. If Python rejects it and it's not Cython specific syntax, we > should reject it as well. (also, when compiling .py files, any non-Python > syntax should be rejected) Python 3 accepts a bare ellipsis, so I think we can leave it in. >>> It seems you also have a fix for #477. That would be another candidate for >>> 0.13.1. Note that the related test case doesn't actually test that the >>> argument typing has the expected effect. This could be done using a tree >>> assertion based on coercion nodes. >> >> The fix for #477 is part of the big pure Python enhancement patch. >> Would it be worth to extract it out as a smaller patch for just this >> issue? > > I'm not a big fan of large cumulative patches. Smaller feature patches are > easier to review and apply. That allows them to go in earlier, so they can > start to receive user side testing. +1 >> And yes, to test argument typing what I did is to test whether >> the coercion is done by, eg. feed in the function an integer and then >> see if it returns a float. Checking for the coercion node is a better >> idea. I guess use the typeof() should also work? > > I think so. I think typeof() is the most explicit and doesn't depend on implementation details. >> The other patches that could be considered to be included in the 0.13.1 release: >> >> #422 - bug in setting __module__: This is just bug-fixing. So should >> be safe to be included. > > I didn't give it a review yet but it looks like a suitable candidate. > > >> #423 - Support explicit exception chaining syntax (PEP 3134): This may >> also lack of sufficient tests. >> >> #542 - Support for relative import: I'm not sure about this. Should be >> able to be included since the syntax is not valid before. > > Both are pure feature additions, so, yes, if the code is ok, merging them > into 0.13.1 should be fine. +1 to both of these. I'm also very excited about http://codereview.appspot.com/1943048/ . Our pure mode documentation is in huge need of an update as well. Given that it's user facing, I wonder if some of that could be stored in docstrings on the Shadow.py functions. - Robert From robertwb at math.washington.edu Tue Aug 24 17:42:30 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Aug 2010 08:42:30 -0700 Subject: [Cython] testsuite in very good shape In-Reply-To: References: <4C6EBAF5.7080908@behnel.de> <4C6F8F14.3090101@behnel.de> Message-ID: On Mon, Aug 23, 2010 at 6:33 AM, Lisandro Dalcin wrote: > On 21 August 2010 05:32, Stefan Behnel wrote: >> Kurt Smith, 20.08.2010 20:02: >>> test_embed (__main__.EmbedTest) ... i686-apple-darwin10-gcc-4.2.1: >>> Python.framework/Versions/6.0.1/Python: No such file or directory >>> make: *** [embedded] Error 1 >>> FAIL >>> >>> ====================================================================== >>> FAIL: test_embed (__main__.EmbedTest) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> ? ?File "runtests.py", line 649, in test_embed >>> ? ? ?"make PYTHON='%s' test> ?make.output" % sys.executable) == 0) >>> AssertionError >> >> The embedding test is a bit fragile. It keeps failing in Python 2.3/4/5 in >> Hudson as well. >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py25-c/lastCompletedBuild/testReport/%28root%29/EmbedTest/test_embed/ >> >> ? test_embed (__main__.EmbedTest) ... >> ? /usr/bin/ld: cannot find -lpython2.5 >> ? collect2: ld returned 1 exit status >> ? make: *** [embedded] Error 1 >> >> It seems to depend on a shared Python build, whereas the older Python >> builds do not use a shared library (which I think is the best way of >> building CPython for testing purposes). >> > > Actually, the Makefile is really broken, it only works for system > Python installs. It works (or tries to work) with whatever Python it finds in your path. I've run it against the non-system Sage Python many times. >> Any volunteer to make the test more robust? >> > > http://hg.cython.org/cython-devel/rev/c9a7095ef019 Thanks! > PS: I'm not sure about OS X. I do not expect framework Python builds to work. Works for me (both with the System Python and with Sage's). - Robert From dalcinl at gmail.com Tue Aug 24 18:06:30 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Aug 2010 13:06:30 -0300 Subject: [Cython] testsuite in very good shape In-Reply-To: References: <4C6EBAF5.7080908@behnel.de> <4C6F8F14.3090101@behnel.de> Message-ID: On 24 August 2010 12:42, Robert Bradshaw wrote: > >> PS: I'm not sure about OS X. I do not expect framework Python builds to work. > > Works for me (both with the System Python and with Sage's). > Are these framework builds? What's the value of distutils.sysconfig.get_config_var('LINKFORSHARED') in both Python's? I think I still need to improve this. Brian Granger is using EPD Python and the test does not link in his Mac, because of of wrong LINKFORSHARED. PS: Robert, could you send me privately your Python's Makefile's? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From njs at pobox.com Tue Aug 24 18:48:00 2010 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 24 Aug 2010 09:48:00 -0700 Subject: [Cython] Nuitka 0.1.1 (was Releasing Nuitka 0.1) In-Reply-To: <4C721BDC.3050300@gmx.de> References: <4C6CBAD2.6080003@gmx.de> <4C721BDC.3050300@gmx.de> Message-ID: On Sun, Aug 22, 2010 at 11:57 PM, Kay Hayen wrote: > I currently only have "pystone" as a benchmark, which is a bit > difficult to look at, because it's not good at telling you why it has > that speed, aggregating everything into one. You might want to take advantage of the work the Unladen Swallow folks have put into benchmark tooling: http://code.google.com/p/unladen-swallow/wiki/Benchmarks -- Nathaniel From robertwb at math.washington.edu Tue Aug 24 19:00:12 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Aug 2010 10:00:12 -0700 Subject: [Cython] testsuite in very good shape In-Reply-To: References: <4C6EBAF5.7080908@behnel.de> <4C6F8F14.3090101@behnel.de> Message-ID: On Tue, Aug 24, 2010 at 9:06 AM, Lisandro Dalcin wrote: > On 24 August 2010 12:42, Robert Bradshaw wrote: >> >>> PS: I'm not sure about OS X. I do not expect framework Python builds to work. >> >> Works for me (both with the System Python and with Sage's). >> > > Are these framework builds? What's the value of > distutils.sysconfig.get_config_var('LINKFORSHARED') in both Python's? > I think I still need to improve this. Brian Granger is using EPD > Python and the test does not link in his Mac, because of of wrong > LINKFORSHARED. System Python (looks like a framework to me): >>> import distutils.sysconfig >>> distutils.sysconfig.get_config_var('LINKFORSHARED') '-u _PyMac_Error /System/Library/Frameworks/Python.framework/Versions/2.6/Python' Sage Python: >>> import distutils.sysconfig >>> distutils.sysconfig.get_config_var('LINKFORSHARED') '' > PS: Robert, could you send me privately your Python's Makefile's? Sure. - Robert From dsdale24 at gmail.com Tue Aug 24 22:40:57 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 24 Aug 2010 16:40:57 -0400 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: References: Message-ID: On Thu, Aug 19, 2010 at 5:05 AM, Robert Bradshaw wrote: > Ignore that last one, the release candidate is > http://cython.org/release/Cython-0.13.rc1.tar.gz > > On Thu, Aug 19, 2010 at 1:48 AM, Robert Bradshaw > wrote: >> The Cython 0.13 release candidate is out at >> http://cython.org/release/Cython-0.13.rc0.tar.gz . Unless there any >> last-minute catastrophic failures, this will be the official release. I just tried generating the c files for h5py (http://code.google.com/p/h5py/), and got an error I have not seen with previous cython versions: Error converting Pyrex file to C: ------------------------------------------------------------ ... hid_t plist, void *buf) except * ctypedef herr_t (*H5D_operator_t)(void *elem, hid_t type_id, unsigned ndim, hsize_t *point, void *operator_data) except -1 herr_t H5Diterate(void *buf, hid_t type_id, hid_t space_id, H5D_operator_t operator, void* operator_data) except * ^ ------------------------------------------------------------ /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator ',' not yet supported. I am not the maintainer of h5py, and my cython skills are underdeveloped, but I will try to provide additional information if needed. Cheers, Darren From robertwb at math.washington.edu Tue Aug 24 23:05:39 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Aug 2010 14:05:39 -0700 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: References: Message-ID: On Tue, Aug 24, 2010 at 1:40 PM, Darren Dale wrote: > On Thu, Aug 19, 2010 at 5:05 AM, Robert Bradshaw > wrote: >> Ignore that last one, the release candidate is >> http://cython.org/release/Cython-0.13.rc1.tar.gz >> >> On Thu, Aug 19, 2010 at 1:48 AM, Robert Bradshaw >> wrote: >>> The Cython 0.13 release candidate is out at >>> http://cython.org/release/Cython-0.13.rc0.tar.gz . Unless there any >>> last-minute catastrophic failures, this will be the official release. > > I just tried generating the c files for h5py > (http://code.google.com/p/h5py/), and got an error I have not seen > with previous cython versions: > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > ? ? ? ? ? ? ? ? ? ? ? ? ? ?hid_t plist, void *buf) except * > > ?ctypedef ?herr_t (*H5D_operator_t)(void *elem, hid_t type_id, unsigned ndim, > ? ? ? ? ? ? ? ? ? ?hsize_t *point, void *operator_data) except -1 > ?herr_t ? ?H5Diterate(void *buf, hid_t type_id, hid_t space_id, > ? ? ? ? ? ? ? ? ? ? ? ?H5D_operator_t operator, void* operator_data) except * > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ > ------------------------------------------------------------ > > /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator > ',' not yet supported. > > I am not the maintainer of h5py, and my cython skills are > underdeveloped, but I will try to provide additional information if > needed. I think I have an idea of where the problem lies, but it might be too late to fix it right now. A workaround is to rename your argument to something other than "operator." - Robert From greg.ewing at canterbury.ac.nz Wed Aug 25 02:34:13 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Aug 2010 12:34:13 +1200 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: References: Message-ID: <4C746505.2090107@canterbury.ac.nz> Darren Dale wrote: > herr_t H5Diterate(void *buf, hid_t type_id, hid_t space_id, > H5D_operator_t operator, void* operator_data) except * > ^ > ------------------------------------------------------------ > > /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator > ',' not yet supported. Looks like "operator" has become a keyword, and it thinks "operator," is referring to the C++ comma operator. -- Greg From robertwb at math.washington.edu Wed Aug 25 02:37:53 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Aug 2010 17:37:53 -0700 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: <4C746505.2090107@canterbury.ac.nz> References: <4C746505.2090107@canterbury.ac.nz> Message-ID: On Tue, Aug 24, 2010 at 5:34 PM, Greg Ewing wrote: > Darren Dale wrote: > >> ? herr_t ? ?H5Diterate(void *buf, hid_t type_id, hid_t space_id, >> ? ? ? ? ? ? ? ? ? ? ? ? H5D_operator_t operator, void* operator_data) except * >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ >> ------------------------------------------------------------ >> >> /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator >> ',' not yet supported. > > Looks like "operator" has become a keyword, and it thinks > "operator," is referring to the C++ comma operator. Yes, that is exactly the problem (and also the problem with namespaces in function arguments), the question is how to best fix it. - Robert From stefan_ml at behnel.de Wed Aug 25 07:30:45 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 07:30:45 +0200 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: References: <4C6E763A.80602@behnel.de> <4C73DCB9.1050601@behnel.de> Message-ID: <4C74AA85.40706@behnel.de> Robert Bradshaw, 24.08.2010 17:14: > On Tue, Aug 24, 2010 at 7:52 AM, Stefan Behnel wrote: >> Haoyu Bai, 24.08.2010 16:11: >>> Looking into Python's test suite, one potential issue is Python >>> recognize ". .." as syntax error while my implementation will still >>> accept it as Ellipsis. Could we tolerate this kind of divergent? >> >> IMHO, no. If Python rejects it and it's not Cython specific syntax, we >> should reject it as well. (also, when compiling .py files, any non-Python >> syntax should be rejected) > > Python 3 accepts a bare ellipsis, so I think we can leave it in. From what Haoyu wrote, I think he meant that the literal ". .." (mind the space character) would be accepted as "...". That should be rejected. Only "..." is a valid spelling for Ellipsis. I considered changing the scanner to merge consecutive sequences of "." into one literal. That might also help with relative imports. I didn't look into the details yet, though. > I'm also very excited about http://codereview.appspot.com/1943048/ . > Our pure mode documentation is in huge need of an update as well. > Given that it's user facing, I wonder if some of that could be stored > in docstrings on the Shadow.py functions. Yes, that's a good idea. Shadow.py is what users will see when they write their code, but it has no impact on the compiler itself. So that's a perfect place to put documentation. Stefan From stefan_ml at behnel.de Wed Aug 25 07:35:17 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 07:35:17 +0200 Subject: [Cython] benchmarking (was: Nuitka 0.1.1) In-Reply-To: References: <4C6CBAD2.6080003@gmx.de> <4C721BDC.3050300@gmx.de> Message-ID: <4C74AB95.5060906@behnel.de> Nathaniel Smith, 24.08.2010 18:48: > On Sun, Aug 22, 2010 at 11:57 PM, Kay Hayen wrote: >> I currently only have "pystone" as a benchmark, which is a bit >> difficult to look at, because it's not good at telling you why it has >> that speed, aggregating everything into one. > > You might want to take advantage of the work the Unladen Swallow folks > have put into benchmark tooling: > http://code.google.com/p/unladen-swallow/wiki/Benchmarks Yes, pystone and pybench are particularly bad benchmarks for translation to C code. Pybench even has a benchmark that simply assigns numbers to variables. When compiled into optimised C code, this becomes useless code that the C compiler discards. So the benchmark sometimes fails because it can happen that it runs a tiny bit faster than even the calibration loop. Stefan From stefan_ml at behnel.de Wed Aug 25 07:50:11 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 07:50:11 +0200 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: References: <4C746505.2090107@canterbury.ac.nz> Message-ID: <4C74AF13.2050901@behnel.de> Robert Bradshaw, 25.08.2010 02:37: > On Tue, Aug 24, 2010 at 5:34 PM, Greg Ewing wrote: >> Darren Dale wrote: >> >>> herr_t H5Diterate(void *buf, hid_t type_id, hid_t space_id, >>> H5D_operator_t operator, void* operator_data) except * >>> ^ >>> ------------------------------------------------------------ >>> >>> /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator >>> ',' not yet supported. >> >> Looks like "operator" has become a keyword, and it thinks >> "operator," is referring to the C++ comma operator. > > Yes, that is exactly the problem (and also the problem with namespaces > in function arguments), the question is how to best fix it. Well, the right fix, first of all, would be to disallow this kind of C++ syntax extensions in Cython code that is known to compile into C code (as opposed to C++ code), similar to what we do for .py files. Then, I'm pretty sure the above would not be a valid position for "operator," in C++. So, most likely, "operator" shouldn't be a keyword at all, but behave more like qualifiers, i.e. it should only be special cased in a specific syntactical context. If that can be fixed for 0.13 or a later minor version is another question. Stefan From robertwb at math.washington.edu Wed Aug 25 08:46:50 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Aug 2010 23:46:50 -0700 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: <4C74AF13.2050901@behnel.de> References: <4C746505.2090107@canterbury.ac.nz> <4C74AF13.2050901@behnel.de> Message-ID: On Tue, Aug 24, 2010 at 10:50 PM, Stefan Behnel wrote: > Robert Bradshaw, 25.08.2010 02:37: >> On Tue, Aug 24, 2010 at 5:34 PM, Greg Ewing wrote: >>> Darren Dale wrote: >>> >>>> ? ?herr_t ? ?H5Diterate(void *buf, hid_t type_id, hid_t space_id, >>>> ? ? ? ? ? ? ? ? ? ? ? ? ?H5D_operator_t operator, void* operator_data) except * >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ >>>> ------------------------------------------------------------ >>>> >>>> /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator >>>> ',' not yet supported. >>> >>> Looks like "operator" has become a keyword, and it thinks >>> "operator," is referring to the C++ comma operator. >> >> Yes, that is exactly the problem (and also the problem with namespaces >> in function arguments), the question is how to best fix it. > > Well, the right fix, first of all, would be to disallow this kind of C++ > syntax extensions in Cython code that is known to compile into C code (as > opposed to C++ code), similar to what we do for .py files. Then, I'm pretty > sure the above would not be a valid position for "operator," in C++. So, > most likely, "operator" shouldn't be a keyword at all, but behave more like > qualifiers, i.e. it should only be special cased in a specific syntactical > context. It is, the context in question is "c variable/function declaration" which needs to be narrowed. - Robert From dagss at student.matnat.uio.no Wed Aug 25 08:54:16 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 25 Aug 2010 08:54:16 +0200 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: References: <4C746505.2090107@canterbury.ac.nz> <4C74AF13.2050901@behnel.de> Message-ID: <4C74BE18.30802@student.matnat.uio.no> On 08/25/2010 08:46 AM, Robert Bradshaw wrote: > On Tue, Aug 24, 2010 at 10:50 PM, Stefan Behnel wrote: > >> Robert Bradshaw, 25.08.2010 02:37: >> >>> On Tue, Aug 24, 2010 at 5:34 PM, Greg Ewing wrote: >>> >>>> Darren Dale wrote: >>>> >>>> >>>>> herr_t H5Diterate(void *buf, hid_t type_id, hid_t space_id, >>>>> H5D_operator_t operator, void* operator_data) except * >>>>> ^ >>>>> ------------------------------------------------------------ >>>>> >>>>> /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator >>>>> ',' not yet supported. >>>>> >>>> Looks like "operator" has become a keyword, and it thinks >>>> "operator," is referring to the C++ comma operator. >>>> >>> Yes, that is exactly the problem (and also the problem with namespaces >>> in function arguments), the question is how to best fix it. >>> >> Well, the right fix, first of all, would be to disallow this kind of C++ >> syntax extensions in Cython code that is known to compile into C code (as >> opposed to C++ code), similar to what we do for .py files. Then, I'm pretty >> sure the above would not be a valid position for "operator," in C++. So, >> most likely, "operator" shouldn't be a keyword at all, but behave more like >> qualifiers, i.e. it should only be special cased in a specific syntactical >> context. >> > It is, the context in question is "c variable/function declaration" > which needs to be narrowed. > Can't you do it conditional on being in a "cdef cppclass" block? While not perfect, that at least gives us backwards compatability with currently working Cython code. Dag Sverre From robertwb at math.washington.edu Wed Aug 25 08:57:40 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Aug 2010 23:57:40 -0700 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: <4C74BE18.30802@student.matnat.uio.no> References: <4C746505.2090107@canterbury.ac.nz> <4C74AF13.2050901@behnel.de> <4C74BE18.30802@student.matnat.uio.no> Message-ID: On Tue, Aug 24, 2010 at 11:54 PM, Dag Sverre Seljebotn wrote: > On 08/25/2010 08:46 AM, Robert Bradshaw wrote: >> On Tue, Aug 24, 2010 at 10:50 PM, Stefan Behnel ?wrote: >> >>> Robert Bradshaw, 25.08.2010 02:37: >>> >>>> On Tue, Aug 24, 2010 at 5:34 PM, Greg Ewing wrote: >>>> >>>>> Darren Dale wrote: >>>>> >>>>> >>>>>> ? ? herr_t ? ?H5Diterate(void *buf, hid_t type_id, hid_t space_id, >>>>>> ? ? ? ? ? ? ? ? ? ? ? ? ? H5D_operator_t operator, void* operator_data) except * >>>>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ >>>>>> ------------------------------------------------------------ >>>>>> >>>>>> /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator >>>>>> ',' not yet supported. >>>>>> >>>>> Looks like "operator" has become a keyword, and it thinks >>>>> "operator," is referring to the C++ comma operator. >>>>> >>>> Yes, that is exactly the problem (and also the problem with namespaces >>>> in function arguments), the question is how to best fix it. >>>> >>> Well, the right fix, first of all, would be to disallow this kind of C++ >>> syntax extensions in Cython code that is known to compile into C code (as >>> opposed to C++ code), similar to what we do for .py files. Then, I'm pretty >>> sure the above would not be a valid position for "operator," in C++. So, >>> most likely, "operator" shouldn't be a keyword at all, but behave more like >>> qualifiers, i.e. it should only be special cased in a specific syntactical >>> context. >>> >> It is, the context in question is "c variable/function declaration" >> which needs to be narrowed. >> > > Can't you do it conditional on being in a "cdef cppclass" block? While > not perfect, that at least gives us backwards compatability with > currently working Cython code. Not easily, given the structure of the parser. As I mentioned, this is the same bug that causes C++ namespaces to get prepended to argument names (I think). Essentially, we need to make parsing function arguments and parsing variable declarations more distinct. The contexts are not granular enough. - Robert From kayhayen at gmx.de Wed Aug 25 08:59:39 2010 From: kayhayen at gmx.de (Kay Hayen) Date: Wed, 25 Aug 2010 08:59:39 +0200 Subject: [Cython] benchmarking In-Reply-To: <4C74AB95.5060906@behnel.de> References: <4C6CBAD2.6080003@gmx.de> <4C721BDC.3050300@gmx.de> <4C74AB95.5060906@behnel.de> Message-ID: <4C74BF5B.60807@gmx.de> Hello Nathaniel, Am 25.08.2010 07:35, schrieb Stefan Behnel: > Nathaniel Smith, 24.08.2010 18:48: >> On Sun, Aug 22, 2010 at 11:57 PM, Kay Hayen wrote: >>> I currently only have "pystone" as a benchmark, which is a bit >>> difficult to look at, because it's not good at telling you why it has >>> that speed, aggregating everything into one. >> >> You might want to take advantage of the work the Unladen Swallow folks >> have put into benchmark tooling: >> http://code.google.com/p/unladen-swallow/wiki/Benchmarks > > Yes, pystone and pybench are particularly bad benchmarks for translation to > C code. Pybench even has a benchmark that simply assigns numbers to > variables. When compiled into optimised C code, this becomes useless code > that the C compiler discards. So the benchmark sometimes fails because it > can happen that it runs a tiny bit faster than even the calibration loop. I have looked at these and I believe it may indeed be a realistic benchmark to see how fast "hg" starts, the kind of things that Google uses there make a lot of sense. I also like the statistically significant analysis, so you know if it's just noise or you really achieved something. And I think I will use it, or at least part of it, to get an overall feeling of how an application benefits from using a compiler. But keeping track of the amount of CPU cycles and memory like Valgrind does, seems to give a lot less noise from the start. plus I would use Valgrind anyway. The thing is, we want to see, when a certain function increases the number of cycles, because something is no longer discarded, or even worse, one more or slower call into CPython library is done. Most of the time its the implicit coercions that surprise you. The good thing about it is that - if done right - you need no timing loop at all. You can use very basic tests and need no loop to make these tests. The bad thing about that is that caches will not be measured. But to me that's a hard to measure secondary effect anyway, more important it is to avoid calls into Python and branches not needed. Yours, Kay From stefan_ml at behnel.de Wed Aug 25 09:02:19 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 09:02:19 +0200 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: References: <4C746505.2090107@canterbury.ac.nz> <4C74AF13.2050901@behnel.de> <4C74BE18.30802@student.matnat.uio.no> Message-ID: <4C74BFFB.6010602@behnel.de> Robert Bradshaw, 25.08.2010 08:57: > On Tue, Aug 24, 2010 at 11:54 PM, Dag Sverre Seljebotn wrote: >> Can't you do it conditional on being in a "cdef cppclass" block? While >> not perfect, that at least gives us backwards compatability with >> currently working Cython code. > > Not easily, given the structure of the parser. As I mentioned, this is > the same bug that causes C++ namespaces to get prepended to argument > names (I think). Essentially, we need to make parsing function > arguments and parsing variable declarations more distinct. The > contexts are not granular enough. Can't we put that information into the parser "Ctx" context? It currently doesn't even hold the information if we are parsing C or C++ ... Stefan From binet at cern.ch Wed Aug 25 08:57:08 2010 From: binet at cern.ch (Sebastien Binet) Date: Wed, 25 Aug 2010 08:57:08 +0200 Subject: [Cython] Releasing Nuitka 0.1 In-Reply-To: <4C73A810.6070406@gmx.de> References: <4C6CBAD2.6080003@gmx.de> <4C73A810.6070406@gmx.de> Message-ID: <877hjfqiwr.fsf@cern.ch> hi, On Tue, 24 Aug 2010 13:08:00 +0200, Kay Hayen wrote: > Hello Neal, > > Am 23.08.2010 13:50, schrieb Neal Becker: > > What is it? > > It is a Python compiler. It takes Python source code of a module or a > whole program, and translates it to C++ that is then using libpython to > execute the code. > > It's similar or even an offspring to Cython, but with a different focus, > namely on using pure Python and supporting all the language constructs, > however dynamic they are. > > Currently it is quite new, but it passes the largest part of the CPython > test suite, so it should be very compatible. don't take it wrong but, how different is that from ShedSkin[0] ? cheers, sebastien. [0] http://shed-skin.blogspot.com/ http://code.google.com/p/shedskin/ From kayhayen at gmx.de Wed Aug 25 09:09:18 2010 From: kayhayen at gmx.de (Kay Hayen) Date: Wed, 25 Aug 2010 09:09:18 +0200 Subject: [Cython] ShedSkin (was Releasing Nuitka 0.1) In-Reply-To: <877hjfqiwr.fsf@cern.ch> References: <4C6CBAD2.6080003@gmx.de> <4C73A810.6070406@gmx.de> <877hjfqiwr.fsf@cern.ch> Message-ID: <4C74C19E.2050800@gmx.de> Hello Sebastien, >> Currently it is quite new, but it passes the largest part of the CPython >> test suite, so it should be very compatible. > > don't take it wrong but, how different is that from ShedSkin[0] ? I looked at it, and it seems to use C++ STL instead if libpython, and has this constraint: "can translate pure, but implicitly statically typed Python programs into optimized C++". So Cython (and alikes) are about compiling Python to C, but "object" remains a legal type. Writing if x: a = None elif y: a = "112" else: a = 3 return a is not going to get you into any trouble, worst case is loss of performance, but not necessarily. Yours, Kay From stefan_ml at behnel.de Wed Aug 25 09:09:40 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 09:09:40 +0200 Subject: [Cython] Releasing Nuitka 0.1 In-Reply-To: <877hjfqiwr.fsf@cern.ch> References: <4C6CBAD2.6080003@gmx.de> <4C73A810.6070406@gmx.de> <877hjfqiwr.fsf@cern.ch> Message-ID: <4C74C1B4.4000808@behnel.de> Sebastien Binet, 25.08.2010 08:57: > On Tue, 24 Aug 2010 13:08:00 +0200, Kay Hayen wrote: >> Am 23.08.2010 13:50, schrieb Neal Becker: >>> What is it? >> >> It is a Python compiler. It takes Python source code of a module or a >> whole program, and translates it to C++ that is then using libpython to >> execute the code. >> >> It's similar or even an offspring to Cython, but with a different focus, >> namely on using pure Python and supporting all the language constructs, >> however dynamic they are. >> >> Currently it is quite new, but it passes the largest part of the CPython >> test suite, so it should be very compatible. > > don't take it wrong but, how different is that from ShedSkin[0] ? He explained that before. It's different because his code runs against the CPython runtime, whereas ShedSkin is an independent implementation of a restricted Python-like language. Stefan From binet at cern.ch Wed Aug 25 09:20:35 2010 From: binet at cern.ch (Sebastien Binet) Date: Wed, 25 Aug 2010 09:20:35 +0200 Subject: [Cython] ShedSkin (was Releasing Nuitka 0.1) In-Reply-To: <4C74C19E.2050800@gmx.de> References: <4C6CBAD2.6080003@gmx.de> <4C73A810.6070406@gmx.de> <877hjfqiwr.fsf@cern.ch> <4C74C19E.2050800@gmx.de> Message-ID: <871v9nqhto.fsf@cern.ch> Kay, (and Stefan) On Wed, 25 Aug 2010 09:09:18 +0200, Kay Hayen wrote: > > Hello Sebastien, > > >> Currently it is quite new, but it passes the largest part of the CPython > >> test suite, so it should be very compatible. > > > > don't take it wrong but, how different is that from ShedSkin[0] ? > > I looked at it, and it seems to use C++ STL instead of libpython, and > has this constraint: "can translate pure, but implicitly statically > typed Python programs into optimized C++". > > So Cython (and alikes) are about compiling Python to C, but "object" > remains a legal type. ah! ok, I see the not-so-subtle difference now :) cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From craigcitro at gmail.com Wed Aug 25 10:00:53 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 25 Aug 2010 01:00:53 -0700 Subject: [Cython] ANN: Cython 0.13 released! Message-ID: It is with *great* pleasure that I email to announce the release of Cython version 0.13! This release sets another milestone on the path towards Python compatibility and brings major new features and improvements for the usability of the Cython language. Download it here: http://cython.org/release/Cython-0.13.tar.gz == New Features == * Closures are fully supported for Python functions. Cython supports inner functions and lambda expressions. Generators and generator expressions are __not__ supported in this release. * Proper C++ support. Cython knows about C++ classes, templates and overloaded function signatures, so that Cython code can interact with them in a straight forward way. * Type inference is enabled by default for safe C types (e.g. double, bint, C++ classes) and known extension types. This reduces the need for explicit type declarations and can improve the performance of untyped code in some cases. There is also a verbose compile mode for testing the impact on user code. * Cython's for-in-loop can iterate over C arrays and sliced pointers. The type of the loop variable will be inferred automatically in this case. * The Py_UNICODE integer type for Unicode code points is fully supported, including for-loops and 'in' tests on unicode strings. It coerces from and to single character unicode strings. Note that untyped for-loop variables will automatically be inferred as Py_UNICODE when iterating over a unicode string. In most cases, this will be much more efficient than yielding sliced string objects, but can also have a negative performance impact when the variable is used in a Python context multiple times, so that it needs to coerce to a unicode string object more than once. If this happens, typing the loop variable as unicode or object will help. * The built-in functions any(), all(), sum(), list(), set() and dict() are inlined as plain `for` loops when called on generator expressions. Note that generator expressions are not generally supported apart from this feature. Also, tuple(genexpr) is not currently supported - use tuple([listcomp]) instead. * More shipped standard library declarations. The python_* and stdlib/stdio .pxd files have been deprecated in favor of clib.* and cpython[.*] and may get removed in a future release. == Python compatibility == * Pure Python mode no longer disallows non-Python keywords like 'cdef', 'include' or 'cimport'. It also no longer recognises syntax extensions like the for-from loop. * Parsing has improved for Python 3 syntax in Python code, although not all features are correctly supported. The missing Python 3 features are being worked on for the next release. * from __future__ import print_function is supported in Python 2.6 and later. Note that there is currently no emulation for earlier Python versions, so code that uses print() with this future import will require at least Python 2.6. * New compiler directive language_level (valid values: 2 or 3) with corresponding command line options -2 and -3 requests source code compatibility with Python 2.x or Python 3.x respectively. Language level 3 currently enforces unicode literals for unprefixed string literals, enables the print function (requires Python 2.6 or later) and keeps loop variables in list comprehensions from leaking. * Loop variables in set/dict comprehensions no longer leak into the surrounding scope (following Python 2.7). List comprehensions are unchanged in language level 2. == Incompatible changes == * The availability of type inference by default means that Cython will also infer the type of pointers on assignments. Previously, code like this cdef char* s = ... untyped_variable = s would convert the char* to a Python bytes string and assign that. This is no longer the case and no coercion will happen in the example above. The correct way of doing this is through an explicit cast or by typing the target variable, i.e. cdef char* s = ... untyped_variable1 = s untyped_variable2 = s cdef object py_object = s cdef bytes bytes_string = s * bool is no longer a valid type name by default. The problem is that it's not clear whether bool should refer to the Python type or the C++ type, and expecting one and finding the other has already led to several hard-to-find bugs. Both types are available for importing: you can use from cpython cimport bool for the Python bool type, and from libcpp cimport bool for the C++ type. == Contributors == Many people contributed to this release, including: * David Barnett * Stefan Behnel * Chuck Blake * Robert Bradshaw * Craig Citro * Bryan Cole * Lisandro Dalcin * Eric Firing * Danilo Freitas * Christoph Gohlke * Dag Sverre Seljebotn * Kurt Smith * Erik Tollerud * Carl Witty -cc From dagss at student.matnat.uio.no Wed Aug 25 10:09:21 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 25 Aug 2010 10:09:21 +0200 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: References: Message-ID: <4C74CFB1.9060705@student.matnat.uio.no> Craig Citro wrote: > It is with *great* pleasure that I email to announce the release of > Cython version 0.13! This release sets another milestone on the path > towards Python compatibility and brings major new features and > improvements for the usability of the Cython language. > > Download it here: http://cython.org/release/Cython-0.13.tar.gz > Thank you very much for such a thorough and informative release announcement! Dag Sverre From stefan_ml at behnel.de Wed Aug 25 10:11:12 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 10:11:12 +0200 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: References: Message-ID: <4C74D020.7060407@behnel.de> Craig Citro, 25.08.2010 10:00: > It is with *great* pleasure that I email to announce the release of > Cython version 0.13! Yay! :) > * bool is no longer a valid type name by default. The problem is that > it's not clear whether bool should refer to the Python type or the C++ > type, and expecting one and finding the other has already led to > several hard-to-find bugs. Both types are available for importing: you > can use from cpython cimport bool for the Python bool type, and from > libcpp cimport bool for the C++ type. Reading this, I noticed that cimporting 'bool' will break valid Python code like "bool(some_pyvalue)". However, given that the Python 'bool' type is so rarely used as a type declaration (mostly because Python's boolean coercion features are much too powerful to restrict values to True/False), I don't think this will pose problems in real life. I'm really happy that 0.13 is out there just the way it is. Stefan From stefan_ml at behnel.de Wed Aug 25 10:24:52 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 10:24:52 +0200 Subject: [Cython] sponsoring generators? (was: ANN: Cython 0.13 released!) In-Reply-To: References: Message-ID: <4C74D354.7000301@behnel.de> > * Closures are fully supported for Python functions. Cython supports > inner functions and lambda expressions. Generators and generator > expressions are __not__ supported in this release. Regarding this bit - is anyone interested in sponsoring the generator implementation? That would allow me to implement it in the next couple of months. The relevant CEP is up here: http://wiki.cython.org/enhancements/generators Stefan From Dave.Cross at cdl.co.uk Wed Aug 25 10:27:23 2010 From: Dave.Cross at cdl.co.uk (Dave Cross) Date: Wed, 25 Aug 2010 09:27:23 +0100 Subject: [Cython] sponsoring generators? (was: ANN: Cython 0.13 released!) In-Reply-To: <4C74D354.7000301@behnel.de> References: <4C74D354.7000301@behnel.de> Message-ID: My company might be. Have you any idea of cost/timescale? (I've just caught this message by chance so excuse me for not having researched the entire thread) Dave Cross > -----Original Message----- > From: cython-dev-bounces at codespeak.net [mailto:cython-dev- > bounces at codespeak.net] On Behalf Of Stefan Behnel > Sent: 25 August 2010 09:25 > To: cython-users at googlegroups.com; Cython-dev > Subject: [Cython] sponsoring generators? (was: ANN: Cython 0.13 released!) > > > * Closures are fully supported for Python functions. Cython supports > > inner functions and lambda expressions. Generators and generator > > expressions are __not__ supported in this release. > > Regarding this bit - is anyone interested in sponsoring the generator > implementation? That would allow me to implement it in the next couple of > months. > > The relevant CEP is up here: > > http://wiki.cython.org/enhancements/generators > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev ********************************************************************** Please consider the environment - do you really need to print this email? This email is intended only for the person(s) named above and may contain private and confidential information. If it has come to you in error, please destroy and permanently delete any copy in your possession and contact us on +44 (0) 161 480 4420. The information in this email is copyright ? CDL Group Holdings Limited. We cannot accept any liability for any loss or damage sustained as a result of software viruses. It is your responsibility to carry out such virus checking as is necessary before opening any attachment. Cheshire Datasystems Limited uses software which automatically screens incoming emails for inappropriate content and attachments. If the software identifies such content or attachment, the email will be forwarded to our Technology Department for checking. You should be aware that any email which you send to Cheshire Datasystems Limited is subject to this procedure. Cheshire Datasystems Limited, Strata House, Kings Reach Road, Stockport SK4 2HD Registered in England and Wales with Company Number 3991057 VAT registration: 727 1188 33 From greg.ewing at canterbury.ac.nz Wed Aug 25 12:45:33 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Aug 2010 22:45:33 +1200 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: <4C74AA85.40706@behnel.de> References: <4C6E763A.80602@behnel.de> <4C73DCB9.1050601@behnel.de> <4C74AA85.40706@behnel.de> Message-ID: <4C74F44D.3040509@canterbury.ac.nz> Stefan Behnel wrote: > From what Haoyu wrote, I think he meant that the literal ". .." (mind the > space character) would be accepted as "...". That should be rejected. Only > "..." is a valid spelling for Ellipsis. Python 2.x would actually accept three separate dots as an ellipsis, but 3.x no longer seems to do that, so it seems reasonable for Cython to follow suit. -- Greg From greg.ewing at canterbury.ac.nz Wed Aug 25 12:54:52 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 25 Aug 2010 22:54:52 +1200 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: <4C74D020.7060407@behnel.de> References: <4C74D020.7060407@behnel.de> Message-ID: <4C74F67C.70604@canterbury.ac.nz> > Craig Citro, 25.08.2010 10:00: > >> * bool is no longer a valid type name by default. ... > Both types are available for importing: What happened to the idea of treating bool like int? -- Greg From stefan_ml at behnel.de Wed Aug 25 13:05:41 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 13:05:41 +0200 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: <4C74F67C.70604@canterbury.ac.nz> References: <4C74D020.7060407@behnel.de> <4C74F67C.70604@canterbury.ac.nz> Message-ID: <4C74F905.6070401@behnel.de> Greg Ewing, 25.08.2010 12:54: >> Craig Citro, 25.08.2010 10:00: >> >>> * bool is no longer a valid type name by default. ... > > Both types are available for importing: > > What happened to the idea of treating bool like int? That's orthogonal. The case of 'bool' as a type only applies to type declarations and doesn't work "as expected" when dealing with C++ code because some users expect the C++ 'bool' type and others expect the Python type. Thus, 'bool' was removed as a Cython type and you have to be explicit about what you use. There are two places where 'bool' values are treated as ints: the literals True/False are treated as ints (or 'bint', actually), and users can declare variables as 'bint' explicitly. Both work nicely and match well with the 'bool' use cases. Stefan From ndbecker2 at gmail.com Wed Aug 25 16:32:28 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 25 Aug 2010 10:32:28 -0400 Subject: [Cython] this link appears to be broken Message-ID: http://cython.org/docs/cython-docs/build/latex/cython.pdf From dsdale24 at gmail.com Wed Aug 25 16:37:08 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 25 Aug 2010 10:37:08 -0400 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: <4C746505.2090107@canterbury.ac.nz> References: <4C746505.2090107@canterbury.ac.nz> Message-ID: On Tue, Aug 24, 2010 at 8:34 PM, Greg Ewing wrote: > Darren Dale wrote: > >> ? herr_t ? ?H5Diterate(void *buf, hid_t type_id, hid_t space_id, >> ? ? ? ? ? ? ? ? ? ? ? ? H5D_operator_t operator, void* operator_data) except * >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ >> ------------------------------------------------------------ >> >> /Users/darren/Projects/h5py/h5py/defs.pxd:190:49: Overloading operator >> ',' not yet supported. > > Looks like "operator" has become a keyword, and it thinks > "operator," is referring to the C++ comma operator. Thanks Greg. Changing "operator" to "operator_" seemed to do the trick. I notified the maintainer of h5py. Cheers, Darren From robertwb at math.washington.edu Wed Aug 25 17:42:25 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 08:42:25 -0700 Subject: [Cython] Cython 0.13 release candidate In-Reply-To: <4C74BFFB.6010602@behnel.de> References: <4C746505.2090107@canterbury.ac.nz> <4C74AF13.2050901@behnel.de> <4C74BE18.30802@student.matnat.uio.no> <4C74BFFB.6010602@behnel.de> Message-ID: On Wed, Aug 25, 2010 at 12:02 AM, Stefan Behnel wrote: > Robert Bradshaw, 25.08.2010 08:57: >> On Tue, Aug 24, 2010 at 11:54 PM, Dag Sverre Seljebotn wrote: >>> Can't you do it conditional on being in a "cdef cppclass" block? While >>> not perfect, that at least gives us backwards compatability with >>> currently working Cython code. >> >> Not easily, given the structure of the parser. As I mentioned, this is >> the same bug that causes C++ namespaces to get prepended to argument >> names (I think). Essentially, we need to make parsing function >> arguments and parsing variable declarations more distinct. The >> contexts are not granular enough. > > Can't we put that information into the parser "Ctx" context? Perhaps. > It currently doesn't even hold the information if we are parsing C or C++ ... This doesn't belong in Ctx as it is global to the module. - Robert From robertwb at math.washington.edu Wed Aug 25 17:43:23 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 08:43:23 -0700 Subject: [Cython] this link appears to be broken In-Reply-To: References: Message-ID: On Wed, Aug 25, 2010 at 7:32 AM, Neal Becker wrote: > http://cython.org/docs/cython-docs/build/latex/cython.pdf Thanks. We're in the process of doing a release and re-generating the docs, so hopefully that link will be up and working again soon. - Robert From craigcitro at gmail.com Wed Aug 25 18:37:59 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 25 Aug 2010 09:37:59 -0700 Subject: [Cython] this link appears to be broken In-Reply-To: References: Message-ID: > Thanks. We're in the process of doing a release and re-generating the > docs, so hopefully that link will be up and working again soon. > Yep -- our script doesn't rebuild the latex/pdf files by default, and trying to run that just now broke. I'm going to look into it some time today. -cc From robertwb at math.washington.edu Wed Aug 25 18:50:27 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 09:50:27 -0700 Subject: [Cython] this link appears to be broken In-Reply-To: References: Message-ID: On Wed, Aug 25, 2010 at 9:37 AM, Craig Citro wrote: >> Thanks. We're in the process of doing a release and re-generating the >> docs, so hopefully that link will be up and working again soon. >> > > Yep -- our script doesn't rebuild the latex/pdf files by default, and > trying to run that just now broke. :( > I'm going to look into it some time today. Thanks. - Robert From robertwb at math.washington.edu Wed Aug 25 18:58:21 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 09:58:21 -0700 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: <4C74D020.7060407@behnel.de> References: <4C74D020.7060407@behnel.de> Message-ID: On Wed, Aug 25, 2010 at 1:11 AM, Stefan Behnel wrote: > Craig Citro, 25.08.2010 10:00: >> It is with *great* pleasure that I email to announce the release of >> Cython version 0.13! > > Yay! :) Wohoo! Now onto 0.13.1. >> ? * bool is no longer a valid type name by default. The problem is that >> it's not clear whether bool should refer to the Python type or the C++ >> type, and expecting one and finding the other has already led to >> several hard-to-find bugs. Both types are available for importing: you >> can use from cpython cimport bool for the Python bool type, and from >> libcpp cimport bool for the C++ type. > > Reading this, I noticed that cimporting 'bool' will break valid Python code > like "bool(some_pyvalue)". However, given that the Python 'bool' type is so > rarely used as a type declaration (mostly because Python's boolean coercion > features are much too powerful to restrict values to True/False), I don't > think this will pose problems in real life. Cimporting bool from stdcpp will break this, but cimporting it from cpython will bring back the old behavior including bool(some_value) still working. What should work is cimporting bool from stdcpp and doing some_value. - Robert From robertwb at math.washington.edu Wed Aug 25 19:05:51 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 10:05:51 -0700 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: <4C74F905.6070401@behnel.de> References: <4C74D020.7060407@behnel.de> <4C74F67C.70604@canterbury.ac.nz> <4C74F905.6070401@behnel.de> Message-ID: On Wed, Aug 25, 2010 at 4:05 AM, Stefan Behnel wrote: > Greg Ewing, 25.08.2010 12:54: >>> Craig Citro, 25.08.2010 10:00: >>> >>>> ? * bool is no longer a valid type name by default. ... >> ? > ?Both types are available for importing: >> >> What happened to the idea of treating bool like int? > > That's orthogonal. The case of 'bool' as a type only applies to type > declarations and doesn't work "as expected" when dealing with C++ code > because some users expect the C++ 'bool' type and others expect the Python > type. Thus, 'bool' was removed as a Cython type and you have to be explicit > about what you use. It should also be noted that g++ happily accepted PyObject* (the pointer pointer) in a bool context and would convert bools into (PyObject*)0 and (PyObject*)1, > There are two places where 'bool' values are treated as ints: the literals > True/False are treated as ints (or 'bint', actually), and users can declare > variables as 'bint' explicitly. Both work nicely and match well with the > 'bool' use cases. Yes, we still have the bint which is still recommended for use (bool doesn't even exist for non-C++ code). It is semantically a bit different, as bint has many truth values (anything but 0) but bool has only one truth value (C++ automatically converts any non-zero value to 1). - Robert From robertwb at math.washington.edu Wed Aug 25 19:10:02 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 10:10:02 -0700 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: <4C74AA85.40706@behnel.de> References: <4C6E763A.80602@behnel.de> <4C73DCB9.1050601@behnel.de> <4C74AA85.40706@behnel.de> Message-ID: On Tue, Aug 24, 2010 at 10:30 PM, Stefan Behnel wrote: > Robert Bradshaw, 24.08.2010 17:14: >> On Tue, Aug 24, 2010 at 7:52 AM, Stefan Behnel wrote: >>> Haoyu Bai, 24.08.2010 16:11: >>>> Looking into Python's test suite, one potential issue is Python >>>> recognize ". .." as syntax error while my implementation will still >>>> accept it as Ellipsis. Could we tolerate this kind of divergent? >>> >>> IMHO, no. If Python rejects it and it's not Cython specific syntax, we >>> should reject it as well. (also, when compiling .py files, any non-Python >>> syntax should be rejected) >> >> Python 3 accepts a bare ellipsis, so I think we can leave it in. > > ?From what Haoyu wrote, I think he meant that the literal ". .." (mind the > space character) would be accepted as "...". That should be rejected. Only > "..." is a valid spelling for Ellipsis. Ah, I totally missed that. Sounds like another case of the -2 vs. -3 flag, where with -2 we should only reject it in -3 mode. - Robert From ndbecker2 at gmail.com Wed Aug 25 19:57:04 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 25 Aug 2010 13:57:04 -0400 Subject: [Cython] ANN: Cython 0.13 released! References: Message-ID: Is there an updated document to c++ support? From robertwb at math.washington.edu Wed Aug 25 20:00:07 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 11:00:07 -0700 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: References: Message-ID: On Wed, Aug 25, 2010 at 10:57 AM, Neal Becker wrote: > Is there an updated document to c++ support? Yep. http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html - Robert From dalcinl at gmail.com Wed Aug 25 20:28:43 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Aug 2010 15:28:43 -0300 Subject: [Cython] C string literals Message-ID: When trying to cythonize my code using the -3 flag, I got many errors like the one below: Error converting Pyrex file to C: ------------------------------------------------------------ ... if not (PetscInitializeCalled): return if (PetscFinalizeCalled): return # deinstall custom error handler ierr = PetscPopErrorHandlerPython() if ierr != 0: fprintf(stderr, "PetscPopErrorHandler() failed " ^ ------------------------------------------------------------ /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:307:24: Unicode literals do not support coercion to C types other than Py_UNICODE. Is this an oversight? The string literal is passed to a C function with a char* argument. Should we accept such usages when -3 is passed? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From ndbecker2 at gmail.com Wed Aug 25 20:40:51 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 25 Aug 2010 14:40:51 -0400 Subject: [Cython] ANN: Cython 0.13 released! References: Message-ID: Robert Bradshaw wrote: > On Wed, Aug 25, 2010 at 10:57 AM, Neal Becker > wrote: >> Is there an updated document to c++ support? > > Yep. http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html > > - Robert I would not have thought to look under 'Old Cython Users Guide' for new info on wrapping c++. From robertwb at math.washington.edu Wed Aug 25 20:44:21 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 11:44:21 -0700 Subject: [Cython] ANN: Cython 0.13 released! In-Reply-To: References: Message-ID: On Wed, Aug 25, 2010 at 11:40 AM, Neal Becker wrote: > Robert Bradshaw wrote: > >> On Wed, Aug 25, 2010 at 10:57 AM, Neal Becker >> wrote: >>> Is there an updated document to c++ support? >> >> Yep. http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html >> >> - Robert > > I would not have thought to look under 'Old Cython Users Guide' for new info > on wrapping c++. I actually just removed the "old" adjective, as that was an artifact of some (unfinished) docs restructuring. - Robert From dalcinl at gmail.com Wed Aug 25 21:00:11 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Aug 2010 16:00:11 -0300 Subject: [Cython] other issue with cython -3 Message-ID: $ cython -3 tmp.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef str a = "abc" ^ ------------------------------------------------------------ /u/dalcinl/tmp/tmp.pyx:1:13: Cannot convert Unicode string to 'str' implicitly. This is not portable and requires explicit encoding. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Wed Aug 25 21:15:54 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 21:15:54 +0200 Subject: [Cython] C string literals In-Reply-To: References: Message-ID: <4C756BEA.40800@behnel.de> Lisandro Dalcin, 25.08.2010 20:28: > When trying to cythonize my code using the -3 flag, I got many errors > like the one below: > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > if not (PetscInitializeCalled): return > if (PetscFinalizeCalled): return > # deinstall custom error handler > ierr = PetscPopErrorHandlerPython() > if ierr != 0: > fprintf(stderr, "PetscPopErrorHandler() failed " > ^ > ------------------------------------------------------------ > > /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:307:24: Unicode > literals do not support coercion to C types other than Py_UNICODE. Right, the parser reads the literal as unicode string here before type analysis figures out that it's really meant to be a bytes literal. This will be hard to change as recovering the original bytes literal is impossible once it's converted to a unicode string (remember that you can use arbitrary character escape sequences in the literal). So I'm leaning towards keeping this as an error. After all, Unicode string literals is one of the things that a user explicitly requests with the -3 switch. Stefan From stefan_ml at behnel.de Wed Aug 25 21:16:50 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 21:16:50 +0200 Subject: [Cython] other issue with cython -3 In-Reply-To: References: Message-ID: <4C756C22.6070308@behnel.de> Lisandro Dalcin, 25.08.2010 21:00: > $ cython -3 tmp.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef str a = "abc" > ^ > ------------------------------------------------------------ > > /u/dalcinl/tmp/tmp.pyx:1:13: Cannot convert Unicode string to 'str' > implicitly. This is not portable and requires explicit encoding. Same thing I said before: if you request unicode string literals, you get unicode literals. Stefan From stefan_ml at behnel.de Wed Aug 25 21:19:26 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 21:19:26 +0200 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: References: <4C6E763A.80602@behnel.de> <4C73DCB9.1050601@behnel.de> <4C74AA85.40706@behnel.de> Message-ID: <4C756CBE.3050000@behnel.de> Robert Bradshaw, 25.08.2010 19:10: > On Tue, Aug 24, 2010 at 10:30 PM, Stefan Behnel wrote: >> Robert Bradshaw, 24.08.2010 17:14: >>> On Tue, Aug 24, 2010 at 7:52 AM, Stefan Behnel wrote: >>>> Haoyu Bai, 24.08.2010 16:11: >>>>> Looking into Python's test suite, one potential issue is Python >>>>> recognize ". .." as syntax error while my implementation will still >>>>> accept it as Ellipsis. Could we tolerate this kind of divergent? >>>> >>>> IMHO, no. If Python rejects it and it's not Cython specific syntax, we >>>> should reject it as well. (also, when compiling .py files, any non-Python >>>> syntax should be rejected) >>> >>> Python 3 accepts a bare ellipsis, so I think we can leave it in. >> >> From what Haoyu wrote, I think he meant that the literal ". .." (mind the >> space character) would be accepted as "...". That should be rejected. Only >> "..." is a valid spelling for Ellipsis. > > Ah, I totally missed that. Sounds like another case of the -2 vs. -3 > flag, where with -2 we should only reject it in -3 mode. Seriously, I don't think Ellipsis is really important enough to inject special casing code into the Parser or Scanner. Even most people who use it regularly won't even know that you can write it as ". . ." in Py2. Stefan From robertwb at math.washington.edu Wed Aug 25 21:37:28 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 12:37:28 -0700 Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1? In-Reply-To: <4C756CBE.3050000@behnel.de> References: <4C6E763A.80602@behnel.de> <4C73DCB9.1050601@behnel.de> <4C74AA85.40706@behnel.de> <4C756CBE.3050000@behnel.de> Message-ID: On Wed, Aug 25, 2010 at 12:19 PM, Stefan Behnel wrote: > Robert Bradshaw, 25.08.2010 19:10: >> On Tue, Aug 24, 2010 at 10:30 PM, Stefan Behnel wrote: >>> Robert Bradshaw, 24.08.2010 17:14: >>>> On Tue, Aug 24, 2010 at 7:52 AM, Stefan Behnel wrote: >>>>> Haoyu Bai, 24.08.2010 16:11: >>>>>> Looking into Python's test suite, one potential issue is Python >>>>>> recognize ". .." as syntax error while my implementation will still >>>>>> accept it as Ellipsis. Could we tolerate this kind of divergent? >>>>> >>>>> IMHO, no. If Python rejects it and it's not Cython specific syntax, we >>>>> should reject it as well. (also, when compiling .py files, any non-Python >>>>> syntax should be rejected) >>>> >>>> Python 3 accepts a bare ellipsis, so I think we can leave it in. >>> >>> ? From what Haoyu wrote, I think he meant that the literal ". .." (mind the >>> space character) would be accepted as "...". That should be rejected. Only >>> "..." is a valid spelling for Ellipsis. >> >> Ah, I totally missed that. Sounds like another case of the -2 vs. -3 >> flag, where with -2 we should only reject it in -3 mode. > > Seriously, I don't think Ellipsis is really important enough to inject > special casing code into the Parser or Scanner. Even most people who use it > regularly won't even know that you can write it as ". . ." in Py2. I think accepting all valid Python code is an important goal--if it's not worth a special case than I'd rather accept invalid Py3 ellipsis than reject valid Py2 ellipsis. - Robert From dalcinl at gmail.com Wed Aug 25 21:57:54 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Aug 2010 16:57:54 -0300 Subject: [Cython] other issue with cython -3 In-Reply-To: <4C756C22.6070308@behnel.de> References: <4C756C22.6070308@behnel.de> Message-ID: On 25 August 2010 16:16, Stefan Behnel wrote: > Lisandro Dalcin, 25.08.2010 21:00: >> $ cython -3 tmp.pyx >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> cdef str a = "abc" >> ? ? ? ? ? ? ?^ >> ------------------------------------------------------------ >> >> /u/dalcinl/tmp/tmp.pyx:1:13: Cannot convert Unicode string to 'str' >> implicitly. This is not portable and requires explicit encoding. > > Same thing I said before: if you request unicode string literals, you get > unicode literals. > But in Python 3, the the Python-level 'str' type actually is an unicode string! So, cdef str a = "xyz" should definitely work. If not, once more, writing Cython code that target both Python 2 and 3 runtimes is a PITA. How should I write my code to get a byte string in Py2 and a unicode string in Py3? Yes, I know, for that to make sense the string should be pure ASCII, but that's a pretty common case (e.g. Python 3 stdlib is guaranteed to have all identifiers in the ASCII range) I'm likely missing something... To start: Is Cython -3 generated code supposed to work in a Python 2 runtime? If the answer is yes, I think we should provide mechanisms letting developers use cython -3 but still support Python 2 without too much extra work. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Wed Aug 25 22:09:34 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Aug 2010 22:09:34 +0200 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> Message-ID: <4C75787E.5070601@behnel.de> Lisandro Dalcin, 25.08.2010 21:57: > On 25 August 2010 16:16, Stefan Behnel wrote: >> Lisandro Dalcin, 25.08.2010 21:00: >>> $ cython -3 tmp.pyx >>> >>> Error converting Pyrex file to C: >>> ------------------------------------------------------------ >>> ... >>> cdef str a = "abc" >>> ^ >>> ------------------------------------------------------------ >>> >>> /u/dalcinl/tmp/tmp.pyx:1:13: Cannot convert Unicode string to 'str' >>> implicitly. This is not portable and requires explicit encoding. >> >> Same thing I said before: if you request unicode string literals, you get >> unicode literals. > > But in Python 3, the the Python-level 'str' type actually is an unicode string! Ah, right, I missed that. Yes, I think it makes sense to statically associate 'str' with 'unicode' on -3. BTW, note that '-3' is mostly meant for people who really want to write code that also runs in plain Python 3, i.e. for pure mode. Having it work on .pyx files is IMHO a nice feature that allows users to write their Cython code with the simpler Py3 syntax and semantics, but I doubt that it will be interesting for existing code, so the above can't currently break anything. It should get fixed for 0.13.1, though. Stefan From carl.witty at gmail.com Wed Aug 25 22:21:19 2010 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 25 Aug 2010 13:21:19 -0700 Subject: [Cython] C string literals In-Reply-To: <4C756BEA.40800@behnel.de> References: <4C756BEA.40800@behnel.de> Message-ID: On Wed, Aug 25, 2010 at 12:15 PM, Stefan Behnel wrote: > Lisandro Dalcin, 25.08.2010 20:28: >> When trying to cythonize my code using the -3 flag, I got many errors >> like the one below: >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> ? ? ?if not (PetscInitializeCalled): return >> ? ? ?if (PetscFinalizeCalled): return >> ? ? ?# deinstall custom error handler >> ? ? ?ierr = PetscPopErrorHandlerPython() >> ? ? ?if ierr != 0: >> ? ? ? ? ?fprintf(stderr, "PetscPopErrorHandler() failed " >> ? ? ? ? ? ? ? ? ? ? ? ? ^ >> ------------------------------------------------------------ >> >> /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:307:24: Unicode >> literals do not support coercion to C types other than Py_UNICODE. > > Right, the parser reads the literal as unicode string here before type > analysis figures out that it's really meant to be a bytes literal. > > This will be hard to change as recovering the original bytes literal is > impossible once it's converted to a unicode string (remember that you can > use arbitrary character escape sequences in the literal). So I'm leaning > towards keeping this as an error. After all, Unicode string literals is one > of the things that a user explicitly requests with the -3 switch. How about allowing it for ASCII literals and leaving it an error if there are any codepoints in the literal outside the 0-127 range? Carl From robertwb at math.washington.edu Wed Aug 25 22:23:37 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Aug 2010 13:23:37 -0700 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> Message-ID: On Wed, Aug 25, 2010 at 12:57 PM, Lisandro Dalcin wrote: > On 25 August 2010 16:16, Stefan Behnel wrote: >> Lisandro Dalcin, 25.08.2010 21:00: >>> $ cython -3 tmp.pyx >>> >>> Error converting Pyrex file to C: >>> ------------------------------------------------------------ >>> ... >>> cdef str a = "abc" >>> ? ? ? ? ? ? ?^ >>> ------------------------------------------------------------ >>> >>> /u/dalcinl/tmp/tmp.pyx:1:13: Cannot convert Unicode string to 'str' >>> implicitly. This is not portable and requires explicit encoding. >> >> Same thing I said before: if you request unicode string literals, you get >> unicode literals. >> > > But in Python 3, the the Python-level 'str' type actually ?is an unicode string! > > So, cdef str a = "xyz" should definitely work. If not, once more, > writing Cython code that target both Python 2 and 3 runtimes is a > PITA. How should I write my code to get a byte string in Py2 and a > unicode string in Py3? Yes, I know, for that to make sense the string > should be pure ASCII, but that's a pretty common case (e.g. Python 3 > stdlib is guaranteed to have all identifiers in the ASCII range) > > I'm likely missing something... To start: Is Cython -3 generated code > supposed to work in a Python 2 runtime? If the answer is yes, I think > we should provide mechanisms letting developers use cython -3 but > still support Python 2 without too much extra work. For one thing, the Python 3 print function is not supported for Python < 2.6 runtime environment. However, the -2 mode will support both the Py2 and Py3 runtimes for the foreseeable future, these flags are just about the input source (in the case that the two have different semantics). - Robert From dalcinl at gmail.com Wed Aug 25 22:26:50 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Aug 2010 17:26:50 -0300 Subject: [Cython] other issue with cython -3 In-Reply-To: <4C75787E.5070601@behnel.de> References: <4C756C22.6070308@behnel.de> <4C75787E.5070601@behnel.de> Message-ID: On 25 August 2010 17:09, Stefan Behnel wrote: > > Having it work > on .pyx files is IMHO a nice feature that allows users to write their > Cython code with the simpler Py3 syntax and semantics, but I doubt that it > will be interesting for existing code, so the above can't currently break > anything. What about people wanting to use Py3 syntax and semantics for pyx files, but still need to support Python 2 runtimes? Are you telling me that getattr(obj, "__doc__", None) will not work as expected with cython -3 under a Python 2 runtime? > > It should get fixed for 0.13.1, though. > -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed Aug 25 22:34:03 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Aug 2010 17:34:03 -0300 Subject: [Cython] C string literals In-Reply-To: References: <4C756BEA.40800@behnel.de> Message-ID: On 25 August 2010 17:21, Carl Witty wrote: > On Wed, Aug 25, 2010 at 12:15 PM, Stefan Behnel wrote: >> Lisandro Dalcin, 25.08.2010 20:28: >>> When trying to cythonize my code using the -3 flag, I got many errors >>> like the one below: >>> >>> Error converting Pyrex file to C: >>> ------------------------------------------------------------ >>> ... >>> ? ? ?if not (PetscInitializeCalled): return >>> ? ? ?if (PetscFinalizeCalled): return >>> ? ? ?# deinstall custom error handler >>> ? ? ?ierr = PetscPopErrorHandlerPython() >>> ? ? ?if ierr != 0: >>> ? ? ? ? ?fprintf(stderr, "PetscPopErrorHandler() failed " >>> ? ? ? ? ? ? ? ? ? ? ? ? ^ >>> ------------------------------------------------------------ >>> >>> /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:307:24: Unicode >>> literals do not support coercion to C types other than Py_UNICODE. >> >> Right, the parser reads the literal as unicode string here before type >> analysis figures out that it's really meant to be a bytes literal. >> >> This will be hard to change as recovering the original bytes literal is >> impossible once it's converted to a unicode string (remember that you can >> use arbitrary character escape sequences in the literal). So I'm leaning >> towards keeping this as an error. After all, Unicode string literals is one >> of the things that a user explicitly requests with the -3 switch. > > How about allowing it for ASCII literals and leaving it an error if > there are any codepoints in the literal outside the 0-127 range? > Definitely +1, your proposal is better than mine :-) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed Aug 25 22:50:50 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Aug 2010 17:50:50 -0300 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> Message-ID: On 25 August 2010 17:23, Robert Bradshaw wrote: > On Wed, Aug 25, 2010 at 12:57 PM, Lisandro Dalcin wrote: >> On 25 August 2010 16:16, Stefan Behnel wrote: >>> Lisandro Dalcin, 25.08.2010 21:00: >>>> $ cython -3 tmp.pyx >>>> >>>> Error converting Pyrex file to C: >>>> ------------------------------------------------------------ >>>> ... >>>> cdef str a = "abc" >>>> ? ? ? ? ? ? ?^ >>>> ------------------------------------------------------------ >>>> >>>> /u/dalcinl/tmp/tmp.pyx:1:13: Cannot convert Unicode string to 'str' >>>> implicitly. This is not portable and requires explicit encoding. >>> >>> Same thing I said before: if you request unicode string literals, you get >>> unicode literals. >>> >> >> But in Python 3, the the Python-level 'str' type actually ?is an unicode string! >> >> So, cdef str a = "xyz" should definitely work. If not, once more, >> writing Cython code that target both Python 2 and 3 runtimes is a >> PITA. How should I write my code to get a byte string in Py2 and a >> unicode string in Py3? Yes, I know, for that to make sense the string >> should be pure ASCII, but that's a pretty common case (e.g. Python 3 >> stdlib is guaranteed to have all identifiers in the ASCII range) >> >> I'm likely missing something... To start: Is Cython -3 generated code >> supposed to work in a Python 2 runtime? If the answer is yes, I think >> we should provide mechanisms letting developers use cython -3 but >> still support Python 2 without too much extra work. > > For one thing, the Python 3 print function is not supported for Python > < 2.6 runtime environment. However, the -2 mode will support both the > Py2 and Py3 runtimes for the foreseeable future, these flags are just > about the input source (in the case that the two have different > semantics). > This is from my own code (mpi4py): from array import array as arraytype ary = arraytype('i', [0]) Thanks to the special handling of string literals under -2, this code works (at runtime) in both Python 2 and Python 3... Howerver, when using cython -3, I get a failure under Python 2: TypeError: array() argument 1 must be char, not unicode -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Thu Aug 26 05:34:51 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Aug 2010 05:34:51 +0200 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> Message-ID: <4C75E0DB.8030101@behnel.de> Lisandro Dalcin, 25.08.2010 22:50: > This is from my own code (mpi4py): > > from array import array as arraytype > ary = arraytype('i', [0]) > > Thanks to the special handling of string literals under -2, this code > works (at runtime) in both Python 2 and Python 3... > > Howerver, when using cython -3, I get a failure under Python 2: > > TypeError: array() argument 1 must be char, not unicode That's a NumPy bug. Unicode strings should be valid string input also in Python 2. Stefan From stefan_ml at behnel.de Thu Aug 26 05:38:51 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Aug 2010 05:38:51 +0200 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> Message-ID: <4C75E1CB.70700@behnel.de> Robert Bradshaw, 25.08.2010 22:23: > For one thing, the Python 3 print function is not supported for Python > < 2.6 runtime environment. Haoyu has been working on a patch that backports it into Cython. http://trac.cython.org/cython_trac/ticket/69 Stefan From stefan_ml at behnel.de Thu Aug 26 05:55:30 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Aug 2010 05:55:30 +0200 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> <4C75787E.5070601@behnel.de> Message-ID: <4C75E5B2.9050807@behnel.de> Lisandro Dalcin, 25.08.2010 22:26: > On 25 August 2010 17:09, Stefan Behnel wrote: >> Having it work >> on .pyx files is IMHO a nice feature that allows users to write their >> Cython code with the simpler Py3 syntax and semantics, but I doubt that it >> will be interesting for existing code, so the above can't currently break >> anything. > > What about people wanting to use Py3 syntax and semantics for pyx > files, but still need to support Python 2 runtimes? Are we talking about concrete problems here? > Are you telling me > that getattr(obj, "__doc__", None) will not work as expected with > cython -3 under a Python 2 runtime? Works for me. Stefan From koepsell at gmail.com Thu Aug 26 07:58:24 2010 From: koepsell at gmail.com (killian koepsell) Date: Wed, 25 Aug 2010 22:58:24 -0700 Subject: [Cython] Preliminary SCons support In-Reply-To: References: <4ADECCA6.4000104@student.matnat.uio.no> Message-ID: Hi, I figured out how to make scons work with cython and C++ code and I thought I'd share it here in case other people are interested in the solution. It was pretty simple. All I had to do was to add the following three lines to Dag's original SConstruct file cited below: env.Replace(CYTHONFLAGS=['--cplus']) env.Replace(CYTHONCFILESUFFIX='.cpp') env.Replace(CXXFILESUFFIX='.cpp') Kilian On Mon, Aug 23, 2010 at 11:02 AM, killian koepsell wrote: > Hi, > > I just found this thread which is a couple of months old. > > I would like to use scons with cython and C++ and the email below > indicates that this is possible. > Has there been any progress on this? Where can I find an example how > to specify C++ as the language in scons? > > Thanks, > ?Kilian > > On Wed, Oct 21, 2009 at 1:56 AM, Dag Sverre Seljebotn > wrote: >> OK, as requested I've uploaded the scons scripts I use. They're mostly >> taken from David Cournapeau's numscons, but works with normal scons. >> I'll start with a small user guide and then go to the TODO-list: >> >> 1) Copy Tools/site_scons to your project dir (sits in cython-devel hg trunk) >> 2) Make SConstruct (not the nicest example, but...): >> >> env = Environment(PYEXT_USE_DISTUTILS=True) >> # The above flag makes Python extensions look up compiler flags and >> include paths etc. from distutils) >> env.Tool("pyext") >> env.Tool("cython") >> env.Append(PYEXTINCPATH=[numpy.get_include()]) >> # Override location of Cython and set some flags >> env.Replace(CYTHON="python /uio/arkimedes/s07/dagss/cython/devel/cython.py", >> ? ? ? ? ? ?CYTHONFLAGS=['-a', '-I/uio/arkimedes/s07/dagss/cmb/cmblib']) >> >> # Specify an extension to be compiled >> env.PythonExtension('mapdatautils', ['mapdatautils.pyx']) >> >> # A more complicated example linking in Fortran code and libraries using >> ifort. >> # ifort must then be used to finally link the extension... >> env2 = env.Environment(F90="ifort") >> # Also omit -O2 and add Cython refnanny. This is hacky, see TODO. >> env2.Replace(PYEXTCFLAGS=['-fno-strict-aliasing', '-DNDEBUG', '-Wall', >> ? ? ? ? ? ? ? ? ? ? ? ? '-fwrapv', '-g', '-Wstrict-prototypes', >> '-DCYTHON_REFNANNY'], >> objs += env2.SharedObject(['comp_normalised_Plm.f90']) >> objs += env2.PythonObject(['healpix.pyx']) # note the use of PythonObject >> >> env2.PythonExtension('healpix', objs, >> ? ? ? ? ? ? ? ? ? ?LIBS=['healpix', 'cfitsio'], >> ? ? ? ? ? ? ? ? ? ?PYEXTLINKFLAGS=['-shared', '-openmp'], >> ? ? ? ? ? ? ? ? ? ?PYEXTLINK='$F90') >> >> >> OK, on to the TODO-list: >> ?- Currently the choice for extension module compilation is between >> PYEXT_USE_DISTUTILS and a set of sensible defaults. Instead I think one >> should be given the choice of a PYTHON to use, and then launch that >> PYTHON executable to fetch distutils information. >> ?- It should be possible to avoid the replacement of PYEXTCFLAGS above, >> perhaps by seperating between optional (-O2) and non-optional >> (-fno-strict-aliasing) flags and put them in seperate vars? >> ?- The support for C++-generated Cython code is too weak (one has to >> specify it in two seperate vars...not shown above) >> ?- Dependency tracking (none at all for pyx files currently) >> ?- I'm uncertain about having to specify PYEXT_USE_DISTUTILS (or, in >> time, PYTHON) prior to loading the pyext tool...is this normal? Or >> should the distutils settings be looked up at compilation time rather >> than configuration time? (But then one wouldn't be able to dynamically >> modify e.g. PYEXTCFLAGS, only "leave it unset to be filled in"...). >> >> Help and input appreciated, I know way too little scons to pull this off >> the right way! But it does what I needed to for, and is already a lot >> better than jumping between my shell scripts, makefiles and setup.py's... >> >> Dag Sverre >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > From anders at embl.de Thu Aug 26 14:12:02 2010 From: anders at embl.de (Simon Anders) Date: Thu, 26 Aug 2010 14:12:02 +0200 Subject: [Cython] Calling 'add' on uninitialized set object Message-ID: <4C765A12.1040009@embl.de> Hi I just spotted an inconsistency, which might be a minor bug: If I declare a variable of type 'list' without initializing it and then try to use it, I get an error that my object is actually None, which makes sense: def test1(): cdef list l l.append( "A" ) $ python -c 'import test; test.test1()' Traceback (most recent call last): File "", line 1, in File "test.pyx", line 3, in test.test1 (test.c:396) l.append( "A" ) AttributeError: 'NoneType' object has no attribute 'append' However, if I do the same with a set, an internal error is triggered: def test2(): cdef set s s.add( "A" ) $ python -c 'import test; test.test2()' Traceback (most recent call last): File "", line 1, in File "test.pyx", line 7, in test.test2 (test.c:434) s.add( "A" ) SystemError: Objects/setobject.c:2303: bad argument to internal function I'm using Python 2.6.1 and Cython 0.13 on CentOS Linux 4.8 x86_64. Simon From dalcinl at gmail.com Thu Aug 26 15:21:00 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 26 Aug 2010 10:21:00 -0300 Subject: [Cython] other issue with cython -3 In-Reply-To: <4C75E0DB.8030101@behnel.de> References: <4C756C22.6070308@behnel.de> <4C75E0DB.8030101@behnel.de> Message-ID: On 26 August 2010 00:34, Stefan Behnel wrote: > Lisandro Dalcin, 25.08.2010 22:50: >> This is from my own code (mpi4py): >> >> from array import array as arraytype >> ary = arraytype('i', [0]) >> >> Thanks to the special handling of string literals under -2, this code >> works (at runtime) in both Python 2 and Python 3... >> >> Howerver, when using cython -3, I get a failure under Python 2: >> >> TypeError: array() argument 1 must be char, not unicode > > That's a NumPy bug. Unicode strings should be valid string input also in > Python 2. > No, it is not NumPy... it is the core Python stdlib array module in all 2.x versions. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Thu Aug 26 15:30:45 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 26 Aug 2010 10:30:45 -0300 Subject: [Cython] other issue with cython -3 In-Reply-To: <4C75E5B2.9050807@behnel.de> References: <4C756C22.6070308@behnel.de> <4C75787E.5070601@behnel.de> <4C75E5B2.9050807@behnel.de> Message-ID: On 26 August 2010 00:55, Stefan Behnel wrote: > Lisandro Dalcin, 25.08.2010 22:26: >> On 25 August 2010 17:09, Stefan Behnel wrote: >>> Having it work >>> on .pyx files is IMHO a nice feature that allows users to write their >>> Cython code with the simpler Py3 syntax and semantics, but I doubt that it >>> will be interesting for existing code, so the above can't currently break >>> anything. >> >> What about people wanting to use Py3 syntax and semantics for pyx >> files, but still need to support Python 2 runtimes? > > Are we talking about concrete problems here? > My concrete problem is that many API's out there do accept 'str', but fail to accept 'unicode'. The 'array' module is an instance of such API's. > >> Are you telling me >> that getattr(obj, "__doc__", None) will not work as expected with >> cython -3 under a Python 2 runtime? > > Works for me. > OK, you are right... because in Python 2 getattr(obj, u"__doc__", None) works as expected, getattr accept both str and unicode. I still think that we should provide a mechanism (likely a compiler directive, off by default) to run -3 but use the behavior of -2 for byte/str/unicode handling (that is, the 'str' type and bare "abc" string literals are PyString in Py2 and PyUnicode in Py3)... As it was suggested, we could even restrict this feature to string literals in the ASCII range. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Thu Aug 26 15:36:39 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Aug 2010 15:36:39 +0200 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> <4C75E0DB.8030101@behnel.de> Message-ID: <4C766DE7.1060905@behnel.de> Lisandro Dalcin, 26.08.2010 15:21: > On 26 August 2010 00:34, Stefan Behnel wrote: >> Lisandro Dalcin, 25.08.2010 22:50: >>> This is from my own code (mpi4py): >>> >>> from array import array as arraytype >>> ary = arraytype('i', [0]) >>> >>> Thanks to the special handling of string literals under -2, this code >>> works (at runtime) in both Python 2 and Python 3... >>> >>> Howerver, when using cython -3, I get a failure under Python 2: >>> >>> TypeError: array() argument 1 must be char, not unicode >> >> That's a NumPy bug. Unicode strings should be valid string input also in >> Python 2. > > No, it is not NumPy... it is the core Python stdlib array module in > all 2.x versions. Argh, I keep misreading imports, sorry ... Ok, it's a bug in the Py2 array module then. Looks like it hasn't even been fixed in 2.7. This is a difference between the runtime environments that you have to work around it in your own code. Seriously, if you want Python 3 semantics for your source code, you get Python 3 semantics for your source code. This has nothing to do with the array module. Cython simply cannot know that the array module fails to accept unicode input and requires bytes input instead. Imagine you did the same thing with a third party module that has the same bug. Would you want Cython to adapt to that, too? Stefan From stefan_ml at behnel.de Thu Aug 26 16:01:02 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Aug 2010 16:01:02 +0200 Subject: [Cython] other issue with cython -3 In-Reply-To: References: <4C756C22.6070308@behnel.de> <4C75787E.5070601@behnel.de> <4C75E5B2.9050807@behnel.de> Message-ID: <4C76739E.60902@behnel.de> Lisandro Dalcin, 26.08.2010 15:30: > On 26 August 2010 00:55, Stefan Behnel wrote: >> Lisandro Dalcin, 25.08.2010 22:26: >>> On 25 August 2010 17:09, Stefan Behnel wrote: >>>> Having it work >>>> on .pyx files is IMHO a nice feature that allows users to write their >>>> Cython code with the simpler Py3 syntax and semantics, but I doubt that it >>>> will be interesting for existing code, so the above can't currently break >>>> anything. >>> >>> What about people wanting to use Py3 syntax and semantics for pyx >>> files, but still need to support Python 2 runtimes? >> >> Are we talking about concrete problems here? > > My concrete problem is that many API's out there do accept 'str', but > fail to accept 'unicode'. The 'array' module is an instance of such > API's. Well, too bad. Then don't use unicode strings in your code when talking to them, or don't use the -3 option. > I still think that we should provide a mechanism (likely a compiler > directive, off by default) to run -3 but use the behavior of -2 for > byte/str/unicode handling (that is, the 'str' type and bare "abc" > string literals are PyString in Py2 and PyUnicode in Py3)... As it was > suggested, we could even restrict this feature to string literals in > the ASCII range. I keep reading "give me a compiler option, off by default" for string proposals over and over. "off by default" is not an excuse for bad design, or for blowing up the size of the language. Cython is not Perl. There *are* incompatibilities between Python 2 and Python 3 that Cython cannot hide. Accept that. It's not always easy to write portable code. If you write a literal in your code, Cython should interpret it in an easy to predict way. Adding options all over the place will make source code harder to read. How many options do you need to create the language "Python 3, except for feature A,B,C"? If you put a couple of compiler directives on the top of your file in order to tweak the language into your language, you have to remember them at any place in the file in order to understand that language. Cython is not C either. Preprocessors and global state are evil, not only for programs, but also for their human readers. Stefan From stefan_ml at behnel.de Thu Aug 26 16:48:39 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Aug 2010 16:48:39 +0200 Subject: [Cython] Calling 'add' on uninitialized set object In-Reply-To: <4C765A12.1040009@embl.de> References: <4C765A12.1040009@embl.de> Message-ID: <4C767EC7.8080109@behnel.de> Simon Anders, 26.08.2010 14:12: > If I declare a variable of type 'list' without initializing it and then > try to use it, I get an error that my object is actually None, which > makes sense: > > def test1(): > cdef list l > l.append( "A" ) > > $ python -c 'import test; test.test1()' > Traceback (most recent call last): > File "", line 1, in > File "test.pyx", line 3, in test.test1 (test.c:396) > l.append( "A" ) > AttributeError: 'NoneType' object has no attribute 'append' > > > However, if I do the same with a set, an internal error is triggered: > > def test2(): > cdef set s > s.add( "A" ) > > $ python -c 'import test; test.test2()' > Traceback (most recent call last): > File "", line 1, in > File "test.pyx", line 7, in test.test2 (test.c:434) > s.add( "A" ) > SystemError: Objects/setobject.c:2303: bad argument to internal function Right, the latter is based on the "old" way of handling builtin type methods. It lacks None tests by design. http://trac.cython.org/cython_trac/ticket/571 Thanks for the report. Stefan From vinjvinj at gmail.com Thu Aug 26 19:00:29 2010 From: vinjvinj at gmail.com (Vineet Jain) Date: Thu, 26 Aug 2010 13:00:29 -0400 Subject: [Cython] Advice with migrating app from python Message-ID: I have an app which allows users to write models in python. The app loads the python module dynamically, and check to see which functions the user has implemented and calls them form the worker. These user functions get called over many data sets. Scheduling Loop -> Model Container [python] -> User defined functions [python] I would like to change it to: Scheduling Loop -> Model Container [cython] -> User defined functions [cython] I would like all the user functions to be cdef and also most of the model container will be cdef functions except for the one that gets called from Scheduling Loop. I have a a couple of questions: 1. The user will submit the source in text through a webservices request. I know how to save the code to disk and compile it with cython. However, what I'm not clear on is how my cdef Model Container can figure out what functions the user has defined or not? cdef user_func1(): pass cdef user_func1(): pass The in the model container: - How would I load the model container dynamically? (is the same way that I do in python?) - How would I check if the loaded module has functions user_func1 and user_func2 defined? - How would I call them using the c interface from model container From ndbecker2 at gmail.com Fri Aug 27 13:32:47 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 27 Aug 2010 07:32:47 -0400 Subject: [Cython] some diffs to boost::python classes Message-ID: I've noticed that cython extension classes are more restrictive than boost::python classes. In particular, I believe boost::python allows attributes to be added by python code to boost::python classes, while cython classes are 'closed'. Is this correct? From stefan_ml at behnel.de Fri Aug 27 14:40:00 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Aug 2010 14:40:00 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: Message-ID: <4C77B220.1070100@behnel.de> Neal Becker, 27.08.2010 13:32: > I've noticed that cython extension classes are more restrictive than > boost::python classes. In particular, I believe boost::python allows > attributes to be added by python code to boost::python classes, while cython > classes are 'closed'. Is this correct? I don't know what kind of classes boost uses, but the above only applies to extension types in Cython. Normal Python classes behave as in normal Python. And it should be possible to add a __dict__ to extension types without much hassle, it's just that most people don't want that. Initialising a dict takes time during type instantiation and takes additional space during the lifetime of the object. BTW, now that you mention it - does anyone know what became of the idea to allow a cdef __dict__ attribute notation (as for __weakref__) to add a dict to an extension type? AFAIR, that was already proposed more than once on this list. Stefan From stefan_ml at behnel.de Fri Aug 27 14:53:22 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Aug 2010 14:53:22 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: Message-ID: <4C77B542.6020600@behnel.de> Neal Becker, 27.08.2010 13:32: > I've noticed that cython extension classes are more restrictive than > boost::python classes. In particular, I believe boost::python allows > attributes to be added by python code to boost::python classes, while cython > classes are 'closed'. Hmm, do you mean attributes of the type or attributes of their instances? Stefan From ndbecker2 at gmail.com Fri Aug 27 15:09:51 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 27 Aug 2010 09:09:51 -0400 Subject: [Cython] some diffs to boost::python classes References: <4C77B220.1070100@behnel.de> Message-ID: Stefan Behnel wrote: > Neal Becker, 27.08.2010 13:32: >> I've noticed that cython extension classes are more restrictive than >> boost::python classes. In particular, I believe boost::python allows >> attributes to be added by python code to boost::python classes, while >> cython >> classes are 'closed'. Is this correct? > > I don't know what kind of classes boost uses, but the above only applies > to extension types in Cython. Normal Python classes behave as in normal > Python. And it should be possible to add a __dict__ to extension types > without much hassle, it's just that most people don't want that. > Initialising a dict takes time during type instantiation and takes > additional space during the lifetime of the object. > > BTW, now that you mention it - does anyone know what became of the idea to > allow a > > cdef __dict__ > > attribute notation (as for __weakref__) to add a dict to an extension > type? AFAIR, that was already proposed more than once on this list. > > Stefan I just tested it. With boost::python you can add attribute to class as well as instance of class. Example: accumulator is a boost::python extension module of mine import pyublas import numpy as np from accumulator import * s = stat2nd_double dir(s) Out[20]: ['Mean', 'RMS', 'Var', '__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__iadd__', '__init__', '__instance_size__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'mean', 'rms', 'var'] s.stuff = 'hi' dir(s) Out[4]: ['Mean', 'RMS', 'Var', '__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__iadd__', '__init__', '__instance_size__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'count', 'mean', 'rms', 'stuff', 'var'] x = s() In [6]: dir(x) Out[6]: ['Mean', 'RMS', 'Var', '__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__iadd__', '__init__', '__instance_size__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'count', 'mean', 'rms', 'stuff', 'var'] In [7]: x.stuff Out[7]: 'hi' In [8]: x.junk='bye' In [9]: dir(x) Out[9]: ['Mean', 'RMS', 'Var', '__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__iadd__', '__init__', '__instance_size__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'count', 'junk', 'mean', 'rms', 'stuff', 'var'] In [10]: From dalcinl at gmail.com Fri Aug 27 16:00:35 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Aug 2010 11:00:35 -0300 Subject: [Cython] some diffs to boost::python classes In-Reply-To: <4C77B220.1070100@behnel.de> References: <4C77B220.1070100@behnel.de> Message-ID: On 27 August 2010 09:40, Stefan Behnel wrote: > Neal Becker, 27.08.2010 13:32: >> I've noticed that cython extension classes are more restrictive than >> boost::python classes. ?In particular, I believe boost::python allows >> attributes to be added by python code to boost::python classes, while cython >> classes are 'closed'. ?Is this correct? > > I don't know what kind of classes boost uses, but the above only applies to > extension types in Cython. Normal Python classes behave as in normal > Python. And it should be possible to add a __dict__ to extension types > without much hassle, it's just that most people don't want that. > Initialising a dict takes time during type instantiation and takes > additional space during the lifetime of the object. > > BTW, now that you mention it - does anyone know what became of the idea to > allow a > > ? ? cdef __dict__ > > attribute notation (as for __weakref__) to add a dict to an extension type? > AFAIR, that was already proposed more than once on this list. > I have a patch somewhere, I remember there were a few minor issues about it. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Aug 27 16:06:01 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Aug 2010 16:06:01 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> Message-ID: <4C77C649.5090003@behnel.de> Neal Becker, 27.08.2010 15:09: > Stefan Behnel wrote: >> Neal Becker, 27.08.2010 13:32: >>> I've noticed that cython extension classes are more restrictive than >>> boost::python classes. In particular, I believe boost::python allows >>> attributes to be added by python code to boost::python classes, while >>> cython >>> classes are 'closed'. Is this correct? >> >> I don't know what kind of classes boost uses, but the above only applies >> to extension types in Cython. Normal Python classes behave as in normal >> Python. And it should be possible to add a __dict__ to extension types >> without much hassle, it's just that most people don't want that. >> Initialising a dict takes time during type instantiation and takes >> additional space during the lifetime of the object. >> >> BTW, now that you mention it - does anyone know what became of the idea to >> allow a >> >> cdef __dict__ >> >> attribute notation (as for __weakref__) to add a dict to an extension >> type? AFAIR, that was already proposed more than once on this list. >> >> Stefan > > I just tested it. With boost::python you can add attribute to class as well > as instance of class. Could you send me the C++ code that it generates for the module? I checked what CPython does when accessing TheType.__dict__, and it actually returns a read-only dict proxy: static PyObject * type_dict(PyTypeObject *type, void *context) { if (type->tp_dict == NULL) { Py_INCREF(Py_None); return Py_None; } return PyDictProxy_New(type->tp_dict); } To work around that, we may have to override the descriptor that returns the __dict__ through the tp_getset slot of the type. Stefan From ndbecker2 at gmail.com Fri Aug 27 17:04:07 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 27 Aug 2010 11:04:07 -0400 Subject: [Cython] some diffs to boost::python classes References: <4C77B220.1070100@behnel.de> <4C77C649.5090003@behnel.de> Message-ID: Stefan Behnel wrote: > Neal Becker, 27.08.2010 15:09: >> Stefan Behnel wrote: >>> Neal Becker, 27.08.2010 13:32: >>>> I've noticed that cython extension classes are more restrictive than >>>> boost::python classes. In particular, I believe boost::python allows >>>> attributes to be added by python code to boost::python classes, while >>>> cython >>>> classes are 'closed'. Is this correct? >>> >>> I don't know what kind of classes boost uses, but the above only applies >>> to extension types in Cython. Normal Python classes behave as in normal >>> Python. And it should be possible to add a __dict__ to extension types >>> without much hassle, it's just that most people don't want that. >>> Initialising a dict takes time during type instantiation and takes >>> additional space during the lifetime of the object. >>> >>> BTW, now that you mention it - does anyone know what became of the idea >>> to allow a >>> >>> cdef __dict__ >>> >>> attribute notation (as for __weakref__) to add a dict to an extension >>> type? AFAIR, that was already proposed more than once on this list. >>> >>> Stefan >> >> I just tested it. With boost::python you can add attribute to class as >> well as instance of class. > > Could you send me the C++ code that it generates for the module? > > I checked what CPython does when accessing TheType.__dict__, and it > actually returns a read-only dict proxy: > > static PyObject * > type_dict(PyTypeObject *type, void *context) > { > if (type->tp_dict == NULL) { > Py_INCREF(Py_None); > return Py_None; > } > return PyDictProxy_New(type->tp_dict); > } > > To work around that, we may have to override the descriptor that returns > the __dict__ through the tp_getset slot of the type. > > Stefan boost::python is not a code generator. Is is (some of the most twisted, obfuscated ever!) native c++ code. From bblais at gmail.com Fri Aug 27 16:11:12 2010 From: bblais at gmail.com (Brian Blais) Date: Fri, 27 Aug 2010 10:11:12 -0400 Subject: [Cython] catch keyboardinterrupt? Message-ID: <25FB7556-FDA6-4E05-A2E6-7C284C422218@gmail.com> Hello, I have a loop like: try: for i in range(1000: do_something_with_a_loop_in_cython() except KeyboardInterrupt: cleanup() but unless I happen to hit Ctrl-C while it is in the python part, I get an interrupt ignore warning. Is there a way to catch the Ctrl-C to prematurely exit out of the cython loop? thanks, bb -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais http://bblais.blogspot.com/ From stefan_ml at behnel.de Fri Aug 27 17:20:38 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Aug 2010 17:20:38 +0200 Subject: [Cython] catch keyboardinterrupt? In-Reply-To: <25FB7556-FDA6-4E05-A2E6-7C284C422218@gmail.com> References: <25FB7556-FDA6-4E05-A2E6-7C284C422218@gmail.com> Message-ID: <4C77D7C6.5020009@behnel.de> Hi, note that this is a question for the cython-users mailing list. Brian Blais, 27.08.2010 16:11: > I have a loop like: > > try: > for i in range(1000: > do_something_with_a_loop_in_cython() > except KeyboardInterrupt: > cleanup() > > but unless I happen to hit Ctrl-C while it is in the python part, I > get an interrupt ignore warning. Is there a way to catch the Ctrl-C > to prematurely exit out of the cython loop? The C-API of CPython has functions to check for interrupts while executing C code. Stefan From robertwb at math.washington.edu Fri Aug 27 17:32:27 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Aug 2010 08:32:27 -0700 Subject: [Cython] catch keyboardinterrupt? In-Reply-To: <25FB7556-FDA6-4E05-A2E6-7C284C422218@gmail.com> References: <25FB7556-FDA6-4E05-A2E6-7C284C422218@gmail.com> Message-ID: On Fri, Aug 27, 2010 at 7:11 AM, Brian Blais wrote: > Hello, > > I have a loop like: > > try: > ? ? for i in range(1000: > ? ? ? ? do_something_with_a_loop_in_cython() > except KeyboardInterrupt: > ? ? cleanup() > > but unless I happen to hit Ctrl-C while it is in the python part, I > get an interrupt ignore warning. ?Is there a way to catch the Ctrl-C > to prematurely exit out of the cython loop? You can also set up your own signal handler like we do in Sage. http://hg.sagemath.org/sage-main/file/5b338f2e484f/c_lib/include/interrupt.h#l1 - Robert From stefan_ml at behnel.de Fri Aug 27 17:42:37 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Aug 2010 17:42:37 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> <4C77C649.5090003@behnel.de> Message-ID: <4C77DCED.6080500@behnel.de> Neal Becker, 27.08.2010 17:04: > boost::python is not a code generator. Is is (some of the most twisted, > obfuscated ever!) native c++ code. Ah, right. It actually seems to have dedicated code for that in libs/python/src/object/class.cpp static PyObject* instance_get_dict(PyObject* op, void*) { instance<>* inst = downcast >(op); if (inst->dict == 0) inst->dict = PyDict_New(); return python::xincref(inst->dict); } static int instance_set_dict(PyObject* op, PyObject* dict, void*) { instance<>* inst = downcast >(op); python::xdecref(inst->dict); inst->dict = python::incref(dict); return 0; } } static PyGetSetDef instance_getsets[] = { {const_cast("__dict__"), instance_get_dict, instance_set_dict, NULL, 0}, {0, 0, 0, 0, 0} }; Stefan From dalcinl at gmail.com Fri Aug 27 18:08:15 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Aug 2010 13:08:15 -0300 Subject: [Cython] some diffs to boost::python classes In-Reply-To: <4C77B220.1070100@behnel.de> References: <4C77B220.1070100@behnel.de> Message-ID: On 27 August 2010 09:40, Stefan Behnel wrote: > > BTW, now that you mention it - does anyone know what became of the idea to > allow a > > ? ? cdef __dict__ > > attribute notation (as for __weakref__) to add a dict to an extension type? > AFAIR, that was already proposed more than once on this list. > Here you have a tentative patch (use "hg import --no-commit ...") There is a small gotcha, tough... CPython makes an optimization and the instance dict in created on-demand. My patch unconditionally creates the dict at tp_new. If not, the dict slot in the object structure has to be set to NULL, but any usage of __dict__ in cdef methods will assume non-NULL and segfault. So better safe than sorry. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 -------------- next part -------------- A non-text attachment was scrubbed... Name: exttypedict.diff Type: text/x-patch Size: 7451 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100827/4ed7c7d5/attachment.bin From stefan_ml at behnel.de Fri Aug 27 18:50:49 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Aug 2010 18:50:49 +0200 Subject: [Cython] testsuite in very good shape In-Reply-To: References: <4C6EBAF5.7080908@behnel.de> <4C6F8F14.3090101@behnel.de> Message-ID: <4C77ECE9.9040501@behnel.de> Lisandro Dalcin, 23.08.2010 15:33: > On 21 August 2010 05:32, Stefan Behnel wrote: >> Kurt Smith, 20.08.2010 20:02: >>> test_embed (__main__.EmbedTest) ... i686-apple-darwin10-gcc-4.2.1: >>> Python.framework/Versions/6.0.1/Python: No such file or directory >>> make: *** [embedded] Error 1 >>> FAIL >>> >>> ====================================================================== >>> FAIL: test_embed (__main__.EmbedTest) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "runtests.py", line 649, in test_embed >>> "make PYTHON='%s' test> make.output" % sys.executable) == 0) >>> AssertionError >> >> The embedding test is a bit fragile. It keeps failing in Python 2.3/4/5 in >> Hudson as well. >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py25-c/lastCompletedBuild/testReport/%28root%29/EmbedTest/test_embed/ >> >> test_embed (__main__.EmbedTest) ... >> /usr/bin/ld: cannot find -lpython2.5 >> collect2: ld returned 1 exit status >> make: *** [embedded] Error 1 >> >> It seems to depend on a shared Python build, whereas the older Python >> builds do not use a shared library (which I think is the best way of >> building CPython for testing purposes). >> > > Actually, the Makefile is really broken, it only works for system > Python installs. I changed it to use a static build now (so that it works on non-shared CPython builds) and added some special casing to the test runner to make sure it finds the static library in all places where different CPython versions keep it. Fixes the test on Hudson. Stefan From stefan_ml at behnel.de Fri Aug 27 22:54:00 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Aug 2010 22:54:00 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> Message-ID: <4C7825E8.7000409@behnel.de> Lisandro Dalcin, 27.08.2010 18:08: > On 27 August 2010 09:40, Stefan Behnel wrote: >> >> BTW, now that you mention it - does anyone know what became of the idea to >> allow a >> >> cdef __dict__ >> >> attribute notation (as for __weakref__) to add a dict to an extension type? >> AFAIR, that was already proposed more than once on this list. >> > > Here you have a tentative patch (use "hg import --no-commit ...") > > There is a small gotcha, tough... CPython makes an optimization and > the instance dict in created on-demand. My patch unconditionally > creates the dict at tp_new. If not, the dict slot in the object > structure has to be set to NULL, but any usage of __dict__ in cdef > methods will assume non-NULL and segfault. Why is that? Does the external lookup code pass through the slot directly? I can't see that from your patch. Also, I don't see why this would be necessary. I don't think access to TheType.__dict__ or obj.__dict__ is commonly performance critical enough to require direct slot access. From my point of view, descriptor based access would be perfectly ok, and on-demand instantiation would work just fine with that. I think the right way to implement this is with custom decriptors for both the type and its instance. Stefan From paul.leopardi at iinet.net.au Fri Aug 27 23:02:34 2010 From: paul.leopardi at iinet.net.au (Paul Leopardi) Date: Sat, 28 Aug 2010 07:02:34 +1000 Subject: [Cython] Wrapping C++ using Cython v0.13 gives strange error messages Message-ID: <201008280702.34582.paul.leopardi@iinet.net.au> Hi all, I'm trying to convert my existing C++ wrapper to use Cython 0.13 and am getting strange error messages. In pyclical.h I have: ----- #include "glucat/glucat.h" typedef glucat::tuning<> Tune_P; #include "glucat/glucat_imp.h" #include #include #include using namespace glucat; using namespace std; typedef index_set IndexSet; typedef double scalar_t; typedef framed_multi Clifford; string index_set_to_repr(const IndexSet& ist) { ostringstream os; os << ist; return os.str(); } string index_set_to_str(const IndexSet& ist) { ostringstream os; os << ist; return os.str(); } [...] ----- In glucat/index_set.h I have: ----- namespace glucat { template class index_set; // forward /// Symmetric set difference: exclusive or template const index_set operator^ (const index_set& lhs, const index_set& rhs); /// Set intersection: and template const index_set operator& (const index_set& lhs, const index_set& rhs); /// Set union: or template const index_set operator| (const index_set& lhs, const index_set& rhs); /// Index set class based on std::bitset<> in Gnu standard C++ library template class index_set : private std::bitset { private: BOOST_STATIC_ASSERT((LO <= 0) && (0 <= HI) && (LO < HI) && \ (-LO < _GLUCAT_BITS_PER_ULONG) && \ ( HI < _GLUCAT_BITS_PER_ULONG) && \ ( HI-LO <= _GLUCAT_BITS_PER_ULONG)); typedef std::bitset bitset_t; typedef error error_t; public: typedef index_set index_set_t; typedef std::pair index_pair_t; static const index_t v_lo = LO; static const index_t v_hi = HI; static const std::string classname(); /// Default constructor creates an empty set index_set () { } /// Constructor from bitset_t index_set (const bitset_t bst); /// Constructor from index index_set (const index_t idx); [...] /// Constructor from string index_set (const std::string& str); /// Equality bool operator== (const index_set_t rhs) const; /// Inequality bool operator!= (const index_set_t rhs) const; /// Set complement: not index_set_t operator~ () const; /// Symmetric set difference: exclusive or index_set_t& operator^= (const index_set_t rhs); /// Set intersection: and index_set_t& operator&= (const index_set_t rhs); /// Set union: or index_set_t& operator|= (const index_set_t rhs); [...] ----- In pyclical.pyx I had: ----- cdef extern from "PyCliCal.h": [...] ctypedef struct c_index_set "IndexSet": int eq "operator=="(c_index_set rhs) c_index_set invert "operator~"() c_index_set *new_index_set_from_index_set "new IndexSet" (c_index_set ist) c_index_set *new_index_set_from_string "new IndexSet" (char* str) c_index_set *new_index_set_from_int "new IndexSet" (int ndx) void del_index_set "delete" (c_index_set *mv) c_index_set c_or "operator|"(c_index_set lhs, c_index_set rhs) c_index_set c_and "operator&"(c_index_set lhs, c_index_set rhs) c_index_set c_xor "operator^"(c_index_set lhs, c_index_set rhs) cpp_string index_set_to_repr(c_index_set ist) cpp_string index_set_to_str(c_index_set ist) ----- I now have: ----- cdef extern from "PyCliCal.h": [...] cdef cppclass IndexSet: int operator==(IndexSet rhs) IndexSet operator~() IndexSet (IndexSet ist) IndexSet (char* str) IndexSet (int ndx) IndexSet operator|(IndexSet lhs, IndexSet rhs) IndexSet operator&(IndexSet lhs, IndexSet rhs) IndexSet operator^(IndexSet lhs, IndexSet rhs) cpp_string index_set_to_repr(IndexSet ist) cpp_string index_set_to_str(IndexSet ist) void del_index_set "delete" (IndexSet *mv) ----- The output from cython 0.13 looks like: ----- cythoning PyCliCal.pyx to PyCliCal.cpp Error converting Pyrex file to C: ------------------------------------------------------------ ... int operator==(IndexSet rhs) IndexSet operator~() IndexSet (IndexSet ist) IndexSet (char* str) IndexSet (int ndx) IndexSet operator|(IndexSet lhs, IndexSet rhs) ^ ------------------------------------------------------------ /home/leopardi/src/Working/Working-0.5.1/glucat/pyclical/PyCliCal.pyx:17:4: 'IndexSet' is not a type identifier Error converting Pyrex file to C: ------------------------------------------------------------ ... int operator==(IndexSet rhs) IndexSet operator~() IndexSet (IndexSet ist) IndexSet (char* str) IndexSet (int ndx) IndexSet operator|(IndexSet lhs, IndexSet rhs) ^ ------------------------------------------------------------ /home/leopardi/src/Working/Working-0.5.1/glucat/pyclical/PyCliCal.pyx:17:23: 'IndexSet' is not a type identifier [...] ----- Why does IndexSet work within cppclass IndexSet but noot outside it? What am I doing wrong, and how do I fix it? Thanks, Paul From dalcinl at gmail.com Fri Aug 27 23:38:06 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Aug 2010 18:38:06 -0300 Subject: [Cython] Fwd: some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> Message-ID: Forgot to reply-to-all ... ---------- Forwarded message ---------- From: Lisandro Dalcin Date: 27 August 2010 18:36 Subject: Re: [Cython] some diffs to boost::python classes To: Stefan Behnel On 27 August 2010 17:54, Stefan Behnel wrote: > Lisandro Dalcin, 27.08.2010 18:08: >> >> On 27 August 2010 09:40, Stefan Behnel wrote: >>> >>> BTW, now that you mention it - does anyone know what became of the idea >>> to >>> allow a >>> >>> ? ? cdef __dict__ >>> >>> attribute notation (as for __weakref__) to add a dict to an extension >>> type? >>> AFAIR, that was already proposed more than once on this list. >>> >> >> Here you have a tentative patch (use "hg import --no-commit ...") >> >> There is a small gotcha, tough... CPython makes an optimization and >> the instance dict in created on-demand. My patch unconditionally >> creates the dict at tp_new. If not, the dict slot in the object >> structure has to be set to NULL, but any usage of __dict__ in cdef >> methods will assume non-NULL and segfault. > > Why is that? Does the external lookup code pass through the slot directly? I > can't see that from your patch. I do not understand you, and perhaps I was not clear enough. When I say that there is a gotcha, it actually means that CPython is a bit more optimized: if you create an instance but never set and attr nor access the __dict__, ?then the instance dict is never created and the object structure maintain a NULL in the slot. In my patch, the __dict__ slot is always filled with PyDict_New() at tp_new. > Also, I don't see why this would be > necessary. I don't think access to TheType.__dict__ or obj.__dict__ is > commonly performance critical enough to require direct slot access. From my > point of view, descriptor based access would be perfectly ok, and on-demand > instantiation would work just fine with that. > My current implementation permits Cython code to access the instance __dict__ from the object structure, like any other cdef attribute. Moreover, by using 'public' and 'readonly', cdef classes can decide how to expose instance dictionaries to Python code... You can see that in action in the testcase. > I think the right way to implement this is with custom decriptors for both > the type and its instance. > Do you mean adding a descriptor for __dict__ ? If you use cdef public|readonly __dict__, you get it. But Cython code will make a direct pointer access, like for any other cdef member... What's wrong with this approach? In short, I think the patch in its current status (modulo some possible oversight) is safe, Python code accessing instances will work as expected, Cython code can access dict safely and fast... The only improvement I can imagine right now is to hack on type inference to attach 'dict' type to __dict__ regardless of the user declaration, my patch does not includes that just because I do not know how to implement the feature. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Sat Aug 28 08:12:32 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 28 Aug 2010 08:12:32 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> Message-ID: <4C78A8D0.6020403@behnel.de> Lisandro Dalcin, 27.08.2010 23:38: >> I think the right way to implement this is with custom decriptors for both >> the type and its instance. > > Do you mean adding a descriptor for __dict__ ? If you use cdef > public|readonly __dict__, you get it. Right, I think __dict__ should be "public" by default, just like __weakref__ is always usable from outside. > But Cython code will make a > direct pointer access, like for any other cdef member... What's wrong > with this approach? Well, it wastes instance memory if __dict__ is not used. I would expect that most use cases for __dict__ do not come from Cython code but from Python code that uses the types. As I said, going through a descriptor would allow instantiating the dict at first request rather than *always*. That's important for library code. Potentially, we could also add a NULL check in the Cython access code wherever __dict__ is used, and go through the descriptor only if it's not set up yet. But that would be an optimisation. I think safe and memory friendly access should go first. > The only > improvement I can imagine right now is to hack on type inference to > attach 'dict' type to __dict__ regardless of the user declaration, my > patch does not includes that just because I do not know how to > implement the feature. Yes, I think it makes sense to always infer 'dict', especially when we go through the tp_dict* slots directly. Stefan From kayhayen at gmx.de Sat Aug 28 17:42:48 2010 From: kayhayen at gmx.de (Kay Hayen) Date: Sat, 28 Aug 2010 17:42:48 +0200 Subject: [Cython] Announcing Nuitka 0.2 Message-ID: <4C792E78.6030503@gmx.de> Hello, this is to inform you about the release 0.2 of Nuitka, my attempt of making a compiler that understands all of Python: http://kayhayen24x7.homelinux.org/blog/2010/08/release-nuitka-0-2/ Most noteworthy is that is passes now the almost complete CPython test suite nearly unchanged, provides support for all language constructs, etc. It's still not useful, although usable. Don't use it because it's faster, that may be a wrong statement at this time. And I have decided that Nuitka will live it's life independent of Cython, i.e. this is the last announcement of a release here. I will still follow the mailing list and will share my insights with you, but only where it's about Cython. I believe some benchmarking fun is ahead with future releases, that's for sure. For those of you that are interested in Nuitka, there is now a dedicated mailing list available: http://www.freelists.org/list/nuitka-dev Best regards, Kay Hayen From dalcinl at gmail.com Sun Aug 29 03:39:17 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 28 Aug 2010 22:39:17 -0300 Subject: [Cython] some diffs to boost::python classes In-Reply-To: <4C78A8D0.6020403@behnel.de> References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> <4C78A8D0.6020403@behnel.de> Message-ID: On 28 August 2010 03:12, Stefan Behnel wrote: > Lisandro Dalcin, 27.08.2010 23:38: >>> I think the right way to implement this is with custom decriptors for both >>> the type and its instance. >> >> Do you mean adding a descriptor for __dict__ ? If you use cdef >> public|readonly __dict__, you get it. > > Right, I think __dict__ should be "public" by default, just like > __weakref__ is always usable from outside. > Do you think that "cdef private __dict__" would have any use? We cannot write "private"... > >> But Cython code will make a >> direct pointer access, like for any other cdef member... What's wrong >> with this approach? > > Well, it wastes instance memory if __dict__ is not used. I would expect > that most use cases for __dict__ do not come from Cython code but from > Python code that uses the types. As I said, going through a descriptor > would allow instantiating the dict at first request rather than *always*. > That's important for library code. > > Potentially, we could also add a NULL check in the Cython access code > wherever __dict__ is used, and go through the descriptor only if it's not > set up yet. But that would be an optimisation. I think safe and memory > friendly access should go first. > On one hand, I think that who adds a __dict__ to cdef classes do know the trade-offs, as __dict__ is not provided unless explicitly requested, I do not consider a big deal to unconditionally create the __dict__ and have fast access, we can optimize things later. My current patch do work already, users needing this could start using the feature in 0.13.1 On the other hand, I do not care too much if this is implemented going through a descriptor, but as I'm not going to use this feature anyway. This implementations have to be written (and certainly not by me, I have no clear idea how to use cdef __dict__ and make Cython go through the descriptor). It is hard follow the criteria of core Cython developers and contributors :-)... For some things, they like it fast despite being unsafe, for others this it is the other way around, some idiot guys as for compiler directives controlling every detail... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Sun Aug 29 09:36:36 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Aug 2010 09:36:36 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> <4C78A8D0.6020403@behnel.de> Message-ID: <4C7A0E04.6030603@behnel.de> Lisandro Dalcin, 29.08.2010 03:39: > On 28 August 2010 03:12, Stefan Behnel wrote: >> Lisandro Dalcin, 27.08.2010 23:38: >>>> I think the right way to implement this is with custom decriptors for both >>>> the type and its instance. >>> >>> Do you mean adding a descriptor for __dict__ ? If you use cdef >>> public|readonly __dict__, you get it. >> >> Right, I think __dict__ should be "public" by default, just like >> __weakref__ is always usable from outside. > > Do you think that "cdef private __dict__" would have any use? A usecase might be a type that allows arbitrary attribute changes from C code but not from Python code. Not completely unrealistic IMHO. It might be possible to actually provide such semantics by letting the descriptor raise an exception on access and passing through tp_dict* from Cython code. However, I'm not sure CPython goes through the descriptor in all cases when accessing tp_dict*, so it's left to be seen how well this would actually work. But I think the decision can be taken at a later time regardless of what we do now, so making __dict__ public by default seems the most reasonable thing to do. > We cannot write "private"... I know. That's another issue that we should solve one day. There should be a way to say: "this is not part of the public API of this module", also for extension types. http://trac.cython.org/cython_trac/ticket/263 >>> But Cython code will make a >>> direct pointer access, like for any other cdef member... What's wrong >>> with this approach? >> >> Well, it wastes instance memory if __dict__ is not used. I would expect >> that most use cases for __dict__ do not come from Cython code but from >> Python code that uses the types. As I said, going through a descriptor >> would allow instantiating the dict at first request rather than *always*. >> That's important for library code. >> >> Potentially, we could also add a NULL check in the Cython access code >> wherever __dict__ is used, and go through the descriptor only if it's not >> set up yet. But that would be an optimisation. I think safe and memory >> friendly access should go first. > > On one hand, I think that who adds a __dict__ to cdef classes do know > the trade-offs, as __dict__ is not provided unless explicitly > requested, I do not consider a big deal to unconditionally create the > __dict__ and have fast access, we can optimize things later. My > current patch do work already, users needing this could start using > the feature in 0.13.1 You're right, memory efficiency is also just a kind of optimisation, and the new feature doesn't impact existing code. However, it's already known that the "fast access" may have to be disabled later on when applying that optimisation or may at least receive a slight performance hit due to the NULL check and the potentially multiplicated lazy initialisation code. So code that gets written now that relies on the now-to-be-set behaviour may behave slightly different later on. I think I'm ok with applying your patch to get the feature working now, although I still think that the memory hit and the impact on type instantiation time is a much more important factor than the access performance. If users wanted fast access from their Cython code, they'd use static cdef attributes in the first place. > On the other hand, I do not care too much if this is implemented going > through a descriptor, but as I'm not going to use this feature anyway. I think that describes the main use case already: "I won't use it in my code, but the users of my code *may* want to have it". That's the main reason why I'm advocating a low impact for the unused case. Note that your patch only solves half of the problem that Neal describes: it implements __dict__ for instances but not for types. I wouldn't care so much about the memory impact for a type dict, but there are usually a lot more instances than types, and if only a couple of instances actively use the __dict__, then the memory overhead of the instances that do not use it will be important. For example, lxml will definitely not use this for its main API type in the current state, although it would be a really nice addition. Even adding an empty dict pointer (4/8 bytes) to each of the instances is worth some careful consideration when the number of instances can become huge. But the real killer argument here is the performance hit on instantiation which happens tons of times all over the place in lxml. BTW, the patch needs to be revised when we implement support for subtyping PyVarObject (bytes/tuple). http://trac.cython.org/cython_trac/ticket/152 > It is hard follow the criteria of core Cython developers and > contributors :-)... For some things, they like it fast despite being > unsafe, for others this it is the other way around, some idiot guys as > for compiler directives controlling every detail... I can see that this may appear arbitrary at times. I hope I made it a bit clearer above why I think that this particular patch provides an implementation trade-off that doesn't match well with the most common use case. It just doesn't make sense to think about optimisations without a couple of important use cases in mind, especially when there *are* drawbacks. Stefan From dalcinl at gmail.com Sun Aug 29 18:13:39 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Aug 2010 13:13:39 -0300 Subject: [Cython] some diffs to boost::python classes In-Reply-To: <4C7A0E04.6030603@behnel.de> References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> <4C78A8D0.6020403@behnel.de> <4C7A0E04.6030603@behnel.de> Message-ID: On 29 August 2010 04:36, Stefan Behnel wrote: > Lisandro Dalcin, 29.08.2010 03:39: > >> On the other hand, I do not care too much if this is implemented going >> through a descriptor, but as I'm not going to use this feature anyway. > > I think that describes the main use case already: "I won't use it in my > code, but the users of my code *may* want to have it". That's the main > reason why I'm advocating a low impact for the unused case. > If users want a dict in their instances, then use a subclass. That the way to have to do it for built-in types like list or dict. Take into account that adding a __dict__ will waste a pointer in your object structure. > Note that your patch only solves half of the problem that Neal describes: > it implements __dict__ for instances but not for types. Adding support for modifying type dict's is a completely different feature. It will require more careful implementation because of the type cache in Py>=2.6, and it would likely require some dedicated syntax of @cython decorator, because cdef __dict__ would apply to instances and not to the type. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Sun Aug 29 19:06:01 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Aug 2010 19:06:01 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> <4C78A8D0.6020403@behnel.de> <4C7A0E04.6030603@behnel.de> Message-ID: <4C7A9379.30909@behnel.de> Lisandro Dalcin, 29.08.2010 18:13: > On 29 August 2010 04:36, Stefan Behnel wrote: >> Lisandro Dalcin, 29.08.2010 03:39: >> >>> On the other hand, I do not care too much if this is implemented going >>> through a descriptor, but as I'm not going to use this feature anyway. >> >> I think that describes the main use case already: "I won't use it in my >> code, but the users of my code *may* want to have it". That's the main >> reason why I'm advocating a low impact for the unused case. > > If users want a dict in their instances, then use a subclass. That the > way to have to do it for built-in types like list or dict. That works if it's the user who instantiates the type. It won't work for module internal types that are only instantiated by a library. That's not uncommon. Also imagine the case of a number-like type that implements all sorts of numeric methods. If you want to use a subtype that provides a __dict__, you may have to override all sorts of (special) methods to make sure it returns the new subtype instead of the extension type itself. That's a lot of overhead just for being able to store an attribute on an object. >> Note that your patch only solves half of the problem that Neal describes: >> it implements __dict__ for instances but not for types. > > Adding support for modifying type dict's is a completely different > feature. It will require more careful implementation because of the > type cache in Py>=2.6 Sure. > and it would likely require some dedicated > syntax of @cython decorator, because cdef __dict__ would apply to > instances and not to the type. Agreed for the "cdef __dict__" syntax. However, is there a reason why extension types can't provide modifiable __dict__ support by default? After all, subtypes can basically override all sorts of internals anyway, so there's not much of a difference if __dict__ is used to override them or if a subtype does that. Does anyone know of a reason why it could result in unexpected/unwanted behaviour if we make all extension type dicts modifiable? If the problems are just related to types that are not supposed to be subtyped in the first place, the long-proposed "final" modifier would be a better solution (and it would simply make the type __dict__ read-only as before). Stefan From dalcinl at gmail.com Sun Aug 29 19:36:15 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 29 Aug 2010 14:36:15 -0300 Subject: [Cython] some diffs to boost::python classes In-Reply-To: <4C7A9379.30909@behnel.de> References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> <4C78A8D0.6020403@behnel.de> <4C7A0E04.6030603@behnel.de> <4C7A9379.30909@behnel.de> Message-ID: On 29 August 2010 14:06, Stefan Behnel wrote: > > Agreed for the "cdef __dict__" syntax. However, is there a reason why > extension types can't provide modifiable __dict__ support by default? I do not care if you what type dict's modifiable by default, as long as you grant me a compiler directive to switch the feature off. > After > all, subtypes can basically override all sorts of internals anyway, so > there's not much of a difference if __dict__ is used to override them or if > a subtype does that. Does anyone know of a reason why it could result in > unexpected/unwanted behaviour if we make all extension type dicts modifiable? > Perhaps the ext type could break if users are able to change things in __dict__. Suppose you have two methods, both updating some internal cache or other internals... then if you let users to change one of these methods, the other could behave bad. > If the problems are just related to types that are not supposed to be > subtyped in the first place, the long-proposed "final" modifier would be a > better solution (and it would simply make the type __dict__ read-only as > before). > It could work, too. But IMHO, a directive would be better: no need for special syntax and it can be globally set. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Sun Aug 29 21:22:51 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Aug 2010 21:22:51 +0200 Subject: [Cython] some diffs to boost::python classes In-Reply-To: References: <4C77B220.1070100@behnel.de> <4C7825E8.7000409@behnel.de> <4C78A8D0.6020403@behnel.de> <4C7A0E04.6030603@behnel.de> <4C7A9379.30909@behnel.de> Message-ID: <4C7AB38B.4030307@behnel.de> Lisandro Dalcin, 29.08.2010 19:36: > On 29 August 2010 14:06, Stefan Behnel wrote: >> Agreed for the "cdef __dict__" syntax. However, is there a reason why >> extension types can't provide modifiable __dict__ support by default? > > I do not care if you what type dict's modifiable by default, as long > as you grant me a compiler directive to switch the feature off. I agree that a decorator makes sense here. I'm just considering whether the feature should be on or off by default. In general, I think Python class compatibility should win, so making type dicts modifiable by default would be the right thing to do. Then we could add a @read_only_type_dict class decorator that would enable the current (standard CPython) behaviour and a @no_type_dict decorator that would disable it completely. Although I'd prefer a single decorator, given that they are mutually exclusive. So maybe @type_dict("readonly") and @type_dict("none") would be better, with a default of @type_dict("public") and a potential extension to @type_dict("private"), in case that can be made to work in a meaningful way. Note that "none" must actually create a decriptor that raises an exception on access. Simply setting tp_dict to NULL will not let CPython handle the access automatically, thus resulting in a "readonly" type dict. >> After >> all, subtypes can basically override all sorts of internals anyway, so >> there's not much of a difference if __dict__ is used to override them or if >> a subtype does that. Does anyone know of a reason why it could result in >> unexpected/unwanted behaviour if we make all extension type dicts modifiable? > > Perhaps the ext type could break if users are able to change things in > __dict__. Suppose you have two methods, both updating some internal > cache or other internals... then if you let users to change one of > these methods, the other could behave bad. Sure, but the question is: can this break in a way that it couldn't if a subtype did the same thing? >> If the problems are just related to types that are not supposed to be >> subtyped in the first place, the long-proposed "final" modifier would be a >> better solution (and it would simply make the type __dict__ read-only as >> before). > > It could work, too. But IMHO, a directive would be better: no need for > special syntax and it can be globally set. I think it's much better to use a decorator as that moves the declaration immediately before the type definition, thus making it clearly visible how the type will behave by looking up its definition. Stefan From paul.leopardi at iinet.net.au Mon Aug 30 03:14:33 2010 From: paul.leopardi at iinet.net.au (Paul Leopardi) Date: Mon, 30 Aug 2010 11:14:33 +1000 Subject: [Cython] Wrapping C++ using Cython v0.13 gives strange error messages Message-ID: <201008301114.34074.paul.leopardi@iinet.net.au> Hi all, I'm re-posting this message to cython-user. Please do not reply on cython-dev. Thanks From dsdale24 at gmail.com Mon Aug 30 13:41:59 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 30 Aug 2010 07:41:59 -0400 Subject: [Cython] Wrapping C++ using Cython v0.13 gives strange error messages In-Reply-To: <201008301114.34074.paul.leopardi@iinet.net.au> References: <201008301114.34074.paul.leopardi@iinet.net.au> Message-ID: I posted a similar error message, and got some feedback from the devs. "operator" is special in cython-0.13, try changing the name of that variable. On Sun, Aug 29, 2010 at 9:14 PM, Paul Leopardi wrote: > Hi all, > I'm re-posting this message to cython-user. Please do not reply on cython-dev. > Thanks > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From aphidia at hotmail.com Mon Aug 30 20:30:41 2010 From: aphidia at hotmail.com (velociraptor Genjix) Date: Mon, 30 Aug 2010 18:30:41 +0000 Subject: [Cython] Python3 + Cython Error Message-ID: Hey, Cython a few weeks ago was fine with Python 3. But now after I build (python setup.py build) and then run setup.py, I get: genjix at l:~/Public/cython$ python setup.py build_extrunning build_ext... File "/home/genjix/Downloads/Cython-0.13/Cython/Compiler/Main.py", line 35 print t.dump() ^SyntaxError: invalid syntax After adding (, )'s around that print and going through fixing some syntax errors: # here's the new exception style in python 3 fyiexcept (Exc1, Exc2) as e: ... It runs but I get, failed to import Cython: No module named Codeerror: Cython does not appear to be installed Indeed, /Cython/ has no module named Code:CodeWriter.py Includes Runtime TestsCompiler __init__.py Shadow.py TestUtils.pyDebugging.py __init__.pyc Shadow.pyc Utils.pyDistutils Plex StringIOTree.py Thanks :) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100830/6c8a1b40/attachment.htm From stefan_ml at behnel.de Mon Aug 30 20:53:39 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 30 Aug 2010 20:53:39 +0200 Subject: [Cython] Python3 + Cython Error In-Reply-To: References: Message-ID: <4C7BFE33.5050608@behnel.de> Hi, please try to format your mail a little more readable next time. Also, please use the cython-users mailing list instead of the developers mailing list. velociraptor Genjix, 30.08.2010 20:30: > Cython a few weeks ago was fine with Python 3. But now after I build (python setup.py build) and then run setup.py, I get: > genjix at l:~/Public/cython$ python setup.py build_extrunning build_ext... File "/home/genjix/Downloads/Cython-0.13/Cython/Compiler/Main.py", line 35 print t.dump() ^SyntaxError: invalid syntax Interesting, the "build_ext" target doesn't run the refactoring step. You can install it normally, though, e.g. using the "install" or "bdist" targets. > After adding (, )'s around that print and going through fixing some syntax errors: > # here's the new exception style in python 3 fyiexcept (Exc1, Exc2) as e: ... > It runs but I get, > failed to import Cython: No module named Codeerror: Cython does not appear to be installed > Indeed, /Cython/ has no module named Code:CodeWriter.py Includes Runtime TestsCompiler __init__.py Shadow.py TestUtils.pyDebugging.py __init__.pyc Shadow.pyc Utils.pyDistutils Plex StringIOTree.py The source file in in the archive, so I guess you have broken something while editing the files. Please retry with the unmodified Cython release. Stefan From aphidia at hotmail.com Tue Aug 31 00:30:55 2010 From: aphidia at hotmail.com (velociraptor Genjix) Date: Mon, 30 Aug 2010 22:30:55 +0000 Subject: [Cython] Python3 + Cython Error In-Reply-To: <4C7BFE33.5050608@behnel.de> References: , <4C7BFE33.5050608@behnel.de> Message-ID: Hello, Apologies. This was the only mailing list I found- I think maybe it's kind of appropriate now here anyway since Cython is broken with Python 3. > You can install it normally, though, e.g. using the "install" or "bdist" > targets.I don't install programs unless they have a package because of problems like this. And just building it in the local directory and pointing PYTHONPATH should be enough, right? > The source file in in the archive, so I guess you have broken something > while editing the files. Please retry with the unmodified Cython release.I wasn't clear, my bad. I edited the files *because* it didn't work. They're using Python2 syntax and so will throw exceptions under Python3. I had to fix the files to get them to run. Then it still bails due to a missing module. The same steps I used here worked previously a few weeks ago, so I assume the latest Cython release might be broken :) cya > Date: Mon, 30 Aug 2010 20:53:39 +0200> From: stefan_ml at behnel.de > To: cython-dev at codespeak.net > Subject: Re: [Cython] Python3 + Cython Error > > Hi, > > please try to format your mail a little more readable next time. Also, > please use the cython-users mailing list instead of the developers mailing > list. > > velociraptor Genjix, 30.08.2010 20:30: > > Cython a few weeks ago was fine with Python 3. But now after I build (python setup.py build) and then run setup.py, I get: > > genjix at l:~/Public/cython$ python setup.py build_extrunning build_ext... File "/home/genjix/Downloads/Cython-0.13/Cython/Compiler/Main.py", line 35 print t.dump() ^SyntaxError: invalid syntax > > Interesting, the "build_ext" target doesn't run the refactoring step. > > You can install it normally, though, e.g. using the "install" or "bdist" > targets. > > > > After adding (, )'s around that print and going through fixing some syntax errors: > > # here's the new exception style in python 3 fyiexcept (Exc1, Exc2) as e: ... > > It runs but I get, > > failed to import Cython: No module named Codeerror: Cython does not appear to be installed > > Indeed, /Cython/ has no module named Code:CodeWriter.py Includes Runtime TestsCompiler __init__.py Shadow.py TestUtils.pyDebugging.py __init__.pyc Shadow.pyc Utils.pyDistutils Plex StringIOTree.py > > The source file in in the archive, so I guess you have broken something > while editing the files. Please retry with the unmodified Cython release. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100830/4d40e4a8/attachment.htm From stefan_ml at behnel.de Tue Aug 31 08:18:52 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Aug 2010 08:18:52 +0200 Subject: [Cython] Python3 + Cython Error In-Reply-To: References: , <4C7BFE33.5050608@behnel.de> Message-ID: <4C7C9ECC.7020703@behnel.de> velociraptor Genjix, 31.08.2010 00:30: > Apologies. This was the only mailing list I found Right, the cython-users mailing list is not linked on the front page. > I think maybe it's kind of appropriate now here anyway The users mailing list is the right place for users to bring up and discuss things they do not understand or consider unexpected or misbehaving. > since Cython is broken with Python 3. > The same steps I used here worked previously a few weeks ago, so I assume the latest Cython release might be broken :) I don't think it is. It works perfectly if you just install it and we continuously test it with Python 3.1 and 3.2pre. I don't consider "build_ext" an important build goal for Cython. It's clearly not an appropriate goal for normal use of Cython. Most users will just install it and won't care that it actually *has* extension modules. So the problem you describe is nothing that would keep users from running Cython in Python 3. Stefan From aphidia at hotmail.com Tue Aug 31 12:02:16 2010 From: aphidia at hotmail.com (velociraptor Genjix) Date: Tue, 31 Aug 2010 10:02:16 +0000 Subject: [Cython] Python3 + Cython Error In-Reply-To: <4C7C9ECC.7020703@behnel.de> References: , , <4C7BFE33.5050608@behnel.de>, , <4C7C9ECC.7020703@behnel.de> Message-ID: No I am running: cd Cython-0.13/ python setup.py buildexport PYTHONPATH=`pwd`cd mycythonproject/python setup.py build_ext# first example in tutorial on docs.cython.org webpage# errors here How is this not perfectly standard usage of Cython? Not everyone wishes to install every package to their / install, and so the fact it is throwing an error here is a major issue. I tried to write a fix to get it to work, but as I said, I came up against an error at the end. > Date: Tue, 31 Aug 2010 08:18:52 +0200 > From: stefan_ml at behnel.de > To: cython-dev at codespeak.net > Subject: Re: [Cython] Python3 + Cython Error > > velociraptor Genjix, 31.08.2010 00:30: > > Apologies. This was the only mailing list I found > > Right, the cython-users mailing list is not linked on the front page. > > > > I think maybe it's kind of appropriate now here anyway > > The users mailing list is the right place for users to bring up and discuss > things they do not understand or consider unexpected or misbehaving. > > > > since Cython is broken with Python 3. > > The same steps I used here worked previously a few weeks ago, so I assume the latest Cython release might be broken :) > > I don't think it is. It works perfectly if you just install it and we > continuously test it with Python 3.1 and 3.2pre. I don't consider > "build_ext" an important build goal for Cython. It's clearly not an > appropriate goal for normal use of Cython. Most users will just install it > and won't care that it actually *has* extension modules. So the problem you > describe is nothing that would keep users from running Cython in Python 3. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100831/9416a658/attachment.htm From stefan_ml at behnel.de Tue Aug 31 12:47:14 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Aug 2010 12:47:14 +0200 Subject: [Cython] Python3 + Cython Error In-Reply-To: References: , , <4C7BFE33.5050608@behnel.de>, , <4C7C9ECC.7020703@behnel.de> Message-ID: <4C7CDDB2.3050702@behnel.de> Hi, please format your e-mails in a readable way, please don't top-post, and please use the cython-users list next time. velociraptor Genjix, 31.08.2010 12:02: > No I am running: > cd Cython-0.13/ > python setup.py buildexport PYTHONPATH=`pwd`cd mycythonproject/python setup.py build_ext# first example in tutorial on docs.cython.org webpage# errors here Ah, ok, that's not what you presented in your original report, but this explains your problem. Your PYTHONPATH points to the wrong directory. It needs to point to the right subdirectory under "build", usually something like "build/lib-...". You can look up the machinery in the distutils docs (and sources). > I tried to write a fix to get it to work, but as I said, I came up against an error at the end. Well, you incorrectly assumed that there was a problem with the sources, that's why you didn't manage to write a fix. Stefan From aphidia at hotmail.com Tue Aug 31 14:01:31 2010 From: aphidia at hotmail.com (velociraptor Genjix) Date: Tue, 31 Aug 2010 12:01:31 +0000 Subject: [Cython] Python3 + Cython Error In-Reply-To: <4C7CDDB2.3050702@behnel.de> References: , , , <4C7BFE33.5050608@behnel.de>, , , , <4C7C9ECC.7020703@behnel.de>, , <4C7CDDB2.3050702@behnel.de> Message-ID: > Date: Tue, 31 Aug 2010 12:47:14 +0200 > From: stefan_ml at behnel.de > To: cython-dev at codespeak.net > Subject: Re: [Cython] Python3 + Cython Error > > Hi, > > please format your e-mails in a readable way, please don't top-post, and > please use the cython-users list next time. Aha, okay. I wasn't sure what you meant last time. Apologies for having posting here also. > velociraptor Genjix, 31.08.2010 12:02: > > No I am running: > > cd Cython-0.13/ > > python setup.py buildexport PYTHONPATH=`pwd`cd mycythonproject/python setup.py build_ext# first example in tutorial on docs.cython.org webpage# errors here > > Ah, ok, that's not what you presented in your original report, but this > explains your problem. Your PYTHONPATH points to the wrong directory. It > needs to point to the right subdirectory under "build", usually something > like "build/lib-...". You can look up the machinery in the distutils docs > (and sources). I see. That worked perfectly. Thank you. Might I suggest this change in the INSTALL file? >From this:(2) If you prefer not to modify your Python installation, arrange for the directory containing this file (INSTALL.txt) to be in your PYTHONPATH. On unix, also put the bin directory on your PATH. To this:(2) If you prefer not to modify your Python installation, then run setup.py in this directory containing this file (INSTALL.txt) as follows: python setup.py build And add the relevant build/lib* directory for your platform to your PYTHONPATH. On Linux this could be: cd build/lib.linux-x86_64-3.1/ export PYTHONPATH=$(pwd) On Unix, also put the bin directory in your path. > > I tried to write a fix to get it to work, but as I said, I came up against an error at the end. > > Well, you incorrectly assumed that there was a problem with the sources, > that's why you didn't manage to write a fix. Sorry, bad assumption by me. But thanks for your help :) cya > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100831/49fa791e/attachment-0001.htm From stefan_ml at behnel.de Tue Aug 31 14:44:47 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Aug 2010 14:44:47 +0200 Subject: [Cython] Python3 + Cython Error In-Reply-To: <201008310838.44081.nixsoftdev@gmail.com> References: <4C7CDDB2.3050702@behnel.de> <201008310838.44081.nixsoftdev@gmail.com> Message-ID: <4C7CF93F.5020506@behnel.de> Alan Boudreault, 31.08.2010 14:38: > All the posts you mentionned the bad formarting look all right for me. Could > the problem be from your client side ? No, if you check the source of the e-mails, you'll see that it's all mangled. You may be viewing the HTML version of the e-mails by default, which is also mangled, but has embedded formatting that should make it readable. However, HTML is not an appropriate format for public mailing lists. Stefan From nixsoftdev at gmail.com Tue Aug 31 14:38:44 2010 From: nixsoftdev at gmail.com (Alan Boudreault) Date: Tue, 31 Aug 2010 08:38:44 -0400 Subject: [Cython] Python3 + Cython Error In-Reply-To: <4C7CDDB2.3050702@behnel.de> References: <4C7CDDB2.3050702@behnel.de> Message-ID: <201008310838.44081.nixsoftdev@gmail.com> Hi Stefan, All the posts you mentionned the bad formarting look all right for me. Could the problem be from your client side ? Alan On August 31, 2010 06:47:14 am Stefan Behnel wrote: > Hi, > > please format your e-mails in a readable way, please don't top-post, and > please use the cython-users list next time. > > velociraptor Genjix, 31.08.2010 12:02: > > No I am running: > > cd Cython-0.13/ > > python setup.py buildexport PYTHONPATH=`pwd`cd mycythonproject/python > > setup.py build_ext# first example in tutorial on docs.cython.org webpage# > > errors here > > Ah, ok, that's not what you presented in your original report, but this > explains your problem. Your PYTHONPATH points to the wrong directory. It > needs to point to the right subdirectory under "build", usually something > like "build/lib-...". You can look up the machinery in the distutils docs > (and sources). > > > I tried to write a fix to get it to work, but as I said, I came up > > against an error at the end. > > Well, you incorrectly assumed that there was a problem with the sources, > that's why you didn't manage to write a fix. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev >