From greg.ewing at canterbury.ac.nz Mon Dec 1 00:08:26 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Dec 2008 12:08:26 +1300 Subject: [Cython] Temp allocation flow In-Reply-To: <49327CD5.5040807@student.matnat.uio.no> References: <49303890.7040501@behnel.de> <49304504.7040004@student.matnat.uio.no> <49312FEF.2020606@student.matnat.uio.no> <493137FE.9010407@student.matnat.uio.no> <49313F1F.2070202@behnel.de> <49314A68.5000200@student.matnat.uio.no> <49314F72.8050307@student.matnat.uio.no> <1DA4A5EA-33CF-4833-9A2A-7E5CD07437C9@math.washington.edu> <4931C54E.3030902@student.matnat.uio.no> <49327CD5.5040807@student.matnat.uio.no> Message-ID: <49331CEA.9050405@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > It > now takes two parameters, "decref" and "free_temp", which both defaults > to True, and which can be toggled. Rather than a free_temp parameter, I'd be more inclined to have an owns_temp flag on the node, set according to whether the node allocated the temp itself or it was supplied by the parent (currently done via an optional parameter to allocate_temps(), which would become a parameter to generate_evaluation_code()). -- Greg From dagss at student.matnat.uio.no Mon Dec 1 02:09:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 01 Dec 2008 02:09:03 +0100 Subject: [Cython] Temp allocation flow In-Reply-To: <49331CEA.9050405@canterbury.ac.nz> References: <49303890.7040501@behnel.de> <49304504.7040004@student.matnat.uio.no> <49312FEF.2020606@student.matnat.uio.no> <493137FE.9010407@student.matnat.uio.no> <49313F1F.2070202@behnel.de> <49314A68.5000200@student.matnat.uio.no> <49314F72.8050307@student.matnat.uio.no> <1DA4A5EA-33CF-4833-9A2A-7E5CD07437C9@math.washington.edu> <4931C54E.3030902@student.matnat.uio.no> <49327CD5.5040807@student.matnat.uio.no> <49331CEA.9050405@canterbury.ac.nz> Message-ID: <4933392F.4060202@student.matnat.uio.no> Greg Ewing wrote: > Dag Sverre Seljebotn wrote: >> It >> now takes two parameters, "decref" and "free_temp", which both defaults >> to True, and which can be toggled. > > Rather than a free_temp parameter, I'd be more inclined > to have an owns_temp flag on the node, set according to > whether the node allocated the temp itself or it was > supplied by the parent (currently done via an optional > parameter to allocate_temps(), which would become a > parameter to generate_evaluation_code()). I didn't supply enough context. It was meant for situations like this: self.arg.generate_evaluate_code(code) ... code.putln("if (...) {") ... self.arg.generate_disposal_code(code) ... code.putln("} else {") ... self.arg.generate_disposal_code(code) ... code.putln("}") Here, one can't free the temp in the first generate_disposal_code (because then the else-block could reuse and overwrite the temp etc.). As mentioned this is now solved by considering temp freeing as a seperate but parallell step to disposal. To make the example up-to-date, simply add at the end, but NOT inside the C if-blocks, self.arg.free_temps(code) I even found one example of the opposite construction, where evaluation_code was called multiple times but disposal once! (In TupleNode.generate_assignment_code). Luckily I was able to shuffle it a bit to avoid the problem. There might be more of this stuff in Cython than in Pyrex? This all points to a fundamental problem with the current scheme: Temp allocation doesn't branch when the code branches -- so with code like above the temp allocation is non-intuitive and suboptimal. That's one reason I long-term believe in more transforms even for the stuff that nodes do today -- if code branches like the above were always done using nodes rather than code.putln, temp allocation would both be much simpler conceptually and more optimal theoretically. -- Dag Sverre From dagss at student.matnat.uio.no Mon Dec 1 02:12:02 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 01 Dec 2008 02:12:02 +0100 Subject: [Cython] Temp allocation flow In-Reply-To: <4933392F.4060202@student.matnat.uio.no> References: <49303890.7040501@behnel.de> <49304504.7040004@student.matnat.uio.no> <49312FEF.2020606@student.matnat.uio.no> <493137FE.9010407@student.matnat.uio.no> <49313F1F.2070202@behnel.de> <49314A68.5000200@student.matnat.uio.no> <49314F72.8050307@student.matnat.uio.no> <1DA4A5EA-33CF-4833-9A2A-7E5CD07437C9@math.washington.edu> <4931C54E.3030902@student.matnat.uio.no> <49327CD5.5040807@student.matnat.uio.no> <49331CEA.9050405@canterbury.ac.nz> <4933392F.4060202@student.matnat.uio.no> Message-ID: <493339E2.9050900@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Greg Ewing wrote: >> Dag Sverre Seljebotn wrote: >>> It >>> now takes two parameters, "decref" and "free_temp", which both defaults >>> to True, and which can be toggled. >> Rather than a free_temp parameter, I'd be more inclined >> to have an owns_temp flag on the node, set according to >> whether the node allocated the temp itself or it was >> supplied by the parent (currently done via an optional >> parameter to allocate_temps(), which would become a >> parameter to generate_evaluation_code()). > > I didn't supply enough context. It was meant for situations like this: > > self.arg.generate_evaluate_code(code) > ... > code.putln("if (...) {") > ... > self.arg.generate_disposal_code(code) > ... > code.putln("} else {") > ... > self.arg.generate_disposal_code(code) > ... > code.putln("}") > > Here, one can't free the temp in the first generate_disposal_code > (because then the else-block could reuse and overwrite the temp etc.). > As mentioned this is now solved by considering temp freeing as a > seperate but parallell step to disposal. To make the example up-to-date, > simply add at the end, but NOT inside the C if-blocks, > That's one reason I long-term believe in more transforms even for the > stuff that nodes do today -- if code branches like the above were always > done using nodes rather than code.putln, temp allocation would both be > much simpler conceptually and more optimal theoretically. > Actually, this is not true. Pushing if-tests like the above to tree nodes would amount simply to moving the generate_disposal_code to the end of the C if-test. -- Dag Sverre From cournape at gmail.com Mon Dec 1 07:27:42 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 1 Dec 2008 15:27:42 +0900 Subject: [Cython] Recursive structure, typedef Message-ID: <5b8d13220811302227q14da1771t3e8137ba9ea12b58@mail.gmail.com> Hi, I am using cython to wrap some fairly large C API, with a big numbers of enumerations. Those API are big enough so that I got tired of manually declaring the definitions (in particular enum), and start working on a code generator based on gccxml. It almost works for some non trivial headers (CoreAudio framework, Alsa headers), but I have a problem with the following kind of C declaration: typedef struct foo foo_t; struct foo { foo_t *foo; } AFAIK, cython does support typedefing a struct directly, and you should use something like that instead: cdef struct foo: foo_t *foo ctypdef foo foo_t Which does not work as the recursive reference uses the typedef'd type name. The following works, though: cdef struct foo: foo *foo ctypdef foo foo_t For now, I can get away by modifying on the fly the gccxml parsed tree when I detect a recursive reference, but this feels hackish. Is there a better way ? David From greg.ewing at canterbury.ac.nz Mon Dec 1 11:19:05 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Dec 2008 23:19:05 +1300 Subject: [Cython] Temp allocation flow In-Reply-To: <4933392F.4060202@student.matnat.uio.no> References: <49303890.7040501@behnel.de> <49304504.7040004@student.matnat.uio.no> <49312FEF.2020606@student.matnat.uio.no> <493137FE.9010407@student.matnat.uio.no> <49313F1F.2070202@behnel.de> <49314A68.5000200@student.matnat.uio.no> <49314F72.8050307@student.matnat.uio.no> <1DA4A5EA-33CF-4833-9A2A-7E5CD07437C9@math.washington.edu> <4931C54E.3030902@student.matnat.uio.no> <49327CD5.5040807@student.matnat.uio.no> <49331CEA.9050405@canterbury.ac.nz> <4933392F.4060202@student.matnat.uio.no> Message-ID: <4933BA19.1020602@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > self.arg.generate_evaluate_code(code) > ... > code.putln("if (...) {") > ... > self.arg.generate_disposal_code(code) > ... > code.putln("} else {") > ... > self.arg.generate_disposal_code(code) > ... > code.putln("}") Are there any examples of this in Pyrex, or is this from Cython? -- Greg From dagss at student.matnat.uio.no Mon Dec 1 12:49:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 01 Dec 2008 12:49:00 +0100 Subject: [Cython] Temp allocation flow Message-ID: <3310980577.8672117@smtp.netcom.no> I don't know Pyrex that well, and weren't around at the time of the fork. I assume it is Cython-only then. We could probably change these cases. Then again they were probably added in the first place as features/optimizations. Anyway, now the necessary changes to support it is up in -devel. Dag Sverre Seljebotn -----Original Message----- From: Greg Ewing Date: Monday, Dec 1, 2008 11:16 am Subject: Re: [Cython] Temp allocation flow To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Dag Sverre Seljebotn wrote: > >> self.arg.generate_evaluate_code(code) > ... > code.putln("if (...) {") > ... > self.arg.generate_disposal_code(code) > ... > code.putln("} else {") > ... > self.arg.generate_disposal_code(code) > ... > code.putln("}") > >Are there any examples of this in Pyrex, or is this >from Cython? > >-- >Greg >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Mon Dec 1 13:20:35 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 1 Dec 2008 10:20:35 -0200 Subject: [Cython] Python aborts when running Cython on petsc4py sources Message-ID: I've got this when running Cython to generate petsc4py sources. cython --cleanup 9 -I. -Iinclude -w src petsc4py.PETSc.pyx -o petsc4py_PETSc.c python: Modules/gcmodule.c:276: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. I've tried to manually remove the Cython extension modules $ rm `find ~/lib/python/Cython -name '*.so'` but then (I reported this some time ago), things fails (because of pure-python mode) cython --cleanup 9 -I. -Iinclude -w src petsc4py.PETSc.pyx -o petsc4py_PETSc.c Traceback (most recent call last): File "/u/dalcinl/bin/cython", line 7, in from Cython.Compiler.Main import main File "/u/dalcinl/lib/python/Cython/Compiler/Main.py", line 17, in import Code File "/u/dalcinl/lib/python/Cython/Compiler/Code.py", line 11, in from Scanning import SourceDescriptor File "/u/dalcinl/lib/python/Cython/Compiler/Scanning.py", line 14, in import cython ImportError: No module named cython I have no time to dive onto this. I'm in the process of releasing petsc4py alongside with core PETSc. Could any of you tell me what cython-devel revision would be more or less safe to use? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Dec 1 13:31:40 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 1 Dec 2008 10:31:40 -0200 Subject: [Cython] Recursive structure, typedef In-Reply-To: <5b8d13220811302227q14da1771t3e8137ba9ea12b58@mail.gmail.com> References: <5b8d13220811302227q14da1771t3e8137ba9ea12b58@mail.gmail.com> Message-ID: Hi, David. Did you tried latest release, http://cython.org/Cython-0.10.2.tar.gz ? It contains fixes for recursive typedef struct. Please, give a try a let us know of any problem. On Mon, Dec 1, 2008 at 4:27 AM, David Cournapeau wrote: > Hi, > > I am using cython to wrap some fairly large C API, with a big numbers > of enumerations. Those API are big enough so that I got tired of > manually declaring the definitions (in particular enum), and start > working on a code generator based on gccxml. It almost works for some > non trivial headers (CoreAudio framework, Alsa headers), but I have a > problem with the following kind of C declaration: > > typedef struct foo foo_t; > > struct foo { > foo_t *foo; > } > > AFAIK, cython does support typedefing a struct directly, and you > should use something like that instead: > > cdef struct foo: > foo_t *foo > ctypdef foo foo_t > > Which does not work as the recursive reference uses the typedef'd type > name. The following works, though: > > cdef struct foo: > foo *foo > ctypdef foo foo_t > > For now, I can get away by modifying on the fly the gccxml parsed tree > when I detect a recursive reference, but this feels hackish. Is there > a better way ? > > David > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From cournape at gmail.com Mon Dec 1 13:40:02 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 1 Dec 2008 21:40:02 +0900 Subject: [Cython] Recursive structure, typedef In-Reply-To: References: <5b8d13220811302227q14da1771t3e8137ba9ea12b58@mail.gmail.com> Message-ID: <5b8d13220812010440n4853cccev20f475e00cb0417d@mail.gmail.com> On Mon, Dec 1, 2008 at 9:31 PM, Lisandro Dalcin wrote: > Hi, David. Did you tried latest release, > http://cython.org/Cython-0.10.2.tar.gz ? It contains fixes for > recursive typedef struct. Please, give a try a let us know of any > problem. > Yes, I use 0.10.2. I upgraded because indeed, before, the following did not work: cdef struct foo: foo *foo ctypdef foo foo_t It works on 0.10.2. But my problem is slightly different: cdef struct foo: foo_t *foo ctypdef foo foo_t Notice how the recursive is referenced through a typedef. In C, this is solved by forward declaring the struct through a typedef before the struct declaration. But AFAIK, you can't do that in cython. My question may be stupid: I know nothing about parsing languages, and maybe this is just a problem in my own code generator which is "hand-built" from the AST given by gccxml. cheers, David From dagss at student.matnat.uio.no Mon Dec 1 14:03:25 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 01 Dec 2008 14:03:25 +0100 Subject: [Cython] Python aborts when running Cython on petsc4py sources In-Reply-To: References: Message-ID: <4933E09D.8050805@student.matnat.uio.no> Lisandro Dalcin wrote: > I have no time to dive onto this. I'm in the process of releasing > petsc4py alongside with core PETSc. Could any of you tell me what > cython-devel revision would be more or less safe to use? If you're in the process of releasing software, use the latest stable. -devel has had bugs for at least a week because of temps stuff, but the latest release should be safe. Or is this happening in the latest release? -- Dag Sverre From dalcinl at gmail.com Mon Dec 1 14:04:22 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 1 Dec 2008 11:04:22 -0200 Subject: [Cython] Python aborts when running Cython on petsc4py sources In-Reply-To: <4933E09D.8050805@student.matnat.uio.no> References: <4933E09D.8050805@student.matnat.uio.no> Message-ID: On Mon, Dec 1, 2008 at 11:03 AM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: > > If you're in the process of releasing software, use the latest stable. > -devel has had bugs for at least a week because of temps stuff, but the > latest release should be safe. OK > Or is this happening in the latest release? No, no. This is happening with cython-devel. BTW, I switched to the revision above (a few commits away for latest release), and all seems to work. Revision: 1393 Branch: default Author: Stefan Behnel 2008-11-26 18:00:35 Committer: Stefan Behnel 2008-11-26 18:00:35 Parent: 1392:a53116f21261 (fixed typo) Child: 1394:71d8a06957eb (make TupleNode and ListNode a NewTempExprNode) let types inherit from object to work around hashing problems > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Dec 1 14:29:22 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 01 Dec 2008 14:29:22 +0100 Subject: [Cython] Python aborts when running Cython on petsc4py sources In-Reply-To: References: <4933E09D.8050805@student.matnat.uio.no> Message-ID: <4933E6B2.20907@student.matnat.uio.no> Lisandro Dalcin wrote: > On Mon, Dec 1, 2008 at 11:03 AM, Dag Sverre Seljebotn > wrote: >> Lisandro Dalcin wrote: >> >> If you're in the process of releasing software, use the latest stable. >> -devel has had bugs for at least a week because of temps stuff, but the >> latest release should be safe. > > OK > >> Or is this happening in the latest release? > > No, no. This is happening with cython-devel. > > BTW, I switched to the revision above (a few commits away for latest > release), and all seems to work. > > Revision: 1393 > Branch: default > Author: Stefan Behnel 2008-11-26 18:00:35 > Committer: Stefan Behnel 2008-11-26 18:00:35 > Parent: 1392:a53116f21261 (fixed typo) > Child: 1394:71d8a06957eb (make TupleNode and ListNode a NewTempExprNode) > > let types inherit from object to work around hashing problems Yes, and that is the very commit that broke a lot of things :-) It just doesn't show up because the testsuite is not complete, and it only strikes in special circumstances. Anything including the new dict for-loop optimizations is NOT SAFE. Please, use the latest stable. It's great that you /develop/ using -devel so that bugs are found, but when it comes to doing a release of your software, I strongly advice to always go back to the latest release. -- Dag Sverre From dagss at student.matnat.uio.no Mon Dec 1 14:30:52 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 01 Dec 2008 14:30:52 +0100 Subject: [Cython] Python aborts when running Cython on petsc4py sources In-Reply-To: <4933E6B2.20907@student.matnat.uio.no> References: <4933E09D.8050805@student.matnat.uio.no> <4933E6B2.20907@student.matnat.uio.no> Message-ID: <4933E70C.1010702@student.matnat.uio.no> Dag Sverre Seljebotn wrote: >> Revision: 1393 >> Branch: default >> Author: Stefan Behnel 2008-11-26 18:00:35 >> Committer: Stefan Behnel 2008-11-26 18:00:35 >> Parent: 1392:a53116f21261 (fixed typo) >> Child: 1394:71d8a06957eb (make TupleNode and ListNode a NewTempExprNode) >> >> let types inherit from object to work around hashing problems > > Yes, and that is the very commit that broke a lot of things :-) Opps, I was looking at the description for "Child". So this was wrong, but my point still stands :-) -- Dag Sverre From stefan_ml at behnel.de Mon Dec 1 14:32:45 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 1 Dec 2008 14:32:45 +0100 (CET) Subject: [Cython] Python aborts when running Cython on petsc4py sources In-Reply-To: <4933E70C.1010702@student.matnat.uio.no> References: <4933E09D.8050805@student.matnat.uio.no> <4933E6B2.20907@student.matnat.uio.no> <4933E70C.1010702@student.matnat.uio.no> Message-ID: <41763.213.61.181.86.1228138365.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >>> Revision: 1393 >>> Branch: default >>> Author: Stefan Behnel 2008-11-26 18:00:35 >>> Committer: Stefan Behnel 2008-11-26 18:00:35 >>> Parent: 1392:a53116f21261 (fixed typo) >>> Child: 1394:71d8a06957eb (make TupleNode and ListNode a >>> NewTempExprNode) >>> >>> let types inherit from object to work around hashing problems >> >> Yes, and that is the very commit that broke a lot of things :-) > > Opps, I was looking at the description for "Child". So this was wrong, I would also consider that revision safe. But I agree that for releasing software, the latest official Cython release is the safest bet. Stefan From robertwb at math.washington.edu Mon Dec 1 19:01:41 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Dec 2008 10:01:41 -0800 Subject: [Cython] Disabled FlattenInListTransform In-Reply-To: <49327FB9.3050301@student.matnat.uio.no> References: <49327FB9.3050301@student.matnat.uio.no> Message-ID: <32AD9F63-F18C-4545-9291-F1E5D519CF82@math.washington.edu> On Nov 30, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote: > I needed to temporarily disable FlattenInListTransform. Don't think > I'll > have time to reenable it myself soon, as this mostly marks the end > of my > Cython sprint for this time. > > http://trac.cython.org/cython_trac/ticket/146 Sure. Your new temps framework, once it all gets shaken out, makes more sense for something like this anyways. - Robert From robertwb at math.washington.edu Mon Dec 1 19:06:37 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Dec 2008 10:06:37 -0800 Subject: [Cython] Refcount safe-mode? In-Reply-To: <49328820.3030900@student.matnat.uio.no> References: <49328820.3030900@student.matnat.uio.no> Message-ID: On Nov 30, 2008, at 4:33 AM, Dag Sverre Seljebotn wrote: > Here's an idea: > > Have a babysit mode for refcounting. > > a) When entering an exiting from a function, "__Pyx_PushRefCountFrame" > and "__Pyx_PopRefcountFrame(__pyx_r)" are called. > > b) Use __Pyx_INCREF and __Pyx_DECREF instead > > c) Whenever a function call returns an object with a reference, > __Pyx_GOTREF should be called on the returned object. > > (Point c) could be done with refactoring "the call of a Python C API > function" to code.call_func or similar -- I'm noticing that a lot of > code is duplicated all over the place currently.) > > With these, I think it should be possible to (conditionally on a > #define) define things in such a way that a) __Pyx_DECREF can log > useful > information and exit more gracefully than a segfault, b) > __Pyx_PopRefcountFrame can report any memory leaks, locally from the > perspective of the function. > > Am I overlooking anything obvious? What do you think? I'm merely > wondering if it is a good enough idea to warrant creating a ticket for > it, not volunteering at this stage. > > If this was done before moving more temps it could perhaps help with > gaining confidence in a change quickly, and it would be a useful > addition to the regression test framework for the future as well. This is to ease development, not for end-users, right? I think this could be a good idea, though it's unclear exactly what "__Pyx_PushRefCountFrame" and "__Pyx_PopRefcountFrame" would do. And would __Pyx_DECREF be more useful than running under gdb? - Robert From dagss at student.matnat.uio.no Mon Dec 1 19:30:26 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 01 Dec 2008 19:30:26 +0100 Subject: [Cython] Refcount safe-mode? In-Reply-To: References: <49328820.3030900@student.matnat.uio.no> Message-ID: <49342D42.80109@student.matnat.uio.no> Robert Bradshaw wrote: > On Nov 30, 2008, at 4:33 AM, Dag Sverre Seljebotn wrote: > >> Here's an idea: >> >> Have a babysit mode for refcounting. >> >> a) When entering an exiting from a function, "__Pyx_PushRefCountFrame" >> and "__Pyx_PopRefcountFrame(__pyx_r)" are called. >> >> b) Use __Pyx_INCREF and __Pyx_DECREF instead >> >> c) Whenever a function call returns an object with a reference, >> __Pyx_GOTREF should be called on the returned object. >> >> (Point c) could be done with refactoring "the call of a Python C API >> function" to code.call_func or similar -- I'm noticing that a lot of >> code is duplicated all over the place currently.) >> >> With these, I think it should be possible to (conditionally on a >> #define) define things in such a way that a) __Pyx_DECREF can log >> useful >> information and exit more gracefully than a segfault, b) >> __Pyx_PopRefcountFrame can report any memory leaks, locally from the >> perspective of the function. >> >> Am I overlooking anything obvious? What do you think? I'm merely >> wondering if it is a good enough idea to warrant creating a ticket for >> it, not volunteering at this stage. >> >> If this was done before moving more temps it could perhaps help with >> gaining confidence in a change quickly, and it would be a useful >> addition to the regression test framework for the future as well. > > > This is to ease development, not for end-users, right? I think this Right. It's something to enable while running runtests.py, in order to get more proper regression tests on refcounting than a segfault. (Also leaks are currently not regression tested as far as I know.) It wouldn't help for leaks in end-user code, it would help ensure that Cython itself doesn't introduce leaks. > could be a good idea, though it's unclear exactly what > "__Pyx_PushRefCountFrame" and "__Pyx_PopRefcountFrame" would do. And > would __Pyx_DECREF be more useful than running under gdb? The former would construct a new object which is stored in a global stack (basically a dict of pointers-casted-to-integer to a "refcount that the curren function holds"), and which __Pyx_???REF would record information in. (Hmm. Perhaps it would be better to conditinally define a local variable with a fixed name. But these are details unecesarry to discuss at this point.) The latter would count up everything and make sure that there was exactly zero references (that the function knows that it holds) to everything but the parameter, __pyx_r, which should have exactly one reference (which the function holds). Schemes using the Python ob_refcnt could work as well, but in general I thought something that is unit-testing the refcount things that happen in the same function only, independent of possible wrong reference counting going on in other functions, is more in line with the goals I wanted to achieve. The test implementations could be written quickly in Cython and compiled and imported by runtests.py. Of course, when not regression testing, they'd be #defined as noops/Py_INCREF/Py_DECREF. Anyway, I always have too many ideas and too little time. But I thought I'd at least archive it somewhere if it seemed ok. -- Dag Sverre From greg.ewing at canterbury.ac.nz Tue Dec 2 00:06:14 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 02 Dec 2008 12:06:14 +1300 Subject: [Cython] Temp allocation flow In-Reply-To: <3310980577.8672117@smtp.netcom.no> References: <3310980577.8672117@smtp.netcom.no> Message-ID: <49346DE6.3010100@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > I don't know Pyrex that well, and weren't around at the time of the fork. > I assume it is Cython-only then. Maybe you could provide a bit more context still? I'm just wondering why you can't generate the disposal code after the whole if-statement, rather than duplicating it in each branch. -- Greg From greg.ewing at canterbury.ac.nz Tue Dec 2 00:27:52 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 02 Dec 2008 12:27:52 +1300 Subject: [Cython] Recursive structure, typedef In-Reply-To: <5b8d13220812010440n4853cccev20f475e00cb0417d@mail.gmail.com> References: <5b8d13220811302227q14da1771t3e8137ba9ea12b58@mail.gmail.com> <5b8d13220812010440n4853cccev20f475e00cb0417d@mail.gmail.com> Message-ID: <493472F8.3020302@canterbury.ac.nz> David Cournapeau wrote: > Notice how the recursive is referenced through a typedef. In C, this > is solved by forward declaring the struct through a typedef before the > struct declaration. But AFAIK, you can't do that in cython. Not sure about Cython, but the following works in Pyrex: cdef struct foo ctypedef foo foo_t cdef struct foo: foo_t *f i.e. although you can't forward-declare and typedef in one go, you can forward declare the struct and then typedef it. -- Greg From dagss at student.matnat.uio.no Tue Dec 2 00:49:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 02 Dec 2008 00:49:00 +0100 Subject: [Cython] Temp allocation flow Message-ID: <3311023764.8975443@smtp.netcom.no> The best way of getting the context would be searching for disposal_code in the source until you find such a case. My situation is that I don't know how that code got there or who wrote it, I just have to adapt to it when moving the temps to code generation, preferably while making as few other changes at the same time as possible. I can guess though .. take BoolBinopNode as an example. def f(): return [] def g(n): if n == 0: return 5 else: return f() or g(n-1) print g(100000000) Since the "or" can decref the result of f() inside the if-branch evaluating g(..), memory usage for the code in Cython is O(1) rather than O(n). (Not saying that it is not O(1) in Pyrex, I wouldn't know). I'm not 100 percent sure about this but it is what I figure. Dag Sverre Seljebotn -----Original Message----- From: Greg Ewing Date: Tuesday, Dec 2, 2008 12:03 am Subject: Re: [Cython] Temp allocation flow To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Dag Sverre Seljebotn wrote: > >> I don't know Pyrex that well, and weren't around at the time of the fork. > I assume it is Cython-only then. > >Maybe you could provide a bit more context still? I'm just >wondering why you can't generate the disposal code after >the whole if-statement, rather than duplicating it in each >branch. > >-- >Greg >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Tue Dec 2 06:07:49 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Dec 2008 21:07:49 -0800 Subject: [Cython] Recursive structure, typedef In-Reply-To: <493472F8.3020302@canterbury.ac.nz> References: <5b8d13220811302227q14da1771t3e8137ba9ea12b58@mail.gmail.com> <5b8d13220812010440n4853cccev20f475e00cb0417d@mail.gmail.com> <493472F8.3020302@canterbury.ac.nz> Message-ID: <699F985C-353E-4FA0-9332-03CF6042A6FB@math.washington.edu> On Dec 1, 2008, at 3:27 PM, Greg Ewing wrote: > David Cournapeau wrote: > >> Notice how the recursive is referenced through a typedef. In C, this >> is solved by forward declaring the struct through a typedef before >> the >> struct declaration. But AFAIK, you can't do that in cython. > > Not sure about Cython, but the following works in Pyrex: > > cdef struct foo > > ctypedef foo foo_t > > cdef struct foo: > foo_t *f Yes, this works in Cython too. - Robert From robertwb at math.washington.edu Tue Dec 2 08:21:44 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Dec 2008 23:21:44 -0800 Subject: [Cython] dynamic memory management In-Reply-To: <492B3F47.80502@gmail.com> References: <492A7DB7.10107@behnel.de> <492B3F47.80502@gmail.com> Message-ID: On Nov 24, 2008, at 3:56 PM, Michael Abshoff wrote: > Stefan Behnel wrote: >> Hi, > > Hi, > >> when re-reading an older thread about the struct syntax, I had a >> funny idea >> I wanted to share. >> >> Robert Bradshaw wrote: >>> I'm with Stefan that it is very >>> dangerous to hide the malloc and expect the user to explicitly >>> provide the free. If the user wants to manage their own memory, they >>> can do so with malloc and friends (understanding the associated >>> dangers), and any special memory-management stuff we add should >>> be as >>> implicit and easy to use as Python's garbage collection (and >>> probably >>> piggyback off of it). >> >> The "piggybacking" here makes me wonder if it would work to provide a >> dedicated "Memory" object, that would be a Python object but could >> be used >> as in >> >> def do_stuff_with_dynamic_memory(Py_ssize_t size): >> cdef Memory mem >> cdef void* ptr >> mem = Memory(10*size) # equiv to malloc() >> ptr = mem # automagic coercion to a pointer to the >> memory buffer >> # do stuff with *ptr, e.g. hand it around in C code >> mem = None # => Py_DECREF(mem), equiv to free() >> > > In general I consider this a great idea since people will be less > likely > to shoot themselves in the foot and as you mentioned for small allocs > this should really be a win. Having memory automatically freed on > exceptions would also be great since that is tricky to write. The less > the average user has to deal with memory allocs and deallocs in Cython > code the better Cython will be. I think this is a great idea too. Memory could have two C attributes like len and ptr (to avoid automagic coercion), and a realloc method (which may of course move ptr). Could probably be implemented on top of the str/bytes object with a new entry in the builtins table and perhaps some tree transformations to make creating one very fast. > But I am concerned to some extend about the success of hunting memory > leaks if this is the default allocator in some future version of > Cython > and there is no way to use malloc() [currenly Sage provide > sagemalloc(), > so that for example could be made to use > do_stuff_with_dynamic_memory() > since we only need to change one define]. Looking for heap objects in > Python that are leaked or not properly dereferenced is a giant pain > right now and valgrind is next to useless here. muppy is starting to > solve that problem, but there are at least in Sage issues with ipython > allocating tens of thousands of heap objects at load time impacting > performance of muppy badly, but those issues will be hopefully fixed. > Anyway, no need to rant about python memory heap debugging here :) > > Since in Sage I have to deal with loads of small, but potentially > large > cumulative leaks I would consider it very nice if the allocator could > have some magic debug fallback to malloc(). That should certainly > be not > the default behavior, just like --without-pymalloc is not the default > for Python. +1 to this idea. - Robert From stefan_ml at behnel.de Tue Dec 2 09:13:48 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 2 Dec 2008 09:13:48 +0100 (CET) Subject: [Cython] Temp allocation flow In-Reply-To: <49346DE6.3010100@canterbury.ac.nz> References: <3310980577.8672117@smtp.netcom.no> <49346DE6.3010100@canterbury.ac.nz> Message-ID: <35416.213.61.181.86.1228205628.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> I don't know Pyrex that well, and weren't around at the time of the >> fork. >> I assume it is Cython-only then. > > Maybe you could provide a bit more context still? I'm just > wondering why you can't generate the disposal code after > the whole if-statement, rather than duplicating it in each > branch. I think it's there because it was simple to put it there at the time when the special case was split off the general case. It may or may not be fine to move it after the if-else branches. The problem is that in the case that the disposal involves a DECREF of a temporary Python object, the object will be held until after the complete if-else has been handled, which may be long depending on the code. If it gets in the way, it's safe to do that, but it's not memory efficient. I don't remember the exact code that's being discussed here (and can't check at the moment), but if it's known to be short, it's best to just move the disposal and be done. Stefan From ondrej at certik.cz Tue Dec 2 12:43:26 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 2 Dec 2008 12:43:26 +0100 Subject: [Cython] convert-range is default? Message-ID: <85b5c3130812020343k8817647vc2b1b5750d44a2e@mail.gmail.com> Hi, I noticed that convert-range was enabled by default now? It seems to have been enabled a long time ago: http://hg.cython.org/cython/rev/5450c26066e9 So I am a little bit confused, because I have the feeling that I tried that after it as well without the "--convert-range" and it didn't produce the nice C for loop. But when I tried today, it indeed works. Is this the reason why it was removed from the help? http://hg.cython.org/cython/rev/7b8970eb4837 Btw, Dag, how is convert-range connected with "Option to emit #line directives"? Or maybe you needed space for the new directive? So in any case, "for i in range(10)" are now automatically converted by default (if i is declared as an integer), so one doesn't need to use the old syntax ("for i from 1 < 5" or something), right? Ondrej From ondrej at certik.cz Tue Dec 2 14:08:45 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 2 Dec 2008 14:08:45 +0100 Subject: [Cython] Python aborts when running Cython on petsc4py sources In-Reply-To: References: Message-ID: <85b5c3130812020508u1d507bd8h85ac2d3882442a3c@mail.gmail.com> On Mon, Dec 1, 2008 at 1:20 PM, Lisandro Dalcin wrote: > I have no time to dive onto this. I'm in the process of releasing > petsc4py alongside with core PETSc. Just curious, will you release it here: http://code.google.com/p/petsc4py/ or somewhere else? Ondrej From kay at fiber-space.de Wed Dec 3 06:48:27 2008 From: kay at fiber-space.de (Kay Schluehr) Date: Wed, 03 Dec 2008 06:48:27 +0100 Subject: [Cython] C methods broken? Message-ID: <49361DAB.8090000@fiber-space.de> Dear Cython list, while investigating the type semantics and constraints of Cython I stumbled over a problem I consider a bug. Here is some brief description using an example # point.pyx # # defines an extension type and a conversion between a C struct and an extension type # import random cdef struct CPoint: int x int y cdef class ExtPoint: cdef int x cdef int y cdef convert(self, CPoint pt): self.x = pt.x self.y = pt.y cdef convert(self, CPoint pt): self.x = pt.x self.y = pt.y cpdef test(): cdef CPoint pt pt.x = random.randrange(100) pt.y = random.randrange(100) ep = ExtPoint() convert(ep, pt) # o.k - compiles ep.convert(pt) # n.o.k - fails to compile return ep # end of point.pyx The standalone convert() function equals the convert() C-method of the ExtPoint extension type. However the compiler accepts the function call but rejects the method call: Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef CPoint pt pt.x = random.randrange(100) pt.y = random.randrange(100) ep = ExtPoint() convert(ep, pt) # o.k - compiles ep.convert(pt) # n.o.k - fails to compile ^ ------------------------------------------------------------ point.pyx:91:17: Cannot convert 'CPoint' to Python object There is no indication why the compiler tries to perform an automatic type conversion since the C method isn't exposed to Python anyway. Regards, Kay From robertwb at math.washington.edu Wed Dec 3 06:55:29 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 2 Dec 2008 21:55:29 -0800 Subject: [Cython] C methods broken? In-Reply-To: <49361DAB.8090000@fiber-space.de> References: <49361DAB.8090000@fiber-space.de> Message-ID: <3787B272-7274-4CCD-8D76-B87BDCCA8999@math.washington.edu> On Dec 2, 2008, at 9:48 PM, Kay Schluehr wrote: > Dear Cython list, > > while investigating the type semantics and constraints of Cython I > stumbled over a problem I consider a bug. Here is some brief > description > using an example > > # point.pyx > # > # defines an extension type and a conversion between a C struct and an > extension type > # > import random > > cdef struct CPoint: > int x > int y > > cdef class ExtPoint: > cdef int x > cdef int y > > cdef convert(self, CPoint pt): > self.x = pt.x > self.y = pt.y > > cdef convert(self, CPoint pt): > self.x = pt.x > self.y = pt.y > > cpdef test(): > cdef CPoint pt > pt.x = random.randrange(100) > pt.y = random.randrange(100) > ep = ExtPoint() > convert(ep, pt) # o.k - compiles > ep.convert(pt) # n.o.k - fails to compile > return ep > > # end of point.pyx > > The standalone convert() function equals the convert() C-method of the > ExtPoint extension type. However the compiler accepts the function > call > but rejects the method call: > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef CPoint pt > pt.x = random.randrange(100) > pt.y = random.randrange(100) > ep = ExtPoint() > convert(ep, pt) # o.k - compiles > ep.convert(pt) # n.o.k - fails to compile > ^ > ------------------------------------------------------------ > > point.pyx:91:17: Cannot convert 'CPoint' to Python object > > There is no indication why the compiler tries to perform an automatic > type conversion since the C method isn't exposed to Python anyway. You need to declare ep to be of type ExtPoint, i.e. cdef ExtPoint ep = ExtPoint() otherwise, it thinks it's just an ordinary Python object and an ordinary python method to be looked up at runtime. - Robert From stefan_ml at behnel.de Wed Dec 3 06:58:06 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 03 Dec 2008 06:58:06 +0100 Subject: [Cython] C methods broken? In-Reply-To: <49361DAB.8090000@fiber-space.de> References: <49361DAB.8090000@fiber-space.de> Message-ID: <49361FEE.4090805@behnel.de> Hi, Kay Schluehr wrote: > > cdef class ExtPoint: > cdef int x > cdef int y > > cdef convert(self, CPoint pt): > self.x = pt.x > self.y = pt.y > > cpdef test(): > cdef CPoint pt > pt.x = random.randrange(100) > pt.y = random.randrange(100) > ep = ExtPoint() > convert(ep, pt) # o.k - compiles > ep.convert(pt) # n.o.k - fails to compile You forgot to cdef ExtPoint ep so that Cython knows that ep.convert() even exists as a C method. Otherwise, it considers it a Python method on an unknown Python object. Stefan From kay at fiber-space.de Wed Dec 3 07:02:34 2008 From: kay at fiber-space.de (Kay Schluehr) Date: Wed, 03 Dec 2008 07:02:34 +0100 Subject: [Cython] C methods broken? In-Reply-To: <49361FEE.4090805@behnel.de> References: <49361DAB.8090000@fiber-space.de> <49361FEE.4090805@behnel.de> Message-ID: <493620FA.1000104@fiber-space.de> Stefan Behnel schrieb: > You forgot to > > cdef ExtPoint ep > > so that Cython knows that ep.convert() even exists as a C method. > Otherwise, it considers it a Python method on an unknown Python object. > > Stefan > Bingo! From cournape at gmail.com Wed Dec 3 07:57:50 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 3 Dec 2008 15:57:50 +0900 Subject: [Cython] Recursive structure, typedef In-Reply-To: <699F985C-353E-4FA0-9332-03CF6042A6FB@math.washington.edu> References: <5b8d13220811302227q14da1771t3e8137ba9ea12b58@mail.gmail.com> <5b8d13220812010440n4853cccev20f475e00cb0417d@mail.gmail.com> <493472F8.3020302@canterbury.ac.nz> <699F985C-353E-4FA0-9332-03CF6042A6FB@math.washington.edu> Message-ID: <5b8d13220812022257o14820674g6f0d13c5dd9dc04b@mail.gmail.com> On Tue, Dec 2, 2008 at 2:07 PM, Robert Bradshaw wrote: >> Not sure about Cython, but the following works in Pyrex: >> >> cdef struct foo >> >> ctypedef foo foo_t >> >> cdef struct foo: >> foo_t *f > > Yes, this works in Cython too. Ok, thanks, that should solve my problem. David From cournape at gmail.com Wed Dec 3 08:02:51 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 3 Dec 2008 16:02:51 +0900 Subject: [Cython] Speed up cython for enum declarations Message-ID: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> Hi, I was wondering whether it would be easy to speed up cython compilation of .pxd files which have a lot of enumerations. For my small code generator, since I generate all enumerations from a set of header files, I have a big number of them (> 1000), and this makes cython compilation quite slow (several seconds). I may well just miss something fundamental, but parsing/generating C code for enums sounds relatively uncomplicated compared to other kind of cython code, right ? It is something which was not sped up because nobody cared, or is it harder than I think ? thanks, David From robertwb at math.washington.edu Wed Dec 3 08:06:06 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 2 Dec 2008 23:06:06 -0800 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> Message-ID: On Dec 2, 2008, at 11:02 PM, David Cournapeau wrote: > Hi, > > I was wondering whether it would be easy to speed up cython > compilation of .pxd files which have a lot of enumerations. For my > small code generator, since I generate all enumerations from a set of > header files, I have a big number of them (> 1000), and this makes > cython compilation quite slow (several seconds). I may well just miss > something fundamental, but parsing/generating C code for enums sounds > relatively uncomplicated compared to other kind of cython code, right > ? It is something which was not sped up because nobody cared, or is it > harder than I think ? No, this is a big issue for me with Sage as well (not just enumerations, but long lists of declarations in general). I've got some code that speeds things up by a factor of two or so, but haven't checked it in yet (it was mixed in with a bunch of failed speedups, and I didn't have time to clean it up at the time). Unfortunately the devel branch is even less stable than usual, as temporary variable handling is undergoing a major shift, but I'll try to check it in and you can see if that helps. - Robert From stefan_ml at behnel.de Wed Dec 3 09:04:41 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 3 Dec 2008 09:04:41 +0100 (CET) Subject: [Cython] Speed up cython for enum declarations In-Reply-To: References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> Message-ID: <50463.213.61.181.86.1228291481.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Unfortunately the devel branch is even less stable than usual, as > temporary variable handling is undergoing a major shift "less stable" is a clear understatement. I actually consider it broken in a couple of places. A lot of code will still work, but crashes (both compiler crashes and runtime crashes) should be expected. Stefan From cournape at gmail.com Wed Dec 3 09:34:47 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 3 Dec 2008 17:34:47 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> Message-ID: <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> On Wed, Dec 3, 2008 at 4:06 PM, Robert Bradshaw wrote: > No, this is a big issue for me with Sage as well (not just > enumerations, but long lists of declarations in general). I've got > some code that speeds things up by a factor of two or so, but haven't > checked it in yet (it was mixed in with a bunch of failed speedups, > and I didn't have time to clean it up at the time). Ok, that would already be helpful. May I ask why long list of declarations are slow in general ? Parsing them should fast, no ? Would it be easy to get a notion of 'precompiled' pxd files otherwise ? > > Unfortunately the devel branch is even less stable than usual, as > temporary variable handling is undergoing a major shift, but I'll try > to check it in and you can see if that helps. Cool, thanks. David From robertwb at math.washington.edu Wed Dec 3 09:58:57 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 3 Dec 2008 00:58:57 -0800 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> Message-ID: <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> On Dec 3, 2008, at 12:34 AM, David Cournapeau wrote: > On Wed, Dec 3, 2008 at 4:06 PM, Robert Bradshaw > wrote: > >> No, this is a big issue for me with Sage as well (not just >> enumerations, but long lists of declarations in general). I've got >> some code that speeds things up by a factor of two or so, but haven't >> checked it in yet (it was mixed in with a bunch of failed speedups, >> and I didn't have time to clean it up at the time). > > Ok, that would already be helpful. May I ask why long list of > declarations are slow in general ? Parsing them should fast, no ? Actually, the parsing takes up a significant amount of the compile time. > Would it be easy to get a notion of 'precompiled' pxd files otherwise Yes and no. This is something I've tried to do before, and still want to do. One difficulty is handling dependancies between pxd files, for example if a typedef or cdef class changes in one, then everything that cimports and uses it changes. > >> >> Unfortunately the devel branch is even less stable than usual, as >> temporary variable handling is undergoing a major shift, but I'll try >> to check it in and you can see if that helps. > > Cool, thanks. > > David > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From cournape at gmail.com Wed Dec 3 10:13:58 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 3 Dec 2008 18:13:58 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> Message-ID: <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> On Wed, Dec 3, 2008 at 5:58 PM, Robert Bradshaw wrote: > On Dec 3, 2008, at 12:34 AM, David Cournapeau wrote: > >> On Wed, Dec 3, 2008 at 4:06 PM, Robert Bradshaw >> wrote: >> >>> No, this is a big issue for me with Sage as well (not just >>> enumerations, but long lists of declarations in general). I've got >>> some code that speeds things up by a factor of two or so, but haven't >>> checked it in yet (it was mixed in with a bunch of failed speedups, >>> and I didn't have time to clean it up at the time). >> >> Ok, that would already be helpful. May I ask why long list of >> declarations are slow in general ? Parsing them should fast, no ? > > Actually, the parsing takes up a significant amount of the compile time. Is it because the parser cannot assume there is only declarations, or is parsing declarations is inherently slow ? > Yes and no. This is something I've tried to do before, and still want > to do. One difficulty is handling dependancies between pxd files, for > example if a typedef or cdef class changes in one, then everything > that cimports and uses it changes. I don't understand: how is this a cython problem ? This should be handled by your build tool, no ? cheers, David From stefan_ml at behnel.de Wed Dec 3 11:04:34 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 3 Dec 2008 11:04:34 +0100 (CET) Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> Message-ID: <41814.213.61.181.86.1228298674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Actually, the parsing takes up a significant amount of the compile time. Which reminds me: I hope the OP uses an installed (i.e. compiled) version of Cython. That should be a lot faster than running it from a working directory. Another thing that comes to mind: in case the OP restarts Cython for each module, that would definitely slow down the compilation for a larger set of modules. I think the current build_ext extension doesn't reuse parser contexts either. >> Would it be easy to get a notion of 'precompiled' pxd files otherwise > > Yes and no. This is something I've tried to do before, and still want > to do. I remember a discussion about pickling parsed .pxd files. I think that would still make sense. > One difficulty is handling dependancies between pxd files, for > example if a typedef or cdef class changes in one, then everything > that cimports and uses it changes. Cython already keeps track of module dependencies (a Pyrex feature). But I'm not sure if that extends to inter-pxd dependencies. It should be easy to support that, though. Then the dependency list would tell you if you have to re-parse a pickled .pxd. Stefan From stefan_ml at behnel.de Wed Dec 3 11:08:44 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 3 Dec 2008 11:08:44 +0100 (CET) Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> Message-ID: <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> David Cournapeau wrote: > On Wed, Dec 3, 2008 at 5:58 PM, Robert Bradshaw wrote: >> Actually, the parsing takes up a significant amount of the compile time. > > Is it because the parser cannot assume there is only declarations, or > is parsing declarations is inherently slow ? Parsing is slow. That's why we switched to C-compiling the parser. >> Yes and no. This is something I've tried to do before, and still want >> to do. One difficulty is handling dependancies between pxd files, for >> example if a typedef or cdef class changes in one, then everything >> that cimports and uses it changes. > > I don't understand: how is this a cython problem ? This should be > handled by your build tool, no ? Only Cython can know the inter-dependencies of .pxd, .pxi and .pyx files, which are based on cimports and includes in the source. Look out for .dep files in your source directory after the generation. Stefan From michael.abshoff at googlemail.com Wed Dec 3 11:14:14 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Wed, 03 Dec 2008 02:14:14 -0800 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <41814.213.61.181.86.1228298674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <41814.213.61.181.86.1228298674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49365BF6.40008@gmail.com> Stefan Behnel wrote: > Robert Bradshaw wrote: >> One difficulty is handling dependancies between pxd files, for >> example if a typedef or cdef class changes in one, then everything >> that cimports and uses it changes. > > Cython already keeps track of module dependencies (a Pyrex feature). But > I'm not sure if that extends to inter-pxd dependencies. It should be easy > to support that, though. Then the dependency list would tell you if you > have to re-parse a pickled .pxd. > > Stefan There is code in Sage that does that written by RobertWB and two other Sage devs. And we even opened a ticket to get that code into Cython itself since it is useful to other projects. For the ticket see http://trac.sagemath.org/sage_trac/ticket/4664 Writing that code took a while to get right and every time we touch it new and interesting bugs are fixed and some times even new ones added :) Cheers, Michael > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From cournape at gmail.com Wed Dec 3 12:57:05 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 3 Dec 2008 20:57:05 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> On Wed, Dec 3, 2008 at 7:08 PM, Stefan Behnel wrote: > David Cournapeau wrote: >> On Wed, Dec 3, 2008 at 5:58 PM, Robert Bradshaw wrote: >>> Actually, the parsing takes up a significant amount of the compile time. >> >> Is it because the parser cannot assume there is only declarations, or >> is parsing declarations is inherently slow ? > > Parsing is slow. That's why we switched to C-compiling the parser. I understand parsing in slow in general; I was wondering why parsing pxd is. Something like the following: ctypedef enum: enum1 = 0 ... enum1000 = 0 Takes a significant amount of time, which surprised me a bit. Now, "it is slow because nobody bothered speeding it up" is a totally vaild answer :) > > Only Cython can know the inter-dependencies of .pxd, .pxi and .pyx files, > which are based on cimports and includes in the source. Look out for .dep > files in your source directory after the generation. I don't understand why only cython can know about it; many build tools handle dependencies without the compiler's help, I don't see why it is any different for cython ? I don't have .dep files, but then, I don't use distutils either, so that may explain it. David From dagss at student.matnat.uio.no Wed Dec 3 13:07:53 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 03 Dec 2008 13:07:53 +0100 Subject: [Cython] Temps again In-Reply-To: <50463.213.61.181.86.1228291481.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <50463.213.61.181.86.1228291481.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49367699.8040707@student.matnat.uio.no> Stefan Behnel wrote: > Robert Bradshaw wrote: > >> Unfortunately the devel branch is even less stable than usual, as >> temporary variable handling is undergoing a major shift >> > > "less stable" is a clear understatement. I actually consider it broken in > a couple of places. A lot of code will still work, but crashes (both > compiler crashes and runtime crashes) should be expected. > The only changes that's been done should be backwards-compatible with the old temp system though. So if you feel that this is the wrong time for fixing up those things and that it halts other kinds of development, you "should" be able to simply change the nodes back to ExprNode and have everything go back to a relatively normal, stable state. Except your dict optimizations, but those can be disabled easily. (I suppose BoolBinopNode might be an exception to this BTW, but that's localized to one place.) I won't be able to lend you a hand on those bugs for a week or two at least. Dag Sverre From dagss at student.matnat.uio.no Wed Dec 3 13:09:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 03 Dec 2008 13:09:03 +0100 Subject: [Cython] Temps again In-Reply-To: <49367699.8040707@student.matnat.uio.no> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <50463.213.61.181.86.1228291481.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49367699.8040707@student.matnat.uio.no> Message-ID: <493676DF.4090307@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: > >> Robert Bradshaw wrote: >> >> >>> Unfortunately the devel branch is even less stable than usual, as >>> temporary variable handling is undergoing a major shift >>> >>> >> "less stable" is a clear understatement. I actually consider it broken in >> a couple of places. A lot of code will still work, but crashes (both >> compiler crashes and runtime crashes) should be expected. >> >> > The only changes that's been done should be backwards-compatible with > the old temp system though. So if you feel that this is the wrong time > for fixing up those things and that it halts other kinds of development, > you "should" be able to simply change the nodes back to ExprNode and > have everything go back to a relatively normal, stable state. Except > your dict optimizations, but those can be disabled easily. (I suppose > BoolBinopNode might be an exception to this BTW, but that's localized to > one place.) > TupleNode.generate_assignment_code is another possible exception to this. Dag Sverre From stefan_ml at behnel.de Wed Dec 3 14:58:06 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 3 Dec 2008 14:58:06 +0100 (CET) Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> Message-ID: <36041.213.61.181.86.1228312686.squirrel@groupware.dvs.informatik.tu-darmstadt.de> David Cournapeau wrote: > Something like the following: > > ctypedef enum: > enum1 = 0 > ... > enum1000 = 0 > > Takes a significant amount of time, which surprised me a bit. Now, "it > is slow because nobody bothered speeding it up" is a totally vaild > answer :) Sounds like a bug to me that's worth filing a bug report for. Some profiling (using an uncompiled Cython!) will tell you where the problem is. >> Only Cython can know the inter-dependencies of .pxd, .pxi and .pyx >> files, >> which are based on cimports and includes in the source. Look out for >> .dep files in your source directory after the generation. > > I don't understand why only cython can know about it; many build tools > handle dependencies without the compiler's help, I don't see why it is > any different for cython ? I don't have .dep files, but then, I don't > use distutils either, so that may explain it. If parsing .h files slowed down C compilation significantly, would you try to improve 'make' to speed it up? This is a problem that needs to be solved in Cython, not in any external build tool or dependency tracker. Tracking the dependencies of a future pre-parsed .pxd file is only a minor issue. Stefan From dalcinl at gmail.com Wed Dec 3 21:18:19 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Dec 2008 18:18:19 -0200 Subject: [Cython] could someone look at this and fix the expected test message? Message-ID: ====================================================================== FAIL: compiling (c) e_pxdimpl ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 201, in runTest self.runCompileTest() File "runtests.py", line 205, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 294, in compile self.assertEquals(expected, error) AssertionError: '6:4: function definition not allowed here' != "6:4: function definition in pxd file must be declared 'cdef inline'" -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Wed Dec 3 23:42:02 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 03 Dec 2008 23:42:02 +0100 Subject: [Cython] Temps again In-Reply-To: <49367699.8040707@student.matnat.uio.no> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <50463.213.61.181.86.1228291481.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49367699.8040707@student.matnat.uio.no> Message-ID: <49370B3A.4000805@behnel.de> Hi, Dag Sverre Seljebotn wrote: > if you feel that this is the wrong time > for fixing up those things and that it halts other kinds of development, > you "should" be able to simply change the nodes back to ExprNode and > have everything go back to a relatively normal, stable state. I hacked a bit into the other direction and it seems that we're getting closer. I first got lxml to compile again :) and then raised the crash bar in its test suite step by step from 5% over 57% to 89%. I'm now stuck with an extremely weird crash due to a tuple item being NULL on read, while all places I can see where such a tuple is constructed do the correct INCREF before writing the item, so it would surely have crashed before if the NULL item had come from there. It happens on a tuple key in a dictionary that is stored in an attribute of an ext-type, so it's actually not the best place to trace value changes (and yes, I tried disabling the iter-dict transform, but that didn't fix it). I'll stop for tonight, but any ideas on this will be warmly appreciated. Stefan From greg.ewing at canterbury.ac.nz Thu Dec 4 00:11:48 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Dec 2008 12:11:48 +1300 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <41814.213.61.181.86.1228298674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <41814.213.61.181.86.1228298674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49371234.80005@canterbury.ac.nz> Stefan Behnel wrote: > Cython already keeps track of module dependencies (a Pyrex feature). But > I'm not sure if that extends to inter-pxd dependencies. It should be easy > to support that, though. Then the dependency list would tell you if you > have to re-parse a pickled .pxd. The difficulty as I see it would be figuring out how to pickle just the declarations that came from a particular .pxd file. Simply pickling the symbol table as it's currently implemented would end up pickling far too much, and unpickling it would create duplicates of objects that you don't want duplicated. -- Greg From greg.ewing at canterbury.ac.nz Thu Dec 4 00:25:40 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Dec 2008 12:25:40 +1300 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> Message-ID: <49371574.3090107@canterbury.ac.nz> David Cournapeau wrote: > I understand parsing in slow in general; I was wondering why parsing > pxd is. Are you sure that it is? Do you have timings indicating that parsing 1000 lines of enum declarations is significantly slower than parsing 1000 lines of any other kind of Cython code? > Now, "it > is slow because nobody bothered speeding it up" is a totally vaild > answer :) Another valid answer is might be "there is no particular bottleneck causing this problem that can be optimized". > I don't understand why only cython can know about it; many build tools > handle dependencies without the compiler's help If you're willing to manually declare all the dependencies in your Makefile or whatever, that's okay. > I don't have .dep files, but then, I don't > use distutils either, so that may explain it. You won't get .dep files from using distutils -- you need to invoke the compiler from the command line with appropriate options, or call it from a Python script. If you do it the right way, then a given .pxd file will only be parsed once for each compilation rather than once for each .pyx file that imports it, which can speed up the compilation process considerably. The relevant section of the Pyrex docs is here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/source_files.html#mozTocId233830 -- Greg From cournape at gmail.com Thu Dec 4 00:24:03 2008 From: cournape at gmail.com (David Cournapeau) Date: Thu, 4 Dec 2008 08:24:03 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <36041.213.61.181.86.1228312686.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <36041.213.61.181.86.1228312686.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <5b8d13220812031524h7ad004damb017f5822bf9d5b5@mail.gmail.com> On Wed, Dec 3, 2008 at 10:58 PM, Stefan Behnel wrote: > > Sounds like a bug to me that's worth filing a bug report for. Some > profiling (using an uncompiled Cython!) will tell you where the problem > is. Will do. > If parsing .h files slowed down C compilation significantly, would you try > to improve 'make' to speed it up? No, but when parsing C++ is slow, pre-compiled header may be a solution. It does not solve the very issue of slow parsing, but it solves the slow build from a user POV :) Anyway, I will do some profile and some dummy .pxd and see if there is an obvious spot. thanks, David From cournape at gmail.com Thu Dec 4 00:36:00 2008 From: cournape at gmail.com (David Cournapeau) Date: Thu, 4 Dec 2008 08:36:00 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <49371574.3090107@canterbury.ac.nz> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> Message-ID: <5b8d13220812031536m21f9c1b1tb7bb7a8cd4a57e3e@mail.gmail.com> On Thu, Dec 4, 2008 at 8:25 AM, Greg Ewing wrote: > > Are you sure that it is? Do you have timings indicating that > parsing 1000 lines of enum declarations is significantly slower > than parsing 1000 lines of any other kind of Cython code? As far as I can tell, enum are not slower to parse than any other kind of cython code, but they "seem" much easier to parse. Since some API have an order of magnitude more enums than functions, it has became a problem for me. > > The relevant section of the Pyrex docs is here: > > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/source_files.html#mozTocId233830 I was not aware of this feature, thanks. cheers, David From robertwb at math.washington.edu Thu Dec 4 03:16:54 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 3 Dec 2008 18:16:54 -0800 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <49371234.80005@canterbury.ac.nz> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <41814.213.61.181.86.1228298674.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49371234.80005@canterbury.ac.nz> Message-ID: <96D472A5-F2EA-4A60-B6D0-2FA47CBA4B7D@math.washington.edu> On Dec 3, 2008, at 3:11 PM, Greg Ewing wrote: > Stefan Behnel wrote: > >> Cython already keeps track of module dependencies (a Pyrex >> feature). But >> I'm not sure if that extends to inter-pxd dependencies. It should >> be easy >> to support that, though. Then the dependency list would tell you >> if you >> have to re-parse a pickled .pxd. > > The difficulty as I see it would be figuring out how to > pickle just the declarations that came from a particular > .pxd file. Simply pickling the symbol table as it's > currently implemented would end up pickling far too much, > and unpickling it would create duplicates of objects > that you don't want duplicated. Yes, these are *exactly* the kind of issues that I've been looking at, and it's not simple. Things were interconnected enough that it felt like I was pickling the entire state of the compiler. Extracting just the new declarations (and then making sure when they're unpickled that uniqueness and consistency is preserved) is a messy task. Pickling the (unprocessed) parse tree is not as optimal, but has shown some hope. - Robert From robertwb at math.washington.edu Thu Dec 4 03:21:36 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 3 Dec 2008 18:21:36 -0800 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <49371574.3090107@canterbury.ac.nz> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> Message-ID: <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> On Dec 3, 2008, at 3:25 PM, Greg Ewing wrote: > David Cournapeau wrote: > >> I understand parsing in slow in general; I was wondering why parsing >> pxd is. > > Are you sure that it is? Do you have timings indicating that > parsing 1000 lines of enum declarations is significantly slower > than parsing 1000 lines of any other kind of Cython code? I would guess that they're about the same speed. Perhaps a 1000 line enum is a bit faster, but it seems like it should be a lot faster (which it probably isn't). Take this hunch with a grain of salt, I haven't timed anything here. >> Now, "it >> is slow because nobody bothered speeding it up" is a totally vaild >> answer :) > > Another valid answer is might be "there is no particular > bottleneck causing this problem that can be optimized". And I think the truth is a bit of both. - Robert From dagss at student.matnat.uio.no Thu Dec 4 06:39:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 04 Dec 2008 06:39:00 +0100 Subject: [Cython] Temps again Message-ID: <3311217559.495138@smtp.netcom.no> Well, the entire tuple (or the object that holds it) could have been decrefed too many times and the memory reused for something else that is similar enough to bring you to the point of the crash.... Not very helpful :-) perhaps implementing my proposal from a few days ago (refcount babysitter) could provide the testing one needs to debug these things in a quicker way. Dag Sverre Seljebotn -----Original Message----- From: Stefan Behnel Date: Wednesday, Dec 3, 2008 11:42 pm Subject: Re: [Cython] Temps again To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hi, > >Dag Sverre Seljebotn wrote: > if you feel that this is the wrong time > for fixing up those things and that it halts other kinds of development, > you "should" be able to simply change the nodes back to ExprNode and > have everything go back to a relatively normal, stable state. > >I hacked a bit into the other direction and it seems that we're getting closer. I first got lxml to compile again :) and then raised the crash bar in its test suite step by step from 5% over 57% to 89%. I'm now stuck with an extremely weird crash due to a tuple item being NULL on read, while all places I can see where such a tuple is constructed do the correct INCREF before writing the item, so it would surely have crashed before if the NULL item had come from there. It happens on a tuple key in a dictionary that is stored in an attribute of an ext-type, so it's actually not the best place to trace value changes (and yes, I tried disabling the iter-dict transform, but that didn't fix it). > >I'll stop for tonight, but any ideas on this will be warmly appreciated. > >Stefan >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Thu Dec 4 08:20:58 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 04 Dec 2008 08:20:58 +0100 Subject: [Cython] Temps again In-Reply-To: <3311217559.495138@smtp.netcom.no> References: <3311217559.495138@smtp.netcom.no> Message-ID: <493784DA.9010803@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Well, the entire tuple (or the object that holds it) could have been > decrefed too many times and the memory reused for something else that is > similar enough to bring you to the point of the crash.... > > Not very helpful :-) ;) What bothers me is that it really doesn't look like the average ref-count bug at all. I end up with a tuple like (, 'a string') in gdb and it has a ref-count of 2, which makes perfect sense seeing the code. Stefan From ggellner at uoguelph.ca Thu Dec 4 16:49:31 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Thu, 4 Dec 2008 10:49:31 -0500 Subject: [Cython] Nice work! Message-ID: <20081204154931.GA7270@workerbee> Just reading the what's new in Python 3 and saw that Cython is recommended for new extensions! http://docs.python.org/dev/3.0/howto/cporting.html#cporting-howto (bottom of page). Nicely done! Gabriel From stefan_ml at behnel.de Thu Dec 4 20:52:42 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 04 Dec 2008 20:52:42 +0100 Subject: [Cython] Temps again In-Reply-To: <493784DA.9010803@behnel.de> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> Message-ID: <4938350A.50003@behnel.de> Hi, the problem really was a ref-counting bug and it was in BoolBinopNode. This Cython code a or b generated the following C code: /* * a or b # <<<<<<<<<<<<<< */ /* __pyx_t_1 allocated */ __pyx_t_1 = __Pyx_PyObject_IsTrue(__pyx_v_a); if (unlikely(__pyx_t_1 < 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 39; __pyx_clineno = __LINE__; goto __pyx_L1_error;} if (!__pyx_t_1) { /* __pyx_t_1 released */ /* __pyx_t_2 allocated */ __pyx_t_2 = __pyx_v_b; } else { __pyx_t_2 = __pyx_v_a; } Py_DECREF(__pyx_t_2); __pyx_t_2 = 0; /* __pyx_t_2 released */ So there's a deadly DECREF at the end. The fix is that __pyx_t_2 must receive an owned reference in both cases. Fixed now, and lxml builds and runs like a charm. :) I also added a test for the 'or' operator that nicely crashes before applying the fix. Stefan From robertwb at math.washington.edu Thu Dec 4 21:20:12 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 4 Dec 2008 12:20:12 -0800 Subject: [Cython] Temps again In-Reply-To: <4938350A.50003@behnel.de> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> Message-ID: On Dec 4, 2008, at 11:52 AM, Stefan Behnel wrote: > Hi, > > the problem really was a ref-counting bug and it was in > BoolBinopNode. This > Cython code > > a or b > > generated the following C code: > > /* > * a or b # <<<<<<<<<<<<<< > */ > /* __pyx_t_1 allocated */ > __pyx_t_1 = __Pyx_PyObject_IsTrue(__pyx_v_a); if (unlikely > (__pyx_t_1 < > 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 39; __pyx_clineno = > __LINE__; goto __pyx_L1_error;} > if (!__pyx_t_1) { > /* __pyx_t_1 released */ > /* __pyx_t_2 allocated */ > __pyx_t_2 = __pyx_v_b; > } else { > __pyx_t_2 = __pyx_v_a; > } > Py_DECREF(__pyx_t_2); __pyx_t_2 = 0; > /* __pyx_t_2 released */ > > So there's a deadly DECREF at the end. The fix is that __pyx_t_2 must > receive an owned reference in both cases. Fixed now, and lxml > builds and > runs like a charm. :) Excellent news! What about the other direction, do you see any leaks? - Robert From stefan_ml at behnel.de Thu Dec 4 21:29:34 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 04 Dec 2008 21:29:34 +0100 Subject: [Cython] Temps again In-Reply-To: References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> Message-ID: <49383DAE.2030603@behnel.de> Hi, Robert Bradshaw wrote: > What about the other direction, do you see any leaks? I fixed a couple of leaks already, so it's well possible that there are more. cython-devel definitely needs some more testing. Dag's refcount nanny would be helpful here, although it might be enough to just run the doctests of the test suite in a loop a couple of times and watch the memory consumption. Stefan From greg.ewing at canterbury.ac.nz Fri Dec 5 00:35:01 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 05 Dec 2008 12:35:01 +1300 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> Message-ID: <49386925.2010408@canterbury.ac.nz> Robert Bradshaw wrote: > Perhaps a 1000 line > enum is a bit faster, but it seems like it should be a lot faster It might be informative to find out what proportion of the time is spent on scanning, parsing, building the parse tree, traversing the parse tree, etc. If building and traversing the parse tree turns out to be a significant component, maybe a more compact way of representing an enum declaration in the parse tree could be found. E.g. in the simple case where all the values are int literals, have the parser turn it into a name->value dict. -- Greg From cournape at gmail.com Fri Dec 5 01:11:58 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 5 Dec 2008 09:11:58 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <49386925.2010408@canterbury.ac.nz> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> Message-ID: <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> On Fri, Dec 5, 2008 at 8:35 AM, Greg Ewing wrote: > Robert Bradshaw wrote: >> Perhaps a 1000 line >> enum is a bit faster, but it seems like it should be a lot faster > > It might be informative to find out what proportion of > the time is spent on scanning, parsing, building the > parse tree, traversing the parse tree, etc. What kind of profiling would be most useful for cython/pyrex developers ? I did not see any option for profiling cython itself in cython code, but is hotshot kind of information preferred over cProfile ? David From robert.kern at gmail.com Fri Dec 5 02:06:09 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 04 Dec 2008 19:06:09 -0600 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> Message-ID: David Cournapeau wrote: > On Fri, Dec 5, 2008 at 8:35 AM, Greg Ewing wrote: >> Robert Bradshaw wrote: >>> Perhaps a 1000 line >>> enum is a bit faster, but it seems like it should be a lot faster >> It might be informative to find out what proportion of >> the time is spent on scanning, parsing, building the >> parse tree, traversing the parse tree, etc. > > What kind of profiling would be most useful for cython/pyrex > developers ? I did not see any option for profiling cython itself in > cython code, but is hotshot kind of information preferred over > cProfile ? May I suggest my kernprof.py script to invoke cProfile? It has a nice feature such that it will locate scripts like cython on your $PATH. You don't have to write any particular profiling support. http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/raw-file/582df342463a/kernprof.py E.g. $ kernprof.py cython e_2_2000.pyx Wrote profile results to cython.prof $ python -m pstats cython.prof Welcome to the profile statistics browser. % strip % reverse % sort time % stats 20 Thu Dec 4 18:51:19 2008 cython.prof 1092804 function calls (830642 primitive calls) in 2.267 CPU seconds Ordered by: internal time List reduced from 963 to 20 due to restriction <20> ncalls tottime percall cumtime percall filename:lineno(function) 8219 0.412 0.000 0.500 0.000 Scanning.py:401(next) 52164/15 0.347 0.000 1.286 0.086 Visitor.py:18(visit) 212836 0.242 0.000 0.242 0.000 {getattr} 52164/15 0.159 0.000 1.204 0.080 Visitor.py:138(visitchildren) 52149/23 0.136 0.000 1.204 0.052 Visitor.py:86(visitchild) 52164/15 0.135 0.000 1.204 0.080 Visitor.py:92(visitchildren) 1 0.047 0.047 0.073 0.073 Scanning.py:6() 52160 0.041 0.000 0.059 0.000 inspect.py:291(getmro) 4083 0.038 0.000 0.057 0.000 Symtab.py:273(declare) 36057/14 0.038 0.000 0.929 0.066 Visitor.py:179(visit_Node) 4000 0.029 0.000 0.160 0.000 Parsing.py:2066(p_c_enum_item) 4 0.029 0.007 0.029 0.007 hashlib.py:55(__get_builtin_constructor) 52573 0.021 0.000 0.021 0.000 {isinstance} 4037 0.020 0.000 0.033 0.000 Scanning.py:359(indentation_action) 117371 0.018 0.000 0.018 0.000 {method 'append' of 'list' objects} 52163 0.018 0.000 0.018 0.000 {hasattr} 8096 0.015 0.000 0.085 0.000 Actions.py:46(perform) 1 0.015 0.015 0.015 0.015 Regexps.py:9() 1 0.015 0.015 0.044 0.044 hashlib.py:52() 1 0.015 0.015 0.015 0.015 Annotate.py:3() It looks like a fair chunk of time is getting spent by the getattr() on line 26 of Visitor.py:BasicVisitor.visit() because of missing entries in the dispatch_table. Per the comment there, that's possibly optimizable. While we're at it, here are the line-base profiles for the top two functions ("kernprof.py -l cython e_2_2000.pyx" if you install the whole line_profiler package): $ python -m line_profiler cython.lprof Timer unit: 1e-06 s File: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Cython-0.10-py2.5-macosx-10.3-fat.egg/Cython/Compiler/Scanning.py Function: next at line 401 Total time: 0.697081 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 401 @profile 402 def next(self): 403 8219 17393 2.1 2.5 try: 404 8219 576929 70.2 82.8 sy, systring = self.read() 405 except UnrecognizedInput: 406 self.error("Unrecognized character") 407 8219 20516 2.5 2.9 if sy == 'IDENT': 408 4077 9589 2.4 1.4 if systring in self.resword_dict: 409 23 42 1.8 0.0 sy = systring 410 else: 411 4054 12283 3.0 1.8 systring = EncodedString(systring) 412 4054 9845 2.4 1.4 systring.encoding = self.source_encoding 413 8219 16423 2.0 2.4 self.sy = sy 414 8219 16570 2.0 2.4 self.systring = systring 415 8219 17491 2.1 2.5 if debug_scanner: 416 _, line, col = self.position() 417 if not self.systring or self.sy == self.systring: 418 t = self.sy 419 else: 420 t = "%s %s" % (self.sy, self.systring) 421 print("--- %3d %2d %s" % (line, col, t)) File: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/Cython-0.10-py2.5-macosx-10.3-fat.egg/Cython/Compiler/Visitor.py Function: visit at line 18 Total time: 3.09749 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 18 @profile 19 def visit(self, obj): 20 52164 117647 2.3 3.8 cls = obj.__class__ 21 52164 140536 2.7 4.5 m = self.dispatch_table.get(cls.__name__) 22 52164 89747 1.7 2.9 if m is None: 23 # Must resolve, try entire hierarchy 24 52160 85652 1.6 2.8 pattern = "visit_%s" 25 52160 238251 4.6 7.7 mro = inspect.getmro(cls) 26 152410 258036 1.7 8.3 for cls in mro: 27 152410 623635 4.1 20.1 m = getattr(self, pattern % cls.__name__, None) 28 152410 277683 1.8 9.0 if m is not None: 29 52160 92241 1.8 3.0 break 30 else: 31 print type(self), type(obj) 32 print self.access_path 33 print self.access_path[-1][0].pos 34 print self.access_path[-1][0].__dict__ 35 raise RuntimeError("Visitor does not accept object: %s" % obj) 36 52160 128948 2.5 4.2 self.dispatch_table[cls.__name__] = m 37 52164 1045118 20.0 33.7 return m(obj) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan_ml at behnel.de Fri Dec 5 09:00:00 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 09:00:00 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> Message-ID: <4938DF80.8020602@behnel.de> Hi, Robert Kern wrote: > % stats 20 > Thu Dec 4 18:51:19 2008 cython.prof > > 1092804 function calls (830642 primitive calls) in 2.267 CPU seconds > > Ordered by: internal time > List reduced from 963 to 20 due to restriction <20> > > ncalls tottime percall cumtime percall filename:lineno(function) > 8219 0.412 0.000 0.500 0.000 Scanning.py:401(next) > 52164/15 0.347 0.000 1.286 0.086 Visitor.py:18(visit) > 212836 0.242 0.000 0.242 0.000 {getattr} > 52164/15 0.159 0.000 1.204 0.080 Visitor.py:138(visitchildren) > 52149/23 0.136 0.000 1.204 0.052 Visitor.py:86(visitchild) > 52164/15 0.135 0.000 1.204 0.080 Visitor.py:92(visitchildren) > 1 0.047 0.047 0.073 0.073 Scanning.py:6() > 52160 0.041 0.000 0.059 0.000 inspect.py:291(getmro) > 4083 0.038 0.000 0.057 0.000 Symtab.py:273(declare) > 36057/14 0.038 0.000 0.929 0.066 Visitor.py:179(visit_Node) > 4000 0.029 0.000 0.160 0.000 Parsing.py:2066(p_c_enum_item) > 4 0.029 0.007 0.029 0.007 > hashlib.py:55(__get_builtin_constructor) > 52573 0.021 0.000 0.021 0.000 {isinstance} > 4037 0.020 0.000 0.033 0.000 Scanning.py:359(indentation_action) > 117371 0.018 0.000 0.018 0.000 {method 'append' of 'list' objects} > 52163 0.018 0.000 0.018 0.000 {hasattr} > 8096 0.015 0.000 0.085 0.000 Actions.py:46(perform) > 1 0.015 0.015 0.015 0.015 Regexps.py:9() > 1 0.015 0.015 0.044 0.044 hashlib.py:52() > 1 0.015 0.015 0.015 0.015 Annotate.py:3() > > It looks like a fair chunk of time is getting spent by the getattr() on line 26 > of Visitor.py:BasicVisitor.visit() because of missing entries in the > dispatch_table. Per the comment there, that's possibly optimizable. When you install Cython, it will compile the scanner, the parser and the visitor module. That gives me the following for parsing a .pyx file that contains only a ctypedef enum with 2000 entries. -------------------------------- % stats 20 Fri Dec 5 08:39:55 2008 cython.prof 162946 function calls (159696 primitive calls) in 2.905 CPU seconds Ordered by: internal time List reduced from 828 to 20 due to restriction <20> ncalls tottime percall cumtime percall filename:lineno(function) 3 0.456 0.152 0.573 0.191 {Parsing.p_module} 1 0.331 0.331 1.060 1.060 Main.py:5() 1 0.322 0.322 0.498 0.498 Code.py:5() 1 0.136 0.136 0.552 0.552 Main.py:77(create_pipeline) 1 0.080 0.080 2.905 2.905 {execfile} 1 0.078 0.078 0.078 0.078 __init__.py:71(search_function) 1 0.071 0.071 0.137 0.137 Builtin.py:5() 1 0.071 0.071 0.120 0.120 __init__.py:31() 1 0.069 0.069 2.825 2.825 cython:7() 3 0.056 0.019 0.191 0.064 TreeFragment.py:32(parse_from_strings) 1 0.043 0.043 0.183 0.183 inspect.py:25() 4094 0.042 0.000 0.066 0.000 Actions.py:46(perform) 2116 0.033 0.000 0.092 0.000 Symtab.py:278(declare) 1 0.033 0.033 0.230 0.230 ParseTreeTransforms.py:1() 1 0.032 0.032 0.032 0.032 collections.py:1() 1 0.031 0.031 0.031 0.031 hashlib.py:55() 2028/13 0.031 0.000 0.049 0.004 ParseTreeTransforms.py:75(visit_StatNode) 1 0.028 0.028 0.198 0.198 Code.py:380(commented_file_contents) 1 0.027 0.027 0.030 0.030 ParseTreeTransforms.py:162(visit_ModuleNode) 1 0.026 0.026 0.029 0.029 Annotate.py:3() -------------------------------- It surprises me a bit that it calls p_module three times when parsing a single .pyx file. Anyway, a single parser run takes 200 msecs even under profiler control, which doesn't sound *that* unreasonable to me. Stefan From stefan_ml at behnel.de Fri Dec 5 09:20:52 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 09:20:52 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030034gf143c52t1014767a0ea5f3db@mail.gmail.com> <2739C783-9DF8-43CA-A618-111B80A9467B@math.washington.edu> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> Message-ID: <4938E464.2050201@behnel.de> Robert Kern wrote: > It looks like a fair chunk of time is getting spent by the getattr() on line 26 > of Visitor.py:BasicVisitor.visit() because of missing entries in the > dispatch_table. BTW, I just noticed that I forgot to push my fix for that code section to the release branch. There's a bug in there that renders the dispatch_table mostly useless. If you try with cython-devel, it should compile a *lot* faster. Stefan From cournape at gmail.com Fri Dec 5 09:58:40 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 5 Dec 2008 17:58:40 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <4938E464.2050201@behnel.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> Message-ID: <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> On Fri, Dec 5, 2008 at 5:20 PM, Stefan Behnel wrote: > > Robert Kern wrote: >> It looks like a fair chunk of time is getting spent by the getattr() on line 26 >> of Visitor.py:BasicVisitor.visit() because of missing entries in the >> dispatch_table. > > BTW, I just noticed that I forgot to push my fix for that code section to > the release branch. There's a bug in there that renders the dispatch_table > mostly useless. > > If you try with cython-devel, it should compile a *lot* faster. Which revision are you using ? Using the last cython-devel crashes on my files, unfortunately. David From stefan_ml at behnel.de Fri Dec 5 10:10:18 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 10:10:18 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030113j6a725821x1c2dc6e69aef1490@mail.gmail.com> <46272.213.61.181.86.1228298924.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> Message-ID: <4938EFFA.8020800@behnel.de> David Cournapeau wrote: > On Fri, Dec 5, 2008 at 5:20 PM, Stefan Behnel wrote: >> Robert Kern wrote: >>> It looks like a fair chunk of time is getting spent by the getattr() on line 26 >>> of Visitor.py:BasicVisitor.visit() because of missing entries in the >>> dispatch_table. >> BTW, I just noticed that I forgot to push my fix for that code section to >> the release branch. There's a bug in there that renders the dispatch_table >> mostly useless. >> >> If you try with cython-devel, it should compile a *lot* faster. > > Which revision are you using? The newest one. I also pushed the Visitor fix to the cython release branch now so that you can try it from there. But there were other improvements in cython-devel that will improve the compilation speed for your case (and I can't push those to the stable bug-fix branch). > Using the last cython-devel crashes on my files, unfortunately. That's almost certainly due to the temp refactoring. Please report any crashes you get, preferably with a short code snippet that shows the problem, so that we can fix them. Stefan From cournape at gmail.com Fri Dec 5 10:25:35 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 5 Dec 2008 18:25:35 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <4938EFFA.8020800@behnel.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> Message-ID: <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> On Fri, Dec 5, 2008 at 6:10 PM, Stefan Behnel wrote: > The newest one. I also pushed the Visitor fix to the cython release branch > now so that you can try it from there. But there were other improvements in > cython-devel that will improve the compilation speed for your case (and I > can't push those to the stable bug-fix branch). Ok, will do. > > That's almost certainly due to the temp refactoring. Please report any > crashes you get, preferably with a short code snippet that shows the > problem, so that we can fix them. I managed to get a quite short code snippet: it crashes on an empty file :) It also crashes on a trivial file (to make sure the empty file itself was not the issue) cheers, David From stefan_ml at behnel.de Fri Dec 5 10:31:58 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 10:31:58 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> Message-ID: <4938F50E.3040802@behnel.de> Hi, David Cournapeau wrote: > I managed to get a quite short code snippet: it crashes on an empty > file Not on my side. Do you mean that the compiler crashes or that the generated extension crashes? > It also crashes on a trivial file (to make sure the empty file > itself was not the issue) Could you send in the error output you get? Stefan From cournape at gmail.com Fri Dec 5 10:36:40 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 5 Dec 2008 18:36:40 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <4938F50E.3040802@behnel.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> <4938F50E.3040802@behnel.de> Message-ID: <5b8d13220812050136t53c963e3v67b513b43077c0c8@mail.gmail.com> On Fri, Dec 5, 2008 at 6:31 PM, Stefan Behnel wrote: > > >> It also crashes on a trivial file (to make sure the empty file >> itself was not the issue) > > Could you send in the error output you get? cython crashes, with a segfault, so I don't get anything really useful. I can give you the backtrace, but I am not sure it is that useful either: #0 PyObject_RichCompare (v=0x0, w=0xb7e52b40, op=2) at ../Objects/object.c:903 #1 0xb7cec5a2 in __pyx_f_6Cython_8Compiler_7Parsing_p_statement (__pyx_v_s=0x8a46ca4, __pyx_v_ctx=0x8b0cfac, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfc64654) at /usr/media/src/python/cython/cython-devel/Cython/Compiler/Parsing.c:28511 #2 0xb7cedbc6 in __pyx_pf_6Cython_8Compiler_7Parsing_p_statement (__pyx_self=0x0, __pyx_args=0x8afad6c, __pyx_kwds=0x8b160b4) at /usr/media/src/python/cython/cython-devel/Cython/Compiler/Parsing.c:28844 #3 0x0805d867 in PyObject_Call (func=0x81455e0, arg=0x8afad6c, kw=0x8b160b4) at ../Objects/abstract.c:1861 #4 0x080c850c in PyEval_CallObjectWithKeywords (func=0xb7d653ec, arg=0x8afad6c, kw=0x8b160b4) at ../Python/ceval.c:3464 #5 0xb7c8542b in __pyx_f_6Cython_8Compiler_7Parsing_p_statement_list (__pyx_v_s=0x8a46ca4, __pyx_v_ctx=0x8b0cfac, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfc64734) at /usr/media/src/python/cython/cython-devel/Cython/Compiler/Parsing.c:28970 #6 0xb7c85bd6 in __pyx_pf_6Cython_8Compiler_7Parsing_p_statement_list (__pyx_self=0x0, __pyx_args=0x8af428c, __pyx_kwds=0x8b161c4) at /usr/media/src/python/cython/cython-devel/Cython/Compiler/Parsing.c:29135 #7 0x0805d867 in PyObject_Call (func=0x81455e0, arg=0x8af428c, kw=0x8b161c4) at ../Objects/abstract.c:1861 #8 0x080c850c in PyEval_CallObjectWithKeywords (func=0xb7d6540c, arg=0x8af428c, kw=0x8b161c4) at ../Python/ceval.c:3464 #9 0xb7cc62f6 in __pyx_f_6Cython_8Compiler_7Parsing_p_module (__pyx_v_s=0x8a46ca4, __pyx_v_pxd=0x88fe0ac, __pyx_v_full_module_name=0x8b04688, __pyx_skip_dispatch=0) at /usr/media/src/python/cython/cython-devel/Cython/Compiler/Parsing.c:46078 #10 0xb7cc6ed5 in __pyx_pf_6Cython_8Compiler_7Parsing_p_module (__pyx_self=0x0, __pyx_args=0x8b048ec, __pyx_kwds=0x0) at /usr/media/src/python/cython/cython-devel/Cython/Compiler/Parsing.c:46283 #11 0x080cea39 in PyEval_EvalFrameEx (f=0x8aa38b4, throwflag=0) at ../Python/ceval.c:3595 #12 0x080d0345 in PyEval_EvalCodeEx (co=0x8b03a88, globals=0x8af8714, locals=0x0, args=0x8aa31a0, argcount=3, kws=0x8aa31ac, kwcount=2, defs=0x8b04808, defcount=3, closure=0x0) at ../Python/ceval.c:2858 #13 0x080ce728 in PyEval_EvalFrameEx (f=0x8aa3024, throwflag=0) at ../Python/ceval.c:3691 #14 0x080d0345 in PyEval_EvalCodeEx (co=0x8b08260, globals=0x8af8714, locals=0x0, args=0x8af2718, argcount=2, kws=0x8a90f78, kwcount=2, defs=0x8b01938, defcount=6, closure=0x0) at ../Python/ceval.c:2858 #15 0x0811797e in function_call (func=0x8b06a74, arg=0x8af270c, kw=0x8ae957c) at ../Objects/funcobject.c:517 #16 0x0805d867 in PyObject_Call (func=0x81455e0, arg=0x8af270c, kw=0x8ae957c) at ../Objects/abstract.c:1861 #17 0x08063a7a in instancemethod_call (func=0x8b06a74, arg=0x8af270c, kw=0x8ae957c) at ../Objects/classobject.c:2519 #18 0x0805d867 in PyObject_Call (func=0x81455e0, arg=0x8af9b2c, kw=0x8ae957c) at ../Objects/abstract.c:1861 #19 0x080a6168 in slot_tp_init (self=0x8afa1cc, args=0x8af9b2c, kwds=0x8ae957c) at ../Objects/typeobject.c:4943 #20 0x080ad971 in type_call (type=0x8aa118c, args=0x8af9b2c, kwds=0x8ae957c) at ../Objects/typeobject.c:436 #21 0x0805d867 in PyObject_Call (func=0x81455e0, arg=0x8af9b2c, kw=0x8ae957c) at ../Objects/abstract.c:1861 #22 0x080cc8a7 in PyEval_EvalFrameEx (f=0x8a87a14, throwflag=0) at ../Python/ceval.c:3806 #23 0x080d0345 in PyEval_EvalCodeEx (co=0x8affa88, globals=0x8af813c, locals=0x0, args=0x8a99098, argcount=0, kws=0x8a99098, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2858 #24 0x080ce728 in PyEval_EvalFrameEx (f=0x8a98f54, throwflag=0) at ../Python/ceval.c:3691 #25 0x080d0345 in PyEval_EvalCodeEx (co=0x8b036e0, globals=0x8af813c, locals=0x8af813c, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2858 #26 0x080d0557 in PyEval_EvalCode (co=0x8b036e0, globals=0x8af813c, locals=0x8af813c) at ../Python/ceval.c:514 #27 0x080e3a49 in PyImport_ExecCodeModuleEx (name=0xbfc67627 "Cython.Compiler.ParseTreeTransforms", co=0x8b036e0, pathname=0xbfc65557 "/home/david/local/lib/python2.5/site-packages/Cython/Compiler/ParseTreeTransforms.pyc") at ../Python/import.c:675 #28 0x080e3d1c in load_source_module (name=0xbfc67627 "Cython.Compiler.ParseTreeTransforms", pathname=0xbfc65557 "/home/david/local/lib/python2.5/site-packages/Cython/Compiler/ParseTreeTransforms.pyc", fp=) at ../Python/import.c:959 #29 0x080e51c9 in import_submodule (mod=0xb7e4659c, subname=0xbfc67637 "ParseTreeTransforms", fullname=0xbfc67627 "Cython.Compiler.ParseTreeTransforms") at ../Python/import.c:2400 #30 0x080e546a in load_next (mod=0xb7e4659c, altmod=0x814bef8, p_name=, buf=0xbfc67627 "Cython.Compiler.ParseTreeTransforms", p_buflen=0xbfc67620) at ../Python/import.c:2220 #31 0x080e5a77 in import_module_level (name=0x0, globals=0xb7e2879c, locals=, fromlist=0xb7e1ec84, level=-1) at ../Python/import.c:2001 #32 0x080e5dd3 in PyImport_ImportModuleLevel (name=0xb7e18a84 "ParseTreeTransforms", globals=0xb7e2879c, locals=0x814bef8, fromlist=0xb7e1ec84, level=-1) at ../Python/import.c:2072 #33 0x080c8034 in builtin___import__ (self=0x0, args=0xb7e0e11c, kwds=0x0) at ../Python/bltinmodule.c:47 #34 0x0805d867 in PyObject_Call (func=0x81455e0, arg=0xb7e0e11c, kw=0x0) at ../Objects/abstract.c:1861 #35 0x080c850c in PyEval_CallObjectWithKeywords (func=0xb7e4ee6c, arg=0xb7e0e11c, kw=0x0) at ../Python/ceval.c:3464 #36 0x080cc2b8 in PyEval_EvalFrameEx (f=0x8a877d4, throwflag=0) at ../Python/ceval.c:2089 #37 0x080d0345 in PyEval_EvalCodeEx (co=0xb7e212a8, globals=0xb7e2879c, locals=0x0, args=0x8a877a4, argcount=1, kws=0x8a877a8, kwcount=2, defs=0xb7d6a418, defcount=1, closure=0x0) at ../Python/ceval.c:2858 #38 0x080ce728 in PyEval_EvalFrameEx (f=0x8a87644, throwflag=0) at ../Python/ceval.c:3691 #39 0x080d0345 in PyEval_EvalCodeEx (co=0xb7e21698, globals=0xb7e2879c, locals=0x0, args=0x89509d4, argcount=3, kws=0x89509e0, kwcount=0, defs=0xb7d6a638, defcount=1, closure=0x0) at ../Python/ceval.c:2858 #40 0x080ce728 in PyEval_EvalFrameEx (f=0x8950874, throwflag=0) at ../Python/ceval.c:3691 #41 0x080d0345 in PyEval_EvalCodeEx (co=0xb7e21f08, globals=0xb7e2879c, locals=0x0, args=0x8a02cec, argcount=2, kws=0x8a02cf4, kwcount=0, defs=0x8adb638, defcount=1, closure=0x0) at ../Python/ceval.c:2858 #42 0x080ce728 in PyEval_EvalFrameEx (f=0x8a02b7c, throwflag=0) at ../Python/ceval.c:3691 #43 0x080cfbf5 in PyEval_EvalFrameEx (f=0x897c834, throwflag=0) at ../Python/ceval.c:3681 #44 0x080d0345 in PyEval_EvalCodeEx (co=0xb7e2e2f0, globals=0xb7e2879c, locals=0x0, args=0x8954c18, argcount=2, kws=0x8954c20, kwcount=0, defs=0xb7e2d6f0, defcount=4, closure=0x0) at ../Python/ceval.c:2858 #45 0x080ce728 in PyEval_EvalFrameEx (f=0x8954abc, throwflag=0) at ../Python/ceval.c:3691 ---Type to continue, or q to quit--- #46 0x080d0345 in PyEval_EvalCodeEx (co=0xb7e2e338, globals=0xb7e2879c, locals=0x0, args=0x8953388, argcount=0, kws=0x8953388, kwcount=1, defs=0x8adb8b8, defcount=1, closure=0x0) at ../Python/ceval.c:2858 #47 0x080ce728 in PyEval_EvalFrameEx (f=0x895324c, throwflag=0) at ../Python/ceval.c:3691 #48 0x080d0345 in PyEval_EvalCodeEx (co=0xb7e0f608, globals=0xb7e5facc, locals=0xb7e5facc, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2858 #49 0x080d0557 in PyEval_EvalCode (co=0xb7e0f608, globals=0xb7e5facc, locals=0xb7e5facc) at ../Python/ceval.c:514 #50 0x080edf8f in PyRun_FileExFlags (fp=0x88fa008, filename=0xbfc6b636 "/home/david/local/bin/cython", start=257, globals=0xb7e5facc, locals=0xb7e5facc, closeit=1, flags=0xbfc69848) at ../Python/pythonrun.c:1273 #51 0x080ee25a in PyRun_SimpleFileExFlags (fp=0x88fa008, filename=0xbfc6b636 "/home/david/local/bin/cython", closeit=1, flags=0xbfc69848) at ../Python/pythonrun.c:879 #52 0x080595e7 in Py_Main (argc=2, argv=0xbfc69914) at ../Modules/main.c:532 #53 0x08058962 in main (argc=205, argv=0x8145520) at ../Modules/python.c:23 David From cournape at gmail.com Fri Dec 5 10:37:45 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 5 Dec 2008 18:37:45 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <4938EFFA.8020800@behnel.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> Message-ID: <5b8d13220812050137x3f3f81d9w3161f9d9ec8d1aa0@mail.gmail.com> On Fri, Dec 5, 2008 at 6:10 PM, Stefan Behnel wrote: > > The newest one. I also pushed the Visitor fix to the cython release branch > now so that you can try it from there. But there were other improvements in > cython-devel that will improve the compilation speed for your case (and I > can't push those to the stable bug-fix branch). This is indeed significantly faster - around 20 % or so. Still 20-30 %, and gcc will become the main culprit :) David From stefan_ml at behnel.de Fri Dec 5 11:25:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 11:25:07 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812050136t53c963e3v67b513b43077c0c8@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> <4938F50E.3040802@behnel.de> <5b8d13220812050136t53c963e3v67b513b43077c0c8@mail.gmail.com> Message-ID: <49390183.7060908@behnel.de> Hi, David Cournapeau wrote: > On Fri, Dec 5, 2008 at 6:31 PM, Stefan Behnel wrote: >>> It also crashes on a trivial file (to make sure the empty file >>> itself was not the issue) >> Could you send in the error output you get? > > cython crashes, with a segfault, so I don't get anything really > useful. I can give you the backtrace, but I am not sure it is that > useful either: > [...] Is that from a fresh Cython install? I.e. do you use an up-to-date build of the compiled modules in Cython? I just want to make sure we are not hunting a bug that's already fixed. You can also install Cython uncompiled (i.e. pure Python) by passing the "--no-cython-compile" option to setup.py. That will prevent any C level crashes of the compiler itself. If you still want good performance, deleting individual C modules (such as the crashing Cython/Compiler/Parsing.so in your case) will make Cython fall back to the normal Python modules that are installed in parallel. Stefan From stefan_ml at behnel.de Fri Dec 5 11:28:13 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 11:28:13 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812050137x3f3f81d9w3161f9d9ec8d1aa0@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> <5b8d13220812050137x3f3f81d9w3161f9d9ec8d1aa0@mail.gmail.com> Message-ID: <4939023D.7080602@behnel.de> Hi, David Cournapeau wrote: > On Fri, Dec 5, 2008 at 6:10 PM, Stefan Behnel wrote: > >> The newest one. I also pushed the Visitor fix to the cython release branch >> now so that you can try it from there. But there were other improvements in >> cython-devel that will improve the compilation speed for your case (and I >> can't push those to the stable bug-fix branch). > > This is indeed significantly faster - around 20 % or so. It was the same for lxml at the time of the fix. > Still 20-30%, and gcc will become the main culprit :) For lxml, that's been the case for ages. I never timed it, but it takes gcc several times longer to compile the generated C code than it takes Cython to generate it. To name some numbers, that's about 18000 lines of Cython code (counting pyx,pxi,pxd) and some 155000 lines of generated C code. Stefan From stefan_ml at behnel.de Fri Dec 5 11:34:14 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 11:34:14 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> Message-ID: <493903A6.50004@behnel.de> Hi, David Cournapeau wrote: > I managed to get a quite short code snippet: it crashes on an empty > file :) It also crashes on a trivial file (to make sure the empty file > itself was not the issue) It still doesn't crash for an empty file, but I get it to crash on this: print "test" I'll look into this. Stefan From stefan_ml at behnel.de Fri Dec 5 12:23:54 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 12:23:54 +0100 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <493903A6.50004@behnel.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <5b8d13220812030357v702c5888kc37bc52e5266c36c@mail.gmail.com> <49371574.3090107@canterbury.ac.nz> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> <5b8d13220812050125l21c08ddci7a31cac08db80551@mail.gmail.com> <493903A6.50004@behnel.de> Message-ID: <49390F4A.6090005@behnel.de> Hi, Stefan Behnel wrote: > David Cournapeau wrote: >> I managed to get a quite short code snippet: it crashes on an empty >> file :) It also crashes on a trivial file (to make sure the empty file >> itself was not the issue) > > It still doesn't crash for an empty file, but I get it to crash on this: > > print "test" > > I'll look into this. Should be fixed now. It was the FlattenInListTransform that still wasn't as safe as I thought. Stefan From dagss at student.matnat.uio.no Fri Dec 5 13:51:47 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 05 Dec 2008 13:51:47 +0100 Subject: [Cython] Temps again In-Reply-To: <4938350A.50003@behnel.de> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> Message-ID: <493923E3.8030402@student.matnat.uio.no> Stefan Behnel wrote: > So there's a deadly DECREF at the end. The fix is that __pyx_t_2 must > receive an owned reference in both cases. Fixed now, and lxml builds and > runs like a charm. :) > Thanks, great! (And sorry for causing that error in the first place.) When it comes to FlattenInListTransform, I was thinking about something along the lines of an "StatAsExprNode", which would be sort of the inverse of ExprStatNode: It would contain a block which could contain "ReturnFromStatAsExprNodeNode" (though a better name is needed :-)) which would yield the value of the statement list, evaluated as an expression. To be able to do anything with transforms, something like this is probably needed. I'm uncertain about it though, it might be that there are assumptions everywhere that StatNodes doesn't appear within ExprNodes which would lead to a new avalanche of hard-to-find and untested-for bugs. Dag Sverre From dagss at student.matnat.uio.no Fri Dec 5 13:59:52 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 05 Dec 2008 13:59:52 +0100 Subject: [Cython] Temps again In-Reply-To: <493923E3.8030402@student.matnat.uio.no> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> Message-ID: <493925C8.5020007@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: > >> So there's a deadly DECREF at the end. The fix is that __pyx_t_2 must >> receive an owned reference in both cases. Fixed now, and lxml builds and >> runs like a charm. :) >> >> > Thanks, great! (And sorry for causing that error in the first place.) > > When it comes to FlattenInListTransform, I was thinking about something > along the lines of an "StatAsExprNode", which would be sort of the > inverse of ExprStatNode: It would contain a block which could contain > "ReturnFromStatAsExprNodeNode" (though a better name is needed :-)) > which would yield the value of the statement list, evaluated as an > expression. > Actually, "SetBlockResultNode" could give the same power, it just requires that the transforms write code without "gotos". But it would be much easier to implement. Dag Sverre From stefan_ml at behnel.de Fri Dec 5 14:10:01 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 14:10:01 +0100 Subject: [Cython] Temps again In-Reply-To: <493923E3.8030402@student.matnat.uio.no> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> Message-ID: <49392829.1080102@behnel.de> Hi, Dag Sverre Seljebotn wrote: > When it comes to FlattenInListTransform, I was thinking about something > along the lines of an "StatAsExprNode", which would be sort of the > inverse of ExprStatNode: It would contain a block which could contain > "ReturnFromStatAsExprNodeNode" (though a better name is needed :-)) > which would yield the value of the statement list, evaluated as an > expression. I thought about this, too. The main problems with TempsBlockNode are that it a) requires the type of a temp variable before hand (which is bad for transforms that run before type analysis), and b) is a block statement, not an expression. Making it an ExprNode and moving the current use into a ExprStatNode would fix that. However, there is still a problem with things like except blocks, where special subexpressions have a dedicated type that cannot easily be replaced by a generic temp handling class. This currently leads to sub-optimal temps usage e.g. for the exception value that the with-transform must catch. Not sure if that remains a problem once we moved everything to new-style temps. (and yes, I fear migrating NameNode, too). Stefan From stefan_ml at behnel.de Fri Dec 5 14:13:11 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 14:13:11 +0100 Subject: [Cython] Temps again In-Reply-To: <493925C8.5020007@student.matnat.uio.no> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <493925C8.5020007@student.matnat.uio.no> Message-ID: <493928E7.4020608@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> When it comes to FlattenInListTransform, I was thinking about something >> along the lines of an "StatAsExprNode", which would be sort of the >> inverse of ExprStatNode: It would contain a block which could contain >> "ReturnFromStatAsExprNodeNode" (though a better name is needed :-)) >> which would yield the value of the statement list, evaluated as an >> expression. >> > Actually, "SetBlockResultNode" could give the same power, it just > requires that the transforms write code without "gotos". But it would be > much easier to implement. Or the other way round, i.e. allow injecting preparatory statements into an expression that will be generated before generating the actual evaluation code of the ExprNode itself. Stefan From cournape at gmail.com Fri Dec 5 15:01:19 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 5 Dec 2008 23:01:19 +0900 Subject: [Cython] Speed up cython for enum declarations In-Reply-To: <4939023D.7080602@behnel.de> References: <5b8d13220812022302v5176c84brfb65029ded0ed1f5@mail.gmail.com> <1D60D7EC-5EDE-4552-BCE1-5A38F9FE3E04@math.washington.edu> <49386925.2010408@canterbury.ac.nz> <5b8d13220812041611r66ac0c55q733e2d5c9f7ab02b@mail.gmail.com> <4938E464.2050201@behnel.de> <5b8d13220812050058k27c62ca5q53f3981ac70e7c02@mail.gmail.com> <4938EFFA.8020800@behnel.de> <5b8d13220812050137x3f3f81d9w3161f9d9ec8d1aa0@mail.gmail.com> <4939023D.7080602@behnel.de> Message-ID: <5b8d13220812050601g6efbd0eds787a797e36e0c339@mail.gmail.com> On Fri, Dec 5, 2008 at 7:28 PM, Stefan Behnel wrote: > > For lxml, that's been the case for ages. I never timed it, but it takes gcc > several times longer to compile the generated C code than it takes Cython > to generate it. To name some numbers, that's about 18000 lines of Cython > code (counting pyx,pxi,pxd) and some 155000 lines of generated C code. I am not sure, but I suspect that gcc scales relatively badly with the size of source files - compilation times, even at -O0 feel significantly slower above a certain size. For example, when I started using numpy, a few years ago, I was surprised first that compiling numpy was slow wrt the amount of code, and I wonder if this due to the file size (numpy core source files are ~10 000 lines, and include each other to deal with the lack of a standard way to limit visibility of symbols). Maybe long names for symbols as typically used in generated code is harmful, too (again, I know nothing about compiler technologies, but I suspect strings operations involved in parsing to be quite demanding if only on the memory allocator). clang is supposed to be several times faster to compile C - it is stated as one of the reason for its funding from Apple. It will be interesting to see how this ends up. David From dalcinl at gmail.com Fri Dec 5 17:05:05 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 5 Dec 2008 14:05:05 -0200 Subject: [Cython] import/cimport gotcha? Message-ID: I'm doing this: # newmodule.pyx from package.module import Class from package.module cimport Class but then Class in not put in the newmodule's dict. Is this expected? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Fri Dec 5 17:44:23 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 05 Dec 2008 17:44:23 +0100 Subject: [Cython] Temps again In-Reply-To: <49392829.1080102@behnel.de> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> Message-ID: <49395A67.3000306@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Dag Sverre Seljebotn wrote: > >> When it comes to FlattenInListTransform, I was thinking about something >> along the lines of an "StatAsExprNode", which would be sort of the >> inverse of ExprStatNode: It would contain a block which could contain >> "ReturnFromStatAsExprNodeNode" (though a better name is needed :-)) >> which would yield the value of the statement list, evaluated as an >> expression. >> > > I thought about this, too. The main problems with TempsBlockNode are that > it a) requires the type of a temp variable before hand (which is bad for > transforms that run before type analysis), and b) is a block statement, not > an expression. Making it an ExprNode and moving the current use into a > ExprStatNode would fix that. > BTW, I don't worry that much about transforms that run before analysis. All it means is that the constructs which they "transform away" must still keep implementations around for doing the analysis (which are already present anyway in most cases...) Pure optimization transforms should always be done after analysis IMO, so I guess I consider FlattenInListTransform misplaced in the pipeline (though I don't know it well so I'm open for there being reasons I'm overlooking). Dag Sverre From dagss at student.matnat.uio.no Fri Dec 5 17:49:21 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 05 Dec 2008 17:49:21 +0100 Subject: [Cython] Temps again In-Reply-To: <49395A67.3000306@student.matnat.uio.no> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> <49395A67.3000306@student.matnat.uio.no> Message-ID: <49395B91.5090102@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Pure optimization transforms should always be done after analysis IMO, > so I guess I consider FlattenInListTransform misplaced in the pipeline > (though I don't know it well so I'm open for there being reasons I'm > overlooking). > Hehm. Such as getting the BoolBinopNodes analysed :-) So when BoolBinopNode was on the old temp system it had to be done before analysis, but now that it isn't one should be able to move FlattenInListTransform until after analysis. One just needs to add some code copying over type information etc. is needed (i.e construct the new BoolBinopNodes be specifying type=item.type...). Of course, if we can manage to create a TempsBlockNode/StatAsExprNode-combo that can be properly analysed the implementation will be slightly less verbose. Dag Sverre From stefan_ml at behnel.de Fri Dec 5 17:53:23 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 05 Dec 2008 17:53:23 +0100 Subject: [Cython] Temps again In-Reply-To: <49395B91.5090102@student.matnat.uio.no> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> <49395A67.3000306@student.matnat.uio.no> <49395B91.5090102@student.matnat.uio.no> Message-ID: <49395C83.9010506@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Pure optimization transforms should always be done after analysis IMO, >> so I guess I consider FlattenInListTransform misplaced in the pipeline >> (though I don't know it well so I'm open for there being reasons I'm >> overlooking). >> > Hehm. Such as getting the BoolBinopNodes analysed :-) > > So when BoolBinopNode was on the old temp system it had to be done > before analysis, but now that it isn't one should be able to move > FlattenInListTransform until after analysis. One just needs to add some > code copying over type information etc. is needed (i.e construct the new > BoolBinopNodes be specifying type=item.type...). Sounds like a plan to me. Will you give it a try? Stefan From cournape at gmail.com Sat Dec 6 14:27:03 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 6 Dec 2008 22:27:03 +0900 Subject: [Cython] IFDEF in cython ? Message-ID: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> Hi, I would like to know whether adding an IFDEF-like feature in cython would be feasible ? I would like to define some cython code depending on some defines in the header of the library I am wrapping. thanks, David From ggellner at uoguelph.ca Sat Dec 6 16:08:25 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 6 Dec 2008 10:08:25 -0500 Subject: [Cython] IFDEF in cython ? In-Reply-To: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> Message-ID: <20081206150825.GA28464@encolpuis> On Sat, Dec 06, 2008 at 10:27:03PM +0900, David Cournapeau wrote: > Hi, > > I would like to know whether adding an IFDEF-like feature in cython > would be feasible ? I would like to define some cython code depending > on some defines in the header of the library I am wrapping. > Not sure if this is what you mean: http://docs.cython.org/docs/language_basics.html#conditional-compilation If not it is probably the place to look for how hard it would be to add. Gabriel From jelleferinga at gmail.com Sat Dec 6 16:23:10 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Sat, 6 Dec 2008 16:23:10 +0100 Subject: [Cython] __next__ | not being able to use cpdef Message-ID: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> Hi, I just started to use Cython, for writing a set of fast iterators. Go easy on me, this is all new territory ;') Now, the thing is that if I understood correctly, it would be faster to have the __next__ method types a cpdef int __next__(self ) When I do so the module fails to compile. Is there a specific reason why this is so? Still, using the object below, accessing objects from an iterable is twice as fast. Thanks, -jelle cdef class FastLoopObj1d: cdef int N, i cdef object obj def __init__(self, n, obj ): self.N = n-1 self.i = 0 self.obj = obj def __iter__(self): return self # no cpdef allowed? def __next__( self ): if self.i < self.N: self.i+=1 return self.obj[self.i] else: raise StopIteration -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20081206/dcd68519/attachment.htm From dalcinl at gmail.com Sat Dec 6 19:31:34 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 6 Dec 2008 15:31:34 -0300 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> Message-ID: I __next__ falls in the category of special methods, please read the fine docs http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html Declare __next__ with 'def', they are treated specially and are as fast as the CPython implementation permits. On Sat, Dec 6, 2008 at 12:23 PM, Jelle Feringa wrote: > Hi, > I just started to use Cython, for writing a set of fast iterators. > Go easy on me, this is all new territory ;') > Now, the thing is that if I understood correctly, it would be faster to have > the __next__ method types a cpdef int __next__(self ) > When I do so the module fails to compile. > Is there a specific reason why this is so? > Still, using the object below, accessing objects from an iterable is twice > as fast. > Thanks, > -jelle > cdef class FastLoopObj1d: > > > > cdef int N, i > cdef object obj > > > > def __init__(self, n, obj ): > self.N = n-1 > self.i = 0 > self.obj = obj > def __iter__(self): > return self > > > > # no cpdef allowed? > def __next__( self ): > if self.i < self.N: > self.i+=1 > return self.obj[self.i] > else: > raise StopIteration > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Sat Dec 6 19:56:24 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Dec 2008 19:56:24 +0100 Subject: [Cython] IFDEF in cython ? In-Reply-To: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> Message-ID: <493ACAD8.4090308@behnel.de> Hi, David Cournapeau wrote: > I would like to know whether adding an IFDEF-like feature in cython > would be feasible ? I would like to define some cython code depending > on some defines in the header of the library I am wrapping. You can use a separate .h file and do something like this #ifdef SOME_EXTERNAL_DEFINE # define MYVAL 1 #else # define MYVAL 0 #endif then you can write this: cdef extern from "myheader.h": cdef int SOMETHING if SOMETHING: print "ON" else: print OFF and the C compiler will do the right thing. I wouldn't mind making this a general Cython feature, though. Maybe something like if cython.defined(SOME_NAME): ... would work. Stefan From jelleferinga at gmail.com Sat Dec 6 19:38:08 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Sat, 6 Dec 2008 19:38:08 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> Message-ID: <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> > I __next__ falls in the category of special methods, please read the > fine docs > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html Right, I've read the document. > Declare __next__ with 'def', they are treated specially and are as > fast as the CPython implementation permits. Aha, I see, that I didn't know. I noticed that the cython iterator is about twice as fast compared to the python version, when indexing an iterable object. A respectable speedup, given how trivial the implementation is. Would you consider the implementation optimal? Is this the fastest way of indexing an iterable object? Many thanks Lisandro. -jelle From dagss at student.matnat.uio.no Sat Dec 6 20:46:22 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 06 Dec 2008 20:46:22 +0100 Subject: [Cython] IFDEF in cython ? In-Reply-To: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> Message-ID: <493AD68E.1030809@student.matnat.uio.no> David Cournapeau wrote: > Hi, > > I would like to know whether adding an IFDEF-like feature in cython > would be feasible ? I would like to define some cython code depending > on some defines in the header of the library I am wrapping. Could you tell us what your exact usecase is? Some things are possible to do in other ways. For instance, if there's a type that's sometimes "long" and sometimes "short", then Cython doesn't really care about that, you can always use "int" in Cython anyway. There might be similar workarounds for whatever your case is. (That doesn't really improve matters of course, I'll agree to this being a weakness in Cython.) -- Dag Sverre From cournape at gmail.com Sun Dec 7 03:49:42 2008 From: cournape at gmail.com (David Cournapeau) Date: Sun, 7 Dec 2008 11:49:42 +0900 Subject: [Cython] IFDEF in cython ? In-Reply-To: <493AD68E.1030809@student.matnat.uio.no> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> <493AD68E.1030809@student.matnat.uio.no> Message-ID: <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> On Sun, Dec 7, 2008 at 4:46 AM, Dag Sverre Seljebotn wrote: > David Cournapeau wrote: >> Hi, >> >> I would like to know whether adding an IFDEF-like feature in cython >> would be feasible ? I would like to define some cython code depending >> on some defines in the header of the library I am wrapping. > > Could you tell us what your exact usecase is? Sure. I am wrapping a library which supports several file formats: for each format, there is an enum value: enum { FORMAT1, ... FORMATN } New versions add some new enums, which can be used in the library. I would like to support the new formats, without forcing people to use the new library. For now, I hardcode the enum value for the recent added values, but that's obviously not a nice solution. > Some things are possible > to do in other ways. For instance, if there's a type that's sometimes > "long" and sometimes "short", then Cython doesn't really care about > that, you can always use "int" in Cython anyway. There might be similar > workarounds for whatever your case is. it won't work in that case: I want to use a symbol which may or not exist depending on the header version, not a symbol whose value may change. David From robertwb at math.washington.edu Sun Dec 7 05:46:10 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 6 Dec 2008 20:46:10 -0800 Subject: [Cython] IFDEF in cython ? In-Reply-To: <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> <493AD68E.1030809@student.matnat.uio.no> <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> Message-ID: <655C5956-3020-424E-90C3-74783E295D47@math.washington.edu> On Dec 6, 2008, at 6:49 PM, David Cournapeau wrote: > On Sun, Dec 7, 2008 at 4:46 AM, Dag Sverre Seljebotn > wrote: >> David Cournapeau wrote: >>> Hi, >>> >>> I would like to know whether adding an IFDEF-like feature in cython >>> would be feasible ? I would like to define some cython code >>> depending >>> on some defines in the header of the library I am wrapping. >> >> Could you tell us what your exact usecase is? > > Sure. I am wrapping a library which supports several file formats: for > each format, there is an enum value: > > enum { > FORMAT1, > ... > FORMATN > } > > New versions add some new enums, which can be used in the library. I > would like to support the new formats, without forcing people to use > the new library. For now, I hardcode the enum value for the recent > added values, but that's obviously not a nice solution. > >> Some things are possible >> to do in other ways. For instance, if there's a type that's sometimes >> "long" and sometimes "short", then Cython doesn't really care about >> that, you can always use "int" in Cython anyway. There might be >> similar >> workarounds for whatever your case is. > > it won't work in that case: I want to use a symbol which may or not > exist depending on the header version, not a symbol whose value may > change. Cython doesn't actually read the .h files, so it can't figure out if a certain symbol is or is not defined. But if I understand your question correction, you want to be able to write some code with emits the #ifdef's? I would be in favor of adding something like this to the language, but cython.defined(SOME_NAME) makes it sound like its asking if SOME_NAME is defined in the current scope (e.g. with a DEF statement). Perhaps something like if cython.cmacro(cython.defined(SOME_NAME)): ... else: ... would generate #if defined(SOME_NAME) ... #else ... #endif One could then use it for more general macros as well. (Note that it still couldn't be used inside struct/enum/class definitions). As a hack, you can do cdef extern from *: cdef void emit_ifdef "#if defined(SOME_NAME) //" () cdef void emit_endif "#endif //" () emit_ifdef() print "hi" emit_endif() - Robert From robertwb at math.washington.edu Sun Dec 7 08:20:29 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 6 Dec 2008 23:20:29 -0800 Subject: [Cython] import/cimport gotcha? In-Reply-To: References: Message-ID: On Dec 5, 2008, at 8:05 AM, Lisandro Dalcin wrote: > I'm doing this: > > # newmodule.pyx > from package.module import Class > from package.module cimport Class > > but then Class in not put in the newmodule's dict. > > Is this expected? I'm not surprised, but I think this should work. - Robert From stefan_ml at behnel.de Sun Dec 7 08:39:59 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 07 Dec 2008 08:39:59 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> Message-ID: <493B7DCF.8070409@behnel.de> Hi, Jelle Feringa wrote: > cdef class FastLoopObj1d: > > cdef int N, i > cdef object obj > > def __init__(self, n, obj ): > self.N = n-1 > self.i = 0 > self.obj = obj > > def __iter__(self): > return self > > # no cpdef allowed? > def __next__( self ): > if self.i < self.N: > self.i+=1 > return self.obj[self.i] > else: > raise StopIteration > > I noticed that the cython iterator is about twice as fast compared to > the python version, when indexing an iterable object. > A respectable speedup, given how trivial the implementation is. Would > you consider the implementation optimal? Is this the fastest way of > indexing an iterable object? This looks a bit like a stripped down version of itertools.islice() to me. Have you tried that instead? Stefan From robertwb at math.washington.edu Sun Dec 7 10:12:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 7 Dec 2008 01:12:33 -0800 Subject: [Cython] Nice work! In-Reply-To: <20081204154931.GA7270@workerbee> References: <20081204154931.GA7270@workerbee> Message-ID: <97AEA859-35C2-4AD7-AFDC-B0F9F1473896@math.washington.edu> Yes, very cool. On Dec 4, 2008, at 7:49 AM, Gabriel Gellner wrote: > Just reading the what's new in Python 3 and saw that Cython is > recommended for > new extensions! > > http://docs.python.org/dev/3.0/howto/cporting.html#cporting-howto > (bottom of > page). > > Nicely done! > > Gabriel > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sun Dec 7 10:23:03 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 7 Dec 2008 01:23:03 -0800 Subject: [Cython] convert-range is default? In-Reply-To: <85b5c3130812020343k8817647vc2b1b5750d44a2e@mail.gmail.com> References: <85b5c3130812020343k8817647vc2b1b5750d44a2e@mail.gmail.com> Message-ID: On Dec 2, 2008, at 3:43 AM, Ondrej Certik wrote: > Hi, > > I noticed that convert-range was enabled by default now? > > It seems to have been enabled a long time ago: > > http://hg.cython.org/cython/rev/5450c26066e9 Yes. > So I am a little bit confused, because I have the feeling that I tried > that after it as well without the "--convert-range" and it didn't > produce the nice C for loop. But when I tried today, it indeed works. The most common cause is not cdef'ing the local variable. > Is this the reason why it was removed from the help? Yes. > http://hg.cython.org/cython/rev/7b8970eb4837 > > Btw, Dag, how is convert-range connected with "Option to emit #line > directives"? Or maybe you needed space for the new directive? It's not. I just deleted it when I was editing the help statement, because it hadn't been deleted earlier. (Technically, I guess that should have been a separate commit). > So in any case, "for i in range(10)" are now automatically converted > by default (if i is declared as an integer), so one doesn't need to > use the old syntax ("for i from 1 < 5" or something), right? Yes. - Robert From ggellner at uoguelph.ca Sun Dec 7 17:33:37 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sun, 7 Dec 2008 11:33:37 -0500 Subject: [Cython] Comments on example code? In-Reply-To: <20081125150522.GA17693@workerbee> References: <20081125150522.GA17693@workerbee> Message-ID: <20081207163337.GA8787@encolpuis> On Tue, Nov 25, 2008 at 10:05:22AM -0500, Gabriel Gellner wrote: > So I am giving a talk to my lab about doing some fast ODE solving using > python. Traditionally I have used f2py to define the callback function, but I > think Cython is a better fit for some of the newer students that don't know > fortran (though in this case it is easy to teach them). > > Now using the cython/numpy tutorial I can get my code to run around 50% slower > than the f2py generated code (which is still a healthy order of magnitude > faster than the python callback . . .). If there are any easy things I can do > to make the code faster (without sacrificing readability) I would be very > grateful! > So I made some code that uses the Numpy C-api for the call to empty and now the Cython code runs faster (or as fast) as the f2py! Joy. I will be getting it to the point that I can give a patch to numpy.pxd soon. Thanks for all the comments and help. Gabriel From aaron.devore at gmail.com Sun Dec 7 19:37:56 2008 From: aaron.devore at gmail.com (Aaron DeVore) Date: Sun, 7 Dec 2008 10:37:56 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <493B7DCF.8070409@behnel.de> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> Message-ID: <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> Out of curiosity, would most of the time be spent on the line "return self.obj[self.i]"? It seems like a __getitem__ call would be much slower than anything else in the class. -Aaron On Sat, Dec 6, 2008 at 11:39 PM, Stefan Behnel wrote: > Hi, > > Jelle Feringa wrote: >> cdef class FastLoopObj1d: >> >> cdef int N, i >> cdef object obj >> >> def __init__(self, n, obj ): >> self.N = n-1 >> self.i = 0 >> self.obj = obj >> >> def __iter__(self): >> return self >> >> # no cpdef allowed? >> def __next__( self ): >> if self.i < self.N: >> self.i+=1 >> return self.obj[self.i] >> else: >> raise StopIteration >> >> I noticed that the cython iterator is about twice as fast compared to >> the python version, when indexing an iterable object. >> A respectable speedup, given how trivial the implementation is. Would >> you consider the implementation optimal? Is this the fastest way of >> indexing an iterable object? > > This looks a bit like a stripped down version of itertools.islice() to me. > Have you tried that instead? > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Sun Dec 7 21:34:19 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 07 Dec 2008 21:34:19 +0100 Subject: [Cython] Temps again In-Reply-To: <49392829.1080102@behnel.de> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> Message-ID: <493C334B.2060406@behnel.de> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> When it comes to FlattenInListTransform, I was thinking about something >> along the lines of an "StatAsExprNode", which would be sort of the >> inverse of ExprStatNode: It would contain a block which could contain >> "ReturnFromStatAsExprNodeNode" (though a better name is needed :-)) >> which would yield the value of the statement list, evaluated as an >> expression. > > I thought about this, too. The main problems with TempsBlockNode are that > it a) requires the type of a temp variable before hand (which is bad for > transforms that run before type analysis), and b) is a block statement, not > an expression. Making it an ExprNode and moving the current use into a > ExprStatNode would fix that. I added a new node in UtilNodes that handles this. It's called EvalWithTempExprNode - better name pending. The idea is that you wrap an ExprNode in a EvalWithTempExprNode and replace the expression node that is used more than once inside the ExprNode with a ResultRefNode. The ResultRefNode will copy its type from the original expression during type analysis, so that it doesn't need to know it at creation time. The TempBlockExprNode then sets the result_code of the ResultRefNode after evaluating its expression and before evaluating the original ExprNode that needs the result. That way, evaluation of the expression and its use are separated by the EvalWithTempExprNode which can handle the temp allocation as required. Example from FlattenInListTransform: lhs = UtilNodes.ResultRefNode(node.operand1) conds = [] for arg in args: cond = ExprNodes.PrimaryCmpNode( pos = node.pos, operand1 = lhs, operator = eq_or_neq, operand2 = arg, cascade = None) conds.append(ExprNodes.TypecastNode( pos = node.pos, operand = cond, type = PyrexTypes.c_bint_type)) def concat(left, right): return ExprNodes.BoolBinopNode( pos = node.pos, operator = conjunction, operand1 = left, operand2 = right) condition = reduce(concat, conds) return UtilNodes.EvalWithTempExprNode(lhs, condition) I wonder if this can also be useful for common subexpression elimination - and if it makes sense to try such an optimisation in Cython... Stefan From greg.ewing at canterbury.ac.nz Mon Dec 8 00:47:14 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 08 Dec 2008 12:47:14 +1300 Subject: [Cython] IFDEF in cython ? In-Reply-To: <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> <493AD68E.1030809@student.matnat.uio.no> <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> Message-ID: <493C6082.4050400@canterbury.ac.nz> David Cournapeau wrote: > it won't work in that case: I want to use a symbol which may or not > exist depending on the header version, not a symbol whose value may > change. Is there some reason you need IFDEF in particular rather than just IF? Note that there's no way you're going to be able to test whether a symbol is defined in a .h file from Cython, since it knows nothing about the contents of .h files apart from what you tell it. -- Greg From michael.abshoff at googlemail.com Mon Dec 8 14:05:46 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Mon, 08 Dec 2008 05:05:46 -0800 Subject: [Cython] Cython wiki down Message-ID: <493D1BAA.8010709@gmail.com> Hello folks, unfortunately I accidentally too the Cython wiki down while restarting the other wikis hosted at sagemath.org. The Cython trac is alive and well. It will take me a little while to get it back up again since it is not part of the usual infrastructure we use for wikis on sagemath.org, so my apologies. But once I have figured this out I will add it to that machinery so this won't happen again the future. One thing I noticed is that the wiki contains a huge number of pages (around 8,000) of which most are spam. We are in the process of converting the various wikis related to Sage to a much more current moinmoin release so I can offer to upgrade the Cython wiki, too. The biggest benefit there would be much better Spam protection by adding a captcha. Since adding it to the Sage wiki a couple weeks ago we have not had a single spam account registered or an actual spamming which is down from about a dozen spam accounts registered per day. The conversion to the new wiki format is 99.9% automatic and accurate, i.e. so far we have had to fix two trivial formatting issues with the new pages. Thoughts? Cheers, Michael From michael.abshoff at googlemail.com Mon Dec 8 14:13:17 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Mon, 08 Dec 2008 05:13:17 -0800 Subject: [Cython] Cython wiki down In-Reply-To: <493D1BAA.8010709@gmail.com> References: <493D1BAA.8010709@gmail.com> Message-ID: <493D1D6D.6090202@gmail.com> Michael Abshoff wrote: > Hello folks, > It will take me a little while to get it back up again since it is not > part of the usual infrastructure we use for wikis on sagemath.org, so my > apologies. The wiki is up again - this was easier to find than I thought :) Cheers, Michael From dagss at student.matnat.uio.no Mon Dec 8 14:38:16 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 08 Dec 2008 14:38:16 +0100 Subject: [Cython] Comments on example code? In-Reply-To: <20081207163337.GA8787@encolpuis> References: <20081125150522.GA17693@workerbee> <20081207163337.GA8787@encolpuis> Message-ID: <493D2348.3090201@student.matnat.uio.no> Gabriel Gellner wrote: > On Tue, Nov 25, 2008 at 10:05:22AM -0500, Gabriel Gellner wrote: > >> So I am giving a talk to my lab about doing some fast ODE solving using >> python. Traditionally I have used f2py to define the callback function, but I >> think Cython is a better fit for some of the newer students that don't know >> fortran (though in this case it is easy to teach them). >> >> Now using the cython/numpy tutorial I can get my code to run around 50% slower >> than the f2py generated code (which is still a healthy order of magnitude >> faster than the python callback . . .). If there are any easy things I can do >> to make the code faster (without sacrificing readability) I would be very >> grateful! >> >> > So I made some code that uses the Numpy C-api for the call to empty and now > the Cython code runs faster (or as fast) as the f2py! Joy. I will be getting > it to the point that I can give a patch to numpy.pxd soon. > > Thanks for all the comments and help. > That's great news! I saw your post on numpy-discuss -- what what the solution?/why did it crash? I couldn't figure it out either... Dag Sverre From robertwb at math.washington.edu Mon Dec 8 20:34:38 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 8 Dec 2008 11:34:38 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> Message-ID: <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> On Dec 7, 2008, at 10:37 AM, Aaron DeVore wrote: > Out of curiosity, would most of the time be spent on the line "return > self.obj[self.i]"? It seems like a __getitem__ call would be much > slower than anything else in the class. Yes, almost certainly. > > -Aaron > > On Sat, Dec 6, 2008 at 11:39 PM, Stefan Behnel > wrote: >> Hi, >> >> Jelle Feringa wrote: >>> cdef class FastLoopObj1d: >>> >>> cdef int N, i >>> cdef object obj >>> >>> def __init__(self, n, obj ): >>> self.N = n-1 >>> self.i = 0 >>> self.obj = obj >>> >>> def __iter__(self): >>> return self >>> >>> # no cpdef allowed? >>> def __next__( self ): >>> if self.i < self.N: >>> self.i+=1 >>> return self.obj[self.i] >>> else: >>> raise StopIteration >>> >>> I noticed that the cython iterator is about twice as fast >>> compared to >>> the python version, when indexing an iterable object. >>> A respectable speedup, given how trivial the implementation is. >>> Would >>> you consider the implementation optimal? Is this the fastest way of >>> indexing an iterable object? >> >> This looks a bit like a stripped down version of itertools.islice >> () to me. >> Have you tried that instead? >> >> Stefan >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From jelleferinga at gmail.com Mon Dec 8 23:24:46 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Mon, 8 Dec 2008 23:24:46 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> Message-ID: <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> >> Out of curiosity, would most of the time be spent on the line "return >> self.obj[self.i]"? It seems like a __getitem__ call would be much >> slower than anything else in the class. > > Yes, almost certainly. Hi Aaron, Robert, Thanks for your response; any suggestion what might be more efficient than a __getitem__ call? Or is this class doomed by design? Cheers, -jelle From aaron.devore at gmail.com Tue Dec 9 00:31:07 2008 From: aaron.devore at gmail.com (Aaron DeVore) Date: Mon, 8 Dec 2008 15:31:07 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> Message-ID: <2ead2fb0812081531x18d8ad13v2d2c3b28f7096107@mail.gmail.com> If there's an internal Python/extension type data structure you could use the iterator from that. class Foo: def __init__(self): self.var = [] def __iter__(self): return iter(self.var) Besides that I can't think of anything. -Aaron On Mon, Dec 8, 2008 at 2:24 PM, Jelle Feringa wrote: >>> Out of curiosity, would most of the time be spent on the line "return >>> self.obj[self.i]"? It seems like a __getitem__ call would be much >>> slower than anything else in the class. >> >> Yes, almost certainly. > > Hi Aaron, Robert, > > Thanks for your response; any suggestion what might be more efficient > than a __getitem__ call? > Or is this class doomed by design? > > Cheers, > > -jelle > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From aaron.devore at gmail.com Tue Dec 9 00:42:06 2008 From: aaron.devore at gmail.com (Aaron DeVore) Date: Mon, 8 Dec 2008 15:42:06 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <2ead2fb0812081531x18d8ad13v2d2c3b28f7096107@mail.gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <2ead2fb0812081531x18d8ad13v2d2c3b28f7096107@mail.gmail.com> Message-ID: <2ead2fb0812081542i4921ff1ds1ea73d3083e03305@mail.gmail.com> That code example needs a bit of a modification. class Foo: def __init__(self): self.contents = [] # ...code that manipulates self.contents... def chunk(self, n): return iter(self.contents[:n]) On Mon, Dec 8, 2008 at 3:31 PM, Aaron DeVore wrote: > If there's an internal Python/extension type data structure you could > use the iterator from that. > > class Foo: > def __init__(self): > self.var = [] > def __iter__(self): > return iter(self.var) > > Besides that I can't think of anything. > > -Aaron > > > On Mon, Dec 8, 2008 at 2:24 PM, Jelle Feringa wrote: >>>> Out of curiosity, would most of the time be spent on the line "return >>>> self.obj[self.i]"? It seems like a __getitem__ call would be much >>>> slower than anything else in the class. >>> >>> Yes, almost certainly. >> >> Hi Aaron, Robert, >> >> Thanks for your response; any suggestion what might be more efficient >> than a __getitem__ call? >> Or is this class doomed by design? >> >> Cheers, >> >> -jelle >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > From jelleferinga at gmail.com Tue Dec 9 00:20:44 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Tue, 9 Dec 2008 00:20:44 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <2ead2fb0812081542i4921ff1ds1ea73d3083e03305@mail.gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <2ead2fb0812081531x18d8ad13v2d2c3b28f7096107@mail.gmail.com> <2ead2fb0812081542i4921ff1ds1ea73d3083e03305@mail.gmail.com> Message-ID: <54C52518-8BCC-4685-BCBC-3D7CEE7A8537@gmail.com> Right, but this isn't any different from looping through a vanilla python iterable, right? Simply typing the number that indexes the iterable already loops twice as fast, although obviously its rather trivial code. Actually, it would be interesting to see if a fast __getitem__ call would be available, since that would facilitate writing fast specialized iterators easily. -jelle On Dec 9, 2008, at 12:42 AM, Aaron DeVore wrote: > That code example needs a bit of a modification. > > class Foo: > def __init__(self): > self.contents = [] > # ...code that manipulates self.contents... > def chunk(self, n): > return iter(self.contents[:n]) > > On Mon, Dec 8, 2008 at 3:31 PM, Aaron DeVore > wrote: >> If there's an internal Python/extension type data structure you could >> use the iterator from that. >> >> class Foo: >> def __init__(self): >> self.var = [] >> def __iter__(self): >> return iter(self.var) >> >> Besides that I can't think of anything. >> >> -Aaron From hoytak at cs.ubc.ca Tue Dec 9 01:31:00 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 8 Dec 2008 16:31:00 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <54C52518-8BCC-4685-BCBC-3D7CEE7A8537@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <2ead2fb0812081531x18d8ad13v2d2c3b28f7096107@mail.gmail.com> <2ead2fb0812081542i4921ff1ds1ea73d3083e03305@mail.gmail.com> <54C52518-8BCC-4685-BCBC-3D7CEE7A8537@gmail.com> Message-ID: <4db580fd0812081631i4be5ef1es63bef20f7cd387a1@mail.gmail.com> > Actually, it would be interesting to see if a fast __getitem__ call > would be available, since that would facilitate writing fast > specialized iterators easily. There are faster ways of doing this, but only if you know the type of object that you're accessing with the [ ]. Cython uses optimized versions to access lists (and tuples I think), which do speed things up. The fastest is if it is a numpy array (http://docs.scipy.org/doc/), where cython uses the underlying c pointers to directly get the item, which is be orders of magnitude faster. --Hoyt -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Tue Dec 9 07:59:06 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Dec 2008 07:59:06 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> Message-ID: <493E173A.4070208@behnel.de> Hi, Jelle Feringa wrote: > Thanks for your response; any suggestion what might be more efficient > than a __getitem__ call? > Or is this class doomed by design? I would call it doomed by design. If you use it from Python, I don't think you can beat direct iteration through __getitem__ *of the object* that you iterate over by introducing another level of indirection that introduces a method call. I tried islice() and it's still a bit faster than the class you presented. Maybe you could give us some more insight into your actual use case, that might allow us to come up with better alternatives. Stefan From dagss at student.matnat.uio.no Tue Dec 9 22:49:02 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 09 Dec 2008 22:49:02 +0100 Subject: [Cython] Temps again In-Reply-To: <493C334B.2060406@behnel.de> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> <493C334B.2060406@behnel.de> Message-ID: <493EE7CE.7000708@student.matnat.uio.no> Stefan Behnel wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> When it comes to FlattenInListTransform, I was thinking about something >>> along the lines of an "StatAsExprNode", which would be sort of the >>> inverse of ExprStatNode: It would contain a block which could contain >>> "ReturnFromStatAsExprNodeNode" (though a better name is needed :-)) >>> which would yield the value of the statement list, evaluated as an >>> expression. >> I thought about this, too. The main problems with TempsBlockNode are that >> it a) requires the type of a temp variable before hand (which is bad for >> transforms that run before type analysis), and b) is a block statement, not >> an expression. Making it an ExprNode and moving the current use into a >> ExprStatNode would fix that. > > I added a new node in UtilNodes that handles this. It's called > EvalWithTempExprNode - better name pending. > > The idea is that you wrap an ExprNode in a EvalWithTempExprNode and replace > the expression node that is used more than once inside the ExprNode with a > ResultRefNode. The ResultRefNode will copy its type from the original > expression during type analysis, so that it doesn't need to know it at > creation time. The TempBlockExprNode then sets the result_code of the > ResultRefNode after evaluating its expression and before evaluating the > original ExprNode that needs the result. That way, evaluation of the > expression and its use are separated by the EvalWithTempExprNode which can > handle the temp allocation as required. Very nice! I don't have time to dive into the finer points of this yet, but some thoughts: - It took me some minutes to understand the concept. I definitely think another name (as you suggest) would help a lot and perhaps make the concept immediately graspable. How about "LetNode" for EvalWithTempExprNode? (I don't have a good suggestion for the reference node, "LetRefNode" or similar in lack of something better). - It seems to mostly be "TempsBlockNode" except with a different (better?) API and designed to work as an ExprNode. Is it redundant with both? I.e. "LetNode" could perhaps implement both interfaces and play the role both as ExprNode and StatNode. (No hurry though!) -- Dag Sverre From dagss at student.matnat.uio.no Tue Dec 9 23:17:20 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 09 Dec 2008 23:17:20 +0100 Subject: [Cython] Temps again In-Reply-To: <493C334B.2060406@behnel.de> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> <493C334B.2060406@behnel.de> Message-ID: <493EEE70.1010905@student.matnat.uio.no> Stefan Behnel wrote: > I wonder if this can also be useful for common subexpression elimination - > and if it makes sense to try such an optimisation in Cython... I think almost all Python expressions can have potential side-effects which makes such optimisation impossible -- and with non-Python subexpressions it can at least technically be considered the job of the C compiler. Or? -- Dag Sverre From stefan_ml at behnel.de Wed Dec 10 07:16:30 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Dec 2008 07:16:30 +0100 Subject: [Cython] Temps again In-Reply-To: <493EE7CE.7000708@student.matnat.uio.no> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> <493C334B.2060406@behnel.de> <493EE7CE.7000708@student.matnat.uio.no> Message-ID: <493F5EBE.5070600@behnel.de> Hi, Dag Sverre Seljebotn wrote: > - It took me some minutes to understand the concept. I definitely think > another name (as you suggest) would help a lot and perhaps make the > concept immediately graspable. How about "LetNode" for > EvalWithTempExprNode? (I don't have a good suggestion for the reference > node, "LetRefNode" or similar in lack of something better). Sounds like what it does, yes. "EvalWithTempExprNode" was the third name already, as the underlying concept changed with trial-and-error. > - It seems to mostly be "TempsBlockNode" except with a different > (better?) API and designed to work as an ExprNode. Is it redundant with > both? It's a bit hackish in that it sets the result_code from outside. It also only handles a single expression, but I actually think that this makes it cleaner. You can easily chain it. It should also be used in a more fine-granular way than the TempsBlockNode, to allow early temp deallocation. There's a difference if you don't have an expression to wrap and only want to allocate temps. It doesn't handle that case. Although you could argue that you could always use an expression that returns the type specific init value (NULL, 0, None, ...). BTW, I'm not 100% sure yet that it works in all places where you might want to put it. But if it fails somewhere, that should not be too hard to fix. It will not fix the with statement for you, though. I start to think that a dedicated node might still be the best way to make this work. > I.e. "LetNode" could perhaps implement both interfaces and play > the role both as ExprNode and StatNode. (No hurry though!) You should be able to wrap the node in an ExprStatNode, no need to implement two interfaces here (although it might simplify things). Stefan From stefan_ml at behnel.de Wed Dec 10 07:32:55 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Dec 2008 07:32:55 +0100 Subject: [Cython] Temps again In-Reply-To: <493EEE70.1010905@student.matnat.uio.no> References: <3311217559.495138@smtp.netcom.no> <493784DA.9010803@behnel.de> <4938350A.50003@behnel.de> <493923E3.8030402@student.matnat.uio.no> <49392829.1080102@behnel.de> <493C334B.2060406@behnel.de> <493EEE70.1010905@student.matnat.uio.no> Message-ID: <493F6297.6050609@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> I wonder if this can also be useful for common subexpression elimination - >> and if it makes sense to try such an optimisation in Cython... > > I think almost all Python expressions can have potential side-effects > which makes such optimisation impossible -- and with non-Python > subexpressions it can at least technically be considered the job of the > C compiler. Or? Right, that was my feeling, too. Another thing is stuff that the C compiler can't optimise due to INCREF/DECREF use, but I can't think of an example here, either. Stefan From cournape at gmail.com Wed Dec 10 08:27:28 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 10 Dec 2008 16:27:28 +0900 Subject: [Cython] IFDEF in cython ? In-Reply-To: <655C5956-3020-424E-90C3-74783E295D47@math.washington.edu> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> <493AD68E.1030809@student.matnat.uio.no> <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> <655C5956-3020-424E-90C3-74783E295D47@math.washington.edu> Message-ID: <5b8d13220812092327u23bee1eagffff966cc1187f8e@mail.gmail.com> On Sun, Dec 7, 2008 at 1:46 PM, Robert Bradshaw wrote: > > > Cython doesn't actually read the .h files, so it can't figure out if > a certain symbol is or is not defined. But if I understand your > question correction, you want to be able to write some code with > emits the #ifdef's? Yes, exactly. > > One could then use it for more general macros as well. (Note that it > still couldn't be used inside struct/enum/class definitions). I am not so familiar with cython, so I don't have anything to say on which syntax is best. > > As a hack, you can do > > cdef extern from *: > cdef void emit_ifdef "#if defined(SOME_NAME) //" () > cdef void emit_endif "#endif //" () > > emit_ifdef() > print "hi" > emit_endif() Hm, did not think about that. I guess that would do for the time being, thanks, David From cournape at gmail.com Wed Dec 10 08:34:07 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 10 Dec 2008 16:34:07 +0900 Subject: [Cython] IFDEF in cython ? In-Reply-To: <493C6082.4050400@canterbury.ac.nz> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> <493AD68E.1030809@student.matnat.uio.no> <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> <493C6082.4050400@canterbury.ac.nz> Message-ID: <5b8d13220812092334x3709786fpda6df1433a57d32f@mail.gmail.com> On Mon, Dec 8, 2008 at 8:47 AM, Greg Ewing wrote: > David Cournapeau wrote: > >> it won't work in that case: I want to use a symbol which may or not >> exist depending on the header version, not a symbol whose value may >> change. > > Is there some reason you need IFDEF in particular > rather than just IF? Yes, since the symbol exists in some versions of the header only. > > Note that there's no way you're going to be able to > test whether a symbol is defined in a .h file from > Cython, since it knows nothing about the contents > of .h files apart from what you tell it. Yes, I understand that I can't use the symbol in cython: I just want to control it in the generated C file. Robert solution with fake functions which translate to strings does exactly what I want to do. David From cournape at gmail.com Wed Dec 10 09:10:23 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 10 Dec 2008 17:10:23 +0900 Subject: [Cython] cdef class and docstrings Message-ID: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> Hi, Is it possible to include docstring in a cdef class exactly as it would work for python classes, or do I have to use python wrappers around the cython class ? As an example: class Yo: def __init__(self, arg1='arg1'): "some doc" This python class can be queried easily from ipython for documentation, including the signature and default argument values. But the docstring is not available if the class is a cdef class in cython: cdef Yo: def __init__(selg, arg1='arg1'): "Some doc" The Yo.__init__.__doc__ is not "some doc", and I can't get the signature in ipython. Is this expected, or am I doing something wrong here ? cheers, David From stefan_ml at behnel.de Wed Dec 10 09:35:31 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Dec 2008 09:35:31 +0100 (CET) Subject: [Cython] cdef class and docstrings In-Reply-To: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> Message-ID: <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> David Cournapeau wrote: > Is it possible to include docstring in a cdef class exactly as it > would work for python classes, or do I have to use python wrappers > around the cython class ? As an example: > > cdef Yo: > def __init__(selg, arg1='arg1'): > "Some doc" > > The Yo.__init__.__doc__ is not "some doc", and I can't get the > signature in ipython. Is this expected, or am I doing something wrong > here ? I find this unexpected, too. However, have you tried putting the docstring where it actually belongs? As in cdef class Yo: "Some doc" ... Stefan From robertwb at math.washington.edu Wed Dec 10 09:36:48 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 10 Dec 2008 00:36:48 -0800 Subject: [Cython] IFDEF in cython ? In-Reply-To: <5b8d13220812092327u23bee1eagffff966cc1187f8e@mail.gmail.com> References: <5b8d13220812060527m23921d55wdaafc191a6bfd6fe@mail.gmail.com> <493AD68E.1030809@student.matnat.uio.no> <5b8d13220812061849s6428f943q89c4cf520e4b771a@mail.gmail.com> <655C5956-3020-424E-90C3-74783E295D47@math.washington.edu> <5b8d13220812092327u23bee1eagffff966cc1187f8e@mail.gmail.com> Message-ID: <8DCE73B8-1718-4099-9640-C2157016C470@math.washington.edu> On Dec 9, 2008, at 11:27 PM, David Cournapeau wrote: > On Sun, Dec 7, 2008 at 1:46 PM, Robert Bradshaw > wrote: > >> >> >> Cython doesn't actually read the .h files, so it can't figure out if >> a certain symbol is or is not defined. But if I understand your >> question correction, you want to be able to write some code with >> emits the #ifdef's? > > Yes, exactly. > >> >> One could then use it for more general macros as well. (Note that it >> still couldn't be used inside struct/enum/class definitions). > > I am not so familiar with cython, so I don't have anything to say on > which syntax is best. A better question would be what's a "pythonic" syntax, which is the first question I ask when thinking of a new feature to add. >> As a hack, you can do >> >> cdef extern from *: >> cdef void emit_ifdef "#if defined(SOME_NAME) //" () >> cdef void emit_endif "#endif //" () >> >> emit_ifdef() >> print "hi" >> emit_endif() > > Hm, did not think about that. I guess that would do for the time > being, Yeah, it's pretty hackish, but it works. From cournape at gmail.com Wed Dec 10 09:57:05 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 10 Dec 2008 17:57:05 +0900 Subject: [Cython] cdef class and docstrings In-Reply-To: <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> On Wed, Dec 10, 2008 at 5:35 PM, Stefan Behnel wrote: > David Cournapeau wrote: >> Is it possible to include docstring in a cdef class exactly as it >> would work for python classes, or do I have to use python wrappers >> around the cython class ? As an example: >> >> cdef Yo: >> def __init__(selg, arg1='arg1'): >> "Some doc" >> >> The Yo.__init__.__doc__ is not "some doc", and I can't get the >> signature in ipython. Is this expected, or am I doing something wrong >> here ? > > I find this unexpected, too. However, have you tried putting the docstring > where it actually belongs? As in > > cdef class Yo: > "Some doc" Hm, why should the __init__ method belong to the class string ? Those are different, in my mind: the Yo.__doc__ tells about the Yo class purpose, and the Yo.__init__.__doc__ should give details about the construction (arguments, + signature in the case of ipython advanced help options). The class docstring does work, though; the __init__ is: x.__init__(...) initializes x; see x.__class__.__doc__ for signature And Yo.__class__.__doc__ is: "type(object) -> the object's type\ntype(name, bases, dict) -> a new type" David From stefan_ml at behnel.de Wed Dec 10 10:10:56 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Dec 2008 10:10:56 +0100 (CET) Subject: [Cython] cdef class and docstrings In-Reply-To: <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> Message-ID: <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> David Cournapeau wrote: > On Wed, Dec 10, 2008 at 5:35 PM, Stefan Behnel wrote: >> cdef class Yo: >> "Some doc" > > Hm, why should the __init__ method belong to the class string ? Those > are different, in my mind: the Yo.__doc__ tells about the Yo class > purpose, and the Yo.__init__.__doc__ should give details about the > construction So, to construct an instance of a class, you would write yo = Yo.__init__(...) is it that what you are saying? > The class docstring does work, though; the __init__ is: > > x.__init__(...) initializes x; see x.__class__.__doc__ for signature > > And Yo.__class__.__doc__ is: > > "type(object) -> the object's type\ntype(name, bases, dict) -> a new type" As you can see, the docstring of __init__() is fine, but the class docstring isn't. Extension classes behave like Python builtins (or vice versa). Their special methods simply don't have a docstring (try tuple.__init__.__doc__, for example). That's why the standard docstring refers to the class docstring. Stefan From cournape at gmail.com Wed Dec 10 10:27:49 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 10 Dec 2008 18:27:49 +0900 Subject: [Cython] cdef class and docstrings In-Reply-To: <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <5b8d13220812100127q5794423ep5a5ef08d6891c1fc@mail.gmail.com> On Wed, Dec 10, 2008 at 6:10 PM, Stefan Behnel wrote: > David Cournapeau wrote: >> On Wed, Dec 10, 2008 at 5:35 PM, Stefan Behnel wrote: >>> cdef class Yo: >>> "Some doc" >> >> Hm, why should the __init__ method belong to the class string ? Those >> are different, in my mind: the Yo.__doc__ tells about the Yo class >> purpose, and the Yo.__init__.__doc__ should give details about the >> construction > > So, to construct an instance of a class, you would write > > yo = Yo.__init__(...) No :) > > is it that what you are saying? I meant that for a class, I find both class and __init__ docstrings useful. In particular, I prefer not to write any signature in the docstring, which means in that case I don't have the information at all. > > As you can see, the docstring of __init__() is fine, but the class > docstring isn't. Extension classes behave like Python builtins (or vice > versa). Their special methods simply don't have a docstring (try > tuple.__init__.__doc__, for example). That's why the standard docstring > refers to the class docstring. Ah, that explains it. I have never really look into the doc of builtins - which are rarely built from __init__ now that I think about it - which explains my surprise. Still, I would prefer to have at least the signature of __init__ (for default arguments), but I guess my only option is to use a pure python class wrapper, in that case, David From jelleferinga at gmail.com Wed Dec 10 11:24:33 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Wed, 10 Dec 2008 11:24:33 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <493E173A.4070208@behnel.de> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> Message-ID: <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> > I would call it doomed by design. If you use it from Python, I don't > think > you can beat direct iteration through __getitem__ *of the object* > that you > iterate over by introducing another level of indirection that > introduces a > method call. I tried islice() and it's still a bit faster than the > class > you presented. Given that islice does the same thing better definitely doomed ;') > Maybe you could give us some more insight into your actual use case, > that > might allow us to come up with better alternatives. Thanks, that's very kind. Thing is that currently I'm working on sampling b-spline surfaces, and compute the curvature of such a surface. Basically that comes down to making a 500*500 grid of curvature samples. Computing the curvature is really quick ( call to a SWIG wrapped object ). My code looks like: for a in xrange(1, 500): for b in xrange(1, 500): do_curvature() When I place pass instead of do_curvature() I notice that the time for such a loop is basically the same, which is why I'm interested in moving the for loops to cython. This is a pattern that is happens often in my code, hence it would be lovely to make a function something like: loop_fast2d( do_curvature, (1,500), (1,500) ) What makes this pattern hard is that you would have to loop again through the results, which reduces the time saved by a fast loop_fast2d function by half. So perhaps this pattern is more effective: loop_fast2d( do_curvature, do_result, (1,500), (1,500) ) What do you think? Is this pattern generic enough to be sped up in Cython? Thanks, -jelle From greg.ewing at canterbury.ac.nz Wed Dec 10 11:39:40 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 10 Dec 2008 23:39:40 +1300 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> Message-ID: <493F9C6C.9080108@canterbury.ac.nz> Jelle Feringa wrote: > loop_fast2d( do_curvature, do_result, (1,500), (1,500) ) > > What do you think? Is this pattern generic enough to be sped up in > Cython? I'd be worried that the cost of two Python function calls for every iteration would swamp any gains made from doing the looping part in Cython. -- Greg From robertwb at math.washington.edu Wed Dec 10 11:48:11 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 10 Dec 2008 02:48:11 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> Message-ID: On Dec 10, 2008, at 2:24 AM, Jelle Feringa wrote: >> I would call it doomed by design. If you use it from Python, I don't >> think >> you can beat direct iteration through __getitem__ *of the object* >> that you >> iterate over by introducing another level of indirection that >> introduces a >> method call. I tried islice() and it's still a bit faster than the >> class >> you presented. > > Given that islice does the same thing better definitely doomed ;') You might be able to get speedups depending on your particular application. For example, if the think you're indexing into is slow, you could cache the result on the first lookup. Or, if it's a (small enough) int or floating point value, you could cache that and provide a nice C interface to it. But islice is about as fast as it's going to get and still be completely generic. >> Maybe you could give us some more insight into your actual use case, >> that >> might allow us to come up with better alternatives. > > Thanks, that's very kind. > Thing is that currently I'm working on sampling b-spline surfaces, and > compute the curvature of such a surface. > Basically that comes down to making a 500*500 grid of curvature > samples. > Computing the curvature is really quick ( call to a SWIG wrapped > object ). We must have different definitions of quick :). > My code looks like: > > for a in xrange(1, 500): > for b in xrange(1, 500): > do_curvature() > > When I place pass instead of do_curvature() I notice that the time for > such a loop is basically the same, which is why I'm interested in > moving the for loops to cython. The loops in Cython will be very fast. A 500x500 loop will be so fast it will be hard to measure. Calling a Python function or two inside the loop, however, will erase much of the benifit. > This is a pattern that is happens often in my code, hence it would be > lovely to make a function something like: > > loop_fast2d( do_curvature, (1,500), (1,500) ) > > What makes this pattern hard is that you would have to loop again > through the results, which reduces the time saved by a fast > loop_fast2d function by half. In Cython, the loop overhead is will be orders of magnitude smaller than the overhead of the function calls, so I don't think it'd matter. > So perhaps this pattern is more effective: > > loop_fast2d( do_curvature, do_result, (1,500), (1,500) ) > > What do you think? Is this pattern generic enough to be sped up in > Cython? The question should be "is this pattern too generic to be sped up in Cython." If, as your experiments indicate, the empty loop is almost as expensive as the do_curvature loop, then Cython can help you there. However, it can't speed up the 250000 calls to do_curvature. - Robert From jelleferinga at gmail.com Wed Dec 10 11:41:44 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Wed, 10 Dec 2008 11:41:44 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <493F9C6C.9080108@canterbury.ac.nz> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> <493F9C6C.9080108@canterbury.ac.nz> Message-ID: <32DCD591-62A0-4E35-B1C8-AFCB1B6B7025@gmail.com> > I'd be worried that the cost of two Python function calls > for every iteration would swamp any gains made from doing > the looping part in Cython. Well, the do_curvature call is a call to a SWIG wrapped module. The call is really quick. What's slowing down is the python for loops, not the call. From dagss at student.matnat.uio.no Wed Dec 10 12:15:17 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 10 Dec 2008 12:15:17 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> Message-ID: <493FA4C5.5060509@student.matnat.uio.no> Jelle Feringa wrote: >> I would call it doomed by design. If you use it from Python, I don't >> think >> you can beat direct iteration through __getitem__ *of the object* >> that you >> iterate over by introducing another level of indirection that >> introduces a >> method call. I tried islice() and it's still a bit faster than the >> class >> you presented. >> > > Given that islice does the same thing better definitely doomed ;') > > >> Maybe you could give us some more insight into your actual use case, >> that >> might allow us to come up with better alternatives. >> > > Thanks, that's very kind. > Thing is that currently I'm working on sampling b-spline surfaces, and > compute the curvature of such a surface. > Basically that comes down to making a 500*500 grid of curvature samples. > Computing the curvature is really quick ( call to a SWIG wrapped > object ). > My code looks like: > > for a in xrange(1, 500): > for b in xrange(1, 500): > do_curvature() > > When I place pass instead of do_curvature() I notice that the time for > such a loop is basically the same, which is why I'm interested in > moving the for loops to cython. > This is a pattern that is happens often in my code, hence it would be > lovely to make a function something like: > > loop_fast2d( do_curvature, (1,500), (1,500) ) > > What makes this pattern hard is that you would have to loop again > through the results, which reduces the time saved by a fast > loop_fast2d function by half. > So perhaps this pattern is more effective: > > loop_fast2d( do_curvature, do_result, (1,500), (1,500) ) > > What do you think? Is this pattern generic enough to be sped up in > Cython? > You should read through the recent thread "Comments on example code" which touches on the same issue. As Robert said our definitions of "quick" likely vary a great deal. This is how to get quick code though: You need to drop the SWIG wrapper and instead wrap the code using Cython. OTOH, that will likely give between 50x to 1000x speedup depending on the usecase. Basically you need to make sure that "do_curvature" and "do_result" can be called in a fast way. One way of doing that is cdef class Float2DGridFunc: cdef float evaluate(int i, int j): pass cdef extern from "yourheader.h": # define C function you are calling the Cython way cdef class DoCurvature(Float2DGridFunc): cdef float evaluate(int i, int j): return direct_call_to_c_function(i, j) And keep the implementation of DoCurvature in Cython. Then you will be able to do this in plain Python: import your_cython_mod your_cython_mode.loop_fast_2d(your_cython_mode.DoCurvature(), ...) Dag Sverre From jelleferinga at gmail.com Wed Dec 10 12:30:34 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Wed, 10 Dec 2008 12:30:34 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> Message-ID: <9933BD9E-821D-4987-B4B6-7D00FF79C5D8@gmail.com> Hi Robert, Thanks so much for your insight. > The question should be "is this pattern too generic to be sped up in > Cython." Oops, that's what I meant actually... > If, as your experiments indicate, the empty loop is almost > as expensive as the do_curvature loop, then Cython can help you > there. Correct, that is the case. > However, it can't speed up the 250000 calls to do_curvature. Of course, though these are nearly as quick as the C++ call. Which is why the looping overhead is so nasty. Certainly when looping through reals rather than integers, things do get slow. Here's an example of a typical pattern. Profiling shows that computing curvature and storing these in a wrapped C++ datatype is very quick. # computing the def curvature( u,v ): return gaussian_curvature # storing the results: a_wrapped_datatype( indx, gaussian_curvature ) # naive really slow version from numpy import ogrid def curvature_grid( curvature, a_wrapped_datatype, n_samples ): n=1 for a in ogrid[ 0:1:complex(n_samples)].tolist(): for b in ogrid[ 0:1:complex(n_samples)].tolist(): curv = curvature( a, b ) a_wrapped_datatype( n, curv ) n+=1 return a_wrapped_datatype I'm hoping Cython could help me to make an efficient n dimensional loop. Such a pattern would be quite elegant: although I have got fast function calls, but now python is introducing serious overhead when looping. Since you do this all the time, it would be really interesting to see if it can be sped up. Though thanks due to all the replied I understand that this is in fact pretty difficult to do in a generic way. The following would be an wonderful function to have: loop_fastnd( curvature, store_curvature, ( (0, n), (0, n) ) ) -jelle From jelleferinga at gmail.com Wed Dec 10 12:42:53 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Wed, 10 Dec 2008 12:42:53 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <493FA4C5.5060509@student.matnat.uio.no> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> <493FA4C5.5060509@student.matnat.uio.no> Message-ID: <4BF31E94-83CB-4A24-8D1A-234B9FC9C8C5@gmail.com> Hi Dag Severre, Thanks, I will read through the "Comments on example code" thread, sorry I missed out on that. Thanks also for the example you gave. I did not understood why its necessary to wrap the do_curvature call in Cython though: the call is close enough to C++ execution speed. The API I'm working with is *huge*, in C++ and already wrapped in SWIG ( using introspecting in gcc_xml ) , so moving it to Cython is really not an option. If the function is fast, why is it necessary to move it to Cython? Like Robert points out: I can measure a 500*500 loop in Python, it would be hard to measure it in Cython. That's the bit I'm interested in ;') Perhaps my question is naive, but I'm trying to understand why. -jelle From dagss at student.matnat.uio.no Wed Dec 10 14:16:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 10 Dec 2008 14:16:00 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <4BF31E94-83CB-4A24-8D1A-234B9FC9C8C5@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> <493FA4C5.5060509@student.matnat.uio.no> <4BF31E94-83CB-4A24-8D1A-234B9FC9C8C5@gmail.com> Message-ID: <493FC110.3090404@student.matnat.uio.no> Jelle Feringa wrote: > Hi Dag Severre, > > Thanks, I will read through the "Comments on example code" thread, > sorry I missed out on that. > Thanks also for the example you gave. > > I did not understood why its necessary to wrap the do_curvature call > in Cython though: the call is close enough to C++ execution speed. > This is the point we disagree on :-) When wrapping the function using SWIG, it means that calling it has to happen by calling a Python function -- meaning amongst other things that the arguments and results are packed into Python objects and unpacked again etc. Of course, the relative impact of this depends on how much time is spent in the C++ function. > Like Robert points out: I can measure a 500*500 loop in Python, it > would be hard to measure it in Cython. > That's the bit I'm interested in ;') > But it is certainly measurable if you put a SWIG function call within it! If the call overhead of calling the SWIG function times 500^2 is acceptable, then you are indeed right, the following can be implemented and will give the speed you are after: loop_fast2d( do_curvature, do_result, (1,500), (1,500) ) It is just that we don't consider it fast enough. This is likely to give a 2x speed increase at best, while the speed increases we seek using Cython are typically in the order of 100x. Dag Sverre From jelleferinga at gmail.com Wed Dec 10 14:13:03 2008 From: jelleferinga at gmail.com (Jelle Feringa) Date: Wed, 10 Dec 2008 14:13:03 +0100 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <493FC110.3090404@student.matnat.uio.no> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> <493FA4C5.5060509@student.matnat.uio.no> <4BF31E94-83CB-4A24-8D1A-234B9FC9C8C5@gmail.com> <493FC110.3090404@student.matnat.uio.no> Message-ID: Hi Dag Severre, Thanks so much for your reply; I perfectly understood the aspects involved now. I understand the argument why to wrap the function in Cython now. I did got your predicted 2* speedup, given how trivial the code is, already something. Thanks again, -jelle On Dec 10, 2008, at 2:16 PM, Dag Sverre Seljebotn wrote: > Jelle Feringa wrote: >> Hi Dag Severre, >> >> Thanks, I will read through the "Comments on example code" thread, >> sorry I missed out on that. >> Thanks also for the example you gave. >> >> I did not understood why its necessary to wrap the do_curvature call >> in Cython though: the call is close enough to C++ execution speed. >> > This is the point we disagree on :-) When wrapping the function using > SWIG, it means that calling it has to happen by calling a Python > function -- meaning amongst other things that the arguments and > results > are packed into Python objects and unpacked again etc. > > Of course, the relative impact of this depends on how much time is > spent > in the C++ function. >> Like Robert points out: I can measure a 500*500 loop in Python, it >> would be hard to measure it in Cython. >> That's the bit I'm interested in ;') >> > But it is certainly measurable if you put a SWIG function call > within it! > > If the call overhead of calling the SWIG function times 500^2 is > acceptable, then you are indeed right, the following can be > implemented > and will give the speed you are after: > > loop_fast2d( do_curvature, do_result, (1,500), (1,500) ) > > > It is just that we don't consider it fast enough. This is likely to > give > a 2x speed increase at best, while the speed increases we seek using > Cython are typically in the order of 100x. > > Dag Sverre From greg.ewing at canterbury.ac.nz Thu Dec 11 00:24:06 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Dec 2008 12:24:06 +1300 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <4BF31E94-83CB-4A24-8D1A-234B9FC9C8C5@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> <493FA4C5.5060509@student.matnat.uio.no> <4BF31E94-83CB-4A24-8D1A-234B9FC9C8C5@gmail.com> Message-ID: <49404F96.8040106@canterbury.ac.nz> Jelle Feringa wrote: > I did not understood why its necessary to wrap the do_curvature call > in Cython though: the call is close enough to C++ execution speed. If you're finding it fast enough for your purposes, then that's fine. It's just that it would be faster still if you got rid of the overhead of the Python function call mechanism and called the underlying C or C++ function directly. -- Greg From dalcinl at gmail.com Thu Dec 11 02:27:01 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 10 Dec 2008 22:27:01 -0300 Subject: [Cython] want to implement support for C99/C++ complex Message-ID: Could any of you (Greg?) give me a executive summary about should I change/add to Cython/Pyrex scanner/parser/generator to make this possible? I believe the scanner does not need much (if any) work, but up to now I cannot get Cython properly recognize something like "cdef float complex z" or "cdef double complex z". -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ajaksu at gmail.com Thu Dec 11 06:07:59 2008 From: ajaksu at gmail.com (Daniel (ajax) Diniz) Date: Thu, 11 Dec 2008 03:07:59 -0200 Subject: [Cython] __next__ | not being able to use cpdef Message-ID: <2d75d7660812102107t77aba200q8fab8ffef675404e@mail.gmail.com> Am I the only one thinking Jelle might know which bits of his function are slow? :) Jelle Feringa wrote: > n=1 > for a in ogrid[ 0:1:complex(n_samples)].tolist(): > for b in ogrid[ 0:1:complex(n_samples)].tolist(): > curv = curvature( a, b ) > a_wrapped_datatype( n, curv ) > n+=1 > return a_wrapped_datatype I think your loop might be doing unnecessary work. See if this helps: cdef int a, b, arrlen myarray = ogrid[ 0:1:complex(n_samples)] arrlen = len(myarray) for a from 0 <= a < arrlen: for b from 0 <= b < arrlen: curv = curvature( myarr[a], myarr[b]) The "cdef int" shouldn't give you any benefit right now, rather being slower. If you can index the array (or its buffer) with non-Python ints, it should be faster (look at numpy integration, Dag's work :). If not, just avoiding the calls to tolist() in the inner loop could be nice. Check http://docs.cython.org/docs/numpy_tutorial.html Now, if you could bypass the python layer in certain places, like calling the real C++_curvature with a real float instead of a python float, cdef'ing n to [numpy]int and a_wrapped_datatype to (int, float?), you'd get things going faster without having to start a new wrapper from zero. Regards, Daniel From stefan_ml at behnel.de Thu Dec 11 07:14:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 11 Dec 2008 07:14:36 +0100 Subject: [Cython] cdef class and docstrings In-Reply-To: <5b8d13220812100127q5794423ep5a5ef08d6891c1fc@mail.gmail.com> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100127q5794423ep5a5ef08d6891c1fc@mail.gmail.com> Message-ID: <4940AFCC.1000501@behnel.de> Hi, David Cournapeau wrote: > I would prefer to have at > least the signature of __init__ (for default arguments) Cython can embed the signature of a function or method in its docstring. However, my guess is that this doesn't work for __init__ yet, where it would have to put it into the respective class (at least, there's no test for that). Sounds like a bug to me. Stefan From dagss at student.matnat.uio.no Thu Dec 11 08:11:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 11 Dec 2008 08:11:00 +0100 Subject: [Cython] want to implement support for C99/C++ complex Message-ID: <3311827901.792492@smtp.netcom.no> Great! I already wrote some descriptions along those lines at http://wiki.cython.org/smallprojects (#2 IIRC). What I suggest there would work for all C compilers, but using C99 complex instead shouldn't be much different (but not much less work either...) Dag Sverre Seljebotn -----Original Message----- From: "Lisandro Dalcin" Date: Thursday, Dec 11, 2008 2:27 am Subject: [Cython] want to implement support for C99/C++ complex To: cython-dev Reply-To: cython-dev at codespeak.net Could any of you (Greg?) give me a executive summary about should I >change/add to Cython/Pyrex scanner/parser/generator to make this >possible? I believe the scanner does not need much (if any) work, but >up to now I cannot get Cython properly recognize something like "cdef >float complex z" or "cdef double complex z". > >-- >Lisandro Dalc?n >--------------- >Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >PTLC - G?emes 3450, (3000) Santa Fe, Argentina >Tel/Fax: +54-(0)342-451.1594 >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Thu Dec 11 08:51:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 10 Dec 2008 23:51:22 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> Message-ID: On Dec 10, 2008, at 2:24 AM, Jelle Feringa wrote: >> Thing is that currently I'm working on sampling b-spline surfaces, >> and > compute the curvature of such a surface. > Basically that comes down to making a 500*500 grid of curvature > samples. > Computing the curvature is really quick ( call to a SWIG wrapped > object ). > My code looks like: > > for a in xrange(1, 500): > for b in xrange(1, 500): > do_curvature() Just to follow up on this, I did some timings. %cython def do_loop_c(f, int n, int m): cdef int i, j for i in range(n): for j in range(m): f() cdef foo_c(): pass def do_loop_c_call(int n, int m): cdef int i, j for i in range(n): for j in range(m): foo_c() %python def foo(): """A "fast" function to call.""" pass def do_loop_py(f, n, m): for i in xrange(n): for j in xrange(m): f() def do_loop_empty(f, n, m): for i in xrange(n): for j in xrange(m): pass sage: time do_loop_py(foo, 5000, 500) CPU time: 0.56 s, Wall time: 0.56 s sage: time do_loop_empty(foo, 5000, 500) CPU time: 0.10 s, Wall time: 0.10 s sage: time do_loop_c(foo, 5000, 500) CPU time: 0.34 s, Wall time: 0.34 s sage: time do_loop_c_call(5000, 500) CPU time: 0.00 s, Wall time: 0.00 s sage: time do_loop_c_call(5000, 50000) # 100x anything above CPU time: 0.22 s, Wall time: 0.22 s The moral of the story is that the loop overhead can be reduced to essentially a clock cycle or two per iteration (nanoseconds) but the cost to call a Python function is several of orders of magnitude more that that. - Robert From stefan_ml at behnel.de Thu Dec 11 10:13:37 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 11 Dec 2008 10:13:37 +0100 (CET) Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> Message-ID: <42689.213.61.181.86.1228986817.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > %cython > > cdef foo_c(): > pass > > def do_loop_c_call(int n, int m): > cdef int i, j > for i in range(n): > for j in range(m): > foo_c() > [...] > sage: time do_loop_c_call(5000, 500) > CPU time: 0.00 s, Wall time: 0.00 s > > The moral of the story is that the loop overhead can be reduced to > essentially a clock cycle or two per iteration (nanoseconds) Have you looked at the assembly that the above code compiles into? I wouldn't be surprised if the C compiler inlined the call to foo_c(), and then removed the entire loop as useless code. Stefan From robertwb at math.washington.edu Thu Dec 11 10:17:01 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 11 Dec 2008 01:17:01 -0800 Subject: [Cython] __next__ | not being able to use cpdef In-Reply-To: <42689.213.61.181.86.1228986817.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <77CE8443-87D6-42A5-9E96-A4ACAD7527F5@gmail.com> <3DCEB23D-5837-4B44-A7D2-E090EECF860B@gmail.com> <493B7DCF.8070409@behnel.de> <2ead2fb0812071037ue66eb97sa38b0213aa6232a0@mail.gmail.com> <3FFA1C92-C478-4EC7-AA94-AC3B6E501287@math.washington.edu> <737D2BB3-CAC1-42E2-A89F-8FEFB9DF65E3@gmail.com> <493E173A.4070208@behnel.de> <3349A1C5-E842-4096-B88F-F950373FC4F0@gmail.com> <42689.213.61.181.86.1228986817.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <23D3F0D7-D678-47AB-9847-57582263394B@math.washington.edu> On Dec 11, 2008, at 1:13 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> %cython >> >> cdef foo_c(): >> pass >> >> def do_loop_c_call(int n, int m): >> cdef int i, j >> for i in range(n): >> for j in range(m): >> foo_c() >> [...] >> sage: time do_loop_c_call(5000, 500) >> CPU time: 0.00 s, Wall time: 0.00 s >> >> The moral of the story is that the loop overhead can be reduced to >> essentially a clock cycle or two per iteration (nanoseconds) > > Have you looked at the assembly that the above code compiles into? I > wouldn't be surprised if the C compiler inlined the call to foo_c > (), and > then removed the entire loop as useless code. Note: sage: time do_loop_c_call(5000, 50000) # 100x anything above CPU time: 0.22 s, Wall time: 0.22 s (It can't optimize away the function call because of the implicit incref/decref None. If it was cdef int foo_c() then it would probably have optimized the whole thing away. - Robert From greg.ewing at canterbury.ac.nz Thu Dec 11 11:02:54 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Dec 2008 23:02:54 +1300 Subject: [Cython] want to implement support for C99/C++ complex In-Reply-To: References: Message-ID: <4940E54E.4070307@canterbury.ac.nz> Lisandro Dalcin wrote: > I cannot get Cython properly recognize something like "cdef > float complex z" or "cdef double complex z". The place to start hacking would probably be in p_simple_base_type in Parsing. -- Greg From dalcinl at gmail.com Thu Dec 11 11:22:35 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 11 Dec 2008 07:22:35 -0300 Subject: [Cython] want to implement support for C99/C++ complex In-Reply-To: <3311827901.792492@smtp.netcom.no> References: <3311827901.792492@smtp.netcom.no> Message-ID: On Thu, Dec 11, 2008 at 4:11 AM, Dag Sverre Seljebotn wrote: > Great! > > I already wrote some descriptions along those lines at http://wiki.cython.org/smallprojects (#2 IIRC). The required steps you mention would be a really good guideline. > What I suggest there would work for all C compilers, but using C99 complex instead shouldn't be much different (but not much less work either...) I'm particularly interested on C99/C++ complex because I need them to support in petsc4py a PETSc build with complex scalars. If you build PETSc using a C compiler, the it used the C99 complex type, if you build with C++, it uses std::complex. In general, if you want to wrap or access from Cython any kind of "external" code that uses native C/C++ complex numbers, then we need this. Of course, in the future, via appropriate compiler directives, we could support non-native complex (I mean, what you are proposing), probably re-using the Py_complex struct and some Py_XXX macros related to it? > -----Original Message----- > From: "Lisandro Dalcin" > Date: Thursday, Dec 11, 2008 2:27 am > Subject: [Cython] want to implement support for C99/C++ complex > To: cython-dev Reply-To: cython-dev at codespeak.net > > Could any of you (Greg?) give me a executive summary about should I >>change/add to Cython/Pyrex scanner/parser/generator to make this >>possible? I believe the scanner does not need much (if any) work, but >>up to now I cannot get Cython properly recognize something like "cdef >>float complex z" or "cdef double complex z". >> >>-- >>Lisandro Dalc?n >>--------------- >>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >>PTLC - G?emes 3450, (3000) Santa Fe, Argentina >>Tel/Fax: +54-(0)342-451.1594 >>_______________________________________________ >>Cython-dev mailing list >>Cython-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/cython-dev >> > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Thu Dec 11 11:24:17 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 11 Dec 2008 07:24:17 -0300 Subject: [Cython] want to implement support for C99/C++ complex In-Reply-To: <4940E54E.4070307@canterbury.ac.nz> References: <4940E54E.4070307@canterbury.ac.nz> Message-ID: I believe that's the root of my failure! I missed to look at that place. Many thanks. On Thu, Dec 11, 2008 at 7:02 AM, Greg Ewing wrote: > Lisandro Dalcin wrote: >> I cannot get Cython properly recognize something like "cdef >> float complex z" or "cdef double complex z". > > The place to start hacking would probably be in > p_simple_base_type in Parsing. > > -- > Greg > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Thu Dec 11 16:19:49 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 11 Dec 2008 16:19:49 +0100 Subject: [Cython] convert-range is default? In-Reply-To: References: <85b5c3130812020343k8817647vc2b1b5750d44a2e@mail.gmail.com> Message-ID: <85b5c3130812110719k51bdadbfs76e8e25fe22c4ec8@mail.gmail.com> On Sun, Dec 7, 2008 at 10:23 AM, Robert Bradshaw wrote: > On Dec 2, 2008, at 3:43 AM, Ondrej Certik wrote: > >> Hi, >> >> I noticed that convert-range was enabled by default now? >> >> It seems to have been enabled a long time ago: >> >> http://hg.cython.org/cython/rev/5450c26066e9 > > Yes. > >> So I am a little bit confused, because I have the feeling that I tried >> that after it as well without the "--convert-range" and it didn't >> produce the nice C for loop. But when I tried today, it indeed works. > > The most common cause is not cdef'ing the local variable. > >> Is this the reason why it was removed from the help? > > Yes. > >> http://hg.cython.org/cython/rev/7b8970eb4837 >> >> Btw, Dag, how is convert-range connected with "Option to emit #line >> directives"? Or maybe you needed space for the new directive? > > It's not. I just deleted it when I was editing the help statement, > because it hadn't been deleted earlier. (Technically, I guess that > should have been a separate commit). > >> So in any case, "for i in range(10)" are now automatically converted >> by default (if i is declared as an integer), so one doesn't need to >> use the old syntax ("for i from 1 < 5" or something), right? > > Yes. Excellent, that's great news. Ondrej From stefan_ml at behnel.de Thu Dec 11 18:07:23 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 11 Dec 2008 18:07:23 +0100 Subject: [Cython] cdef class and docstrings In-Reply-To: <4940AFCC.1000501@behnel.de> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100127q5794423ep5a5ef08d6891c1fc@mail.gmail.com> <4940AFCC.1000501@behnel.de> Message-ID: <494148CB.4070101@behnel.de> Stefan Behnel wrote: > David Cournapeau wrote: >> I would prefer to have at >> least the signature of __init__ (for default arguments) > > Cython can embed the signature of a function or method in its docstring. > However, my guess is that this doesn't work for __init__ yet, where it > would have to put it into the respective class (at least, there's no test > for that). Works now. http://hg.cython.org/cython-devel/rev/31644ed14440 Stefan From Joris.DeRidder at ster.kuleuven.be Fri Dec 12 01:33:22 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Fri, 12 Dec 2008 01:33:22 +0100 Subject: [Cython] Cython vs ctypes + C++ Message-ID: Hi, Below you can find a small cython function that implements a simple max-filter. That is, given an image (a 2d numpy ndarray), each pixel is replaced by the maximum value in a neighborhood [-dx:dx, -dy:dy] around that pixel. For simplicity it just ignores the edges of the image. I implemented the same function using C++ & ctypes, and the latter function turns out to be about 100 times faster than the cython one, for a 500x500 image, and dx,dy = 5,5. This surprises me, because of the simplicity of the function. I checked the C file, and also the translation of the maxfilter() function in C is quite simple. The bulk of the time is of course spent in the loops, and the only bottleneck that I can see is accessing the numpy arrays. Is there anything I can do in Cython to speed that up? Cheers, Joris import numpy as np cimport cython cimport numpy as np DTYPE = np.int # our image type ctypedef np.int_t DTYPE_t # our compile time image type @cython.boundscheck(False) def maxfilter(np.ndarray[DTYPE_t, ndim=2] image, int dx, int dy): cdef int nRow = image.shape[0] cdef int nCol = image.shape[1] cdef np.ndarray[DTYPE_t, ndim=2] result = np.zeros((nRow,nCol), dtype=DTYPE) cdef DTYPE_t maxvalue cdef int i,j,k,n for i from dx <= i < nRow-dx: for j from dy <= j < nCol-dy: maxvalue = image[i,j] for k from -dx <= k <= dx: for n from -dy <= n <= dy: if maxvalue < image[i+k,j+n]: maxvalue = image[i +k,j+n] result[i,j] = maxvalue return result Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20081212/8e5f0085/attachment.htm From cournape at gmail.com Fri Dec 12 06:59:09 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 12 Dec 2008 14:59:09 +0900 Subject: [Cython] Cython vs ctypes + C++ In-Reply-To: References: Message-ID: <5b8d13220812112159u2e03f93ev57854792408037f7@mail.gmail.com> On Fri, Dec 12, 2008 at 9:33 AM, Joris De Ridder wrote: > Hi, > Below you can find a small cython function that implements a simple > max-filter. That is, given an image (a 2d numpy ndarray), each pixel is > replaced by the maximum value in a neighborhood [-dx:dx, -dy:dy] around that > pixel. For simplicity it just ignores the edges of the image. > I implemented the same function using C++ & ctypes, and the latter function > turns out to be about 100 times faster than the cython one, for a 500x500 > image, and dx,dy = 5,5. What is the implementation in ctypes ? David From robertwb at math.washington.edu Fri Dec 12 07:29:30 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 11 Dec 2008 22:29:30 -0800 Subject: [Cython] Cython vs ctypes + C++ In-Reply-To: References: Message-ID: <5E9F73C5-C95E-4C57-82CE-43584B494D81@math.washington.edu> On Dec 11, 2008, at 4:33 PM, Joris De Ridder wrote: > Hi, > > Below you can find a small cython function that implements a simple > max-filter. That is, given an image (a 2d numpy ndarray), each > pixel is replaced by the maximum value in a neighborhood [-dx:dx, - > dy:dy] around that pixel. For simplicity it just ignores the edges > of the image. > > I implemented the same function using C++ & ctypes, and the latter > function turns out to be about 100 times faster than the cython > one, for a 500x500 image, and dx,dy = 5,5. This surprises me, > because of the simplicity of the function. I checked the C file, > and also the translation of the maxfilter() function in C is quite > simple. The bulk of the time is of course spent in the loops, and > the only bottleneck that I can see is accessing the numpy arrays. > Is there anything I can do in Cython to speed that up? Try declaring i and j to be unsigned ints, and see if that speeds anything up. (Currently it checks to see if the index is negative and does the Python-style indexing backwards in that case). You could also try passing in mode="c" to your ndarray type declaration. Another thing to attempt is to avoid looking up image[i+k,j+n] more than once by doing value = image[i+k,j+n] if maxvalue < value: maxvalue = value 100x difference sounds like what one would expect between Python and C, not between Cython and C++. Is it *exactly* the same algorithm? (Perhaps, since it's such a simple algorithm you could post the C++ code as well?) It seems one could use a "sliding window" of maximum values to avoid re-computing the max of a 11x11 square on each iteration which could easily account for most of the speed difference you see here. > Cheers, > Joris > > > > import numpy as np > cimport cython > cimport numpy as np > > DTYPE = np.int # our image type > ctypedef np.int_t DTYPE_t # our compile time image type > > @cython.boundscheck(False) > def maxfilter(np.ndarray[DTYPE_t, ndim=2] image, int dx, int dy): > > cdef int nRow = image.shape[0] > cdef int nCol = image.shape[1] > cdef np.ndarray[DTYPE_t, ndim=2] result = np.zeros((nRow,nCol), > dtype=DTYPE) > cdef DTYPE_t maxvalue > > cdef int i,j,k,n > for i from dx <= i < nRow-dx: > for j from dy <= j < nCol-dy: > maxvalue = image[i,j] > for k from -dx <= k <= dx: > for n from -dy <= n <= dy: > if maxvalue < image[i+k,j+n]: maxvalue = image[i > +k,j+n] > result[i,j] = maxvalue > return result > > > Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm for > more information. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Fri Dec 12 09:55:55 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 12 Dec 2008 09:55:55 +0100 Subject: [Cython] Cython vs ctypes + C++ In-Reply-To: <5E9F73C5-C95E-4C57-82CE-43584B494D81@math.washington.edu> References: <5E9F73C5-C95E-4C57-82CE-43584B494D81@math.washington.edu> Message-ID: <4942271B.5050703@student.matnat.uio.no> Robert Bradshaw wrote: > On Dec 11, 2008, at 4:33 PM, Joris De Ridder wrote: > > >> Hi, >> >> Below you can find a small cython function that implements a simple >> max-filter. That is, given an image (a 2d numpy ndarray), each >> pixel is replaced by the maximum value in a neighborhood [-dx:dx, - >> dy:dy] around that pixel. For simplicity it just ignores the edges >> of the image. >> >> I implemented the same function using C++ & ctypes, and the latter >> function turns out to be about 100 times faster than the cython >> one, for a 500x500 image, and dx,dy = 5,5. This surprises me, >> because of the simplicity of the function. I checked the C file, >> and also the translation of the maxfilter() function in C is quite >> simple. The bulk of the time is of course spent in the loops, and >> the only bottleneck that I can see is accessing the numpy arrays. >> Is there anything I can do in Cython to speed that up? >> > > Try declaring i and j to be unsigned ints, and see if that speeds > anything up. (Currently it checks to see if the index is negative and > does the Python-style indexing backwards in that case). You could > Note that you can also use the negative_indices parameter, i.e. ndarray[DTYPE_t, ndim=2, negative_indices=False]. This is guaranteed not to emit those checks. Also I'd like to add these things to check: - Make absolutely sure that your Cython-generated C code is compiled using -O2 - How does your profiling work? 500x500 isn't all that much, and the calls to "np.zeros" takes some time that C++ might not take (Gabriel Gellner has work in progress to make this faster in Cython). Can you try with 5000x5000 and see if there's still the same speed difference? If this theory is right (i.e. you profile the entire function and not just the loop) then the difference in speed should converge to a ratio between 1 and 3 as N goes to infinity. - Is the array statically allocated in C++? I.e. does it say "int data[500][500]", or is it dynamically allocated? In the former case C++ could be theoretically a little bit faster (but if it makes a real impact I'll fix Cython to make it as fast :-)) 100x difference is definitely too much, in fact I'd be worried with a 3x difference (if only the loop is profiled). Dag Sverre From cournape at gmail.com Fri Dec 12 13:58:32 2008 From: cournape at gmail.com (David Cournapeau) Date: Fri, 12 Dec 2008 21:58:32 +0900 Subject: [Cython] cdef class and docstrings In-Reply-To: <494148CB.4070101@behnel.de> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100127q5794423ep5a5ef08d6891c1fc@mail.gmail.com> <4940AFCC.1000501@behnel.de> <494148CB.4070101@behnel.de> Message-ID: <5b8d13220812120458l1320116di7709bf015a19e090@mail.gmail.com> On Fri, Dec 12, 2008 at 2:07 AM, Stefan Behnel wrote: > > Stefan Behnel wrote: >> David Cournapeau wrote: >>> I would prefer to have at >>> least the signature of __init__ (for default arguments) >> >> Cython can embed the signature of a function or method in its docstring. >> However, my guess is that this doesn't work for __init__ yet, where it >> would have to put it into the respective class (at least, there's no test >> for that). > > Works now. > > http://hg.cython.org/cython-devel/rev/31644ed14440 What kind of docstring should I expect for something like: cdef class Yo: "Some doc." def __init__(self, yo=None): "init doc" pass I tried using cython-devel with your including changes, and I can't see any difference, but maybe I am not looking at the right place (the signature is supposed to be in Yo.__doc__, right ?) David From stefan_ml at behnel.de Fri Dec 12 14:54:25 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 12 Dec 2008 14:54:25 +0100 Subject: [Cython] cdef class and docstrings In-Reply-To: <5b8d13220812120458l1320116di7709bf015a19e090@mail.gmail.com> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100127q5794423ep5a5ef08d6891c1fc@mail.gmail.com> <4940AFCC.1000501@behnel.de> <494148CB.4070101@behnel.de> <5b8d13220812120458l1320116di7709bf015a19e090@mail.gmail.com> Message-ID: <49426D11.5030005@behnel.de> Hi, David Cournapeau wrote: > I tried using cython-devel with your including changes, and I can't > see any difference, but maybe I am not looking at the right place (the > signature is supposed to be in Yo.__doc__, right ?) Yes, but the feature is not enabled by default. Put #cython: embedsignature=True at the top of your file. Does anyone know if that's already documented somewhere? I.e., is there a place where all compiler directives are listed? Stefan From dagss at student.matnat.uio.no Fri Dec 12 15:08:50 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 12 Dec 2008 15:08:50 +0100 Subject: [Cython] cdef class and docstrings In-Reply-To: <49426D11.5030005@behnel.de> References: <5b8d13220812100010w1c9f103fu72a4f1a6cbaf3689@mail.gmail.com> <41055.213.61.181.86.1228898131.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100057s3226924cs3ef26cbde06335ca@mail.gmail.com> <50345.213.61.181.86.1228900256.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5b8d13220812100127q5794423ep5a5ef08d6891c1fc@mail.gmail.com> <4940AFCC.1000501@behnel.de> <494148CB.4070101@behnel.de> <5b8d13220812120458l1320116di7709bf015a19e090@mail.gmail.com> <49426D11.5030005@behnel.de> Message-ID: <49427072.4060807@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > David Cournapeau wrote: > >> I tried using cython-devel with your including changes, and I can't >> see any difference, but maybe I am not looking at the right place (the >> signature is supposed to be in Yo.__doc__, right ?) >> > > Yes, but the feature is not enabled by default. Put > > #cython: embedsignature=True > > at the top of your file. > > Does anyone know if that's already documented somewhere? I.e., is there a > place where all compiler directives are listed? > I listed the ones I added at http://wiki.cython.org/enhancements/compilerdirectives, it should probably be updated. Dag Sverre From Joris.DeRidder at ster.kuleuven.be Fri Dec 12 23:36:40 2008 From: Joris.DeRidder at ster.kuleuven.be (Joris De Ridder) Date: Fri, 12 Dec 2008 23:36:40 +0100 Subject: [Cython] Cython vs ctypes + C++ In-Reply-To: References: Message-ID: On 12 Dec 2008, at 1:33 , Joris De Ridder wrote: > I implemented the same function using C++ & ctypes, and the latter > function turns out to be about 100 times faster than the cython one, > for a 500x500 image, and dx,dy = 5,5. This surprises me, because of > the simplicity of the function. I checked the C file, and also the > translation of the maxfilter() function in C is quite simple. The > bulk of the time is of course spent in the loops, and the only > bottleneck that I can see is accessing the numpy arrays. Is there > anything I can do in Cython to speed that up? My statement turned out to be wrong: I misread the unit of the timing. :-/ Cython took 168 ms, ctypes+Cpp took 1.41 s. Prejudiced about the speed of C++, I misread the latter as 1.41 ms. So, it turns out that Cython was actually 10 times faster! I'm still a bit puzzled about this difference, but it's on the ctypes side I have to look, not on the cython side. Thanks for the responses, and sorry for the noise. Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From cournape at gmail.com Sat Dec 13 01:26:51 2008 From: cournape at gmail.com (David Cournapeau) Date: Sat, 13 Dec 2008 09:26:51 +0900 Subject: [Cython] Cython vs ctypes + C++ In-Reply-To: References: Message-ID: <5b8d13220812121626u538a03eeh3f6ab65356692112@mail.gmail.com> On Sat, Dec 13, 2008 at 7:36 AM, Joris De Ridder wrote: > > On 12 Dec 2008, at 1:33 , Joris De Ridder wrote: > >> I implemented the same function using C++ & ctypes, and the latter >> function turns out to be about 100 times faster than the cython one, >> for a 500x500 image, and dx,dy = 5,5. This surprises me, because of >> the simplicity of the function. I checked the C file, and also the >> translation of the maxfilter() function in C is quite simple. The >> bulk of the time is of course spent in the loops, and the only >> bottleneck that I can see is accessing the numpy arrays. Is there >> anything I can do in Cython to speed that up? > > My statement turned out to be wrong: I misread the unit of the > timing. :-/ Cython took 168 ms, ctypes+Cpp took 1.41 s. Prejudiced > about the speed of C++, I misread the latter as 1.41 ms. :) > So, it turns > out that Cython was actually 10 times faster! I'm still a bit puzzled > about this difference, but it's on the ctypes side I have to look, not > on the cython side. Note that ctypes has a non negligeable cost (because you can easily go through several function calls for each call to the underlying dll, which is costly in python), depending on how you use it. Your cython code can also be sped up by using direct index computing on the data buffer instead of using numpy indexing, but only by 30 % or though, at least on my machine, David From robertwb at math.washington.edu Sat Dec 13 20:00:03 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 13 Dec 2008 11:00:03 -0800 Subject: [Cython] [sage-devel] sorting+lambdas in Cython In-Reply-To: <8cf963450812130812q3c6d5330s1c7b26f5be0536ad@mail.gmail.com> References: <8cf963450812130812q3c6d5330s1c7b26f5be0536ad@mail.gmail.com> Message-ID: <6C703985-32DB-4389-B7A1-B66BA2AE6AA3@math.washington.edu> On Dec 13, 2008, at 8:12 AM, David Joyner wrote: > Hi: > > Perhaps this should go to the Cython devel list > but I looked at the archives and FAQ and guide on > cython.org and did not see this issue discussed. > (BTW, the links on http://docs.cython.org/ > under "Indices and tables" are broken, and also > the docs page has no link back to the main cython page.) Thanks. This is relevant here and there. > Basic idea is that I would like to rewrite the coding theory > modules in Cython. One of the basic tasks is to sort > a collection of vectors by the "Hamming weight" (the number of > non-zero positions). This is very easy to do using > Python since you can feed sort a comparison function, > like this: > > def my_sort1(): > L = [2,1,3,0] > #L.sort(cmp=lambda x,y: x-y, key=None, reverse=False) > L.sort() > return L > > or like this: > > def my_order(x,y): > return x-y > > def my_sort2(): > L = [2,1,3,0] > #L.sort(cmp=my_order, key=None, reverse=False) > L.sort() > return L > > However, I cannot figure out how to do something like this > in Cython. The second works just fine for me (with our without the custom cmp command). > I resorted to trying to build my own sort function > in Cython, but it turned out to be slower than simply using the > Python sort. Python's sort it rather good, it'd be hard to beat it generically. If I were trying to speed things up, I would make sure your hamming weight computation is extremely fast, and (safely) cached. Equivalently, you could sort the list [(v.hamming_weight(), v) for v in L] using a cmp function that only looks at the first element of the tuple. If this wasn't fast enough, you could get the hamming weights as a list of c ints, sort that (keeping track of the permutation) and then return the permuted list. Sorting a list of ints can be written faster than Python's sort. > Also, I'd like to know if lambdas are implemented in Cython. No, not yet. I would love to do this, and think I know how, but don't know when I'll have the time. - Robert From stefan_ml at behnel.de Sat Dec 13 22:57:53 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 13 Dec 2008 22:57:53 +0100 Subject: [Cython] constant folding Message-ID: <49442FE1.3080006@behnel.de> Hi, I started working on ticket #119, which deals with constant folding and figuring out at code generation time if a subexpression is constant. An initial implementation is here: http://hg.cython.org/cython-devel/rev/8a2e7b51e770 It happily borrows from the compile_time_value() evaluation which is done at parse time, but moves it into a separate ConstantFolding transform (which runs after type analysis). This makes it a lot simpler to calculate the result in a node method, as all the error handling and preconditions are now handled in the transform code. The constant result is stored in node.constant_result, which is set to ExprNodes.not_a_constant if the result cannot be determined. Apart from slowing down the compilation a bit, the transform does not yet have *any* impact on code generation :). It does not replace any nodes with a ConstNode, neither does the code generation phase use the results so far. Replacing a node requires knowing which node can represent the type of the expression, which needs to be defined first, but which would have the best impact on code generation and transforms later in the pipeline. Being able to read the constant result from any node might however be just as useful during code generation. I also skipped any name lookups so far, so constant calculation is currently mostly restricted to arithmetic expressions and builtin type literals (lists, tuples, dicts, sets). It's also tested very little, there's a tiny run/consts.pyx test, which will only get interesting once we generate optimised code for calculated constants. At least, building up the constant values at compile time doesn't break lxml's build so far. :) Any use cases and further hacking appreciated. :) Stefan From magnus at hetland.org Mon Dec 15 17:09:24 2008 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon, 15 Dec 2008 17:09:24 +0100 Subject: [Cython] Automatic buffer dtype conversion? Message-ID: Hi! I've been trying to get the simple Cython+NumPy example from Robert Bradshaw's slides to work: > # footest.pyx > cimport numpy > > def sum(x): > cdef numpy.ndarray[int, ndim=1] arr = x > cdef int i, s = 0 > for i in range(arr.shape[0]): > s += arr[i] > return s I get it to compile/link, but then I try to actually use it, with the following code: > import footest > import numpy as np > > a = np.arange(1000, dtype=np.int) > print footest.sum(a) The np.int shouldn't be necessary -- and I started without it -- but even *with* it, I get the following error: > File "footest.pyx", line 4, in footest.sum (footest.c:392) > cdef numpy.ndarray[int, ndim=1] arr = x > ValueError: Buffer dtype mismatch (expected int, got long) In other words, there seems to have been a conversion from int to long. If I try to print out repr(x), it seems to have a dtype of int32. If I change the offending line from the pyx file to > cdef numpy.ndarray[long, ndim=1] arr = x it suddenly works. I'm currently using Cython 0.10.2 with Python 2.5.2 and NumPy 1.2.1 in Mac OS X (Leopard). Am I missing something obvious? Any ideas? -- Magnus Lie Hetland http://hetland.org From dagss at student.matnat.uio.no Mon Dec 15 17:57:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 15 Dec 2008 17:57:39 +0100 Subject: [Cython] Automatic buffer dtype conversion? In-Reply-To: References: Message-ID: <49468C83.6070402@student.matnat.uio.no> Hei der! Magnus Lie Hetland wrote: > Hi! > > I've been trying to get the simple Cython+NumPy example from Robert > Bradshaw's slides to work: > >> # footest.pyx > >> cimport numpy >> >> def sum(x): >> cdef numpy.ndarray[int, ndim=1] arr = x >> cdef int i, s = 0 >> for i in range(arr.shape[0]): >> s += arr[i] >> return s > > I get it to compile/link, but then I try to actually use it, with the > following code: > >> import footest >> import numpy as np >> >> a = np.arange(1000, dtype=np.int) >> print footest.sum(a) > > The np.int shouldn't be necessary -- and I started without it -- but > even *with* it, I get the following error: > >> File "footest.pyx", line 4, in footest.sum (footest.c:392) >> cdef numpy.ndarray[int, ndim=1] arr = x >> ValueError: Buffer dtype mismatch (expected int, got long) Yes, this is very expected on 64-bit systems, I suppose Robert's slides are in error. Basically the "np.int" type object is defined (by NumPy) to be long (probably on merit of being the most convenient int type; read "np.int" as saying "Integer", not a C int). To be safe with these things, you should use the compile-time types defined in the numpy cimport: cdef numpy.ndarray[numpy.int_t, ndim=1] arr = x numpy.int_t is typedef-ed to always be whatever np.int refers to. If you really want a C int, use e.g. numpy.int32(_t). -- Dag Sverre From dagss at student.matnat.uio.no Mon Dec 15 18:00:11 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 15 Dec 2008 18:00:11 +0100 Subject: [Cython] Automatic buffer dtype conversion? In-Reply-To: References: Message-ID: <49468D1B.1030400@student.matnat.uio.no> Magnus Lie Hetland wrote: > Hi! > > I've been trying to get the simple Cython+NumPy example from Robert > Bradshaw's slides to work: > >> # footest.pyx > >> cimport numpy >> >> def sum(x): >> cdef numpy.ndarray[int, ndim=1] arr = x >> cdef int i, s = 0 >> for i in range(arr.shape[0]): >> s += arr[i] >> return s > > I get it to compile/link, but then I try to actually use it, with the > following code: > >> import footest >> import numpy as np >> >> a = np.arange(1000, dtype=np.int) >> print footest.sum(a) > > The np.int shouldn't be necessary -- and I started without it -- but > even *with* it, I get the following error: BTW I believe NumPy has switched the default dtype to float in recent versions, so better leave it in. (But I'm not 100% sure about this) -- Dag Sverre From magnus at hetland.org Mon Dec 15 19:57:33 2008 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon, 15 Dec 2008 19:57:33 +0100 Subject: [Cython] Automatic buffer dtype conversion? In-Reply-To: <49468C83.6070402@student.matnat.uio.no> References: <49468C83.6070402@student.matnat.uio.no> Message-ID: <39447A57-9A93-4B52-A6B1-C8BA3A78A71E@hetland.org> On Dec 15, 2008, at 17:57 , Dag Sverre Seljebotn wrote: > Hei der! Hei, ja :) > Yes, this is very expected on 64-bit systems, I suppose Robert's > slides > are in error. > > Basically the "np.int" type object is defined (by NumPy) to be long > (probably on merit of being the most convenient int type; read > "np.int" > as saying "Integer", not a C int). I see... But, as I mentioned, x.dtype is Int32 (or seems to be, when I print out repr(x))... I also believe I tried explicitly setting the type to int32 (although I am now on a different machine, without the proper software installed, so I can't check right now :) > To be safe with these things, you should use the compile-time types > defined in the numpy cimport: > > cdef numpy.ndarray[numpy.int_t, ndim=1] arr = x > > numpy.int_t is typedef-ed to always be whatever np.int refers to. > > If you really want a C int, use e.g. numpy.int32(_t). Right. That might be an issue at times, I guess. I'll have a look to see what happens if I use int32. As I said, I thought I had tried that (and failed), but I wouldn't rule out a bit of PEBKAC there ;-) Thanks, - M -- Magnus Lie Hetland http://hetland.org From lev at columbia.edu Tue Dec 16 00:38:18 2008 From: lev at columbia.edu (Lev Givon) Date: Mon, 15 Dec 2008 18:38:18 -0500 Subject: [Cython] cross-compiling Python extensions Message-ID: <20081215233818.GK19703@localhost.ee.columbia.edu> Not strictly a Cython question, but is there a way to cross-compile a Python extension written in C for a specific platform using distutils or setuptools? L.G. From dalcinl at gmail.com Tue Dec 16 01:20:57 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 15 Dec 2008 21:20:57 -0300 Subject: [Cython] cross-compiling Python extensions In-Reply-To: <20081215233818.GK19703@localhost.ee.columbia.edu> References: <20081215233818.GK19703@localhost.ee.columbia.edu> Message-ID: AFAIK, distutils does not support cross-compilation, just in case google yourself :-) On Mon, Dec 15, 2008 at 8:38 PM, Lev Givon wrote: > Not strictly a Cython question, but is there a way to cross-compile a > Python extension written in C for a specific platform using distutils > or setuptools? > > L.G. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From lev at columbia.edu Tue Dec 16 01:32:57 2008 From: lev at columbia.edu (Lev Givon) Date: Mon, 15 Dec 2008 19:32:57 -0500 Subject: [Cython] cross-compiling Python extensions In-Reply-To: References: <20081215233818.GK19703@localhost.ee.columbia.edu> Message-ID: <20081216003257.GM19703@localhost.ee.columbia.edu> Received from Lisandro Dalcin on Mon, Dec 15, 2008 at 07:20:57PM EST: > On Mon, Dec 15, 2008 at 8:38 PM, Lev Givon wrote: > > Not strictly a Cython question, but is there a way to > > cross-compile a Python extension written in C for a specific > > platform using distutils or setuptools? > > > > AFAIK, distutils does not support cross-compilation, just in case > google yourself :-) I saw that, but I had hoped that folks who deal with Python extensions more than myself might be privy to some arcane Python wisdom :-) Thanks, L.G. From hoytak at cs.ubc.ca Tue Dec 16 01:55:34 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 15 Dec 2008 16:55:34 -0800 Subject: [Cython] cython errors installing cython Message-ID: <4db580fd0812151655m6f262148g36a1f4da0b2a0f0b@mail.gmail.com> Hello, On my machine at least, it seems that changest 1478 in cython-devel (by Stephen) introduced some errors while compiling module Cython.Compiler.Parsing. Here's the output from my attempt to install cython. hoytak at hk-laptop:~/sysroot/src/cython> rm -Rf build ; python setup.py install --prefix=$SYSROOT Compiling module Cython.Plex.Scanners ... Compiling module Cython.Compiler.Scanning ... Compiling module Cython.Compiler.Parsing ... Error converting Pyrex file to C: ------------------------------------------------------------ ... cpdef p_string_literal(PyrexScanner s) cpdef p_list_maker(PyrexScanner s) cpdef p_list_iter(PyrexScanner s) cpdef p_list_for(PyrexScanner s) cpdef p_list_if(PyrexScanner s) cpdef p_dict_maker(PyrexScanner s) ^ ------------------------------------------------------------ /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.pxd:47:19: Non-extern C function 'p_dict_maker' declared but not defined Error converting Pyrex file to C: ------------------------------------------------------------ ... cpdef p_list_maker(PyrexScanner s) cpdef p_list_iter(PyrexScanner s) cpdef p_list_for(PyrexScanner s) cpdef p_list_if(PyrexScanner s) cpdef p_dict_maker(PyrexScanner s) cpdef p_dict_item(PyrexScanner s) ^ ------------------------------------------------------------ /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.pxd:48:18: Non-extern C function 'p_dict_item' declared but not defined Error converting Pyrex file to C: Compilation failed Compiling module Cython.Compiler.Visitor ... running install running build running build_py ... And the rest appears to be go as usual... I'm not sure what's going on, and I suspect that I'm running python version 2.5.2 on opensuse 11.0. If you need any more information, I'd be happy to give it. Thanks! --Hoyt P.S. Stephen -- It looks like you've put a lot of great work into the last few changesets; thanks a ton!!! They look awesome! ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dalcinl at gmail.com Tue Dec 16 06:49:09 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 16 Dec 2008 02:49:09 -0300 Subject: [Cython] cython errors installing cython In-Reply-To: <4db580fd0812151655m6f262148g36a1f4da0b2a0f0b@mail.gmail.com> References: <4db580fd0812151655m6f262148g36a1f4da0b2a0f0b@mail.gmail.com> Message-ID: Try again... just pushed a fix... On Mon, Dec 15, 2008 at 9:55 PM, Hoyt Koepke wrote: > Hello, > > On my machine at least, it seems that changest 1478 in cython-devel > (by Stephen) introduced some errors while compiling module > Cython.Compiler.Parsing. Here's the output from my attempt to install > cython. > > hoytak at hk-laptop:~/sysroot/src/cython> rm -Rf build ; python setup.py > install --prefix=$SYSROOT > Compiling module Cython.Plex.Scanners ... > Compiling module Cython.Compiler.Scanning ... > Compiling module Cython.Compiler.Parsing ... > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cpdef p_string_literal(PyrexScanner s) > cpdef p_list_maker(PyrexScanner s) > cpdef p_list_iter(PyrexScanner s) > cpdef p_list_for(PyrexScanner s) > cpdef p_list_if(PyrexScanner s) > cpdef p_dict_maker(PyrexScanner s) > ^ > ------------------------------------------------------------ > > /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.pxd:47:19: > Non-extern C function 'p_dict_maker' declared but not defined > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cpdef p_list_maker(PyrexScanner s) > cpdef p_list_iter(PyrexScanner s) > cpdef p_list_for(PyrexScanner s) > cpdef p_list_if(PyrexScanner s) > cpdef p_dict_maker(PyrexScanner s) > cpdef p_dict_item(PyrexScanner s) > ^ > ------------------------------------------------------------ > > /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.pxd:48:18: > Non-extern C function 'p_dict_item' declared but not defined > > Error converting Pyrex file to C: > > > Compilation failed > Compiling module Cython.Compiler.Visitor ... > running install > running build > running build_py > ... > > And the rest appears to be go as usual... > > I'm not sure what's going on, and I suspect that I'm running python > version 2.5.2 on opensuse 11.0. If you need any more information, I'd > be happy to give it. > > Thanks! > --Hoyt > > P.S. Stephen -- It looks like you've put a lot of great work into the > last few changesets; thanks a ton!!! They look awesome! > > > ++++++++++++++++++++++++++++++++++++++++++++++++ > + Hoyt Koepke > + University of Washington Department of Statistics > + http://www.stat.washington.edu/~hoytak/ > + hoytak at gmail.com > ++++++++++++++++++++++++++++++++++++++++++ > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Dec 16 06:53:09 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 16 Dec 2008 02:53:09 -0300 Subject: [Cython] cross-compiling Python extensions In-Reply-To: <20081216003257.GM19703@localhost.ee.columbia.edu> References: <20081215233818.GK19703@localhost.ee.columbia.edu> <20081216003257.GM19703@localhost.ee.columbia.edu> Message-ID: On Mon, Dec 15, 2008 at 9:32 PM, Lev Givon wrote: > > I saw that, but I had hoped that folks who deal with Python extensions > more than myself might be privy to some arcane Python wisdom :-) > Well, I deall with them all the time, but not cross-compiling... ping the numpy mailing list, people is really helpful there and perhaps someone could give you some hints... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Dec 16 10:17:17 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 16 Dec 2008 10:17:17 +0100 Subject: [Cython] Automatic buffer dtype conversion? In-Reply-To: <39447A57-9A93-4B52-A6B1-C8BA3A78A71E@hetland.org> References: <49468C83.6070402@student.matnat.uio.no> <39447A57-9A93-4B52-A6B1-C8BA3A78A71E@hetland.org> Message-ID: <4947721D.7010700@student.matnat.uio.no> Magnus Lie Hetland wrote: > On Dec 15, 2008, at 17:57 , Dag Sverre Seljebotn wrote: > >> Yes, this is very expected on 64-bit systems, I suppose Robert's >> slides >> are in error. >> >> Basically the "np.int" type object is defined (by NumPy) to be long >> (probably on merit of being the most convenient int type; read >> "np.int" >> as saying "Integer", not a C int). >> > > I see... But, as I mentioned, x.dtype is Int32 (or seems to be, when I > print out repr(x))... I also believe I tried explicitly setting the > type to int32 (although I am now on a different machine, without the > proper software installed, so I can't check right now :) > Actually, this was an error with Cython (which actually would occur on 32-bit systems only). You can retrieve a fixed version from http://hg.cython.org/cython (and in the next bugfix release which shouldn't be too far off). Or, use "np.int" as the type instead which will work with all recent releases. (Note that my point still stands, your code would have failed to work on a 64-bit system, but that was not the issue you were running into.) Thanks for bringing this to our attention! Dag Sverre From cournape at gmail.com Tue Dec 16 16:19:02 2008 From: cournape at gmail.com (David Cournapeau) Date: Wed, 17 Dec 2008 00:19:02 +0900 Subject: [Cython] cross-compiling Python extensions In-Reply-To: <20081215233818.GK19703@localhost.ee.columbia.edu> References: <20081215233818.GK19703@localhost.ee.columbia.edu> Message-ID: <5b8d13220812160719j69b1f931r2c760a65cb46827b@mail.gmail.com> On Tue, Dec 16, 2008 at 8:38 AM, Lev Givon wrote: > Not strictly a Cython question, but is there a way to cross-compile a > Python extension written in C for a specific platform using distutils > or setuptools? No, except for special cases (windows 32 <-> 64 bits). You will be better of using a real build system, like make, scons or waf (for scons, I have a tool which implements a python builder, and works on quite a few platforms). You will still need to handle the cross compilation part, but it will be doable. Distutils is such a mess that I would not wish my worst enemy to have to do implement cross compilation on the top of it, David From hoytak at cs.ubc.ca Tue Dec 16 17:02:45 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Tue, 16 Dec 2008 08:02:45 -0800 Subject: [Cython] cython errors installing cython In-Reply-To: References: <4db580fd0812151655m6f262148g36a1f4da0b2a0f0b@mail.gmail.com> Message-ID: <4db580fd0812160802s1295af8dx925e212bcca8d4a3@mail.gmail.com> Yup, works fine now! Thanks!! --Hoyt On Mon, Dec 15, 2008 at 9:49 PM, Lisandro Dalcin wrote: > Try again... just pushed a fix... > > On Mon, Dec 15, 2008 at 9:55 PM, Hoyt Koepke wrote: >> Hello, >> >> On my machine at least, it seems that changest 1478 in cython-devel >> (by Stephen) introduced some errors while compiling module >> Cython.Compiler.Parsing. Here's the output from my attempt to install >> cython. >> >> hoytak at hk-laptop:~/sysroot/src/cython> rm -Rf build ; python setup.py >> install --prefix=$SYSROOT >> Compiling module Cython.Plex.Scanners ... >> Compiling module Cython.Compiler.Scanning ... >> Compiling module Cython.Compiler.Parsing ... >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> cpdef p_string_literal(PyrexScanner s) >> cpdef p_list_maker(PyrexScanner s) >> cpdef p_list_iter(PyrexScanner s) >> cpdef p_list_for(PyrexScanner s) >> cpdef p_list_if(PyrexScanner s) >> cpdef p_dict_maker(PyrexScanner s) >> ^ >> ------------------------------------------------------------ >> >> /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.pxd:47:19: >> Non-extern C function 'p_dict_maker' declared but not defined >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> cpdef p_list_maker(PyrexScanner s) >> cpdef p_list_iter(PyrexScanner s) >> cpdef p_list_for(PyrexScanner s) >> cpdef p_list_if(PyrexScanner s) >> cpdef p_dict_maker(PyrexScanner s) >> cpdef p_dict_item(PyrexScanner s) >> ^ >> ------------------------------------------------------------ >> >> /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.pxd:48:18: >> Non-extern C function 'p_dict_item' declared but not defined >> >> Error converting Pyrex file to C: >> >> >> Compilation failed >> Compiling module Cython.Compiler.Visitor ... >> running install >> running build >> running build_py >> ... >> >> And the rest appears to be go as usual... >> >> I'm not sure what's going on, and I suspect that I'm running python >> version 2.5.2 on opensuse 11.0. If you need any more information, I'd >> be happy to give it. >> >> Thanks! >> --Hoyt >> >> P.S. Stephen -- It looks like you've put a lot of great work into the >> last few changesets; thanks a ton!!! They look awesome! >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++ >> + Hoyt Koepke >> + University of Washington Department of Statistics >> + http://www.stat.washington.edu/~hoytak/ >> + hoytak at gmail.com >> ++++++++++++++++++++++++++++++++++++++++++ >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From robertwb at math.washington.edu Wed Dec 17 10:59:28 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 17 Dec 2008 01:59:28 -0800 Subject: [Cython] Cython 0.10.3 Message-ID: <78677217-94A5-43E2-A5F4-5CB863508AEC@math.washington.edu> Another bug fix release is out, fixing an exception-handling memory leak (Robert Bradshaw), transformation code (Stefan Behnel, Robert Bradshaw), and buffer type checking (Dag Sverre Seljebotn). It does not include most of the work that has been going on in the cython- devel branch, which will be released in Cython 0.11. Full details at http://trac.cython.org/cython_trac/query? status=closed&group=component&milestone=0.10.3&order=priority - Robert From michael.abshoff at googlemail.com Wed Dec 17 11:34:40 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Wed, 17 Dec 2008 02:34:40 -0800 Subject: [Cython] Cython 0.10.3 In-Reply-To: <78677217-94A5-43E2-A5F4-5CB863508AEC@math.washington.edu> References: <78677217-94A5-43E2-A5F4-5CB863508AEC@math.washington.edu> Message-ID: <4948D5C0.50000@gmail.com> Robert Bradshaw wrote: Hi, > Another bug fix release is out, fixing an exception-handling memory > leak (Robert Bradshaw), FYI: that leak can hit you extremely hard depending on the code base, i.e. in Sage running the following two lines for E in cremona_curves(range(11,10000)): print E.cremona_label(); ld=E.local_data() goes from about 100 minutes down to 10 minutes CPU time and a leak of *8 GB* is gone due to the above fix. The speed improvement is mostly due to the avoidance of garbage collection, but the cause of the above leak had eluded us for a while. So, thanks Robert for fixing this :) transformation code (Stefan Behnel, Robert > Bradshaw), and buffer type checking (Dag Sverre Seljebotn). It does > not include most of the work that has been going on in the cython- > devel branch, which will be released in Cython 0.11. > > Full details at > > http://trac.cython.org/cython_trac/query? > status=closed&group=component&milestone=0.10.3&order=priority > > - Robert Cheers, Michael > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From magnus at hetland.org Wed Dec 17 15:45:46 2008 From: magnus at hetland.org (Magnus Lie Hetland) Date: Wed, 17 Dec 2008 15:45:46 +0100 Subject: [Cython] Setuptools and Cython? Message-ID: <244B1AA7-E71A-4C9F-9706-11E5FFEEA174@hetland.org> Hi! I'm thinking of using nose for some tests, and it seems that it plays nicely with setuptools -- and setuptools has built-in support for automatically running Pyrex on .pyx files ... which isn't exactly what I want, of course :-> Anyone else using setuptools with Cython here? Will the Cython stuff for Distutils work with setuptools as well? (I could always hack things about a bit; it would just be nice if there were a "smooth" solution...) -- Magnus Lie Hetland http://hetland.org From ggellner at uoguelph.ca Wed Dec 17 16:34:40 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Wed, 17 Dec 2008 10:34:40 -0500 Subject: [Cython] Setuptools and Cython? In-Reply-To: <244B1AA7-E71A-4C9F-9706-11E5FFEEA174@hetland.org> References: <244B1AA7-E71A-4C9F-9706-11E5FFEEA174@hetland.org> Message-ID: <20081217153440.GA13758@workerbee> On Wed, Dec 17, 2008 at 03:45:46PM +0100, Magnus Lie Hetland wrote: > Hi! > > I'm thinking of using nose for some tests, and it seems that it plays > nicely with setuptools -- and setuptools has built-in support for > automatically running Pyrex on .pyx files ... which isn't exactly what > I want, of course :-> > > Anyone else using setuptools with Cython here? Will the Cython stuff > for Distutils work with setuptools as well? (I could always hack > things about a bit; it would just be nice if there were a "smooth" > solution...) > I use setuptools exclusively and have no problems. Though my needs are simple, so I am not sure if everything you want would work. But for simple stuff (develop mode, automatic Cythoning etc) it works great. Gabriel From stephane.drouard at st.com Wed Dec 17 18:55:15 2008 From: stephane.drouard at st.com (Stephane DROUARD) Date: Wed, 17 Dec 2008 18:55:15 +0100 Subject: [Cython] Scope rules vs. Python behaviour Message-ID: <013101c96070$9dd4c360$b9ad810a@gnb.st.com> Hi, Robert Bradshaw wrote: >>> >>> foo.py: >>> print __name__ >>> >>> But under Python it returns "foo", "__builtin__" when cython'ized. >>> >>> As the documentation specifies, this can be fixed by: >>> >>> foo.py: >>> global __name__ >>> print __name__ >>> >>> Then both return "foo". >> >> Ah, good catch. Cython (and, incidentally I) didn't know that >> __name__ is an already declared module-level variable, hence it >> looked it up as a builtin (and found it there, so no error). This is >> easily fixed by always having __name__ in the module namespace (what >> other magic variables are there?) > > $ touch foo.py > $ python2.5 -c 'import foo; print foo.__dict__.keys()' > ['__builtins__', '__name__', '__file__', '__doc__'] > > To clarify, other than these (thanks for the list Lisandro) the only > time this can cause issues is if one manually adds names to the > module externally, or if one uses "import *" to overwrite a builtin. Do you plan to fix it (0.10.3 does not)? Cheers, Stephane From robertwb at math.washington.edu Wed Dec 17 19:33:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 17 Dec 2008 10:33:22 -0800 Subject: [Cython] Scope rules vs. Python behaviour In-Reply-To: <013101c96070$9dd4c360$b9ad810a@gnb.st.com> References: <013101c96070$9dd4c360$b9ad810a@gnb.st.com> Message-ID: On Dec 17, 2008, at 9:55 AM, Stephane DROUARD wrote: > Hi, > > Robert Bradshaw wrote: >>>> >>>> foo.py: >>>> print __name__ >>>> >>>> But under Python it returns "foo", "__builtin__" when cython'ized. >>>> >>>> As the documentation specifies, this can be fixed by: >>>> >>>> foo.py: >>>> global __name__ >>>> print __name__ >>>> >>>> Then both return "foo". >>> >>> Ah, good catch. Cython (and, incidentally I) didn't know that >>> __name__ is an already declared module-level variable, hence it >>> looked it up as a builtin (and found it there, so no error). This is >>> easily fixed by always having __name__ in the module namespace (what >>> other magic variables are there?) >> >> $ touch foo.py >> $ python2.5 -c 'import foo; print foo.__dict__.keys()' >> ['__builtins__', '__name__', '__file__', '__doc__'] >> >> To clarify, other than these (thanks for the list Lisandro) the only >> time this can cause issues is if one manually adds names to the >> module externally, or if one uses "import *" to overwrite a builtin. > > Do you plan to fix it (0.10.3 does not)? Yes, these are fixed in the devel branch (along with many other improvements). As soon as all the temp allocation stuff has stabled, we will release that as 0.11. - Robert From stefan_ml at behnel.de Wed Dec 17 22:37:25 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 17 Dec 2008 22:37:25 +0100 Subject: [Cython] moving the iter-range optimisation into a transform Message-ID: <49497115.8070209@behnel.de> Hi, initially, the iter-range transform was done during the analysis phase inside the For*StatNode, which feels a bit misplaced now that we have nicely working tree transformations. I tried moving it out into a transform (actually next to the iter-dict transform), but it's currently a "works for me". I'm pretty sure there are cases I forgot where this doesn't work (yet). It's definitely worth a review, and it would be nice if a few people could give it a try with their code base and check if the generated code comes out as expected. http://hg.cython.org/cython-devel/rev/c1a7180ac974 Thanks! Stefan From malkarouri at gmail.com Thu Dec 18 00:54:49 2008 From: malkarouri at gmail.com (Muhammad Alkarouri) Date: Wed, 17 Dec 2008 23:54:49 +0000 Subject: [Cython] Cannot coerce list to type 'list' Message-ID: Hi everyone, I have code that used to work in Cython 0.9.8.1.1 but doesn't work in the current version (0.10.3). I would like to know if this is a bug or is it something wrong in my code, and how to correct it either way. Can you please help me? The code and error are: from c_numpy cimport import_array, ndarray from c_python cimport list ... cdef class Scope: """A multidimensional interval""" cdef readonly ndarray centre cdef Scope parent cdef int height cdef list refinements def __init__(self, ndarray centre, Scope parent, int height): self.centre = centre self.parent = parent self.height = height self.refinements = [] ^ Cannot coerce list to type 'list' Best Regards, Muhammad Alkarouri From robertwb at math.washington.edu Thu Dec 18 01:05:11 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 17 Dec 2008 16:05:11 -0800 Subject: [Cython] Cannot coerce list to type 'list' In-Reply-To: References: Message-ID: <64A2BF1D-51C0-4346-B827-0E3261E5358B@math.washington.edu> On Dec 17, 2008, at 3:54 PM, Muhammad Alkarouri wrote: > Hi everyone, > > I have code that used to work in Cython 0.9.8.1.1 but doesn't work in > the current version (0.10.3). > > I would like to know if this is a bug or is it something wrong in my > code, and how to correct it either way. Can you please help me? The > code and error are: > > from c_numpy cimport import_array, ndarray > from c_python cimport list > ... > cdef class Scope: > """A multidimensional interval""" > cdef readonly ndarray centre > cdef Scope parent > cdef int height > cdef list refinements > def __init__(self, ndarray centre, Scope parent, int height): > self.centre = centre > self.parent = parent > self.height = height > self.refinements = [] > ^ > Cannot coerce list to type 'list' Hmm... looks like the c_python declaration of "list" is conflicting with the builtin list type which Cython now understands. Could you try with Cython 0.10.2? For now you can do self.refinements = [] What is the definition of list in c_python? If it's a cdef extern class of the actual python list, then this is a bug and should work. Also, you might not need c_python anymore. - Robert From malkarouri at gmail.com Thu Dec 18 01:25:18 2008 From: malkarouri at gmail.com (Muhammad Alkarouri) Date: Thu, 18 Dec 2008 00:25:18 +0000 Subject: [Cython] Cannot coerce list to type 'list' Message-ID: On Dec 17, 2008, Robert Bradshaw wrote: > What is the definition of list in c_python? If it's a cdef extern > class of the actual python list, then this is a bug and should work. > Also, you might not need c_python anymore. Thanks, that does the trick. The __builtin__.list was cdef extern in the c_python.pxd (shown below), I simply commented it and the code worked fine. Is there a fault with how I have it written, or should I report it as a bug now? # -*- Mode: Python -*- Not really, but close enough # Expose as much of the Python C API as we need here cdef extern from "stdlib.h": ctypedef int size_t cdef extern from "Python.h": ctypedef int Py_intptr_t void* PyMem_Malloc(size_t) void* PyMem_Realloc(void *p, size_t n) void PyMem_Free(void *p) char* PyString_AsString(object string) object PyString_FromString(char *v) object PyString_InternFromString(char *v) int PyErr_CheckSignals() object PyFloat_FromDouble(double v) void Py_XINCREF(object o) void Py_XDECREF(object o) void Py_CLEAR(object o) # use instead of decref ctypedef class __builtin__.dict [object PyDictObject]: pass ctypedef class __builtin__.list [object PyListObject]: pass Regards, Muhammad Alkarouri From robertwb at math.washington.edu Thu Dec 18 01:39:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 17 Dec 2008 16:39:33 -0800 Subject: [Cython] Cannot coerce list to type 'list' In-Reply-To: References: Message-ID: On Dec 17, 2008, at 4:25 PM, Muhammad Alkarouri wrote: > On Dec 17, 2008, Robert Bradshaw wrote: > >> What is the definition of list in c_python? If it's a cdef extern >> class of the actual python list, then this is a bug and should work. >> Also, you might not need c_python anymore. > > Thanks, that does the trick. > The __builtin__.list was cdef extern in the c_python.pxd (shown > below), I simply commented it and the code worked fine. > Is there a fault with how I have it written, or should I report it > as a bug now? I'm not sure what the best thing to do here is (certainly it could be fixed to ignore these extern types that match), but there's no advantage to declaring the empty dict and list types as Cython understands those types natively now. - Robert From stephane.drouard at st.com Thu Dec 18 10:09:20 2008 From: stephane.drouard at st.com (Stephane DROUARD) Date: Thu, 18 Dec 2008 10:09:20 +0100 Subject: [Cython] Scope rules vs. Python behaviour In-Reply-To: Message-ID: <001401c960f0$5162d960$b9ad810a@gnb.st.com> > Robert Bradshaw wrote: > > Yes, these are fixed in the devel branch (along with many other improvements). As soon > as all the temp allocation stuff has stabled, we will release that as 0.11. Great. Thanks. Stephane From robertwb at math.washington.edu Thu Dec 18 11:13:55 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 18 Dec 2008 02:13:55 -0800 Subject: [Cython] moving the iter-range optimisation into a transform In-Reply-To: <49497115.8070209@behnel.de> References: <49497115.8070209@behnel.de> Message-ID: On Dec 17, 2008, at 1:37 PM, Stefan Behnel wrote: > Hi, > > initially, the iter-range transform was done during the analysis phase > inside the For*StatNode, which feels a bit misplaced now that we have > nicely working tree transformations. Nice. It certainly fits here better. > I tried moving it out into a transform > (actually next to the iter-dict transform), but it's currently a > "works for > me". I'm pretty sure there are cases I forgot where this doesn't work > (yet). It's definitely worth a review, and it would be nice if a > few people > could give it a try with their code base and check if the generated > code > comes out as expected. > > http://hg.cython.org/cython-devel/rev/c1a7180ac974 I noticed that it does extra int -> object -> int conversions, e.g. * def foo(int a, int b): * cdef int n * for n in range(a, b): # <<<<<<<<<<<<<< * print n * */ __pyx_t_1 = PyInt_FromLong(__pyx_v_a); if (unlikely(!__pyx_t_1)) ... __pyx_t_2 = __pyx_PyInt_long(__pyx_t_1); if (unlikely((__pyx_t_2 == (long)-1) && PyErr_Occurred())) ... Py_DECREF(__pyx_t_1); __pyx_t_1 = 0; __pyx_t_1 = PyInt_FromLong(__pyx_v_b); if (unlikely(!__pyx_t_1)) ... __pyx_t_3 = __pyx_PyInt_long(__pyx_t_1); if (unlikely((__pyx_t_3 == (long)-1) && PyErr_Occurred())) ... Py_DECREF(__pyx_t_1); __pyx_t_1 = 0; for (__pyx_v_n = __pyx_t_2; __pyx_v_n < __pyx_t_3; __pyx_v_n+=1) { Originally, it would simply emit the line for (__pyx_v_n = __pyx_v_a; __pyx_v_n < __pyx_v_b; __pyx_v_n+=1) { - Robert From malkarouri at gmail.com Thu Dec 18 12:53:04 2008 From: malkarouri at gmail.com (Muhammad Alkarouri) Date: Thu, 18 Dec 2008 11:53:04 +0000 Subject: [Cython] Cannot coerce list to type 'list' In-Reply-To: References: Message-ID: 2008/12/18 Robert Bradshaw : > On Dec 17, 2008, at 4:25 PM, Muhammad Alkarouri wrote: > >> On Dec 17, 2008, Robert Bradshaw wrote: >> >>> What is the definition of list in c_python? If it's a cdef extern >>> class of the actual python list, then this is a bug and should work. >>> Also, you might not need c_python anymore. >> >> Thanks, that does the trick. >> The __builtin__.list was cdef extern in the c_python.pxd (shown >> below), I simply commented it and the code worked fine. >> Is there a fault with how I have it written, or should I report it >> as a bug now? > > I'm not sure what the best thing to do here is (certainly it could be > fixed to ignore these extern types that match), but there's no > advantage to declaring the empty dict and list types as Cython > understands those types natively now. > I suggested to report it according to your earlier suggestion, but I would say the best way forward for Cython is to explicitly deprecate the practice of declaring types that are defined by Cython. Say, cython should fail with a kind of duplicate error or 'new definition overrides cython built in' kind of message. The idea is that: 1. The will be less hairy code and more maintainable for cython developers. 2. For a cython user like me, such a failure would be clear enough to modify the source in accordance. 3. Cython will probably grow more capabilities in types and other optimizations in the future. These may replace user defined ones. If users are used to this kind of change this will give Cython developers more leeway in doing improvements in types and functions. In short, I would fix that by giving a clearer error and also clearly noting that in the documentation. The user should expect to have to delete the offending duplicate type or function. Does that make sense? Cheers, Muhammad From stefan_ml at behnel.de Thu Dec 18 17:51:07 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 18 Dec 2008 17:51:07 +0100 Subject: [Cython] prevent comprehension variables from leaking Message-ID: <494A7F7B.70408@behnel.de> Hi, I think this change might be a bit controversial, so I would like to hear some comments on it. I added a transform that keeps the run variable in comprehensions from leaking by replacing it with a temp variable. This mimics the way comprehensions work in Py3 and generator expressions work in Py2. I.e. this will no longer work: [x for x in range(10)] print x Well, it may still work, but it will then print "None" instead of "9", as x is not initialised outside of the loop. Even worse, BTW, it will currently print an arbitrary integer if 'x' is an uninitialised C int... Doing this change, I noticed that this makes it somewhat tricky to set a type for 'x'. This still works: cdef int x [x for x in range(10)] and it still makes 'x' and int, but it feels somewhat wrong that you have to do it this way. We might be able to allow something like this: [x for x in range(10)] but to me that doesn't feel much better. Any comments or ideas? Would it be enough to document this behaviour? Something like: "the original value of the loop variable in a comprehension is restored on exit" or something like that? Stefan From stefan_ml at behnel.de Thu Dec 18 18:05:51 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 18 Dec 2008 18:05:51 +0100 Subject: [Cython] moving the iter-range optimisation into a transform In-Reply-To: References: <49497115.8070209@behnel.de> Message-ID: <494A82EF.8060805@behnel.de> Robert Bradshaw wrote: > I noticed that it does extra int -> object -> int conversions Fixed. Stefan From dagss at student.matnat.uio.no Thu Dec 18 18:48:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 18 Dec 2008 18:48:39 +0100 Subject: [Cython] prevent comprehension variables from leaking In-Reply-To: <494A7F7B.70408@behnel.de> References: <494A7F7B.70408@behnel.de> Message-ID: <494A8CF7.5080009@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > I think this change might be a bit controversial, so I would like to hear > some comments on it. I added a transform that keeps the run variable in > comprehensions from leaking by replacing it with a temp variable. This > mimics the way comprehensions work in Py3 and generator expressions work in > Py2. I.e. this will no longer work: > > [x for x in range(10)] > print x > > Well, it may still work, but it will then print "None" instead of "9", as x > is not initialised outside of the loop. Even worse, BTW, it will currently > print an arbitrary integer if 'x' is an uninitialised C int... > No, doesn't sound good. But as you suggest this may be because Cython already departs from Python variable declaration rules. This is probably another instance of "flow control would fix this" :-) (because we could then let __pyx_v_x = NULL initially, and check for NULL on every access and raise NameError in the right way, and flow control could optimize it). > Doing this change, I noticed that this makes it somewhat tricky to set a > type for 'x'. This still works: > > cdef int x > [x for x in range(10)] > > and it still makes 'x' and int, but it feels somewhat wrong that you have > to do it this way. We might be able to allow something like this: > > [x for x in range(10)] > > but to me that doesn't feel much better. > > Any comments or ideas? > I think either a) leave it like it is, or b) the [] sets up an entirely new scope (if this is how Python 3 does it, I don't know myself). Anything in-between seems only confusing. This means that if this change is done, it must be OK to do cdef char* x print [x for x in range(10)] because the iterator names used in the [] is in no way connected with the "x" in the outer scope. So as you say, new syntax may be needed. However: a) For builtins like range, we can always infer the variable as an int, similar to the ForIn-optimization. b) Otherwise, typing is often not needed on the variable -- i.e. unless you use it multiple times, one can always cast in the expression instead and it will do the same thing. The exception seems to be cases like [(x * x) for x in f()] or [((x)[0] * (x)[1] for x in f()] (I can imagine this occuring a lot in code like Sage though, but perhaps only on the Python interpreter level...) Hmm. Some additional proposals: a) To force the user to write code like this, but "pull out" the casts (i.e. if "x" only occurs as "x", pull up the int conversion as a common subexpression). A bit hairy to implement though. b) [x for x in cython.typediter(cython.int, f())] b) [x for int x in f()] # I don't really like this one, too hard to read. Dag Sverre From robertwb at math.washington.edu Thu Dec 18 18:55:38 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 18 Dec 2008 09:55:38 -0800 Subject: [Cython] Cannot coerce list to type 'list' In-Reply-To: References: Message-ID: On Dec 18, 2008, at 3:53 AM, Muhammad Alkarouri wrote: > 2008/12/18 Robert Bradshaw : >> On Dec 17, 2008, at 4:25 PM, Muhammad Alkarouri wrote: >> >>> On Dec 17, 2008, Robert Bradshaw wrote: >>> >>>> What is the definition of list in c_python? If it's a cdef extern >>>> class of the actual python list, then this is a bug and should >>>> work. >>>> Also, you might not need c_python anymore. >>> >>> Thanks, that does the trick. >>> The __builtin__.list was cdef extern in the c_python.pxd (shown >>> below), I simply commented it and the code worked fine. >>> Is there a fault with how I have it written, or should I report it >>> as a bug now? >> >> I'm not sure what the best thing to do here is (certainly it could be >> fixed to ignore these extern types that match), but there's no >> advantage to declaring the empty dict and list types as Cython >> understands those types natively now. >> > > I suggested to report it according to your earlier suggestion, but I > would say the best way forward for Cython is to explicitly deprecate > the practice of declaring types that are defined by Cython. Say, > cython should fail with a kind of duplicate error or 'new definition > overrides cython built in' kind of message. The idea is that: > > 1. The will be less hairy code and more maintainable for cython > developers. > 2. For a cython user like me, such a failure would be clear enough to > modify the source in accordance. > 3. Cython will probably grow more capabilities in types and other > optimizations in the future. These may replace user defined ones. If > users are used to this kind of change this will give Cython developers > more leeway in doing improvements in types and functions. > > In short, I would fix that by giving a clearer error and also clearly > noting that in the documentation. The user should expect to have to > delete the offending duplicate type or function. Does that make sense? Yes, that sounds very reasonable. http://trac.cython.org/cython_trac/ticket/170 - Robert From dagss at student.matnat.uio.no Thu Dec 18 18:59:23 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 18 Dec 2008 18:59:23 +0100 Subject: [Cython] prevent comprehension variables from leaking In-Reply-To: <494A8CF7.5080009@student.matnat.uio.no> References: <494A7F7B.70408@behnel.de> <494A8CF7.5080009@student.matnat.uio.no> Message-ID: <494A8F7B.4010107@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > > The exception seems to be cases like > > [(x * x) for x in f()] > or > [((x)[0] * (x)[1] for x in f()] > > (I can imagine this occuring a lot in code like Sage though, but perhaps > only on the Python interpreter level...) Hmm. Some additional proposals: > a) To force the user to write code like this, but "pull out" the casts > (i.e. if "x" only occurs as "x", pull up the int conversion as a > common subexpression). A bit hairy to implement though. > Nope, we cannot pull out casts as common subexpressions in some cases (though we can in some that can be checked for). Counterexample (contrived warning!): call_f= g() # If False, an object from f() could be non-int, and we want [0,0,0,...,0] [(0 if not call_f else f(x*x)) for x in f()] Dag Sverre From dalcinl at gmail.com Thu Dec 18 22:47:32 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Dec 2008 19:47:32 -0200 Subject: [Cython] cython-devel broken in 1507:d87254afb524 Message-ID: Installed Cython segfaults, tested a bit and likely the problem is just in the second to last commit: changeset: 1507:d87254afb524 user: Stefan Behnel date: Thu Dec 18 17:48:44 2008 +0100 summary: new transform that hides the loop variable in a comprehension BTW, just in case, I'll ask something (forget me if this is already the current case) Why you never experience such breakages when running the testsuite before pushing? Should't runtest.py forcibly compile the Scanning,Parsing, etc extension modules? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Thu Dec 18 23:06:21 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 18 Dec 2008 23:06:21 +0100 Subject: [Cython] cython-devel broken in 1507:d87254afb524 In-Reply-To: References: Message-ID: <494AC95D.8040606@student.matnat.uio.no> Lisandro Dalcin wrote: > Installed Cython segfaults, tested a bit and likely the problem is > just in the second to last commit: > > changeset: 1507:d87254afb524 > user: Stefan Behnel > date: Thu Dec 18 17:48:44 2008 +0100 > summary: new transform that hides the loop variable in a comprehension > > > BTW, just in case, I'll ask something (forget me if this is already > the current case) > > Why you never experience such breakages when running the testsuite > before pushing? > Should't runtest.py forcibly compile the Scanning,Parsing, etc > extension modules? Ideally runtests.py should offer as clear diagnostics as possible and allow you to focus on the spot of the problem. A self-compiled Cython segfaulting is a very indirect way of spotting a problem. (I'm now convinced a refcount nanny is needed and have started some work on it, but it might take a few weeks at this pace.) -- Dag Sverre From dalcinl at gmail.com Thu Dec 18 23:20:58 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Dec 2008 20:20:58 -0200 Subject: [Cython] cython-devel broken in 1507:d87254afb524 In-Reply-To: <494AC95D.8040606@student.matnat.uio.no> References: <494AC95D.8040606@student.matnat.uio.no> Message-ID: On Thu, Dec 18, 2008 at 8:06 PM, Dag Sverre Seljebotn wrote: > Ideally runtests.py should offer as clear diagnostics as possible and > allow you to focus on the spot of the problem. A self-compiled Cython > segfaulting is a very indirect way of spotting a problem. However, I can remember 4 of 5 times that those segfaults pointed me out something was broken :-). > (I'm now convinced a refcount nanny is needed and have started some work > on it, but it might take a few weeks at this pace.) > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Dec 19 10:23:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 10:23:36 +0100 Subject: [Cython] cython-devel broken in 1507:d87254afb524 In-Reply-To: <494AC95D.8040606@student.matnat.uio.no> References: <494AC95D.8040606@student.matnat.uio.no> Message-ID: <494B6818.4050109@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Should't runtest.py forcibly compile the Scanning,Parsing, etc >> extension modules? > > Ideally runtests.py should offer as clear diagnostics as possible and > allow you to focus on the spot of the problem. A self-compiled Cython > segfaulting is a very indirect way of spotting a problem. I agree. The whole purpose of the test suite is finding bugs in Cython. Running it with a compiled Cython compiler would actually degrade the quality of error reports by killing the stack trace. Stefan From stefan_ml at behnel.de Fri Dec 19 10:29:37 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 10:29:37 +0100 Subject: [Cython] cython-devel broken in 1507:d87254afb524 In-Reply-To: References: Message-ID: <494B6981.6040708@behnel.de> Hi, Lisandro Dalcin wrote: > Installed Cython segfaults I guess that's the bug I spotted when testing lxml with it. I fixed it tonight but couldn't push it before now. > changeset:1507:d87254afb524 > user: Stefan Behnel > date: Thu Dec 18 17:48:44 2008 +0100 > summary: new transform that hides the loop variable in a comprehension I pushed this change to have people comment on it. I don't mind if it has bugs, as it's not certain that it will stay in (it's a separate transform, so it's easy to disable). Stefan From stefan_ml at behnel.de Fri Dec 19 15:44:06 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 15:44:06 +0100 Subject: [Cython] prevent comprehension variables from leaking In-Reply-To: <494A8CF7.5080009@student.matnat.uio.no> References: <494A7F7B.70408@behnel.de> <494A8CF7.5080009@student.matnat.uio.no> Message-ID: <494BB336.2020801@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> I think this change might be a bit controversial, so I would like to hear >> some comments on it. I added a transform that keeps the run variable in >> comprehensions from leaking by replacing it with a temp variable. > > I think either a) leave it like it is, or b) the [] sets up an entirely > new scope (if this is how Python 3 does it, I don't know myself). > Anything in-between seems only confusing. I agree that a local scope would be the right way to do it. The problem of leaking comprehension run variables was fixed in Py3, and the current support for set comprehensions and dict comprehensions in Cython fails to comply here. Having list comprehensions leak is bad enough, and I think it should be fixed in Cython even if that means a slight incompatibility with Py2.x. > This means that if this change is done, it must be OK to do > > cdef char* x > print [x for x in range(10)] That would be expected, yes. However, type inference will not work in most cases, as there is no way to specify what an iterator will return. And most use cases will not involve range() but an arbitrary list or dict. And I agree that a suitable syntax is hard to come up with. Stefan From stefan_ml at behnel.de Fri Dec 19 16:13:24 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 16:13:24 +0100 Subject: [Cython] prevent comprehension variables from leaking In-Reply-To: <494A8CF7.5080009@student.matnat.uio.no> References: <494A7F7B.70408@behnel.de> <494A8CF7.5080009@student.matnat.uio.no> Message-ID: <494BBA14.2050502@behnel.de> Dag Sverre Seljebotn wrote: > a) For builtins like range, we can always infer the variable as an int, > similar to the ForIn-optimization. Do you mean "For-From" optimisation? I actually think the fact that the for-from loop makes its run variable a C long even if not explicitly typed is worth a general optimisation that should also apply to for-in-range loops. Maybe restricting this to the case where we find a compile-time known value range that fits into 31 bits would catch most use cases and always be portable, regardless of sizeof(long). Stefan From stefan_ml at behnel.de Fri Dec 19 17:06:37 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 17:06:37 +0100 Subject: [Cython] separate 'initial typing' phase? Message-ID: <494BC68D.3090103@behnel.de> Hi, I noticed that many of the transforms I implemented lately are supposed to run after type analysis for mainly one reason: they must be able to distinguish between builtin types and other stuff. For example, they must know that "range" denotes the builtin range function/type, that [] or a list comprehension return a list, and that a variable declared 'cdef dict' can only contain a dict (ok, possibly None). Would it make sense to add something like an initial pre-typing step that tries to annotate as many nodes as possible with their definite type, so that the transforms could run between this step and the final type analysis? That would make the final tree available for type analysis (and thus avoid unnecessary coercions), and still provide the required type information to transforms. Stefan From hoytak at cs.ubc.ca Fri Dec 19 17:25:06 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Fri, 19 Dec 2008 08:25:06 -0800 Subject: [Cython] FYI: current cython parser module has compile error Message-ID: <4db580fd0812190825n1b5119f1o83c73173a7ca7b4e@mail.gmail.com> Hello, Just an FYI: In the current cython revision, 1517:c0e5bd815f52, I get the following error while rebuilding it: Compiling module Cython.Plex.Scanners ... Compiling module Cython.Compiler.Scanning ... Compiling module Cython.Compiler.Parsing ... Error converting Pyrex file to C: ------------------------------------------------------------ ... s.next() exprs += p_simple_expr_list(s) s.expect(']') return ExprNodes.ListNode(pos, args = exprs) def p_list_iter(s, body): ^ ------------------------------------------------------------ /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.py:717:0: wrong number of arguments ERROR: local variable 'declarator' referenced before assignment Extension module compilation failed, using plain Python implementation It seems there's a discrepancy between the .pxd file definition of p_list_iter (and a couple others) and the .py file. (Perhaps a file missing in a commit?) Thanks again for all the work! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Fri Dec 19 18:16:09 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 18:16:09 +0100 Subject: [Cython] FYI: current cython parser module has compile error In-Reply-To: <4db580fd0812190825n1b5119f1o83c73173a7ca7b4e@mail.gmail.com> References: <4db580fd0812190825n1b5119f1o83c73173a7ca7b4e@mail.gmail.com> Message-ID: <494BD6D9.5020007@behnel.de> Hi, Hoyt Koepke wrote: > Just an FYI: In the current cython revision, 1517:c0e5bd815f52, I get > the following error while rebuilding it: > > Compiling module Cython.Plex.Scanners ... > Compiling module Cython.Compiler.Scanning ... > Compiling module Cython.Compiler.Parsing ... > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > s.next() > exprs += p_simple_expr_list(s) > s.expect(']') > return ExprNodes.ListNode(pos, args = exprs) > > def p_list_iter(s, body): > ^ > ------------------------------------------------------------ Yep, sorry. I guess it would still be good to integrate the build of Cython itself into the test run *somewhere*... Stefan From dalcinl at gmail.com Fri Dec 19 18:19:41 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 19 Dec 2008 15:19:41 -0200 Subject: [Cython] removing ALL those nasty warnings about string constants Message-ID: I'm working on release petsc4py today. If core PETSc is compiled with C++, then Cython-generated sources are also compiled with C++. When using Cython release 0.10.3, I can count about 260 warnings like this: * In generated C sources src/petsc4py_PETSc.c:82219: warning: deprecated conversion from string constant to 'char*' * In geneted C-API headers: src/include/petsc4py/petsc4py_PETSc_api.h:197: warning: deprecated conversion from string constant to 'char*' This is a complete mess, and of course the mess is not on Cython side, but in Python 2.x headers. Python 3.x has improved a lot, though still there are a few API calls that use bare "char*" instead of "const char*". I want to ask all you if I could make the required changes to silence all those warnings, even when this will lead to introduce explicit casts to (char*). I know I'm too much pedantic about this stuff, but please, please, let me fix this righ now (well, actually, tomorrow...) Thanks, -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Fri Dec 19 19:25:42 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 19 Dec 2008 19:25:42 +0100 (CET) Subject: [Cython] separate 'initial typing' phase? In-Reply-To: <494BC68D.3090103@behnel.de> References: <494BC68D.3090103@behnel.de> Message-ID: <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> Stefan Behnel wrote: > Hi, > > I noticed that many of the transforms I implemented lately are supposed to > run after type analysis for mainly one reason: they must be able to > distinguish between builtin types and other stuff. For example, they must > know that "range" denotes the builtin range function/type, that [] or a > list comprehension return a list, and that a variable declared 'cdef dict' > can only contain a dict (ok, possibly None). > > Would it make sense to add something like an initial pre-typing step that > tries to annotate as many nodes as possible with their definite type, so > that the transforms could run between this step and the final type > analysis? That would make the final tree available for type analysis (and > thus avoid unnecessary coercions), and still provide the required type > information to transforms. I guess it could work within the features we have today. But: Overloaded functions would mean that call nodes would be pushed to the latter stage, while any kind of simple type inference would push local variable types to the latter stage, and so on. In general in seems to be a web that must be untangled with a keen eye on the exact functionality that we provide in the Cython language, and as that is subject to change, at least a little, those kind of designs make me rather careful... But I would really like something of the kind too. Perhaps (and I think this might have been Robert's idea) coercion and typing could be split. This requires more thinking but the end-result would be a more flexible and natural design. I.e. something like this: In the typing phase each node declaratively specifies a *set* of possible types for input(s), and rules for resulting output types. Then a seperate pass is made which follows simple rules to insert coercion nodes. I was pushing rather loudly for something along these lines back in May, and I remember that there were some concerns that Greg and Robert raised with this that put me off the idea eventually (but the concept wasn't as clear in my mind at the time as it seems to be in yours, so we may get farther this time). Dag Sverre From dagss at student.matnat.uio.no Fri Dec 19 19:58:36 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 19 Dec 2008 19:58:36 +0100 Subject: [Cython] prevent comprehension variables from leaking In-Reply-To: <494BBA14.2050502@behnel.de> References: <494A7F7B.70408@behnel.de> <494A8CF7.5080009@student.matnat.uio.no> <494BBA14.2050502@behnel.de> Message-ID: <494BEEDC.3050704@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> a) For builtins like range, we can always infer the variable as an int, >> similar to the ForIn-optimization. > > Do you mean "For-From" optimisation? Well, yes. BTW, thanks a lot for refactoring the For-From-optimization, I've been itching to get rid of that "self.__class__ = ..." thing since I started on the transform business, but it kept falling behind other things... > I actually think the fact that the for-from loop makes its run variable a C > long even if not explicitly typed is worth a general optimisation that > should also apply to for-in-range loops. Maybe restricting this to the case > where we find a compile-time known value range that fits into 31 bits would > catch most use cases and always be portable, regardless of sizeof(long). OK I wasn't aware of the details here. Actually I'd be OK myself to make a detour from Python compatability in this specific case -- make range by default only cover the int range, and require to use "cython.bigrange" otherwise. I know we aim for 100% compatability but this is such a common case, and how often are people iterating over things larger than 2^31 anyway? (Well, in Sage they probably do, but they have their own integer types anyway.) One would raise an exception before entering the loop if the integer was too big, which could inform about the bigrange function. -- Dag Sverre From malkarouri at gmail.com Fri Dec 19 19:59:38 2008 From: malkarouri at gmail.com (Muhammad Alkarouri) Date: Fri, 19 Dec 2008 18:59:38 +0000 Subject: [Cython] Cannot coerce list to type 'list' In-Reply-To: References: Message-ID: 2008/12/18 Robert Bradshaw : > ... > Yes, that sounds very reasonable. > > http://trac.cython.org/cython_trac/ticket/170 > > - Robert Very much appreciated. Muhammad From robertwb at math.washington.edu Fri Dec 19 20:01:40 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 11:01:40 -0800 Subject: [Cython] prevent comprehension variables from leaking In-Reply-To: <494BEEDC.3050704@student.matnat.uio.no> References: <494A7F7B.70408@behnel.de> <494A8CF7.5080009@student.matnat.uio.no> <494BBA14.2050502@behnel.de> <494BEEDC.3050704@student.matnat.uio.no> Message-ID: On Dec 19, 2008, at 10:58 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> a) For builtins like range, we can always infer the variable as >>> an int, >>> similar to the ForIn-optimization. >> >> Do you mean "For-From" optimisation? > > Well, yes. BTW, thanks a lot for refactoring the For-From- > optimization, > I've been itching to get rid of that "self.__class__ = ..." thing > since > I started on the transform business, but it kept falling behind other > things... Me too... that first one was a hack, but the best I could do before we had any kind of transformation structure in place. > >> I actually think the fact that the for-from loop makes its run >> variable a C >> long even if not explicitly typed is worth a general optimisation >> that >> should also apply to for-in-range loops. Maybe restricting this to >> the case >> where we find a compile-time known value range that fits into 31 >> bits would >> catch most use cases and always be portable, regardless of sizeof >> (long). > > OK I wasn't aware of the details here. > > Actually I'd be OK myself to make a detour from Python > compatability in > this specific case -- make range by default only cover the int range, > and require to use "cython.bigrange" otherwise. > > I know we aim for 100% > compatability but this is such a common case, and how often are people > iterating over things larger than 2^31 anyway? (Well, in Sage they > probably do, but they have their own integer types anyway.) > > One would raise an exception before entering the loop if the > integer was > too big, which could inform about the bigrange function. I am -1 on the idea of always doing it and throwing an error, but if the value is known at compile time to fit into a long (e.g. 31 bits max) then this certainly sounds worth it, though the gains will not be huge if it has to construct the index as a python object anyways (at least, not compared to an actual c for loop). I think if we do type inference on the index, it should be the widest type of the range parameters. - Robert From hoytak at cs.ubc.ca Fri Dec 19 20:01:28 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Fri, 19 Dec 2008 11:01:28 -0800 Subject: [Cython] FYI: current cython parser module has compile error In-Reply-To: <494BD6D9.5020007@behnel.de> References: <4db580fd0812190825n1b5119f1o83c73173a7ca7b4e@mail.gmail.com> <494BD6D9.5020007@behnel.de> Message-ID: <4db580fd0812191101m5fd496c1qbbbc759d50090af3@mail.gmail.com> Hey Stefan, Thanks! It builds fine now. However, cython now segfaults when compiling my code, but I guess that's a separate issue that's currently being looked at. Here's the backtrace from the coredump in case it's helpful to anyone. The suspicious thing seems to be the __pyx_self=0x0 in #3. If the core dump would be helpful to anyone, I can email it directly or post it online somewhere. --Hoyt #0 0xb7f49206 in PyDict_SetItem () from /usr/lib/libpython2.5.so.1.0 #1 0xb7b3131b in __pyx_f_6Cython_8Compiler_7Parsing_p_buffer_access (__pyx_v_s=0xb719c87c, __pyx_v_base_type_node=0xb6dde86c, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:32910 #2 0xb7b33976 in __pyx_f_6Cython_8Compiler_7Parsing_p_c_simple_base_type (__pyx_v_s=0xb719c87c, __pyx_v_self_flag=0xb7ff3760, __pyx_v_nonempty=0xb7ff376c, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:32530 #3 0xb7b342f5 in __pyx_pf_6Cython_8Compiler_7Parsing_p_c_simple_base_type (__pyx_self=0x0, __pyx_args=0xb6e9948c, __pyx_kwds=0xb6de0b54) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:32640 #4 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #5 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #6 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #7 0xb7afd7ef in __pyx_f_6Cython_8Compiler_7Parsing_p_c_base_type (__pyx_v_s=0xb719c87c, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfa55ab0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:31414 #8 0xb7b063f6 in __pyx_pf_6Cython_8Compiler_7Parsing_p_c_base_type (__pyx_self=0x0, __pyx_args=0xb7165b4c, __pyx_kwds=0xb6de0a44) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:31506 #9 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #10 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #11 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #12 0xb7b118ee in __pyx_f_6Cython_8Compiler_7Parsing_p_c_func_or_var_declaration (__pyx_v_s=0xb719c87c, __pyx_v_pos=0xb6ed034c, __pyx_v_ctx=0xb6dde80c, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:41471 #13 0xb7b775de in __pyx_f_6Cython_8Compiler_7Parsing_p_cdef_statement (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6dde80c, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:38931 #14 0xb7b7fad7 in __pyx_f_6Cython_8Compiler_7Parsing_p_statement (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6dde7ac, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfa55ca4) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:28857 #15 0xb7b82f86 in __pyx_pf_6Cython_8Compiler_7Parsing_p_statement (__pyx_self=0x0, __pyx_args=0xb6e997ec, __pyx_kwds=0xb6ddfbdc) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:29689 #16 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #17 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #18 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #19 0xb7b14e38 in __pyx_f_6Cython_8Compiler_7Parsing_p_statement_list (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6dde7ac, __pyx_skip_dispatch=0, __pyx_optional_args=0x0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:29822 #20 0xb7b73ace in __pyx_f_6Cython_8Compiler_7Parsing_p_suite (__pyx_v_s=0xb719c87c, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfa55e0c) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:30177 #21 0xb7b74c4a in __pyx_pf_6Cython_8Compiler_7Parsing_p_suite (__pyx_self=0x0, __pyx_args=0xb6e996ec, __pyx_kwds=0xb6ddd8ac) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:30496 #22 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #23 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #24 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #25 0xb7b12afe in __pyx_f_6Cython_8Compiler_7Parsing_p_c_func_or_var_declaration (__pyx_v_s=0xb719c87c, __pyx_v_pos=0xb6ec8e8c, __pyx_v_ctx=0xb6dde3ac, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:41656 #26 0xb7b775de in __pyx_f_6Cython_8Compiler_7Parsing_p_cdef_statement (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6dde3ac, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:38931 #27 0xb7b7fad7 in __pyx_f_6Cython_8Compiler_7Parsing_p_statement (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6dd974c, __pyx_skip_dispatch=0, ---Type to continue, or q to quit--- __pyx_optional_args=0xbfa56004) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:28857 #28 0xb7b82f86 in __pyx_pf_6Cython_8Compiler_7Parsing_p_statement (__pyx_self=0x0, __pyx_args=0xb6e9980c, __pyx_kwds=0xb6ddb934) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:29689 #29 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #30 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #31 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #32 0xb7b14e38 in __pyx_f_6Cython_8Compiler_7Parsing_p_statement_list (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6dd974c, __pyx_skip_dispatch=0, __pyx_optional_args=0x0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:29822 #33 0xb7b73ace in __pyx_f_6Cython_8Compiler_7Parsing_p_suite (__pyx_v_s=0xb719c87c, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfa5616c) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:30177 #34 0xb7b74c4a in __pyx_pf_6Cython_8Compiler_7Parsing_p_suite (__pyx_self=0x0, __pyx_args=0xb6e9968c, __pyx_kwds=0xb6f2ef0c) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:30496 #35 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #36 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #37 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #38 0xb7b29652 in __pyx_f_6Cython_8Compiler_7Parsing_p_c_class_definition (__pyx_v_s=0xb719c87c, __pyx_v_pos=0xb6f33dec, __pyx_v_ctx=0xb6f3664c, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:44879 #39 0xb7b7693f in __pyx_f_6Cython_8Compiler_7Parsing_p_cdef_statement (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6f3664c, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:38712 #40 0xb7b7fad7 in __pyx_f_6Cython_8Compiler_7Parsing_p_statement (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6f2ffcc, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfa56374) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:28857 #41 0xb7b82f86 in __pyx_pf_6Cython_8Compiler_7Parsing_p_statement (__pyx_self=0x0, __pyx_args=0xb6e996cc, __pyx_kwds=0xb6f3479c) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:29689 #42 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #43 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #44 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #45 0xb7b14e38 in __pyx_f_6Cython_8Compiler_7Parsing_p_statement_list (__pyx_v_s=0xb719c87c, __pyx_v_ctx=0xb6f2ffcc, __pyx_skip_dispatch=0, __pyx_optional_args=0xbfa56484) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:29822 #46 0xb7b155b6 in __pyx_pf_6Cython_8Compiler_7Parsing_p_statement_list (__pyx_self=0x0, __pyx_args=0xb719004c, __pyx_kwds=0xb6f3413c) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:29987 #47 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #48 0xb7f18157 in PyObject_Call () from /usr/lib/libpython2.5.so.1.0 #49 0xb7f9075c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.5.so.1.0 #50 0xb7b595b6 in __pyx_f_6Cython_8Compiler_7Parsing_p_module (__pyx_v_s=0xb719c87c, __pyx_v_pxd=0x804f0ac, __pyx_v_full_module_name=0xb7197bb0, __pyx_skip_dispatch=0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:46999 #51 0xb7b5a195 in __pyx_pf_6Cython_8Compiler_7Parsing_p_module (__pyx_self=0x0, __pyx_args=0xb6ec8464, __pyx_kwds=0x0) at /home/hoytak/sysroot/src/cython/Cython/Compiler/Parsing.c:47204 #52 0xb7f4a3c9 in PyCFunction_Call () from /usr/lib/libpython2.5.so.1.0 #53 0xb7f96e49 in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #54 0xb7f9869d in PyEval_EvalCodeEx () from /usr/lib/libpython2.5.so.1.0 #55 0xb7f96c8a in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #56 0xb7f9869d in PyEval_EvalCodeEx () from /usr/lib/libpython2.5.so.1.0 #57 0xb7f96c8a in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #58 0xb7f9795f in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #59 0xb7f9869d in PyEval_EvalCodeEx () from /usr/lib/libpython2.5.so.1.0 #60 0xb7f96c8a in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #61 0xb7f9869d in PyEval_EvalCodeEx () from /usr/lib/libpython2.5.so.1.0 #62 0xb7f96c8a in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #63 0xb7f9869d in PyEval_EvalCodeEx () from /usr/lib/libpython2.5.so.1.0 #64 0xb7f96c8a in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #65 0xb7f9795f in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #66 0xb7f9795f in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #67 0xb7f9795f in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #68 0xb7f9795f in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #69 0xb7f9795f in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #70 0xb7f9869d in PyEval_EvalCodeEx () from /usr/lib/libpython2.5.so.1.0 #71 0xb7f96c8a in PyEval_EvalFrameEx () from /usr/lib/libpython2.5.so.1.0 #72 0xb7f9869d in PyEval_EvalCodeEx () from /usr/lib/libpython2.5.so.1.0 #73 0xb7f988b3 in PyEval_EvalCode () from /usr/lib/libpython2.5.so.1.0 #74 0xb7fb2e82 in ?? () from /usr/lib/libpython2.5.so.1.0 #75 0xb7fb2f42 in PyRun_FileExFlags () from /usr/lib/libpython2.5.so.1.0 #76 0xb7fb46cc in PyRun_SimpleFileExFlags () from /usr/lib/libpython2.5.so.1.0 #77 0xb7fb4e3a in PyRun_AnyFileExFlags () from /usr/lib/libpython2.5.so.1.0 #78 0xb7fbe8ff in Py_Main () from /usr/lib/libpython2.5.so.1.0 On Fri, Dec 19, 2008 at 9:16 AM, Stefan Behnel wrote: > Hi, > > Hoyt Koepke wrote: >> Just an FYI: In the current cython revision, 1517:c0e5bd815f52, I get >> the following error while rebuilding it: >> >> Compiling module Cython.Plex.Scanners ... >> Compiling module Cython.Compiler.Scanning ... >> Compiling module Cython.Compiler.Parsing ... >> >> Error converting Pyrex file to C: >> ------------------------------------------------------------ >> ... >> s.next() >> exprs += p_simple_expr_list(s) >> s.expect(']') >> return ExprNodes.ListNode(pos, args = exprs) >> >> def p_list_iter(s, body): >> ^ >> ------------------------------------------------------------ > > Yep, sorry. I guess it would still be good to integrate the build of Cython > itself into the test run *somewhere*... > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Fri Dec 19 20:10:16 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 20:10:16 +0100 Subject: [Cython] FYI: current cython parser module has compile error In-Reply-To: <4db580fd0812191101m5fd496c1qbbbc759d50090af3@mail.gmail.com> References: <4db580fd0812190825n1b5119f1o83c73173a7ca7b4e@mail.gmail.com> <494BD6D9.5020007@behnel.de> <4db580fd0812191101m5fd496c1qbbbc759d50090af3@mail.gmail.com> Message-ID: <494BF198.8080202@behnel.de> Hi, Hoyt Koepke wrote: > Thanks! It builds fine now. > However, cython now segfaults when compiling my code Same here, I just noticed that. I'm looking into it. Stefan From stefan_ml at behnel.de Fri Dec 19 20:16:50 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 20:16:50 +0100 Subject: [Cython] FYI: current cython parser module has compile error In-Reply-To: <494BF198.8080202@behnel.de> References: <4db580fd0812190825n1b5119f1o83c73173a7ca7b4e@mail.gmail.com> <494BD6D9.5020007@behnel.de> <4db580fd0812191101m5fd496c1qbbbc759d50090af3@mail.gmail.com> <494BF198.8080202@behnel.de> Message-ID: <494BF322.3080702@behnel.de> Stefan Behnel wrote: > Hoyt Koepke wrote: >> Thanks! It builds fine now. >> However, cython now segfaults when compiling my code > > Same here, I just noticed that. I'm looking into it. Fixed. Stefan From robertwb at math.washington.edu Fri Dec 19 20:16:47 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 11:16:47 -0800 Subject: [Cython] separate 'initial typing' phase? In-Reply-To: <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> Message-ID: On Dec 19, 2008, at 10:25 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> I noticed that many of the transforms I implemented lately are >> supposed to >> run after type analysis for mainly one reason: they must be able to >> distinguish between builtin types and other stuff. For example, >> they must >> know that "range" denotes the builtin range function/type, that [] >> or a >> list comprehension return a list, and that a variable declared >> 'cdef dict' >> can only contain a dict (ok, possibly None). >> >> Would it make sense to add something like an initial pre-typing >> step that >> tries to annotate as many nodes as possible with their definite >> type, so >> that the transforms could run between this step and the final type >> analysis? That would make the final tree available for type >> analysis (and >> thus avoid unnecessary coercions), and still provide the required >> type >> information to transforms. > > I guess it could work within the features we have today. But: > Overloaded > functions would mean that call nodes would be pushed to the latter > stage, > while any kind of simple type inference would push local variable > types to > the latter stage, and so on. In general in seems to be a web that > must be > untangled with a keen eye on the exact functionality that we > provide in > the Cython language, and as that is subject to change, at least a > little, > those kind of designs make me rather careful... > > But I would really like something of the kind too. Perhaps (and I > think > this might have been Robert's idea) coercion and typing could be > split. > This requires more thinking but the end-result would be a more > flexible > and natural design. I.e. something like this: In the typing phase each > node declaratively specifies a *set* of possible types for input > (s), and > rules for resulting output types. Then a seperate pass is made which > follows simple rules to insert coercion nodes. Not sure if this set is finite, but perhaps it could somehow be represented. > I was pushing rather loudly for something along these lines back in > May, > and I remember that there were some concerns that Greg and Robert > raised > with this that put me off the idea eventually (but the concept > wasn't as > clear in my mind at the time as it seems to be in yours, so we may get > farther this time). Yes, for sure we need to split this up. I've been thinking something like - Type declared variables (this includes all "cdef T x" and builtins) - high-level transformations - [type inference on untyped variables] - lower-level transformations - type analysis and coercions - code generation One issue is that in the current code base, there is a lot of code that early on branches on whether or not the type is a python type or not. I think much of this can be differed to right before code generation. - Robert From robertwb at math.washington.edu Fri Dec 19 20:19:35 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 11:19:35 -0800 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: References: Message-ID: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> I'm all for suppressing warnings as long as you are sure they are not real ones, and it doesn't make the code too verbose. I assume it's a lot of changes like SomePyCAPI(%s) to SomePyCAPI((char*)%s) ? - Robert On Dec 19, 2008, at 9:19 AM, Lisandro Dalcin wrote: > I'm working on release petsc4py today. If core PETSc is compiled with > C++, then Cython-generated sources are also compiled with C++. > > When using Cython release 0.10.3, I can count about 260 warnings > like this: > > * In generated C sources > src/petsc4py_PETSc.c:82219: warning: deprecated conversion from string > constant to 'char*' > > * In geneted C-API headers: > src/include/petsc4py/petsc4py_PETSc_api.h:197: warning: deprecated > conversion from string constant to 'char*' > > > This is a complete mess, and of course the mess is not on Cython side, > but in Python 2.x headers. Python 3.x has improved a lot, though still > there are a few API calls that use bare "char*" instead of "const > char*". > > I want to ask all you if I could make the required changes to silence > all those warnings, even when this will lead to introduce explicit > casts to (char*). I know I'm too much pedantic about this stuff, but > please, please, let me fix this righ now (well, actually, tomorrow...) > > Thanks, > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Fri Dec 19 20:24:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 20:24:27 +0100 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> Message-ID: <494BF4EB.5050702@behnel.de> Robert Bradshaw wrote: > I'm all for suppressing warnings as long as you are sure they are not > real ones, and it doesn't make the code too verbose. I assume it's a > lot of changes like > > SomePyCAPI(%s) to SomePyCAPI((char*)%s) > > ? Then I'd prefer using a macro like #define CHARPTR (char*) so that we can redefine it to nothing in Py3 where it's not needed (if I understood correctly). Stefan From robertwb at math.washington.edu Fri Dec 19 20:26:59 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 11:26:59 -0800 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: <494BF4EB.5050702@behnel.de> References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> Message-ID: On Dec 19, 2008, at 11:24 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> I'm all for suppressing warnings as long as you are sure they are not >> real ones, and it doesn't make the code too verbose. I assume it's a >> lot of changes like >> >> SomePyCAPI(%s) to SomePyCAPI((char*)%s) >> >> ? > > Then I'd prefer using a macro like > > #define CHARPTR (char*) > > so that we can redefine it to nothing in Py3 where it's not needed > (if I > understood correctly). +1 to this idea. - Robert From jasone at canonware.com Fri Dec 19 20:40:30 2008 From: jasone at canonware.com (Jason Evans) Date: Fri, 19 Dec 2008 11:40:30 -0800 Subject: [Cython] Is trac.cython.org effective for communication? Message-ID: <494BF8AE.5030906@canonware.com> I've been using the trac system to report bugs, but is simply emailing this list the preferred approach for communicating issues? I can't assign tickets to likely owners, and it looks like reports just go unnoticed for as long as it takes for the right person to peruse the open tickets. Also, the one enhancement I've submitted thus far (#125) has been untouched for a month now. Do you guys do a sweep of such tickets before major releases? Thanks, Jason From stefan_ml at behnel.de Fri Dec 19 20:46:30 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 20:46:30 +0100 Subject: [Cython] separate 'initial typing' phase? In-Reply-To: References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> Message-ID: <494BFA16.2030409@behnel.de> Robert Bradshaw wrote: > - Type declared variables (this includes all "cdef T x" and builtins) > - high-level transformations That's where the major tree restructuring takes place, right? That's most of what is currently in Optimize.py, plus the with-statement. > - [type inference on untyped variables] > - lower-level transformations Would the constant folding go here as well? In the current state, it doesn't take much care about types (or even type casts), although it probably should do that for correctness. > - type analysis and coercions What would this step do with the already known or previously assigned types? I guess that's node specific, right? > - code generation > > One issue is that in the current code base, there is a lot of code > that early on branches on whether or not the type is a python type or > not. I think much of this can be differed to right before code > generation. I don't fear this that much. According to the main pipeline, we already have a lot of the code running before type analysis. The refactoring would be more about moving interesting stuff out of the type analysis phase and into the earlier steps. Stefan From robertwb at math.washington.edu Fri Dec 19 21:00:08 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 12:00:08 -0800 Subject: [Cython] Is trac.cython.org effective for communication? In-Reply-To: <494BF8AE.5030906@canonware.com> References: <494BF8AE.5030906@canonware.com> Message-ID: On Dec 19, 2008, at 11:40 AM, Jason Evans wrote: > I've been using the trac system to report bugs, but is simply emailing > this list the preferred approach for communicating issues? I can't > assign tickets to likely owners, and it looks like reports just go > unnoticed for as long as it takes for the right person to peruse the > open tickets. At this point, I think the best answer is both. The email list is actually read by people on a regular basis, and is good for getting feedback. However, putting stuff on the trac sever makes sure that it doesn't get forgotten. Thanks for your patches. I do want to note that it's preferable to use mercurial's export (rather than plain diffs) which has metadata like a description and authorship, but that's just a minor detail. > Also, the one enhancement I've submitted thus far (#125) has been > untouched for a month now. Do you guys do a sweep of such tickets > before major releases? Yes, I do. This doesn't mean I fix all the bugs before major releases--if they're not fixed I bump them to the next milestone--but that wouldn't be the case with this one. Note that there hasn't been a major release since that ticket was posted. I think that ticket is a good one, and should go in, but I haven't had time to try it out. (Lately, I've had too many higher-priority things (e.g. my studies) taking up my time to work on Cython as much as I would have liked.) There are also so many invasive (but very needed) changes going on in cython-devel that I don't have a timeline for when we'll be releasing next, other than bugfixes (within a month I'd hope). - Robert From robertwb at math.washington.edu Fri Dec 19 21:14:16 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 12:14:16 -0800 Subject: [Cython] separate 'initial typing' phase? In-Reply-To: <494BFA16.2030409@behnel.de> References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> <494BFA16.2030409@behnel.de> Message-ID: On Dec 19, 2008, at 11:46 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> - Type declared variables (this includes all "cdef T x" and builtins) >> - high-level transformations > > That's where the major tree restructuring takes place, right? > That's most > of what is currently in Optimize.py, plus the with-statement. Yep. >> - [type inference on untyped variables] >> - lower-level transformations > > Would the constant folding go here as well? In the current state, it > doesn't take much care about types (or even type casts), although it > probably should do that for correctness. Yeah. Typing constants would happen in the phase above, so it could probably go even higher. >> - type analysis and coercions > > What would this step do with the already known or previously assigned > types? I guess that's node specific, right? This would essentially be much of what analyse_types does now for non- name/attribute nodes. >> - code generation >> >> One issue is that in the current code base, there is a lot of code >> that early on branches on whether or not the type is a python type or >> not. I think much of this can be differed to right before code >> generation. > > I don't fear this that much. According to the main pipeline, we > already > have a lot of the code running before type analysis. The > refactoring would > be more about moving interesting stuff out of the type analysis > phase and > into the earlier steps. True. Let's try and hit a stable 0.11 before we attack this too much though. - Robert From dalcinl at gmail.com Fri Dec 19 21:27:56 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 19 Dec 2008 18:27:56 -0200 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: <494BF4EB.5050702@behnel.de> References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> Message-ID: On Fri, Dec 19, 2008 at 5:24 PM, Stefan Behnel wrote: > > Robert Bradshaw wrote: >> I'm all for suppressing warnings as long as you are sure they are not >> real ones, and it doesn't make the code too verbose. I assume it's a >> lot of changes like >> >> SomePyCAPI(%s) to SomePyCAPI((char*)%s) >> >> ? > > Then I'd prefer using a macro like > > #define CHARPTR (char*) > > so that we can redefine it to nothing in Py3 where it's not needed (if I > understood correctly). Yes, the only problem is that it should be better to use __Pyx_CHARP("abc") ... Let's avoid possible colisions in the preprocessor/compiler namespace.... In short, I'm thinking on implementing like this #if Py2 #define __Pyx_CHARP(p) ((char*)(p)) #else #define __Pyx_CHARP(p) (p) #endif What do you think? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Dec 19 21:35:28 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 21:35:28 +0100 Subject: [Cython] separate 'initial typing' phase? In-Reply-To: References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> <494BFA16.2030409@behnel.de> Message-ID: <494C0590.9080109@behnel.de> Robert Bradshaw wrote: > On Dec 19, 2008, at 11:46 AM, Stefan Behnel wrote: > >> Robert Bradshaw wrote: >>> - Type declared variables (this includes all "cdef T x" and builtins) >>> - high-level transformations >> That's where the major tree restructuring takes place, right? >> That's most >> of what is currently in Optimize.py, plus the with-statement. > > Yep. > >>> - [type inference on untyped variables] >>> - lower-level transformations >> Would the constant folding go here as well? In the current state, it >> doesn't take much care about types (or even type casts), although it >> probably should do that for correctness. > > Yeah. Typing constants would happen in the phase above, so it could > probably go even higher. It actually has to, now that you say it. It needs to be done right after type declaration and before the major tree transforms run, as some of them need access to pre-evaluated expression results (the step in the for-range transform, for instance). >>> - type analysis and coercions >> What would this step do with the already known or previously assigned >> types? I guess that's node specific, right? > > This would essentially be much of what analyse_types does now for non- > name/attribute nodes. I think so, too. >>> - code generation >>> >>> One issue is that in the current code base, there is a lot of code >>> that early on branches on whether or not the type is a python type or >>> not. I think much of this can be differed to right before code >>> generation. >> I don't fear this that much. According to the main pipeline, we >> already >> have a lot of the code running before type analysis. The >> refactoring would >> be more about moving interesting stuff out of the type analysis >> phase and >> into the earlier steps. > > True. Let's try and hit a stable 0.11 before we attack this too much > though. :) +1 The only major bug I currently see is that you can't short-cut out of the with-statement (return/break/continue). But that's been broken since the beginning and isn't trivial to fix. We should try to get some of the remaining 0.11 bugs fixed and do an alpha release soon. Stefan From stefan_ml at behnel.de Fri Dec 19 21:39:46 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 21:39:46 +0100 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> Message-ID: <494C0692.1040003@behnel.de> Lisandro Dalcin wrote: > On Fri, Dec 19, 2008 at 5:24 PM, Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> I'm all for suppressing warnings as long as you are sure they are not >>> real ones, and it doesn't make the code too verbose. I assume it's a >>> lot of changes like >>> >>> SomePyCAPI(%s) to SomePyCAPI((char*)%s) >>> >>> ? >> Then I'd prefer using a macro like >> >> #define CHARPTR (char*) >> >> so that we can redefine it to nothing in Py3 where it's not needed (if I >> understood correctly). > > Yes, the only problem is that it should be better to use > __Pyx_CHARP("abc") ... Let's avoid possible colisions in the > preprocessor/compiler namespace.... > > In short, I'm thinking on implementing like this > > #if Py2 > #define __Pyx_CHARP(p) ((char*)(p)) > #else > #define __Pyx_CHARP(p) (p) > #endif > > > What do you think? Fine with me. Whatever you take, the hardest part is to put it where it belongs. Changing the name later on is a simple search&replace. Stefan From dagss at student.matnat.uio.no Fri Dec 19 21:47:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 19 Dec 2008 21:47:00 +0100 (CET) Subject: [Cython] separate 'initial typing' phase? In-Reply-To: <494C0590.9080109@behnel.de> References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> <494BFA16.2030409@behnel.de> <494C0590.9080109@behnel.de> Message-ID: Stefan Behnel wrote: > > Robert Bradshaw wrote: >> On Dec 19, 2008, at 11:46 AM, Stefan Behnel wrote: >> >>> Robert Bradshaw wrote: >>>> - Type declared variables (this includes all "cdef T x" and builtins) >>>> - high-level transformations >>> That's where the major tree restructuring takes place, right? >>> That's most >>> of what is currently in Optimize.py, plus the with-statement. >> >> Yep. >> >>>> - [type inference on untyped variables] >>>> - lower-level transformations >>> Would the constant folding go here as well? In the current state, it >>> doesn't take much care about types (or even type casts), although it >>> probably should do that for correctness. >> >> Yeah. Typing constants would happen in the phase above, so it could >> probably go even higher. > > It actually has to, now that you say it. It needs to be done right after > type declaration and before the major tree transforms run, as some of them > need access to pre-evaluated expression results (the step in the for-range > transform, for instance). > > >>>> - type analysis and coercions >>> What would this step do with the already known or previously assigned >>> types? I guess that's node specific, right? >> >> This would essentially be much of what analyse_types does now for non- >> name/attribute nodes. > > I think so, too. > > >>>> - code generation >>>> >>>> One issue is that in the current code base, there is a lot of code >>>> that early on branches on whether or not the type is a python type or >>>> not. I think much of this can be differed to right before code >>>> generation. >>> I don't fear this that much. According to the main pipeline, we >>> already >>> have a lot of the code running before type analysis. The >>> refactoring would >>> be more about moving interesting stuff out of the type analysis >>> phase and >>> into the earlier steps. >> >> True. Let's try and hit a stable 0.11 before we attack this too much >> though. > > :) +1 > > The only major bug I currently see is that you can't short-cut out of the > with-statement (return/break/continue). But that's been broken since the > beginning and isn't trivial to fix. > > We should try to get some of the remaining 0.11 bugs fixed and do an alpha > release soon. So much temp migration has been going on already -- wouldn't it make sense to migrate *all* temps and remove the old system before releasing? I may, or may not, have time for doing the refcount nanny and some more migration during the next few days. Dag Sverre From stefan_ml at behnel.de Fri Dec 19 21:48:45 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 21:48:45 +0100 Subject: [Cython] separate 'initial typing' phase? In-Reply-To: <494C0590.9080109@behnel.de> References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> <494BFA16.2030409@behnel.de> <494C0590.9080109@behnel.de> Message-ID: <494C08AD.8080508@behnel.de> Stefan Behnel wrote: > Robert Bradshaw wrote: >> On Dec 19, 2008, at 11:46 AM, Stefan Behnel wrote: >>> Robert Bradshaw wrote: >>>> - Type declared variables (this includes all "cdef T x" and builtins) >>>> - high-level transformations >>>> - [type inference on untyped variables] >>>> - lower-level transformations >>> Would the constant folding go here as well? In the current state, it >>> doesn't take much care about types (or even type casts), although it >>> probably should do that for correctness. >> Yeah. Typing constants would happen in the phase above, so it could >> probably go even higher. > > It actually has to, now that you say it. It needs to be done right after > type declaration and before the major tree transforms run, as some of them > need access to pre-evaluated expression results (the step in the for-range > transform, for instance). Although this might have implications. When a transform changes a subtree, and a later transform reads the constant expression of its parent... I doubt that there will be many cases where we actually change the structure of a subtree that was already found constant, though, especially with tree modifications that change the constant value. So maybe this is a non-issue right from the start. Stefan From dalcinl at gmail.com Fri Dec 19 21:49:34 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 19 Dec 2008 18:49:34 -0200 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: <494C0692.1040003@behnel.de> References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> <494C0692.1040003@behnel.de> Message-ID: On Fri, Dec 19, 2008 at 6:39 PM, Stefan Behnel wrote: >> >> #if Py2 >> #define __Pyx_CHARP(p) ((char*)(p)) >> #else >> #define __Pyx_CHARP(p) (p) >> #endif >> >> >> What do you think? > > Fine with me. Whatever you take, the hardest part is to put it where it > belongs. Changing the name later on is a simple search&replace. > OK. Some time ago I've already pushed some fixes, but by using an explicit (char*) cast. Now I'll revert that to use the __Pyx_CHARP macro.... BTW, Python 2.5 is better about those problems than 2.4, with in turns is better than 2.3 .... Python 2.3 is probably near to be unsupported for some very popular packages, like numpy (funny enough, last time I've tried numpy 1.2.1 did not work on 2.3 because of a generator)... so the question is: Could I target all those fixes for 2.4 and above? If not, I'll have to put the casting macro at ever more places... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Dec 19 21:52:41 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 21:52:41 +0100 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> <494C0692.1040003@behnel.de> Message-ID: <494C0999.7050402@behnel.de> Lisandro Dalcin wrote: > On Fri, Dec 19, 2008 at 6:39 PM, Stefan Behnel wrote: >>> #if Py2 >>> #define __Pyx_CHARP(p) ((char*)(p)) >>> #else >>> #define __Pyx_CHARP(p) (p) >>> #endif >>> >>> What do you think? >> Fine with me. Whatever you take, the hardest part is to put it where it >> belongs. Changing the name later on is a simple search&replace. > > OK. Some time ago I've already pushed some fixes, but by using an > explicit (char*) cast. Now I'll revert that to use the __Pyx_CHARP > macro.... > > BTW, Python 2.5 is better about those problems than 2.4, with in turns > is better than 2.3 .... Python 2.3 is probably near to be unsupported > for some very popular packages, like numpy (funny enough, last time > I've tried numpy 1.2.1 did not work on 2.3 because of a generator)... > so the question is: Could I target all those fixes for 2.4 and above? > If not, I'll have to put the casting macro at ever more places... +1 These are warnings, there's nothing broken in Cython. Stefan From stefan_ml at behnel.de Fri Dec 19 21:54:03 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 21:54:03 +0100 Subject: [Cython] separate 'initial typing' phase? In-Reply-To: References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> <494BFA16.2030409@behnel.de> <494C0590.9080109@behnel.de> Message-ID: <494C09EB.10206@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> We should try to get some of the remaining 0.11 bugs fixed and do an alpha >> release soon. > > So much temp migration has been going on already -- wouldn't it make sense > to migrate *all* temps and remove the old system before releasing? That would be nice. But there are still major obstacles (such as NameNode), and I don't see it as a requirement. > I may, or may not, have time for doing the refcount nanny and some more > migration during the next few days. Great. I already changed the trac milestone of the refcount nanny to 'wishlist', but if we can get it in now that would be very helpful. Stefan From dagss at student.matnat.uio.no Fri Dec 19 21:55:46 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 19 Dec 2008 21:55:46 +0100 (CET) Subject: [Cython] Best way to link in Cython utility code? Message-ID: For the refcount nanny, I have a file refcheck.pyx which exports several functions which is called from Cython-generated C code if a flag is set, to do the reference nannying (basically they are "filters" for Py_...REF calls, called (__Pyx_RefcheckerIncRef and so on). Currently, I hack this to work by manually including "cimport refcheck" in my (single) testcase, and declare the functions as "api", leaving it to the Cython-specific protocol to do the "linking"/pointer retrieval. This give me what I need for developing it, but no more... I could build on this and use the __pyx_capi protocol under the hood etc., but it would be kind of nicer to do the linking at the C level (especially as one would like to refcount the module importing etc. as well) -- so that my functions (__Pyx_RefcheckerIncRef etc.) was linked in the same way modules links to the C/Python API. Is this easy/possible? Starting the Python interpreter in some special way would be acceptable in this case. I'm afraid this question may demonstrate my ignorance of C linkage :-) Dag Sverre From stefan_ml at behnel.de Fri Dec 19 22:02:08 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 22:02:08 +0100 Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: References: Message-ID: <494C0BD0.3040400@behnel.de> Dag Sverre Seljebotn wrote: > For the refcount nanny, I have a file refcheck.pyx which exports several > functions which is called from Cython-generated C code if a flag is set, > to do the reference nannying (basically they are "filters" for Py_...REF > calls, called (__Pyx_RefcheckerIncRef and so on). > > Currently, I hack this to work by manually including "cimport refcheck" in > my (single) testcase, and declare the functions as "api", leaving it to > the Cython-specific protocol to do the "linking"/pointer retrieval. This > give me what I need for developing it, but no more... I'd try to use the .pxi include mechanism (somehow) from within Cython, to get it into the same file. Stefan From michael.abshoff at googlemail.com Fri Dec 19 22:08:20 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Fri, 19 Dec 2008 13:08:20 -0800 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> <494C0692.1040003@behnel.de> Message-ID: <494C0D44.8010102@gmail.com> Lisandro Dalcin wrote: > On Fri, Dec 19, 2008 at 6:39 PM, Stefan Behnel wrote: Hi Lisandro, > BTW, Python 2.5 is better about those problems than 2.4, with in turns > is better than 2.3 .... Python 2.3 is probably near to be unsupported > for some very popular packages, like numpy (funny enough, last time > I've tried numpy 1.2.1 did not work on 2.3 because of a generator)... > so the question is: Could I target all those fixes for 2.4 and above? > If not, I'll have to put the casting macro at ever more places... numpy still supports Python 2.3 and probably will so in the future. The bug you mentioned has been fixed IIRC and ought to be in 1.2.2 and 1.3. By the way: OSX 10.4 still ships Python 2.3.5 per default: varro:~ mabshoff$ uname -a Darwin varro 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc varro:~ mabshoff$ which python /usr/bin/python varro:~ mabshoff$ python Python 2.3.5 (#1, Nov 26 2007, 09:23:30) [GCC 3.3 20030304 (Apple Computer, Inc. build 1819)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cheers, Michael From stefan_ml at behnel.de Fri Dec 19 22:09:19 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 22:09:19 +0100 Subject: [Cython] Is trac.cython.org effective for communication? In-Reply-To: References: <494BF8AE.5030906@canonware.com> Message-ID: <494C0D7F.8090605@behnel.de> Robert Bradshaw wrote: > On Dec 19, 2008, at 11:40 AM, Jason Evans wrote: >> Also, the one enhancement I've submitted thus far (#125) has been >> untouched for a month now. Do you guys do a sweep of such tickets >> before major releases? > > Yes, I do. This doesn't mean I fix all the bugs before major > releases--if they're not fixed I bump them to the next milestone--but > that wouldn't be the case with this one. Note that there hasn't been > a major release since that ticket was posted. > > I think that ticket is a good one, and should go in, but I haven't > had time to try it out. I think that's the main point. I had looked at it before and this /looks/ trivial and might even make a bug fix for 0.10, but changing module names may break things in a very subtle way, so it can't go in untested. Stefan From stefan_ml at behnel.de Fri Dec 19 22:14:52 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Dec 2008 22:14:52 +0100 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: <494C0D44.8010102@gmail.com> References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> <494C0692.1040003@behnel.de> <494C0D44.8010102@gmail.com> Message-ID: <494C0ECC.7090203@behnel.de> Michael Abshoff wrote: > By the way: OSX 10.4 still ships Python 2.3.5 per default: > > varro:~ mabshoff$ uname -a > Darwin varro 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 > PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc > varro:~ mabshoff$ which python > /usr/bin/python > varro:~ mabshoff$ python > Python 2.3.5 (#1, Nov 26 2007, 09:23:30) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1819)] on darwin > Type "help", "copyright", "credits" or "license" for more information. Apple has shipped outdated system libraries for ages. I have this problem with libxml2 and libxslt which are just plain buggy on every MacOS system out there. BTW, with the latest non-release of a Python 2.3.8 (which would have fixed a couple of security problems), it's clear that 2.3 is now officially taken out of maintenance, too. Stefan From dagss at student.matnat.uio.no Fri Dec 19 22:42:04 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 19 Dec 2008 22:42:04 +0100 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: <494C0D44.8010102@gmail.com> References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> <494C0692.1040003@behnel.de> <494C0D44.8010102@gmail.com> Message-ID: <494C152C.7010808@student.matnat.uio.no> Michael Abshoff wrote: > Lisandro Dalcin wrote: >> On Fri, Dec 19, 2008 at 6:39 PM, Stefan Behnel wrote: > > Hi Lisandro, > >> BTW, Python 2.5 is better about those problems than 2.4, with in turns >> is better than 2.3 .... Python 2.3 is probably near to be unsupported >> for some very popular packages, like numpy (funny enough, last time >> I've tried numpy 1.2.1 did not work on 2.3 because of a generator)... >> so the question is: Could I target all those fixes for 2.4 and above? >> If not, I'll have to put the casting macro at ever more places... > > numpy still supports Python 2.3 and probably will so in the future. The > bug you mentioned has been fixed IIRC and ought to be in 1.2.2 and 1.3. Does this mean that this (from the 1.2.1 release notes) has changed? "Please note that NumPy 1.2.1 requires Python 2.4 or greater." Dag Sverre From robertwb at math.washington.edu Fri Dec 19 22:37:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 13:37:07 -0800 Subject: [Cython] separate 'initial typing' phase? In-Reply-To: <494C09EB.10206@behnel.de> References: <494BC68D.3090103@behnel.de> <16fe64d252dc8955a9329131d463646a.squirrel@webmail.uio.no> <494BFA16.2030409@behnel.de> <494C0590.9080109@behnel.de> <494C09EB.10206@behnel.de> Message-ID: <8CD3EA0C-B65C-4A51-B209-335C8526612C@math.washington.edu> On Dec 19, 2008, at 12:54 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> We should try to get some of the remaining 0.11 bugs fixed and do >>> an alpha >>> release soon. >> >> So much temp migration has been going on already -- wouldn't it >> make sense >> to migrate *all* temps and remove the old system before releasing? > > That would be nice. But there are still major obstacles (such as > NameNode), > and I don't see it as a requirement. I was going to say "that depends on how close we are." But it sounds like there's still a lot of work to do, so I'd vote for trying to get something stable sooner. >> I may, or may not, have time for doing the refcount nanny and some >> more >> migration during the next few days. > > Great. I already changed the trac milestone of the refcount nanny to > 'wishlist', but if we can get it in now that would be very helpful. One can "easily" make segfautls away by adding increfs around, but the other direction will be harder to verify, so I would really like to see something like this. - Robert From michael.abshoff at googlemail.com Fri Dec 19 22:43:04 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Fri, 19 Dec 2008 13:43:04 -0800 Subject: [Cython] removing ALL those nasty warnings about string constants In-Reply-To: <494C152C.7010808@student.matnat.uio.no> References: <0ADC31F8-188B-4DA8-9621-DF673047FB83@math.washington.edu> <494BF4EB.5050702@behnel.de> <494C0692.1040003@behnel.de> <494C0D44.8010102@gmail.com> <494C152C.7010808@student.matnat.uio.no> Message-ID: <494C1568.1070905@gmail.com> Dag Sverre Seljebotn wrote: > Michael Abshoff wrote: >> Lisandro Dalcin wrote: >>> On Fri, Dec 19, 2008 at 6:39 PM, Stefan Behnel wrote: >> Hi Lisandro, Hi, >>> BTW, Python 2.5 is better about those problems than 2.4, with in turns >>> is better than 2.3 .... Python 2.3 is probably near to be unsupported >>> for some very popular packages, like numpy (funny enough, last time >>> I've tried numpy 1.2.1 did not work on 2.3 because of a generator)... >>> so the question is: Could I target all those fixes for 2.4 and above? >>> If not, I'll have to put the casting macro at ever more places... >> numpy still supports Python 2.3 and probably will so in the future. The >> bug you mentioned has been fixed IIRC and ought to be in 1.2.2 and 1.3. > > Does this mean that this (from the 1.2.1 release notes) has changed? Sorry, I was confused. > "Please note that NumPy 1.2.1 requires Python 2.4 or greater." I just checked and you are correct: "If you're right (the operation not being possible under python 2.3), then this is a bug. Numpy 1.1.* still supports python 2.3. Starting from 1.2, we will only support 2.4 (and above)." > Dag Sverre Cheers, Michael > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From jasone at canonware.com Sat Dec 20 02:08:00 2008 From: jasone at canonware.com (Jason Evans) Date: Fri, 19 Dec 2008 17:08:00 -0800 Subject: [Cython] Is trac.cython.org effective for communication? In-Reply-To: <494C0D7F.8090605@behnel.de> References: <494BF8AE.5030906@canonware.com> <494C0D7F.8090605@behnel.de> Message-ID: <494C4570.80406@canonware.com> Stefan Behnel wrote: > Robert Bradshaw wrote: >> On Dec 19, 2008, at 11:40 AM, Jason Evans wrote: >>> [trac #125] >> >> I think that ticket is a good one, and should go in, but I haven't >> had time to try it out. > > I think that's the main point. I had looked at it before and this /looks/ > trivial and might even make a bug fix for 0.10, but changing module names > may break things in a very subtle way, so it can't go in untested. I've been using the patch to build Crux (http://canonware.com/cgi-bin/hg_crux), and it's working as intended. I originally developed the patch by diffing Cython output with/without the patch, and the patch changes precisely the right things in the output, as near as I can tell. If there's anything else I can do to make the change less daunting, please let me know. Thanks, Jason From hoytak at cs.ubc.ca Sat Dec 20 02:29:55 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Fri, 19 Dec 2008 17:29:55 -0800 Subject: [Cython] FYI: current cython parser module has compile error In-Reply-To: <494BF322.3080702@behnel.de> References: <4db580fd0812190825n1b5119f1o83c73173a7ca7b4e@mail.gmail.com> <494BD6D9.5020007@behnel.de> <4db580fd0812191101m5fd496c1qbbbc759d50090af3@mail.gmail.com> <494BF198.8080202@behnel.de> <494BF322.3080702@behnel.de> Message-ID: <4db580fd0812191729j23d82eaci70122ef5edbff490@mail.gmail.com> Works fine now, thanks! --Hoyt On Fri, Dec 19, 2008 at 11:16 AM, Stefan Behnel wrote: > > Stefan Behnel wrote: >> Hoyt Koepke wrote: >>> Thanks! It builds fine now. >>> However, cython now segfaults when compiling my code >> >> Same here, I just noticed that. I'm looking into it. > > Fixed. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From robertwb at math.washington.edu Sat Dec 20 02:48:21 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 17:48:21 -0800 Subject: [Cython] Is trac.cython.org effective for communication? In-Reply-To: <494C4570.80406@canonware.com> References: <494BF8AE.5030906@canonware.com> <494C0D7F.8090605@behnel.de> <494C4570.80406@canonware.com> Message-ID: <69287D3E-3073-4C04-AE46-9C2992FDBFB3@math.washington.edu> On Dec 19, 2008, at 5:08 PM, Jason Evans wrote: > Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> On Dec 19, 2008, at 11:40 AM, Jason Evans wrote: >>>> [trac #125] >>> >>> I think that ticket is a good one, and should go in, but I haven't >>> had time to try it out. >> >> I think that's the main point. I had looked at it before and this / >> looks/ >> trivial and might even make a bug fix for 0.10, but changing >> module names >> may break things in a very subtle way, so it can't go in untested. > > I've been using the patch to build Crux > (http://canonware.com/cgi-bin/hg_crux), and it's working as > intended. I > originally developed the patch by diffing Cython output with/ > without the > patch, and the patch changes precisely the right things in the output, > as near as I can tell. > > If there's anything else I can do to make the change less daunting, > please let me know. It's a pretty easy patch, the only hard part is making a set of files to test it with (which, honestly, wasn't any harder than writing this email, I just had to sit down and do it). Works great for me. I wasn't able to easily add this to the testing framework though. - Robert From robertwb at math.washington.edu Sat Dec 20 06:57:17 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 19 Dec 2008 21:57:17 -0800 Subject: [Cython] cython-devel-py3 branch Message-ID: <3D636753-C4C9-4627-92AA-D8F2E6D1E646@math.washington.edu> It looks like http://hg.cython.org/cython-devel-py3/ served its purpose, and everything is now merged back into the main line. Any objections to killing this repository online? - Robert From stefan_ml at behnel.de Sat Dec 20 11:07:56 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Dec 2008 11:07:56 +0100 Subject: [Cython] cython-devel-py3 branch In-Reply-To: <3D636753-C4C9-4627-92AA-D8F2E6D1E646@math.washington.edu> References: <3D636753-C4C9-4627-92AA-D8F2E6D1E646@math.washington.edu> Message-ID: <494CC3FC.80402@behnel.de> Robert Bradshaw wrote: > It looks like http://hg.cython.org/cython-devel-py3/ served its > purpose, and everything is now merged back into the main line. Any > objections to killing this repository online? Nope, time to take it down. Stefan From stefan_ml at behnel.de Sat Dec 20 11:19:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Dec 2008 11:19:57 +0100 Subject: [Cython] Is trac.cython.org effective for communication? In-Reply-To: <69287D3E-3073-4C04-AE46-9C2992FDBFB3@math.washington.edu> References: <494BF8AE.5030906@canonware.com> <494C0D7F.8090605@behnel.de> <494C4570.80406@canonware.com> <69287D3E-3073-4C04-AE46-9C2992FDBFB3@math.washington.edu> Message-ID: <494CC6CD.1050803@behnel.de> Robert Bradshaw wrote: > It's a pretty easy patch, the only hard part is making a set of files > to test it with (which, honestly, wasn't any harder than writing this > email, I just had to sit down and do it). Works great for me. I > wasn't able to easily add this to the testing framework though. Yes, the test runner is pretty much made to test one file at a time, so it's not obvious how to test packages or imports. We could let it recognise test packages in addition to .pyx files, and it could compile all modules in the package (including subpackages recursively) and then run a file "test.pyx" in that package. That would allow us to test imports as well as package behaviour. Stefan From dagss at student.matnat.uio.no Sat Dec 20 12:25:09 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 20 Dec 2008 12:25:09 +0100 Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: <494C0BD0.3040400@behnel.de> References: <494C0BD0.3040400@behnel.de> Message-ID: <494CD615.7080204@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> For the refcount nanny, I have a file refcheck.pyx which exports several >> functions which is called from Cython-generated C code if a flag is set, >> to do the reference nannying (basically they are "filters" for Py_...REF >> calls, called (__Pyx_RefcheckerIncRef and so on). >> >> Currently, I hack this to work by manually including "cimport refcheck" in >> my (single) testcase, and declare the functions as "api", leaving it to >> the Cython-specific protocol to do the "linking"/pointer retrieval. This >> give me what I need for developing it, but no more... > > I'd try to use the .pxi include mechanism (somehow) from within Cython, to > get it into the same file. Could be done. However there's a problem: I wouldn't like the refcont nanny itself to be refcount nannied, to avoid infinite loops. I suppose I can put in locks to (rather easily) overcome this, still compiling the refcount nanny in a seperate stage with different flags and linking it in seems rather more elegant *shrug*. -- Dag Sverre From dalcinl at gmail.com Sat Dec 20 16:33:19 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 20 Dec 2008 12:33:19 -0300 Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: <494CD615.7080204@student.matnat.uio.no> References: <494C0BD0.3040400@behnel.de> <494CD615.7080204@student.matnat.uio.no> Message-ID: Dag, sorry about my complete ignorance. This refcounting nanny is supposed to be a tool for Cython developement and testing, or rather a general mechanism for Cython-generated extension modules? On Sat, Dec 20, 2008 at 8:25 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> For the refcount nanny, I have a file refcheck.pyx which exports several >>> functions which is called from Cython-generated C code if a flag is set, >>> to do the reference nannying (basically they are "filters" for Py_...REF >>> calls, called (__Pyx_RefcheckerIncRef and so on). >>> >>> Currently, I hack this to work by manually including "cimport refcheck" in >>> my (single) testcase, and declare the functions as "api", leaving it to >>> the Cython-specific protocol to do the "linking"/pointer retrieval. This >>> give me what I need for developing it, but no more... >> >> I'd try to use the .pxi include mechanism (somehow) from within Cython, to >> get it into the same file. > > Could be done. However there's a problem: I wouldn't like the refcont > nanny itself to be refcount nannied, to avoid infinite loops. > > I suppose I can put in locks to (rather easily) overcome this, still > compiling the refcount nanny in a seperate stage with different flags > and linking it in seems rather more elegant *shrug*. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Sat Dec 20 17:53:33 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Dec 2008 17:53:33 +0100 Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: References: <494C0BD0.3040400@behnel.de> <494CD615.7080204@student.matnat.uio.no> Message-ID: <494D230D.2010602@behnel.de> Hi, Lisandro Dalcin wrote: > Dag, sorry about my complete ignorance. This refcounting nanny is > supposed to be a tool for Cython developement and testing, or rather a > general mechanism for Cython-generated extension modules? It's meant for testing the code Cython generates and making sure it gets ref-counting right when doing things like assigning Python objects back and forth between temp variables or (I assume) jumping into exception handlers or returning from the middle of a function. When Cython gets this right, there shouldn't be any need for such a functionality on the user side (read: no performance impact or the like). Stefan From dalcinl at gmail.com Sat Dec 20 19:21:24 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 20 Dec 2008 15:21:24 -0300 Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: <494D230D.2010602@behnel.de> References: <494C0BD0.3040400@behnel.de> <494CD615.7080204@student.matnat.uio.no> <494D230D.2010602@behnel.de> Message-ID: Then It would be really easy to implement some ctypes hackery to make globally visible the API required for nanny refcounting. The only gotcha is that you would need to "install" this feature before importing Cython-generated ext modules, I mean, something similar to pyximport, where you have first to import andinstall() it for make use of their features... But a really nice thing about this approach is that you can run in with an unmodified Python interpreter, and there is no need to link extension modules with especial shared libs or other modules... On Sat, Dec 20, 2008 at 1:53 PM, Stefan Behnel wrote: > Hi, > > Lisandro Dalcin wrote: >> Dag, sorry about my complete ignorance. This refcounting nanny is >> supposed to be a tool for Cython developement and testing, or rather a >> general mechanism for Cython-generated extension modules? > > It's meant for testing the code Cython generates and making sure it gets > ref-counting right when doing things like assigning Python objects back and > forth between temp variables or (I assume) jumping into exception handlers > or returning from the middle of a function. > > When Cython gets this right, there shouldn't be any need for such a > functionality on the user side (read: no performance impact or the like). > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Dec 20 19:32:30 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 20 Dec 2008 10:32:30 -0800 Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: References: <494C0BD0.3040400@behnel.de> <494CD615.7080204@student.matnat.uio.no> <494D230D.2010602@behnel.de> Message-ID: On Dec 20, 2008, at 10:21 AM, Lisandro Dalcin wrote: > Then It would be really easy to implement some ctypes hackery to make > globally visible the API required for nanny refcounting. This won't work for the trickiest part, all the temp variables and exceptions that need to be handled correctly. On the other hand, the external variables can't always be refcounted automatically, because the semantics may dictate that the refcounts increase or decrease (e.g. inserting into a list). > The only gotcha is that you would need to "install" this feature > before importing Cython-generated ext modules, I mean, something > similar to pyximport, where you have first to import andinstall() it > for make use of their features... > > But a really nice thing about this approach is that you can run in > with an unmodified Python interpreter, and there is no need to link > extension modules with especial shared libs or other modules... > > > > On Sat, Dec 20, 2008 at 1:53 PM, Stefan Behnel > wrote: >> Hi, >> >> Lisandro Dalcin wrote: >>> Dag, sorry about my complete ignorance. This refcounting nanny is >>> supposed to be a tool for Cython developement and testing, or >>> rather a >>> general mechanism for Cython-generated extension modules? >> >> It's meant for testing the code Cython generates and making sure >> it gets >> ref-counting right when doing things like assigning Python objects >> back and >> forth between temp variables or (I assume) jumping into exception >> handlers >> or returning from the middle of a function. >> >> When Cython gets this right, there shouldn't be any need for such a >> functionality on the user side (read: no performance impact or the >> like). >> >> Stefan >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Sat Dec 20 19:45:34 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 20 Dec 2008 19:45:34 +0100 (CET) Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: References: <494C0BD0.3040400@behnel.de> <494CD615.7080204@student.matnat.uio.no> <494D230D.2010602@behnel.de> Message-ID: Lisandro Dalcin wrote: > On Dec 20, 2008, at 10:21 AM, Lisandro Dalcin wrote: > >> Then It would be really easy to implement some ctypes hackery to make >> globally visible the API required for nanny refcounting. > > This won't work for the trickiest part, all the temp variables and > exceptions that need to be handled correctly. On the other hand, the > external variables can't always be refcounted automatically, because > the semantics may dictate that the refcounts increase or decrease > (e.g. inserting into a list). If I understand correctly, Lisandro simply recommends loading the library loading my refchecker.so utility code (basically a twenty-liner raising the right diagnostics at the right times) using ctypes with the RTLD_GLOBAL flag (which I didn't know about until I just now read the ctypes docs). Over to something else: Does anyone have an opinion on a choice between these two: - I change Cython to always generate __Pyx_INCREF/DECREF/GOTREF/GIVEREF. Whether to install the refcount nanny or not only affects the definition of these macros (the latter two being noops when the nanny is not in operation). I'm +1 on this one. - If the refcount nanny is not installed, generate direct Py_INCREF/DECREF and do not generate GOTREF/GIVEREF. Dag Sverre From dalcinl at gmail.com Sat Dec 20 22:47:29 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 20 Dec 2008 18:47:29 -0300 Subject: [Cython] Best way to link in Cython utility code? In-Reply-To: References: <494C0BD0.3040400@behnel.de> <494CD615.7080204@student.matnat.uio.no> <494D230D.2010602@behnel.de> Message-ID: On Sat, Dec 20, 2008 at 3:45 PM, Dag Sverre Seljebotn > > If I understand correctly, Lisandro simply recommends loading the library > loading my refchecker.so utility code (basically a twenty-liner raising > the right diagnostics at the right times) using ctypes with the > RTLD_GLOBAL flag (which I didn't know about until I just now read the > ctypes docs). That's exactly what I was talking about... Sorry I was too lazy to explain it first time.. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Dec 22 13:12:48 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 22 Dec 2008 09:12:48 -0300 Subject: [Cython] suspicious (but correct) code generation Message-ID: While working on my patch for suppressing compiler warnings, I've noticed this: /* "/u/dalcinl/Devel/cython-devel/tests/compile/del.pyx":3 * def f(a, b): * global g * del g # <<<<<<<<<<<<<< * del a[b] * del a[b][42] */ if (PyObject_DelAttrString(__pyx_m, "g") < 0) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 3; __pyx_clineno = __LINE__; goto __pyx_L1_error;} I would expect this instead: PyObject_DelAttr(__pyx_m, __pyx_kp_g) Not sure how to fix this, though I guess that would be trivial... Could someone or you give a try? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Dec 22 14:29:49 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 22 Dec 2008 14:29:49 +0100 (CET) Subject: [Cython] suspicious (but correct) code generation In-Reply-To: References: Message-ID: <77c3bdd3f06b90bdf5122bf9eb4f017f.squirrel@webmail.uio.no> Lisandro Dalcin wrote: > While working on my patch for suppressing compiler warnings, I've noticed > this: > > > /* "/u/dalcinl/Devel/cython-devel/tests/compile/del.pyx":3 > * def f(a, b): > * global g > * del g # <<<<<<<<<<<<<< > * del a[b] > * del a[b][42] > */ > if (PyObject_DelAttrString(__pyx_m, "g") < 0) {__pyx_filename = > __pyx_f[0]; __pyx_lineno = 3; __pyx_clineno = __LINE__; goto > __pyx_L1_error;} > > > I would expect this instead: > > PyObject_DelAttr(__pyx_m, __pyx_kp_g) If it works (which it should) currently I'd leave it for now, because this must anyway be redone when ticket 144 is fixed: http://trac.cython.org/cython_trac/ticket/144 No point in doing work twice. Dag Sverre From dalcinl at gmail.com Mon Dec 22 22:47:22 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 22 Dec 2008 19:47:22 -0200 Subject: [Cython] problem with PySequenceMethods in Py3k Message-ID: While running the testsuite, I got this.... compiling (c) extsetslice ... ok extsetslice.cpp:316: error: invalid conversion from 'int (*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)' to 'void*' I bet you never noticed the warning, just because the C++ compilation is usually contaminated with the warning about -Wstrict-prototypes not being a valid flag in C++ ;-) ... Can you see why I hate so much to have warnings that we can avoid... At some point, I'll try to hot-fix that stupid -Wstrict-prototypes for the C++ case... Now the actual problem... Apparently, in "PySequenceMethods", the "sq_slice" and "sq_ass_slice" slots changed to be typed "void*" and their names prefixed with "was_". I bet this change intended to provide backward compatibility in the struct layout. IMHO, this was a really bad idea, people porting extension modules to Py3k will not notice the change unless they give a try with C++. Those "was_sq_[ass_]slice" are just ignored in Py3k, and then I assume the Cython-generated code could not work as expected... What do you think? Unfortunately, I'll not be able to work for fixing this... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Dec 23 08:06:43 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 23 Dec 2008 08:06:43 +0100 Subject: [Cython] problem with PySequenceMethods in Py3k In-Reply-To: References: Message-ID: <49508E03.9090604@behnel.de> Hi Lisandro, Lisandro Dalcin wrote: > compiling (c) extsetslice ... ok > extsetslice.cpp:316: error: invalid conversion from 'int > (*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)' to 'void*' > > Apparently, in "PySequenceMethods", the "sq_slice" and "sq_ass_slice" > slots changed to be typed "void*" and their names prefixed with > "was_". I bet this change intended to provide backward compatibility > in the struct layout. IMHO, this was a really bad idea, people porting > extension modules to Py3k will not notice the change unless they give > a try with C++. Yes, I'm pretty sure that slipped through. There was a point during the development of Py3k up to which they intended to keep the C-API compatible, and afterwards they turned to really fixing it up for the future language. Looks like this change went in before that and didn't get fixed later on. > Those "was_sq_[ass_]slice" are just ignored in Py3k, and then I assume > the Cython-generated code could not work as expected... I'm fine with a warning from Cython that this will no longer work in Py3, and a preprocessor #if that puts 0 into the method struct in that case. We should just disable that test case for Py3. The 'new' slicing API works in all Python versions that Cython supports, so old code should be fixed instead of trying to find a work-around in Cython. A warning will help people to do the right thing. > I bet you never noticed the warning, just because the C++ compilation > is usually contaminated with the warning about -Wstrict-prototypes not > being a valid flag in C++ ;-) ... Can you see why I hate so much to > have warnings that we can avoid... At some point, I'll try to hot-fix > that stupid -Wstrict-prototypes for the C++ case... Yes, I think everybody noticed this by now. :) It's because Python itself was built with that option, and it passes its CFLAGS on to the extensions. I never got around to looking for a fix either. I'd say this needs fixing in distutils. Another problem with the C++ tests is that the doctests do not run on the C++ compiled modules, as the C compiled modules with the same name were already imported. But that's a minor issue IMHO, as the compilation is the thing that matters here. Stefan From ondrej at certik.cz Tue Dec 23 10:56:47 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 23 Dec 2008 10:56:47 +0100 Subject: [Cython] calling gmp automatically Message-ID: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> Hi, currently Cython converts all "ints" as int32 in C (right?). How difficult would be to improve Cython, so that it can automatically convert all ints to gmp calls, so that one can use arbitrary integers? Or to put the question differently, obviously, for most of my codes I just need int32 and enjoy the speedups. However, sometimes it's handy to use arbitrary ints, and since in Python both of it is just "int" (resp. long in py2.6) --- what is the best way to have both? Of course, one can call let's say gmpy directly from Python or Cython and then all would be fine. But I think it would be easier if one can just use Python integers and be done with it. And then just (optionally) run it through Cython (let's say in pure Python mode) and enjoy the speedups for free. Ondrej From stefan_ml at behnel.de Tue Dec 23 13:53:36 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 23 Dec 2008 13:53:36 +0100 (CET) Subject: [Cython] calling gmp automatically In-Reply-To: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> References: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> Message-ID: <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi, Ondrej Certik wrote: > currently Cython converts all "ints" as int32 in C (right?). Depends on what you mean with "ints". Python ints are Python objects. Only things defined "cdef int" will become C ints. > How difficult would be to improve Cython, so that it can automatically > convert all ints to gmp calls, so that one can use arbitrary integers? > > Or to put the question differently, obviously, for most of my codes I > just need int32 and enjoy the speedups. However, sometimes it's handy > to use arbitrary ints, and since in Python both of it is just "int" > (resp. long in py2.6) --- what is the best way to have both? We have special support for NumPy (and buffer objects in general), so I wouldn't mind getting special support for other common high-performance libraries in as well. However, I'm not sure what we could do more than what gmpy already does. Stefan From dalcinl at gmail.com Tue Dec 23 14:21:47 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 23 Dec 2008 11:21:47 -0200 Subject: [Cython] pushed a fix for -Wstrict-prototypes for C++ Message-ID: Although I believe this should always work for any compiler/flags/platform, if you use a compiler other than GCC or a OS other than Linux, please give a try. changeset: 1538:0638ab853b00 tag: tip user: Lisandro Dalcin date: Tue Dec 23 11:18:16 2008 -0200 summary: fix distutils GCC compilation flags to remove -Wstrict-prototypes for the C++ case -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Tue Dec 23 14:29:51 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 23 Dec 2008 14:29:51 +0100 Subject: [Cython] calling gmp automatically In-Reply-To: <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <85b5c3130812230529t4cae25bfuf537fbeaaf09b24a@mail.gmail.com> On Tue, Dec 23, 2008 at 1:53 PM, Stefan Behnel wrote: > Hi, > > Ondrej Certik wrote: >> currently Cython converts all "ints" as int32 in C (right?). > > Depends on what you mean with "ints". Python ints are Python objects. Only > things defined "cdef int" will become C ints. Yeah, that's right. > > >> How difficult would be to improve Cython, so that it can automatically >> convert all ints to gmp calls, so that one can use arbitrary integers? >> >> Or to put the question differently, obviously, for most of my codes I >> just need int32 and enjoy the speedups. However, sometimes it's handy >> to use arbitrary ints, and since in Python both of it is just "int" >> (resp. long in py2.6) --- what is the best way to have both? > > We have special support for NumPy (and buffer objects in general), so I > wouldn't mind getting special support for other common high-performance > libraries in as well. However, I'm not sure what we could do more than > what gmpy already does. Is there some example in Sage (or elsewhere) how to do this? Looking into sage/ext/arith_gmp.pyx I read: See polynomial_pyx.pyx for an example. and looking into sage/rings/polynomial/polynomial_pyx.pyx I find code like this: cdef mpz_t tmp # Allocate memory for answer, and raise exception on failure. vec[0] = sage_malloc(sizeof(mpz_t) * (f.degree+1)) if vec[0] == 0: raise MemoryError # Go through and find the least commmon multiple of the denominators mpz_set_si(den, 1) for i from 0 <= i <= f.degree: mpz_lcm(den, den, mpq_denref(f.v[i])) # Fill vec with the products c*d, where c are the coefficients if mpz_sgn(den) == 0: # denominator is 1 for i from 0 <= i <= f.degree: mpz_init(vec[0][i]) mpz_set(vec[0][i], mpq_numref(f.v[i])) else: mpz_init(tmp) for i from 0 <= i <= f.degree: mpz_init(vec[0][i]) mpz_divexact(tmp, den, mpq_denref(f.v[i])) mpz_mul(vec[0][i], mpq_numref(f.v[i]), tmp) mpz_clear(tmp) Preferably one could use it like this: cdef mpz tmp tmp = den/f.v[i] vec[0][i] = f.v[i]*tmp or something like that and leave it to Cython to call the correct methods. I don't like the fact that Cython would have to be gmp aware --- maybe there is some way to just implement a cdef class mpz that would support __add__, __mull__ and things like that, so as to behave just like python integer, but still be as fast as the explicit code above. Ondrej From dagss at student.matnat.uio.no Tue Dec 23 14:46:11 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 23 Dec 2008 14:46:11 +0100 (CET) Subject: [Cython] calling gmp automatically In-Reply-To: <85b5c3130812230529t4cae25bfuf537fbeaaf09b24a@mail.gmail.com> References: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <85b5c3130812230529t4cae25bfuf537fbeaaf09b24a@mail.gmail.com> Message-ID: <7403be6aa55d452ce2efb0319feed1af.squirrel@webmail.uio.no> Ondrej Certik wrote: > Preferably one could use it like this: > > cdef mpz tmp > tmp = den/f.v[i] > vec[0][i] = f.v[i]*tmp > > or something like that and leave it to Cython to call the correct > methods. I don't like the fact that Cython would have to be gmp aware > --- maybe there is some way to just implement a cdef class mpz that > would support __add__, __mull__ and things like that, so as to behave > just like python integer, but still be as fast as the explicit code > above. Yep. Operator overloading using inline cdef functions should be a very non-controversial feature to add to Cython, and not very difficult to add either (transform from eg. BinopNode to SimpleCallNode in the right circumstances), and that would solve this problem in a much more generic way. At first that would require one to create a "cdef class" wrapper class containing the struct as a member and the overloaded operators, but when that is all working one could discuss allowing such overloads in structs as well? Dag Sverre From ondrej at certik.cz Tue Dec 23 14:55:41 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 23 Dec 2008 14:55:41 +0100 Subject: [Cython] calling gmp automatically In-Reply-To: <7403be6aa55d452ce2efb0319feed1af.squirrel@webmail.uio.no> References: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <85b5c3130812230529t4cae25bfuf537fbeaaf09b24a@mail.gmail.com> <7403be6aa55d452ce2efb0319feed1af.squirrel@webmail.uio.no> Message-ID: <85b5c3130812230555g7400489bx549eb3a194c87a57@mail.gmail.com> On Tue, Dec 23, 2008 at 2:46 PM, Dag Sverre Seljebotn wrote: > Ondrej Certik wrote: >> Preferably one could use it like this: >> >> cdef mpz tmp >> tmp = den/f.v[i] >> vec[0][i] = f.v[i]*tmp >> >> or something like that and leave it to Cython to call the correct >> methods. I don't like the fact that Cython would have to be gmp aware >> --- maybe there is some way to just implement a cdef class mpz that >> would support __add__, __mull__ and things like that, so as to behave >> just like python integer, but still be as fast as the explicit code >> above. > > Yep. Operator overloading using inline cdef functions should be a very > non-controversial feature to add to Cython, and not very difficult to add > either (transform from eg. BinopNode to SimpleCallNode in the right > circumstances), and that would solve this problem in a much more generic > way. At first that would require one to create a "cdef class" wrapper > class containing the struct as a member and the overloaded operators, but > when that is all working one could discuss allowing such overloads in > structs as well? Yes, I was thinking of something like this. How would this work technically? cdef class mpz: cdef __cinit__(...) pass cdef inline __add__(a, b): cdef mpz tmp mpz_add(a.thisptr, b.thisptr, tmp) return tmp So Cython would inline this. Ondrej From ondrej at certik.cz Tue Dec 23 14:58:03 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 23 Dec 2008 14:58:03 +0100 Subject: [Cython] calling gmp automatically In-Reply-To: <85b5c3130812230555g7400489bx549eb3a194c87a57@mail.gmail.com> References: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <85b5c3130812230529t4cae25bfuf537fbeaaf09b24a@mail.gmail.com> <7403be6aa55d452ce2efb0319feed1af.squirrel@webmail.uio.no> <85b5c3130812230555g7400489bx549eb3a194c87a57@mail.gmail.com> Message-ID: <85b5c3130812230558m54d89d77o2a57839d821764c8@mail.gmail.com> > cdef class mpz: > cdef __cinit__(...) > pass > cdef inline __add__(a, b): > cdef mpz tmp > mpz_add(a.thisptr, b.thisptr, tmp) > return tmp ^^ this should be: cdef mpz_t tmp mpz_init(tmp) mpz_add(a.thisptr, b.thisptr, tmp) return mpz(tmp) Ondrej From stefan_ml at behnel.de Tue Dec 23 08:43:23 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 23 Dec 2008 08:43:23 +0100 Subject: [Cython] overriding cpdef methods Message-ID: <4950969B.3020108@behnel.de> Hi, while cythonising Visitor.py with an external .pxd, I noticed that I can't currently override a cpdef method in a subtype. The hierarchy is like this: -------------------- cdef class BasicVisitor: cdef dict dispatch_table cpdef visit(self, obj) cdef class TreeVisitor(BasicVisitor): cdef public list access_path cpdef visitchild(self, child, parent, attrname, idx) cpdef visitchildren(self, parent, attrs=*) # <----- cdef class VisitorTransform(TreeVisitor): cdef object _super_visitchildren cpdef visitchildren(self, parent, attrs=*) # <----- cpdef recurse_to_children(self, node) -------------------- and the error I get out of Symtab.py is: ------------------------------------------------------------ ... def visitchildren(self, parent, attrs=None): ^ ------------------------------------------------------------ .../Cython/Compiler/Visitor.py:150:4: Self argument of C method does not match parent type Does anyone know if this is an anticipated problem, or just a bug that needs fixing? I don't see any general obstacles for overriding cpdef methods. Stefan From dagss at student.matnat.uio.no Tue Dec 23 15:06:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 23 Dec 2008 15:06:00 +0100 (CET) Subject: [Cython] calling gmp automatically In-Reply-To: <85b5c3130812230555g7400489bx549eb3a194c87a57@mail.gmail.com> References: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <85b5c3130812230529t4cae25bfuf537fbeaaf09b24a@mail.gmail.com> <7403be6aa55d452ce2efb0319feed1af.squirrel@webmail.uio.no> <85b5c3130812230555g7400489bx549eb3a194c87a57@mail.gmail.com> Message-ID: <0f83ad85d12c9eddd9589050369403a3.squirrel@webmail.uio.no> Ondrej Certik wrote: > On Tue, Dec 23, 2008 at 2:46 PM, Dag Sverre Seljebotn > wrote: >> Ondrej Certik wrote: >>> Preferably one could use it like this: >>> >>> cdef mpz tmp >>> tmp = den/f.v[i] >>> vec[0][i] = f.v[i]*tmp >>> >>> or something like that and leave it to Cython to call the correct >>> methods. I don't like the fact that Cython would have to be gmp aware >>> --- maybe there is some way to just implement a cdef class mpz that >>> would support __add__, __mull__ and things like that, so as to behave >>> just like python integer, but still be as fast as the explicit code >>> above. >> >> Yep. Operator overloading using inline cdef functions should be a very >> non-controversial feature to add to Cython, and not very difficult to >> add >> either (transform from eg. BinopNode to SimpleCallNode in the right >> circumstances), and that would solve this problem in a much more generic >> way. At first that would require one to create a "cdef class" wrapper >> class containing the struct as a member and the overloaded operators, >> but >> when that is all working one could discuss allowing such overloads in >> structs as well? > > Yes, I was thinking of something like this. How would this work > technically? > > cdef class mpz: > cdef __cinit__(...) > pass > cdef inline __add__(a, b): > cdef mpz tmp > mpz_add(a.thisptr, b.thisptr, tmp) > return tmp > > So Cython would inline this. Well, it's not supported now, so right now Cython wouldn't do anything. But if supported then the add function would be emitted to C with the INLINE modifier, and gcc can then inline it. That does give the overhead of creating and refcounting a Python object, but it is a good first goal at least. To get around that one could discuss allowing something like this: cdef extern from ...: cdef struct mpz_t: cdef inline mpz_t __add__(mpz_t a, mpz_t b): mpz_t result ... return result or perhaps cdef extern from ...: cdef struct mpz_t: cdef inline int __add__(mpz_t* a, mpz_t* b, mpz_t* result) except -1: ... However it's a question of just how much of C++ we want to reimplement :-) Dag Sverre From dagss at student.matnat.uio.no Tue Dec 23 15:20:17 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 23 Dec 2008 15:20:17 +0100 (CET) Subject: [Cython] overriding cpdef methods In-Reply-To: <4950969B.3020108@behnel.de> References: <4950969B.3020108@behnel.de> Message-ID: Stefan Behnel wrote: > Does anyone know if this is an anticipated problem, or just a bug that > needs fixing? I don't see any general obstacles for overriding cpdef > methods. I don't have an answer to your question, but I want to note that this potentially also applies to return types of overloaded functions. E.g. in C++ you can do something like this: cdef class Subclass(Superclass): cdef Subclass clone(Subclass self): # overrides "Superclass clone(Superclass self)" ... What other languages allow and don't allow in this area is listed here: http://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science) Dag Sverre From robertwb at math.washington.edu Tue Dec 23 19:54:33 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Dec 2008 10:54:33 -0800 Subject: [Cython] overriding cpdef methods In-Reply-To: <4950969B.3020108@behnel.de> References: <4950969B.3020108@behnel.de> Message-ID: On Dec 22, 2008, at 11:43 PM, Stefan Behnel wrote: > Hi, > > while cythonising Visitor.py with an external .pxd, Excellent. > I noticed that I can't > currently override a cpdef method in a subtype. The hierarchy is > like this: > > -------------------- > cdef class BasicVisitor: > cdef dict dispatch_table > cpdef visit(self, obj) > > cdef class TreeVisitor(BasicVisitor): > cdef public list access_path > cpdef visitchild(self, child, parent, attrname, idx) > cpdef visitchildren(self, parent, attrs=*) # <----- > > cdef class VisitorTransform(TreeVisitor): > cdef object _super_visitchildren > cpdef visitchildren(self, parent, attrs=*) # <----- > cpdef recurse_to_children(self, node) > -------------------- > > and the error I get out of Symtab.py is: > > ------------------------------------------------------------ > ... > def visitchildren(self, parent, attrs=None): > ^ > ------------------------------------------------------------ > > .../Cython/Compiler/Visitor.py:150:4: Self argument of C method > does not > match parent type > > Does anyone know if this is an anticipated problem, or just a bug that > needs fixing? I don't see any general obstacles for overriding > cpdef methods. This *should* work, and we use it all the time in Sage (with pyx, not py files). I bet it's a bug in copying the method signatures into the .py parse tree. - Robert From robertwb at math.washington.edu Tue Dec 23 19:58:27 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Dec 2008 10:58:27 -0800 Subject: [Cython] calling gmp automatically In-Reply-To: <7403be6aa55d452ce2efb0319feed1af.squirrel@webmail.uio.no> References: <85b5c3130812230156y9feb20enb50a858667dcd740@mail.gmail.com> <36728.213.61.181.86.1230036816.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <85b5c3130812230529t4cae25bfuf537fbeaaf09b24a@mail.gmail.com> <7403be6aa55d452ce2efb0319feed1af.squirrel@webmail.uio.no> Message-ID: <63C3FFA4-B892-42A7-A8B0-86F3D801D1EB@math.washington.edu> On Dec 23, 2008, at 5:46 AM, Dag Sverre Seljebotn wrote: > Ondrej Certik wrote: >> Preferably one could use it like this: >> >> cdef mpz tmp >> tmp = den/f.v[i] >> vec[0][i] = f.v[i]*tmp >> >> or something like that and leave it to Cython to call the correct >> methods. I don't like the fact that Cython would have to be gmp aware >> --- maybe there is some way to just implement a cdef class mpz that >> would support __add__, __mull__ and things like that, so as to behave >> just like python integer, but still be as fast as the explicit code >> above. > > Yep. Operator overloading using inline cdef functions should be a very > non-controversial feature to add to Cython, and not very difficult > to add > either (transform from eg. BinopNode to SimpleCallNode in the right > circumstances), and that would solve this problem in a much more > generic > way. At first that would require one to create a "cdef class" wrapper > class containing the struct as a member and the overloaded > operators, but > when that is all working one could discuss allowing such overloads in > structs as well? Yes, this has been on my want-to-do list for a long time. In fact, I've been thinking about this in terms of adding support for native (double) complex types. One question is weather/how to handle memory allocation if we implemented gmp integers in a more native way. - Robert From boogaloojb at yahoo.fr Fri Dec 26 11:32:00 2008 From: boogaloojb at yahoo.fr (aucun aucun) Date: Fri, 26 Dec 2008 10:32:00 +0000 (GMT) Subject: [Cython] Numpy - Chararray and efficient indexing Message-ID: <871600.72511.qm@web28516.mail.ukl.yahoo.com> Hello, I've tried to write a small function to merge two Numpy arrays of strings with Cython. I need to use efficient indexing. What I've done for array of integers : import numpy as np cimport numpy as np ctypedef np.int_t DTYPE_t cdef np.ndarray[DTYPE_t, ndim=1] vecteur_n = np.zeros(vecteur.shape[0], dtype=int) ... But I don't know the equivalent of "ctypedef np.int_t DTYPE_t" for strings, or at least a way of getting around this. Thank you for the help, JB -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20081226/11e426ef/attachment.htm From dagss at student.matnat.uio.no Sat Dec 27 11:59:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 27 Dec 2008 11:59:00 +0100 Subject: [Cython] Numpy - Chararray and efficient indexing Message-ID: <3313223986.2323175@smtp.netcom.no> If you have stored Python strings, use "object" as the datatype. If you have stored NumPy fixed-width strings, then the answer is that this is not supported by Cython. The problem is that there's no obvious semantics -- does one copy the whole string on access (wasteful + how to manage the memory?) or get a pointer to it (contrary to how the other types work), which Cython type would one use for the elements (perhaps a single struct containing a fixed-size char array?), and so on. As I never had a use for this myself I have a hard time figuring out how it should work, but if someone has good ideas I may be able to implement it. A more detailed description of your usecase may help. Dag Sverre Seljebotn -----Original Message----- From: aucun aucun Date: Friday, Dec 26, 2008 11:32 am Subject: [Cython] Numpy - Chararray and efficient indexing To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net >Hello, > > >I've tried to write a small function to merge two Numpy arrays of strings with Cython. > > >I need to use efficient indexing. What I've done for array of integers : > > >import numpy as np >cimport numpy as np >ctypedef np.int_t DTYPE_t >cdef np.ndarray[DTYPE_t, ndim=1] vecteur_n = np.zeros(vecteur.shape[0], dtype=int) >... > > >But I don't know the equivalent of "ctypedef np.int_t DTYPE_t" for strings, or at least a way of getting around this. > > >Thank you for the help, > > >JB > > > > > > > > > From boogaloojb at yahoo.fr Mon Dec 29 14:57:20 2008 From: boogaloojb at yahoo.fr (Jean-Baptiste Rudant) Date: Mon, 29 Dec 2008 13:57:20 +0000 (GMT) Subject: [Cython] Re : Numpy - Chararray and efficient indexing References: <3313223986.2323175@smtp.netcom.no> Message-ID: <108221.49241.qm@web28503.mail.ukl.yahoo.com> Thank you for your answer. My english is bad so I tried to describe my usecase with a small piece of code in python. JB ________________________________ De : Dag Sverre Seljebotn ? : cython-dev at codespeak.net Envoy? le : Samedi, 27 D?cembre 2008, 11h59mn 00s Objet : Re: [Cython] Numpy - Chararray and efficient indexing If you have stored Python strings, use "object" as the datatype. If you have stored NumPy fixed-width strings, then the answer is that this is not supported by Cython. The problem is that there's no obvious semantics -- does one copy the whole string on access (wasteful + how to manage the memory?) or get a pointer to it (contrary to how the other types work), which Cython type would one use for the elements (perhaps a single struct containing a fixed-size char array?), and so on. As I never had a use for this myself I have a hard time figuring out how it should work, but if someone has good ideas I may be able to implement it. A more detailed description of your usecase may help. Dag Sverre Seljebotn -----Original Message----- From: aucun aucun Date: Friday, Dec 26, 2008 11:32 am Subject: [Cython] Numpy - Chararray and efficient indexing To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net >Hello, > > >I've tried to write a small function to merge two Numpy arrays of strings with Cython. > > >I need to use efficient indexing. What I've done for array of integers : > > >import numpy as np >cimport numpy as np >ctypedef np.int_t DTYPE_t >cdef np.ndarray[DTYPE_t, ndim=1] vecteur_n = np.zeros(vecteur.shape[0], dtype=int) >... > > >But I don't know the equivalent of "ctypedef np.int_t DTYPE_t" for strings, or at least a way of getting around this. > > >Thank you for the help, > > >JB > > > > > > > > > _______________________________________________ Cython-dev mailing list Cython-dev at codespeak.net http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20081229/75fb45ad/attachment.htm -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Usecase_efficient_indexing_chararray.py Url: http://codespeak.net/pipermail/cython-dev/attachments/20081229/75fb45ad/attachment.diff From dalcinl at gmail.com Mon Dec 29 18:46:29 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Dec 2008 15:46:29 -0200 Subject: [Cython] C99/C++ complex type support Message-ID: I have some working code. If any of you have some time, I can send a preliminary patch. There are a lot of issues that need to be addressed, mainly: * This new complex support should be enabled by using a complier directive. If not, there are backward compatibility issues, now 'complex' means the native C99/C++ type, and not the Python type object. Furthermore, what would happen if the C compiler does not fully implement C99 (Sun?). Other way will be to also support an implementation based on Python's "Py_complex" struct, but I have not idea how to transform airthmetic operations in appropriate function calls. * Regarding the above issue, I'm not sure if we should use "complex" or something new, like "ccomplex", I mean, how to specify the C native type, like "cdef complex z" or "cdef ccomplex z", or something different. The second will clearly help to dissambiguate and also solve backward compatibility issues. * Up to now, I'm only managing the "double complex" case, supporting "float complex" or "long double complex" will require the introduction of a 'precission' concept in the spirit of 'longness' for integers (What do you think?). Unfortunately, I'll not be available on all January to further work on this. Just let me know if any you want to look at my patch. At least, it serves to figure out what parts needs to be touched to make Cython understand "complex" as a native C type. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Mon Dec 29 20:25:34 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Dec 2008 11:25:34 -0800 Subject: [Cython] C99/C++ complex type support In-Reply-To: References: Message-ID: <8931B21F-2623-495B-AD74-FD77BA3F42A2@math.washington.edu> On Dec 29, 2008, at 9:46 AM, Lisandro Dalcin wrote: > I have some working code. If any of you have some time, I can send a > preliminary patch. I don't know how soon I'll have time, but I'm very interested in this. > There are a lot of issues that need to be addressed, mainly: > > * This new complex support should be enabled by using a complier > directive. If not, there are backward compatibility issues, now > 'complex' means the native C99/C++ type, and not the Python type > object. Furthermore, what would happen if the C compiler does not > fully implement C99 (Sun?). Other way will be to also support an > implementation based on Python's "Py_complex" struct, but I have not > idea how to transform airthmetic operations in appropriate function > calls. I think we should use our own struct, and transforming the arithmetic into function calls is something I've already thought about (in fact, this is what I intended to do, with macros being defined in the case the compiler supports complex types natively). > * Regarding the above issue, I'm not sure if we should use "complex" > or something new, like "ccomplex", I mean, how to specify the C native > type, like "cdef complex z" or "cdef ccomplex z", or something > different. The second will clearly help to dissambiguate and also > solve backward compatibility issues. I think we should handle this the same way we handle int, float, long, etc, i.e. the name specifies the C type. > * Up to now, I'm only managing the "double complex" case, supporting > "float complex" or "long double complex" will require the introduction > of a 'precission' concept in the spirit of 'longness' for integers > (What do you think?). We should certainly support this. > Unfortunately, I'll not be available on all January to further work on > this. Just let me know if any you want to look at my patch. At least, > it serves to figure out what parts needs to be touched to make Cython > understand "complex" as a native C type. I need this for something I'm working on, so I'll try and pick up where you left off. - Robert From robert.kern at gmail.com Mon Dec 29 20:42:48 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Dec 2008 14:42:48 -0500 Subject: [Cython] C99/C++ complex type support In-Reply-To: References: Message-ID: Lisandro Dalcin wrote: > I have some working code. If any of you have some time, I can send a > preliminary patch. > > There are a lot of issues that need to be addressed, mainly: > > * This new complex support should be enabled by using a complier > directive. If not, there are backward compatibility issues, now > 'complex' means the native C99/C++ type, and not the Python type > object. Furthermore, what would happen if the C compiler does not > fully implement C99 (Sun?). Other way will be to also support an > implementation based on Python's "Py_complex" struct, but I have not > idea how to transform airthmetic operations in appropriate function > calls. > > * Regarding the above issue, I'm not sure if we should use "complex" > or something new, like "ccomplex", I mean, how to specify the C native > type, like "cdef complex z" or "cdef ccomplex z", or something > different. The second will clearly help to dissambiguate and also > solve backward compatibility issues. > > * Up to now, I'm only managing the "double complex" case, supporting > "float complex" or "long double complex" will require the introduction > of a 'precission' concept in the spirit of 'longness' for integers > (What do you think?). You might want to follow numpy's lead and define complex64 (float complex), complex128 (double complex), etc. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dalcinl at gmail.com Mon Dec 29 21:43:12 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Dec 2008 18:43:12 -0200 Subject: [Cython] C99/C++ complex type support In-Reply-To: <8931B21F-2623-495B-AD74-FD77BA3F42A2@math.washington.edu> References: <8931B21F-2623-495B-AD74-FD77BA3F42A2@math.washington.edu> Message-ID: On Mon, Dec 29, 2008 at 5:25 PM, Robert Bradshaw wrote: > I need this for something I'm working on, so I'll try and pick up > where you left off. > OK, here you have... Sorry about the mess... * the "CCOMPLEX.diff" patch was generated against cython-devel (updated about 3 hours ago) * The "ccompleximpl.h" header has some macros that should end-up verbatim in the Cython-generated sources, perhaps by using utility code. * The "ccomplex.pyx" is a testcase that should go to "tests/run" (and assumes the "ccompleximpl.h" is also in "tests/run"). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: CCOMPLEX.diff Type: application/octet-stream Size: 4890 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20081229/0b0dbbe4/attachment.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: ccompleximpl.h Type: text/x-chdr Size: 794 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20081229/0b0dbbe4/attachment.h -------------- next part -------------- A non-text attachment was scrubbed... Name: ccomplex.pyx Type: application/octet-stream Size: 395 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20081229/0b0dbbe4/attachment-0001.obj From dalcinl at gmail.com Mon Dec 29 21:45:32 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Dec 2008 18:45:32 -0200 Subject: [Cython] C99/C++ complex type support In-Reply-To: References: Message-ID: On Mon, Dec 29, 2008 at 5:42 PM, Robert Kern wrote: > You might want to follow numpy's lead and define complex64 (float complex), > complex128 (double complex), etc. > I really like your idea... Let's see what other people think about this... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Mon Dec 29 21:52:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Dec 2008 12:52:42 -0800 Subject: [Cython] C99/C++ complex type support In-Reply-To: References: Message-ID: <1F71F4C2-E444-4857-BC19-163CD647250D@math.washington.edu> On Dec 29, 2008, at 12:45 PM, Lisandro Dalcin wrote: > On Mon, Dec 29, 2008 at 5:42 PM, Robert Kern > wrote: >> You might want to follow numpy's lead and define complex64 (float >> complex), >> complex128 (double complex), etc. >> > > I really like your idea... Let's see what other people think about > this... Yes. Note that "double complex" typically means a complex with double- prec real and imaginary parts, not 128 bits. - Robert From dagss at student.matnat.uio.no Mon Dec 29 21:55:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 29 Dec 2008 21:55:00 +0100 Subject: [Cython] C99/C++ complex type support Message-ID: <3313432543.2946327@smtp.netcom.no> Great! I might have some time in January, and (better) might need it myself then. It is a chicken and egg problem though: I do not want to say whether I can spend time until I see the nature of the patch and how easy it is to build on to fix the shortcomings you mention :-) This is all much easier to talk about when one can see the code, so even if the patch isn't applied immediately it is helpful to have to glance on. I recommend uploading a diff to a trac ticket (I think one exists but not sure), or if you do not want to do that just send me a copy. In short, this is in pretty high demand and it looks like either myself or Robert will pick it up if you make the patch available. Dag Sverre Seljebotn -----Original Message----- From: "Lisandro Dalcin" Date: Monday, Dec 29, 2008 6:46 pm Subject: [Cython] C99/C++ complex type support To: cython-dev Reply-To: cython-dev at codespeak.net I have some working code. If any of you have some time, I can send a >preliminary patch. > >There are a lot of issues that need to be addressed, mainly: > >* This new complex support should be enabled by using a complier >directive. If not, there are backward compatibility issues, now >'complex' means the native C99/C++ type, and not the Python type >object. Furthermore, what would happen if the C compiler does not >fully implement C99 (Sun?). Other way will be to also support an >implementation based on Python's "Py_complex" struct, but I have not >idea how to transform airthmetic operations in appropriate function >calls. > >* Regarding the above issue, I'm not sure if we should use "complex" >or something new, like "ccomplex", I mean, how to specify the C native >type, like "cdef complex z" or "cdef ccomplex z", or something >different. The second will clearly help to dissambiguate and also >solve backward compatibility issues. > >* Up to now, I'm only managing the "double complex" case, supporting >"float complex" or "long double complex" will require the introduction >of a 'precission' concept in the spirit of 'longness' for integers >(What do you think?). > >Unfortunately, I'll not be available on all January to further work on >this. Just let me know if any you want to look at my patch. At least, >it serves to figure out what parts needs to be touched to make Cython >understand "complex" as a native C type. > > >-- >Lisandro Dalc?n >--------------- >Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >PTLC - G?emes 3450, (3000) Santa Fe, Argentina >Tel/Fax: +54-(0)342-451.1594 >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From robert.kern at gmail.com Mon Dec 29 22:07:46 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Dec 2008 16:07:46 -0500 Subject: [Cython] C99/C++ complex type support In-Reply-To: <1F71F4C2-E444-4857-BC19-163CD647250D@math.washington.edu> References: <1F71F4C2-E444-4857-BC19-163CD647250D@math.washington.edu> Message-ID: Robert Bradshaw wrote: > On Dec 29, 2008, at 12:45 PM, Lisandro Dalcin wrote: > >> On Mon, Dec 29, 2008 at 5:42 PM, Robert Kern >> wrote: >>> You might want to follow numpy's lead and define complex64 (float >>> complex), >>> complex128 (double complex), etc. >>> >> I really like your idea... Let's see what other people think about >> this... > > Yes. Note that "double complex" typically means a complex with double- > prec real and imaginary parts, not 128 bits. Right. numpy goes with the convention that the bit-width suffix is the width of the entire item, not the component real and imaginary parts for the complex type. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robertwb at math.washington.edu Mon Dec 29 22:13:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Dec 2008 13:13:31 -0800 Subject: [Cython] C99/C++ complex type support In-Reply-To: References: <1F71F4C2-E444-4857-BC19-163CD647250D@math.washington.edu> Message-ID: <5036B9B4-88F5-4A1D-8221-23CA4287D442@math.washington.edu> On Dec 29, 2008, at 1:07 PM, Robert Kern wrote: > Robert Bradshaw wrote: >> On Dec 29, 2008, at 12:45 PM, Lisandro Dalcin wrote: >> >>> On Mon, Dec 29, 2008 at 5:42 PM, Robert Kern >>> wrote: >>>> You might want to follow numpy's lead and define complex64 (float >>>> complex), >>>> complex128 (double complex), etc. >>>> >>> I really like your idea... Let's see what other people think about >>> this... >> >> Yes. Note that "double complex" typically means a complex with >> double- >> prec real and imaginary parts, not 128 bits. > > Right. numpy goes with the convention that the bit-width suffix is > the width of > the entire item, not the component real and imaginary parts for the > complex type. Ah, good. Thanks for the clarification. - Robert From robertwb at math.washington.edu Mon Dec 29 22:18:11 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Dec 2008 13:18:11 -0800 Subject: [Cython] [Pyrex] const in function type signature In-Reply-To: <200812251209.19013.dominic.sacre@gmx.de> References: <200812251209.19013.dominic.sacre@gmx.de> Message-ID: <403025FB-5684-43B6-B2DD-DCCE6D9C597F@math.washington.edu> On Dec 25, 2008, at 3:09 AM, Dominic Sacr? wrote: > Hi, > > I'm using Pyrex to wrap a C library that uses callbacks. Some of the > parameters to the callback functions are const, e.g. > void(*callback)(int num, const char *msg, const char *where) > > Pyrex doesn't allow const, so i have to write this as > void(*callback)(int num, char *msg, char *where) > > This works, as long as the function doesn't actually try to modify its > arguments, but passing the function pointer to the library produces > warnings like: > passing argument 2 of ?lo_server_new? from incompatible pointer type > > Is there some way to get rid of these warnings? Basically, just > allowing the > use of const in the Pyrex source, and literally placing it in the > generated > C code would suffice... Yes, see http://wiki.cython.org/FAQ#head- af019db12f683abe6bc226926e8a2a92425b8e57 - Robert From dalcinl at gmail.com Mon Dec 29 22:19:27 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Dec 2008 19:19:27 -0200 Subject: [Cython] C99/C++ complex type support In-Reply-To: <5036B9B4-88F5-4A1D-8221-23CA4287D442@math.washington.edu> References: <1F71F4C2-E444-4857-BC19-163CD647250D@math.washington.edu> <5036B9B4-88F5-4A1D-8221-23CA4287D442@math.washington.edu> Message-ID: On Mon, Dec 29, 2008 at 7:13 PM, Robert Bradshaw wrote: > On Dec 29, 2008, at 1:07 PM, Robert Kern wrote: > >> Robert Bradshaw wrote: >>> On Dec 29, 2008, at 12:45 PM, Lisandro Dalcin wrote: >>> >>>> On Mon, Dec 29, 2008 at 5:42 PM, Robert Kern >>>> wrote: >>>>> You might want to follow numpy's lead and define complex64 (float >>>>> complex), >>>>> complex128 (double complex), etc. >>>>> >>>> I really like your idea... Let's see what other people think about >>>> this... >>> >>> Yes. Note that "double complex" typically means a complex with >>> double- >>> prec real and imaginary parts, not 128 bits. >> >> Right. numpy goes with the convention that the bit-width suffix is >> the width of >> the entire item, not the component real and imaginary parts for the >> complex type. > > Ah, good. Thanks for the clarification. > However, remember this... sizeof(float) <= sizeof(double) <= sizeof(long double) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Dec 29 22:39:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 29 Dec 2008 22:39:00 +0100 Subject: [Cython] C99/C++ complex type support Message-ID: <3313435169.3063287@smtp.netcom.no> My opinion is that specifying exact bit lengths should be an orthogonal issue/feature to complex float types. Cython already uses the C type system, and making an exception for complex types seems inconsistent and harder to learn. So I vote for using the syntax "complex long double" and so on, and what the "long double" part means is up to the C compiler (like other types work). Note that no matter what we do, "complex64" etc. should be made available as typedefs in numpy.pxd, making use of the bit-length-detection already in the numpy C headers. Dag Sverre Seljebotn -----Original Message----- From: Robert Kern Date: Monday, Dec 29, 2008 10:07 pm Subject: Re: [Cython] C99/C++ complex type support To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Robert Bradshaw wrote: > On Dec 29, 2008, at 12:45 PM, Lisandro Dalcin wrote: > >> On Mon, Dec 29, 2008 at 5:42 PM, Robert Kern >> wrote: >>> You might want to follow numpy's lead and define complex64 (float >>> complex), >>> complex128 (double complex), etc. >>> >> I really like your idea... Let's see what other people think about >> this... > > Yes. Note that "double complex" typically means a complex with double- > prec real and imaginary parts, not 128 bits. > >Right. numpy goes with the convention that the bit-width suffix is the width of >the entire item, not the component real and imaginary parts for the complex type. > >-- >Robert Kern > >"I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Mon Dec 29 22:50:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Dec 2008 13:50:22 -0800 Subject: [Cython] C99/C++ complex type support In-Reply-To: <3313435169.3063287@smtp.netcom.no> References: <3313435169.3063287@smtp.netcom.no> Message-ID: On Dec 29, 2008, at 1:39 PM, Dag Sverre Seljebotn wrote: > My opinion is that specifying exact bit lengths should be an > orthogonal issue/feature to complex float types. Cython already > uses the C type system, and making an exception for complex types > seems inconsistent and harder to learn. So I vote for using the > syntax "complex long double" and so on, and what the "long double" > part means is up to the C compiler (like other types work). Yes, I meant that we let the complex64/128 type coerce back and forth between these, I certainly agree with you that one should define them the same way as in C. > Note that no matter what we do, "complex64" etc. should be made > available as typedefs in numpy.pxd, making use of the bit-length- > detection already in the numpy C headers. > > Dag Sverre Seljebotn > -----Original Message----- > From: Robert Kern > Date: Monday, Dec 29, 2008 10:07 pm > Subject: Re: [Cython] C99/C++ complex type support > To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > > Robert Bradshaw wrote: >> On Dec 29, 2008, at 12:45 PM, Lisandro Dalcin wrote: >> >>> On Mon, Dec 29, 2008 at 5:42 PM, Robert Kern >>> wrote: >>>> You might want to follow numpy's lead and define complex64 (float >>>> complex), >>>> complex128 (double complex), etc. >>>> >>> I really like your idea... Let's see what other people think about >>> this... >> >> Yes. Note that "double complex" typically means a complex with >> double- >> prec real and imaginary parts, not 128 bits. >> >> Right. numpy goes with the convention that the bit-width suffix is >> the width of >> the entire item, not the component real and imaginary parts for >> the complex type. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a >> harmless enigma >> that is made terrible by our own mad attempt to interpret it as >> though it had >> an underlying truth." >> -- Umberto Eco >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Tue Dec 30 00:28:45 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Dec 2008 21:28:45 -0200 Subject: [Cython] catched a reference leak problem in try/except block (temp allocation?) Message-ID: Sorry in advance if this is a know issue ... Anyway, I have no idea about how to fix it ;-( ... I have the following function, which expects a (possibly empty) list: def gettok(tokens): try: return tokens.pop(0) except IndexError: return None However, when I call "gettok" in a loop passing a empty list, 2 references are leaked per iteration. I've distilled the problem to this: when the call "tokens.pop(0)" actually raises IndexError, then the temporary 1-tuple with item "0" (zero) that Cython build in order to make the method call is never DECREF'ed; then you leak two references (the 1-tuple plus the integer "0"). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Dec 30 12:21:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 30 Dec 2008 12:21:57 +0100 Subject: [Cython] catched a reference leak problem in try/except block (temp allocation?) In-Reply-To: References: Message-ID: <495A0455.6000808@behnel.de> Hi Lisandro, Lisandro Dalcin wrote: > def gettok(tokens): > try: > return tokens.pop(0) > except IndexError: > return None > > I've distilled the problem to this: when the call "tokens.pop(0)" > actually raises IndexError, then the temporary 1-tuple with item "0" > (zero) that Cython build in order to make the method call is never > DECREF'ed; then you leak two references (the 1-tuple plus the integer > "0"). Thanks for catching this. New style temps were not cleaned up during exception handling. Should be fixed now. Stefan