From cournape at gmail.com Sun Nov 1 09:32:27 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 1 Nov 2009 17:32:27 +0900 Subject: [Cython] [patch] Optional setuptools-based cython build Message-ID: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> Hi, I needed to be able to build an egg for cython on windows, and thought it could be interesting to integrate it to cython sources. Besides an egg, it provides cython as a console_script, which is easier to handle on windows. The patch is minimal and non invasive, as by default, setuptools is *not* run if you use setup.py. I attached the patch here (I can't connect to trac ATM), cheers, David -------------- next part -------------- A non-text attachment was scrubbed... Name: eggify.patch Type: application/octet-stream Size: 1846 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20091101/25aa7ab7/attachment.obj From stefan_ml at behnel.de Sun Nov 1 14:02:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Nov 2009 14:02:40 +0100 Subject: [Cython] getting rid of the PY_NEW() hack In-Reply-To: References: <4AEC392B.9080406@behnel.de> Message-ID: <4AED86F0.3010404@behnel.de> Lisandro Dalcin, 31.10.2009 22:12: > On Sat, Oct 31, 2009 at 10:18 AM, Stefan Behnel wrote: >> To fix this, I pushed a change that optimises this >> >> cdef type some_type = ... >> obj = some_type.__new__(some_type) >> > > So it seems that the optimization only works if __new__ is called with > just a single arg some_type... but your __new__ optimization will > definitely call __cinit__. Do you think that optimizing a many-args > __cinit__ is a nonsense? Not nonsense, but certainly less important. Since the arguments are passed as Python arguments in that case, it doesn't really fit the use case of fast instantiation. Without having benchmarked it any recently, I assume it's still quite a bit faster to do cdef MyType obj = MyType.__new__(MyType) obj.some_attr = something than to do cdef MyType obj = MyType.__new__(MyType, something) and set the attribute in __cinit__(). It may not be that a big difference if you are only passing Python arguments, but it certainly is a bigger difference if you additionally need to convert arguments from C types to Python types and back, just to get them passed into __cinit__(). So I assume that the main use case is fast instantiation from within a factory function. > BTW, You did not taken into account __cinit__ special method in your > testing... I've tried to add __cinit__ in your testcase with a "print" > inside, but I do not know why the test hangs (this is a bit strange, > isn't it?)... I added that to the tests and it absolutely works for me. Stefan From stefan_ml at behnel.de Sun Nov 1 14:44:22 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Nov 2009 14:44:22 +0100 Subject: [Cython] getting rid of the PY_NEW() hack In-Reply-To: References: <4AEC392B.9080406@behnel.de> Message-ID: <4AED90B6.4070601@behnel.de> Hi Lisandro, thanks for the feed-back. I should note that I actually consider the current implementation also a bit of a hack. The right place to implement this would be as part of the builtin 'type' object, so that we could more easily support efficient references to __new__, __init__, __call__, __add__, etc. But I think what's there now will make life easier for users, as they no longer have to keep header files around to do this. I'll be very happy if this can be ripped back out in favour of a more general solution, but until then, it'll do its job IMO. Lisandro Dalcin, 31.10.2009 21:43: > On Sat, Oct 31, 2009 at 10:18 AM, Stefan Behnel wrote: >> I think this is safe, but I wanted to mention it here so that others can >> give it another bit of thought. > > Stefan, could you cythonize the line below: > > cdef str s = str.__new__(str) > > In the generated C code you will see a superfluous check "if(str is > None): raise TypeError()" ... Can you figure out why this is > happening? Well, the above is also part of the tests, so I actually know that this is happening - and I even know why! :) The problem is that I wasn't sure how to figure out that a variable refers to a builtin type. There isn't a None check generated for extension types as they are easy to recognise, so if someone knows a good way to disable it also for builtin types, please fix it. That said, I doubt that the above has a major use case, and even with the None pointer comparison (assuming that the C compiler actually generates code for it), the optimised tp_new() call will still be faster than a Python call to the __new__() method. Note that there's also a str type check generated during the above assignment, unless you add an additional cast to the rhs. We could potentially discard it, but given that __new__() can actually return an arbitrary object (and given that it's not trivial to know when it's safe to disable that type check), I chose to leave it in for now. > BTW, this piece of code if far from being what CPython actually does :-) ... > > 1064 elif type_arg.type_entry != obj.type_entry: > 1065 # different types - do what CPython does at runtime > 1066 error(type_arg.pos, "%s.__new__(%s) is not safe, > use %s.__new__()" % > 1067 (obj.type_entry.name, type_arg.type_entry.name, > 1068 type_arg.type_entry.name)) > 1069 return node You think so? I didn't read the code before, just did a couple of tests, but I wasn't able to get anything but an exception out of __new__() whenever I used two different types on both sides. Anyway, the above check means that a user is trying to call tp_new() of one type to instantiate another type. Given that Cython generates a separate tp_new() for each extension type, this is close enough to the tp_new pointer comparison check that CPython does here. But I wouldn't mind dropping the above error in favour of a runtime failure. That's not a major concern of the optimisation. Stefan From ndbecker2 at gmail.com Mon Nov 2 19:25:08 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 02 Nov 2009 13:25:08 -0500 Subject: [Cython] =?utf-8?b?ZXJyb3I6IOKAmF9fUHl4X1NhZmVSZWxlYXNlQnVmZmVy?= =?utf-8?q?=E2=80=99_was_not_declared_in_this_scope?= References: Message-ID: Neal Becker wrote: > This code: > ------------------- > import numpy as np > cimport numpy as np > > np.import_array() > > cdef inline int x_ (int k, int n): > cdef int center = (n-1)/2 > return k-center > > def lagrange (float x, int n): > > cdef np.ndarray[np.float64_t, ndim=1] ret # = np.empty (n, > dtype=float) cdef int j, k > cdef float prod, top, bot > > for j in range (n): > prod = 1 > for k in xrange (0, n): > if (k != j): > top = x - x_(k, n) > bot = x_(j, n) - x_(k, n) > prod *= top/bot > ret[j] = prod > return ret > -------------- > > gives: > > _lagrange_coef.cpp: In function ?PyObject* > __pyx_pf_14_lagrange_coef_lagrange(PyObject*, PyObject*, PyObject*)?: > _lagrange_coef.cpp:941: error: ?__Pyx_SafeReleaseBuffer? was not declared > in this scope > _lagrange_coef.cpp:947: error: ?__Pyx_SafeReleaseBuffer? was not declared > in this scope > > No error with the comment (np.empty) removed. Bug or user error? > Any ideas on this? From dagss at student.matnat.uio.no Mon Nov 2 20:27:49 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 02 Nov 2009 20:27:49 +0100 Subject: [Cython] =?utf-8?b?ZXJyb3I6IOKAmF9fUHl4X1NhZmVSZWxlYXNlQnVmZmVy?= =?utf-8?q?=E2=80=99_was_not_declared_in_this_scope?= In-Reply-To: References: Message-ID: <4AEF32B5.4070903@student.matnat.uio.no> Neal Becker wrote: > Neal Becker wrote: > >> This code: >> ------------------- >> import numpy as np >> cimport numpy as np >> >> np.import_array() >> >> cdef inline int x_ (int k, int n): >> cdef int center = (n-1)/2 >> return k-center >> >> def lagrange (float x, int n): >> >> cdef np.ndarray[np.float64_t, ndim=1] ret # = np.empty (n, >> dtype=float) cdef int j, k >> cdef float prod, top, bot >> >> for j in range (n): >> prod = 1 >> for k in xrange (0, n): >> if (k != j): >> top = x - x_(k, n) >> bot = x_(j, n) - x_(k, n) >> prod *= top/bot >> ret[j] = prod >> return ret >> -------------- >> >> gives: >> >> _lagrange_coef.cpp: In function ?PyObject* >> __pyx_pf_14_lagrange_coef_lagrange(PyObject*, PyObject*, PyObject*)?: >> _lagrange_coef.cpp:941: error: ?__Pyx_SafeReleaseBuffer? was not declared >> in this scope >> _lagrange_coef.cpp:947: error: ?__Pyx_SafeReleaseBuffer? was not declared >> in this scope >> >> No error with the comment (np.empty) removed. Bug or user error? >> > > Any ideas on this? Thanks for the report, this is now fixed (reported and closed as #444). (Of course, the only effect this can have is that you'll get a segfault at runtime instead... :-) but Cython should never create unexpected noncompileable code and a segfault is better in this case.) -- Dag Sverre From dagss at student.matnat.uio.no Mon Nov 2 20:53:49 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 02 Nov 2009 20:53:49 +0100 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> Message-ID: <4AEF38CD.20309@student.matnat.uio.no> Robert Bradshaw wrote: > On Oct 28, 2009, at 8:39 AM, Dag Sverre Seljebotn wrote: > >> I just added http://trac.cython.org/cython_trac/ticket/441 >> >> Basically, >> >> ctypedef float myfloat; >> ... >> print some_myfloat * some_complex >> >> miscompiles as "some_myfloat" is coerced to "myfloat complex" (which, >> interestingly, is a type which can't be created or used in any other >> way!) >> >> Fixing this requires some thought. >> >> This probably broke because (in order to fix another bug) I stepped >> away >> from "the order something is needed in Cython decides output order in >> C". I believe that was a correct decision and don't want to step back. >> The proper solution is a DAG of all types and their dependencies. I >> believe that is a bit heavy/destabilizing for 0.12 though. Perhaps >> output all very simply typedefs (typedefs of ints and floats) first, >> then complex, then the rest? > > Yep, I don't see the full DAG analysis happening in 0.12. I think any > arbitrary order we come up with is potentially prone to errors. I'm OK > with just disallowing that for 0.12, it's not a regression. OK, I disabled it as I likely won't have time for a proper fix: http://hg.cython.org/cython-devel/rev/c6a27fd42d87 (Did you do all of http://hg.cython.org/cython-devel/rev/82d312a9f1fc by hand? Either way I'm impressed, thanks a lot!) -- Dag Sverre From robertwb at math.washington.edu Mon Nov 2 23:14:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 2 Nov 2009 14:14:07 -0800 Subject: [Cython] More complex bugs In-Reply-To: <4AEF38CD.20309@student.matnat.uio.no> References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> Message-ID: <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> On Nov 2, 2009, at 11:53 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Oct 28, 2009, at 8:39 AM, Dag Sverre Seljebotn wrote: >> >>> I just added http://trac.cython.org/cython_trac/ticket/441 >>> >>> Basically, >>> >>> ctypedef float myfloat; >>> ... >>> print some_myfloat * some_complex >>> >>> miscompiles as "some_myfloat" is coerced to "myfloat >>> complex" (which, >>> interestingly, is a type which can't be created or used in any other >>> way!) >>> >>> Fixing this requires some thought. >>> >>> This probably broke because (in order to fix another bug) I stepped >>> away >>> from "the order something is needed in Cython decides output order >>> in >>> C". I believe that was a correct decision and don't want to step >>> back. >>> The proper solution is a DAG of all types and their dependencies. I >>> believe that is a bit heavy/destabilizing for 0.12 though. Perhaps >>> output all very simply typedefs (typedefs of ints and floats) first, >>> then complex, then the rest? >> >> Yep, I don't see the full DAG analysis happening in 0.12. I think any >> arbitrary order we come up with is potentially prone to errors. I'm >> OK >> with just disallowing that for 0.12, it's not a regression. > > OK, I disabled it as I likely won't have time for a proper fix: > > http://hg.cython.org/cython-devel/rev/c6a27fd42d87 Thanks. I'm still hoping to get an alpha out very soon... There's another regression, "cdef int complex" doesn't work anymore (our structs used to support this, as does gcc, even if it's non-c99). > (Did you do all of http://hg.cython.org/cython-devel/rev/ > 82d312a9f1fc by > hand? Either way I'm impressed, thanks a lot!) I did write a script to do most of that (though I'll admit it was kind of hackish and still required too much manual touchup in some files). - Robert From robertwb at math.washington.edu Tue Nov 3 06:35:44 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 2 Nov 2009 21:35:44 -0800 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> Message-ID: <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> On Nov 1, 2009, at 1:32 AM, David Cournapeau wrote: > Hi, > > I needed to be able to build an egg for cython on windows, and thought > it could be interesting to integrate it to cython sources. Besides an > egg, it provides cython as a console_script, which is easier to handle > on windows. > > The patch is minimal and non invasive, as by default, setuptools is > *not* run if you use setup.py. I attached the patch here (I can't > connect to trac ATM), This sounds like a good idea, though distutils and packaging are outside my area of expertise, especially on Windows. I am a bit wary of the line if 'setuptools' in sys.modules is this always safe? - Robert From robertwb at math.washington.edu Tue Nov 3 06:40:34 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 2 Nov 2009 21:40:34 -0800 Subject: [Cython] "real" module support In-Reply-To: <200910292031.38904.binet@cern.ch> References: <200910292031.38904.binet@cern.ch> Message-ID: <6D95D03E-B5B1-4E39-A92C-BA12854BE710@math.washington.edu> On Oct 29, 2009, at 12:31 PM, Sebastien Binet wrote: > hi there, > > I have some freecycles these days and I was wondering if I couldn't > dive a > little bit more into cython's codebase. > > my pet peeve would be to add some more user friendly module support > in cython, > so one could write: > > # mymodule.pyx > __all__ = [ 'fct1', 'fct2', 'cls1', 'obj', ] > # definitions follow > # EOF > > and be able to cimport those in various other cython modules, > instead of > having to create .pxd and .pyx files. > > so, where should I start ? That is a highly requested feature, even without the __all__. To start I would look at find_module in Main.py, and change it to look for .pyx files if the search for .pxd files failed. There may or may not be other changes that are needed, but that's where to start. - Robert From dagss at student.matnat.uio.no Tue Nov 3 09:45:31 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 09:45:31 +0100 Subject: [Cython] More complex bugs In-Reply-To: <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> Message-ID: <4AEFEDAB.6020306@student.matnat.uio.no> Robert Bradshaw wrote: > On Nov 2, 2009, at 11:53 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On Oct 28, 2009, at 8:39 AM, Dag Sverre Seljebotn wrote: >>> >>>> I just added http://trac.cython.org/cython_trac/ticket/441 >>>> >>>> Basically, >>>> >>>> ctypedef float myfloat; >>>> ... >>>> print some_myfloat * some_complex >>>> >>>> miscompiles as "some_myfloat" is coerced to "myfloat >>>> complex" (which, >>>> interestingly, is a type which can't be created or used in any other >>>> way!) >>>> >>>> Fixing this requires some thought. >>>> >>>> This probably broke because (in order to fix another bug) I stepped >>>> away >>>> from "the order something is needed in Cython decides output order >>>> in >>>> C". I believe that was a correct decision and don't want to step >>>> back. >>>> The proper solution is a DAG of all types and their dependencies. I >>>> believe that is a bit heavy/destabilizing for 0.12 though. Perhaps >>>> output all very simply typedefs (typedefs of ints and floats) first, >>>> then complex, then the rest? >>> Yep, I don't see the full DAG analysis happening in 0.12. I think any >>> arbitrary order we come up with is potentially prone to errors. I'm >>> OK >>> with just disallowing that for 0.12, it's not a regression. >> OK, I disabled it as I likely won't have time for a proper fix: >> >> http://hg.cython.org/cython-devel/rev/c6a27fd42d87 > > Thanks. I'm still hoping to get an alpha out very soon... There's > another regression, "cdef int complex" doesn't work anymore (our > structs used to support this, as does gcc, even if it's non-c99). Turns out this broke nearly all my own code :-( I'll give a proper solution a go. -- Dag Sverre From robertwb at math.washington.edu Tue Nov 3 10:20:39 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 01:20:39 -0800 Subject: [Cython] More complex bugs In-Reply-To: <4AEFEDAB.6020306@student.matnat.uio.no> References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> Message-ID: On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: > >> There's >> another regression, "cdef int complex" doesn't work anymore (our >> structs used to support this, as does gcc, even if it's non-c99). > > Turns out this broke nearly all my own code :-( I'll give a proper > solution a go. Thanks! It think it should be pretty easy (probably can use the type name, rather than the math postfix marker to name the methods...) http://trac.cython.org/cython_trac/ticket/446 - Robert From robertwb at math.washington.edu Tue Nov 3 10:22:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 01:22:05 -0800 Subject: [Cython] optimizing list.pop In-Reply-To: References: <4AE48D6D.3080503@molden.no> Message-ID: <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> On Oct 25, 2009, at 11:53 AM, Lisandro Dalcin wrote: > On Sun, Oct 25, 2009 at 2:39 PM, Sturla Molden > wrote: >> I just realized the Python does not provide a fast way of popping an >> item from a list, just append. I.e. there is a PyList_Append, but no >> PyList_Pop. And in Python's code for _listobject.c, the function >> listpop >> is declared static. When using a list as a stack (a common case for >> Python lists), a fast pop is just as important as a fast append. >> > > Indeed. +1 for the idea, but... > >> Since the function listpop is static, the fastest way to pop a list >> is >> to grab the method named "pop" from the PyTypeObject. This might >> not be >> obvious to everyone, so I think it is an optimization that Cython >> should do. >> > > -0.5 for the implementation... The hack is cute, but still have to > make a listpop(self,args), were you need to create a tuple to make the > call... I think Cython could generate a custom, inline function > implementing l.pop([n]) and fast dispatch to it if the object is > exactly a list ... http://hg.cython.org/cython-devel/rev/2e3dda4a7d23 - Robert From cournape at gmail.com Tue Nov 3 12:12:43 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 3 Nov 2009 20:12:43 +0900 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> Message-ID: <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> On Tue, Nov 3, 2009 at 2:35 PM, Robert Bradshaw wrote: > > if 'setuptools' in sys.modules > > is this always safe? What kind of unsafe behavior would you have in mind ? David From stefan_ml at behnel.de Tue Nov 3 12:50:27 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 03 Nov 2009 12:50:27 +0100 Subject: [Cython] optimizing list.pop In-Reply-To: <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> References: <4AE48D6D.3080503@molden.no> <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> Message-ID: <4AF01903.8060909@behnel.de> Robert Bradshaw, 03.11.2009 10:22: > On Oct 25, 2009, at 11:53 AM, Lisandro Dalcin wrote: >> I think Cython could generate a custom, inline function >> implementing l.pop([n]) and fast dispatch to it if the object is >> exactly a list ... > > http://hg.cython.org/cython-devel/rev/2e3dda4a7d23 Hmm, is this really worth the code duplication with CPython? I mean, we are partly copying the list-shrink trigger algorithm here, which I would prefer to consider an implementation detail of the list type. Stefan From dagss at student.matnat.uio.no Tue Nov 3 15:28:45 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 15:28:45 +0100 Subject: [Cython] optimizing list.pop In-Reply-To: <4AF01903.8060909@behnel.de> References: <4AE48D6D.3080503@molden.no> <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> <4AF01903.8060909@behnel.de> Message-ID: <4AF03E1D.4000706@student.matnat.uio.no> Stefan Behnel wrote: > Robert Bradshaw, 03.11.2009 10:22: > >> On Oct 25, 2009, at 11:53 AM, Lisandro Dalcin wrote: >> >>> I think Cython could generate a custom, inline function >>> implementing l.pop([n]) and fast dispatch to it if the object is >>> exactly a list ... >>> >> http://hg.cython.org/cython-devel/rev/2e3dda4a7d23 >> > > Hmm, is this really worth the code duplication with CPython? I mean, we are > partly copying the list-shrink trigger algorithm here, which I would prefer > to consider an implementation detail of the list type. > I second this, this is the kind of code I don't like to see Cython output. At the very least, one should #IFDEF it to only kick in with known good Python versions...it's not the kind of thing we'll automatically remember to check when Python 3.4 comes out. It should be possible to find a middle route somewhere though? (At the very least prefetch "pop" of the list type rather than use getattr every time.) Dag Sverre From dagss at student.matnat.uio.no Tue Nov 3 16:01:04 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 16:01:04 +0100 Subject: [Cython] runtests.py locks Message-ID: <4AF045B0.4090109@student.matnat.uio.no> I lost some time over what appears to be a problem in runtests.py; the line PartialTestResult.join_results(result, pickle.load(input)) never returns if and only if some test fails (!!) My current strategy is to remove the multiprocessing stuff locally...I'm not in a mood to spend more time on this now... Dag Sverre From dalcinl at gmail.com Tue Nov 3 16:01:27 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 3 Nov 2009 13:01:27 -0200 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> Message-ID: On Tue, Nov 3, 2009 at 7:20 AM, Robert Bradshaw wrote: > On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >> >>> There's >>> another regression, "cdef int complex" doesn't work anymore (our >>> structs used to support this, as does gcc, even if it's non-c99). >> >> Turns out this broke nearly all my own code :-( I'll give a proper >> solution a go. > > Thanks! It think it should be pretty easy (probably can use the type > name, rather than the math postfix marker to name the methods...) > > http://trac.cython.org/cython_trac/ticket/446 > Or use other approach for handling "int complex" types. Honestly, I did not take into account this use case. Mmm... I'm thinking that trying to support (floating) complex in C89 is not a good idea, some things are really hard to do (if ever possible)... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Nov 3 16:06:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 16:06:29 +0100 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> Message-ID: <4AF046F5.8010308@student.matnat.uio.no> Lisandro Dalcin wrote: > On Tue, Nov 3, 2009 at 7:20 AM, Robert Bradshaw > wrote: > >> On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: >> >> >>> Robert Bradshaw wrote: >>> >>> >>>> There's >>>> another regression, "cdef int complex" doesn't work anymore (our >>>> structs used to support this, as does gcc, even if it's non-c99). >>>> >>> Turns out this broke nearly all my own code :-( I'll give a proper >>> solution a go. >>> >> Thanks! It think it should be pretty easy (probably can use the type >> name, rather than the math postfix marker to name the methods...) >> >> http://trac.cython.org/cython_trac/ticket/446 >> >> > > Or use other approach for handling "int complex" types. Honestly, I > did not take into account this use case. > > Mmm... I'm thinking that trying to support (floating) complex in C89 > is not a good idea, some things are really hard to do (if ever > possible)... > Huh? Do you mean what's there already? Dag Sverre From dalcinl at gmail.com Tue Nov 3 16:08:58 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 3 Nov 2009 13:08:58 -0200 Subject: [Cython] optimizing list.pop In-Reply-To: <4AF03E1D.4000706@student.matnat.uio.no> References: <4AE48D6D.3080503@molden.no> <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> <4AF01903.8060909@behnel.de> <4AF03E1D.4000706@student.matnat.uio.no> Message-ID: On Tue, Nov 3, 2009 at 12:28 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Robert Bradshaw, 03.11.2009 10:22: >> >>> On Oct 25, 2009, at 11:53 AM, Lisandro Dalcin wrote: >>> >>>> I think Cython could generate a custom, inline function >>>> implementing l.pop([n]) and fast dispatch to it if the object is >>>> exactly a list ... >>>> >>> http://hg.cython.org/cython-devel/rev/2e3dda4a7d23 >>> >> >> Hmm, is this really worth the code duplication with CPython? I mean, we are >> partly copying the list-shrink trigger algorithm here, which I would prefer >> to consider an implementation detail of the list type. >> > I second this, this is the kind of code I don't like to see Cython > output. At the very least, one should #IFDEF it to only kick in with > known good Python versions...it's not the kind of thing we'll > automatically remember to check when Python 3.4 comes out. > > It should be possible to find a middle route somewhere though? (At the > very least prefetch "pop" of the list type rather than use getattr every > time.) > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > I agree with Stefan and Dag... Why not use PyList_GET_ITEM(L) + Py_INCREF() + PyList_SetSlice() ?? This way, you could easily support "L.pop(index)"... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Nov 3 16:12:44 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 3 Nov 2009 13:12:44 -0200 Subject: [Cython] More complex bugs In-Reply-To: <4AF046F5.8010308@student.matnat.uio.no> References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> Message-ID: On Tue, Nov 3, 2009 at 1:06 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Tue, Nov 3, 2009 at 7:20 AM, Robert Bradshaw >> wrote: >> >>> On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: >>> >>> >>>> Robert Bradshaw wrote: >>>> >>>> >>>>> There's >>>>> another regression, "cdef int complex" doesn't work anymore (our >>>>> structs used to support this, as does gcc, even if it's non-c99). >>>>> >>>> Turns out this broke nearly all my own code :-( I'll give a proper >>>> solution a go. >>>> >>> Thanks! It think it should be pretty easy (probably can use the type >>> name, rather than the math postfix marker to name the methods...) >>> >>> http://trac.cython.org/cython_trac/ticket/446 >>> >>> >> >> Or use other approach for handling "int complex" types. Honestly, I >> did not take into account this use case. >> >> Mmm... I'm thinking that trying to support (floating) complex in C89 >> is not a good idea, some things are really hard to do (if ever >> possible)... >> > Huh? Do you mean what's there already? > No, just the "Seamless" support... I mean, if the 'ccomplex' directive is not on, then make Cython generate an error when external ctypedefs are used, and any other situation where there is ambiguity... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Nov 3 16:42:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 16:42:55 +0100 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> Message-ID: <4AF04F7F.4010903@student.matnat.uio.no> Robert Bradshaw wrote: > On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: > > >> Robert Bradshaw wrote: >> >> >>> There's >>> another regression, "cdef int complex" doesn't work anymore (our >>> structs used to support this, as does gcc, even if it's non-c99). >>> >> Turns out this broke nearly all my own code :-( I'll give a proper >> solution a go. >> > > Thanks! It think it should be pretty easy (probably can use the type > name, rather than the math postfix marker to name the methods...) > I fixed the basic case of ctypedef double mydouble print some_mydouble * some_complex however external ctypedefs will still raise errors. It's not easy; nearly all calls rely on widest_type to decide their signature, and widest_type information is unavailable. I'll start a new thread. Dag Sverre From dagss at student.matnat.uio.no Tue Nov 3 16:58:53 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 16:58:53 +0100 Subject: [Cython] More complex bugs In-Reply-To: <4AF04F7F.4010903@student.matnat.uio.no> References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF04F7F.4010903@student.matnat.uio.no> Message-ID: <4AF0533D.5000509@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: >> >> >>> Robert Bradshaw wrote: >>> >>> >>>> There's >>>> another regression, "cdef int complex" doesn't work anymore (our >>>> structs used to support this, as does gcc, even if it's non-c99). >>>> >>> Turns out this broke nearly all my own code :-( I'll give a proper >>> solution a go. >>> >> >> Thanks! It think it should be pretty easy (probably can use the type >> name, rather than the math postfix marker to name the methods...) >> > I fixed the basic case of > > ctypedef double mydouble > print some_mydouble * some_complex > > however external ctypedefs will still raise errors. > > It's not easy; nearly all calls rely on widest_type to decide their > signature, and widest_type information is unavailable. I'll start a > new thread. I didn't start a new thread, it quickly degraded to rehashing. But in the specific case of complex, the problem is: a) The only reason stuff like "a * b" even works for external typedefs is because this can mean *different things* to the C compiler and Cython. I.e. to Cython "a" may be the widest type and to the C compiler "b" may be the widest type, but "a * b" is output anyhow so it doesn't matter. b) For complex types, with "__pyx_c_mul_mytype(a, b)", Cython defines the widening behaviour (and Cython is wrong!) I guess it's the same issue that we're facing with division. This is the kind of trouble we get for half-way allowing something impossible... Dag Sverre From dagss at student.matnat.uio.no Tue Nov 3 17:12:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 17:12:43 +0100 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> Message-ID: <4AF0567B.6070908@student.matnat.uio.no> Lisandro Dalcin wrote: > On Tue, Nov 3, 2009 at 1:06 PM, Dag Sverre Seljebotn > wrote: > >> Lisandro Dalcin wrote: >> >>> On Tue, Nov 3, 2009 at 7:20 AM, Robert Bradshaw >>> wrote: >>> >>> >>>> On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: >>>> >>>> >>>> >>>>> Robert Bradshaw wrote: >>>>> >>>>> >>>>> >>>>>> There's >>>>>> another regression, "cdef int complex" doesn't work anymore (our >>>>>> structs used to support this, as does gcc, even if it's non-c99). >>>>>> >>>>>> >>>>> Turns out this broke nearly all my own code :-( I'll give a proper >>>>> solution a go. >>>>> >>>>> >>>> Thanks! It think it should be pretty easy (probably can use the type >>>> name, rather than the math postfix marker to name the methods...) >>>> >>>> http://trac.cython.org/cython_trac/ticket/446 >>>> >>>> >>>> >>> Or use other approach for handling "int complex" types. Honestly, I >>> did not take into account this use case. >>> >>> Mmm... I'm thinking that trying to support (floating) complex in C89 >>> is not a good idea, some things are really hard to do (if ever >>> possible)... >>> >>> >> Huh? Do you mean what's there already? >> >> > > No, just the "Seamless" support... I mean, if the 'ccomplex' directive > is not on, then make Cython generate an error when external ctypedefs > are used, and any other situation where there is ambiguity... > Hmm. I don't think C89 complex is really the problem. There's plenty of similar problems like the functions for integer division and (in future) overloaded functions. Dag Sverre From dalcinl at gmail.com Tue Nov 3 17:49:12 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 3 Nov 2009 14:49:12 -0200 Subject: [Cython] More complex bugs In-Reply-To: <4AF0567B.6070908@student.matnat.uio.no> References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> <4AF0567B.6070908@student.matnat.uio.no> Message-ID: On Tue, Nov 3, 2009 at 2:12 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Tue, Nov 3, 2009 at 1:06 PM, Dag Sverre Seljebotn >> wrote: >> >>> Lisandro Dalcin wrote: >>> >>>> On Tue, Nov 3, 2009 at 7:20 AM, Robert Bradshaw >>>> wrote: >>>> >>>> >>>>> On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: >>>>> >>>>> >>>>> >>>>>> Robert Bradshaw wrote: >>>>>> >>>>>> >>>>>> >>>>>>> There's >>>>>>> another regression, "cdef int complex" doesn't work anymore (our >>>>>>> structs used to support this, as does gcc, even if it's non-c99). >>>>>>> >>>>>>> >>>>>> Turns out this broke nearly all my own code :-( I'll give a proper >>>>>> solution a go. >>>>>> >>>>>> >>>>> Thanks! It think it should be pretty easy (probably can use the type >>>>> name, rather than the math postfix marker to name the methods...) >>>>> >>>>> http://trac.cython.org/cython_trac/ticket/446 >>>>> >>>>> >>>>> >>>> Or use other approach for handling "int complex" types. Honestly, I >>>> did not take into account this use case. >>>> >>>> Mmm... I'm thinking that trying to support (floating) complex in C89 >>>> is not a good idea, some things are really hard to do (if ever >>>> possible)... >>>> >>>> >>> Huh? Do you mean what's there already? >>> >>> >> >> No, just the "Seamless" support... I mean, if the 'ccomplex' directive >> is not on, then make Cython generate an error when external ctypedefs >> are used, and any other situation where there is ambiguity... >> > Hmm. I don't think C89 complex is really the problem. There's plenty of > similar problems like the functions for integer division and (in future) > overloaded functions. > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Do you remember that I advocated for "ctypedef some int MyInt" or something similar? If right now we had that, Cython could reject some things, like binops on different "ctypedef some .." types, requiring users to introduce casts to resolve the ambiguity... A said it before, and I say it again: second-guessing is a bad thing (or, in Python's Zen words: "In the face of ambiguity, refuse the temptation to guess.") -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Nov 3 18:06:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 18:06:29 +0100 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> <4AEF38CD.20309@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> <4AF0567B.6070908@student.matnat.uio.no> Message-ID: <4AF06315.8040005@student.matnat.uio.no> Lisandro Dalcin wrote: > On Tue, Nov 3, 2009 at 2:12 PM, Dag Sverre Seljebotn > wrote: > >> Lisandro Dalcin wrote: >> >>> On Tue, Nov 3, 2009 at 1:06 PM, Dag Sverre Seljebotn >>> wrote: >>> >>> >>>> Lisandro Dalcin wrote: >>>> >>>> >>>>> On Tue, Nov 3, 2009 at 7:20 AM, Robert Bradshaw >>>>> wrote: >>>>> >>>>> >>>>> >>>>>> On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Robert Bradshaw wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> There's >>>>>>>> another regression, "cdef int complex" doesn't work anymore (our >>>>>>>> structs used to support this, as does gcc, even if it's non-c99). >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Turns out this broke nearly all my own code :-( I'll give a proper >>>>>>> solution a go. >>>>>>> >>>>>>> >>>>>>> >>>>>> Thanks! It think it should be pretty easy (probably can use the type >>>>>> name, rather than the math postfix marker to name the methods...) >>>>>> >>>>>> http://trac.cython.org/cython_trac/ticket/446 >>>>>> >>>>>> >>>>>> >>>>>> >>>>> Or use other approach for handling "int complex" types. Honestly, I >>>>> did not take into account this use case. >>>>> >>>>> Mmm... I'm thinking that trying to support (floating) complex in C89 >>>>> is not a good idea, some things are really hard to do (if ever >>>>> possible)... >>>>> >>>>> >>>>> >>>> Huh? Do you mean what's there already? >>>> >>>> >>>> >>> No, just the "Seamless" support... I mean, if the 'ccomplex' directive >>> is not on, then make Cython generate an error when external ctypedefs >>> are used, and any other situation where there is ambiguity... >>> >>> >> Hmm. I don't think C89 complex is really the problem. There's plenty of >> similar problems like the functions for integer division and (in future) >> overloaded functions. >> >> Dag Sverre >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> >> > > Do you remember that I advocated for "ctypedef some int MyInt" or > something similar? If right now we had that, Cython could reject some > things, like binops on different "ctypedef some .." types, requiring > users to introduce casts to resolve the ambiguity... A said it before, > and I say it again: second-guessing is a bad thing (or, in Python's > Zen words: "In the face of ambiguity, refuse the temptation to > guess.") > I remember very well, and I've been thinking about it the whole day. I wholeheartedly agree with you that the situation is not good. Another, perhaps more "backwards-compatible" approach is (backwards-compatible in the following sense: It will break code that should be broken and fixed, and not suddenly introduce bugs that are hard to find): a) Tighten down everything so that explicit casts are needed even for stuff like this: cdef long i cdef external_typedef_int j print i / j # not allowed, must do i / j b) Add "exact" instead of "some" cdef extern ...: ctypedef long! mylong Code is then inserted during module loading which ensures sizeof(long) == sizeof(mylong). However I seem to remember that this was rejected as well by Robert (who, for some reason I can't understand, seems to think the current situation is OK.) Dag Sverre From dalcinl at gmail.com Tue Nov 3 19:15:32 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 3 Nov 2009 16:15:32 -0200 Subject: [Cython] More complex bugs In-Reply-To: <4AF06315.8040005@student.matnat.uio.no> References: <4AE86598.10204@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> <4AF0567B.6070908@student.matnat.uio.no> <4AF06315.8040005@student.matnat.uio.no> Message-ID: On Tue, Nov 3, 2009 at 3:06 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Tue, Nov 3, 2009 at 2:12 PM, Dag Sverre Seljebotn >> wrote: >> >>> Lisandro Dalcin wrote: >>> >>>> On Tue, Nov 3, 2009 at 1:06 PM, Dag Sverre Seljebotn >>>> wrote: >>>> >>>> >>>>> Lisandro Dalcin wrote: >>>>> >>>>> >>>>>> On Tue, Nov 3, 2009 at 7:20 AM, Robert Bradshaw >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> On Nov 3, 2009, at 12:45 AM, Dag Sverre Seljebotn wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Robert Bradshaw wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> There's >>>>>>>>> another regression, "cdef int complex" doesn't work anymore (our >>>>>>>>> structs used to support this, as does gcc, even if it's non-c99). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Turns out this broke nearly all my own code :-( I'll give a proper >>>>>>>> solution a go. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Thanks! It think it should be pretty easy (probably can use the type >>>>>>> name, rather than the math postfix marker to name the methods...) >>>>>>> >>>>>>> http://trac.cython.org/cython_trac/ticket/446 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> Or use other approach for handling "int complex" types. Honestly, I >>>>>> did not take into account this use case. >>>>>> >>>>>> Mmm... I'm thinking that trying to support (floating) complex in C89 >>>>>> is not a good idea, some things are really hard to do (if ever >>>>>> possible)... >>>>>> >>>>>> >>>>>> >>>>> Huh? Do you mean what's there already? >>>>> >>>>> >>>>> >>>> No, just the "Seamless" support... I mean, if the 'ccomplex' directive >>>> is not on, then make Cython generate an error when external ctypedefs >>>> are used, and any other situation where there is ambiguity... >>>> >>>> >>> Hmm. I don't think C89 complex is really the problem. There's plenty of >>> similar problems like the functions for integer division and (in future) >>> overloaded functions. >>> >>> Dag Sverre >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >>> >> >> Do you remember that I advocated for "ctypedef some int MyInt" or >> something similar? If right now we had that, Cython could reject some >> things, like binops on different "ctypedef some .." types, requiring >> users to introduce casts to resolve the ambiguity... A said it before, >> and I say it again: second-guessing is a bad thing (or, in Python's >> Zen words: "In the face of ambiguity, refuse the temptation to >> guess.") >> > I remember very well, and I've been thinking about it the whole day. I > wholeheartedly agree with you that the situation is not good. > > Another, perhaps more "backwards-compatible" approach is > (backwards-compatible in the following sense: It will break code that > should be broken and fixed, and not suddenly introduce bugs that are > hard to find): > > ?a) Tighten down everything so that explicit casts are needed even for > stuff like this: > > cdef long i > cdef external_typedef_int j > print i / j # not allowed, must do i / j > OK, +1 on this... > > ?b) Add "exact" instead of "some" > cdef extern ...: > ? ?ctypedef long! ?mylong > Well, I'm still thinking that "ctypedef int MyInt" should actually mean exact... However, I realize that this is going to be a source of code breakage. > Code is then inserted during module loading which ensures sizeof(long) > == sizeof(mylong). > Mmm... Do "ensure" means a import failure? Perhaps a warning would better? Perhaps a mismatch does not necessarily means the code is broken... > However I seem to remember that this was rejected as well by Robert > (who, for some reason I can't understand, seems to think the current > situation is OK.) > Perhaps we were not clear enough at that time, and Robert rejected the idea because it introduced new syntax? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Tue Nov 3 19:26:26 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 10:26:26 -0800 Subject: [Cython] runtests.py locks In-Reply-To: <4AF045B0.4090109@student.matnat.uio.no> References: <4AF045B0.4090109@student.matnat.uio.no> Message-ID: On Nov 3, 2009, at 7:01 AM, Dag Sverre Seljebotn wrote: > I lost some time over what appears to be a problem in runtests.py; > the line > > PartialTestResult.join_results(result, pickle.load(input)) > > never returns if and only if some test fails (!!) > > My current strategy is to remove the multiprocessing stuff > locally...I'm > not in a mood to spend more time on this now... I've been having the same issue, with little luck chasing it down (though admittedly little time put into it). Try the recently introduced --no-fork option... - Robert From robertwb at math.washington.edu Tue Nov 3 19:27:57 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 10:27:57 -0800 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> Message-ID: On Nov 3, 2009, at 3:12 AM, David Cournapeau wrote: > On Tue, Nov 3, 2009 at 2:35 PM, Robert Bradshaw > wrote: > >> >> if 'setuptools' in sys.modules >> >> is this always safe? > > What kind of unsafe behavior would you have in mind ? How would you know, in general, that setuptools has not previously been imported? - Robert From dagss at student.matnat.uio.no Tue Nov 3 19:40:04 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 03 Nov 2009 19:40:04 +0100 Subject: [Cython] runtests.py locks In-Reply-To: References: <4AF045B0.4090109@student.matnat.uio.no> Message-ID: <4AF07904.1020204@student.matnat.uio.no> Robert Bradshaw wrote: > On Nov 3, 2009, at 7:01 AM, Dag Sverre Seljebotn wrote: > > >> I lost some time over what appears to be a problem in runtests.py; >> the line >> >> PartialTestResult.join_results(result, pickle.load(input)) >> >> never returns if and only if some test fails (!!) >> >> My current strategy is to remove the multiprocessing stuff >> locally...I'm >> not in a mood to spend more time on this now... >> > > I've been having the same issue, with little luck chasing it down > (though admittedly little time put into it). Try the recently > introduced --no-fork option... > > How about enabling this for version 2.6+ only and use multiprocessing? Dag Sverre From stefan_ml at behnel.de Tue Nov 3 19:41:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 03 Nov 2009 19:41:03 +0100 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> Message-ID: <4AF0793F.6040509@behnel.de> Robert Bradshaw, 03.11.2009 19:27: > On Nov 3, 2009, at 3:12 AM, David Cournapeau wrote: > >> On Tue, Nov 3, 2009 at 2:35 PM, Robert Bradshaw >> wrote: >> >>> if 'setuptools' in sys.modules >>> >>> is this always safe? >> What kind of unsafe behavior would you have in mind ? > > How would you know, in general, that setuptools has not previously > been imported? If it has been imported (or any of its submodules), it will appear in sys.modules, unless someone deliberately removed it from there after the import. So I'd say it's safe to test for it being in sys.modules to figure out if it's being used in a setup.py script or not. Note, however, that there's also the "distribute" package, which is mostly setuptools compatible, minus a couple of its bugs and quirks. http://pypi.python.org/pypi/distribute If we special case setuptools, we might want to support that, too - although I'm not sure how far this special casing business should go. Stefan From stefan_ml at behnel.de Tue Nov 3 20:00:19 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 03 Nov 2009 20:00:19 +0100 Subject: [Cython] runtests.py locks In-Reply-To: <4AF07904.1020204@student.matnat.uio.no> References: <4AF045B0.4090109@student.matnat.uio.no> <4AF07904.1020204@student.matnat.uio.no> Message-ID: <4AF07DC3.1060301@behnel.de> Dag Sverre Seljebotn, 03.11.2009 19:40: > Robert Bradshaw wrote: >> On Nov 3, 2009, at 7:01 AM, Dag Sverre Seljebotn wrote: >> >>> I lost some time over what appears to be a problem in runtests.py; >>> the line >>> >>> PartialTestResult.join_results(result, pickle.load(input)) >>> >>> never returns if and only if some test fails (!!) Not true in general, but I also noticed that this can happen in some cases. I guess it's a problem when a test crashes the Python interpreter, for example. >>> My current strategy is to remove the multiprocessing stuff >>> locally...I'm not in a mood to spend more time on this now... >>> >> I've been having the same issue, with little luck chasing it down >> (though admittedly little time put into it). Try the recently >> introduced --no-fork option... >> > How about enabling this for version 2.6+ only and use multiprocessing? What I would like to see is a way to gather the output of gcc, and then let the tests run in parallel using whatever support fits best (note that multiprocessing has the advantage of being available for Windows, and the disadvantage of requiring Python 2.6+). To get there, we'd have to patch into distutils, and that's far from trivial as there isn't really a hook to do that. Another possibility would be to divert stdout/stderr into a pipe when we fork. Does anyone know how to do that in Python? Stefan From robertwb at math.washington.edu Tue Nov 3 20:20:00 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 11:20:00 -0800 Subject: [Cython] optimizing list.pop In-Reply-To: References: <4AE48D6D.3080503@molden.no> <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> <4AF01903.8060909@behnel.de> <4AF03E1D.4000706@student.matnat.uio.no> Message-ID: On Nov 3, 2009, at 7:08 AM, Lisandro Dalcin wrote: > On Tue, Nov 3, 2009 at 12:28 PM, Dag Sverre Seljebotn > wrote: >> Stefan Behnel wrote: >>> Robert Bradshaw, 03.11.2009 10:22: >>> >>>> On Oct 25, 2009, at 11:53 AM, Lisandro Dalcin wrote: >>>> >>>>> I think Cython could generate a custom, inline function >>>>> implementing l.pop([n]) and fast dispatch to it if the object is >>>>> exactly a list ... >>>>> >>>> http://hg.cython.org/cython-devel/rev/2e3dda4a7d23 >>>> >>> >>> Hmm, is this really worth the code duplication with CPython? I >>> mean, we are >>> partly copying the list-shrink trigger algorithm here, which I >>> would prefer >>> to consider an implementation detail of the list type. >>> >> I second this, this is the kind of code I don't like to see Cython >> output. At the very least, one should #IFDEF it to only kick in with >> known good Python versions...it's not the kind of thing we'll >> automatically remember to check when Python 3.4 comes out. I concede that we're partially copying the (half-line) list-shrink trigger algorithm here, but if that changed nothing would break. It's not like we're using any secret APIs. >> It should be possible to find a middle route somewhere though? (At >> the >> very least prefetch "pop" of the list type rather than use getattr >> every >> time.) >> >> Dag Sverre >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > I agree with Stefan and Dag... For the no argument case, there really is hardly any code duplication. Also, we don't do any reallocation or anything, we just note that if the number of items get small enough we dispatch to the generic case. I think the resulting 5x speedup is worth it. > Why not use PyList_GET_ITEM(L) + Py_INCREF() + PyList_SetSlice() ?? > This way, you could easily support "L.pop(index)"... Have you seen the code for PyList_SetSlice? It's way more generic (and complicated) than is needed in this case. And if one were to use the methods mentioned above, one would still need all the bounds checking code that's there (the only savings would be calling PyList_SetSlice rather than having a manual loop, at the expense of extra memory management). Again, I'd argue it's worth an 10x speedup. Ideally there should be a PyList_Pop method in the C API, and more efficient than the one that's there, but that's not the case. - Robert ----------------------------------------------------------- In [1]: from benchmark_pop import * In [2]: L = range(10**7) In [3]: time cython_pop(L, 10**7) CPU times: user 0.15 s, sys: 0.00 s, total: 0.16 s Wall time: 0.16 s In [5]: L = range(10**7) In [6]: time python_pop(L, 10**7) CPU times: user 0.78 s, sys: 0.00 s, total: 0.78 s Wall time: 0.78 s In [8]: L = range(10**7) In [9]: time cython_pop_index(L, 10**7-1, -2) CPU times: user 0.21 s, sys: 0.00 s, total: 0.21 s Wall time: 0.21 s In [11]: L = range(10**7) In [12]: time python_pop_index(L, 10**7-1, -2) CPU times: user 2.38 s, sys: 0.01 s, total: 2.39 s Wall time: 2.40 s -------------- next part -------------- A non-text attachment was scrubbed... Name: benchmark_pop.pyx Type: application/octet-stream Size: 485 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20091103/d41218e6/attachment.obj From robertwb at math.washington.edu Tue Nov 3 21:03:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 12:03:22 -0800 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> <4AF0567B.6070908@student.matnat.uio.no> <4AF06315.8040005@student.matnat.uio.no> Message-ID: <0D7E932E-7E8C-47ED-8900-1898AA95D7FC@math.washington.edu> On Nov 3, 2009, at 10:15 AM, Lisandro Dalcin wrote: > On Tue, Nov 3, 2009 at 3:06 PM, Dag Sverre Seljebotn > wrote: >> Lisandro Dalcin wrote: >> >>> >>> Do you remember that I advocated for "ctypedef some int MyInt" or >>> something similar? If right now we had that, Cython could reject >>> some >>> things, like binops on different "ctypedef some .." types, requiring >>> users to introduce casts to resolve the ambiguity... A said it >>> before, >>> and I say it again: second-guessing is a bad thing (or, in Python's >>> Zen words: "In the face of ambiguity, refuse the temptation to >>> guess.") >>> >> I remember very well, and I've been thinking about it the whole >> day. I >> wholeheartedly agree with you that the situation is not good. I think in generally this is an unsoveable problem as the Cython stage completes before the C stage starts (and generating code for all possible permutations isn't feasible in general). This means also that the author of the code doesn't (necessarily) know the size of the external types before C compile time, and they may change from platform to platform. We're still going to have to deal with this for function overloading (and there it's even nastier, as sizeof(a) >= sizeof(b) has different semantics from sizeof(a) == sizeof(b)). >> Another, perhaps more "backwards-compatible" approach is >> (backwards-compatible in the following sense: It will break code that >> should be broken and fixed, and not suddenly introduce bugs that are >> hard to find): >> >> a) Tighten down everything so that explicit casts are needed even >> for >> stuff like this: >> >> cdef long i >> cdef external_typedef_int j >> print i / j # not allowed, must do i / j >> > > OK, +1 on this... I think this may be too much of a backwards incompatible change. Would one then have to do cdef external_typedef_int j print j / 2 print j / 2 ? The phrase "should be broken and fixed" is somewhat subjective. Care should be taken when marking something that is potentially broken as something that is definitely broken. At the very least we should start with a directive that emits warnings or errors to see how much stuff it breaks and how constraining it is to work with first. Note that if we require casts, I can see people putting them in just to get their code to compile (without thinking of the full ramifications--or even if they do they might not know which of the two is actually wider, or it may be different on their system rather than their users). If this causes bugs down the line, it will be harder to track down than if it simply emitted an (optional or not) warning at that point. >> b) Add "exact" instead of "some" >> cdef extern ...: >> ctypedef long! mylong >> > > Well, I'm still thinking that "ctypedef int MyInt" should actually > mean exact... However, I realize that this is going to be a source of > code breakage. Lots of code breakage in Sage I could imagine. (And I don't think any of that code is actually broken, its stuff like flags (boolean ops count as arithmatic) or the mpfr precision type, etc. where size may be platform dependent but doesn't really need to be known). >> Code is then inserted during module loading which ensures >> sizeof(long) >> == sizeof(mylong). >> > > Mmm... Do "ensure" means a import failure? Perhaps a warning would > better? Perhaps a mismatch does not necessarily means the code is > broken... If you really want this (and it would seem to create highly non- portable code if you did, which mostly defeats the purpose of using external types in the first place) then you could stick an assert sizeof(long) == sizeof(mylong) right at the top of your module. Much more explicit, and needs no extra syntax. >> However I seem to remember that this was rejected as well by Robert >> (who, for some reason I can't understand, seems to think the current >> situation is OK.) > > Perhaps we were not clear enough at that time, and Robert rejected the > idea because it introduced new syntax? If I recall, my two primary concerns were that it was too large of a backwards incompatible change (at least to make it so by default right off), and that we hadn't come up with the right syntax yet. In particular, I really liked the idea of partially ranked types, so one could have short <= int <= long <= long long int16 < int32 < int64 int16 <= short ... which I think would make things much less painful, but we have yet to come up with a good way of specifying that. I think we're to the point that only ranking matters, not exact size. I'm more in favor of being able to specify a type as unranked, than have all extern types be unranked by default and having a keyword to specify types as exact. - Robert From dalcinl at gmail.com Tue Nov 3 23:37:41 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 3 Nov 2009 19:37:41 -0300 Subject: [Cython] optimizing list.pop In-Reply-To: References: <4AE48D6D.3080503@molden.no> <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> <4AF01903.8060909@behnel.de> <4AF03E1D.4000706@student.matnat.uio.no> Message-ID: On Tue, Nov 3, 2009 at 4:20 PM, Robert Bradshaw wrote: > On Nov 3, 2009, at 7:08 AM, Lisandro Dalcin wrote: > >> On Tue, Nov 3, 2009 at 12:28 PM, Dag Sverre Seljebotn >> wrote: >>> >>> Stefan Behnel wrote: >>>> >>>> Robert Bradshaw, 03.11.2009 10:22: >>>> >>>>> On Oct 25, 2009, at 11:53 AM, Lisandro Dalcin wrote: >>>>> >>>>>> I think Cython could generate a custom, inline function >>>>>> implementing l.pop([n]) and fast dispatch to it if the object is >>>>>> exactly a list ... >>>>>> >>>>> http://hg.cython.org/cython-devel/rev/2e3dda4a7d23 >>>>> >>>> >>>> Hmm, is this really worth the code duplication with CPython? I mean, we >>>> are >>>> partly copying the list-shrink trigger algorithm here, which I would >>>> prefer >>>> to consider an implementation detail of the list type. >>>> >>> I second this, this is the kind of code I don't like to see Cython >>> output. At the very least, one should #IFDEF it to only kick in with >>> known good Python versions...it's not the kind of thing we'll >>> automatically remember to check when Python 3.4 comes out. > > I concede that we're partially copying the (half-line) list-shrink trigger > algorithm here, but if that changed nothing would break. It's not like we're > using any secret APIs. > So L->allocated is public API? If you can get rid of that, I think no one would object your patch... >>> It should be possible to find a middle route somewhere though? (At the >>> very least prefetch "pop" of the list type rather than use getattr every >>> time.) >>> >>> Dag Sverre >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >> I agree with Stefan and Dag... > > For the no argument case, there really is hardly any code duplication. Also, > we don't do any reallocation or anything, we just note that if the number of > items get small enough we dispatch to the generic case. I think the > resulting 5x speedup is worth it. > >> Why not use PyList_GET_ITEM(L) + Py_INCREF() + PyList_SetSlice() ?? >> This way, you could easily support "L.pop(index)"... > > Have you seen the code for PyList_SetSlice? It's way more generic (and > complicated) than is needed in this case. And if one were to use the methods > mentioned above, one would still need all the bounds checking code that's > there (the only savings would be calling PyList_SetSlice rather than having > a manual loop, at the expense of extra memory management). Again, I'd argue > it's worth an 10x speedup. > Robert, "premature optimization is the root of all evil"... You are trying to be faster than CPython!! list.pop() is implemented more or less with PyList_SetSlice !! > Ideally there should be a PyList_Pop method in the C API, and more efficient > than the one that's there, but that's not the case. > Yes, of course... And if it were, it would likely be implemented with PyList_SetSlice, as the list.pop() method is currently is... Or core CPython devs are truly lazy, or you Robert are wasting valuable time in micro-optimizations ... > ----------------------------------------------------------- > > In [1]: from benchmark_pop import * > > In [2]: L = range(10**7) > > In [3]: time cython_pop(L, 10**7) > CPU times: user 0.15 s, sys: 0.00 s, total: 0.16 s > Wall time: 0.16 s > > In [5]: L = range(10**7) > > In [6]: time python_pop(L, 10**7) > CPU times: user 0.78 s, sys: 0.00 s, total: 0.78 s > Wall time: 0.78 s > > In [8]: L = range(10**7) > > In [9]: time cython_pop_index(L, 10**7-1, -2) > CPU times: user 0.21 s, sys: 0.00 s, total: 0.21 s > Wall time: 0.21 s > > In [11]: L = range(10**7) > > In [12]: time python_pop_index(L, 10**7-1, -2) > CPU times: user 2.38 s, sys: 0.01 s, total: 2.39 s > Wall time: 2.40 s > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Wed Nov 4 00:45:35 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 15:45:35 -0800 Subject: [Cython] optimizing list.pop In-Reply-To: References: <4AE48D6D.3080503@molden.no> <6ED2907C-490E-41B5-8554-9531960D9433@math.washington.edu> <4AF01903.8060909@behnel.de> <4AF03E1D.4000706@student.matnat.uio.no> Message-ID: <804ED31B-0BE0-4B84-8C57-342DBD783E62@math.washington.edu> On Nov 3, 2009, at 2:37 PM, Lisandro Dalcin wrote: > On Tue, Nov 3, 2009 at 4:20 PM, Robert Bradshaw > wrote: >> I concede that we're partially copying the (half-line) list-shrink >> trigger >> algorithm here, but if that changed nothing would break. It's not >> like we're >> using any secret APIs. >> > > So L->allocated is public API? If you can get rid of that, I think no > one would object your patch... Well, it's clearly documented in listobject.h, and we didn't have to import any special headers to get it (like we do for frame objects and exception handling). There's nothing clean and simple to use instead, at least not without giving up nearly all the potential savings. >> For the no argument case, there really is hardly any code >> duplication. Also, >> we don't do any reallocation or anything, we just note that if the >> number of >> items get small enough we dispatch to the generic case. I think the >> resulting 5x speedup is worth it. >> >>> Why not use PyList_GET_ITEM(L) + Py_INCREF() + PyList_SetSlice() ?? >>> This way, you could easily support "L.pop(index)"... >> >> Have you seen the code for PyList_SetSlice? It's way more generic >> (and >> complicated) than is needed in this case. And if one were to use >> the methods >> mentioned above, one would still need all the bounds checking code >> that's >> there (the only savings would be calling PyList_SetSlice rather >> than having >> a manual loop, at the expense of extra memory management). Again, >> I'd argue >> it's worth an 10x speedup. >> > > Robert, "premature optimization is the root of all evil"... You are > trying to be faster than CPython!! list.pop() is implemented more or > less with PyList_SetSlice !! You say this as if it's a bad thing to be faster than CPython :) >> Ideally there should be a PyList_Pop method in the C API, and more >> efficient >> than the one that's there, but that's not the case. > > Yes, of course... And if it were, it would likely be implemented with > PyList_SetSlice, as the list.pop() method is currently is... Or core > CPython devs are truly lazy, or you Robert are wasting valuable time > in micro-optimizations ... Well, at least I spent less time on that micro-optimization than answering these emails :). (Not that I think that either is a waste.) My hunch is that since there was no (natural) way to pop without a Python function call, and the attribute lookup and argument unpacking took more time each than the PyList_SetSlice code overhead, there was no incentive for the core CPython devs to put any time into optimizing this, if it even crossed their radar. - Robert From cournape at gmail.com Wed Nov 4 01:13:39 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 4 Nov 2009 09:13:39 +0900 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> Message-ID: <5b8d13220911031613x21104e13iffbfab7234da8970@mail.gmail.com> On Wed, Nov 4, 2009 at 3:27 AM, Robert Bradshaw wrote: > > How would you know, in general, that setuptools has not previously > been imported? The reason for this line is precisely to detect whether setuptools has been previously imported or not. David From robertwb at math.washington.edu Wed Nov 4 01:29:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 16:29:53 -0800 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <5b8d13220911031613x21104e13iffbfab7234da8970@mail.gmail.com> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> <5b8d13220911031613x21104e13iffbfab7234da8970@mail.gmail.com> Message-ID: <30586B4A-3108-478F-A50F-092A224550AE@math.washington.edu> On Nov 3, 2009, at 4:13 PM, David Cournapeau wrote: > On Wed, Nov 4, 2009 at 3:27 AM, Robert Bradshaw > wrote: > >> >> How would you know, in general, that setuptools has not previously >> been imported? > > The reason for this line is precisely to detect whether setuptools has > been previously imported or not. Yes, I understand that. I should have been more clear--what makes me wary is that you are departing from normal behavior if, anywhere in the interpreters past history, a certain module has been imported. Now maybe (probably) I'm missing something here, does "import setuptools" in your setupegg.py do something more than just pass a flag to setup.py? If so, I'd really like to understand the magic. - Robert From robert.kern at gmail.com Wed Nov 4 02:51:38 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 03 Nov 2009 19:51:38 -0600 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <30586B4A-3108-478F-A50F-092A224550AE@math.washington.edu> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> <5b8d13220911031613x21104e13iffbfab7234da8970@mail.gmail.com> <30586B4A-3108-478F-A50F-092A224550AE@math.washington.edu> Message-ID: On 2009-11-03 18:29 PM, Robert Bradshaw wrote: > On Nov 3, 2009, at 4:13 PM, David Cournapeau wrote: > >> On Wed, Nov 4, 2009 at 3:27 AM, Robert Bradshaw >> wrote: >> >>> >>> How would you know, in general, that setuptools has not previously >>> been imported? >> >> The reason for this line is precisely to detect whether setuptools has >> been previously imported or not. > > Yes, I understand that. I should have been more clear--what makes me > wary is that you are departing from normal behavior if, anywhere in > the interpreters past history, a certain module has been imported. > > Now maybe (probably) I'm missing something here, does "import > setuptools" in your setupegg.py do something more than just pass a > flag to setup.py? If so, I'd really like to understand the magic. Importing setuptools automatically monkeypatches a couple of things in distutils and activates setuptools' features. The "if 'setuptools' in sys.modules:" check is the safest way to optionally provide those extra setup() options only when the person installing the package explicitly uses easy_install or setupegg.py. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robertwb at math.washington.edu Wed Nov 4 03:04:34 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 3 Nov 2009 18:04:34 -0800 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> <5b8d13220911031613x21104e13iffbfab7234da8970@mail.gmail.com> <30586B4A-3108-478F-A50F-092A224550AE@math.washington.edu> Message-ID: <2124F20F-8189-45DA-9532-1BC04FEDA4F1@math.washington.edu> On Nov 3, 2009, at 5:51 PM, Robert Kern wrote: > On 2009-11-03 18:29 PM, Robert Bradshaw wrote: >> On Nov 3, 2009, at 4:13 PM, David Cournapeau wrote: >> >>> On Wed, Nov 4, 2009 at 3:27 AM, Robert Bradshaw >>> wrote: >>> >>>> >>>> How would you know, in general, that setuptools has not previously >>>> been imported? >>> >>> The reason for this line is precisely to detect whether setuptools >>> has >>> been previously imported or not. >> >> Yes, I understand that. I should have been more clear--what makes me >> wary is that you are departing from normal behavior if, anywhere in >> the interpreters past history, a certain module has been imported. >> >> Now maybe (probably) I'm missing something here, does "import >> setuptools" in your setupegg.py do something more than just pass a >> flag to setup.py? If so, I'd really like to understand the magic. > > Importing setuptools automatically monkeypatches a couple of things > in distutils > and activates setuptools' features. The "if 'setuptools' in > sys.modules:" check > is the safest way to optionally provide those extra setup() options > only when > the person installing the package explicitly uses easy_install or > setupegg.py. Ah, that is exactly the kind of thing I was wondering about. In this case I'm +1 for the implementation. I agree with Stefan that we should probably support distribute as well. - Robert From dalcinl at gmail.com Wed Nov 4 05:11:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 4 Nov 2009 01:11:48 -0300 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> Message-ID: On Sun, Nov 1, 2009 at 5:32 AM, David Cournapeau wrote: > Hi, > > I needed to be able to build an egg for cython on windows, and thought > it could be interesting to integrate it to cython sources. Besides an > egg, it provides cython as a console_script, which is easier to handle > on windows. > > The patch is minimal and non invasive, as by default, setuptools is > *not* run if you use setup.py. I attached the patch here (I can't > connect to trac ATM), > Mmm... your patch add a new "setuptools_main()" entry point, likely just because you need main(command_line=1) and not the default to 0... Do any of Cython devs know if why that defaults to 0? If there is not good reason, we could change the default to 1... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From cournape at gmail.com Wed Nov 4 09:32:07 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 4 Nov 2009 17:32:07 +0900 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> Message-ID: <5b8d13220911040032r76c282ahab301ba4cdba5e23@mail.gmail.com> On Wed, Nov 4, 2009 at 1:11 PM, Lisandro Dalcin wrote: > > Mmm... your patch add a new "setuptools_main()" entry point, likely > just because you need main(command_line=1) and not the default to 0... Yes, that's correct. > Do any of Cython devs know if why that defaults to 0? If there is not > good reason, we could change the default to 1... I think it is cleared to have a wrapper for the tools which intend to be used as entry points nonetheless - you may want to deal with things differently in your main letter, having an explicit function for this helps you doing that. That's not a fundamental issue, obviously. cheers, David From seb.binet at gmail.com Wed Nov 4 11:51:59 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Wed, 4 Nov 2009 11:51:59 +0100 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <2124F20F-8189-45DA-9532-1BC04FEDA4F1@math.washington.edu> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <2124F20F-8189-45DA-9532-1BC04FEDA4F1@math.washington.edu> Message-ID: <200911041151.59460.binet@cern.ch> On Wednesday 04 November 2009 03:04:34 Robert Bradshaw wrote: > On Nov 3, 2009, at 5:51 PM, Robert Kern wrote: > > On 2009-11-03 18:29 PM, Robert Bradshaw wrote: > >> On Nov 3, 2009, at 4:13 PM, David Cournapeau wrote: > >>> On Wed, Nov 4, 2009 at 3:27 AM, Robert Bradshaw > >>> > >>> wrote: > >>>> How would you know, in general, that setuptools has not previously > >>>> been imported? > >>> > >>> The reason for this line is precisely to detect whether setuptools > >>> has > >>> been previously imported or not. > >> > >> Yes, I understand that. I should have been more clear--what makes me > >> wary is that you are departing from normal behavior if, anywhere in > >> the interpreters past history, a certain module has been imported. > >> > >> Now maybe (probably) I'm missing something here, does "import > >> setuptools" in your setupegg.py do something more than just pass a > >> flag to setup.py? If so, I'd really like to understand the magic. > > > > Importing setuptools automatically monkeypatches a couple of things > > in distutils > > and activates setuptools' features. The "if 'setuptools' in > > sys.modules:" check > > is the safest way to optionally provide those extra setup() options > > only when > > the person installing the package explicitly uses easy_install or > > setupegg.py. > > Ah, that is exactly the kind of thing I was wondering about. In this > case I'm +1 for the implementation. I agree with Stefan that we should > probably support distribute as well. for ease of migration, distribute also puts (among other things) a 'setuptools' entry in sys.modules. so nothing else to do on the cython side. cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From seb.binet at gmail.com Wed Nov 4 17:13:51 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Wed, 4 Nov 2009 17:13:51 +0100 Subject: [Cython] runtests.py locks In-Reply-To: <4AF07DC3.1060301@behnel.de> References: <4AF045B0.4090109@student.matnat.uio.no> <4AF07904.1020204@student.matnat.uio.no> <4AF07DC3.1060301@behnel.de> Message-ID: <200911041713.51743.binet@cern.ch> On Tuesday 03 November 2009 20:00:19 Stefan Behnel wrote: > Dag Sverre Seljebotn, 03.11.2009 19:40: > > Robert Bradshaw wrote: > >> On Nov 3, 2009, at 7:01 AM, Dag Sverre Seljebotn wrote: > >>> I lost some time over what appears to be a problem in runtests.py; > >>> the line > >>> > >>> PartialTestResult.join_results(result, pickle.load(input)) > >>> > >>> never returns if and only if some test fails (!!) > > Not true in general, but I also noticed that this can happen in some cases. > I guess it's a problem when a test crashes the Python interpreter, for > example. > > >>> My current strategy is to remove the multiprocessing stuff > >>> locally...I'm not in a mood to spend more time on this now... > >> > >> I've been having the same issue, with little luck chasing it down > >> (though admittedly little time put into it). Try the recently > >> introduced --no-fork option... > > > > How about enabling this for version 2.6+ only and use multiprocessing? > > What I would like to see is a way to gather the output of gcc, and then let > the tests run in parallel using whatever support fits best (note that > multiprocessing has the advantage of being available for Windows, and the > disadvantage of requiring Python 2.6+). To get there, we'd have to patch > into distutils, and that's far from trivial as there isn't really a hook to > do that. > > Another possibility would be to divert stdout/stderr into a pipe when we > fork. Does anyone know how to do that in Python? this would probably help: http://alxr.usatlas.bnl.gov/lxr/source/atlas/Tools/PyUtils/python/Decorators.py?v=head#086 (I just ripped it off from: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/511474 ) cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From craigcitro at gmail.com Wed Nov 4 18:36:29 2009 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 4 Nov 2009 09:36:29 -0800 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <200911041151.59460.binet@cern.ch> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <2124F20F-8189-45DA-9532-1BC04FEDA4F1@math.washington.edu> <200911041151.59460.binet@cern.ch> Message-ID: > for ease of migration, distribute also puts (among other things) a > 'setuptools' entry in sys.modules. > so nothing else to do on the cython side. > Will this still be true when the Distribute 0.7 series gets released? -cc From dalcinl at gmail.com Wed Nov 4 19:30:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 4 Nov 2009 16:30:33 -0200 Subject: [Cython] runtests.py locks In-Reply-To: <4AF07DC3.1060301@behnel.de> References: <4AF045B0.4090109@student.matnat.uio.no> <4AF07904.1020204@student.matnat.uio.no> <4AF07DC3.1060301@behnel.de> Message-ID: On Tue, Nov 3, 2009 at 5:00 PM, Stefan Behnel wrote: > > To get there, we'd have to patch > into distutils, and that's far from trivial as there isn't really a hook to > do that. > The natural hook is monkeypatching distutils.spawn.spawn, I think... Actually: from distutils import ccompiler # the module ccompiler.spawn = my_new_posix_spawn > Another possibility would be to divert stdout/stderr into a pipe when we > fork. Does anyone know how to do that in Python? > pid = os.fork() if pid = 0: err = open("filename.ext", 'w') os.dup2(err.fileno(), sys.stderr.fileno()) try: spawn(cmd) finally: err.close() else: os.wait(pid) err = open("filename.ext", 'r') try: data = err.read() finally: err.close() Something like that... Actually, we should modify _spawn_posix() in distutils.spawn adding the os.dup2 stuff. Right now I'm very busy to implement it, sorry. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Thu Nov 5 07:55:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 4 Nov 2009 22:55:28 -0800 Subject: [Cython] Cython 0.12 alpha0 Message-ID: <7BCB5169-31E9-4349-B4B6-E085FC5FE066@math.washington.edu> I've posted an alpha up at http://cython.org/release/Cython-0.12.alpha0.tar.gz , which is also the current tip of cython-devel. There is a lot of new code in this release, so we'd like to test it thoroughly, and please get back to us with any feedback so we don't end up breaking your code. (Not that I think we will, but just in case, better safe than sorry...) - Robert From stefan_ml at behnel.de Thu Nov 5 08:20:16 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 05 Nov 2009 08:20:16 +0100 Subject: [Cython] [cython-users] Re: Cython 0.12 alpha0 In-Reply-To: <9bb68f9f-b6aa-4a7d-a36d-cf98957d7134@m38g2000yqd.googlegroups.com> References: <7BCB5169-31E9-4349-B4B6-E085FC5FE066@math.washington.edu> <9bb68f9f-b6aa-4a7d-a36d-cf98957d7134@m38g2000yqd.googlegroups.com> Message-ID: <4AF27CB0.9090908@behnel.de> Yosef, 05.11.2009 08:06: > On 5 ??????, 08:55, Robert Bradshaw wrote: That's about the weirdest date format I've ever seen. :) >> I've posted an alpha up athttp://cython.org/release/Cython-0.12.alpha0.tar.gz >> , which is also the current tip of cython-devel. There is a lot of >> new code in this release, so we'd like to test it thoroughly, and >> please get back to us with any feedback so we don't end up breaking >> your code. (Not that I think we will, but just in case, better safe >> than sorry...) >> >> - Robert > > It would be useful to have a short list of changes so we'll know what > we're asked to test. That's also half way to a release announsement. Does this help? http://trac.cython.org/cython_trac/query?status=closed&group=resolution&milestone=0.12 Stefan From robertwb at math.washington.edu Thu Nov 5 08:21:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 4 Nov 2009 23:21:21 -0800 Subject: [Cython] [cython-users] Re: Cython 0.12 alpha0 In-Reply-To: <9bb68f9f-b6aa-4a7d-a36d-cf98957d7134@m38g2000yqd.googlegroups.com> References: <7BCB5169-31E9-4349-B4B6-E085FC5FE066@math.washington.edu> <9bb68f9f-b6aa-4a7d-a36d-cf98957d7134@m38g2000yqd.googlegroups.com> Message-ID: On Nov 4, 2009, at 11:06 PM, Yosef wrote: > On 5 ??????, 08:55, Robert Bradshaw > > wrote: >> I've posted an alpha up athttp://cython.org/release/Cython-0.12.alpha0.tar.gz >> , which is also the current tip of cython-devel. There is a lot of >> new code in this release, so we'd like to test it thoroughly, and >> please get back to us with any feedback so we don't end up breaking >> your code. (Not that I think we will, but just in case, better safe >> than sorry...) >> >> - Robert > > It would be useful to have a short list of changes so we'll know what > we're asked to test. That's also half way to a release announsement. Thanks for your willingness to help out. I don't have a good list off hand, but that is a good point. http://trac.cython.org/cython_trac/query?group=component&milestone=0.12 provides an (incomplete) overview. The big semantics changes are that division and boolean operations now follow Python semantics. Unmarked string literals are now the str type in both Python 2 and Python 3. There's an improved numpy.pxd, and many new optimizations. Another big change was a re-writing how temp allocation is done, and of course there are lots of bugfixes. Even running and testing all the Cython code you have access to would be a great help for increasing the confidence we haven't introduced any obscure regressions (or finding them if we have). Another thing that would be very helpful is running the test suite on a large variety of platforms. This can be done by going into the unpacked tarball and running python runtests.py (If that hangs, try python runtests.py --no-fork, this is one bug we have yet to work out...) - Robert From dagss at student.matnat.uio.no Thu Nov 5 11:31:57 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 05 Nov 2009 11:31:57 +0100 Subject: [Cython] More complex bugs In-Reply-To: <0D7E932E-7E8C-47ED-8900-1898AA95D7FC@math.washington.edu> References: <4AE86598.10204@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> <4AF0567B.6070908@student.matnat.uio.no> <4AF06315.8040005@student.matnat.uio.no> <0D7E932E-7E8C-47ED-8900-1898AA95D7FC@math.washington.edu> Message-ID: <4AF2A99D.7080105@student.matnat.uio.no> Robert Bradshaw wrote: > On Nov 3, 2009, at 10:15 AM, Lisandro Dalcin wrote: > >> On Tue, Nov 3, 2009 at 3:06 PM, Dag Sverre Seljebotn >> wrote: >>> Lisandro Dalcin wrote: >>> >>>> Do you remember that I advocated for "ctypedef some int MyInt" or >>>> something similar? If right now we had that, Cython could reject >>>> some >>>> things, like binops on different "ctypedef some .." types, requiring >>>> users to introduce casts to resolve the ambiguity... A said it >>>> before, >>>> and I say it again: second-guessing is a bad thing (or, in Python's >>>> Zen words: "In the face of ambiguity, refuse the temptation to >>>> guess.") >>>> >>> I remember very well, and I've been thinking about it the whole >>> day. I >>> wholeheartedly agree with you that the situation is not good. > > I think in generally this is an unsoveable problem as the Cython stage > completes before the C stage starts (and generating code for all > possible permutations isn't feasible in general). This means also that > the author of the code doesn't (necessarily) know the size of the > external types before C compile time, and they may change from > platform to platform. It's definitely unsolveable -- and yet we can seek a "better local optimum" than what we have today (mainly because the current behaviour feels inconsistent, and inconsistency leads to hard-to-learn and bugs). > We're still going to have to deal with this for function overloading > (and there it's even nastier, as sizeof(a) >= sizeof(b) has different > semantics from sizeof(a) == sizeof(b)). For talking with C++, the C++ compiler deals with it. For Cython-specific/C-only, I think we should seriously consider disallowing overloading based on size. We simply don't allow cdef foo(double x) cdef foo(float x) But let's cross that bridge when we come to it... >>> Another, perhaps more "backwards-compatible" approach is >>> (backwards-compatible in the following sense: It will break code that >>> should be broken and fixed, and not suddenly introduce bugs that are >>> hard to find): >>> >>> a) Tighten down everything so that explicit casts are needed even >>> for >>> stuff like this: >>> >>> cdef long i >>> cdef external_typedef_int j >>> print i / j # not allowed, must do i / j >>> >> OK, +1 on this... > > I think this may be too much of a backwards incompatible change. Would > one then have to do > > cdef external_typedef_int j > print j / 2 > print j / 2 > > ? Hmm. But today, this would lead to a bug on 32-bit Linux: cdef extern from "header.h": ctypedef short my_longlong # really long long cdef my_longlong f(my_longlong x): return x // 2 I'm pretty sure this would result in 32-bit division rather than 64-bit. Long-term, as we've pretty much decided to have literals have Python semantics (i.e. Stefan's work on folding compile-time expressions), I think this problem can be worked around by changing the typing phase somewhat and have literals be the widest possible type (signed or unsigned depending on context...or something). > The phrase "should be broken and fixed" is somewhat subjective. Care > should be taken when marking something that is potentially broken as > something that is definitely broken. > > At the very least we should start with a directive that emits warnings > or errors to see how much stuff it breaks and how constraining it is > to work with first. Very good idea. I'll direct any future efforts in this area towards such a feature rather than on the mailing list. (Not that I promise to do it, I think memoryviews are ahead of it in my queue.) -- Dag Sverre From ndbecker2 at gmail.com Thu Nov 5 12:53:10 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 05 Nov 2009 06:53:10 -0500 Subject: [Cython] optimizing numpy slice Message-ID: Is there a trac ticket for optimizing numpy slicing (and a milestone)? From josef.pktd at gmail.com Thu Nov 5 16:46:31 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Nov 2009 10:46:31 -0500 Subject: [Cython] runtests results WindowXP MingW Message-ID: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> After adding the following to the runtests (copied from pyxbuild.py, I don't know why config_files are parsed twice), the tests run with MingW (3.4.5) as compiler. I get (failures=4, errors=14) which I have no idea what they mean (setup problem or real), see full output below. Josef ######## added to runtests.py config_files = distutils_distro.find_config_files() try: config_files.remove('setup.cfg') except ValueError: pass distutils_distro.parse_config_files(config_files) cfgfiles = distutils_distro.find_config_files() try: cfgfiles.remove('setup.cfg') except ValueError: pass distutils_distro.parse_config_files(cfgfiles) ############# C:\Josef\work-oth\Cython-0.12.alpha0>python runtests.py Running tests against Cython 0.12.alpha0 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] bufaccess_noassignT444.c:1061: warning: '__Pyx_GetBufferAndValidate' defined but not used bufaccess_noassignT444.cpp:1061: warning: 'int __Pyx_GetBufferAndValidate(Py_buf fer*, PyObject*, __Pyx_TypeInfo*, int, int, int, __Pyx_BufFmt_StackElem*)' defin ed but not used C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x35b): undefined reference to `p1' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x35f): undefined reference to `f1' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x365): undefined reference to `p2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x369): undefined reference to `f2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x36f): undefined reference to `p3' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x373): undefined reference to `f3 at 0' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x379): undefined reference to `p4' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\callingconvention.o:calling convention.c:(.text+0x37d): undefined reference to `@f4 at 0' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x35b): undefined reference to `p1' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x35f): undefined reference to `f1' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x365): undefined reference to `p2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x369): undefined reference to `f2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x36f): undefined reference to `p3' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x373): undefined reference to `f3 at 0' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x379): undefined reference to `p4' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\callingconvention.o:calli ngconvention.cpp:(.text+0x37d): undefined reference to `@f4 at 0' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\cnamespec.o:cnamespec.c:(.t ext+0xb4): undefined reference to `c_a' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\cnamespec.o:cnamespec.c:(.t ext+0xc4): undefined reference to `c_b' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\cnamespec.o:cnamespec.cpp :(.text+0xb4): undefined reference to `c_a' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\cnamespec.o:cnamespec.cpp :(.text+0xc4): undefined reference to `c_b' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp _exceptions_T265.cpp:(.text+0x2c1): undefined reference to `generic_error()' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp _exceptions_T265.cpp:(.text+0x335): undefined reference to `specified_error()' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp _exceptions_T265.cpp:(.text+0x42b): undefined reference to `dynamic_error()' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\declarations.o:declarations .c:(.text+0xae): undefined reference to `ifnp' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\declarations.o:declaratio ns.cpp:(.text+0xae): undefined reference to `ifnp' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\excvalcheck.o:excvalcheck.c :(.text+0xad): undefined reference to `spam' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\excvalcheck.o:excvalcheck.c :(.text+0xd9): undefined reference to `grail' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\excvalcheck.o:excvalcheck.c :(.text+0x10a): undefined reference to `tomato' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\excvalcheck.o:excvalcheck .cpp:(.text+0xad): undefined reference to `spam' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\excvalcheck.o:excvalcheck .cpp:(.text+0xd9): undefined reference to `grail' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\excvalcheck.o:excvalcheck .cpp:(.text+0x10a): undefined reference to `tomato' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\nogil.o:nogil.c:(.text+0x9d ): undefined reference to `g2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\nogil.o:nogil.c:(.text+0xa5 ): undefined reference to `g2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\nogil.o:nogil.c:(.text+0xaa ): undefined reference to `e1' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\c\nogil.o:nogil.c:(.text+0xaf ): undefined reference to `e2' collect2: ld returned 1 exit status C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+ 0x9e): undefined reference to `g2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+ 0xa6): undefined reference to `g2' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+ 0xab): undefined reference to `e1()' C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+ 0xb0): undefined reference to `e2()' collect2: ld returned 1 exit status specmethdocstring.c:401: warning: '__pyx_doc_17specmethdocstring_1C___init__' de fined but not used specmethdocstring.c:423: warning: '__pyx_doc_17specmethdocstring_1C_3foo___get__ ' defined but not used specmethdocstring.c:442: warning: '__pyx_doc_17specmethdocstring_1C_3foo___set__ ' defined but not used specmethdocstring.cpp:401: warning: '__pyx_doc_17specmethdocstring_1C___init__' defined but not used specmethdocstring.cpp:423: warning: '__pyx_doc_17specmethdocstring_1C_3foo___get __' defined but not used specmethdocstring.cpp:442: warning: '__pyx_doc_17specmethdocstring_1C_3foo___set __' defined but not used compile_time_unraisable_T370.c:382: warning: '__pyx_f_28compile_time_unraisable_ T370_raiseit' defined but not used autotestdict.c:463: warning: '__pyx_f_12autotestdict_cdeffunc' defined but not u sed autotestdict.c:945: warning: '__pyx_doc_12autotestdict_11MyCdefClass___cinit__' defined but not used autotestdict.c:967: warning: '__pyx_doc_12autotestdict_11MyCdefClass___dealloc__ ' defined but not used autotestdict.c:983: warning: '__pyx_doc_12autotestdict_11MyCdefClass___richcmp__ ' defined but not used autotestdict.c:1003: warning: '__pyx_doc_12autotestdict_11MyCdefClass___nonzero_ _' defined but not used autotestdict.cpp:463: warning: 'PyObject* __pyx_f_12autotestdict_cdeffunc()' def ined but not used autotestdict.cpp:945: warning: '__pyx_doc_12autotestdict_11MyCdefClass___cinit__ ' defined but not used autotestdict.cpp:967: warning: '__pyx_doc_12autotestdict_11MyCdefClass___dealloc __' defined but not used autotestdict.cpp:983: warning: '__pyx_doc_12autotestdict_11MyCdefClass___richcmp __' defined but not used autotestdict.cpp:1003: warning: '__pyx_doc_12autotestdict_11MyCdefClass___nonzer o__' defined but not used big_indices.cpp: In function `PyObject* __pyx_pf_11big_indices_test(PyObject*, P yObject*)': big_indices.cpp:487: warning: converting of negative value `-0x000000002' to `lo ng unsigned int' bufaccess.c: In function `__pyx_pf_9bufaccess_printbuf': bufaccess.c:1849: warning: unused variable `__pyx_bstride_0_buf' bufaccess.c:1850: warning: unused variable `__pyx_bstride_1_buf' bufaccess.c:1851: warning: unused variable `__pyx_bshape_0_buf' bufaccess.c:1852: warning: unused variable `__pyx_bshape_1_buf' bufaccess.c:1853: warning: unused variable `__pyx_boffset_0_buf' bufaccess.c:1854: warning: unused variable `__pyx_boffset_1_buf' bufaccess.cpp: In function `PyObject* __pyx_pf_9bufaccess_printbuf(PyObject*, Py Object*)': bufaccess.cpp:1849: warning: unused variable '__pyx_bstride_0_buf' bufaccess.cpp:1850: warning: unused variable '__pyx_bstride_1_buf' bufaccess.cpp:1851: warning: unused variable '__pyx_bshape_0_buf' bufaccess.cpp:1852: warning: unused variable '__pyx_bshape_1_buf' bufaccess.cpp:1853: warning: unused variable '__pyx_boffset_0_buf' bufaccess.cpp:1854: warning: unused variable '__pyx_boffset_1_buf' bufaccess.cpp: In function `PyObject* __pyx_pf_9bufaccess_get_int_2d_uintindex(P yObject*, PyObject*, PyObject*)': bufaccess.cpp:5781: warning: comparison between signed and unsigned integer expr essions bufaccess.cpp:5782: warning: comparison between signed and unsigned integer expr essions bufaccess.cpp: In function `PyObject* __pyx_pf_9bufaccess_coercions(PyObject*, P yObject*)': bufaccess.cpp:7623: warning: converting of negative value `-0x000000001' to `uns igned char' buffer.c:525: warning: '__pyx_pf_6buffer_10TestBuffer___getbuffer__' defined but not used buffer.c:690: warning: '__pyx_pf_6buffer_17TestBufferRelease___releasebuffer__' defined but not used buffer.cpp:525: warning: 'int __pyx_pf_6buffer_10TestBuffer___getbuffer__(PyObje ct*, Py_buffer*, int)' defined but not used buffer.cpp:690: warning: 'void __pyx_pf_6buffer_17TestBufferRelease___releasebuf fer__(PyObject*, Py_buffer*)' defined but not used cpp_exceptions.cpp:139:37: cpp_exceptions_helper.cpp: No such file or directory cpp_exceptions.cpp: In function `PyObject* __pyx_pf_14cpp_exceptions_test_int_ra w(PyObject*, PyObject*)': cpp_exceptions.cpp:503: error: `raise_int' was not declared in this scope cpp_exceptions.cpp:503: warning: unused variable 'raise_int' cpp_exceptions.cpp: In function `PyObject* __pyx_pf_14cpp_exceptions_test_int_va lue(PyObject*, PyObject*)': cpp_exceptions.cpp:550: error: `raise_int' was not declared in this scope cpp_exceptions.cpp:550: warning: unused variable 'raise_int' cpp_exceptions.cpp: In function `PyObject* __pyx_pf_14cpp_exceptions_test_int_cu stom(PyObject*, PyObject*)': cpp_exceptions.cpp:597: error: `raise_int' was not declared in this scope cpp_exceptions.cpp:597: warning: unused variable 'raise_int' cpp_exceptions.cpp: In function `PyObject* __pyx_pf_14cpp_exceptions_test_index_ raw(PyObject*, PyObject*)': cpp_exceptions.cpp:644: error: `raise_index' was not declared in this scope cpp_exceptions.cpp:644: warning: unused variable 'raise_index' cpp_exceptions.cpp: In function `PyObject* __pyx_pf_14cpp_exceptions_test_index_ value(PyObject*, PyObject*)': cpp_exceptions.cpp:691: error: `raise_index' was not declared in this scope cpp_exceptions.cpp:691: warning: unused variable 'raise_index' cpp_exceptions.cpp: In function `PyObject* __pyx_pf_14cpp_exceptions_test_index_ custom(PyObject*, PyObject*)': cpp_exceptions.cpp:736: error: `raise_index' was not declared in this scope cpp_exceptions.cpp:736: warning: unused variable 'raise_index' c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__multiarray_api .h:969: warning: '_import_array' defined but not used c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__ufunc_api.h:18 3: warning: '_import_umath' defined but not used numpy_bufacc_T155.c:3856: warning: '__Pyx_UnpackItem' defined but not used numpy_bufacc_T155.c:3866: warning: '__Pyx_EndUnpack' defined but not used c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__multiarray_api .h:969: warning: 'int _import_array()' defined but not used c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__ufunc_api.h:18 3: warning: 'int _import_umath()' defined but not used numpy_bufacc_T155.cpp:3856: warning: 'PyObject* __Pyx_UnpackItem(PyObject*, Py_s size_t)' defined but not used numpy_bufacc_T155.cpp:3866: warning: 'int __Pyx_EndUnpack(PyObject*)' defined bu t not used c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__multiarray_api .h:969: warning: '_import_array' defined but not used c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__ufunc_api.h:18 3: warning: '_import_umath' defined but not used numpy_test.c:10249: warning: '__Pyx_UnpackItem' defined but not used numpy_test.c:10259: warning: '__Pyx_EndUnpack' defined but not used c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__multiarray_api .h:969: warning: 'int _import_array()' defined but not used c:/programs/python25/lib/site-packages/numpy/core/include/numpy/__ufunc_api.h:18 3: warning: 'int _import_umath()' defined but not used numpy_test.cpp:10249: warning: 'PyObject* __Pyx_UnpackItem(PyObject*, Py_ssize_t )' defined but not used numpy_test.cpp:10259: warning: 'int __Pyx_EndUnpack(PyObject*)' defined but not used === Expected errors: === === Got errors: === 2:21: Error in compile-time expression: ValueError: invalid literal for float(): nan 3:22: Error in compile-time expression: ValueError: invalid literal for float(): +inf 4:22: Error in compile-time expression: ValueError: invalid literal for float(): -inf 10:21: Invalid type for compile-time constant: NoneType 11:23: Invalid type for compile-time constant: NoneType 12:23: Invalid type for compile-time constant: NoneType 29:17: Invalid type for compile-time constant: NoneType 49:17: Invalid type for compile-time constant: NoneType 60:18: Invalid type for compile-time constant: NoneType 86:18: Invalid type for compile-time constant: NoneType 97:18: Invalid type for compile-time constant: NoneType 123:18: Invalid type for compile-time constant: NoneType === Expected errors: === === Got errors: === 2:21: Error in compile-time expression: ValueError: invalid literal for float(): nan 3:22: Error in compile-time expression: ValueError: invalid literal for float(): +inf 4:22: Error in compile-time expression: ValueError: invalid literal for float(): -inf 10:21: Invalid type for compile-time constant: NoneType 11:23: Invalid type for compile-time constant: NoneType 12:23: Invalid type for compile-time constant: NoneType 29:17: Invalid type for compile-time constant: NoneType 49:17: Invalid type for compile-time constant: NoneType 60:18: Invalid type for compile-time constant: NoneType 86:18: Invalid type for compile-time constant: NoneType 97:18: Invalid type for compile-time constant: NoneType 123:18: Invalid type for compile-time constant: NoneType ====================================================================== ERROR: compiling (c) callingconvention ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'gcc' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) callingconvention ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'g++' failed with exit status 1 ====================================================================== ERROR: compiling (c) cnamespec ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'gcc' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) cnamespec ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'g++' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) cpp_exceptions_T265 ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'g++' failed with exit status 1 ====================================================================== ERROR: compiling (c) declarations ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'gcc' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) declarations ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'g++' failed with exit status 1 ====================================================================== ERROR: compiling (c) excvalcheck ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'gcc' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) excvalcheck ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'g++' failed with exit status 1 ====================================================================== ERROR: compiling (c) nogil ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'gcc' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) nogil ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 267, in runTest self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 513, in b uild_extension target_lang=language) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 845, in link_shar ed_object extra_preargs, extra_postargs, build_temp, target_lang) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 246, in lin k target_lang) File "C:\Programs\Python25\lib\distutils\unixccompiler.py", line 254, in link raise LinkError, msg LinkError: command 'g++' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) and running cpp_exceptions ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 398, in run self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 386, in compile self.run_distutils(module, workdir, incdir) File "runtests.py", line 346, in run_distutils build_extension.run() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 290, in r un self.build_extensions() File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 416, in b uild_extensions self.build_extension(ext) File "runtests.py", line 82, in build_extension _build_ext.build_extension(self, ext) File "C:\Programs\Python25\lib\distutils\command\build_ext.py", line 481, in b uild_extension depends=ext.depends) File "C:\Programs\Python25\lib\distutils\ccompiler.py", line 697, in compile self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) File "C:\Programs\Python25\lib\distutils\cygwinccompiler.py", line 152, in _co mpile raise CompileError, msg CompileError: command 'gcc' failed with exit status 1 ====================================================================== ERROR: compiling (c) and running specialfloat ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 398, in run self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 376, in compile self.assertEquals(None, unexpected_error) AssertionError: None != u'2:21: Error in compile-time expression: ValueError: in valid literal for float(): nan' ====================================================================== ERROR: compiling (cpp) and running specialfloat ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 398, in run self.runCompileTest() File "runtests.py", line 271, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 376, in compile self.assertEquals(None, unexpected_error) AssertionError: None != u'2:21: Error in compile-time expression: ValueError: in valid literal for float(): nan' ====================================================================== FAIL: Doctest: float_floor_division_T260.__test__.floor_div_float (line 1) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for float_floor_division_T260.__test__.floor _div_float (line 1) File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\c\float_floor_division_T2 60.pyd", line unknown line number, in floor_div_float (line 1) ---------------------------------------------------------------------- File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\c\float_floor_division_T260 .pyd", line ?, in float_floor_division_T260.__test__.floor_div_float (line 1) Failed example: floor_div_float(1e10, 1e-10) Expected: 1e+20 Got: 1e+020 ====================================================================== FAIL: Doctest: float_floor_division_T260.__test__.floor_div_float (line 1) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for float_floor_division_T260.__test__.floor _div_float (line 1) File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp\float_floor_division_ T260.pyd", line unknown line number, in floor_div_float (line 1) ---------------------------------------------------------------------- File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp\float_floor_division_T2 60.pyd", line ?, in float_floor_division_T260.__test__.floor_div_float (line 1) Failed example: floor_div_float(1e10, 1e-10) Expected: 1e+20 Got: 1e+020 ====================================================================== FAIL: Doctest: profile_test ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for profile_test File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\c\profile_test.pyd", line 40, in profile_test ---------------------------------------------------------------------- File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\c\profile_test.pyd", line 6 1, in profile_test Failed example: os.unlink(statsfile) Exception raised: Traceback (most recent call last): File "C:\Programs\Python25\lib\doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in os.unlink(statsfile) WindowsError: [Error 32] The process cannot access the file because it is be ing used by another process: 'c:\\docume~1\\carrasco\\locals~1\\temp\\tmpzbldpp' ====================================================================== FAIL: Doctest: profile_test ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for profile_test File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp\profile_test.pyd", li ne 36, in profile_test ---------------------------------------------------------------------- File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp\profile_test.pyd", line 57, in profile_test Failed example: os.unlink(statsfile) Exception raised: Traceback (most recent call last): File "C:\Programs\Python25\lib\doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in os.unlink(statsfile) WindowsError: [Error 32] The process cannot access the file because it is be ing used by another process: 'c:\\docume~1\\carrasco\\locals~1\\temp\\tmpm6kkx5' ---------------------------------------------------------------------- Ran 2950 tests in 1474.453s FAILED (failures=4, errors=14) From robertwb at math.washington.edu Thu Nov 5 19:42:03 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Nov 2009 10:42:03 -0800 Subject: [Cython] More complex bugs In-Reply-To: <4AF2A99D.7080105@student.matnat.uio.no> References: <4AE86598.10204@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> <4AF0567B.6070908@student.matnat.uio.no> <4AF06315.8040005@student.matnat.uio.no> <0D7E932E-7E8C-47ED-8900-1898AA95D7FC@math.washington.edu> <4AF2A99D.7080105@student.matnat.uio.no> Message-ID: On Nov 5, 2009, at 2:31 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Nov 3, 2009, at 10:15 AM, Lisandro Dalcin wrote: >> >>> On Tue, Nov 3, 2009 at 3:06 PM, Dag Sverre Seljebotn >>> wrote: >>>> Lisandro Dalcin wrote: >>>> >>>>> Do you remember that I advocated for "ctypedef some int MyInt" or >>>>> something similar? If right now we had that, Cython could reject >>>>> some >>>>> things, like binops on different "ctypedef some .." types, >>>>> requiring >>>>> users to introduce casts to resolve the ambiguity... A said it >>>>> before, >>>>> and I say it again: second-guessing is a bad thing (or, in >>>>> Python's >>>>> Zen words: "In the face of ambiguity, refuse the temptation to >>>>> guess.") >>>>> >>>> I remember very well, and I've been thinking about it the whole >>>> day. I >>>> wholeheartedly agree with you that the situation is not good. >> >> I think in generally this is an unsoveable problem as the Cython >> stage >> completes before the C stage starts (and generating code for all >> possible permutations isn't feasible in general). This means also >> that >> the author of the code doesn't (necessarily) know the size of the >> external types before C compile time, and they may change from >> platform to platform. > > It's definitely unsolveable -- and yet we can seek a "better local > optimum" than what we have today (mainly because the current behaviour > feels inconsistent, and inconsistency leads to hard-to-learn and > bugs). I agree that we can do better. >> We're still going to have to deal with this for function overloading >> (and there it's even nastier, as sizeof(a) >= sizeof(b) has different >> semantics from sizeof(a) == sizeof(b)). > > For talking with C++, the C++ compiler deals with it. > > For Cython-specific/C-only, I think we should seriously consider > disallowing overloading based on size. We simply don't allow > > cdef foo(double x) > cdef foo(float x) > > But let's cross that bridge when we come to it... Well, cdef long max(long a, long b) cdef int max(int a, int b) and cdef float sin(float) cdef double sin(double) May both come in handy, but abuse of overloading based on size could be really nasty. (I don't think this is specific to Cython, but to writing portable code in general.) >>>> Another, perhaps more "backwards-compatible" approach is >>>> (backwards-compatible in the following sense: It will break code >>>> that >>>> should be broken and fixed, and not suddenly introduce bugs that >>>> are >>>> hard to find): >>>> >>>> a) Tighten down everything so that explicit casts are needed even >>>> for >>>> stuff like this: >>>> >>>> cdef long i >>>> cdef external_typedef_int j >>>> print i / j # not allowed, must do i / j >>>> >>> OK, +1 on this... >> >> I think this may be too much of a backwards incompatible change. >> Would >> one then have to do >> >> cdef external_typedef_int j >> print j / 2 >> print j / 2 >> >> ? > > Hmm. But today, this would lead to a bug on 32-bit Linux: > > cdef extern from "header.h": > ctypedef short my_longlong # really long long > > cdef my_longlong f(my_longlong x): > return x // 2 > > I'm pretty sure this would result in 32-bit division rather than 64- > bit. Yes, that's true, and I could be bad (though only if you're storing values larger than shorts, in which case the extern definition is clearly wrong, though that might not always be obvious). > Long-term, as we've pretty much decided to have literals have Python > semantics (i.e. Stefan's work on folding compile-time expressions), I > think this problem can be worked around by changing the typing phase > somewhat and have literals be the widest possible type (signed or > unsigned depending on context...or something). One thing to avoid is, with type inference we don't want to make x = 3 create a long-long typed x. Perhaps they could be typed according to context somehow (if this doesn't get too complicated to reason about). >> The phrase "should be broken and fixed" is somewhat subjective. Care >> should be taken when marking something that is potentially broken as >> something that is definitely broken. >> >> At the very least we should start with a directive that emits >> warnings >> or errors to see how much stuff it breaks and how constraining it is >> to work with first. > > Very good idea. I'll direct any future efforts in this area towards > such > a feature rather than on the mailing list. Cool, thanks. And an eventual CEP. > (Not that I promise to do it, I think memoryviews are ahead of it in > my > queue.) Sure. I think memoryviews will be more valuable to have as well (though if we both had an infinite amount of time to work on things...). - Robert From dagss at student.matnat.uio.no Thu Nov 5 20:22:45 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 05 Nov 2009 20:22:45 +0100 Subject: [Cython] More complex bugs In-Reply-To: References: <4AE86598.10204@student.matnat.uio.no> <8C3AAD65-2082-484A-B0BF-291426014DF3@math.washington.edu> <4AEFEDAB.6020306@student.matnat.uio.no> <4AF046F5.8010308@student.matnat.uio.no> <4AF0567B.6070908@student.matnat.uio.no> <4AF06315.8040005@student.matnat.uio.no> <0D7E932E-7E8C-47ED-8900-1898AA95D7FC@math.washington.edu> <4AF2A99D.7080105@student.matnat.uio.no> Message-ID: <4AF32605.5060109@student.matnat.uio.no> Robert Bradshaw wrote: > On Nov 5, 2009, at 2:31 AM, Dag Sverre Seljebotn wrote: >> For talking with C++, the C++ compiler deals with it. >> >> For Cython-specific/C-only, I think we should seriously consider >> disallowing overloading based on size. We simply don't allow >> >> cdef foo(double x) >> cdef foo(float x) >> >> But let's cross that bridge when we come to it... > > Well, > > cdef long max(long a, long b) > cdef int max(int a, int b) > > and > > cdef float sin(float) > cdef double sin(double) > > May both come in handy, but abuse of overloading based on size could > be really nasty. (I don't think this is specific to Cython, but to > writing portable code in general.) I hope templates can deal with most of these cases. >> Hmm. But today, this would lead to a bug on 32-bit Linux: >> >> cdef extern from "header.h": >> ctypedef short my_longlong # really long long >> >> cdef my_longlong f(my_longlong x): >> return x // 2 >> >> I'm pretty sure this would result in 32-bit division rather than 64- >> bit. > Actually, "int" will have the same problem and is much less obvious (because the "2" literal is apparently "long", which I guess trumps "int"). > Yes, that's true, and I could be bad (though only if you're storing > values larger than shorts, in which case the extern definition is > clearly wrong, though that might not always be obvious). Well, if you know that short is 16 bits long, then the problem is a bit different. You don't in general with a C compiler (although I don't know of any exceptions personally). (I was wondering if I should suggest doing like Java and *define* short to be 16 bits while we have the chance, but I guess it would only bring confusion and we can always support int16_t in C99). -- Dag Sverre From robertwb at math.washington.edu Fri Nov 6 06:26:38 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Nov 2009 21:26:38 -0800 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> Message-ID: <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> Thanks for the feedback. Obviously we don't do enough testing on your system. On Nov 5, 2009, at 7:46 AM, josef.pktd at gmail.com wrote: > After adding the following to the runtests (copied from pyxbuild.py, I > don't know why config_files are parsed twice), > the tests run with MingW (3.4.5) as compiler. > > I get (failures=4, errors=14) which I have no idea what they mean > (setup problem or real), see full output below. > > Josef > > ######## added to runtests.py > config_files = distutils_distro.find_config_files() > try: config_files.remove('setup.cfg') > except ValueError: pass > distutils_distro.parse_config_files(config_files) > > cfgfiles = distutils_distro.find_config_files() > try: cfgfiles.remove('setup.cfg') > except ValueError: pass > distutils_distro.parse_config_files(cfgfiles) > ############# > > > C:\Josef\work-oth\Cython-0.12.alpha0>python runtests.py > Running tests against Cython 0.12.alpha0 > Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit > (Intel)] > > bufaccess_noassignT444.c:1061: warning: '__Pyx_GetBufferAndValidate' > defined but > not used [...] we have got to cut down on the number of warnings we are starting to generate... > numpy_test.cpp:10259: warning: 'int __Pyx_EndUnpack(PyObject*)' > defined but not > used > > === Expected errors: === > > > > === Got errors: === > 2:21: Error in compile-time expression: ValueError: invalid literal > for float(): > nan > 3:22: Error in compile-time expression: ValueError: invalid literal > for float(): > +inf > 4:22: Error in compile-time expression: ValueError: invalid literal > for float(): > -inf What happens when you do this in your Python? >>> float('+inf'), float('-inf'), float('nan') (inf, -inf, nan) >>> 1e300 * 1e300 inf >>> 1e300 * 1e300 * 0 nan > ====================================================================== > ERROR: compiling (c) callingconvention > ERROR: compiling (c) cnamespec > ERROR: compiling (c) declarations > ERROR: compiling (c) excvalcheck > ERROR: compiling (c) nogil > ---------------------------------------------------------------------- > LinkError: command 'gcc' failed with exit status 1 > > ====================================================================== > ERROR: compiling (cpp) callingconvention > ERROR: compiling (cpp) cnamespec > ERROR: compiling (cpp) cpp_exceptions_T265 > ERROR: compiling (cpp) declarations > ERROR: compiling (cpp) excvalcheck > ERROR: compiling (cpp) nogil > ---------------------------------------------------------------------- > LinkError: command 'g++' failed with exit status 1 These look like http://trac.cython.org/cython_trac/ticket/106 . Unfortunately they've been around for a while, but I think it'll take a windows developer to fix them. I'm keeping them on the radar though. Anyone have any ideas? It'd be great to finally knock these out. Note these are all compile (not run). They might be referring to non- existent methods. In that case, I bet re-writing the tests could get rid of these for once and for all. > ====================================================================== > ERROR: compiling (cpp) and running cpp_exceptions > ---------------------------------------------------------------------- > CompileError: command 'gcc' failed with exit status 1 There's a better test in tests/run that is a superset of this one (and actually gets tested, not just compiled). I've just removed this test, as it breaks on Windows due to using fake functions. > ====================================================================== > ERROR: compiling (c) and running specialfloat > ERROR: compiling (cpp) and running specialfloat > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "runtests.py", line 398, in run > self.runCompileTest() > File "runtests.py", line 271, in runCompileTest > self.directory, self.expect_errors, self.annotate) > File "runtests.py", line 376, in compile > self.assertEquals(None, unexpected_error) > AssertionError: None != u'2:21: Error in compile-time expression: > ValueError: in > valid literal for float(): nan' See above. > ====================================================================== > FAIL: Doctest: float_floor_division_T260.__test__.floor_div_float > (line 1) > FAIL: Doctest: float_floor_division_T260.__test__.floor_div_float > (line 1) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest > raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for > float_floor_division_T260.__test__.floor > _div_float (line 1) > File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp > \float_floor_division_ > T260.pyd", line unknown line number, in floor_div_float (line 1) > > ---------------------------------------------------------------------- > File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp > \float_floor_division_T2 > 60.pyd", line ?, in > float_floor_division_T260.__test__.floor_div_float (line 1) > Failed example: > floor_div_float(1e10, 1e-10) > Expected: > 1e+20 > Got: > 1e+020 What do you get for >>> float('1e20') 1e+20 in your Python? This (and the errors above) might be http://bugs.python.org/issue1635 > ====================================================================== > FAIL: Doctest: profile_test > FAIL: Doctest: profile_test > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest > raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for profile_test > File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp > \profile_test.pyd", li > ne 36, in profile_test > > ---------------------------------------------------------------------- > File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp > \profile_test.pyd", line > 57, in profile_test > Failed example: > os.unlink(statsfile) > Exception raised: > Traceback (most recent call last): > File "C:\Programs\Python25\lib\doctest.py", line 1228, in __run > compileflags, 1) in test.globs > File "", line 1, in > os.unlink(statsfile) > WindowsError: [Error 32] The process cannot access the file > because it is be > ing used by another process: 'c:\\docume~1\\carrasco\\locals~1\\temp\ > \tmpm6kkx5' That's a strange one. If os.fork is available (though it shouldn't be on Windows), could you try python runtests.py --no-fork profile_test > ---------------------------------------------------------------------- > Ran 2950 tests in 1474.453s > > FAILED (failures=4, errors=14) Well, at least most of them passed :). Was that on a heavily loaded system, or is MingW just a really slow compiler? Thanks again, - Robert From robertwb at math.washington.edu Fri Nov 6 06:34:37 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Nov 2009 21:34:37 -0800 Subject: [Cython] optimizing numpy slice In-Reply-To: References: Message-ID: <64831E43-589C-49CF-AB6F-45812AFC6269@math.washington.edu> On Nov 5, 2009, at 3:53 AM, Neal Becker wrote: > Is there a trac ticket for optimizing numpy slicing (and a milestone)? Not that I know of, but it's on the radar as part of the memoryview concept. - Robert From josef.pktd at gmail.com Fri Nov 6 06:59:03 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 6 Nov 2009 00:59:03 -0500 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> Message-ID: <1cd32cbb0911052159n8173c41y7c1e6a4242b51c8@mail.gmail.com> On Fri, Nov 6, 2009 at 12:26 AM, Robert Bradshaw wrote: > Thanks for the feedback. Obviously we don't do enough testing on your > system. > > On Nov 5, 2009, at 7:46 AM, josef.pktd at gmail.com wrote: > >> After adding the following to the runtests (copied from pyxbuild.py, I >> don't know why config_files are parsed twice), >> the tests run with MingW (3.4.5) as compiler. >> >> I get (failures=4, errors=14) which I have no idea what they mean >> (setup problem or real), see full output below. >> >> Josef >> >> ######## added to runtests.py >> config_files = distutils_distro.find_config_files() >> try: config_files.remove('setup.cfg') >> except ValueError: pass >> distutils_distro.parse_config_files(config_files) >> >> cfgfiles = distutils_distro.find_config_files() >> try: cfgfiles.remove('setup.cfg') >> except ValueError: pass >> distutils_distro.parse_config_files(cfgfiles) >> ############# >> >> >> C:\Josef\work-oth\Cython-0.12.alpha0>python runtests.py >> Running tests against Cython 0.12.alpha0 >> Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit >> (Intel)] >> >> bufaccess_noassignT444.c:1061: warning: '__Pyx_GetBufferAndValidate' >> defined but >> not used > > [...] we have got to cut down on the number of warnings we are > starting to generate... > >> numpy_test.cpp:10259: warning: 'int __Pyx_EndUnpack(PyObject*)' >> defined but not >> used >> >> === Expected errors: === >> >> >> >> === Got errors: === >> 2:21: Error in compile-time expression: ValueError: invalid literal >> for float(): >> nan >> 3:22: Error in compile-time expression: ValueError: invalid literal >> for float(): >> +inf >> 4:22: Error in compile-time expression: ValueError: invalid literal >> for float(): >> -inf > > What happens when you do this in your Python? > > ?>>> float('+inf'), float('-inf'), float('nan') > (inf, -inf, nan) > ?>>> 1e300 * 1e300 > inf > ?>>> 1e300 * 1e300 * 0 > nan only exceptions for string to float conversion: >>> float('+inf'), float('-inf'), float('nan') Traceback (most recent call last): ValueError: invalid literal for float(): +inf >>> 1e300 * 1e300 1.#INF >>> 1e300 * 1e300 * 0 -1.#IND >>> float('inf') Traceback (most recent call last): ValueError: invalid literal for float(): inf >>> ss = str(1e300 * 1e300) >>> ss '1.#INF' >>> float(ss) Traceback (most recent call last): ValueError: invalid literal for float(): 1.#INF >>> float(+np.inf), float(-np.inf), float(np.nan) (1.#INF, -1.#INF, 1.#QNAN) > >> ====================================================================== >> ERROR: compiling (c) callingconvention >> ERROR: compiling (c) cnamespec >> ERROR: compiling (c) declarations >> ERROR: compiling (c) excvalcheck >> ERROR: compiling (c) nogil >> ---------------------------------------------------------------------- >> LinkError: command 'gcc' failed with exit status 1 >> >> ====================================================================== >> ERROR: compiling (cpp) callingconvention >> ERROR: compiling (cpp) cnamespec >> ERROR: compiling (cpp) cpp_exceptions_T265 >> ERROR: compiling (cpp) declarations >> ERROR: compiling (cpp) excvalcheck >> ERROR: compiling (cpp) nogil >> ---------------------------------------------------------------------- >> LinkError: command 'g++' failed with exit status 1 > > These look like http://trac.cython.org/cython_trac/ticket/106 . > Unfortunately they've been around for a while, but I think it'll take > a windows developer to fix them. I'm keeping them on the radar though. > Anyone have any ideas? It'd be great to finally knock these out. Note > these are all compile (not run). They might be referring to non- > existent methods. In that case, I bet re-writing the tests could get > rid of these for once and for all. > >> ====================================================================== >> ERROR: compiling (cpp) and running cpp_exceptions >> ---------------------------------------------------------------------- >> CompileError: command 'gcc' failed with exit status 1 > > There's a better test in tests/run that is a superset of this one (and > actually gets tested, not just compiled). I've just removed this test, > as it breaks on Windows due to using fake functions. > >> ====================================================================== >> ERROR: compiling (c) and running specialfloat >> ERROR: compiling (cpp) and running specialfloat >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "runtests.py", line 398, in run >> ? ?self.runCompileTest() >> ?File "runtests.py", line 271, in runCompileTest >> ? ?self.directory, self.expect_errors, self.annotate) >> ?File "runtests.py", line 376, in compile >> ? ?self.assertEquals(None, unexpected_error) >> AssertionError: None != u'2:21: Error in compile-time expression: >> ValueError: in >> valid literal for float(): nan' > > See above. > >> ====================================================================== >> FAIL: Doctest: float_floor_division_T260.__test__.floor_div_float >> (line 1) >> FAIL: Doctest: float_floor_division_T260.__test__.floor_div_float >> (line 1) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest >> ? ?raise self.failureException(self.format_failure(new.getvalue())) >> AssertionError: Failed doctest test for >> float_floor_division_T260.__test__.floor >> _div_float (line 1) >> ?File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp >> \float_floor_division_ >> T260.pyd", line unknown line number, in floor_div_float (line 1) >> >> ---------------------------------------------------------------------- >> File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp >> \float_floor_division_T2 >> 60.pyd", line ?, in >> float_floor_division_T260.__test__.floor_div_float (line 1) >> Failed example: >> ? ?floor_div_float(1e10, 1e-10) >> Expected: >> ? ?1e+20 >> Got: >> ? ?1e+020 > > What do you get for > > ?>>> float('1e20') > 1e+20 > > in your Python? >>> float('1e20') 1e+020 > > This (and the errors above) might be http://bugs.python.org/issue1635 > >> ====================================================================== >> FAIL: Doctest: profile_test >> FAIL: Doctest: profile_test >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "C:\Programs\Python25\lib\doctest.py", line 2128, in runTest >> ? ?raise self.failureException(self.format_failure(new.getvalue())) >> AssertionError: Failed doctest test for profile_test >> ?File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp >> \profile_test.pyd", li >> ne 36, in profile_test >> >> ---------------------------------------------------------------------- >> File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp >> \profile_test.pyd", line >> 57, in profile_test >> Failed example: >> ? ?os.unlink(statsfile) >> Exception raised: >> ? ?Traceback (most recent call last): >> ? ? ?File "C:\Programs\Python25\lib\doctest.py", line 1228, in __run >> ? ? ? ?compileflags, 1) in test.globs >> ? ? ?File "", line 1, in >> ? ? ? ?os.unlink(statsfile) >> ? ?WindowsError: [Error 32] The process cannot access the file >> because it is be >> ing used by another process: 'c:\\docume~1\\carrasco\\locals~1\\temp\ >> \tmpm6kkx5' This is a pretty common error on windows, in many packages temp files don't get cleaned up because Windows still has an open file handle. Many packages just skip the unlink on Windows and leave some tempfiles on the system. I usually delete them manually every once in a while > > That's a strange one. If os.fork is available (though it shouldn't be > on Windows), could you try > > python runtests.py --no-fork profile_test I will run this tomorrow. > >> ---------------------------------------------------------------------- >> Ran 2950 tests in 1474.453s >> >> FAILED (failures=4, errors=14) > > Well, at least most of them passed :). Was that on a heavily loaded > system, or is MingW just a really slow compiler? Heavy use, and at memory capacity, I guess a lot of swapping might be done. I only use MingW so I cannot compare, but building scipy or running the full scipy test suite is usually the time for a coffee break. Thanks, Josef > > Thanks again, > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Fri Nov 6 08:15:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 5 Nov 2009 23:15:33 -0800 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: <1cd32cbb0911052159n8173c41y7c1e6a4242b51c8@mail.gmail.com> References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> <1cd32cbb0911052159n8173c41y7c1e6a4242b51c8@mail.gmail.com> Message-ID: <062F0AAD-4967-4443-B9FA-CEEDB4461A21@math.washington.edu> On Nov 5, 2009, at 9:59 PM, josef.pktd at gmail.com wrote: >> What happens when you do this in your Python? >> >> >>> float('+inf'), float('-inf'), float('nan') >> (inf, -inf, nan) >> >>> 1e300 * 1e300 >> inf >> >>> 1e300 * 1e300 * 0 >> nan > > only exceptions for string to float conversion: Thanks. This is all now http://trac.cython.org/cython_trac/ticket/450 I'm sure we're not the first to deal with it. >>> File "C:\Josef\work-oth\Cython-0.12.alpha0\BUILD\run\cpp >>> \profile_test.pyd", line >>> 57, in profile_test >>> Failed example: >>> os.unlink(statsfile) >>> Exception raised: >>> Traceback (most recent call last): >>> File "C:\Programs\Python25\lib\doctest.py", line 1228, in __run >>> compileflags, 1) in test.globs >>> File "", line 1, in >>> os.unlink(statsfile) >>> WindowsError: [Error 32] The process cannot access the file >>> because it is be >>> ing used by another process: 'c:\\docume~1\\carrasco\\locals~1\ >>> \temp\ >>> \tmpm6kkx5' > > This is a pretty common error on windows, in many packages temp files > don't get cleaned up because Windows still has an open file handle. > Many packages just skip the unlink on Windows and leave some tempfiles > on the system. I usually delete them manually every once in a while > >> >> That's a strange one. If os.fork is available (though it shouldn't be >> on Windows), could you try >> >> python runtests.py --no-fork profile_test > > I will run this tomorrow. No need, I bet the above explains it. I always wondered why Windows users periodically have to delete temp files... I've surrounded the unlink with a try...except block. > >> >>> ---------------------------------------------------------------------- >>> Ran 2950 tests in 1474.453s >>> >>> FAILED (failures=4, errors=14) >> >> Well, at least most of them passed :). Was that on a heavily loaded >> system, or is MingW just a really slow compiler? > > Heavy use, and at memory capacity, I guess a lot of swapping might > be done. > I only use MingW so I cannot compare, but building scipy or running > the full scipy test suite is usually the time for a coffee break. OK. It just took longer than I would have expected. - Robert From stefan_ml at behnel.de Fri Nov 6 09:00:36 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 06 Nov 2009 09:00:36 +0100 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> Message-ID: <4AF3D7A4.5020805@behnel.de> Robert Bradshaw, 06.11.2009 06:26: >> ====================================================================== >> ERROR: compiling (c) callingconvention >> ERROR: compiling (c) cnamespec >> ERROR: compiling (c) declarations >> ERROR: compiling (c) excvalcheck >> ERROR: compiling (c) nogil >> ---------------------------------------------------------------------- >> LinkError: command 'gcc' failed with exit status 1 > > These look like http://trac.cython.org/cython_trac/ticket/106 . > Unfortunately they've been around for a while, but I think it'll take > a windows developer to fix them. I'm keeping them on the radar though. > Anyone have any ideas? It'd be great to finally knock these out. Note > these are all compile (not run). They might be referring to non- > existent methods. In that case, I bet re-writing the tests could get > rid of these for once and for all. Yes, I think that's the problem. They use "cdef extern" declarations that refer to non-existing header files. A left-over of the Pyrex test suite. I think I fixed one or two of those, but some are really meant to test "cdef extern" stuff, so they require an external header file at least. IMHO, these tests actually merit a rewrite. Stefan From dalcinl at gmail.com Fri Nov 6 15:23:53 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Nov 2009 12:23:53 -0200 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: <4AF3D7A4.5020805@behnel.de> References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> <4AF3D7A4.5020805@behnel.de> Message-ID: On Fri, Nov 6, 2009 at 6:00 AM, Stefan Behnel wrote: > > Robert Bradshaw, 06.11.2009 06:26: >>> ====================================================================== >>> ERROR: compiling (c) callingconvention >>> ERROR: compiling (c) cnamespec >>> ERROR: compiling (c) declarations >>> ERROR: compiling (c) excvalcheck >>> ERROR: compiling (c) nogil >>> ---------------------------------------------------------------------- >>> LinkError: command 'gcc' failed with exit status 1 >> >> These look like http://trac.cython.org/cython_trac/ticket/106 . >> Unfortunately they've been around for a while, but I think it'll take >> a windows developer to fix them. I'm keeping them on the radar though. >> Anyone have any ideas? It'd be great to finally knock these out. Note >> these are all compile (not run). They might be referring to non- >> existent methods. In that case, I bet re-writing the tests could get >> rid of these for once and for all. > > Yes, I think that's the problem. They use "cdef extern" declarations that > refer to non-existing header files. A left-over of the Pyrex test suite. > I pointed this issue out one or two times in the past... But it is not my business to take decisions on things related to Windows. > > I think I fixed one or two of those, but some are really meant to test > "cdef extern" stuff, so they require an external header file at least. > and an external DLL perhaps ?? cdef extern void foo() generates this C code: __PYX_EXTERN_C DL_EXPORT(void) foo(void); /*proto*/ > > IMHO, these tests actually merit a rewrite. > Mmm, I would completely remove these kind of declarations, and deprecate them right now, in favor of "using cdef extern from ..." blocks. Then it is clear that a external header is required, and users are then charge of deciding what the C compiler actually see as a prototype. Better safe than sorry... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Nov 6 18:32:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Nov 2009 09:32:07 -0800 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> <4AF3D7A4.5020805@behnel.de> Message-ID: <3152FDBA-27A8-4A73-8DA1-A310EE03B70B@math.washington.edu> On Nov 6, 2009, at 6:23 AM, Lisandro Dalcin wrote: > On Fri, Nov 6, 2009 at 6:00 AM, Stefan Behnel > wrote: >> >> Robert Bradshaw, 06.11.2009 06:26: >>>> = >>>> = >>>> = >>>> =================================================================== >>>> ERROR: compiling (c) callingconvention >>>> ERROR: compiling (c) cnamespec >>>> ERROR: compiling (c) declarations >>>> ERROR: compiling (c) excvalcheck >>>> ERROR: compiling (c) nogil >>>> ---------------------------------------------------------------------- >>>> LinkError: command 'gcc' failed with exit status 1 >>> >>> These look like http://trac.cython.org/cython_trac/ticket/106 . >>> Unfortunately they've been around for a while, but I think it'll >>> take >>> a windows developer to fix them. I'm keeping them on the radar >>> though. >>> Anyone have any ideas? It'd be great to finally knock these out. >>> Note >>> these are all compile (not run). They might be referring to non- >>> existent methods. In that case, I bet re-writing the tests could get >>> rid of these for once and for all. >> >> Yes, I think that's the problem. They use "cdef extern" >> declarations that >> refer to non-existing header files. A left-over of the Pyrex test >> suite. >> > > I pointed this issue out one or two times in the past... Yes, I do recall this. > But it is not my business to take decisions on things related to > Windows. On the other hand, you're probably more knowledgeable about using Cython on Windows than most of us here. > >> I think I fixed one or two of those, but some are really meant to >> test >> "cdef extern" stuff, so they require an external header file at >> least. >> > > and an external DLL perhaps ?? As Lisandro said, I don't think the header file alone would help--cdef extern from and cdef extern are two different things. To turn these into run tests we would require a separate DLL that gets compiled in. Alternatively, we could test against something that's in the Python library but not in the Python header files (?). Or is there a way to force lazy symbol lookup? > cdef extern void foo() > > generates this C code: > > __PYX_EXTERN_C DL_EXPORT(void) foo(void); /*proto*/ > >> >> IMHO, these tests actually merit a rewrite. >> > > Mmm, I would completely remove these kind of declarations, and > deprecate them right now, in favor of "using cdef extern from ..." > blocks. Then it is clear that a external header is required, and users > are then charge of deciding what the C compiler actually see as a > prototype. Better safe than sorry... The tests are failing (due to obvious abuse of extern) but I don't see much user confusion over the feature so I don't think we should be removing it. For one thing, it makes writing cross-platform code easier (otherwise the user needs to create a separate .h file, and then worry about whether or not they're on Windows, if they're in a C+ + file, etc). - Robert From dalcinl at gmail.com Fri Nov 6 18:51:39 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Nov 2009 15:51:39 -0200 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: <3152FDBA-27A8-4A73-8DA1-A310EE03B70B@math.washington.edu> References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> <4AF3D7A4.5020805@behnel.de> <3152FDBA-27A8-4A73-8DA1-A310EE03B70B@math.washington.edu> Message-ID: On Fri, Nov 6, 2009 at 3:32 PM, Robert Bradshaw wrote: > >> But it is not my business to take decisions on things related to >> Windows. > > On the other hand, you're probably more knowledgeable about using > Cython on Windows than most of us here. > OK. Can I take that as a BDLF pronouncement about trusting me to fix annoying Windows issues? > > As Lisandro said, I don't think the header file alone would help--cdef > extern from and cdef extern are two different things. To turn these > into run tests we would require a separate DLL that gets compiled in. Indeed. > > Alternatively, we could test against something that's in the Python > library but not in the Python header files (?). > Yes, that could do the trick.. Or something in libc (and perhaps libm?) > > Or is there a way to force lazy symbol lookup? > AFAIK, no ... > > The tests are failing (due to obvious abuse of extern) but I don't see > much user confusion over the feature so I don't think we should be > removing it. > Let see... > For one thing, it makes writing cross-platform code > easier (otherwise the user needs to create a separate .h file, and > then worry about whether or not they're on Windows, if they're in a C+ > + file, etc). > Robert, I have to say it once more, second guessing for the sake of making it simple can actually make a lot of confusion: 1) What about if the extern declaration refers to a function implemented in a companion C source? The {DL_EXTERN} part is clearly wrong. 2) What about if the extern declaration refers to a function in a STATIC library? Then the {DL_EXTERN} part is perhaps wrong (not 100% sure what Win static libs are). 3) Why Cython should assume that the extern declaration is {extern "C"} ? So If do C++ I cannot use a C++ extern function? So... you see, things are not so easy...What's the point of supporting something that can really bit WinDog users (1)-(2) and has constrained use in C++ (and can also bit, if the extern function is actually a C++ one). I bet you will not want to introduce new syntax for letting users disambiguate (1)-(2) and (3), right? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Nov 6 19:42:11 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Nov 2009 10:42:11 -0800 Subject: [Cython] runtests results WindowXP MingW In-Reply-To: References: <1cd32cbb0911050746g2187e7a0md5e8349b16f1af93@mail.gmail.com> <7C46592D-D2E4-47BF-A118-E95CB1C8B03C@math.washington.edu> <4AF3D7A4.5020805@behnel.de> <3152FDBA-27A8-4A73-8DA1-A310EE03B70B@math.washington.edu> Message-ID: On Nov 6, 2009, at 9:51 AM, Lisandro Dalcin wrote: > On Fri, Nov 6, 2009 at 3:32 PM, Robert Bradshaw > wrote: >> >>> But it is not my business to take decisions on things related to >>> Windows. >> >> On the other hand, you're probably more knowledgeable about using >> Cython on Windows than most of us here. > > OK. Can I take that as a BDLF pronouncement about trusting me to fix > annoying Windows issues? I would *love* for someone to take up this banner. So yes, just document what you do, and don't introduce regressions. >> As Lisandro said, I don't think the header file alone would help-- >> cdef >> extern from and cdef extern are two different things. To turn these >> into run tests we would require a separate DLL that gets compiled in. > > Indeed. > >> >> Alternatively, we could test against something that's in the Python >> library but not in the Python header files (?). >> > > Yes, that could do the trick.. Or something in libc (and perhaps > libm?) Yeah, those could be good candidates as well. > >> The tests are failing (due to obvious abuse of extern) but I don't >> see >> much user confusion over the feature so I don't think we should be >> removing it. >> > > Let see... > >> For one thing, it makes writing cross-platform code >> easier (otherwise the user needs to create a separate .h file, and >> then worry about whether or not they're on Windows, if they're in a >> C+ >> + file, etc). >> > > Robert, I have to say it once more, second guessing for the sake of > making it simple can actually make a lot of confusion: > > 1) What about if the extern declaration refers to a function > implemented in a companion C source? The {DL_EXTERN} part is clearly > wrong. That should be an "cdef extern from" right? > 2) What about if the extern declaration refers to a function in a > STATIC library? Then the {DL_EXTERN} part is perhaps wrong (not 100% > sure what Win static libs are). Not sure about this either, but if such a thing exists and it's getting statically linked in I'd imagine it should be a "cdef extern from" as well. > 3) Why Cython should assume that the extern declaration is {extern > "C"} ? So If do C++ I cannot use a C++ extern function? Yes, that's a problem. (You'd have to use an external .h file right now.) > So... you see, things are not so easy...What's the point of supporting > something that can really bit WinDog users (1)-(2) and has constrained > use in C++ (and can also bit, if the extern function is actually a C++ > one). I bet you will not want to introduce new syntax for letting > users disambiguate (1)-(2) and (3), right? I'd rather not introduce a new syntax, but would be open to it if necessary. Thinking about this again, I think my primary reaction was due to backwards compatibility. This feature has been in Cython for a long time, and even in Pyrex, so slating it for removal is not something I'd like to do. It would be different if we were trying to add the feature, in which case more care would have to go into making sure it's always right (though the only real guessing I see is the C/C++, which could be fixed). In terms of ease of use vs. guessing, if it's well documented that cdef extern only handles the simplest case, and for anything more complicated you need to write your own header files, I think that would be a step forward. - Robert From stefan at sun.ac.za Sun Nov 8 10:20:49 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 8 Nov 2009 11:20:49 +0200 Subject: [Cython] Setting cython.cdivision(True) globally Message-ID: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> Hi all, What is the best way to enable cython.cdivision for a whole file? Regards St?fan From stefan at sun.ac.za Sun Nov 8 10:24:47 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 8 Nov 2009 11:24:47 +0200 Subject: [Cython] Releasing the gil in cdef without using cython.cdivision Message-ID: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> Hi all, Why is it necessary to decorate a pure "cdef" with @cython.cdivision(True) in order to release the gil? If I missed the discussion on this somewhere, feel free to just refer me back to the archive. Thanks! St?fan From greg.ewing at canterbury.ac.nz Mon Nov 9 01:40:48 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 09 Nov 2009 13:40:48 +1300 Subject: [Cython] Releasing the gil in cdef without using cython.cdivision In-Reply-To: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> References: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> Message-ID: <4AF76510.70403@canterbury.ac.nz> St?fan van der Walt wrote: > Why is it necessary to decorate a pure "cdef" with > @cython.cdivision(True) in order to release the gil? Probably because otherwise it assumes Python semantics for division, which requires making Python API calls, so the GIL is needed. -- Greg From dalcinl at gmail.com Mon Nov 9 15:55:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 9 Nov 2009 12:55:48 -0200 Subject: [Cython] Setting cython.cdivision(True) globally In-Reply-To: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> References: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> Message-ID: 2009/11/8 St?fan van der Walt : > Hi all, > > What is the best way to enable cython.cdivision for a whole file? > You can set any directive at the very beginning for the whole file, like this: http://code.google.com/p/mpi4py/source/browse/trunk/src/mpi4py.MPI.pyx -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Nov 9 17:40:47 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 9 Nov 2009 17:40:47 +0100 Subject: [Cython] Releasing the gil in cdef without using cython.cdivision In-Reply-To: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> References: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> Message-ID: <76f5d509a8772c92dd212af190e67488.squirrel@webmail.uio.no> > Hi all, > > Why is it necessary to decorate a pure "cdef" with > @cython.cdivision(True) in order to release the gil? > > If I missed the discussion on this somewhere, feel free to just refer > me back to the archive. If there's a division by zero in Python division mode, an exception must be raised, and to create the exception one must have the GIL... Dag Sverre From sccolbert at gmail.com Mon Nov 9 18:40:03 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 9 Nov 2009 18:40:03 +0100 Subject: [Cython] Releasing the gil in cdef without using cython.cdivision In-Reply-To: <76f5d509a8772c92dd212af190e67488.squirrel@webmail.uio.no> References: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> <76f5d509a8772c92dd212af190e67488.squirrel@webmail.uio.no> Message-ID: <7f014ea60911090940l4706cac1wccab2e192fe44364@mail.gmail.com> What was the rationality behind having to explicitly declare C division, versus explicitly declaring python division? Just curious. Cheers! On Mon, Nov 9, 2009 at 5:40 PM, Dag Sverre Seljebotn wrote: >> Hi all, >> >> Why is it necessary to decorate a pure "cdef" with >> @cython.cdivision(True) in order to release the gil? >> >> If I missed the discussion on this somewhere, feel free to just refer >> me back to the archive. > > If there's a division by zero in Python division mode, an exception must > be raised, and to create the exception one must have the GIL... > > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Mon Nov 9 18:44:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Nov 2009 18:44:42 +0100 Subject: [Cython] Releasing the gil in cdef without using cython.cdivision In-Reply-To: <7f014ea60911090940l4706cac1wccab2e192fe44364@mail.gmail.com> References: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> <76f5d509a8772c92dd212af190e67488.squirrel@webmail.uio.no> <7f014ea60911090940l4706cac1wccab2e192fe44364@mail.gmail.com> Message-ID: <4AF8550A.1050604@behnel.de> Chris Colbert, 09.11.2009 18:40: > What was the rationality behind having to explicitly declare C > division, versus explicitly declaring python division? Python code semantics. http://wiki.cython.org/enhancements/division http://thread.gmane.org/gmane.comp.python.cython.devel/4769/focus=4938 Stefan From robertwb at math.washington.edu Mon Nov 9 19:30:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Nov 2009 10:30:09 -0800 Subject: [Cython] Setting cython.cdivision(True) globally In-Reply-To: References: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> Message-ID: On Nov 9, 2009, at 6:55 AM, Lisandro Dalcin wrote: > 2009/11/8 St?fan van der Walt : >> Hi all, >> >> What is the best way to enable cython.cdivision for a whole file? >> > > You can set any directive at the very beginning for the whole file, > like this: > > http://code.google.com/p/mpi4py/source/browse/trunk/src/mpi4py.MPI.pyx See also http://wiki.cython.org/enhancements/compilerdirectives - Robert From sccolbert at gmail.com Mon Nov 9 19:45:45 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 9 Nov 2009 19:45:45 +0100 Subject: [Cython] Setting cython.cdivision(True) globally In-Reply-To: References: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> Message-ID: <7f014ea60911091045l2faa54b9y9521205848e77abd@mail.gmail.com> is there an error in that table: ''' cdivision True/False Version >= 0.12 If set to True, Cython will adjust the remainder and quotient operators C types to match those of Python ints (which differ when the operands have opposite signs) and raise a ZeroDivisionError when the right operand is 0. This has about a 35% speed penalty. See CEP 516 ''' Shouldn't that be, "If set to False,...." ? On Mon, Nov 9, 2009 at 7:30 PM, Robert Bradshaw wrote: > On Nov 9, 2009, at 6:55 AM, Lisandro Dalcin wrote: > >> 2009/11/8 St?fan van der Walt : >>> Hi all, >>> >>> What is the best way to enable cython.cdivision for a whole file? >>> >> >> You can set any directive at the very beginning for the whole file, >> like this: >> >> http://code.google.com/p/mpi4py/source/browse/trunk/src/mpi4py.MPI.pyx > > See also > > http://wiki.cython.org/enhancements/compilerdirectives > > - Robert > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Mon Nov 9 19:52:02 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Nov 2009 10:52:02 -0800 Subject: [Cython] Releasing the gil in cdef without using cython.cdivision In-Reply-To: <4AF8550A.1050604@behnel.de> References: <9457e7c80911080124r2180836bs388aae1817da0658@mail.gmail.com> <76f5d509a8772c92dd212af190e67488.squirrel@webmail.uio.no> <7f014ea60911090940l4706cac1wccab2e192fe44364@mail.gmail.com> <4AF8550A.1050604@behnel.de> Message-ID: <9AB4DA64-5E65-4839-BA2A-1DBB50AF4F91@math.washington.edu> On Nov 9, 2009, at 9:44 AM, Stefan Behnel wrote: > Chris Colbert, 09.11.2009 18:40: >> What was the rationality behind having to explicitly declare C >> division, versus explicitly declaring python division? This is a very good question, and if you look at the links below (and the discussions on several other mailing lists) you can see that the decision was not trivial. > Python code semantics. > > http://wiki.cython.org/enhancements/division > http://thread.gmane.org/gmane.comp.python.cython.devel/4769/focus=4938 One thing that does not come out as clear in the above (perhaps it was not as much on people's radar at the time) is that preserving Python semantics on C types is important for eventual type inference (especially if the inference engine grows stronger with time). Of course there are still possible overflow issues, but that's more "obvious." - Robert From robertwb at math.washington.edu Mon Nov 9 19:54:38 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Nov 2009 10:54:38 -0800 Subject: [Cython] Setting cython.cdivision(True) globally In-Reply-To: <7f014ea60911091045l2faa54b9y9521205848e77abd@mail.gmail.com> References: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> <7f014ea60911091045l2faa54b9y9521205848e77abd@mail.gmail.com> Message-ID: <37349C8C-7BC9-4EB0-BEAB-1E063CE6D79C@math.washington.edu> On Nov 9, 2009, at 10:45 AM, Chris Colbert wrote: > is there an error in that table: > > ''' > > cdivision True/False Version >= 0.12 > > If set to True, Cython will adjust the remainder and quotient > operators C types to match those of Python ints (which differ when the > operands have opposite signs) and raise a ZeroDivisionError when the > right operand is 0. This has about a 35% speed penalty. See CEP 516 > > ''' > > Shouldn't that be, "If set to False,...." ? Oh, oops... Fixed. - Robert From sccolbert at gmail.com Mon Nov 9 19:57:43 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 9 Nov 2009 19:57:43 +0100 Subject: [Cython] Setting cython.cdivision(True) globally In-Reply-To: <37349C8C-7BC9-4EB0-BEAB-1E063CE6D79C@math.washington.edu> References: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> <7f014ea60911091045l2faa54b9y9521205848e77abd@mail.gmail.com> <37349C8C-7BC9-4EB0-BEAB-1E063CE6D79C@math.washington.edu> Message-ID: <7f014ea60911091057p7f349f9t4cf595b077d57aec@mail.gmail.com> I noticed you just changed the version signature too: Version < 0.12 Does that mean we can't turn that off in >= 0.12, or must we just explicitly declare it? On Mon, Nov 9, 2009 at 7:54 PM, Robert Bradshaw wrote: > On Nov 9, 2009, at 10:45 AM, Chris Colbert wrote: > >> is there an error in that table: >> >> ''' >> >> cdivision ? True/False ?Version >= 0.12 >> >> If set to True, Cython will adjust the remainder and quotient >> operators C types to match those of Python ints (which differ when the >> operands have opposite signs) and raise a ZeroDivisionError when the >> right operand is 0. This has about a 35% speed penalty. See CEP 516 >> >> ''' >> >> Shouldn't that be, "If set to False,...." ? > > Oh, oops... Fixed. > > - Robert > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Mon Nov 9 20:03:01 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Nov 2009 11:03:01 -0800 Subject: [Cython] Setting cython.cdivision(True) globally In-Reply-To: <7f014ea60911091057p7f349f9t4cf595b077d57aec@mail.gmail.com> References: <9457e7c80911080120h4b81b53bv78b155c6316908e5@mail.gmail.com> <7f014ea60911091045l2faa54b9y9521205848e77abd@mail.gmail.com> <37349C8C-7BC9-4EB0-BEAB-1E063CE6D79C@math.washington.edu> <7f014ea60911091057p7f349f9t4cf595b077d57aec@mail.gmail.com> Message-ID: <3614F1AE-C1DA-4FF3-BACB-1E5AD7C8AB38@math.washington.edu> On Nov 9, 2009, at 10:57 AM, Chris Colbert wrote: > I noticed you just changed the version signature too: > > Version < 0.12 > > Does that mean we can't turn that off in >= 0.12, or must we just > explicitly declare it? That's the default value. In Cython < 0.12, you had to explicitly declare Python semantics. For Cython 0.12+ you have to explicitly declare C semantics. Yes, it's a huge change, but we decided it was worth the cost. To get Sage to compile I just explicitly enabled c division everywhere. - Robert From stefan_ml at behnel.de Tue Nov 10 08:47:35 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Nov 2009 08:47:35 +0100 Subject: [Cython] Compiling pure Python mode with distutils In-Reply-To: References: <5DF7C4A2-0FD3-44F0-AC93-6AB1DFCC39EA@hetland.org> <026BA7D9-4C2E-4998-AEB2-328899EBA48A@hetland.org> <25DADA9C-D630-438F-957B-6B29D02E11A4@math.washington.edu> <4AB1DA34.4030103@behnel.de> <14307F2D-8951-4BBF-863B-78A3C7F0A2D9@hetland.org> <4AB279F3.2040909@behnel.de> Message-ID: <4AF91A97.8060903@behnel.de> Magnus Lie Hetland, 17.09.2009 21:02: > why would it need to find > the package to do the compilation? Because it needs to know the package layout in order to find related modules and .pxd files at compile time and runtime. That doesn't mean it couldn't potentially look for the module.pxd file in the same directory as the .py or .pyx file. It just means that it needs to know the final package hierarchy. > If there's a .pxd file in the same > directory as the .py(x) file, couldn't that be enough? Even in Python 3, the equivalent isn't enough. In that case, you'd have to import ".module" for a relative import. > 1. I'd like to automate the process of finding compilable files, so I > don't have to manually update my setup.py file all the time. > Previously, I globbed for .pyx files You may consider looking for .pxd files instead and compiling the .py files with the same name. Or you can just try to compile everything and ignore errors. > (because I'm forced to > use .py files to get cython to compile pure Python mode correctly, it > seems) "correctly" is not the right wording here. The Cython language is different from the Python language. Cython can compile both, but that doesn't mean you can use file extensions interchangeably. Renaming ".py" to ".c" doesn't magically translate your Python code to C code, either. > I can't do that. So I need some way of discerning Cython > source files from Python files. I thought I'd keep the Python files in > the actual package hierarchy (where they belong), and the Cython > source somewhere else, putting the .so files in the package hierarchy > (where they, too, belong). You can do that, but you still need to keep them in the correct package hierarchy. Just use separate source directories for them. Note that you can also use __init__.pyx instead of __init__.py, which might help here. > 2. I'm just using standard mechanisms for installing, and Distutils > will happily install the Cython source files (with .py endings) > alongside the corresponding .so files. I don't really want this, > although it doesn't hurt ... except it makes me a bit uneasy to think > that there's a chance that the .py file will be imported instead of > the .so file. Not sure if there's a clearly documented Python behavior > here? Not "Python behaviour", but CPython loads .so files before .py files. Plus, having the .py file next to the .so makes it easier for users to look up the code, so that's actually a feature. > 3. I'd like to be able to run tests and so forth in two ways -- one > interpreting the Cython .py files (for coverage, among other things) > and one using the .so files. Now, that's a good reason to use .py instead of .pyx, don't you think? > Haven't thought through exactly how I'll > implement this, but it seems it'll be easier to tell Python what to > import if the (compiled) .py files and their .so files aren't in the > same directory. Then don't install your code and keep the sources in separate source directories. Setting the PYTHONPATH appropriately will do the trick. Or, you can keep them in the same directory and compile the code only after running the coverage tests. Stefan From seb.binet at gmail.com Tue Nov 10 19:59:59 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Tue, 10 Nov 2009 19:59:59 +0100 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <200911041151.59460.binet@cern.ch> Message-ID: <200911101959.59571.binet@cern.ch> On Wednesday 04 November 2009 18:36:29 Craig Citro wrote: > > for ease of migration, distribute also puts (among other things) a > > 'setuptools' entry in sys.modules. > > so nothing else to do on the cython side. > > Will this still be true when the Distribute 0.7 series gets released? I don't know. it is not clear either from the readme in the bitbucket branch. Tarek surely knows :) cheers, sebastien. > > -cc > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From dalcinl at gmail.com Tue Nov 10 22:32:34 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Nov 2009 18:32:34 -0300 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <200911101959.59571.binet@cern.ch> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <200911041151.59460.binet@cern.ch> <200911101959.59571.binet@cern.ch> Message-ID: On Tue, Nov 10, 2009 at 3:59 PM, Sebastien Binet wrote: > On Wednesday 04 November 2009 18:36:29 Craig Citro wrote: >> > for ease of migration, distribute also puts (among other things) a >> > 'setuptools' entry in sys.modules. >> > so nothing else to do on the cython side. >> >> Will this still be true when the Distribute 0.7 series gets released? > > I don't know. > it is not clear either from the readme in the bitbucket branch. > > Tarek surely knows :) > Tarek just told me that the 'setuptools' entry WILL NOT be put by Distribute >= 0.7 -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From seb.binet at gmail.com Tue Nov 10 22:45:20 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Tue, 10 Nov 2009 22:45:20 +0100 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <200911101959.59571.binet@cern.ch> Message-ID: <200911102245.20610.binet@cern.ch> On Tuesday 10 November 2009 22:32:34 Lisandro Dalcin wrote: > On Tue, Nov 10, 2009 at 3:59 PM, Sebastien Binet wrote: > > On Wednesday 04 November 2009 18:36:29 Craig Citro wrote: > >> > for ease of migration, distribute also puts (among other things) a > >> > 'setuptools' entry in sys.modules. > >> > so nothing else to do on the cython side. > >> > >> Will this still be true when the Distribute 0.7 series gets released? > > > > I don't know. > > it is not clear either from the readme in the bitbucket branch. > > > > Tarek surely knows :) > > Tarek just told me that the 'setuptools' entry WILL NOT be put by > Distribute >= 0.7 yeah, I guess his answer bounced off cython-dev... see in attachement.. cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### -------------- next part -------------- An embedded message was scrubbed... From: =?ISO-8859-1?Q?Tarek_Ziad=E9?= Subject: Re: [Cython] [patch] Optional setuptools-based cython build Date: Tue, 10 Nov 2009 21:40:10 +0100 Size: 4164 Url: http://codespeak.net/pipermail/cython-dev/attachments/20091110/d96d6d5c/attachment.eml From robertwb at math.washington.edu Wed Nov 11 07:44:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Nov 2009 22:44:28 -0800 Subject: [Cython] [patch] Optional setuptools-based cython build In-Reply-To: <5b8d13220911031613x21104e13iffbfab7234da8970@mail.gmail.com> References: <5b8d13220911010132w16f9f3d7u2c388fa4e369a7c1@mail.gmail.com> <1E685FD2-392B-4F69-98FC-9C5652436B72@math.washington.edu> <5b8d13220911030312x52613fc4m80523905aae3df8d@mail.gmail.com> <5b8d13220911031613x21104e13iffbfab7234da8970@mail.gmail.com> Message-ID: On Nov 3, 2009, at 4:13 PM, David Cournapeau wrote: > On Wed, Nov 4, 2009 at 3:27 AM, Robert Bradshaw > wrote: > >> >> How would you know, in general, that setuptools has not previously >> been imported? > > The reason for this line is precisely to detect whether setuptools has > been previously imported or not. OK, I've pushed your patch as is. We'll deal with how to directly support Distribute once 0.7 comes out. - Robert From robertwb at math.washington.edu Wed Nov 11 08:11:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Nov 2009 23:11:58 -0800 Subject: [Cython] runtests.py locks In-Reply-To: <4AF07904.1020204@student.matnat.uio.no> References: <4AF045B0.4090109@student.matnat.uio.no> <4AF07904.1020204@student.matnat.uio.no> Message-ID: <19E25CA7-478B-478B-A1A5-014A380DE687@math.washington.edu> On Nov 3, 2009, at 10:40 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Nov 3, 2009, at 7:01 AM, Dag Sverre Seljebotn wrote: >> >> >>> I lost some time over what appears to be a problem in runtests.py; >>> the line >>> >>> PartialTestResult.join_results(result, pickle.load(input)) >>> >>> never returns if and only if some test fails (!!) >>> >>> My current strategy is to remove the multiprocessing stuff >>> locally...I'm >>> not in a mood to spend more time on this now... >>> >> >> I've been having the same issue, with little luck chasing it down >> (though admittedly little time put into it). Try the recently >> introduced --no-fork option... >> >> > How about enabling this for version 2.6+ only and use multiprocessing? One issue is that without it one never tests both c and cpp as modules are only loaded once (though I suppose we could mangle the module names). In any case, this is the one last blocker for 0.12. - Robert From stefan_ml at behnel.de Wed Nov 11 08:37:11 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Nov 2009 08:37:11 +0100 Subject: [Cython] @classmethod optimisation broken In-Reply-To: References: Message-ID: <4AFA69A7.3030602@behnel.de> Hi Lisandro, good catch! Lisandro Dalcin, 10.11.2009 19:10: > Stefan, the __new__ optimization is not taking place in the case > below... Any good reason? Am I missing something? > > diff -r 7e9af512c2eb tests/run/tp_new.pyx > +cdef class Foo: > + @classmethod > + def new(type cls): > + return cls.__new__(cls) > + > +def make_new_foo(): > + """ > + >>> isinstance(make_new_foo(), Foo) > + True > + """ > + return Foo.new() This isn't because of the __new__ optimisation but because of the way @classmethod is handled. In the tree, "cls" is typed as Foo, which is totally wrong for a classmethod. It even overrides your declaration as "type" and may lead to all sorts of incorrect code, depending on what you do in your factory. http://trac.cython.org/cython_trac/ticket/454 I added the test case above (disabled in bugs.txt). Stefan From dalcinl at gmail.com Wed Nov 11 16:16:37 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Nov 2009 13:16:37 -0200 Subject: [Cython] blocker for 0.12 Message-ID: I marked this as a 0.12 blocker. Perhaps it is too much (though a working patch is already provided), but figuring out the problem took a fair amount of time while helping Chris Colbert to debug the issue. Please review: http://trac.cython.org/cython_trac/ticket/455 -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From sturla at molden.no Wed Nov 11 20:42:36 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 11 Nov 2009 20:42:36 +0100 Subject: [Cython] buffer enhancements in 0.12 or later? Message-ID: <4AFB13AC.2050704@molden.no> I promised Dag Sverre to add test cases for my changes to the buffer syntax. Unfortunately I've been busy lately, and the computer I mainly use for programming broke down (damned Vista now replaced with Windows 7...) I think I could have test cases ready later this week. I am not sure if that is too late for 0.12? Also I would like to add the volatile keyword argument (as discussed here previously), as it belongs together with the readonly and restrict qualifiers. It will only require a small chage to the code I wrote, but be important if a Py_buffer is modified by means external to the thread (hardware interrupt, different thread, another process). The code used for readonly and restrict seems to be stabile. I have compiled rather large numerical routines with them, and the code compiles runs without error. The only issue is that Cython does not yet detect an attempt to write to a readonly buffer, but the C compiler does. I'll try to fix that, but right now it has lower priority. Anyway, do you think these changes should be postponed until a release after 0.12? Sturla Molden From robertwb at math.washington.edu Wed Nov 11 21:50:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Nov 2009 12:50:52 -0800 Subject: [Cython] blocker for 0.12 In-Reply-To: References: Message-ID: <7FDF0671-3469-4AF8-A8B4-4B15E5F20DDC@math.washington.edu> On Nov 11, 2009, at 7:16 AM, Lisandro Dalcin wrote: > I marked this as a 0.12 blocker. Perhaps it is too much (though a > working patch is already provided), but figuring out the problem took > a fair amount of time while helping Chris Colbert to debug the issue. > > Please review: > > http://trac.cython.org/cython_trac/ticket/455 Thanks for tracking this down. It fixed the issue named, but broke cascaded comparison (i.e. you couldn't just "assert not coerce_result"). I pushed what you have and a followup to fix cascaded comparison too (couldn't think of a clean way to do this without introducing two on-line utility functions). - Robert From robertwb at math.washington.edu Wed Nov 11 21:52:10 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Nov 2009 12:52:10 -0800 Subject: [Cython] buffer enhancements in 0.12 or later? In-Reply-To: <4AFB13AC.2050704@molden.no> References: <4AFB13AC.2050704@molden.no> Message-ID: On Nov 11, 2009, at 11:42 AM, Sturla Molden wrote: > I promised Dag Sverre to add test cases for my changes to the buffer > syntax. Unfortunately I've been busy lately, and the computer I mainly > use for programming broke down (damned Vista now replaced with Windows > 7...) Sorry about that... > I think I could have test cases ready later this week. I am not sure > if > that is too late for 0.12? Also I would like to add the volatile > keyword > argument (as discussed here previously), as it belongs together with > the > readonly and restrict qualifiers. It will only require a small > chage to > the code I wrote, but be important if a Py_buffer is modified by means > external to the thread (hardware interrupt, different thread, another > process). > > The code used for readonly and restrict seems to be stabile. I have > compiled rather large numerical routines with them, and the code > compiles runs without error. The only issue is that Cython does not > yet > detect an attempt to write to a readonly buffer, but the C compiler > does. I'll try to fix that, but right now it has lower priority. > > Anyway, do you think these changes should be postponed until a release > after 0.12? Yeah, at this point we shouldn't be introducing anything new into 0.12, but hopefully we won't take as long with 0.12.1. - Robert From dalcinl at gmail.com Wed Nov 11 22:20:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Nov 2009 19:20:48 -0200 Subject: [Cython] blocker for 0.12 In-Reply-To: <7FDF0671-3469-4AF8-A8B4-4B15E5F20DDC@math.washington.edu> References: <7FDF0671-3469-4AF8-A8B4-4B15E5F20DDC@math.washington.edu> Message-ID: On Wed, Nov 11, 2009 at 6:50 PM, Robert Bradshaw wrote: > On Nov 11, 2009, at 7:16 AM, Lisandro Dalcin wrote: > >> I marked this as a 0.12 blocker. Perhaps it is too much (though a >> working patch is already provided), but figuring out the problem took >> a fair amount of time while helping Chris Colbert to debug the issue. >> >> Please review: >> >> http://trac.cython.org/cython_trac/ticket/455 > > Thanks for tracking this down. It fixed the issue named, but broke > cascaded comparison (i.e. you couldn't just "assert not > coerce_result"). Interesting... I ran the whole testsuite and I did not noticed any failure... perhaps I just got confused and did not see the problem... Anyway, I put assert there just to try to spot problems, I knew my patch was broken ;-) > I pushed what you have and a followup to fix cascaded > comparison too (couldn't think of a clean way to do this without > introducing two on-line utility functions). > Yes... it is a bit ugly, but no point in wasting more time to make it look better :-) So, many thanks!!. Should I close the ticket? Or shall you do that? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Wed Nov 11 22:28:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Nov 2009 13:28:09 -0800 Subject: [Cython] blocker for 0.12 In-Reply-To: References: <7FDF0671-3469-4AF8-A8B4-4B15E5F20DDC@math.washington.edu> Message-ID: On Nov 11, 2009, at 1:20 PM, Lisandro Dalcin wrote: > On Wed, Nov 11, 2009 at 6:50 PM, Robert Bradshaw > wrote: >> On Nov 11, 2009, at 7:16 AM, Lisandro Dalcin wrote: >> >>> I marked this as a 0.12 blocker. Perhaps it is too much (though a >>> working patch is already provided), but figuring out the problem >>> took >>> a fair amount of time while helping Chris Colbert to debug the >>> issue. >>> >>> Please review: >>> >>> http://trac.cython.org/cython_trac/ticket/455 >> >> Thanks for tracking this down. It fixed the issue named, but broke >> cascaded comparison (i.e. you couldn't just "assert not >> coerce_result"). > > Interesting... I ran the whole testsuite and I did not noticed any > failure... perhaps I just got confused and did not see the problem... > Anyway, I put assert there just to try to spot problems, I knew my > patch was broken ;-) That's because the test suite is still woefully incomplete... I added some relevant tests in that last patch. >> I pushed what you have and a followup to fix cascaded >> comparison too (couldn't think of a clean way to do this without >> introducing two on-line utility functions). >> > > Yes... it is a bit ugly, but no point in wasting more time to make it > look better :-) > > So, many thanks!!. Should I close the ticket? Or shall you do that? Yes, you can go ahead and close the ticket. - Robert From dalcinl at gmail.com Wed Nov 11 22:38:32 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Nov 2009 19:38:32 -0200 Subject: [Cython] blocker for 0.12 In-Reply-To: References: <7FDF0671-3469-4AF8-A8B4-4B15E5F20DDC@math.washington.edu> Message-ID: On Wed, Nov 11, 2009 at 7:28 PM, Robert Bradshaw wrote: > > Yes, you can go ahead and close the ticket. > Done. Many thanks, Robert! -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Nov 13 04:08:38 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Nov 2009 04:08:38 +0100 Subject: [Cython] Implementing generators Message-ID: <4AFCCDB6.3090209@behnel.de> Hi all, warning: long e-mail ahead, don't read in a hurry! I gave generator functions a couple of thoughts. Implementing them actually sounds simpler than it is, not because of the state keeping, but because of the refactoring (point 1 below) that I would like to see done before going there. Here's what I think should be done: 1) refactor def functions into a Python wrapper and a static C function * Python wrapper does all argument unpacking, return value packing and the final exception propagation * C function contains the complete body of the original function and returns the return value directly a) non-closure functions: - C function has signature as written in the code - Python wrapper calls C function to execute the body b) closure functions: - C function has METH_NOARGS signature - Python wrapper creates closure and fills in arguments - Python wrapper calls C function with closure as 'self' 2) support writing utility code in Cython (does this work already?) * likely just compile TreeFragments inside of the utility_scope? (does the utility_scope actually have a unique mangling prefix or will it interfere with a user provided "utility" module?) 3) implement a generic 'generator' type in Cython code (see code below) * methods: __iter__, __next__, send, throw, close (as in PEP 342, see http://www.python.org/dev/peps/pep-0342/ ) * fields: closure, exception, __weakref__, C function pointer 4) implement generators as extension to 1b) * Python wrapper works mostly as in 1b), but - does not call the C function - creates and returns a generator instance instead and fills in the created closure and the pointer to the C function part of the generator function * generator functions become modified closure functions: - METH_O signature instead of METH_NOARGS to receive the send(x) value directly (note that gen.__next__() is defined as gen.send(None) and gen.throw(exc) could be implemented as gen.send(NULL)) - closures additionally contain function temps (I'm thinking of a union of structs, i.e. one struct for each set of temps that existed during the code generation for a yield node, but I guess storing all temps is just fine to start with - won't impact performance, just memory) - closures have an additional C field to store the execution state (void* to a function label, initially NULL) - "sendval = (yield [expr])" emits the following code: - store away all current temp values in the closure - set "closure._resume_label" to the resume label (see below, uses the C operator "&&") - return the expression result (or None) - return immediately without cleanup (the temp that holds the expression result must be unmanaged to prevent DECREF()-ing on resume; INCREF()-ing the return value will keep it alive for too long) - here goes the resume label ("__Lxyz_resume_from_yield:") - reset all saved temp values from the closure - if an exception is to be raised (gen.throw() was called, which has already set the exception externally), use normal exception path - set the result temp of the yield node to the send value argument that was passed (INCREF or not, as for parameters) * generator C function basically implements gen.send(x) - receives both the closure and the current send value as parameters - if "closure._resume_label" is not NULL, jump to the label; otherwise, check that 'x' is None (raise an exception if not) and execute the function body normally So the main work that's left to be done in 4) will be the closure extension to include the temps and the yield/resume implementation. Here's the (trivial) generic generator type: cdef class generator: cdef object _closure cdef meth_o_func* _run cdef object __weakref__ def __iter__(self): return self def __next__(self): return self._run(self._closure, None) def send(self, value): return self._run(self._closure, value) def throw(self, type, value=None, traceback=None): EXC_RESET(type, value, traceback) return self._run(self._closure, NULL) def close(self): try: EXC_RESET(GeneratorExit, NULL, NULL) self._run(self._closure, NULL) except (GeneratorExit, StopIteration): pass else: raise RuntimeError('generator ignored GeneratorExit') I wonder if there is a way to make it inherit from CPython's GeneratorType. That would enhance the interoperability, but it would also mean that we add some unnecessary instance size overhead and that we have to prevent that base-type from doing anything, including initialisation and final cleanup. The separation in 1a) has also been requested by Lisandro (and likely others) a while ago to make the function setup code more readable. Currently, the argument unpacking code takes so much space that it's easy to get lost when trying to read the generated function code, especially in short functions. The refactoring for 1) actually conflicts a bit with cpdef functions, which do the exact opposite: they create a DefNode for an existing C function. I wonder if it makes sense to swap that while we're at it. That would reduce some redundancy. Ok, this is a rather lengthy e-mail that's a bit akin to a spec already. Does this make sense to everybody? Any objections or ideas? Anyone happy to give a hand? :) Stefan From dagss at student.matnat.uio.no Fri Nov 13 04:52:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Nov 2009 19:52:59 -0800 Subject: [Cython] Implementing generators In-Reply-To: <4AFCCDB6.3090209@behnel.de> References: <4AFCCDB6.3090209@behnel.de> Message-ID: <4AFCD81B.1020608@student.matnat.uio.no> Stefan Behnel wrote: > Hi all, > > warning: long e-mail ahead, don't read in a hurry! > > I gave generator functions a couple of thoughts. Implementing them actually > sounds simpler than it is, not because of the state keeping, but because of > the refactoring (point 1 below) that I would like to see done before going > there. > > Here's what I think should be done: > > 1) refactor def functions into a Python wrapper and a static C function > * Python wrapper does all argument unpacking, return value packing and > the final exception propagation > * C function contains the complete body of the original function and > returns the return value directly > a) non-closure functions: > - C function has signature as written in the code > - Python wrapper calls C function to execute the body > b) closure functions: > - C function has METH_NOARGS signature > - Python wrapper creates closure and fills in arguments > - Python wrapper calls C function with closure as 'self' > > 2) support writing utility code in Cython (does this work already?) > * likely just compile TreeFragments inside of the utility_scope? > (does the utility_scope actually have a unique mangling prefix > or will it interfere with a user provided "utility" module?) > I am in a hurry and am going to be for another two weeks, so I'll just answer this: This is partly implemented in the kurt-gsoc branch. I'm hoping to do some work on getting that merged in December. I'll try to get back to this thread in two weeks. Dag Sverre > 3) implement a generic 'generator' type in Cython code (see code below) > * methods: __iter__, __next__, send, throw, close (as in PEP 342, see > http://www.python.org/dev/peps/pep-0342/ ) > * fields: closure, exception, __weakref__, C function pointer > > 4) implement generators as extension to 1b) > * Python wrapper works mostly as in 1b), but > - does not call the C function > - creates and returns a generator instance instead and fills in the > created closure and the pointer to the C function part of the > generator function > * generator functions become modified closure functions: > - METH_O signature instead of METH_NOARGS to receive the send(x) value > directly (note that gen.__next__() is defined as gen.send(None) and > gen.throw(exc) could be implemented as gen.send(NULL)) > - closures additionally contain function temps (I'm thinking of a > union of structs, i.e. one struct for each set of temps that existed > during the code generation for a yield node, but I guess storing > all temps is just fine to start with - won't impact performance, > just memory) > - closures have an additional C field to store the execution state > (void* to a function label, initially NULL) > - "sendval = (yield [expr])" emits the following code: > - store away all current temp values in the closure > - set "closure._resume_label" to the resume label (see below, uses > the C operator "&&") > - return the expression result (or None) - return immediately > without cleanup (the temp that holds the expression result must be > unmanaged to prevent DECREF()-ing on resume; INCREF()-ing the > return value will keep it alive for too long) > - here goes the resume label ("__Lxyz_resume_from_yield:") > - reset all saved temp values from the closure > - if an exception is to be raised (gen.throw() was called, which has > already set the exception externally), use normal exception path > - set the result temp of the yield node to the send value argument > that was passed (INCREF or not, as for parameters) > * generator C function basically implements gen.send(x) > - receives both the closure and the current send value as parameters > - if "closure._resume_label" is not NULL, jump to the label; > otherwise, check that 'x' is None (raise an exception if not) and > execute the function body normally > > So the main work that's left to be done in 4) will be the closure extension > to include the temps and the yield/resume implementation. > > Here's the (trivial) generic generator type: > > cdef class generator: > cdef object _closure > cdef meth_o_func* _run > cdef object __weakref__ > > def __iter__(self): > return self > > def __next__(self): > return self._run(self._closure, None) > > def send(self, value): > return self._run(self._closure, value) > > def throw(self, type, value=None, traceback=None): > EXC_RESET(type, value, traceback) > return self._run(self._closure, NULL) > > def close(self): > try: > EXC_RESET(GeneratorExit, NULL, NULL) > self._run(self._closure, NULL) > except (GeneratorExit, StopIteration): > pass > else: > raise RuntimeError('generator ignored GeneratorExit') > > I wonder if there is a way to make it inherit from CPython's GeneratorType. > That would enhance the interoperability, but it would also mean that we add > some unnecessary instance size overhead and that we have to prevent that > base-type from doing anything, including initialisation and final cleanup. > > The separation in 1a) has also been requested by Lisandro (and likely > others) a while ago to make the function setup code more readable. > Currently, the argument unpacking code takes so much space that it's easy > to get lost when trying to read the generated function code, especially in > short functions. > > The refactoring for 1) actually conflicts a bit with cpdef functions, which > do the exact opposite: they create a DefNode for an existing C function. I > wonder if it makes sense to swap that while we're at it. That would reduce > some redundancy. > > Ok, this is a rather lengthy e-mail that's a bit akin to a spec already. > Does this make sense to everybody? Any objections or ideas? Anyone happy to > give a hand? :) > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Fri Nov 13 05:04:49 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Nov 2009 20:04:49 -0800 Subject: [Cython] Implementing generators In-Reply-To: <4AFCD81B.1020608@student.matnat.uio.no> References: <4AFCCDB6.3090209@behnel.de> <4AFCD81B.1020608@student.matnat.uio.no> Message-ID: <4AFCDAE1.1060404@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: > >> Hi all, >> >> warning: long e-mail ahead, don't read in a hurry! >> >> I gave generator functions a couple of thoughts. Implementing them actually >> sounds simpler than it is, not because of the state keeping, but because of >> the refactoring (point 1 below) that I would like to see done before going >> there. >> >> Here's what I think should be done: >> >> 1) refactor def functions into a Python wrapper and a static C function >> * Python wrapper does all argument unpacking, return value packing and >> the final exception propagation >> * C function contains the complete body of the original function and >> returns the return value directly >> a) non-closure functions: >> - C function has signature as written in the code >> - Python wrapper calls C function to execute the body >> b) closure functions: >> - C function has METH_NOARGS signature >> - Python wrapper creates closure and fills in arguments >> - Python wrapper calls C function with closure as 'self' >> >> 2) support writing utility code in Cython (does this work already?) >> * likely just compile TreeFragments inside of the utility_scope? >> (does the utility_scope actually have a unique mangling prefix >> or will it interfere with a user provided "utility" module?) >> >> > I am in a hurry and am going to be for another two weeks, so I'll just > answer this: This is partly implemented in the kurt-gsoc branch. I'm > hoping to do some work on getting that merged in December. > partly = It does work, but perhaps not exactly as you'd want it (string-based rather than TreeFragment, though that is easily changed). Dag Sverre > I'll try to get back to this thread in two weeks. > > Dag Sverre > > >> 3) implement a generic 'generator' type in Cython code (see code below) >> * methods: __iter__, __next__, send, throw, close (as in PEP 342, see >> http://www.python.org/dev/peps/pep-0342/ ) >> * fields: closure, exception, __weakref__, C function pointer >> >> 4) implement generators as extension to 1b) >> * Python wrapper works mostly as in 1b), but >> - does not call the C function >> - creates and returns a generator instance instead and fills in the >> created closure and the pointer to the C function part of the >> generator function >> * generator functions become modified closure functions: >> - METH_O signature instead of METH_NOARGS to receive the send(x) value >> directly (note that gen.__next__() is defined as gen.send(None) and >> gen.throw(exc) could be implemented as gen.send(NULL)) >> - closures additionally contain function temps (I'm thinking of a >> union of structs, i.e. one struct for each set of temps that existed >> during the code generation for a yield node, but I guess storing >> all temps is just fine to start with - won't impact performance, >> just memory) >> - closures have an additional C field to store the execution state >> (void* to a function label, initially NULL) >> - "sendval = (yield [expr])" emits the following code: >> - store away all current temp values in the closure >> - set "closure._resume_label" to the resume label (see below, uses >> the C operator "&&") >> - return the expression result (or None) - return immediately >> without cleanup (the temp that holds the expression result must be >> unmanaged to prevent DECREF()-ing on resume; INCREF()-ing the >> return value will keep it alive for too long) >> - here goes the resume label ("__Lxyz_resume_from_yield:") >> - reset all saved temp values from the closure >> - if an exception is to be raised (gen.throw() was called, which has >> already set the exception externally), use normal exception path >> - set the result temp of the yield node to the send value argument >> that was passed (INCREF or not, as for parameters) >> * generator C function basically implements gen.send(x) >> - receives both the closure and the current send value as parameters >> - if "closure._resume_label" is not NULL, jump to the label; >> otherwise, check that 'x' is None (raise an exception if not) and >> execute the function body normally >> >> So the main work that's left to be done in 4) will be the closure extension >> to include the temps and the yield/resume implementation. >> >> Here's the (trivial) generic generator type: >> >> cdef class generator: >> cdef object _closure >> cdef meth_o_func* _run >> cdef object __weakref__ >> >> def __iter__(self): >> return self >> >> def __next__(self): >> return self._run(self._closure, None) >> >> def send(self, value): >> return self._run(self._closure, value) >> >> def throw(self, type, value=None, traceback=None): >> EXC_RESET(type, value, traceback) >> return self._run(self._closure, NULL) >> >> def close(self): >> try: >> EXC_RESET(GeneratorExit, NULL, NULL) >> self._run(self._closure, NULL) >> except (GeneratorExit, StopIteration): >> pass >> else: >> raise RuntimeError('generator ignored GeneratorExit') >> >> I wonder if there is a way to make it inherit from CPython's GeneratorType. >> That would enhance the interoperability, but it would also mean that we add >> some unnecessary instance size overhead and that we have to prevent that >> base-type from doing anything, including initialisation and final cleanup. >> >> The separation in 1a) has also been requested by Lisandro (and likely >> others) a while ago to make the function setup code more readable. >> Currently, the argument unpacking code takes so much space that it's easy >> to get lost when trying to read the generated function code, especially in >> short functions. >> >> The refactoring for 1) actually conflicts a bit with cpdef functions, which >> do the exact opposite: they create a DefNode for an existing C function. I >> wonder if it makes sense to swap that while we're at it. That would reduce >> some redundancy. >> >> Ok, this is a rather lengthy e-mail that's a bit akin to a spec already. >> Does this make sense to everybody? Any objections or ideas? Anyone happy to >> give a hand? :) >> >> Stefan >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From sturla at molden.no Fri Nov 13 05:11:05 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 13 Nov 2009 05:11:05 +0100 Subject: [Cython] buffer enhancements in 0.12 or later? In-Reply-To: References: <4AFB13AC.2050704@molden.no> Message-ID: <4AFCDC59.3080307@molden.no> Robert Bradshaw skrev: > Yeah, at this point we shouldn't be introducing anything new into > 0.12, but hopefully we won't take as long with 0.12.1. > > Ok, I will prepare patches for 0.12 once it is released and submit those for review. Sturla From robertwb at math.washington.edu Fri Nov 13 07:47:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Nov 2009 22:47:07 -0800 Subject: [Cython] Implementing generators In-Reply-To: <4AFCCDB6.3090209@behnel.de> References: <4AFCCDB6.3090209@behnel.de> Message-ID: On Nov 12, 2009, at 7:08 PM, Stefan Behnel wrote: > Hi all, > > warning: long e-mail ahead, don't read in a hurry! > > I gave generator functions a couple of thoughts. Implementing them > actually > sounds simpler than it is, not because of the state keeping, but > because of > the refactoring (point 1 below) that I would like to see done before > going > there. > > Here's what I think should be done: > > 1) refactor def functions into a Python wrapper and a static C > function > * Python wrapper does all argument unpacking, return value packing > and > the final exception propagation > * C function contains the complete body of the original function and > returns the return value directly That could clean things up a lot--it would be nice if this refactoring cleaned up some of the redundancies and inconsistencies between CFuncDefNode and DefNode. I'm not sure exactly what you mean by "final exception propagation" or "return value packing" though, as we still want cdef functions to propagate the exception. This could make cpdef functions more natural as well. I think the extra c function call overhead should be negligible, but we should be sure. Perhaps I'm prematurely optimizing, but I hate the idea of potentially introducing regressions. > a) non-closure functions: > - C function has signature as written in the code > - Python wrapper calls C function to execute the body Sure. > b) closure functions: > - C function has METH_NOARGS signature > - Python wrapper creates closure and fills in arguments > - Python wrapper calls C function with closure as 'self' I'm not sure I'm following you here--the closure is created when the function is created, not when it's called, and the function needs to take its arguments later. We could separate things out as in case (a), letting self be the actual closure argument, and binding it with the bound method type as is done now. I'm probably miss-understanding you, given def add_n(n): def f(x): return x+n return f which C function has METH_NOARGS? Both add_n and f could be handled as in (a), where f carries around a "self" that is the scope inherited from add_n. On that note, I certainly want to support cdef add_n(n): def f(x): return x+n return f and perhaps, if we can figure out a clean way to do it, cdef ??? add_n(n): cdef int f(int x): return x+n return f though this last case is not as important. > 2) support writing utility code in Cython (does this work already?) > * likely just compile TreeFragments inside of the utility_scope? > (does the utility_scope actually have a unique mangling prefix > or will it interfere with a user provided "utility" module?) > > 3) implement a generic 'generator' type in Cython code (see code > below) > * methods: __iter__, __next__, send, throw, close (as in PEP 342, > see > http://www.python.org/dev/peps/pep-0342/ ) > * fields: closure, exception, __weakref__, C function pointer I assume the motivation is that this would be easier than just generating a class with these methods every time? There's overhead to calling virtual methods, and it seems odd to direct throw() and __next__/send() into the same method, only to have an if statement to separate them in its body. > 4) implement generators as extension to 1b) > * Python wrapper works mostly as in 1b), but > - does not call the C function > - creates and returns a generator instance instead and fills in > the > created closure and the pointer to the C function part of the > generator function I was imagining def my_xrange(n): i = 0 while i < n: yield i i += 1 would get transformed into def my_xrange(n): return __Pyx_my_xrange_generator_class(n) > * generator functions become modified closure functions: > - METH_O signature instead of METH_NOARGS to receive the send(x) > value > directly (note that gen.__next__() is defined as > gen.send(None) and > gen.throw(exc) could be implemented as gen.send(NULL)) > - closures additionally contain function temps (I'm thinking of a > union of structs, i.e. one struct for each set of temps that > existed > during the code generation for a yield node, but I guess storing > all temps is just fine to start with - won't impact performance, > just memory) I don't think we should put temps in all closure functions, but for generators we certainly need to. > - closures have an additional C field to store the execution state > (void* to a function label, initially NULL) Ah, function lablels, good idea. It might make sense to initialize it to a label at the entry point of the function, so you can always start with a goto *self->exec_state, but if that label is hard to get at then setting (and checking) for NULL should work just fine. > - "sendval = (yield [expr])" emits the following code: > - store away all current temp values in the closure > - set "closure._resume_label" to the resume label (see below, > uses > the C operator "&&") > - return the expression result (or None) - return immediately > without cleanup (the temp that holds the expression result > must be > unmanaged to prevent DECREF()-ing on resume; INCREF()-ing the > return value will keep it alive for too long) > - here goes the resume label ("__Lxyz_resume_from_yield:") > - reset all saved temp values from the closure > - if an exception is to be raised (gen.throw() was called, > which has > already set the exception externally), use normal exception > path Rather than inserting this after every label, I think throw() should be a completely separate function. > - set the result temp of the yield node to the send value > argument > that was passed (INCREF or not, as for parameters) > * generator C function basically implements gen.send(x) > - receives both the closure and the current send value as > parameters > - if "closure._resume_label" is not NULL, jump to the label; > otherwise, check that 'x' is None (raise an exception if not) > and > execute the function body normally > > So the main work that's left to be done in 4) will be the closure > extension > to include the temps and the yield/resume implementation. > > Here's the (trivial) generic generator type: > > cdef class generator: > cdef object _closure > cdef meth_o_func* _run > cdef object __weakref__ > > def __iter__(self): > return self > > def __next__(self): > return self._run(self._closure, None) > > def send(self, value): > return self._run(self._closure, value) > > def throw(self, type, value=None, traceback=None): > EXC_RESET(type, value, traceback) > return self._run(self._closure, NULL) > > def close(self): > try: > EXC_RESET(GeneratorExit, NULL, NULL) > self._run(self._closure, NULL) > except (GeneratorExit, StopIteration): > pass > else: > raise RuntimeError('generator ignored GeneratorExit') We also need a __dealloc__ method to clean up any stored temps, etc. in case this generator goes out of scope before it is used up. > I wonder if there is a way to make it inherit from CPython's > GeneratorType. > That would enhance the interoperability, but it would also mean that > we add > some unnecessary instance size overhead and that we have to prevent > that > base-type from doing anything, including initialisation and final > cleanup. That could be nice, but it could have unforeseen and nasty consequences, especially if the generator type evolves as well. > The separation in 1a) has also been requested by Lisandro (and likely > others) a while ago to make the function setup code more readable. > Currently, the argument unpacking code takes so much space that it's > easy > to get lost when trying to read the generated function code, > especially in > short functions. > > The refactoring for 1) actually conflicts a bit with cpdef > functions, which > do the exact opposite: they create a DefNode for an existing C > function. I > wonder if it makes sense to swap that while we're at it. That would > reduce > some redundancy. Yep, makes sense to me. > Ok, this is a rather lengthy e-mail that's a bit akin to a spec > already. > Does this make sense to everybody? Any objections or ideas? Anyone > happy to > give a hand? :) I'd love to help implement this with you, but I've got a thesis to write... - Robert From robertwb at math.washington.edu Fri Nov 13 08:37:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Nov 2009 23:37:21 -0800 Subject: [Cython] Cython 0.12 release candidate Message-ID: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> It looks like Cython 0.12 is about ready for release. I've posted http://cython.org/release/Cython-0.12.rc0.tar.gz Sage builds fine and passes all doctests, please test away. The only outstanding issue that must be resolved is the fork issue when running the tests--if we do not come up with a solution I will simply disable it for the release. Until then, if the tests seem to hang, try running them with the --no-fork option. We still haven't resolved the MACOSX_DEPLOYMENT_TARGET testing issue, but no solution is in sight and I don't consider that a blocker. If all goes well, I do not see any other code going in at this point. There's a start at the release notes at http://wiki.cython.org/ReleaseNotes-0.12 - Robert From stefan_ml at behnel.de Fri Nov 13 12:05:16 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Nov 2009 12:05:16 +0100 Subject: [Cython] Implementing generators In-Reply-To: References: <4AFCCDB6.3090209@behnel.de> Message-ID: <4AFD3D6C.70908@behnel.de> Hi Robert, thanks for the feedback, I'll use it to write up a CEP for this. It looks like there's still room for discussion, and this isn't a trivial/obvious thing to implement, so it's worth some planning. Robert Bradshaw, 13.11.2009 07:47: > On Nov 12, 2009, at 7:08 PM, Stefan Behnel wrote: > >> Hi all, >> >> warning: long e-mail ahead, don't read in a hurry! >> >> I gave generator functions a couple of thoughts. Implementing them >> actually >> sounds simpler than it is, not because of the state keeping, but >> because of >> the refactoring (point 1 below) that I would like to see done before >> going >> there. >> >> Here's what I think should be done: >> >> 1) refactor def functions into a Python wrapper and a static C >> function >> * Python wrapper does all argument unpacking, return value packing >> and >> the final exception propagation >> * C function contains the complete body of the original function and >> returns the return value directly > > That could clean things up a lot--it would be nice if this refactoring > cleaned up some of the redundancies and inconsistencies between > CFuncDefNode and DefNode. Hopefully, yes. > I'm not sure exactly what you mean by "final > exception propagation" or "return value packing" though, as we still > want cdef functions to propagate the exception. Right, I guess I got confused here. The return type of a def function is always 'object', so that will be inherited by the underlying C function. > This could make cpdef > functions more natural as well. I think the extra c function call > overhead should be negligible, but we should be sure. Perhaps I'm > prematurely optimizing, but I hate the idea of potentially introducing > regressions. Given that the only case where this matters is when the function is called from Python code, including all the Python call overhead, I honestly doubt that the overhead of a single predictable C function call is even measurable. In the case of very short functions (where the call overhead starts becoming interesting), the C compiler might even end up inlining the C function, so you'd loose the overhead completely. >> b) closure functions: >> - C function has METH_NOARGS signature I actually didn't quite mean METH_NOARGS but just a single object in the signature (maybe even typed as the specific closure type). >> - Python wrapper creates closure and fills in arguments >> - Python wrapper calls C function with closure as 'self' > > I'm not sure I'm following you here--the closure is created when the > function is created, not when it's called, and the function needs to > take its arguments later. True. Sorry for the noise. So we actually need to pass the closure *and* the parameters. No real change here compared to the current workings. > On that note, I certainly want to support > > cdef add_n(n): > def f(x): > return x+n > return f That shouldn't be hard to do, as only f needs a closure here. > and perhaps, if we can figure out a clean way to do it, > > cdef ??? add_n(n): > cdef int f(int x): > return x+n > return f > > though this last case is not as important. It's certainly less important. Although it shouldn't be impossible to do either. "f" could become a C method in the closure object. But the return value would be less obvious, yes. Maybe something similar to the buffer protocol would work here, which transparently unpacks an object into a C value (a function pointer in this case). But the result would have to be a Python object in any case, so maybe we could let it implement __call__, and then provide a builtin type declaration for the value that would allow callers to bypass __call__ and instead call the underlying C method directly. But that's certainly a future add-on, nothing to care about now. >> 2) support writing utility code in Cython (does this work already?) >> * likely just compile TreeFragments inside of the utility_scope? >> (does the utility_scope actually have a unique mangling prefix >> or will it interfere with a user provided "utility" module?) >> >> 3) implement a generic 'generator' type in Cython code (see code >> below) >> * methods: __iter__, __next__, send, throw, close (as in PEP 342, >> see >> http://www.python.org/dev/peps/pep-0342/ ) >> * fields: closure, exception, __weakref__, C function pointer > > I assume the motivation is that this would be easier than just > generating a class with these methods every time? There's overhead to > calling virtual methods, and it seems odd to direct throw() and > __next__/send() into the same method, only to have an if statement to > separate them in its body. No, the if statement must be generated for each yield, so that the exception originates from the correct source line and takes the correct exception handling path. Nothing keeps you from putting a yield into a try-except or try-finally block, for example. So there needs to be some support in the yield node, which basically raises the exception. I just set the exception values outside of the generator function to avoid having to pass them into the function as arguments. >> 4) implement generators as extension to 1b) >> * Python wrapper works mostly as in 1b), but >> - does not call the C function >> - creates and returns a generator instance instead and fills in >> the >> created closure and the pointer to the C function part of the >> generator function > > I was imagining > > def my_xrange(n): > i = 0 > while i < n: > yield i > i += 1 > > would get transformed into > > def my_xrange(n): > return __Pyx_my_xrange_generator_class(n) Yes, I got confused again. So the original generator function itself will just get replaced by a dumb function that lives completely without closures. >> * generator functions become modified closure functions: >> - METH_O signature instead of METH_NOARGS to receive the send(x) >> value >> directly (note that gen.__next__() is defined as >> gen.send(None) and >> gen.throw(exc) could be implemented as gen.send(NULL)) >> - closures additionally contain function temps (I'm thinking of a >> union of structs, i.e. one struct for each set of temps that >> existed >> during the code generation for a yield node, but I guess storing >> all temps is just fine to start with - won't impact performance, >> just memory) > > I don't think we should put temps in all closure functions, but for > generators we certainly need to. That's what I meant. A "union of structs" approach would allow us to safe memory even for longer functions with many temps, as the space needed would match exactly the maximum number of temps that are in use during a yield. >> - closures have an additional C field to store the execution state >> (void* to a function label, initially NULL) > > Ah, function lablels, good idea. It might make sense to initialize it > to a label at the entry point of the function, so you can always start > with a goto *self->exec_state, but if that label is hard to get at > then setting (and checking) for NULL should work just fine. I think it'll be hard to get. Labels only exist inside of functions, as do static local variables, so you'd have to call the function in order to get to the label pointer. >> - "sendval = (yield [expr])" emits the following code: >> - store away all current temp values in the closure >> - set "closure._resume_label" to the resume label (see below, >> uses >> the C operator "&&") >> - return the expression result (or None) - return immediately >> without cleanup (the temp that holds the expression result >> must be >> unmanaged to prevent DECREF()-ing on resume; INCREF()-ing the >> return value will keep it alive for too long) >> - here goes the resume label ("__Lxyz_resume_from_yield:") >> - reset all saved temp values from the closure >> - if an exception is to be raised (gen.throw() was called, >> which has >> already set the exception externally), use normal exception >> path > > Rather than inserting this after every label, I think throw() should > be a completely separate function. It can't be because the exception must follows the normal exception flow inside the generator function. >> - set the result temp of the yield node to the send value >> argument >> that was passed (INCREF or not, as for parameters) >> * generator C function basically implements gen.send(x) >> - receives both the closure and the current send value as >> parameters >> - if "closure._resume_label" is not NULL, jump to the label; >> otherwise, check that 'x' is None (raise an exception if not) >> and >> execute the function body normally >> >> So the main work that's left to be done in 4) will be the closure >> extension >> to include the temps and the yield/resume implementation. >> >> Here's the (trivial) generic generator type: >> >> cdef class generator: >> cdef object _closure >> cdef meth_o_func* _run >> cdef object __weakref__ >> >> def __iter__(self): >> return self >> >> def __next__(self): >> return self._run(self._closure, None) >> >> def send(self, value): >> return self._run(self._closure, value) >> >> def throw(self, type, value=None, traceback=None): >> EXC_RESET(type, value, traceback) >> return self._run(self._closure, NULL) >> >> def close(self): >> try: >> EXC_RESET(GeneratorExit, NULL, NULL) >> self._run(self._closure, NULL) >> except (GeneratorExit, StopIteration): >> pass >> else: >> raise RuntimeError('generator ignored GeneratorExit') > > We also need a __dealloc__ method to clean up any stored temps, etc. > in case this generator goes out of scope before it is used up. True. In that case, we'd also have to know which temps were actually used at some point. Maybe the yield node could automatically emit corresponding cleanup code into a separate code writer that builds the __dealloc__ method. >> I wonder if there is a way to make it inherit from CPython's >> GeneratorType. >> That would enhance the interoperability, but it would also mean that >> we add >> some unnecessary instance size overhead and that we have to prevent >> that >> base-type from doing anything, including initialisation and final >> cleanup. > > That could be nice, but it could have unforeseen and nasty > consequences, especially if the generator type evolves as well. It certainly would be nasty. The CPython generator type keeps a frame alive, for example, and checks for that on type construction. So we'd either have to create a fake frame object to initialise the generator (and kill it afterwards, so that it won't get used), or find a way to disable the type initialisation completely. Rather scary either way. Stefan From stefan_ml at behnel.de Fri Nov 13 12:11:55 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Nov 2009 12:11:55 +0100 Subject: [Cython] Implementing generators In-Reply-To: <4AFCDAE1.1060404@student.matnat.uio.no> References: <4AFCCDB6.3090209@behnel.de> <4AFCD81B.1020608@student.matnat.uio.no> <4AFCDAE1.1060404@student.matnat.uio.no> Message-ID: <4AFD3EFB.7020507@behnel.de> Dag Sverre Seljebotn, 13.11.2009 05:04: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> 2) support writing utility code in Cython (does this work already?) >>> * likely just compile TreeFragments inside of the utility_scope? >>> (does the utility_scope actually have a unique mangling prefix >>> or will it interfere with a user provided "utility" module?) >> >> I am in a hurry and am going to be for another two weeks, so I'll just >> answer this: This is partly implemented in the kurt-gsoc branch. I'm >> hoping to do some work on getting that merged in December. > > partly = It does work, but perhaps not exactly as you'd want it > (string-based rather than TreeFragment, though that is easily changed). I don't currently think we need to do any replacements if we use a dedicated builtin generator class, so a string based approach should work just fine. Stefan From ndbecker2 at gmail.com Fri Nov 13 12:41:33 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 13 Nov 2009 06:41:33 -0500 Subject: [Cython] Cython 0.12 release candidate References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> Message-ID: Robert Bradshaw wrote: > It looks like Cython 0.12 is about ready for release. I've posted > > http://cython.org/release/Cython-0.12.rc0.tar.gz > > Sage builds fine and passes all doctests, please test away. > > The only outstanding issue that must be resolved is the fork issue > when running the tests--if we do not come up with a solution I will > simply disable it for the release. Until then, if the tests seem to > hang, try running them with the --no-fork option. We still haven't > resolved the MACOSX_DEPLOYMENT_TARGET testing issue, but no solution > is in sight and I don't consider that a blocker. If all goes well, I > do not see any other code going in at this point. > > There's a start at the release notes at > http://wiki.cython.org/ReleaseNotes-0.12 > > - Robert All tests OK on fedora F11 x86_64 From robertwb at math.washington.edu Fri Nov 13 19:30:44 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Nov 2009 10:30:44 -0800 Subject: [Cython] Implementing generators In-Reply-To: <4AFD3D6C.70908@behnel.de> References: <4AFCCDB6.3090209@behnel.de> <4AFD3D6C.70908@behnel.de> Message-ID: <01F78C57-B3D2-4A21-A070-EFE2A16F016D@math.washington.edu> On Nov 13, 2009, at 3:05 AM, Stefan Behnel wrote: > Hi Robert, > > thanks for the feedback, I'll use it to write up a CEP for this. Great. > It looks like there's still room for discussion, and this isn't a > trivial/obvious > thing to implement, so it's worth some planning. Yeah. But like you said, it's not too far off either. >> I assume the motivation is that this would be easier than just >> generating a class with these methods every time? There's overhead to >> calling virtual methods, and it seems odd to direct throw() and >> __next__/send() into the same method, only to have an if statement to >> separate them in its body. > > No, the if statement must be generated for each yield, so that the > exception originates from the correct source line and takes the > correct > exception handling path. Nothing keeps you from putting a yield into a > try-except or try-finally block, for example. So there needs to be > some > support in the yield node, which basically raises the exception. I > just set > the exception values outside of the generator function to avoid > having to > pass them into the function as arguments. Oh, yes, you're right here. I wasn't fully thinking through the semantics of the throws method. > True. In that case, we'd also have to know which temps were actually > used > at some point. Maybe the yield node could automatically emit > corresponding > cleanup code into a separate code writer that builds the __dealloc__ > method. If our temps a flat struct instead of a union of structs (and I don't think that'd be too wasteful as temps get freed and re-used), we should be safe just XDecrefing them whole set of them, just like the error exit cleanup in a normal function. - Robert From viyer at facebook.com Fri Nov 13 20:52:05 2009 From: viyer at facebook.com (Venky Iyer) Date: Fri, 13 Nov 2009 19:52:05 +0000 (UTC) Subject: [Cython] Implementing generators References: <4AFCCDB6.3090209@behnel.de> Message-ID: Stefan Behnel writes: > > Hi all, > > warning: long e-mail ahead, don't read in a hurry! > > I gave generator functions a couple of thoughts. Implementing them actually > sounds simpler than it is, not because of the state keeping, but because of > the refactoring (point 1 below) that I would like to see done before going > there. > At the risk of adding noise, I'd like to say that this is one of the most important features for us to use Cython more deeply in our stuff- we do a lot of data processing on very large streams of data and use generators extensively to keep our code modular; it would be awesome to be able to transparently replace the most used generators with Cython-ized versions. I'm glad to see this getting attention- carry on, and thank you for the work you've put into Cython so far! From jonovik at gmail.com Fri Nov 13 22:51:50 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Fri, 13 Nov 2009 21:51:50 +0000 (UTC) Subject: [Cython] Cython 0.12 release candidate References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> Message-ID: Robert Bradshaw writes: > It looks like Cython 0.12 is about ready for release. I've posted > > http://cython.org/release/Cython-0.12.rc0.tar.gz > > Sage builds fine and passes all doctests, please test away. Windows XP SP 3, Enthought Python Distribution 5.1.1 (Python 2.5.4). "python setup.py install" works without error, and I can "python setup.py build_ext --inplace" in the Demos directory, then "import primes" from Python. However, "python runtests.py" fails, telling me that Visual Studio 2003 is not found. It suggests passing the "-c mingw32" option to setup.py, whereas runtests.py doesn't accept this option. Also, the fact that the primes example compiles suggests that Cython should be able to use mingw32 based on my settings in distutils.cfg. Any help in getting the tests to run would be highly appreciated. Best regards, Jon Olav C:\temp\Cython-0.12.rc0>python runtests.py Running tests against Cython 0.12.rc0 Python 2.5.4 |EPD 5.1.1| (r254:67916, Aug 11 2009, 21:11:08) [MSC v.1310 32 bit (Intel)] Traceback (most recent call last): File "runtests.py", line 781, in pyxbuild_dir=os.path.join(WORKDIR, "support")) File "C:\temp\Cython-0.12.rc0\pyximport\pyxbuild.py", line 85, in pyx_to_dll dist.run_commands() File "C:\Python25\Lib\distutils\dist.py", line 974, in run_commands self.run_command(cmd) File "C:\Python25\Lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Python25\lib\distutils\command\build_ext.py", line 264, in run force=self.force) File "C:\Python25\lib\distutils\ccompiler.py", line 1175, in new_compiler return klass (None, dry_run, force) File "C:\Python25\lib\distutils\msvccompiler.py", line 243, in __init__ self.__macros = MacroExpander(self.__version) File "C:\Python25\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "C:\Python25\lib\distutils\msvccompiler.py", line 137, in load_macros you can try compiling with MingW32, by passing "-c mingw32" to setup.py.""") distutils.errors.DistutilsPlatformError: Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py. From robertwb at math.washington.edu Sat Nov 14 00:26:04 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Nov 2009 15:26:04 -0800 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> Message-ID: On Nov 13, 2009, at 1:51 PM, Jon Olav Vik wrote: > Robert Bradshaw writes: > >> It looks like Cython 0.12 is about ready for release. I've posted >> >> http://cython.org/release/Cython-0.12.rc0.tar.gz >> >> Sage builds fine and passes all doctests, please test away. > > Windows XP SP 3, Enthought Python Distribution 5.1.1 (Python 2.5.4). > > "python setup.py install" works without error, and I can "python > setup.py > build_ext --inplace" in the Demos directory, then "import primes" > from Python. > However, "python runtests.py" fails, telling me that Visual Studio > 2003 is not > found. It suggests passing the "-c mingw32" option to setup.py, > whereas > runtests.py doesn't accept this option. Also, the fact that the > primes example > compiles suggests that Cython should be able to use mingw32 based on > my > settings in distutils.cfg. > > Any help in getting the tests to run would be highly appreciated. Given the traceback below, it's looking like it's having trouble compiling the refnanny with pyximport, so you might try to follow some threads on using pyximport with mingw32. However, as a workaround, you could try running the tests with python runtests.py --no-refnanny - Robert > > > C:\temp\Cython-0.12.rc0>python runtests.py > Running tests against Cython 0.12.rc0 > Python 2.5.4 |EPD 5.1.1| (r254:67916, Aug 11 2009, 21:11:08) [MSC v. > 1310 32 bit > (Intel)] > Traceback (most recent call last): > File "runtests.py", line 781, in > pyxbuild_dir=os.path.join(WORKDIR, "support")) > File "C:\temp\Cython-0.12.rc0\pyximport\pyxbuild.py", line 85, in > pyx_to_dll > dist.run_commands() > File "C:\Python25\Lib\distutils\dist.py", line 974, in run_commands > self.run_command(cmd) > File "C:\Python25\Lib\distutils\dist.py", line 994, in run_command > cmd_obj.run() > File "C:\Python25\lib\distutils\command\build_ext.py", line 264, in > run > force=self.force) > File "C:\Python25\lib\distutils\ccompiler.py", line 1175, in > new_compiler > return klass (None, dry_run, force) > File "C:\Python25\lib\distutils\msvccompiler.py", line 243, in > __init__ > self.__macros = MacroExpander(self.__version) > File "C:\Python25\lib\distutils\msvccompiler.py", line 112, in > __init__ > self.load_macros(version) > File "C:\Python25\lib\distutils\msvccompiler.py", line 137, in > load_macros > you can try compiling with MingW32, by passing "-c mingw32" to > setup.py.""") > distutils.errors.DistutilsPlatformError: Python was built with > Visual Studio > 2003; > extensions must be built with a compiler than can generate > compatible binaries. > Visual Studio 2003 was not found on this system. If you have Cygwin > installed, > you can try compiling with MingW32, by passing "-c mingw32" to > setup.py. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From josef.pktd at gmail.com Sat Nov 14 01:18:33 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 13 Nov 2009 19:18:33 -0500 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> Message-ID: <1cd32cbb0911131618s3839ae2eke316593b0ba8c591@mail.gmail.com> On Fri, Nov 13, 2009 at 6:26 PM, Robert Bradshaw wrote: > On Nov 13, 2009, at 1:51 PM, Jon Olav Vik wrote: > >> Robert Bradshaw writes: >> >>> It looks like Cython 0.12 is about ready for release. I've posted >>> >>> http://cython.org/release/Cython-0.12.rc0.tar.gz >>> >>> Sage builds fine and passes all doctests, please test away. >> >> Windows XP SP 3, Enthought Python Distribution 5.1.1 (Python 2.5.4). >> >> "python setup.py install" works without error, and I can "python >> setup.py >> build_ext --inplace" in the Demos directory, then "import primes" >> from Python. >> However, "python runtests.py" fails, telling me that Visual Studio >> 2003 is not >> found. It suggests passing the "-c mingw32" option to setup.py, >> whereas >> runtests.py doesn't accept this option. Also, the fact that the >> primes example >> compiles suggests that Cython should be able to use mingw32 based on >> my >> settings in distutils.cfg. >> >> Any help in getting the tests to run would be highly appreciated. > > Given the traceback below, it's looking like it's having trouble > compiling the refnanny with pyximport, so you might try to follow some > threads on using pyximport with mingw32. However, as a workaround, you > could try running the tests with > > python runtests.py --no-refnanny > > - Robert > >> >> >> C:\temp\Cython-0.12.rc0>python runtests.py >> Running tests against Cython 0.12.rc0 >> Python 2.5.4 |EPD 5.1.1| (r254:67916, Aug 11 2009, 21:11:08) [MSC v. >> 1310 32 bit >> (Intel)] >> Traceback (most recent call last): >> ?File "runtests.py", line 781, in >> ? ?pyxbuild_dir=os.path.join(WORKDIR, "support")) >> ?File "C:\temp\Cython-0.12.rc0\pyximport\pyxbuild.py", line 85, in >> pyx_to_dll >> ? ?dist.run_commands() >> ?File "C:\Python25\Lib\distutils\dist.py", line 974, in run_commands >> ? ?self.run_command(cmd) >> ?File "C:\Python25\Lib\distutils\dist.py", line 994, in run_command >> ? ?cmd_obj.run() >> ?File "C:\Python25\lib\distutils\command\build_ext.py", line 264, in >> run >> ? ?force=self.force) >> ?File "C:\Python25\lib\distutils\ccompiler.py", line 1175, in >> new_compiler >> ? ?return klass (None, dry_run, force) >> ?File "C:\Python25\lib\distutils\msvccompiler.py", line 243, in >> __init__ >> ? ?self.__macros = MacroExpander(self.__version) >> ?File "C:\Python25\lib\distutils\msvccompiler.py", line 112, in >> __init__ >> ? ?self.load_macros(version) >> ?File "C:\Python25\lib\distutils\msvccompiler.py", line 137, in >> load_macros >> ? ?you can try compiling with MingW32, by passing "-c mingw32" to >> setup.py.""") >> distutils.errors.DistutilsPlatformError: Python was built with >> Visual Studio >> 2003; >> extensions must be built with a compiler than can generate >> compatible binaries. >> Visual Studio 2003 was not found on this system. If you have Cygwin >> installed, >> you can try compiling with MingW32, by passing "-c mingw32" to >> setup.py. As explained in the previous thread "runtests results WindowXP MingW", I had to adjust runtests to pick up the config_files to use the compiler option: After adding the following to the runtests (copied from pyxbuild.py, I don't know why config_files are parsed twice), the tests run with MingW (3.4.5) as compiler. ######## added to runtests.py config_files = distutils_distro.find_config_files() try: config_files.remove('setup.cfg') except ValueError: pass distutils_distro.parse_config_files(config_files) cfgfiles = distutils_distro.find_config_files() try: cfgfiles.remove('setup.cfg') except ValueError: pass distutils_distro.parse_config_files(cfgfiles) ############# Josef >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From wing1127aishi at gmail.com Sat Nov 14 15:29:03 2009 From: wing1127aishi at gmail.com (Leon Sit) Date: Sat, 14 Nov 2009 09:29:03 -0500 Subject: [Cython] Copperhead with Cython Message-ID: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> Dear All: The Copperhead project at UCB is aimed to produce code that can be ran on CUDA architecture concurrently. Is current Cython design suitable to integrate such ideas/projecot in the future or by design, they are likely to be a separate project? Thanks a lot for your time. Reference: http://www.google.com/url?sa=t&source=web&ct=res&cd=1&ved=0CAcQFjAA&url=http%3A%2F%2Fparlang.pbworks.com%2Ff%2FCUDA%2B-%2BCopperhead.pptx&ei=8bv-Su3XLM_HlAebopGcCw&usg=AFQjCNGbUCnn1QlJ-vCHc2MneT7q7MPnnw Sincerely, Leon From stefan_ml at behnel.de Sat Nov 14 17:29:21 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 14 Nov 2009 17:29:21 +0100 Subject: [Cython] Copperhead with Cython In-Reply-To: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> References: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> Message-ID: <4AFEDAE1.6000602@behnel.de> Leon Sit, 14.11.2009 15:29: > Reference: http://www.google.com/url?sa=t&source=web&ct=res&cd=1&ved=0CAcQFjAA&url=http%3A%2F%2Fparlang.pbworks.com%2Ff%2FCUDA%2B-%2BCopperhead.pptx&ei=8bv-Su3XLM_HlAebopGcCw&usg=AFQjCNGbUCnn1QlJ-vCHc2MneT7q7MPnnw Here's Google's HTML version of the above file, in case someone actually wants to /read/ it: http://209.85.129.132/search?q=cache:qPIQOEdD84gJ:parlang.pbworks.com/f/CUDA%2B-%2BCopperhead.pptx Stefan From matthew.brett at gmail.com Sun Nov 15 02:09:51 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Nov 2009 17:09:51 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? Message-ID: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> Hi, I've been spending some time with Cython in Scipy: http://projects.scipy.org/scipy/browser/trunk/scipy/io/matlab In general, it's been very enjoyable - a warm thank you to y'all for making it so clear and so useful. Here, I just wanted to check that I had understood the idioms right to call the numpy C-API. I'm doing this for speed. Imagine I have a string 'data', length 'n' bytes, and a numpy dtype 'dt', and I wanted to make a 1d array. In python that is: arr = np.ndarray(shape=(n,), dtype=dt, buffer=data) Am I right, that in Cython, this would be a function like the following (that does work): # start of make_array.pyx from python cimport Py_INCREF cdef extern from "Python.h": ctypedef struct PyTypeObject: pass import numpy as np cimport numpy as cnp cdef extern from "numpy/arrayobject.h": PyTypeObject PyArray_Type object PyArray_NewFromDescr(PyTypeObject *subtype, cnp.dtype newdtype, int nd, cnp.npy_intp* dims, cnp.npy_intp* strides, void* data, int flags, object parent) cdef extern from "workaround.h": void PyArray_Set_BASE(cnp.ndarray arr, object obj) # NOTE: numpy MUST be initialized before any other code is executed. cnp.import_array() def make_1d_array(cnp.npy_intp size, object data, cnp.dtype dt): cdef char *ptr ptr = data Py_INCREF( dt) Py_INCREF( data) narr = PyArray_NewFromDescr(&PyArray_Type, dt, 1, &size, NULL, ptr, 0, NULL) PyArray_Set_BASE(narr, data) return narr # end of make_array.pyx where 'workaround.h' (thanks to Dag Sverre) [1]: #include #define PyArray_Set_BASE(arr, obj) PyArray_BASE(arr) = obj Is that correct? In particular, I don't think I fully understand the right idiom for using the numpy array type PyArray_Type. Did I understand correctly that I have to Py_INCREF both the dtype and the data? [2] [3] Many thanks for any help. Matthew [1] http://codespeak.net/pipermail/cython-dev/2009-June/005979.html [2] http://docs.scipy.org/doc/numpy/reference/c-api.array.html [3] http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI From sccolbert at gmail.com Sun Nov 15 02:13:27 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 15 Nov 2009 02:13:27 +0100 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> Message-ID: <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> AFAIK, you only have to INCREF the dtype. And only for numpy array creation routines that take a dtype object as an argument (they "steal" a dtype reference, which is why you have to incref it). On Sun, Nov 15, 2009 at 2:09 AM, Matthew Brett wrote: > Hi, > > I've been spending some time with Cython in Scipy: > http://projects.scipy.org/scipy/browser/trunk/scipy/io/matlab > > In general, it's been very enjoyable - a warm thank you to y'all for > making it so clear and so useful. > > Here, I just wanted to check that I had understood the idioms right to > call the numpy C-API. ?I'm doing this for speed. > > Imagine I have a string 'data', length 'n' bytes, and a numpy dtype > 'dt', and I wanted to make a 1d array. ?In python that is: > > arr = np.ndarray(shape=(n,), dtype=dt, buffer=data) > > Am I right, that in Cython, this would be a function like the > following (that does work): > > # start of make_array.pyx > from python cimport Py_INCREF > > cdef extern from "Python.h": > ? ?ctypedef struct PyTypeObject: > ? ? ? ?pass > > import numpy as np > cimport numpy as cnp > > cdef extern from "numpy/arrayobject.h": > ? ?PyTypeObject PyArray_Type > ? ?object PyArray_NewFromDescr(PyTypeObject *subtype, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?cnp.dtype newdtype, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?int nd, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?cnp.npy_intp* dims, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?cnp.npy_intp* strides, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?void* data, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?int flags, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?object parent) > > cdef extern from "workaround.h": > ? ?void PyArray_Set_BASE(cnp.ndarray arr, object obj) > > # NOTE: numpy MUST be initialized before any other code is executed. > cnp.import_array() > > > def make_1d_array(cnp.npy_intp size, object data, cnp.dtype dt): > ? ?cdef char *ptr > ? ?ptr = data > ? ?Py_INCREF( dt) > ? ?Py_INCREF( data) > ? ?narr = PyArray_NewFromDescr(&PyArray_Type, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? dt, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? &size, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? NULL, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ptr, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? NULL) > ? ?PyArray_Set_BASE(narr, data) > ? ?return narr > # end of make_array.pyx > > where 'workaround.h' (thanks to Dag Sverre) [1]: > > #include > #define PyArray_Set_BASE(arr, obj) PyArray_BASE(arr) = obj > > Is that correct? ?In particular, I don't think I fully understand the > right idiom for using the numpy array type PyArray_Type. ?Did I > understand correctly that I have to Py_INCREF both the dtype and the > data? [2] [3] > > Many thanks for any help. > > Matthew > > [1] http://codespeak.net/pipermail/cython-dev/2009-June/005979.html > [2] http://docs.scipy.org/doc/numpy/reference/c-api.array.html > [3] http://wiki.cython.org/tutorials/numpy#UsingtheNumpyCAPI > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From matthew.brett at gmail.com Sun Nov 15 02:31:19 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Nov 2009 17:31:19 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> Message-ID: <1e2af89e0911141731l6c51b9j59ba22e643d1d458@mail.gmail.com> Hi, On Sat, Nov 14, 2009 at 5:13 PM, Chris Colbert wrote: > AFAIK, you only have to INCREF the dtype. And only for numpy array > creation routines that take a dtype object as an argument (they > "steal" a dtype reference, which is why you have to incref it). In case it's useful, here's the scratchpad I'm using for testing: http://github.com/matthew-brett/arraymakers (see arraymaker1.pyx). The reason I think that 'data' needs to be INCREF'd is the Warning here: http://docs.scipy.org/doc/numpy/reference/c-api.array.html#from-scratch and the fact that I get segfaults if I don't do it. For example, commenting out the Py_INCREF(data) around line 34 of arraymaker1.pyx in the repository above, results in a segmentation fault in test_arraymaker2.py: mb312 at millroad:~/dev_trees/arraymakers$ nosetests test_arraymaker2.py test_arraymaker2.test_arraymaker2(False,) ... ok test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int16), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int16)) ... ok test_arraymaker2.test_arraymaker2(False,) ... ok test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int16), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int16)) ... ok test_arraymaker2.test_arraymaker2(False,) ... ok test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint32), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint32)) ... ok test_arraymaker2.test_arraymaker2(False,) ... ok test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint32), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint32)) ... ok test_arraymaker2.test_arraymaker2(False,) ... ok test_arraymaker2.test_arraymaker2(array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]), array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])) ... ok test_arraymaker2.test_arraymaker2(False,) ... ok Segmentation fault Best, Matthew From sccolbert at gmail.com Sun Nov 15 02:46:19 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 15 Nov 2009 02:46:19 +0100 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <1e2af89e0911141731l6c51b9j59ba22e643d1d458@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> <1e2af89e0911141731l6c51b9j59ba22e643d1d458@mail.gmail.com> Message-ID: <7f014ea60911141746q55b3fd3sf3d47a5f186969f@mail.gmail.com> Yeah, this is a pretty important statement: "If data is passed to PyArray_NewFromDescr or PyArray_New, this memory must not be deallocated until the new array is deleted." So if you dont keep a reference to "data" in the calling scope, then python will free that memory once you lose the reference, and you will segfault. If you hold on to the data for the duration of the call, you should be ok. But like suggested in the docs, if the data is a python object, just incref it. But dont forget to set base member pointer (PyObject *PyArray_BASE(PyObject* arr)) to point to the python data object, or else you leak a reference, and the memory will never get freed. Cheers, Chris On Sun, Nov 15, 2009 at 2:31 AM, Matthew Brett wrote: > Hi, > > On Sat, Nov 14, 2009 at 5:13 PM, Chris Colbert wrote: >> AFAIK, you only have to INCREF the dtype. And only for numpy array >> creation routines that take a dtype object as an argument (they >> "steal" a dtype reference, which is why you have to incref it). > > In case it's useful, here's the scratchpad I'm using for testing: > > http://github.com/matthew-brett/arraymakers > > (see arraymaker1.pyx). ?The reason I think that 'data' needs to be > INCREF'd is the Warning here: > > http://docs.scipy.org/doc/numpy/reference/c-api.array.html#from-scratch > > and the fact that I get segfaults if I don't do it. ?For example, > commenting out the Py_INCREF(data) around line 34 of arraymaker1.pyx > in the repository above, results in a segmentation fault in > test_arraymaker2.py: > > mb312 at millroad:~/dev_trees/arraymakers$ nosetests test_arraymaker2.py > test_arraymaker2.test_arraymaker2(False,) ... ok > test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, > 9], dtype=int16), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int16)) > ... ok > test_arraymaker2.test_arraymaker2(False,) ... ok > test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, > 9], dtype=int16), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int16)) > ... ok > test_arraymaker2.test_arraymaker2(False,) ... ok > test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, > 9], dtype=uint32), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], > dtype=uint32)) ... ok > test_arraymaker2.test_arraymaker2(False,) ... ok > test_arraymaker2.test_arraymaker2(array([0, 1, 2, 3, 4, 5, 6, 7, 8, > 9], dtype=uint32), array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], > dtype=uint32)) ... ok > test_arraymaker2.test_arraymaker2(False,) ... ok > test_arraymaker2.test_arraymaker2(array([ 0., ?1., ?2., ?3., ?4., ?5., > ?6., ?7., ?8., ?9.]), array([ 0., ?1., ?2., ?3., ?4., ?5., ?6., ?7., > 8., ?9.])) ... ok > test_arraymaker2.test_arraymaker2(False,) ... ok > Segmentation fault > > Best, > > Matthew > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From matthew.brett at gmail.com Sun Nov 15 02:55:55 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Nov 2009 17:55:55 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <7f014ea60911141746q55b3fd3sf3d47a5f186969f@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> <1e2af89e0911141731l6c51b9j59ba22e643d1d458@mail.gmail.com> <7f014ea60911141746q55b3fd3sf3d47a5f186969f@mail.gmail.com> Message-ID: <1e2af89e0911141755s564b2592n1a97eb0e6eb7d4e6@mail.gmail.com> Hi, > But dont forget to set base member pointer > (PyObject *PyArray_BASE(PyObject* arr)) to point to the python data > object, or else you leak a reference, and the memory will never get > freed. Right - for reference that's the PyArray_Set_BASE(narr, data) in the code I posted earlier (brought to you courtesy of Dag Sverre's workaround.h trick). Cheers, Matthew From sccolbert at gmail.com Sun Nov 15 03:04:26 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 15 Nov 2009 03:04:26 +0100 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <1e2af89e0911141755s564b2592n1a97eb0e6eb7d4e6@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> <1e2af89e0911141731l6c51b9j59ba22e643d1d458@mail.gmail.com> <7f014ea60911141746q55b3fd3sf3d47a5f186969f@mail.gmail.com> <1e2af89e0911141755s564b2592n1a97eb0e6eb7d4e6@mail.gmail.com> Message-ID: <7f014ea60911141804g19de8449h57895a68c6197ca8@mail.gmail.com> AH I see ok. I should have read more carefully (a little late here). When you commented out the INCREF on the datat, did you also comment out PyArray_Set_BASE? If not, that may have been causing the problem. If you set the base without INCREF'ing the data, then numpy will DECREF that data when python goes to gc the array. Now you've lost a reference, and if it went to zero, python then tries to gc an object that's still in scope. On Sun, Nov 15, 2009 at 2:55 AM, Matthew Brett wrote: > Hi, > >> But dont forget to set base member pointer >> (PyObject *PyArray_BASE(PyObject* arr)) to point to the python data >> object, or else you leak a reference, and the memory will never get >> freed. > > Right - for reference that's the > > PyArray_Set_BASE(narr, data) > > in the code I posted earlier (brought to you courtesy of Dag Sverre's > workaround.h trick). > > Cheers, > > Matthew > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From sccolbert at gmail.com Sun Nov 15 03:06:31 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 15 Nov 2009 03:06:31 +0100 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <7f014ea60911141804g19de8449h57895a68c6197ca8@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> <1e2af89e0911141731l6c51b9j59ba22e643d1d458@mail.gmail.com> <7f014ea60911141746q55b3fd3sf3d47a5f186969f@mail.gmail.com> <1e2af89e0911141755s564b2592n1a97eb0e6eb7d4e6@mail.gmail.com> <7f014ea60911141804g19de8449h57895a68c6197ca8@mail.gmail.com> Message-ID: <7f014ea60911141806r1c3427del72159d0e7b793a6e@mail.gmail.com> I forgot a disclaimer: This is all to the best of my knowledge. I'm sure one of the awesome Cython devs will jump in and correct me ;) On Sun, Nov 15, 2009 at 3:04 AM, Chris Colbert wrote: > AH I see ok. I should have read more carefully (a little late here). > > When you commented out the INCREF on the datat, did you also comment > out PyArray_Set_BASE? If not, that may have been causing the problem. > If you set the base without INCREF'ing the data, then numpy will > DECREF that data when python goes to gc the array. Now you've lost a > reference, and if it went to zero, python then tries to gc an object > that's still in scope. > > > > On Sun, Nov 15, 2009 at 2:55 AM, Matthew Brett wrote: >> Hi, >> >>> But dont forget to set base member pointer >>> (PyObject *PyArray_BASE(PyObject* arr)) to point to the python data >>> object, or else you leak a reference, and the memory will never get >>> freed. >> >> Right - for reference that's the >> >> PyArray_Set_BASE(narr, data) >> >> in the code I posted earlier (brought to you courtesy of Dag Sverre's >> workaround.h trick). >> >> Cheers, >> >> Matthew >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > From matthew.brett at gmail.com Sun Nov 15 03:30:28 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Nov 2009 18:30:28 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <7f014ea60911141804g19de8449h57895a68c6197ca8@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <7f014ea60911141713oe638340u1bce31d21125c754@mail.gmail.com> <1e2af89e0911141731l6c51b9j59ba22e643d1d458@mail.gmail.com> <7f014ea60911141746q55b3fd3sf3d47a5f186969f@mail.gmail.com> <1e2af89e0911141755s564b2592n1a97eb0e6eb7d4e6@mail.gmail.com> <7f014ea60911141804g19de8449h57895a68c6197ca8@mail.gmail.com> Message-ID: <1e2af89e0911141830o562776f3k6b3cd163932dc158@mail.gmail.com> Hi, > When you commented out the INCREF on the datat, did you also comment > out PyArray_Set_BASE? If not, that may have been causing the problem. > If you set the base without INCREF'ing the data, then numpy will > DECREF that data when python goes to gc the array. Now you've lost a > reference, and if it went to zero, python then tries to gc an object > that's still in scope. Ah - yes - you are right. Commenting out the Base line removes the segfault. Just for my own sake I'm going to summarize, often repeating what you've said above: Given this line: arr = PyArray_NewFromDescr(&PyArray_Type, 1, dt, 1, &size, NULL, ptr, 0, NULL) The array now points into the memory at `ptr`. The memory at `ptr` needs to stay around for as long as I want to use the array. In my example, as you've pointed out, I passed in `data`, from which I got `ptr`, and `data` therefore exists in the surrounding scope, and will support `ptr`, at least in the surrounding scope. When I add: PyArray_Set_BASE(arr, data) this _borrows_ a reference to the object `data`, and stores it in the array. When the array is garbage collected, `data` is DECREF'd so that `data` in the surrounding scope has been busted by the DECREF on array garbage collection. However, if I had created `data` in the original 'make_1d_array` function, rather than passing it in, I'd need to do as I have done, which is INCREF the data so that it does not get garbage collected when I leave the function scope, and PyArray_Set_BASE so that the array knows to DECREF `data` and allow garbage collection when it goes out of scope. I don't know whether that is clear, but it was my best shot... Cheers, Matthew From jonovik at gmail.com Sun Nov 15 08:48:49 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Sun, 15 Nov 2009 07:48:49 +0000 (UTC) Subject: [Cython] Cython 0.12 release candidate References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <1cd32cbb0911131618s3839ae2eke316593b0ba8c591@mail.gmail.com> Message-ID: writes: > As explained in the previous thread "runtests results WindowXP MingW", > I had to adjust runtests to pick up the config_files to use the compiler option: > > After adding the following to the runtests (copied from pyxbuild.py, I > don't know why config_files are parsed twice), > the tests run with MingW (3.4.5) as compiler. > > ######## added to runtests.py > config_files = distutils_distro.find_config_files() > try: config_files.remove('setup.cfg') > except ValueError: pass > distutils_distro.parse_config_files(config_files) > > cfgfiles = distutils_distro.find_config_files() > try: cfgfiles.remove('setup.cfg') > except ValueError: pass > distutils_distro.parse_config_files(cfgfiles) > ############# Thanks. I did read that thread, but didn't understand that my issue was the same... 8-) I've added this and Robert's suggestion to http://wiki.cython.org/InstallingOnWindows Where in runtests.py do you add the above code? I tried adding it after line 28, distutils_distro = Distribution() C:\temp\Cython-0.12.rc0>python runtests_mingw32.py --no-refnanny Running tests against Cython 0.12.rc0 Python 2.5.4 |EPD 5.1.1| (r254:67916, Aug 11 2009, 21:11:08) [MSC v.1310 32 bit (Intel)] [...] Ran 2935 tests in 801.122s FAILED (errors=10) I've pasted the full output to http://pastebin.com/f2bac2cc4 Best regards, Jon Olav From stefan_ml at behnel.de Sun Nov 15 10:54:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Nov 2009 10:54:57 +0100 Subject: [Cython] Copperhead with Cython In-Reply-To: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> References: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> Message-ID: <4AFFCFF1.9060402@behnel.de> Leon Sit, 14.11.2009 15:29: > The Copperhead project at UCB is aimed to produce code that can be ran > on CUDA architecture concurrently. Is current Cython design suitable > to integrate such ideas/projecot in the future or by design, Sure, if "such ideas" is referring to code parallelisation. We've been thinking about integrating OpenMP a while ago, although there hasn't been any major push in that direction since. The current moves are more towards a SIMD data type, which would provide a more natural way to introduce parallelism into the language. http://wiki.cython.org/enhancements/simd But I actually like the idea of optimising map(). What about writing our own version of map() that spawns (OpenMP) threads when called on nogil functions? That's certainly less nice to use than a parallel for-loop, but it would certainly allow all sorts of fast code... Given that you need to constrain the number of parallel executions, it might be better not to overload Python's own map(), though, but to provide a "cython.map(f, *iterables, parallel=1)". > they are likely to be a separate project? Cython is targeting the C programming language, not the CUDA platform or OpenCL. So I don't see an interest in *not* being separate projects. Stefan From josef.pktd at gmail.com Sun Nov 15 13:18:40 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 15 Nov 2009 07:18:40 -0500 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <1cd32cbb0911131618s3839ae2eke316593b0ba8c591@mail.gmail.com> Message-ID: <1cd32cbb0911150418r588f8cefr6fd38a12bb1790ce@mail.gmail.com> On Sun, Nov 15, 2009 at 2:48 AM, Jon Olav Vik wrote: > ? writes: >> As explained in the previous thread "runtests results WindowXP MingW", >> I had to adjust runtests to pick up the config_files to use the compiler > option: >> >> After adding the following to the runtests (copied from pyxbuild.py, I >> don't know why config_files are parsed twice), >> the tests run with MingW (3.4.5) as compiler. >> >> ######## added to runtests.py >> config_files = distutils_distro.find_config_files() >> try: config_files.remove('setup.cfg') >> except ValueError: pass >> distutils_distro.parse_config_files(config_files) >> >> cfgfiles = distutils_distro.find_config_files() >> try: cfgfiles.remove('setup.cfg') >> except ValueError: pass >> distutils_distro.parse_config_files(cfgfiles) >> ############# > > Thanks. I did read that thread, but didn't understand that my issue was the > same... 8-) I've added this and Robert's suggestion to > http://wiki.cython.org/InstallingOnWindows > > Where in runtests.py do you add the above code? I tried adding it after line 28, > distutils_distro = Distribution() Yes, that's where I added it. This time I added the parsing only once. > > C:\temp\Cython-0.12.rc0>python runtests_mingw32.py --no-refnanny > Running tests against Cython 0.12.rc0 > Python 2.5.4 |EPD 5.1.1| (r254:67916, Aug 11 2009, 21:11:08) [MSC v.1310 32 bit > (Intel)] > [...] > Ran 2935 tests in 801.122s > FAILED (errors=10) > > I've pasted the full output to > http://pastebin.com/f2bac2cc4 > > Best regards, > Jon Olav I'm getting the same 10 errors without command line options, a few more warnings and printout because of the refnanny. (I ran the tests from the unzipped directory without installing.) C:\Josef\work-oth\Cython-0.12.rc0\Cython-0.12.rc0>runtests_mingw.py Running tests against Cython 0.12.rc0 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] warning: C:\Josef\work-oth\Cython-0.12.rc0\Cython-0.12.rc0\Cython\Runtime\refnan ny.pyx:90:5: Unraisable exception in function 'refnanny.GOTREF'. warning: C:\Josef\work-oth\Cython-0.12.rc0\Cython-0.12.rc0\Cython\Runtime\refnan ny.pyx:103:5: Unraisable exception in function 'refnanny.GIVEREF_and_report'. warning: C:\Josef\work-oth\Cython-0.12.rc0\Cython-0.12.rc0\Cython\Runtime\refnan ny.pyx:129:5: Unraisable exception in function 'refnanny.FinishContext'. bufaccess_noassignT444.c:1064: warning: '__Pyx_GetBufferAndValidate' defined but not used <...........> ---------------------------------------------------------------------- Ran 2935 tests in 1161.484s FAILED (errors=10) Josef > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Sun Nov 15 19:43:53 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 15 Nov 2009 15:43:53 -0300 Subject: [Cython] Copperhead with Cython In-Reply-To: <4AFFCFF1.9060402@behnel.de> References: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> <4AFFCFF1.9060402@behnel.de> Message-ID: On Sun, Nov 15, 2009 at 6:54 AM, Stefan Behnel wrote: > > Leon Sit, 14.11.2009 15:29: >> The Copperhead project at UCB is aimed to produce code that can be ran >> on CUDA architecture concurrently. Is current Cython design suitable >> to integrate such ideas/projecot in the future or by design, > > Sure, if "such ideas" is referring to code parallelisation. We've been > thinking about integrating OpenMP a while ago, although there hasn't been > any major push in that direction since. The current moves are more towards > a SIMD data type, which would provide a more natural way to introduce > parallelism into the language. > > http://wiki.cython.org/enhancements/simd > > But I actually like the idea of optimising map(). What about writing our > own version of map() that spawns (OpenMP) threads when called on nogil > functions? That's certainly less nice to use than a parallel for-loop, but > it would certainly allow all sorts of fast code... > > Given that you need to constrain the number of parallel executions, it > might be better not to overload Python's own map(), though, but to provide > a "cython.map(f, *iterables, parallel=1)". > > >> they are likely to be a separate project? > > Cython is targeting the C programming language, not the CUDA platform or > OpenCL. So I don't see an interest in *not* being separate projects. > Stefan, I think you are wrong... Targeting CUDA or OpenCL would be more or less conceptually equivalent to Fortran interoperability and Kurt's work on fwrap. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Mon Nov 16 00:34:07 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 15 Nov 2009 15:34:07 -0800 Subject: [Cython] small typo: missing " in the docs Message-ID: <85b5c3130911151534p709755d5nccebfb89304c4167@mail.gmail.com> Hi, http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils instead of: def say_hello_to(name): print(Hello %s!" % name) there should be: def say_hello_to(name): print("Hello %s!" % name) Ondrej P.S. I actually tried to compile it. :) From sturla at molden.no Mon Nov 16 05:36:46 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Nov 2009 05:36:46 +0100 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> Message-ID: <4B00D6DE.9080700@molden.no> Matthew Brett skrev: > Imagine I have a string 'data', length 'n' bytes, and a numpy dtype > 'dt', and I wanted to make a 1d array. In python that is: > > arr = np.ndarray(shape=(n,), dtype=dt, buffer=data) > > Am I right, that in Cython, this would be a function like the > following (that does work): > You are thinking too difficult. Forget about numpy's C API, you don't have to use it. In Cython this is: import numpy as np cimport numpy as np cdef np.ndarray[dt_t, ndim=1, mode="c"] arr arr = np.ndarray(shape=(n,), dtype=dt, buffer=data) The notice the type dt_t. For simple dtypes, there is mapping in numpy.pxd np.int32 -> np.int32_t np.float64 -> np.float64_t For recarrays, you simply ctypedef a struct similar to your dtype, and declare with that. Cython will now optimize arr[index] if index is a C integer, and arr.data gives you a char* pointer to arr's data. Sturla Molden From matthew.brett at gmail.com Mon Nov 16 07:46:34 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 15 Nov 2009 22:46:34 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <4B00D6DE.9080700@molden.no> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <4B00D6DE.9080700@molden.no> Message-ID: <1e2af89e0911152246g212f31adn94f9b96a2268d288@mail.gmail.com> Hi, > You are thinking too difficult. Forget about numpy's C API, you don't > have to use it. > > In Cython this is: > > import numpy as np > cimport numpy as np > > cdef np.ndarray[dt_t, ndim=1, mode="c"] arr > arr = np.ndarray(shape=(n,), dtype=dt, buffer=data) Do I not then need to know the array dtype at compile time? Cheers, Matthew From matthew.brett at gmail.com Mon Nov 16 08:33:30 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Sun, 15 Nov 2009 23:33:30 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <1e2af89e0911152246g212f31adn94f9b96a2268d288@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <4B00D6DE.9080700@molden.no> <1e2af89e0911152246g212f31adn94f9b96a2268d288@mail.gmail.com> Message-ID: <1e2af89e0911152333w5eeeb0c4i18addb65c7e9ad83@mail.gmail.com> >> cdef np.ndarray[dt_t, ndim=1, mode="c"] arr >> arr = np.ndarray(shape=(n,), dtype=dt, buffer=data) > > Do I not then need to know the array dtype at compile time? and - does this in fact optimize the array creation time (as opposed to subsequent array indexing)? I'm creating thousands of arrays of various dtypes known only at run time, Cheers, Matthew From stefan_ml at behnel.de Mon Nov 16 08:44:09 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Nov 2009 08:44:09 +0100 Subject: [Cython] Copperhead with Cython In-Reply-To: References: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> <4AFFCFF1.9060402@behnel.de> Message-ID: <4B0102C9.1070000@behnel.de> Lisandro Dalcin, 15.11.2009 19:43: > On Sun, Nov 15, 2009 at 6:54 AM, Stefan Behnel wrote: >> Leon Sit, 14.11.2009 15:29: >>> they are likely to be a separate project? >> Cython is targeting the C programming language, not the CUDA platform or >> OpenCL. So I don't see an interest in *not* being separate projects. > > Stefan, I think you are wrong... Targeting CUDA or OpenCL would be > more or less conceptually equivalent to Fortran interoperability and > Kurt's work on fwrap. Interfacing with CUDA is different from what Copperhead does, though. I really don't see Cython generate CUDA /code/. Stefan From sturla at molden.no Mon Nov 16 09:55:47 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Nov 2009 09:55:47 +0100 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <1e2af89e0911152333w5eeeb0c4i18addb65c7e9ad83@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <4B00D6DE.9080700@molden.no> <1e2af89e0911152246g212f31adn94f9b96a2268d288@mail.gmail.com> <1e2af89e0911152333w5eeeb0c4i18addb65c7e9ad83@mail.gmail.com> Message-ID: <4B011393.4020402@molden.no> Matthew Brett skrev: > and - does this in fact optimize the array creation time (as opposed > to subsequent array indexing)? I'm creating thousands of arrays of > various dtypes known only at run time, > No, but I doubt using the C API will be much faster. There are things you can do to speed this up, for example storing arguments and np.ndarray in local variables: cdef object _kwargs = {'shape': (n,), dtype=dt, buffer=data} cdef object _ndarray = np.ndarray arr = _ndarray(**_kwargs) Sturla From matthew.brett at gmail.com Mon Nov 16 18:27:07 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 16 Nov 2009 09:27:07 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <4B011393.4020402@molden.no> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <4B00D6DE.9080700@molden.no> <1e2af89e0911152246g212f31adn94f9b96a2268d288@mail.gmail.com> <1e2af89e0911152333w5eeeb0c4i18addb65c7e9ad83@mail.gmail.com> <4B011393.4020402@molden.no> Message-ID: <1e2af89e0911160927j444ee9d3if3622102e20dc06b@mail.gmail.com> Hi, >> and - does this in fact optimize the array creation time (as opposed >> to subsequent array indexing)? ?I'm creating thousands of arrays of >> various dtypes known only at run time, >> > No, but I doubt using the C API will be much faster. > > There are things you can do to speed this up, for example storing > arguments and np.ndarray in local variables: > > cdef object _kwargs = {'shape': ?(n,), ?dtype=dt, ?buffer=data} > cdef object _ndarray = np.ndarray > > arr = _ndarray(**_kwargs) That does seem to make a small difference, but the C-API is still about 4 times faster, in my hands (see http://github.com/matthew-brett/arraymakers): In [1]: run arraymakers/test_arraymaker2.py In [4]: import sturlabench In [5]: n = 50 In [6]: dt = np.dtype('f4') In [7]: data = np.arange(n, dtype=dt).tostring() In [8]: timeit a = py_make_1d_array(n, data, dt) 100000 loops, best of 3: 3.52 ?s per loop In [9]: timeit a = arraymaker2.make_1d_array(n, data, dt) 1000000 loops, best of 3: 875 ns per loop In [10]: timeit a = sturlabench.make_local_1d_array(n, data, dt) 100000 loops, best of 3: 3.39 ?s per loop> Best, Matthew From robertwb at math.washington.edu Mon Nov 16 20:32:42 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Nov 2009 11:32:42 -0800 Subject: [Cython] Copperhead with Cython In-Reply-To: <4B0102C9.1070000@behnel.de> References: <3af1998d0911140629q391936f1w6f70451c12b94667@mail.gmail.com> <4AFFCFF1.9060402@behnel.de> <4B0102C9.1070000@behnel.de> Message-ID: On Nov 15, 2009, at 11:44 PM, Stefan Behnel wrote: > Lisandro Dalcin, 15.11.2009 19:43: >> On Sun, Nov 15, 2009 at 6:54 AM, Stefan Behnel wrote: >>> Leon Sit, 14.11.2009 15:29: >>>> they are likely to be a separate project? >>> Cython is targeting the C programming language, not the CUDA >>> platform or >>> OpenCL. So I don't see an interest in *not* being separate projects. >> >> Stefan, I think you are wrong... Targeting CUDA or OpenCL would be >> more or less conceptually equivalent to Fortran interoperability and >> Kurt's work on fwrap. > > Interfacing with CUDA is different from what Copperhead does, > though. I > really don't see Cython generate CUDA /code/. I agree. At the moment Cython is structured around generating semantically faithful C code, with the expectation that the full Python/C API is available. Copperhead looks like a very interesting project, but sufficiently distinct that it probably makes sense to be a separate project. For example, I don't see a bunch of CUDA specific code making its way into the Cython codebase, not at this stage anyways. It would be interesting to see support for this kind of thing as a plugin (e.g. for the SIMD types already referred to, or for optionally optimizing the map function), though of course we don't even have the concept of a plugin right now. - Robert From sturla at molden.no Tue Nov 17 06:04:00 2009 From: sturla at molden.no (Sturla Molden) Date: Tue, 17 Nov 2009 06:04:00 +0100 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <1e2af89e0911160927j444ee9d3if3622102e20dc06b@mail.gmail.com> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <4B00D6DE.9080700@molden.no> <1e2af89e0911152246g212f31adn94f9b96a2268d288@mail.gmail.com> <1e2af89e0911152333w5eeeb0c4i18addb65c7e9ad83@mail.gmail.com> <4B011393.4020402@molden.no> <1e2af89e0911160927j444ee9d3if3622102e20dc06b@mail.gmail.com> Message-ID: <4B022EC0.7090404@molden.no> Matthew Brett skrev: >> >> cdef object _kwargs = {'shape': (n,), dtype=dt, buffer=data} >> cdef object _ndarray = np.ndarray >> >> arr = _ndarray(**_kwargs) > > That does seem to make a small difference, but the C-API is still > about 4 times faster, in my hands You didn't use it right. You have to manually hoist out the loop invariants. Obviously, declaring two local variables _kwargs and _ndarray doesn't help if you do it for each array you construct. Say you want to construct 1000 similar arrays: cdef object _kwargs = {'shape':(n,),'dtype':dt,'buffer':data} cdef object _ndarray = np.ndarray cdef list arrays = [None]*1000 cdef int i for i in range(1000): arrays[i] = _ndarray(**_kwargs) You should thus modify sturlabench.pyx like this: cdef dict _kwargs = {'shape':(50,),'dtype':cnp.float64,'buffer':None} cdef object _ndarray = cnp.ndarray cdef cnp.npy_intp _size = 50 cdef cnp.dtype _dt = cnp.float64 cdef object _data = None def make_local_1d_array(cnp.npy_intp size, object data, cnp.dtype dt): global _kwargs, _size, _dt, _data, _ndarray if (size != _size): _kwargs['shape'] = (size,) _size = size if (dt is not _dt): _kwargs['dtype'] = dt _dt = dt if (data is not _data): _kwargs['buffer'] = data _data = data return _ndarray(**_kwargs) I'd also like to point out this: Even with the non-optimized "Python" version (py_make_1d_array), creating 1000 arrays will only take 3.5 ms with your timings. "Premature optimization is the root of all evil in computer programming..." (C.A.R. Hoare, according to D. Knuth). Regards, Sturla Molden From matthew.brett at gmail.com Tue Nov 17 07:00:27 2009 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 16 Nov 2009 22:00:27 -0800 Subject: [Cython] Right way to create numpy arrays using C-API? In-Reply-To: <4B022EC0.7090404@molden.no> References: <1e2af89e0911141709jd82dbeet5a0f7db118474bfa@mail.gmail.com> <4B00D6DE.9080700@molden.no> <1e2af89e0911152246g212f31adn94f9b96a2268d288@mail.gmail.com> <1e2af89e0911152333w5eeeb0c4i18addb65c7e9ad83@mail.gmail.com> <4B011393.4020402@molden.no> <1e2af89e0911160927j444ee9d3if3622102e20dc06b@mail.gmail.com> <4B022EC0.7090404@molden.no> Message-ID: <1e2af89e0911162200n50027a46h7c5db3a7ad01cb06@mail.gmail.com> Hi, > You didn't use it right. You have to manually hoist out the loop > invariants. Obviously, declaring two local variables _kwargs and > _ndarray doesn't help if you do it for each array you construct. > > Say you want to construct 1000 similar arrays: Right - but here you are just caching the parameter dictionary in the hope that the size, dtype and/or data are the same from one iteration to the next. Again, it's a bit faster in the best possible case (exactly the same array over and over): In [2]: timeit a = sturlabench.make_local_1d_array(n, data, dt) 100000 loops, best of 3: 2.19 us per loop In [3]: timeit a = py_make_1d_array(n, data, dt) 100000 loops, best of 3: 2.95 us per loop but still quite a bit slower than the C version: In [4]: timeit a = arraymaker2.make_1d_array(n, data, dt) 1000000 loops, best of 3: 773 ns per loop In my case I can have many many arrays being created, which unlikely to have the data, dtype or size the same from iteration to iteration. > I'd also like to point out this: Even with the non-optimized "Python" version (py_make_1d_array), creating 1000 arrays will only take 3.5 ms with your timings. "Premature optimization is the root of all evil in computer programming..." (C.A.R. Hoare, according to D. Knuth). Ah yes, but, I believe the Knuth quote is "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil" [1], and I guess that we Cython users feel ourselves to be towards that 3% at least some of the time. Cheers, Matthew [1] http://en.wikipedia.org/wiki/C._A._R._Hoare#Quotes From stefan_ml at behnel.de Tue Nov 17 07:06:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 Nov 2009 07:06:18 +0100 Subject: [Cython] Implementing generators In-Reply-To: <4AFD3D6C.70908@behnel.de> References: <4AFCCDB6.3090209@behnel.de> <4AFD3D6C.70908@behnel.de> Message-ID: <4B023D5A.4020601@behnel.de> Stefan Behnel, 13.11.2009 12:05: > thanks for the feedback, I'll use it to write up a CEP for this. Here goes the CEP: http://wiki.cython.org/enhancements/generators It's somewhat lengthy, but I also consider it pretty complete. Please comment as you see fit. Stefan From robertwb at math.washington.edu Tue Nov 17 20:22:16 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Nov 2009 11:22:16 -0800 Subject: [Cython] small typo: missing " in the docs In-Reply-To: <85b5c3130911151534p709755d5nccebfb89304c4167@mail.gmail.com> References: <85b5c3130911151534p709755d5nccebfb89304c4167@mail.gmail.com> Message-ID: <2E9687AA-4A59-4BFA-A291-050FE2B9C056@math.washington.edu> Thanks! Fixed. On Nov 15, 2009, at 3:34 PM, Ondrej Certik wrote: > Hi, > > http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils > > instead of: > > def say_hello_to(name): > print(Hello %s!" % name) > > there should be: > > def say_hello_to(name): > print("Hello %s!" % name) > > Ondrej > > P.S. I actually tried to compile it. :) > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Tue Nov 17 22:45:25 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Nov 2009 13:45:25 -0800 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: <1cd32cbb0911150418r588f8cefr6fd38a12bb1790ce@mail.gmail.com> References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <1cd32cbb0911131618s3839ae2eke316593b0ba8c591@mail.gmail.com> <1cd32cbb0911150418r588f8cefr6fd38a12bb1790ce@mail.gmail.com> Message-ID: <51164DE5-93CC-4431-9AC8-9BDF006769D2@math.washington.edu> One last release candidate: http://cython.org/release/Cython-0.12.rc1.tar.gz This fixes the fork issue and setting real/imag members of complex variables on MSVC (version 2008+). The Windows link errors are still there, but other than that it should pass all tests. I anticipate this being the actual release by the end of the week. - Robert From Chris.Barker at noaa.gov Wed Nov 18 23:01:38 2009 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed, 18 Nov 2009 14:01:38 -0800 Subject: [Cython] minor nit about distribution Message-ID: <4B046EC2.9080704@noaa.gov> HI folks, I just downloaded the 0.11.3 tarball from: http://pypi.python.org/pypi/Cython/ It's got the ".hg" tree it, which is about 6.5MB of stuff. Should that be in the distributed tarball? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From stefan_ml at behnel.de Thu Nov 19 08:11:33 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Nov 2009 08:11:33 +0100 Subject: [Cython] minor nit about distribution In-Reply-To: <4B046EC2.9080704@noaa.gov> References: <4B046EC2.9080704@noaa.gov> Message-ID: <4B04EFA5.9030107@behnel.de> Chris Barker, 18.11.2009 23:01: > I just downloaded the 0.11.3 tarball from: > > http://pypi.python.org/pypi/Cython/ > > It's got the ".hg" tree it, which is about 6.5MB of stuff. > > Should that be in the distributed tarball? This has been discussed a couple of times in the past, and the answer has always been "yes" so far. A couple of MB isn't much, and it makes it easier for users to write patches. I think a hint next to the download link would make this more apparent. Stefan From Chris.Barker at noaa.gov Fri Nov 20 18:15:57 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 20 Nov 2009 09:15:57 -0800 Subject: [Cython] minor nit about distribution In-Reply-To: <4B04EFA5.9030107@behnel.de> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> Message-ID: <4B06CECD.5050804@noaa.gov> Stefan Behnel wrote: >> It's got the ".hg" tree it, which is about 6.5MB of stuff. >> >> Should that be in the distributed tarball? > > This has been discussed a couple of times in the past, and the answer has > always been "yes" so far. A couple of MB isn't much, and it makes it easier > for users to write patches. fair enough. I'm not very familiar with Hg -- does that hold the entire history? If so, it'll get too big eventually. I'm still not used to the fact that 6.4 MB is inconsequential! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From stefan_ml at behnel.de Fri Nov 20 19:44:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Nov 2009 19:44:26 +0100 Subject: [Cython] Portability of computed goto's ? Message-ID: <4B06E38A.7080708@behnel.de> Hi, the current generator CEP proposes to use computed goto's, as described here, for example: http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Labels-as-Values.html However, from a web search, it seems that they are not a standard C feature and therefore not completely portable, which is the reasons why CPython 3.1 only enables their use optionally for the interpreter core (ceval). Here is an example for a compiler that doesn't seem to support them: http://code.google.com/p/unladen-swallow/issues/detail?id=22 It's one thing to provide a build time option for a performance enhancement (as CPython 3.1 does), but it's a different beast to make a language feature depend on a non-standard compiler feature (as the CEP suggests). I wasn't able to find a list of compilers that support this, so I can't say how many compilers we'd lock out. Any ideas? Pointers? Opinions? Stefan From robertwb at math.washington.edu Fri Nov 20 19:54:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Nov 2009 10:54:33 -0800 Subject: [Cython] minor nit about distribution In-Reply-To: <4B06CECD.5050804@noaa.gov> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> Message-ID: <41F18780-8213-4F77-B685-9F4B16C500CE@math.washington.edu> On Nov 20, 2009, at 9:15 AM, Christopher Barker wrote: > Stefan Behnel wrote: >>> It's got the ".hg" tree it, which is about 6.5MB of stuff. >>> >>> Should that be in the distributed tarball? >> >> This has been discussed a couple of times in the past, and the >> answer has >> always been "yes" so far. A couple of MB isn't much, and it makes >> it easier >> for users to write patches. > > fair enough. I'm not very familiar with Hg -- does that hold the > entire > history? If so, it'll get too big eventually. Yep. IIRC, I think we decided to revisit the issue if/when we hit 10MB. It might be interesting if there was a way to have a partial history, so that one could easily make patches/pull/move forward, but of course not roll back to arbitrary points in time. I don't know if that's possible with hg. Another option would be a script that would run that would instantly turn the folder into a working directory (by downloading all the hg info from hg.cython.org). That probably wouldn't be to hard. - Robert From robertwb at math.washington.edu Fri Nov 20 19:58:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Nov 2009 10:58:13 -0800 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B06E38A.7080708@behnel.de> References: <4B06E38A.7080708@behnel.de> Message-ID: <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> On Nov 20, 2009, at 10:44 AM, Stefan Behnel wrote: > Hi, > > the current generator CEP proposes to use computed goto's, as > described > here, for example: > > http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Labels-as-Values.html > > However, from a web search, it seems that they are not a standard C > feature > and therefore not completely portable, which is the reasons why > CPython 3.1 > only enables their use optionally for the interpreter core (ceval). > Here is > an example for a compiler that doesn't seem to support them: > > http://code.google.com/p/unladen-swallow/issues/detail?id=22 > > It's one thing to provide a build time option for a performance > enhancement > (as CPython 3.1 does), but it's a different beast to make a language > feature depend on a non-standard compiler feature (as the CEP > suggests). > > I wasn't able to find a list of compilers that support this, so I > can't say > how many compilers we'd lock out. Any ideas? Pointers? Opinions? How much of a performance lost are we talking here? If we're already unpacking python arguments, passing through virtual functions, and returning object, having a list of gotos in a switch statement will probably not even be measurably slower (though worth checking). That would seem sufficient for a first pass, at least until we have cdef- level calling of generators. - Robert From Chris.Barker at noaa.gov Fri Nov 20 21:27:10 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 20 Nov 2009 12:27:10 -0800 Subject: [Cython] minor nit about distribution In-Reply-To: <41F18780-8213-4F77-B685-9F4B16C500CE@math.washington.edu> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> <41F18780-8213-4F77-B685-9F4B16C500CE@math.washington.edu> Message-ID: <4B06FB9E.1020309@noaa.gov> Robert Bradshaw wrote: > It might be interesting if there was a way to have a partial > history, so that one could easily make patches/pull/move forward Wouldn't you need to have and know how to sue Hg to do any of that? In which case, why not just have folks pull from the repository if they need to do that. I do like the idea of lowering the barrier to entry for supplying patches. I know I've had a few cases of minor patches I could have done, but I just didn't want to deal with getting the SVN version and all that. (though I write that, it sure sounds lazy!) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robertwb at math.washington.edu Fri Nov 20 22:35:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Nov 2009 13:35:28 -0800 Subject: [Cython] minor nit about distribution In-Reply-To: <4B06FB9E.1020309@noaa.gov> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> <41F18780-8213-4F77-B685-9F4B16C500CE@math.washington.edu> <4B06FB9E.1020309@noaa.gov> Message-ID: <01880AD4-AFFE-405D-97EE-08799828AA1F@math.washington.edu> On Nov 20, 2009, at 12:27 PM, Christopher Barker wrote: > Robert Bradshaw wrote: >> It might be interesting if there was a way to have a partial >> history, so that one could easily make patches/pull/move forward > > Wouldn't you need to have and know how to sue Hg to do any of that? As you mention below, that's not the barrier. Hg is easy to get, and lots of folks probably know how to use it (or can be taught to commit and export a patch in 5 minutes). > In which case, why not just have folks pull from the repository if > they > need to do that. > > I do like the idea of lowering the barrier to entry for supplying > patches. I know I've had a few cases of minor patches I could have > done, > but I just didn't want to deal with getting the SVN version and all > that. > > (though I write that, it sure sounds lazy!) That is exactly the point. Usually, you have the changes in your working copy, and if you feel inclined to submit them, you don't want to check out the repo (which creates everything in a separate directory), copy your changes over, then create a patch, and forever keep your changes in sync where the copy that you're actually developing is *not* the one with revision control. If we could magically change a directory, possibly with changes, that isn't under revision control into a full repository with a single command (and it may be as simple as acquiring the .hg folder and copying it over, but maybe not), that would be great. - Robert From lists at cheimes.de Sat Nov 21 00:49:33 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 21 Nov 2009 00:49:33 +0100 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B06E38A.7080708@behnel.de> References: <4B06E38A.7080708@behnel.de> Message-ID: <4B072B0D.1090705@cheimes.de> Stefan Behnel wrote: > the current generator CEP proposes to use computed goto's, as described > here, for example: > > http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Labels-as-Values.html > > However, from a web search, it seems that they are not a standard C feature > and therefore not completely portable, which is the reasons why CPython 3.1 > only enables their use optionally for the interpreter core (ceval). Here is > an example for a compiler that doesn't seem to support them: > > http://code.google.com/p/unladen-swallow/issues/detail?id=22 > > It's one thing to provide a build time option for a performance enhancement > (as CPython 3.1 does), but it's a different beast to make a language > feature depend on a non-standard compiler feature (as the CEP suggests). > > I wasn't able to find a list of compilers that support this, so I can't say > how many compilers we'd lock out. Any ideas? Pointers? Opinions? Computed gotos are only available with some compilers. As far as I remember they are supported by recent versions of GCC (IIRC 3.x and newer), Intel's ICC and Sun's C compiler. MSVC doesn't have it for sure and will probably never support, it since Microsoft is focusing on C# rather than extending its C/C++ compiler. Antoine Pitrou has lead the implementation in Python 3.1. He might know more details. I've included him in CC. Christian From arfrever.fta at gmail.com Sat Nov 21 00:59:46 2009 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sat, 21 Nov 2009 00:59:46 +0100 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: <51164DE5-93CC-4431-9AC8-9BDF006769D2@math.washington.edu> References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <1cd32cbb0911150418r588f8cefr6fd38a12bb1790ce@mail.gmail.com> <51164DE5-93CC-4431-9AC8-9BDF006769D2@math.washington.edu> Message-ID: <200911210059.53376.Arfrever.FTA@gmail.com> 2009-11-17 22:45:25 Robert Bradshaw napisa?(a): > One last release candidate: http://cython.org/release/Cython-0.12.rc1.tar.gz 0.12.rc1 tarball contains needless Cython/Compiler/._Version.py and ._runtests.py files. Cython/Compiler/._Version.py causes this problem during running tests with Python 3.1: ... Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/StringEncoding.py Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/StringEncoding.py Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/Buffer.py Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/Buffer.py Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/TypeInference.py Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/TypeInference.py Traceback (most recent call last): File "runtests.py", line 740, in ''') File "/usr/lib64/python3.1/distutils/util.py", line 604, in copydir_run_2to3 fixer_names=fixer_names, options=options, explicit=explicit) File "/usr/lib64/python3.1/distutils/util.py", line 572, in run_2to3 r.refactor(files, write=True) File "/usr/lib64/python3.1/lib2to3/refactor.py", line 276, in refactor self.refactor_file(dir_or_file, write, doctests_only) File "/usr/lib64/python3.1/lib2to3/refactor.py", line 315, in refactor_file input, encoding = self._read_python_source(filename) File "/usr/lib64/python3.1/lib2to3/refactor.py", line 311, in _read_python_source return _from_system_newlines(f.read()), encoding File "/usr/lib64/python3.1/codecs.py", line 300, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf8' codec can't decode byte 0xab in position 45: unexpected code byte After removing this file, `python3.1 runtests.py --no-fork -vv` fails later with segmentation fault: ... compiling (c) nogil ... ok compiling (c) nogilcmeth ... ok compiling (c) nogilfunctype ... ok compiling (c) nonconst_def ... ok compiling (c) notcimportedT418 ... ok compiling (c) pxd_cdef_class_declaration_T286 ... ok compiling (c) pyobjcastdisallow_T313 ... ok compiling (c) return_outside_function_T135 ... ok compiling (c) se_badindent ... ok compiling (c) se_badindent2 ... ok compiling (c) se_conddef ... ok compiling (c) se_mixtabspace ... ok compiling (c) se_multass ... ok compiling (c) se_nestdef ... ok compiling (c) string_assignments ... ok compiling (c) tree_assert ... ok compiling (c) typoT304 ... ok compiling (c) undefinedname ... ok compiling (c) void_as_arg ... ok compiling (c) and running __getattribute__ ... Doctest: __getattribute__.both ... Segmentation fault -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20091121/138c0e08/attachment.pgp From greg.ewing at canterbury.ac.nz Sat Nov 21 07:09:50 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 21 Nov 2009 19:09:50 +1300 Subject: [Cython] minor nit about distribution In-Reply-To: <4B06CECD.5050804@noaa.gov> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> Message-ID: <4B07842E.6050601@canterbury.ac.nz> Christopher Barker wrote: > I'm still not used to the fact that 6.4 MB is inconsequential! It's not inconsequential to some people -- not everyone has a blazingly fast nearly-free internet connection. I'm not sure it's fair to impose this on everyone, even if all they want to do is install and use Cython. You don't expect to get the entire history of a project when you download a tar file of the source. -- Greg From greg.ewing at canterbury.ac.nz Sat Nov 21 07:18:36 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 21 Nov 2009 19:18:36 +1300 Subject: [Cython] minor nit about distribution In-Reply-To: <41F18780-8213-4F77-B685-9F4B16C500CE@math.washington.edu> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> <41F18780-8213-4F77-B685-9F4B16C500CE@math.washington.edu> Message-ID: <4B07863C.5030300@canterbury.ac.nz> Robert Bradshaw wrote: > It might be interesting if there was a way to have a partial > history, so that one could easily make patches/pull/move forward, but > of course not roll back to arbitrary points in time. I don't know if > that's possible with hg. A user can get that effect themselves by initializing a new hg repository in their source before they start to modify it. Then it's easy to generate patches relative to whatever they started with. I did that myself when I developed the yield-from implementation for Python, and it worked very well. You could facilitate it by including a .hgignore file in the source distribution excluding .pyc files, etc. to save the user from having to create his own. -- Greg From robertwb at math.washington.edu Sat Nov 21 07:22:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Nov 2009 22:22:19 -0800 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: <200911210059.53376.Arfrever.FTA@gmail.com> References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <1cd32cbb0911150418r588f8cefr6fd38a12bb1790ce@mail.gmail.com> <51164DE5-93CC-4431-9AC8-9BDF006769D2@math.washington.edu> <200911210059.53376.Arfrever.FTA@gmail.com> Message-ID: On Nov 20, 2009, at 3:59 PM, Arfrever Frehtes Taifersar Arahesis wrote: > 2009-11-17 22:45:25 Robert Bradshaw napisa?(a): >> One last release candidate: http://cython.org/release/Cython-0.12.rc1.tar.gz > > 0.12.rc1 tarball contains needless Cython/Compiler/._Version.py > and ._runtests.py files. > Cython/Compiler/._Version.py causes this problem during running > tests with Python 3.1: Ah, I'll make sure those are not in the final distribution. > ... > Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/StringEncoding.py > Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/StringEncoding.py > Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/Buffer.py > Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/Buffer.py > Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/TypeInference.py > Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/TypeInference.py > Traceback (most recent call last): > File "runtests.py", line 740, in > ''') > File "/usr/lib64/python3.1/distutils/util.py", line 604, in > copydir_run_2to3 > fixer_names=fixer_names, options=options, explicit=explicit) > File "/usr/lib64/python3.1/distutils/util.py", line 572, in run_2to3 > r.refactor(files, write=True) > File "/usr/lib64/python3.1/lib2to3/refactor.py", line 276, in > refactor > self.refactor_file(dir_or_file, write, doctests_only) > File "/usr/lib64/python3.1/lib2to3/refactor.py", line 315, in > refactor_file > input, encoding = self._read_python_source(filename) > File "/usr/lib64/python3.1/lib2to3/refactor.py", line 311, in > _read_python_source > return _from_system_newlines(f.read()), encoding > File "/usr/lib64/python3.1/codecs.py", line 300, in decode > (result, consumed) = self._buffer_decode(data, self.errors, final) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xab in position > 45: unexpected code byte Not sure what this is about... I just checked--all Cython source files are pure ASCII. > After removing this file, `python3.1 runtests.py --no-fork -vv` > fails later with segmentation fault: > > ... > compiling (c) nogil ... ok > compiling (c) nogilcmeth ... ok > compiling (c) nogilfunctype ... ok > compiling (c) nonconst_def ... ok > compiling (c) notcimportedT418 ... ok > compiling (c) pxd_cdef_class_declaration_T286 ... ok > compiling (c) pyobjcastdisallow_T313 ... ok > compiling (c) return_outside_function_T135 ... ok > compiling (c) se_badindent ... ok > compiling (c) se_badindent2 ... ok > compiling (c) se_conddef ... ok > compiling (c) se_mixtabspace ... ok > compiling (c) se_multass ... ok > compiling (c) se_nestdef ... ok > compiling (c) string_assignments ... ok > compiling (c) tree_assert ... ok > compiling (c) typoT304 ... ok > compiling (c) undefinedname ... ok > compiling (c) void_as_arg ... ok > compiling (c) and running __getattribute__ ... Doctest: > __getattribute__.both ... Segmentation fault That is worrisome. Just so I understand you right, you ran it all through 2to3, then ran the tests? (Just trying to run python3 runtests.py fails right away...) I'm not sure that's supported yet (the resulting .c files work with Py3, but the compiler itself doesn't run under Py3 yet), but it least it got that far. Stefan, is this expected? - Robert From robertwb at math.washington.edu Sat Nov 21 07:26:10 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Nov 2009 22:26:10 -0800 Subject: [Cython] Cython 0.12 release candidate release notes In-Reply-To: <51164DE5-93CC-4431-9AC8-9BDF006769D2@math.washington.edu> References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <1cd32cbb0911131618s3839ae2eke316593b0ba8c591@mail.gmail.com> <1cd32cbb0911150418r588f8cefr6fd38a12bb1790ce@mail.gmail.com> <51164DE5-93CC-4431-9AC8-9BDF006769D2@math.washington.edu> Message-ID: If you think anything's missing from the release notes, please go and add it. http://wiki.cython.org/ReleaseNotes-0.12 From greg.ewing at canterbury.ac.nz Sat Nov 21 07:37:24 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 21 Nov 2009 19:37:24 +1300 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> Message-ID: <4B078AA4.4060500@canterbury.ac.nz> Robert Bradshaw wrote: > having a list of gotos in a switch statement will > probably not even be measurably slower You mightn't even need the gotos -- just put a big switch statement around the whole function body. Since the Cython language doesn't have a switch statement itself, you aren't going to be generating any other switch statements to interfere with it. -- Greg From robertwb at math.washington.edu Sat Nov 21 07:46:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Nov 2009 22:46:21 -0800 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B078AA4.4060500@canterbury.ac.nz> References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> <4B078AA4.4060500@canterbury.ac.nz> Message-ID: On Nov 20, 2009, at 10:37 PM, Greg Ewing wrote: > Robert Bradshaw wrote: >> having a list of gotos in a switch statement will >> probably not even be measurably slower > > You mightn't even need the gotos -- just put a big > switch statement around the whole function body. > Since the Cython language doesn't have a switch > statement itself, you aren't going to be generating > any other switch statements to interfere with it. We need to be able to jump into the insides of arbitrary loops and the likes... - Robert From robertwb at math.washington.edu Sat Nov 21 08:18:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Nov 2009 23:18:15 -0800 Subject: [Cython] minor nit about distribution In-Reply-To: <4B07842E.6050601@canterbury.ac.nz> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> <4B07842E.6050601@canterbury.ac.nz> Message-ID: <0076031F-40D5-4192-B79D-8EC63E2F5D69@math.washington.edu> On Nov 20, 2009, at 10:09 PM, Greg Ewing wrote: > Christopher Barker wrote: > >> I'm still not used to the fact that 6.4 MB is inconsequential! > > It's not inconsequential to some people -- not everyone > has a blazingly fast nearly-free internet connection. True, though the target audience is much more likely to have a good internet connection, and when downloading it's compressed to about half that. (I pity the developer trying to use Cython on a machine where half a dozen MB is a consequential burden on their hard drive.) On Nov 20, 2009, at 10:18 PM, Greg Ewing wrote: > Robert Bradshaw wrote: >> It might be interesting if there was a way to have a partial >> history, so that one could easily make patches/pull/move forward, but >> of course not roll back to arbitrary points in time. I don't know if >> that's possible with hg. > > A user can get that effect themselves by initializing > a new hg repository in their source before they start > to modify it. Then it's easy to generate patches relative > to whatever they started with. > > I did that myself when I developed the yield-from > implementation for Python, and it worked very well. Yes, that works, but it changes a (usually insignificant) burden on all users' equipment to a (minor but annoying) burden on potential developers themselves. A key point here is that you started with the intent to work on Python. A typical workflow with an open source project is. 1) Use. Use a lot until it becomes essential. 2) Find a bug that I need to fix to keep using the software (fortunately, I rarely hit this point). Or maybe come up with a feature that I think would be really cool to have. 3) Hack. 4) Submit the fix upstream. People moving onto step (4) is what makes an open source project successful. Having to insert "figure what revision control is used, download the dev version, ..." before step (3) blocks the creative juices--at this point often I'm just looking for a workaround, and might work it into a real fix later, and don't want to be bothered with that even if eventual contribution is as the back of my head. Inserting that step after (3) involves figuring out what I changed and copying stuff over (where one half is missing the revision control, and the dev branch might have changed since). I have to admit that there are times I've not contributed stuff upstream just because I was too lazy to get the dev version and integrate my changes over, especially when doing so was completely orthogonal to my task at hand. One of the great things about the Sage project (which I think has helped greatly, and I try to emulate with Cython) is reducing the gap between "user" and "developer." Many contributors to Sage didn't start out intending to be contributors, but it was so easy that when they saw the need, with the proper encouragement, it just happened. As they got more comfortable with the code the contributions became more significant and more numerous. I think there's more "starter" projects in Sage than in Cython, but we're also in the very lucky position that almost all users of Cython are already Python developers. If people get to (3), it should be really easy to get to (4). Anyway, that's my long-winded defense of those 6.4 MB. A script to turn a simple source directory into a full-blown repository would nearly meet the above goals, and is probably not that hard. Anyone want to submit a patch? :-) > You could facilitate it by including a .hgignore file > in the source distribution excluding .pyc files, etc. > to save the user from having to create his own. .hgignore is itself under revision control. - Robert From stefan_ml at behnel.de Sat Nov 21 08:36:07 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Nov 2009 08:36:07 +0100 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> Message-ID: <4B079867.1010104@behnel.de> Robert Bradshaw, 20.11.2009 19:58: > How much of a performance lost are we talking here? If we're already > unpacking python arguments, Most of this will happen behind METH_O functions or __next__, so I'd expect most generator code to run without tuple (un-)packing. Couroutines and their .send(x) method will require a tuple, though, at least when called from user code. > passing through virtual functions, Yes. Even __next__ will be called through the type slot. > and returning object, having a list of gotos in a switch statement will > probably not even be measurably slower (though worth checking). I agree that the existing overhead is heavy enough to allow for an additional switch statement. We might also be able to generate both using a precompiler switch. The closure class could have a union {void* label, size_t yield_id} yield_target and the generated entry code could use either of them depending on the C compiler being used. > That > would seem sufficient for a first pass, at least until we have cdef- > level calling of generators. Although that shouldn't be too far off once we have working generators. Stefan From solipsis at pitrou.net Sat Nov 21 12:05:52 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 21 Nov 2009 11:05:52 +0000 (UTC) Subject: [Cython] Portability of computed goto's ? References: <4B06E38A.7080708@behnel.de> <4B072B0D.1090705@cheimes.de> Message-ID: Christian Heimes writes: > > Computed gotos are only available with some compilers. As far as I > remember they are supported by recent versions of GCC (IIRC 3.x and > newer), Intel's ICC and Sun's C compiler. MSVC doesn't have it for sure > and will probably never support, it since Microsoft is focusing on C# > rather than extending its C/C++ compiler. > > Antoine Pitrou has lead the implementation in Python 3.1. He might know > more details. I've included him in CC. I don't know anything more than what you said :) Basically, it's a non-standard C feature, so you shouldn't rely on it being available everywhere and everytime. Regards Antoine. From arfrever.fta at gmail.com Sat Nov 21 14:44:17 2009 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sat, 21 Nov 2009 14:44:17 +0100 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <200911210059.53376.Arfrever.FTA@gmail.com> Message-ID: <200911211444.32115.Arfrever.FTA@gmail.com> 2009-11-21 07:22:19 Robert Bradshaw napisa?(a): > On Nov 20, 2009, at 3:59 PM, Arfrever Frehtes Taifersar Arahesis wrote: > > > 2009-11-17 22:45:25 Robert Bradshaw napisa?(a): > >> One last release candidate: http://cython.org/release/Cython-0.12.rc1.tar.gz > > > > 0.12.rc1 tarball contains needless Cython/Compiler/._Version.py > > and ._runtests.py files. > > Cython/Compiler/._Version.py causes this problem during running > > tests with Python 3.1: > > Ah, I'll make sure those are not in the final distribution. > > > ... > > Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/StringEncoding.py > > Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/StringEncoding.py > > Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/Buffer.py > > Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/Buffer.py > > Refactoring /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/TypeInference.py > > Wrote changes to /var/tmp/portage/dev-python/cython-0.12_rc1/work/ > > Cython-0.12.rc1/BUILD/Cy3/Cython/Compiler/TypeInference.py > > Traceback (most recent call last): > > File "runtests.py", line 740, in > > ''') > > File "/usr/lib64/python3.1/distutils/util.py", line 604, in > > copydir_run_2to3 > > fixer_names=fixer_names, options=options, explicit=explicit) > > File "/usr/lib64/python3.1/distutils/util.py", line 572, in run_2to3 > > r.refactor(files, write=True) > > File "/usr/lib64/python3.1/lib2to3/refactor.py", line 276, in > > refactor > > self.refactor_file(dir_or_file, write, doctests_only) > > File "/usr/lib64/python3.1/lib2to3/refactor.py", line 315, in > > refactor_file > > input, encoding = self._read_python_source(filename) > > File "/usr/lib64/python3.1/lib2to3/refactor.py", line 311, in > > _read_python_source > > return _from_system_newlines(f.read()), encoding > > File "/usr/lib64/python3.1/codecs.py", line 300, in decode > > (result, consumed) = self._buffer_decode(data, self.errors, final) > > UnicodeDecodeError: 'utf8' codec can't decode byte 0xab in position > > 45: unexpected code byte > > Not sure what this is about... I just checked--all Cython source files > are pure ASCII. These non-ASCII bytes were in Cython/Compiler/._Version.py. > > After removing this file, `python3.1 runtests.py --no-fork -vv` > > fails later with segmentation fault: > > > > ... > > compiling (c) nogil ... ok > > compiling (c) nogilcmeth ... ok > > compiling (c) nogilfunctype ... ok > > compiling (c) nonconst_def ... ok > > compiling (c) notcimportedT418 ... ok > > compiling (c) pxd_cdef_class_declaration_T286 ... ok > > compiling (c) pyobjcastdisallow_T313 ... ok > > compiling (c) return_outside_function_T135 ... ok > > compiling (c) se_badindent ... ok > > compiling (c) se_badindent2 ... ok > > compiling (c) se_conddef ... ok > > compiling (c) se_mixtabspace ... ok > > compiling (c) se_multass ... ok > > compiling (c) se_nestdef ... ok > > compiling (c) string_assignments ... ok > > compiling (c) tree_assert ... ok > > compiling (c) typoT304 ... ok > > compiling (c) undefinedname ... ok > > compiling (c) void_as_arg ... ok > > compiling (c) and running __getattribute__ ... Doctest: > > __getattribute__.both ... Segmentation fault > > That is worrisome. Just so I understand you right, you ran it all > through 2to3, then ran the tests? I ran `python3.1 setup.py build`. setup.py internally uses lib2to3 and distutils.command.build_py.build_py_2to3. Next I ran the tests with Python 3.1. (All tests pass with Python 2.6.) -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20091121/b766c4e7/attachment.pgp From lodatom at gmail.com Sat Nov 21 19:18:26 2009 From: lodatom at gmail.com (Mark Lodato) Date: Sat, 21 Nov 2009 13:18:26 -0500 Subject: [Cython] minor nit about distribution In-Reply-To: <0076031F-40D5-4192-B79D-8EC63E2F5D69@math.washington.edu> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> <4B07842E.6050601@canterbury.ac.nz> <0076031F-40D5-4192-B79D-8EC63E2F5D69@math.washington.edu> Message-ID: On Sat, Nov 21, 2009 at 2:18 AM, Robert Bradshaw wrote: > True, though the target audience is much more likely to have a good > internet connection, and when downloading it's compressed to about > half that. (I pity the developer trying to use Cython on a machine > where half a dozen MB is a consequential burden on their hard drive.) Just to make this explicit, the .hg directory only adds about 3.4 MB to the download, not 6.4MB (or 7.6MB). Here are some test results: 464K cython-devel.tar.bz2 564K cython-devel.tar.gz 3.1M cython-devel_withgit.tar.bz2 3.2M cython-devel_withgit.tar.gz 3.8M cython-devel_withhg.tar.bz2 3.9M cython-devel_withhg.tar.gz 3.0M cython-devel/.git 7.6M cython-devel/.hg Explanation: * cython-devel is working tree only * cython-devel_withhg is working tree + .hg directory (same as we currently do) * cython-devel_withgit is working tree + .git directory (after running hg-fast-export and doing git gc --aggressive) * cython-devel/.hg is size of extracted .hg directory * cython-devel/.git is size of extracted .git directory > Anyway, that's my long-winded defense of those 6.4 MB. A script to > turn a simple source directory into a full-blown repository would > nearly meet the above goals, and is probably not that hard. Anyone > want to submit a patch? :-) Here's a proof of concept. I haven't tested it, and I don't know Hg well, so this may not be good. It might be better to turn this into a Makefile target rather than a script. #!/bin/bash -x -e tag=0.11.3 repo=http://hg.cython.org/cython-devel tmpdir=`mktemp --tmpdir=. -d` hg clone $repo $tmpdir cd $tmpdir hg update $tag cd .. mv $tmpdir/{.hg,.hgtags} . rm -rf $tmpdir The rest of my email contains a bunch of git commands, for the curious (and for me in the future, in case I forget them). -- Mark ------------ Git Stuff ------------ Here are the steps to reproduce my size numbers: git clone git://repo.or.cz/fast-export.git hg clone http://hg.cython.org/cython-devel cd cython-devel git init ../fast-export/hg-fast-export.sh -r . rm -f .git/hg2git* git reset git gc --aggressive cd .. tar cvzf cython-devel.tar.gz cython-devel --exclude=.git --exclude=.hg tar cvzf cython-devel_withhg.tar.gz cython-devel --exclude=.git tar cvzf cython-devel_withgit.tar.gz cython-devel --exclude=.hg tar cvjf cython-devel.tar.bz2 cython-devel --exclude=.git --exclude=.hg tar cvjf cython-devel_withhg.tar.bz2 cython-devel --exclude=.git tar cvjf cython-devel_withgit.tar.bz2 cython-devel --exclude=.hg du -hs cython-devel*.* cython-devel/.{git,hg} Here's a git version of the update script (not well tested): #!/bin/bash -x -e tag=0.11.3 branch=working-$TAG repo=git://git.cython.org git init git remote add origin $repo git fetch git branch master origin/master git update-ref --no-deref HEAD $tag git reset git checkout -b $branch From stefan_ml at behnel.de Sat Nov 21 20:30:54 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Nov 2009 20:30:54 +0100 Subject: [Cython] Cython 0.12 release candidate In-Reply-To: <200911210059.53376.Arfrever.FTA@gmail.com> References: <4119AE38-6AF4-4B3E-B2E0-7CD573C1677D@math.washington.edu> <1cd32cbb0911150418r588f8cefr6fd38a12bb1790ce@mail.gmail.com> <51164DE5-93CC-4431-9AC8-9BDF006769D2@math.washington.edu> <200911210059.53376.Arfrever.FTA@gmail.com> Message-ID: <4B083FEE.1040404@behnel.de> Arfrever Frehtes Taifersar Arahesis, 21.11.2009 00:59: > `python3.1 runtests.py --no-fork -vv` fails later with segmentation fault: > > ... > compiling (c) void_as_arg ... ok > compiling (c) and running __getattribute__ ... Doctest: __getattribute__.both ... Segmentation fault Yes, that's a known crash bug in CPython 3.1. I already reported it a while ago, but there wasn't any response so far. http://bugs.python.org/issue7173 It's not related to the code Cython generates, so the tests pass successfully when you generate the C files in Python 2.6 or earlier. It just crashes when you run the compiler's tree assertions in Python 3.1, i.e. for the pure Python code of the compiler itself. Stefan From robertwb at math.washington.edu Sat Nov 21 20:46:17 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 21 Nov 2009 11:46:17 -0800 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B079867.1010104@behnel.de> References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> <4B079867.1010104@behnel.de> Message-ID: On Nov 20, 2009, at 11:36 PM, Stefan Behnel wrote: > > I agree that the existing overhead is heavy enough to allow for an > additional switch statement. We might also be able to generate both > using a > precompiler switch. The closure class could have a > > union {void* label, size_t yield_id} yield_target > > and the generated entry code could use either of them depending on > the C > compiler being used. Well, I'd rather a macro than a union for type safety, but we could certainly provide the optimization for compilers that support them. - Robert From greg.ewing at canterbury.ac.nz Sat Nov 21 22:37:41 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 22 Nov 2009 10:37:41 +1300 Subject: [Cython] Portability of computed goto's ? In-Reply-To: References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> <4B078AA4.4060500@canterbury.ac.nz> Message-ID: <4B085DA5.9090804@canterbury.ac.nz> Robert Bradshaw wrote: > We need to be able to jump into the insides of arbitrary loops and the > likes... You can -- look up "Duff's device". -- Greg From stefan_ml at behnel.de Sat Nov 21 22:59:21 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Nov 2009 22:59:21 +0100 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B085DA5.9090804@canterbury.ac.nz> References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> <4B078AA4.4060500@canterbury.ac.nz> <4B085DA5.9090804@canterbury.ac.nz> Message-ID: <4B0862B9.8060708@behnel.de> Greg Ewing, 21.11.2009 22:37: > Robert Bradshaw wrote: > >> We need to be able to jump into the insides of arbitrary loops and the >> likes... > > You can -- look up "Duff's device". The sick definition of switch statements in the C language doesn't help here. Cython actually /does/ generate switch statements for certain if-elif-else constructs, so we can't use Duff's device for generators. However, I'd expect that for the use case at hand, a switch statement full of goto's would lead to exactly the same assembly code as Duff's device in action. Stefan From robertwb at math.washington.edu Sat Nov 21 22:59:03 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 21 Nov 2009 13:59:03 -0800 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B085DA5.9090804@canterbury.ac.nz> References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> <4B078AA4.4060500@canterbury.ac.nz> <4B085DA5.9090804@canterbury.ac.nz> Message-ID: On Nov 21, 2009, at 1:37 PM, Greg Ewing wrote: > Robert Bradshaw wrote: > >> We need to be able to jump into the insides of arbitrary loops and >> the >> likes... > > You can -- look up "Duff's device". I've heard of that trick for loop unrolling, but I never realized that was valid C. Crazy. You're right though, that would work. - Robert From robertwb at math.washington.edu Sat Nov 21 23:02:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 21 Nov 2009 14:02:07 -0800 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B0862B9.8060708@behnel.de> References: <4B06E38A.7080708@behnel.de> <2411C86F-D266-467F-A245-3BE0C0496391@math.washington.edu> <4B078AA4.4060500@canterbury.ac.nz> <4B085DA5.9090804@canterbury.ac.nz> <4B0862B9.8060708@behnel.de> Message-ID: On Nov 21, 2009, at 1:59 PM, Stefan Behnel wrote: > > Greg Ewing, 21.11.2009 22:37: >> Robert Bradshaw wrote: >> >>> We need to be able to jump into the insides of arbitrary loops and >>> the >>> likes... >> >> You can -- look up "Duff's device". > > The sick definition of switch statements in the C language doesn't > help > here. Cython actually /does/ generate switch statements for certain > if-elif-else constructs, so we can't use Duff's device for generators. Oh, yeah, I forgot about that (despite the fact that I'm the one who implemented it...) > However, I'd expect that for the use case at hand, a switch > statement full > of goto's would lead to exactly the same assembly code as Duff's > device in > action. Yeah, me too. - Robert From stefan_ml at behnel.de Sun Nov 22 03:39:10 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 22 Nov 2009 03:39:10 +0100 Subject: [Cython] Portability of computed goto's ? In-Reply-To: References: <4B06E38A.7080708@behnel.de> <4B072B0D.1090705@cheimes.de> Message-ID: <4B08A44E.5010703@behnel.de> Antoine Pitrou, 21.11.2009 12:05: > Christian Heimes writes: >> Computed gotos are only available with some compilers. As far as I >> remember they are supported by recent versions of GCC (IIRC 3.x and >> newer), Intel's ICC and Sun's C compiler. MSVC doesn't have it for sure >> and will probably never support, it since Microsoft is focusing on C# >> rather than extending its C/C++ compiler. >> >> Antoine Pitrou has lead the implementation in Python 3.1. He might know >> more details. I've included him in CC. > > I don't know anything more than what you said :) > Basically, it's a non-standard C feature, so you shouldn't rely on it being > available everywhere and everytime. Just to be sure, not only the computed goto itself is a non-standard feature, but also taking the address of a label using "&&", right? If the latter was portable, it would at least simplify the generated code as we could switch on label addresses... Stefan From robertwb at math.washington.edu Sun Nov 22 04:49:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 21 Nov 2009 19:49:41 -0800 Subject: [Cython] Portability of computed goto's ? In-Reply-To: <4B08A44E.5010703@behnel.de> References: <4B06E38A.7080708@behnel.de> <4B072B0D.1090705@cheimes.de> <4B08A44E.5010703@behnel.de> Message-ID: On Nov 21, 2009, at 6:39 PM, Stefan Behnel wrote: > Antoine Pitrou, 21.11.2009 12:05: >> Christian Heimes writes: >>> Computed gotos are only available with some compilers. As far as I >>> remember they are supported by recent versions of GCC (IIRC 3.x and >>> newer), Intel's ICC and Sun's C compiler. MSVC doesn't have it for >>> sure >>> and will probably never support, it since Microsoft is focusing on >>> C# >>> rather than extending its C/C++ compiler. >>> >>> Antoine Pitrou has lead the implementation in Python 3.1. He might >>> know >>> more details. I've included him in CC. >> >> I don't know anything more than what you said :) >> Basically, it's a non-standard C feature, so you shouldn't rely on >> it being >> available everywhere and everytime. > > Just to be sure, not only the computed goto itself is a non-standard > feature, but also taking the address of a label using "&&", right? Can't imagine why you'd want/need a label if you couldn't use it... > If the latter was portable, it would at least simplify the generated > code > as we could switch on label addresses... You can only switch on integral values, and switching on (extremely) non-consecutive values yields much less efficient code. - Robert From stefan_ml at behnel.de Sun Nov 22 05:55:01 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 22 Nov 2009 05:55:01 +0100 Subject: [Cython] Portability of computed goto's ? In-Reply-To: References: <4B06E38A.7080708@behnel.de> <4B072B0D.1090705@cheimes.de> <4B08A44E.5010703@behnel.de> Message-ID: <4B08C425.3070803@behnel.de> Robert Bradshaw, 22.11.2009 04:49: > On Nov 21, 2009, at 6:39 PM, Stefan Behnel wrote: > >> Antoine Pitrou, 21.11.2009 12:05: >>> Christian Heimes writes: >>>> Computed gotos are only available with some compilers. As far as I >>>> remember they are supported by recent versions of GCC (IIRC 3.x and >>>> newer), Intel's ICC and Sun's C compiler. MSVC doesn't have it for >>>> sure >>>> and will probably never support, it since Microsoft is focusing on >>>> C# >>>> rather than extending its C/C++ compiler. >>>> >>>> Antoine Pitrou has lead the implementation in Python 3.1. He might >>>> know >>>> more details. I've included him in CC. >>> I don't know anything more than what you said :) >>> Basically, it's a non-standard C feature, so you shouldn't rely on >>> it being >>> available everywhere and everytime. >> Just to be sure, not only the computed goto itself is a non-standard >> feature, but also taking the address of a label using "&&", right? > > Can't imagine why you'd want/need a label if you couldn't use it... > >> If the latter was portable, it would at least simplify the generated >> code >> as we could switch on label addresses... > > You can only switch on integral values, and switching on (extremely) > non-consecutive values yields much less efficient code. Ok, I updated the CEP accordingly. http://wiki.cython.org/enhancements/generators#Implementinggeneratorfunctions Stefan From robertwb at math.washington.edu Sun Nov 22 10:15:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 22 Nov 2009 01:15:08 -0800 Subject: [Cython] minor nit about distribution In-Reply-To: References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> <4B07842E.6050601@canterbury.ac.nz> <0076031F-40D5-4192-B79D-8EC63E2F5D69@math.washington.edu> Message-ID: <50DB0CA7-14FC-46C3-9680-7DCCB3A83217@math.washington.edu> On Nov 21, 2009, at 10:18 AM, Mark Lodato wrote: > On Sat, Nov 21, 2009 at 2:18 AM, Robert Bradshaw > wrote: >> True, though the target audience is much more likely to have a good >> internet connection, and when downloading it's compressed to about >> half that. (I pity the developer trying to use Cython on a machine >> where half a dozen MB is a consequential burden on their hard drive.) > > Just to make this explicit, the .hg directory only adds about 3.4 MB > to the download, not 6.4MB (or 7.6MB). Here are some test results: > > 464K cython-devel.tar.bz2 > 564K cython-devel.tar.gz > 3.1M cython-devel_withgit.tar.bz2 > 3.2M cython-devel_withgit.tar.gz > 3.8M cython-devel_withhg.tar.bz2 > 3.9M cython-devel_withhg.tar.gz > 3.0M cython-devel/.git > 7.6M cython-devel/.hg > > Explanation: > * cython-devel is working tree only > * cython-devel_withhg is working tree + .hg directory (same as we > currently do) > * cython-devel_withgit is working tree + .git directory (after running > hg-fast-export and doing git gc --aggressive) > * cython-devel/.hg is size of extracted .hg directory > * cython-devel/.git is size of extracted .git directory Interesting stats. >> Anyway, that's my long-winded defense of those 6.4 MB. A script to >> turn a simple source directory into a full-blown repository would >> nearly meet the above goals, and is probably not that hard. Anyone >> want to submit a patch? :-) > > Here's a proof of concept. I haven't tested it, and I don't know Hg > well, so this may not be good. It might be better to turn this into a > Makefile target rather than a script. > > #!/bin/bash -x -e > tag=0.11.3 > repo=http://hg.cython.org/cython-devel > tmpdir=`mktemp --tmpdir=. -d` > hg clone $repo $tmpdir > cd $tmpdir > hg update $tag > cd .. > mv $tmpdir/{.hg,.hgtags} . > rm -rf $tmpdir Thanks. I was hoping it wouldn't be anything messy, but that seems to do it (except that --tmpdir isn't a standard posix option). I agree, I'll turn this into a Makefile. Probably too big of a change to put into 0.12 at this point, but I'll turn this into a makefile. - Robert From robertwb at math.washington.edu Sun Nov 22 11:36:23 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 22 Nov 2009 02:36:23 -0800 Subject: [Cython] minor nit about distribution In-Reply-To: <50DB0CA7-14FC-46C3-9680-7DCCB3A83217@math.washington.edu> References: <4B046EC2.9080704@noaa.gov> <4B04EFA5.9030107@behnel.de> <4B06CECD.5050804@noaa.gov> <4B07842E.6050601@canterbury.ac.nz> <0076031F-40D5-4192-B79D-8EC63E2F5D69@math.washington.edu> <50DB0CA7-14FC-46C3-9680-7DCCB3A83217@math.washington.edu> Message-ID: On Nov 22, 2009, at 1:15 AM, Robert Bradshaw wrote: > On Nov 21, 2009, at 10:18 AM, Mark Lodato wrote: >> >> >> Here's a proof of concept. I haven't tested it, and I don't know Hg >> well, so this may not be good. It might be better to turn this >> into a >> Makefile target rather than a script. >> >> #!/bin/bash -x -e >> tag=0.11.3 >> repo=http://hg.cython.org/cython-devel >> tmpdir=`mktemp --tmpdir=. -d` >> hg clone $repo $tmpdir >> cd $tmpdir >> hg update $tag >> cd .. >> mv $tmpdir/{.hg,.hgtags} . >> rm -rf $tmpdir > > Thanks. I was hoping it wouldn't be anything messy, and that seems to > do it (except that --tmpdir isn't a standard posix option). I agree, > I'll turn this into a Makefile. Probably too big of a change to put > into 0.12 at this point, but I'll turn this into a makefile. http://trac.cython.org/cython_trac/ticket/458 - Robert From solipsis at pitrou.net Sun Nov 22 18:06:01 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Nov 2009 17:06:01 +0000 (UTC) Subject: [Cython] Portability of computed goto's ? References: <4B06E38A.7080708@behnel.de> <4B072B0D.1090705@cheimes.de> <4B08A44E.5010703@behnel.de> Message-ID: Stefan Behnel writes: > > Just to be sure, not only the computed goto itself is a non-standard > feature, but also taking the address of a label using "&&", right? Indeed. I'm not sure why you care about computed gotos here. Most generators only have a single yield statement, so you'll have a single-case switch statement (or two cases if you take into account the case where the generator is being entered rather than resumed), so the code will be quite simple anyway and the CPU's branch predictor should do a good job. Regards Antoine. From stefan_ml at behnel.de Mon Nov 23 08:57:22 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 23 Nov 2009 08:57:22 +0100 Subject: [Cython] yield dispatch and branch prediction (was: Portability of computed goto's ?) In-Reply-To: References: <4B06E38A.7080708@behnel.de> <4B072B0D.1090705@cheimes.de> <4B08A44E.5010703@behnel.de> Message-ID: <4B0A4062.3070806@behnel.de> Antoine Pitrou, 22.11.2009 18:06: > Stefan Behnel writes: >> Just to be sure, not only the computed goto itself is a non-standard >> feature, but also taking the address of a label using "&&", right? > > Indeed. I'm not sure why you care about computed gotos here. Most generators > only have a single yield statement, so you'll have a single-case switch > statement (or two cases if you take into account the case where the generator is > being entered rather than resumed), so the code will be quite simple anyway and > the CPU's branch predictor should do a good job. Agreed. However, the common single-yield case can always be optimised into if (likely(closure._resume_label == ...)) goto __L1_resume_from_yield; ('likely' being a branch prediction hint here), in which case it wouldn't matter if we use an int or a pointer. One test is needed anyway, as the label is initially 0. As a result, you'd just get an asymptotic 100% branch prediction with a single initial branch. The main advantage of computed gotos for the multiple-yield cases is that the code generation itself becomes simpler, as the generator function wouldn't have to know the yield labels (which are created at code generation time and only at need, so only the yield node itself knows if it gets generated). But given that they are not portable, and assuming that there are cases where knowing the labels is simply faster (such as the single-yield case above), we will need to propagate that information anyway, so we can just as well drop the computed gotos completely. But you got me thinking about the branch prediction. Disclaimer: I'm actually aware that this deals with micro-optimisations that might not turn into any performance gain at all. I just found it interesting to look at what kind of code we can expect to see. For the multiple yield case, I think a scenario like the following would be quite common: def gen(): while 1: # calculate some correlated values x,y,z yield x yield y yield z In that case, there will be zero branch prediction, as the code will jump to a different label on each call, despite them being close in the code and doing exactly the same thing. Looking at http://docs.python.org/library/itertools.html another common scenario would be this: def gen(flag): if flag: while 1: yield ... else: while 1: yield ... in which case the branch prediction would actually work pretty well, but would depend on how the switch code is arranged. Any ideas how computed gotos would perform here? They'd always jump to the same address, after all, but I have no idea if the CPU learns from that... A third scenario uses an initial setup run (also from itertools): def gen(): for x in something: yield f(x) while 1: for x in something_else: yield f(x) Here, the branch prediction would only fail in two cases during the life cycle (at startup and when switching to the while loop), and the final performance would depend on the switch statement again. Stefan From robertwb at math.washington.edu Mon Nov 23 18:53:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Nov 2009 09:53:41 -0800 Subject: [Cython] yield dispatch and branch prediction (was: Portability of computed goto's ?) In-Reply-To: <4B0A4062.3070806@behnel.de> References: <4B06E38A.7080708@behnel.de> <4B072B0D.1090705@cheimes.de> <4B08A44E.5010703@behnel.de> <4B0A4062.3070806@behnel.de> Message-ID: On Nov 22, 2009, at 11:57 PM, Stefan Behnel wrote: > Antoine Pitrou, 22.11.2009 18:06: >> Stefan Behnel writes: >>> Just to be sure, not only the computed goto itself is a non-standard >>> feature, but also taking the address of a label using "&&", right? >> >> Indeed. I'm not sure why you care about computed gotos here. Most >> generators >> only have a single yield statement, so you'll have a single-case >> switch >> statement (or two cases if you take into account the case where the >> generator is >> being entered rather than resumed), so the code will be quite >> simple anyway and >> the CPU's branch predictor should do a good job. > > Agreed. However, the common single-yield case can always be > optimised into > > if (likely(closure._resume_label == ...)) > goto __L1_resume_from_yield; > > ('likely' being a branch prediction hint here), in which case it > wouldn't > matter if we use an int or a pointer. One test is needed anyway, as > the > label is initially 0. As a result, you'd just get an asymptotic 100% > branch > prediction with a single initial branch. > > The main advantage of computed gotos for the multiple-yield cases is > that > the code generation itself becomes simpler, as the generator function > wouldn't have to know the yield labels (which are created at code > generation time and only at need, so only the yield node itself > knows if it > gets generated). But given that they are not portable, and assuming > that > there are cases where knowing the labels is simply faster (such as the > single-yield case above), we will need to propagate that information > anyway, so we can just as well drop the computed gotos completely. > But you > got me thinking about the branch prediction. > > Disclaimer: I'm actually aware that this deals with micro- > optimisations > that might not turn into any performance gain at all. I just found it > interesting to look at what kind of code we can expect to see. > > For the multiple yield case, I think a scenario like the following > would be > quite common: > > def gen(): > while 1: > # calculate some correlated values x,y,z > yield x > yield y > yield z > > In that case, there will be zero branch prediction, as the code will > jump > to a different label on each call, despite them being close in the > code and > doing exactly the same thing. > > Looking at > > http://docs.python.org/library/itertools.html > > another common scenario would be this: > > def gen(flag): > if flag: > while 1: > yield ... > else: > while 1: > yield ... > > in which case the branch prediction would actually work pretty well, > but > would depend on how the switch code is arranged. Any ideas how > computed > gotos would perform here? They'd always jump to the same address, > after > all, but I have no idea if the CPU learns from that... > > A third scenario uses an initial setup run (also from itertools): > > def gen(): > for x in something: > yield f(x) > while 1: > for x in something_else: > yield f(x) > > Here, the branch prediction would only fail in two cases during the > life > cycle (at startup and when switching to the while loop), and the final > performance would depend on the switch statement again. I realize this is just an interesting exercise, but one thing to keep in mind is that unless the calling code is trivial (and maybe even if it is), the branch prediction registers for the generator will probably be flushed by the next time you call it. - Robert From robertwb at math.washington.edu Mon Nov 23 20:33:44 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Nov 2009 11:33:44 -0800 Subject: [Cython] Cython 0.12 released. Message-ID: Cython 0.12 has been officially released. You can get it directly from http://cython.org/ or at http://pypi.python.org/pypi/Cython/. This is the culmination of many months of work, including a mergeback of the experimental branch (after much testing) that was started earlier this year. == New features == - Type inference with the infer_types directive - Seamless C++ complex support - Fast extension type instantiation using the normal Python meme obj = MyType.__new__(MyType). - Improved support for Py3.1 - Cython now runs under Python 3.x using the 2to3 tool - unittest support for doctests in Cython modules - Optimised handling of C strings (char*): for c in cstring[2:50] and cstring.decode() - Looping over c pointers: for i in intptr[:50]. - Many other optimisation, e.g. enumerate() loops, parallel swap assignments (a,b = b,a), and unicode.encode() - More complete numpy.pxd - pyximport improvements - cython_freeze improvements - Many bug fixes There has also been a lot of work behind the scenes to improve temp handling, streamline writing optimizations, and cleanup code in general. A list of tickets closed can be found at http://trac.cython.org/cython_trac/query?group=component&milestone=0.12 == Semantic Changes == This revision to introduces some backwards incompatible changes to more closely align Cython with the Python language, and unify C and Python types to pave the way for seamless type inference. They are: - Division involving negative C integers now follow Python semantics rather than C semantics. Division by zero also now raises a Python exception. See [[enhancements/division|CEP 516]] This has been optional for several releases, and is now the default. It can be disabled on a per-function or per-file basis, or from the command line (see [[enhancements/compilerdirectives]]) - Unmarked strings are now of type str in both Python 2.x (becoming byte strings) and Python 3.x (becoming unicode strings). Byte strings may be marked as b"..." and unicode strings as u"..." no matter what the runtime environment. See [[enhancements/stringliterals|CEP 108]] - The boolean expressions x or y and x and y return either x or y just like in Python, even when x and y are c types. This means that the types must be compatible. Previously x and y were coerced into truth values first, and either 1 or 0 was returned. == Contributors to this release == * Peter Alexander * Stefan Behnel * Robert Bradshaw * David Cournapeau * Lisandro Dalcin * Mark Lodato * Sturla Molden * Dag Sverre Seljebotn * Holger Thanks also to everybody who's helping us out in our discussions on the mailing list. From stefan_ml at behnel.de Tue Nov 24 12:55:21 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Nov 2009 12:55:21 +0100 Subject: [Cython] Cython 0.12 released. In-Reply-To: References: Message-ID: <4B0BC9A9.3070606@behnel.de> Robert Bradshaw, 23.11.2009 20:33: > == New features == > - Type inference with the infer_types directive Now that you mention it - why is there a directive for this? Can't this be enabled by default? Stefan From robertwb at math.washington.edu Tue Nov 24 19:32:48 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Nov 2009 10:32:48 -0800 Subject: [Cython] Cython 0.12 released. In-Reply-To: <4B0BC9A9.3070606@behnel.de> References: <4B0BC9A9.3070606@behnel.de> Message-ID: <359BD084-7CB8-4E45-8D36-02BB6047D0EC@math.washington.edu> On Nov 24, 2009, at 3:55 AM, Stefan Behnel wrote: > > Robert Bradshaw, 23.11.2009 20:33: >> == New features == >> - Type inference with the infer_types directive > > Now that you mention it - why is there a directive for this? Can't > this be > enabled by default? I think this came up before--first, it's a relatively new feature, so it'd be nice to get it out there a tested some, but more importantly it's such a large, backwards incompatible change that it didn't make sense to do it with all the other stuff going on on 0.12. There's a lot of code out there (much written by me, for example, when type inference wasn't even on the horizon) that uses the fact that assignment to an untyped variable is conversion to an object. One thing that I want to do is compile all of Sage with type inference on and see how much breaks. (In general, we're pretty good a typing variables, so I don't think it'll make a huge difference.) Have you tried compiling lxml with type inference enabled? - Robert From jonas at lophus.org Tue Nov 24 19:41:06 2009 From: jonas at lophus.org (jonas at lophus.org) Date: Tue, 24 Nov 2009 19:41:06 +0100 Subject: [Cython] Two docs bugs Message-ID: <4B0C28C2.6010200@lophus.org> Hi guys, first of all: nice work for 0.12! In the pxd docs source, use "files -- they" with spaces rather than "files--they" without any in the first paragraph, so sphinx renders this correctly. The second thing: pygments has experimental support for Cython source code since some weeks' commit ago. Could you please make use of that Cython highlighter? Cython code is very hard to read when not highlighted. Regards, Jonas From stefan_ml at behnel.de Tue Nov 24 21:08:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Nov 2009 21:08:53 +0100 Subject: [Cython] Cython 0.12 released. In-Reply-To: <359BD084-7CB8-4E45-8D36-02BB6047D0EC@math.washington.edu> References: <4B0BC9A9.3070606@behnel.de> <359BD084-7CB8-4E45-8D36-02BB6047D0EC@math.washington.edu> Message-ID: <4B0C3D55.4030701@behnel.de> Robert Bradshaw, 24.11.2009 19:32: > On Nov 24, 2009, at 3:55 AM, Stefan Behnel wrote: > >> Robert Bradshaw, 23.11.2009 20:33: >>> == New features == >>> - Type inference with the infer_types directive >> Now that you mention it - why is there a directive for this? Can't >> this be enabled by default? > > I think this came up before Yes, I recall something like that, too. > first, it's a relatively new feature, so > it'd be nice to get it out there a tested some, but more importantly > it's such a large, backwards incompatible change that it didn't make > sense to do it with all the other stuff going on on 0.12. There's a > lot of code out there (much written by me, for example, when type > inference wasn't even on the horizon) that uses the fact that > assignment to an untyped variable is conversion to an object. Right, that's certainly a feature that people have their code depend on. Breaking it will break tons and tons of code, so it *is* good not to make it the default. How would you want this conversion to work in the future? With an explicit cast? How does the current implementation actually handle diverging assignments? As in if test: items = [] else: items = () or: items = () if test: items = [] or: a = 5 ... a = "test" ... > One thing that I want to do is compile all of Sage with type inference > on and see how much breaks. (In general, we're pretty good a typing > variables, so I don't think it'll make a huge difference.) That certainly applies to C variables, but Python variables are a different thing here. > Have you tried compiling lxml with type inference enabled? Well, I thought I did, but at the time I wasn't aware that it wasn't on by default, and I didn't have the time to figure out why it still worked after pulling your changes. ;-) Doing so now gives me: Internal compiler error: SliceIndexNode.infer_type not implemented I guess there's always a 0.12.1 ... :-/ Stefan From saul at gfz-potsdam.de Tue Nov 24 20:59:28 2009 From: saul at gfz-potsdam.de (Joachim Saul) Date: Tue, 24 Nov 2009 20:59:28 +0100 Subject: [Cython] str arguments Message-ID: <4B0C3B20.4030007@gfz-potsdam.de> Hello all, in Pyrex I do cdef extern from "Python.h": ctypedef class __builtin__.str [object PyStringObject]: pass cdef extern void c_foo(char*) def foo(str bar=""): c_foo(bar) This is works nicely in Pyrex (as long as normal text strings are passed, *not* with binary data *containing* 0-bytes, of course). In Cython 0.12, however, this gives the error Cannot assign type 'str object' to 'str' probably because of the default value "". Is this intentional or am I overlooking something trivial? Python is version 2.6.2 Cheers, Joachim From stefan_ml at behnel.de Tue Nov 24 21:32:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Nov 2009 21:32:53 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0C3B20.4030007@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> Message-ID: <4B0C42F5.1080303@behnel.de> Joachim Saul, 24.11.2009 20:59: > cdef extern from "Python.h": > ctypedef class __builtin__.str [object PyStringObject]: > pass Remove the above and use the 'bytes' type in the rest of your code instead. 'str', 'unicode' and 'bytes' are builtin types in Cython. Stefan From robertwb at math.washington.edu Tue Nov 24 21:32:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Nov 2009 12:32:51 -0800 Subject: [Cython] Cython 0.12 released. In-Reply-To: <4B0C3D55.4030701@behnel.de> References: <4B0BC9A9.3070606@behnel.de> <359BD084-7CB8-4E45-8D36-02BB6047D0EC@math.washington.edu> <4B0C3D55.4030701@behnel.de> Message-ID: On Nov 24, 2009, at 12:08 PM, Stefan Behnel wrote: > > Robert Bradshaw, 24.11.2009 19:32: >> On Nov 24, 2009, at 3:55 AM, Stefan Behnel wrote: >> >>> Robert Bradshaw, 23.11.2009 20:33: >>>> == New features == >>>> - Type inference with the infer_types directive >>> Now that you mention it - why is there a directive for this? Can't >>> this be enabled by default? >> >> I think this came up before > > Yes, I recall something like that, too. > > >> first, it's a relatively new feature, so >> it'd be nice to get it out there a tested some, but more importantly >> it's such a large, backwards incompatible change that it didn't make >> sense to do it with all the other stuff going on on 0.12. There's a >> lot of code out there (much written by me, for example, when type >> inference wasn't even on the horizon) that uses the fact that >> assignment to an untyped variable is conversion to an object. > > Right, that's certainly a feature that people have their code depend > on. > Breaking it will break tons and tons of code, so it *is* good not to > make > it the default. > > How would you want this conversion to work in the future? With an > explicit > cast? Yep. One can always do cdef object a = x as well. Somehow, eventually, I'd like to make it the default, but I don't see a clear path from here to there. Having it explicit is nice as well because it makes me uneasy that the behavior of code could change as the type inference algorithm evolves. (This is one reason that I think the alignment Python and C types was important for division and "x or y") I could also see something like @cython.infer_types(algorithm=...) which would be useful both for experimentation and consistency. > How does the current implementation actually handle diverging > assignments? > As in > > if test: > items = [] > else: > items = () > > or: > > items = () > if test: > items = [] > > or: > > a = 5 > ... > a = "test" > ... In all of these cases, it would deduce object is the only type "wide" enough to contain both options. >> One thing that I want to do is compile all of Sage with type >> inference >> on and see how much breaks. (In general, we're pretty good a typing >> variables, so I don't think it'll make a huge difference.) > > That certainly applies to C variables, but Python variables are a > different > thing here. > > >> Have you tried compiling lxml with type inference enabled? > > Well, I thought I did, but at the time I wasn't aware that it wasn't > on by > default, and I didn't have the time to figure out why it still > worked after > pulling your changes. ;-) > > Doing so now gives me: > > Internal compiler error: SliceIndexNode.infer_type not implemented I guess that's one reason it's not enabled by default yet :-) > I guess there's always a 0.12.1 ... :-/ Yep. - Robert From dalcinl at gmail.com Tue Nov 24 22:18:37 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Nov 2009 18:18:37 -0300 Subject: [Cython] str arguments In-Reply-To: <4B0C42F5.1080303@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> Message-ID: On Tue, Nov 24, 2009 at 5:32 PM, Stefan Behnel wrote: > > Joachim Saul, 24.11.2009 20:59: >> cdef extern from "Python.h": >> ? ? ?ctypedef class __builtin__.str ?[object PyStringObject]: >> ? ? ? ? ?pass > > Remove the above and use the 'bytes' type in the rest of your code instead. > 'str', 'unicode' and 'bytes' are builtin types in Cython. > > Stefan > Anyway, any good reason for such code to fail in Cython? Should we support such declarations of builtin types? I think Cython should be able to compile as much Pyrex code as possible. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Nov 24 22:41:25 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Nov 2009 22:41:25 +0100 Subject: [Cython] str arguments In-Reply-To: References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> Message-ID: <4B0C5305.90200@behnel.de> Lisandro Dalcin, 24.11.2009 22:18: > On Tue, Nov 24, 2009 at 5:32 PM, Stefan Behnel wrote: >> Joachim Saul, 24.11.2009 20:59: >>> cdef extern from "Python.h": >>> ctypedef class __builtin__.str [object PyStringObject]: >>> pass >> Remove the above and use the 'bytes' type in the rest of your code instead. >> 'str', 'unicode' and 'bytes' are builtin types in Cython. > > Anyway, any good reason for such code to fail in Cython? Should we > support such declarations of builtin types? I think Cython should be > able to compile as much Pyrex code as possible. Well, supporting this would mean that we'd have to check the above declaration, including the header file name and all potentially declared type fields. Forbidding it is a lot easier, I think. Stefan From dalcinl at gmail.com Tue Nov 24 23:40:18 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Nov 2009 19:40:18 -0300 Subject: [Cython] str arguments In-Reply-To: <4B0C5305.90200@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> Message-ID: On Tue, Nov 24, 2009 at 6:41 PM, Stefan Behnel wrote: > > Lisandro Dalcin, 24.11.2009 22:18: >> On Tue, Nov 24, 2009 at 5:32 PM, Stefan Behnel wrote: >>> Joachim Saul, 24.11.2009 20:59: >>>> cdef extern from "Python.h": >>>> ? ? ?ctypedef class __builtin__.str ?[object PyStringObject]: >>>> ? ? ? ? ?pass >>> Remove the above and use the 'bytes' type in the rest of your code instead. >>> 'str', 'unicode' and 'bytes' are builtin types in Cython. >> >> Anyway, any good reason for such code to fail in Cython? Should we >> support such declarations of builtin types? I think Cython should be >> able to compile as much Pyrex code as possible. > > Well, supporting this would mean that we'd have to check the above > declaration, including the header file name and all potentially declared > type fields. > As usual, I was not clear enough. I was thinking about just ignoring the declaration, no check at all, and perhaps generate a warning. My whole point is on providing some sort of compatibility with Pyrex. Anyway, this is not a big deal for me. > > Forbidding it is a lot easier, I think. > OK. I understand your point. I'm not going to spend my time on this, nor I'm asking Cython-devs to do so. But If at some point a third party generates a patch proposing to maintain as much compatibility as possible between Cython and Pyrex, I think it should devise some consideration. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From saul at gfz-potsdam.de Tue Nov 24 23:47:11 2009 From: saul at gfz-potsdam.de (Joachim Saul) Date: Tue, 24 Nov 2009 23:47:11 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0C5305.90200@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> Message-ID: <4B0C626F.506@gfz-potsdam.de> Stefan Behnel schrieb: > Lisandro Dalcin, 24.11.2009 22:18: >> On Tue, Nov 24, 2009 at 5:32 PM, Stefan Behnel wrote: >>> Joachim Saul, 24.11.2009 20:59: >>>> cdef extern from "Python.h": >>>> ctypedef class __builtin__.str [object PyStringObject]: >>>> pass >>> Remove the above and use the 'bytes' type in the rest of your code instead. >>> 'str', 'unicode' and 'bytes' are builtin types in Cython. >> Anyway, any good reason for such code to fail in Cython? Should we >> support such declarations of builtin types? I think Cython should be >> able to compile as much Pyrex code as possible. > > Well, supporting this would mean that we'd have to check the above > declaration, including the header file name and all potentially declared > type fields. > > Forbidding it is a lot easier, I think. 1st of all thanks for the helpful replies. Indeed, Pyrex compatibility is a requirement for me. I want to use/test the Cython-generated code but need to be able to revert to Pyrex in case of problems. Anyway, what I have done is def foo(bytes bar=bytes("")): c_foo(bar) This compiles and works under both Cython and Pyrex, so for the sake of compatibility, this seems to be the way to go for the time being. But doesn't the explicit conversion seem pretty redundant/ugly? Cheers, Joachim From dalcinl at gmail.com Wed Nov 25 01:25:13 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Nov 2009 21:25:13 -0300 Subject: [Cython] str arguments In-Reply-To: <4B0C626F.506@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> Message-ID: On Tue, Nov 24, 2009 at 7:47 PM, Joachim Saul wrote: > > Indeed, Pyrex compatibility is a requirement for me. I want to use/test > the Cython-generated code but need to be able to revert to Pyrex in case > of problems. What kind of problems are you worried about? Could you elaborate on this? > Anyway, what I have done is > > def foo(bytes bar=bytes("")): > ? ? c_foo(bar) > Can you try this: ? def foo(bytes bar=b""): c_foo(bar) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Wed Nov 25 04:24:30 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Nov 2009 19:24:30 -0800 Subject: [Cython] str arguments In-Reply-To: References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> Message-ID: <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> On Nov 24, 2009, at 4:25 PM, Lisandro Dalcin wrote: > On Tue, Nov 24, 2009 at 7:47 PM, Joachim Saul > wrote: >> >> Indeed, Pyrex compatibility is a requirement for me. I want to use/ >> test >> the Cython-generated code but need to be able to revert to Pyrex in >> case >> of problems. > > What kind of problems are you worried about? Could you elaborate on > this? I'm curious too. >> Anyway, what I have done is >> >> def foo(bytes bar=bytes("")): >> c_foo(bar) >> > > Can you try this: ? > > def foo(bytes bar=b""): > c_foo(bar) I could be wrong, but I don't think Pyrex supports the b prefix on strings. BTW, it be easier to just do def foo(bar=""): c_foo(bar) - Robert From robertwb at math.washington.edu Wed Nov 25 04:27:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Nov 2009 19:27:33 -0800 Subject: [Cython] str arguments In-Reply-To: References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> Message-ID: <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> On Nov 24, 2009, at 1:18 PM, Lisandro Dalcin wrote: > On Tue, Nov 24, 2009 at 5:32 PM, Stefan Behnel > wrote: >> >> Joachim Saul, 24.11.2009 20:59: >>> cdef extern from "Python.h": >>> ctypedef class __builtin__.str [object PyStringObject]: >>> pass >> >> Remove the above and use the 'bytes' type in the rest of your code >> instead. >> 'str', 'unicode' and 'bytes' are builtin types in Cython. This isn't really addressing the issue, just sidestepping it by enforcing unicode safety. > Anyway, any good reason for such code to fail in Cython? Should we > support such declarations of builtin types? I think Cython should be > able to compile as much Pyrex code as possible. I agree, it shouldn't fail (though perhaps there should be a warning). The underlying issue is that it doesn't recognize that the user- declared type could be compatible with the builtin type. We should recognize this and allow it. - Robert From stefan_ml at behnel.de Wed Nov 25 08:34:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Nov 2009 08:34:32 +0100 Subject: [Cython] str arguments In-Reply-To: <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> Message-ID: <4B0CDE08.9050902@behnel.de> Robert Bradshaw, 25.11.2009 04:24: > On Nov 24, 2009, at 4:25 PM, Lisandro Dalcin wrote: >> On Tue, Nov 24, 2009 at 7:47 PM, Joachim Saul wrote: >>> Anyway, what I have done is >>> >>> def foo(bytes bar=bytes("")): >>> c_foo(bar) >>> >> Can you try this: ? >> >> def foo(bytes bar=b""): >> c_foo(bar) > > I could be wrong, but I don't think Pyrex supports the b prefix on > strings. > > BTW, it be easier to just do > > def foo(bar=""): > c_foo(bar) This is not portable to Py3. This may not be an issue, given that the OP aims for compatibility with Pyrex, which generates code that won't work in Py3 anyway. But I would assume that writing broken code is not the intention of the OP. Stefan From stefan_ml at behnel.de Wed Nov 25 08:39:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Nov 2009 08:39:53 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0C3B20.4030007@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> Message-ID: <4B0CDF49.7080706@behnel.de> Joachim Saul, 24.11.2009 20:59: > in Pyrex I do > > cdef extern from "Python.h": > ctypedef class __builtin__.str [object PyStringObject]: > pass > > cdef extern void c_foo(char*) > > def foo(str bar=""): > c_foo(bar) > > This is works nicely in Pyrex (as long as normal text strings are > passed, *not* with binary data *containing* 0-bytes, of course). You should read this: http://wiki.cython.org/FAQ#HowdoIpassaPythonstringparameterontoaClibrary.3F Stefan From stefan_ml at behnel.de Wed Nov 25 08:51:33 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Nov 2009 08:51:33 +0100 Subject: [Cython] str arguments In-Reply-To: <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> Message-ID: <4B0CE205.9080003@behnel.de> Robert Bradshaw, 25.11.2009 04:27: > On Nov 24, 2009, at 1:18 PM, Lisandro Dalcin wrote: >>> Joachim Saul, 24.11.2009 20:59: >>>> cdef extern from "Python.h": >>>> ctypedef class __builtin__.str [object PyStringObject]: >>>> pass >> >> Anyway, any good reason for such code to fail in Cython? Should we >> support such declarations of builtin types? I think Cython should be >> able to compile as much Pyrex code as possible. > > I agree, it shouldn't fail (though perhaps there should be a warning). > The underlying issue is that it doesn't recognize that the user- > declared type could be compatible with the builtin type. We should > recognize this and allow it. I disagree. This would either require us to check the declaration exactly (in which case there would be zero gain at all, as Cython would need to know the complete type declaration already), or it would allow users to declare all sorts of broken things on the type without checking. The latter means that there could be all sorts of surprises for users that see their code work or fail, depending on what they think they have declared. I think the right way to handle this would be to finish up subtyping support and type fields for builtin types (IIRC, that's what's still missing?), and then emit either an error on the above, or a clear warning that states that the senseless redefinition of a builtin type has been ignored (which IMHO is a very good argument for making it an error). Stefan From robertwb at math.washington.edu Wed Nov 25 09:43:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Nov 2009 00:43:21 -0800 Subject: [Cython] str arguments In-Reply-To: <4B0CE205.9080003@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> Message-ID: <9A2068B0-C917-48B4-BEFD-6BA36910BA5D@math.washington.edu> On Nov 24, 2009, at 11:51 PM, Stefan Behnel wrote: > Robert Bradshaw, 25.11.2009 04:27: >> On Nov 24, 2009, at 1:18 PM, Lisandro Dalcin wrote: >>>> Joachim Saul, 24.11.2009 20:59: >>>>> cdef extern from "Python.h": >>>>> ctypedef class __builtin__.str [object PyStringObject]: >>>>> pass >>> >>> Anyway, any good reason for such code to fail in Cython? Should we >>> support such declarations of builtin types? I think Cython should be >>> able to compile as much Pyrex code as possible. >> >> I agree, it shouldn't fail (though perhaps there should be a >> warning). >> The underlying issue is that it doesn't recognize that the user- >> declared type could be compatible with the builtin type. We should >> recognize this and allow it. > > I disagree. This would either require us to check the declaration > exactly > (in which case there would be zero gain at all, as Cython would need > to > know the complete type declaration already), or it would allow users > to > declare all sorts of broken things on the type without checking. The > latter > means that there could be all sorts of surprises for users that see > their > code work or fail, depending on what they think they have declared. I'm proposing something much simpler--if the name matches for a __builtin__ type, we allow assignment. If the extern declaration is erroneous, that will break things, but that's *always* the case with extern declarations. In general, a gcc error means check your extern declarations. (We should warn that a builtin type is being overridden.) > I think the right way to handle this would be to finish up subtyping > support and type fields for builtin types (IIRC, that's what's still > missing?), Yes, we should do this (though I'm doubtful the public and private APIs are all consistent from 2.3-3.x). > and then emit either an error on the above, or a clear warning > that states that the senseless redefinition of a builtin type has been > ignored (which IMHO is a very good argument for making it an error). The users type declaration would not be ignored, it would override the builtin one, just like cdef extern from *: cdef object range(object) would override the builtin range. - Robert From robertwb at math.washington.edu Wed Nov 25 10:25:27 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Nov 2009 01:25:27 -0800 Subject: [Cython] str arguments In-Reply-To: <9A2068B0-C917-48B4-BEFD-6BA36910BA5D@math.washington.edu> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> <9A2068B0-C917-48B4-BEFD-6BA36910BA5D@math.washington.edu> Message-ID: On Nov 25, 2009, at 12:43 AM, Robert Bradshaw wrote: > On Nov 24, 2009, at 11:51 PM, Stefan Behnel wrote: > >> Robert Bradshaw, 25.11.2009 04:27: >>> On Nov 24, 2009, at 1:18 PM, Lisandro Dalcin wrote: >>>>> Joachim Saul, 24.11.2009 20:59: >>>>>> cdef extern from "Python.h": >>>>>> ctypedef class __builtin__.str [object PyStringObject]: >>>>>> pass >>>> >>>> Anyway, any good reason for such code to fail in Cython? Should we >>>> support such declarations of builtin types? I think Cython should >>>> be >>>> able to compile as much Pyrex code as possible. >>> >>> I agree, it shouldn't fail (though perhaps there should be a >>> warning). >>> The underlying issue is that it doesn't recognize that the user- >>> declared type could be compatible with the builtin type. We should >>> recognize this and allow it. >> >> I disagree. This would either require us to check the declaration >> exactly >> (in which case there would be zero gain at all, as Cython would need >> to >> know the complete type declaration already), or it would allow users >> to >> declare all sorts of broken things on the type without checking. The >> latter >> means that there could be all sorts of surprises for users that see >> their >> code work or fail, depending on what they think they have declared. > > I'm proposing something much simpler--if the name matches for a > __builtin__ type, we allow assignment. If the extern declaration is > erroneous, that will break things, but that's *always* the case with > extern declarations. In general, a gcc error means check your extern > declarations. (We should warn that a builtin type is being > overridden.) > >> I think the right way to handle this would be to finish up subtyping >> support and type fields for builtin types (IIRC, that's what's still >> missing?), > > Yes, we should do this (though I'm doubtful the public and private > APIs are all consistent from 2.3-3.x). > >> and then emit either an error on the above, or a clear warning >> that states that the senseless redefinition of a builtin type has >> been >> ignored (which IMHO is a very good argument for making it an error). > > The users type declaration would not be ignored, it would override the > builtin one, just like > > cdef extern from *: > cdef object range(object) > > would override the builtin range. A clarification--I'm not trying to say that I want to make it easy for users to write broken code, but I'd rather the user be able to override the compiler, even if they're wrong. One of my pet peeves is software that says "I know better than you, so I'm ignoring you," especially if I know what I'm doing and am trying to use a corner case that it gets wrong. - Robert From saul at gfz-potsdam.de Wed Nov 25 10:55:03 2009 From: saul at gfz-potsdam.de (Joachim Saul) Date: Wed, 25 Nov 2009 10:55:03 +0100 Subject: [Cython] str arguments In-Reply-To: <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> Message-ID: <4B0CFEF7.1020604@gfz-potsdam.de> Robert Bradshaw: > On Nov 24, 2009, at 4:25 PM, Lisandro Dalcin wrote: > >> On Tue, Nov 24, 2009 at 7:47 PM, Joachim Saul >> wrote: >>> Indeed, Pyrex compatibility is a requirement for me. I want to use/ >>> test >>> the Cython-generated code but need to be able to revert to Pyrex in >>> case >>> of problems. >> What kind of problems are you worried about? Could you elaborate on >> this? > > I'm curious too. "in case of problems" - I haven't encountered any so far but last time when I tried a couple months ago I did run into issues that forced me to postpone the migration from Pyrex to Cython. Now with 0.12 things have improved quite considerably. Yet it makes me sleep better if during the transition I can translate the codes with both Pyrex and Cython, Pyrex being the fallback. >>> Anyway, what I have done is >>> >>> def foo(bytes bar=bytes("")): >>> c_foo(bar) >>> >> Can you try this: ? >> >> def foo(bytes bar=b""): >> c_foo(bar) > > I could be wrong, but I don't think Pyrex supports the b prefix on > strings. That's the point. > BTW, it be easier to just do > > def foo(bar=""): > c_foo(bar) True, but the resulting code then lacks the static typing I want here for consistency with the rest of the code. Plus, as pointed out by Stefan Behnel, it is not portable to Python 3. I would assume that in addition to None, any value that is acceptable as argument to bytes() should also be acceptable as a default value for a bytes-type parameter, following the Rule of Least Surprize?. This would consequently also allow code such as def foo(bytes bar=5): unless forbidden in Python 3. I am not yet very much into Cython and much less the internals, so this is just from a rather naive user's perspective, and I recognize that there may very well be points speaking against it. Cheers, Joachim From saul at gfz-potsdam.de Wed Nov 25 11:07:18 2009 From: saul at gfz-potsdam.de (Joachim Saul) Date: Wed, 25 Nov 2009 11:07:18 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0CFEF7.1020604@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> Message-ID: <4B0D01D6.9090805@gfz-potsdam.de> Joachim Saul: > I would assume that in addition to None, any value that is acceptable as > argument to bytes() should also be acceptable as a default value for a > bytes-type parameter, following the Rule of Least Surprize?. This would > consequently also allow code such as > > def foo(bytes bar=5): > > unless forbidden in Python 3. Read: Unless 'bytes(5)' etc. is forbidden in Python 3. ;) Cheers, Joachim From stefan_ml at behnel.de Wed Nov 25 13:04:55 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Nov 2009 13:04:55 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0D01D6.9090805@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <4B0D01D6.9090805@gfz-potsdam.de> Message-ID: <4B0D1D67.7030507@behnel.de> Joachim Saul, 25.11.2009 11:07: > Joachim Saul: >> I would assume that in addition to None, any value that is acceptable as >> argument to bytes() should also be acceptable as a default value for a >> bytes-type parameter, following the Rule of Least Surprize?. This would >> consequently also allow code such as >> >> def foo(bytes bar=5): If you want bytes(5), use bytes(5) as default value. The literal 5 is a number, not a bytes string, and thus not a valid value for a variable typed as bytes. >> unless forbidden in Python 3. > > Read: Unless 'bytes(5)' etc. is forbidden in Python 3. ;) I hope you are aware that bytes(5) is not str(5): Python 3.1.1 (r311:74480, Oct 22 2009, 19:34:26) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> bytes(5) b'\x00\x00\x00\x00\x00' >>> str(5) '5' That's one of the differences between the Py2.6 bytes/str type and the 'real' Py3.x bytes type, BTW. Stefan From dalcinl at gmail.com Wed Nov 25 17:23:27 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Nov 2009 13:23:27 -0300 Subject: [Cython] str arguments In-Reply-To: <4B0D1D67.7030507@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <4B0D01D6.9090805@gfz-potsdam.de> <4B0D1D67.7030507@behnel.de> Message-ID: On Wed, Nov 25, 2009 at 9:04 AM, Stefan Behnel wrote: > > Joachim Saul, 25.11.2009 11:07: >> Joachim Saul: >>> I would assume that in addition to None, any value that is acceptable as >>> argument to bytes() should also be acceptable as a default value for a >>> bytes-type parameter, following the Rule of Least Surprize?. This would >>> consequently also allow code such as >>> >>> def foo(bytes bar=5): > > If you want bytes(5), use bytes(5) as default value. The literal 5 is a > number, not a bytes string, and thus not a valid value for a variable typed > as bytes. > > >>> unless forbidden in Python 3. >> >> Read: Unless 'bytes(5)' etc. is forbidden in Python 3. ;) > > I hope you are aware that bytes(5) is not str(5): > > ? ?Python 3.1.1 (r311:74480, Oct 22 2009, 19:34:26) > ? ?[GCC 4.3.2] on linux2 > ? ?Type "help", "copyright", "credits" or "license" for more information. > ? ?>>> bytes(5) > ? ?b'\x00\x00\x00\x00\x00' > ? ?>>> str(5) > ? ?'5' > > That's one of the differences between the Py2.6 bytes/str type and the > 'real' Py3.x bytes type, BTW. > Just other thing we should workaround to alleviate users from the diferences? So bytes(n) would give at b"\0"*n in both Py2/Py3 ? note: only bytes(n), not for str(n) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Wed Nov 25 18:39:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Nov 2009 18:39:26 +0100 Subject: [Cython] str arguments In-Reply-To: References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <4B0D01D6.9090805@gfz-potsdam.de> <4B0D1D67.7030507@behnel.de> Message-ID: <4B0D6BCE.504@behnel.de> Lisandro Dalcin, 25.11.2009 17:23: > On Wed, Nov 25, 2009 at 9:04 AM, Stefan Behnel wrote: >> bytes(5) is not str(5): >> >> Python 3.1.1 (r311:74480, Oct 22 2009, 19:34:26) >> [GCC 4.3.2] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> bytes(5) >> b'\x00\x00\x00\x00\x00' >> >>> str(5) >> '5' >> >> That's one of the differences between the Py2.6 bytes/str type and the >> 'real' Py3.x bytes type. > > Just other thing we should workaround to alleviate users from the > diferences? So bytes(n) would give at b"\0"*n in both Py2/Py3 ? Would you really want to backport the 'bytes' type to Py2.x? There's much more to it than just the constructor. Think of indexing and iteration, for example, which return integers, not substrings. Stefan From dalcinl at gmail.com Wed Nov 25 19:35:50 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Nov 2009 15:35:50 -0300 Subject: [Cython] str arguments In-Reply-To: <4B0D6BCE.504@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <4B0D01D6.9090805@gfz-potsdam.de> <4B0D1D67.7030507@behnel.de> <4B0D6BCE.504@behnel.de> Message-ID: On Wed, Nov 25, 2009 at 2:39 PM, Stefan Behnel wrote: > > Lisandro Dalcin, 25.11.2009 17:23: >> On Wed, Nov 25, 2009 at 9:04 AM, Stefan Behnel wrote: >>> bytes(5) is not str(5): >>> >>> ? ?Python 3.1.1 (r311:74480, Oct 22 2009, 19:34:26) >>> ? ?[GCC 4.3.2] on linux2 >>> ? ?Type "help", "copyright", "credits" or "license" for more information. >>> ? ?>>> bytes(5) >>> ? ?b'\x00\x00\x00\x00\x00' >>> ? ?>>> str(5) >>> ? ?'5' >>> >>> That's one of the differences between the Py2.6 bytes/str type and the >>> 'real' Py3.x bytes type. >> >> Just other thing we should workaround to alleviate users from the >> diferences? So bytes(n) would give at b"\0"*n in both Py2/Py3 ? > > Would you really want to backport the 'bytes' type to Py2.x? There's much > more to it than just the constructor. Think of indexing and iteration, for > example, which return integers, not substrings. > Forget about it, my brain is not working well today. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Wed Nov 25 19:54:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Nov 2009 10:54:20 -0800 Subject: [Cython] str arguments In-Reply-To: <4B0CFEF7.1020604@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> Message-ID: <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> On Nov 25, 2009, at 1:55 AM, Joachim Saul wrote: > Robert Bradshaw: >> On Nov 24, 2009, at 4:25 PM, Lisandro Dalcin wrote: >> >>> On Tue, Nov 24, 2009 at 7:47 PM, Joachim Saul >>> wrote: >>>> Indeed, Pyrex compatibility is a requirement for me. I want to use/ >>>> test >>>> the Cython-generated code but need to be able to revert to Pyrex in >>>> case >>>> of problems. >>> What kind of problems are you worried about? Could you elaborate on >>> this? >> >> I'm curious too. > > "in case of problems" - I haven't encountered any so far but last time > when I tried a couple months ago I did run into issues that forced > me to > postpone the migration from Pyrex to Cython. Now with 0.12 things have > improved quite considerably. Yet it makes me sleep better if during > the > transition I can translate the codes with both Pyrex and Cython, Pyrex > being the fallback. Sure, that makes sense. We certainly try to target multiple C compilers. >>>> Anyway, what I have done is >>>> >>>> def foo(bytes bar=bytes("")): >>>> c_foo(bar) >>>> >>> Can you try this: ? >>> >>> def foo(bytes bar=b""): >>> c_foo(bar) >> >> I could be wrong, but I don't think Pyrex supports the b prefix on >> strings. > > That's the point. > >> BTW, it be easier to just do >> >> def foo(bar=""): >> c_foo(bar) > > True, but the resulting code then lacks the static typing I want here > for consistency with the rest of the code. Plus, as pointed out by > Stefan Behnel, it is not portable to Python 3. Nor is def foo(bytes bar=b""): ... unless you actually expect users to be passing bytes objects (It will reject unicode objects, which string literals will be in Py3). Suggesting that people use the bytes type instead of the string type is probably not a good idea, as the bytes type is so different in Py3 and not usually what people want. > I would assume that in addition to None, any value that is > acceptable as > argument to bytes() should also be acceptable as a default value for a > bytes-type parameter, following the Rule of Least Surprize?. That's not the case, with Pyrex or with Cython. Actually, bytes("") is illegal in Python 3 (because you have to explicitly give an encoding). I think this is similar to how one can't do range("5") even though range expects and int and int has a constructor from a string. > This would consequently also allow code such as > > def foo(bytes bar=5): > > unless forbidden in Python 3. > > I am not yet very much into Cython and much less the internals, so > this > is just from a rather naive user's perspective, and I recognize that > there may very well be points speaking against it. There are two options for what you want to do here: (1) Target Python 2 only (Pyrex forces this) and assume all your strings are ASCII only or already in the correct encoding for the C library calls. Use str everywhere, just as you have been doing (though without pre-declaring it as an extern definition, which I don't think is necessary in Pyrex either). You can still use str in Python 3, it's just that str -> char* will not happen automatically. (2) Explicitly and manually handle all encoding/decoding every time you want to pass from a Python string to a C string, as in http://wiki.cython.org/FAQ#HowdoIpassaPythonstringparameterontoaClibrary.3F In either case, we should be handling __builtin__ redeclarations more gracefully. - Robert From greg.ewing at canterbury.ac.nz Wed Nov 25 21:04:00 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 26 Nov 2009 09:04:00 +1300 Subject: [Cython] str arguments In-Reply-To: <4B0CE205.9080003@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> Message-ID: <4B0D8DB0.3040208@canterbury.ac.nz> Stefan Behnel wrote: > Robert Bradshaw, 25.11.2009 04:27: > >>The underlying issue is that it doesn't recognize that the user- >>declared type could be compatible with the builtin type. We should >>recognize this and allow it. > > I disagree. This would either require us to check the declaration exactly > (in which case there would be zero gain at all, as Cython would need to > know the complete type declaration already), or it would allow users to > declare all sorts of broken things on the type without checking. The code in question works in Pyrex because the type of the default argument is checked dynamically at run time, just like in any other assignment. Cython appears to be making an excessively static determination of the type here. Having full builtin declarations for all the standard types would be useful, but as long as external type definitions are allowed at all, you can't eliminate the possibility that the user will declare a type in a way that doesn't agree with reality. At some point you just have to trust the programmer, lest you cripple them unnecessarily. -- Greg From stefan_ml at behnel.de Wed Nov 25 21:19:27 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Nov 2009 21:19:27 +0100 Subject: [Cython] str arguments In-Reply-To: <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> Message-ID: <4B0D914F.2080408@behnel.de> Robert Bradshaw, 25.11.2009 19:54: > You can still use str in Python 3, it's > just that str -> char* will not happen automatically. Slight clarification: "str -> char* will not happen automatically" is also true for Py2. 'bytes' is the only type that automatically coerces from and to char*. Stefan From stefan_ml at behnel.de Wed Nov 25 22:38:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Nov 2009 22:38:53 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0D8DB0.3040208@canterbury.ac.nz> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> <4B0D8DB0.3040208@canterbury.ac.nz> Message-ID: <4B0DA3ED.5060903@behnel.de> Hi Greg, Greg Ewing, 25.11.2009 21:04: > The code in question works in Pyrex because the type of the > default argument is checked dynamically at run time, just > like in any other assignment. Cython appears to be making an > excessively static determination of the type here. Ah, right, that's due to the str/bytes/unicode separation in 0.12. Cython now generates errors when you try to assign one to the other. I think that strikes here, too, as the OP's code was trying to assign one of them to an unrelated extension type. This makes the behaviour appear more like a side-effect than a well designed feature. > Having full builtin declarations for all the standard types > would be useful, but as long as external type definitions are > allowed at all, you can't eliminate the possibility that the > user will declare a type in a way that doesn't agree with > reality. At some point you just have to trust the programmer, > lest you cripple them unnecessarily. This is sort of a grey area. Cython aims to optimise access to builtin types, so I find it a good thing to generate errors when users try to redefine them. Obviously, this only makes sense once Cython actually makes the redeclarations completely redundant, which we haven't achieved yet. We currently generate warnings when users redefine builtin types. I think that's ok for the time being. Stefan From robertwb at math.washington.edu Wed Nov 25 23:35:03 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Nov 2009 14:35:03 -0800 Subject: [Cython] str arguments In-Reply-To: <4B0DA3ED.5060903@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> <4B0D8DB0.3040208@canterbury.ac.nz> <4B0DA3ED.5060903@behnel.de> Message-ID: On Nov 25, 2009, at 1:38 PM, Stefan Behnel wrote: > Hi Greg, > > Greg Ewing, 25.11.2009 21:04: >> The code in question works in Pyrex because the type of the >> default argument is checked dynamically at run time, just >> like in any other assignment. Cython appears to be making an >> excessively static determination of the type here. > > Ah, right, that's due to the str/bytes/unicode separation in 0.12. > Cython > now generates errors when you try to assign one to the other. I > think that > strikes here, too, as the OP's code was trying to assign one of them > to an > unrelated extension type. > > This makes the behaviour appear more like a side-effect than a well > designed feature. > > >> Having full builtin declarations for all the standard types >> would be useful, but as long as external type definitions are >> allowed at all, you can't eliminate the possibility that the >> user will declare a type in a way that doesn't agree with >> reality. At some point you just have to trust the programmer, >> lest you cripple them unnecessarily. +1 > This is sort of a grey area. Cython aims to optimise access to builtin > types, so I find it a good thing to generate errors when users try to > redefine them. Obviously, this only makes sense once Cython actually > makes > the redeclarations completely redundant, which we haven't achieved > yet. > > We currently generate warnings when users redefine builtin types. I > think > that's ok for the time being. We should allow assignments to these re-defined types too, and let the user use an undocumented feature or shoot themselves in the foot if they want. - Robert From greg.ewing at canterbury.ac.nz Wed Nov 25 23:39:52 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 26 Nov 2009 11:39:52 +1300 Subject: [Cython] str arguments In-Reply-To: <4B0DA3ED.5060903@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> <4B0D8DB0.3040208@canterbury.ac.nz> <4B0DA3ED.5060903@behnel.de> Message-ID: <4B0DB238.2080401@canterbury.ac.nz> Stefan Behnel wrote: > Ah, right, that's due to the str/bytes/unicode separation in 0.12. Ah, I see. It's probably behaving as designed, then, because the user hasn't said that his str class is based on anything that would be compatible with a builtin str. I think the difference is that Cython has an internal notion of the builtin str class, whereas Pyrex doesn't -- to Pyrex, a Python string is just a generic Python object. If Cython does have a str type internally, it needs to be exposed so that users can use it or subclass from it without confusing the type checking. -- Greg From robertwb at math.washington.edu Wed Nov 25 23:37:24 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Nov 2009 14:37:24 -0800 Subject: [Cython] str arguments In-Reply-To: <4B0D914F.2080408@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> Message-ID: On Nov 25, 2009, at 12:19 PM, Stefan Behnel wrote: > > Robert Bradshaw, 25.11.2009 19:54: >> You can still use str in Python 3, it's >> just that str -> char* will not happen automatically. > > Slight clarification: "str -> char* will not happen automatically" > is also > true for Py2. 'bytes' is the only type that automatically coerces > from and > to char*. So, short of a bunch of encoding code (can it be written Pyrex and Cython compatible) there's no way to support this? I'd rather this is allowed with a (runtime) TypeError in Py3 and possibly a compile time warning than disallowed altogether. - Robert From stefan_ml at behnel.de Thu Nov 26 09:05:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Nov 2009 09:05:15 +0100 Subject: [Cython] str arguments In-Reply-To: References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> Message-ID: <4B0E36BB.10504@behnel.de> Robert Bradshaw, 25.11.2009 23:37: > On Nov 25, 2009, at 12:19 PM, Stefan Behnel wrote: >> Robert Bradshaw, 25.11.2009 19:54: >>> You can still use str in Python 3, it's >>> just that str -> char* will not happen automatically. >> Slight clarification: "str -> char* will not happen automatically" >> is also >> true for Py2. 'bytes' is the only type that automatically coerces >> from and to char*. > > So, short of a bunch of encoding code (can it be written Pyrex and > Cython compatible) there's no way to support this? It's easy to write encoding code that works in Python, Pyrex and Cython, and if you encode your strings, it will also easily work in Py2.x and Py3. If you don't, then your code will not work well in both Py3 *and* Py2. > I'd rather this is allowed with a (runtime) TypeError in Py3 and > possibly a compile time warning than disallowed altogether. Well, I prefer disallowing it all together, because it helps users in writing correct code, even if they don't care about Py3 compatibility for now. Don't forget that typing variables as "str" doesn't magically solve all problems. Not in all cases in Py3, but certainly not in Py2. It is important to /not/ type your variables as long as you don't need to, especially for function parameters that may receive both str and unicode in Py2. Then, if you mean "bytes" in your code, write "bytes" and if you mean "unicode", write "unicode" - but don't write "str" just because you think it's text. "str" is just there so users can work with Python text strings in both Py2 and Py3 without too much hassle (so it's actually for people who care about Py3 compatibility), although the hassle is big enough already in Py2 as it requires users to deal with both "str" and "unicode" for essentially the same thing. This is much easier in Py3. Whenever you want to pass strings into C (which is the only case where this restriction /really/ matters IMHO), you have to care about the content of the string anyway, so "str" will simply not work as it is underdefined from a C level perspective. Ignoring the semantics of data will necessarily make you write fragile code. Supporting that should not be a design goal of the Cython language. I'm actually getting tired of rediscussing these things over and over again. Please come up with a valid use case where a currently unsupported automatic conversion between different string types clearly helps in writing simple and correct code. Stefan From stefan_ml at behnel.de Thu Nov 26 10:12:07 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Nov 2009 10:12:07 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0DB238.2080401@canterbury.ac.nz> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> <4B0D8DB0.3040208@canterbury.ac.nz> <4B0DA3ED.5060903@behnel.de> <4B0DB238.2080401@canterbury.ac.nz> Message-ID: <4B0E4667.3070109@behnel.de> Greg Ewing, 25.11.2009 23:39: > Stefan Behnel wrote: >> Ah, right, that's due to the str/bytes/unicode separation in 0.12. > > Ah, I see. It's probably behaving as designed, then, > because the user hasn't said that his str class is > based on anything that would be compatible with a > builtin str. > > I think the difference is that Cython has an internal > notion of the builtin str class, whereas Pyrex > doesn't -- to Pyrex, a Python string is just a > generic Python object. Yes, I know. Cython as departed from this a little lately, especially in 0.12. I'm a big fan of having the compiler fail early if it detects something that would lead to an error at runtime. Coercion between different string types is an area where I think this is particularly helpful, but I think the rules are a little too strict. Unknown/external/underdefined types must always be allowed in assignments as the compiler can't know for sure if they will work or not. The only case where assignments between known distinct types work is when the value is None. But even then, I don't see a use case for allowing this: cdef list l = None cdef str s = l IMHO, if a user explicitly types a variable (which is absolutely optional for Python types), he/she must be aware that this restricts its value set and consequently its use. Disallowing the above (for cases where automatic coercion is not clearly obvious) is something I would want Cython to do at compile time, not runtime. > If Cython does have a str type internally, it > needs to be exposed so that users can use it or > subclass from it without confusing the type > checking. Sure. Stefan From robertwb at math.washington.edu Thu Nov 26 11:03:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 26 Nov 2009 02:03:50 -0800 Subject: [Cython] str arguments In-Reply-To: <4B0E36BB.10504@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> <4B0E36BB.10504@behnel.de> Message-ID: On Nov 26, 2009, at 12:05 AM, Stefan Behnel wrote: > Robert Bradshaw, 25.11.2009 23:37: >> On Nov 25, 2009, at 12:19 PM, Stefan Behnel wrote: >>> Robert Bradshaw, 25.11.2009 19:54: >>>> You can still use str in Python 3, it's >>>> just that str -> char* will not happen automatically. >>> Slight clarification: "str -> char* will not happen automatically" >>> is also >>> true for Py2. 'bytes' is the only type that automatically coerces >>> from and to char*. >> >> So, short of a bunch of encoding code (can it be written Pyrex and >> Cython compatible) there's no way to support this? > > It's easy to write encoding code that works in Python, Pyrex and > Cython, > and if you encode your strings, it will also easily work in Py2.x > and Py3. > If you don't, then your code will not work well in both Py3 *and* Py2. > > >> I'd rather this is allowed with a (runtime) TypeError in Py3 and >> possibly a compile time warning than disallowed altogether. > > Well, I prefer disallowing it all together, because it helps users in > writing correct code, even if they don't care about Py3 > compatibility for now. I think this boils down to a philosophical difference. I think we should encourage users to write correct code, but not require it. That's why there's warnings vs. errors. We should only raise errors for things that could never work. (Well, I'm OK with a compile time error for something like def foo(list L): cdef tuple t = L # might succeed if L is None but that should almost never work.) Should we disallow the use of xrange, file, or reduce because they are no longer available in Python 3? (I concede that their use in Py2 is more correct than sloppy handling of strings.) As another example, if someone declares cdef extern from "Python.h": object PyDict_New(int foo) than I don't think it's our job to notice and raise an error, we should let them use the (incorrect) definition and the C compiler will choke (or, maybe not, if they're working on a hacked or pre-release version of Python). > Don't forget that typing variables as "str" doesn't magically solve > all > problems. Not in all cases in Py3, but certainly not in Py2. It is > important to /not/ type your variables as long as you don't need to, > especially for function parameters that may receive both str and > unicode in > Py2. Then, if you mean "bytes" in your code, write "bytes" and if > you mean > "unicode", write "unicode" - but don't write "str" just because you > think > it's text. "str" is just there so users can work with Python text > strings > in both Py2 and Py3 without too much hassle (so it's actually for > people > who care about Py3 compatibility), although the hassle is big enough > already in Py2 as it requires users to deal with both "str" and > "unicode" > for essentially the same thing. This is much easier in Py3. > > Whenever you want to pass strings into C (which is the only case > where this > restriction /really/ matters IMHO), you have to care about the > content of > the string anyway, so "str" will simply not work as it is > underdefined from > a C level perspective. Ignoring the semantics of data will > necessarily make > you write fragile code. Supporting that should not be a design goal > of the > Cython language. > > I'm actually getting tired of rediscussing these things over and over > again. Me too--this is why for my first several posts I focused on the extern compatibility issue. > Please come up with a valid use case where a currently unsupported > automatic conversion between different string types clearly helps in > writing simple and correct code. Lets look at the example that started this thread: cdef extern void c_foo(char*) def foo(str bar=""): c_foo(bar) Now I understand that your position that this is incorrect, broken, and fragile code just waiting to blow up in the users face, but from a pragmatic point of view it's working code that's part of a larger project and compiles and runs fine with Python 2 and Pyrex. It would compile and work fine with Cython as well except we have a check for just this case to disallow it [1], trying to force the user to write correct code. So what are the options? def foo(bar=""): c_foo(bar) is probably the easiest option, but is still incorrect in your book, and lacks the static typing desired to fit in with the rest of the code. (Personally, I would discourage typing bar, but perhaps foo is more complicated than this two-liner, and we shouldn't prohibit it.) The other proposed option def foo(bytes bar=b""): c_foo(bar) doesn't maintain compatibility with Pyrex, and though I have no idea what foo actually does, I would guess that this is not the API one would want to actually use in Py3. The most correct option is def foo(bar=""): [manual type checking and decoding on bar, perhaps via a helper function] c_foo(bar_c) which is the most correct from a unicode standpoint, but may require substantial changes to an existing codebase (I know, it could be called bugfixing), is certainly less user-friendly, and could introduce a fair amount of overhead as well. My modest proposal is to allow bytes <-> str, char* <-> str, and str <- > unicode *with* typechecking. (Alternatively, we could omit an explicit error at C compile time if conversion is attempted between incompatible types. I'm not sure whether that would be better.) - Robert [1] I'm actually surprised at cdef list L cdef str s cdef char* ss0 = L # this will never work, but is accepted cdef char* ss1 = s # this could succeed in Py2, but is prohibited for being incorrect by not explicitly encoding From stefan_ml at behnel.de Thu Nov 26 13:48:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Nov 2009 13:48:00 +0100 Subject: [Cython] str arguments In-Reply-To: References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> <4B0E36BB.10504@behnel.de> Message-ID: <4B0E7900.9030108@behnel.de> Hi Robert, Robert Bradshaw, 26.11.2009 11:03: > I think this boils down to a philosophical difference. I think we > should encourage users to write correct code, but not require it. > That's why there's warnings vs. errors. We should only raise errors > for things that could never work. I'm fine with that in general. However, for the specific case of strings, I must note that Py3 is as much a reality as Py2, and that the semantics of strings in Cython dictate that certain things will not work in certain Python runtimes. I admit that a warning allows users to actively say "I don't care, my code will never run in a different environment anyway". So I guess there is a case for making it a warning. >> Please come up with a valid use case where a currently unsupported >> automatic conversion between different string types clearly helps in >> writing simple and correct code. > > Lets look at the example that started this thread: > > cdef extern void c_foo(char*) > > def foo(str bar=""): > c_foo(bar) > > Now I understand that your position that this is incorrect, broken, > and fragile code just waiting to blow up in the users face, but from a > pragmatic point of view it's working code that's part of a larger > project and compiles and runs fine with Python 2 and Pyrex. It would > compile and work fine with Cython as well except we have a check for > just this case to disallow it [1], trying to force the user to write > correct code. My position is that "str" is not the right type here. It's just like using "float" where "long" would be appropriate. If you want "bytes", write "bytes". That will also give you an appropriate runtime exception in both Py2 and Py3 when you pass a unicode string. No harm done to anyone. If you use 'str', the exception you get in Py3 later in the code will be much less beautiful. > So what are the options? > > def foo(bar=""): > c_foo(bar) > > is probably the easiest option, but is still incorrect in your book, > and lacks the static typing desired to fit in with the rest of the > code. It's incorrect, sure. However, the difference is that 'bar' is not explicitly typed, so it's a plain Python object. The contract for the Python object type is that it can accept any Python object. And coercing any Python object to char* is allowed because it explicitly moves the type check to runtime. So it may surprise you, but I'm actually fine with not doing anything about the above code, and just accepting it as is, without any warning. It's an entirely different beast to type 'bar', thus restricting it to a specific type of Python object. This type may or may not be able to coerce to char*, but Cython can determine that at compile time if it knows the type. Typing is not only for speed, it's also to say: dear compiler, please check that this variable is only used with the type I give it and tell me if I get it wrong anywhere in my code. Note that you can also do this (warning, unhelpful code ahead): def foo(str bar="") cdef bytes b if isinstance(bar, unicode): b = bar.encode('UTF-8') else: b = bar return b So it's not that these assignments are impossible. They are just not allowed with *implicit* coercions. > The other proposed option > > def foo(bytes bar=b""): > c_foo(bar) > > doesn't maintain compatibility with Pyrex Note that the following works in both Cython and Pyrex: cdef bytes b = "abcdefg" This is actually specified in CEP 108, and it's supported for good reason. So the above could simply be written as def foo(bytes bar=""): c_foo(bar) in a portable way. (If this doesn't work in Cython, I would consider that a bug w.r.t. the CEP). > and though I have no idea > what foo actually does, I would guess that this is not the API one > would want to actually use in Py3. At least the OP mentioned the intention to pass "text", so I'd second that (also in Py2, BTW). > The most correct option is > > def foo(bar=""): > [manual type checking and decoding on bar, perhaps via a helper > function] > c_foo(bar_c) > > which is the most correct from a unicode standpoint, but may require > substantial changes to an existing codebase (I know, it could be > called bugfixing), is certainly less user-friendly, and could > introduce a fair amount of overhead as well. Ok, I understand that you understand that this is required anyway, both in Py2 and Py3, so there's not much to add here. But what about actually making this specific case easier? Not caring too much about syntax for now, we could support something like def foo(bytes[encoding='UTF-8'] bar=""): c_foo(bar_c) and, respectively: def foo(unicode[encoding='UTF-8'] bar=u""): assert isinstance(bar, unicode) def foo(str[encoding='UTF-8'] bar=""): assert isinstance(bar, str) and let Cython generate the argument type checking and conversion code internally, based on what kind of string is passed at runtime. Even 1D char buffers could be supported as argument here... OTOH, this is really only required in Py2.x, and Py3 is the pretty immediate future. Adding syntactic sugar for a soon-to-be-dead platform may just drop legacy stuff into the Cython language. But it would certainly solve the above problem. > My modest proposal ;) > is to allow bytes <-> str, char* <-> str, and str <- > > unicode *with* typechecking. So, you would basically allow this: cdef char* s = "abcdef" cdef bytes bs = s cdef str pys = bs cdef unicode us = pys Not beautiful, if you ask me, and I still haven't seen a use case that supports this. What's so bad about an explicit cast if you really think you need such an assignment? You can even use a safe "" cast to tell the compiler that you are not sure if this works. > (Alternatively, we could omit an > explicit error at C compile time if conversion is attempted between > incompatible types. I'm not sure whether that would be better.) I thought about that, too, but I doubt that it would be more helpful than a runtime error. > I'm actually surprised at > > cdef list L > cdef str s > cdef char* ss0 = L # this will never work, but is accepted > cdef char* ss1 = s # this could succeed in Py2, but is prohibited for > being incorrect by not explicitly encoding As I said in my answer to Greg, I would appreciate it if Cython considered the list assignment an error. But as you see from the current behaviour for external types, it's not trivial to set up the rules correctly. Stefan From dagss at student.matnat.uio.no Thu Nov 26 15:49:13 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 26 Nov 2009 15:49:13 +0100 Subject: [Cython] Cython 0.12 released. In-Reply-To: <359BD084-7CB8-4E45-8D36-02BB6047D0EC@math.washington.edu> References: <4B0BC9A9.3070606@behnel.de> <359BD084-7CB8-4E45-8D36-02BB6047D0EC@math.washington.edu> Message-ID: <4B0E9569.7010908@student.matnat.uio.no> Robert Bradshaw wrote: > On Nov 24, 2009, at 3:55 AM, Stefan Behnel wrote: > >> Robert Bradshaw, 23.11.2009 20:33: >>> == New features == >>> - Type inference with the infer_types directive >> Now that you mention it - why is there a directive for this? Can't >> this be >> enabled by default? > > I think this came up before--first, it's a relatively new feature, so > it'd be nice to get it out there a tested some, but more importantly > it's such a large, backwards incompatible change that it didn't make > sense to do it with all the other stuff going on on 0.12. There's a > lot of code out there (much written by me, for example, when type > inference wasn't even on the horizon) that uses the fact that > assignment to an untyped variable is conversion to an object. This is the first time I hear of us having type inference at all. Did I miss a discussion somewhere? Could someone point me to the right changeset so that I can read up on it? (I guess this is something I should know how works even if I don't work on it so that I don't break it...) -- Dag Sverre From stefan_ml at behnel.de Thu Nov 26 16:04:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 26 Nov 2009 16:04:52 +0100 Subject: [Cython] Cython 0.12 released. In-Reply-To: <4B0E9569.7010908@student.matnat.uio.no> References: <4B0BC9A9.3070606@behnel.de> <359BD084-7CB8-4E45-8D36-02BB6047D0EC@math.washington.edu> <4B0E9569.7010908@student.matnat.uio.no> Message-ID: <4B0E9914.9090200@behnel.de> Dag Sverre Seljebotn, 26.11.2009 15:49: > This is the first time I hear of us having type inference at all. Did I > miss a discussion somewhere? Could someone point me to the right > changeset so that I can read up on it? (I guess this is something I > should know how works even if I don't work on it so that I don't break > it...) The relevant commits start here IIRC: http://hg.cython.org/cython-devel/rev/64f4242db9f0 The basic idea is that assignments 'taint' their targets with the assigned type. As Robert writes, this breaks all sorts of code as it drops the feature that an assignment to an untyped variable converts the value to a Python object. It's currently unclear how to deal with this in the future. It might be nice to just enable this feature for Python types by default, so that optimisations like list.append() strike automatically for lists created inside of the function, or to make extension type attributes and functions directly available after instantiation without the need to type a variable. I think that would break the least amount of existing code, while providing the largest benefit. Stefan From saul at gfz-potsdam.de Thu Nov 26 22:53:11 2009 From: saul at gfz-potsdam.de (Joachim Saul) Date: Thu, 26 Nov 2009 22:53:11 +0100 Subject: [Cython] str arguments In-Reply-To: References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> <4B0E36BB.10504@behnel.de> Message-ID: <4B0EF8C7.3070306@gfz-potsdam.de> Robert Bradshaw [26.11.2009 11:03]: > (Well, I'm OK with a compile time > error for something like > > def foo(list L): > cdef tuple t = L # might succeed if L is None > > but that should almost never work.) Should we disallow the use of > xrange, file, or reduce because they are no longer available in Python > 3? (I concede that their use in Py2 is more correct than sloppy > handling of strings.) As another example, if someone declares > > cdef extern from "Python.h": > object PyDict_New(int foo) > > than I don't think it's our job to notice and raise an error, we > should let them use the (incorrect) definition and the C compiler will > choke (or, maybe not, if they're working on a hacked or pre-release > version of Python). > Exactly. It is not Cython's job to enforce correct (Cython) code. The user (of the Cython-created modules) will normally not be aware of what goes on inside. To me, nevertheless, it would make a lot of sense to allow code like def foo(list L): cdef array arr = L given that at Python level arr = array(L) is likely to work (depending on the *values* of L of course). > Lets look at the example that started this thread: > cdef extern void c_foo(char*) > > def foo(str bar=""): > c_foo(bar) > > Now I understand that your position that this is incorrect, broken, > and fragile code just waiting to blow up in the users face, Under what conditions is this incorrect or fragile? Certainly not because of the typing or the default vaule. Maybe I need to add that the argument to c_foo() is interpreted as 0-terminated character string. This will obviously give incorrect results if bar itself contains 0-characters, which can be ruled out unless such strings are created deliberately. Usually it will be simple ASCII character strings like "ABC" or the like. In this particular case it could be sensor ID's. I recognize that the c_foo() interface is not the most robust, but in the context in which this code is used, the value of bar can be trusted. > but from a > pragmatic point of view it's working code that's part of a larger > project and compiles and runs fine with Python 2 and Pyrex. It would > compile and work fine with Cython as well except we have a check for > just this case to disallow it [1], trying to force the user to write > correct code. So what are the options? > > def foo(bar=""): > c_foo(bar) > > is probably the easiest option, but is still incorrect in your book, > and lacks the static typing desired to fit in with the rest of the > code. (Personally, I would discourage typing bar, but perhaps foo is > more complicated than this two-liner, and we shouldn't prohibit it.) > The other proposed option > > def foo(bytes bar=b""): > c_foo(bar) > > doesn't maintain compatibility with Pyrex, and though I have no idea > what foo actually does, I would guess that this is not the API one > would want to actually use in Py3. Indeed not. I must admit I haven't used Python 3 yet and am not very familiar so far with the new unicode str's, bytes-type etc. What I certainly don't want is some sensor ID "ABC" encoded as bytes. code=b"ABC" foo(code) print(code[1]) 66 Event though formally correct, this is not what the user (at Python level) would consider acceptable. The user wants and shall be able to code="ABC" foo(code) print(code[1]) 'B' so 'code' must be a 'str', there is no alternative to this even though in Python 3 'str' is differently encoded. As a consequence, I need to define a C function that takes care of the de/encoding from/to C strings depending on whether this is Python 2 or 3. I am pretty sure doing this at C level using #ifdef's for different Python versions should be straightforward. > The most correct option is > > def foo(bar=""): > [manual type checking and decoding on bar, perhaps via a helper > function] > Manual type checking only if I expect bar to be anything other than a 'str'. Which I don't :) > c_foo(bar_c) > I agree on the decoding. > which is the most correct from a unicode standpoint, but may require > substantial changes to an existing codebase (I know, it could be > called bugfixing), is certainly less user-friendly, and could > introduce a fair amount of overhead as well. > You probably mean less developer-friendly ;) The changes to the existing codebase would probably only involve the additional decoding step. The overhead is what I'm a bit concerned about. See, often one wants to simply compare such codes, or perhaps sorting a large list of objects with such strings as attributes; the overhead can be expected to be quite substantial in such cases. Cheers, Joachim From saul at gfz-potsdam.de Thu Nov 26 23:40:07 2009 From: saul at gfz-potsdam.de (Joachim Saul) Date: Thu, 26 Nov 2009 23:40:07 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0E7900.9030108@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> <4B0E36BB.10504@behnel.de> <4B0E7900.9030108@behnel.de> Message-ID: <4B0F03C7.2020105@gfz-potsdam.de> Stefan Behnel [26.11.2009 13:48]: >> Lets look at the example that started this thread: >> >> cdef extern void c_foo(char*) >> >> def foo(str bar=""): >> c_foo(bar) >> >> Now I understand that your position that this is incorrect, broken, >> and fragile code just waiting to blow up in the users face, but from a >> pragmatic point of view it's working code that's part of a larger >> project and compiles and runs fine with Python 2 and Pyrex. It would >> compile and work fine with Cython as well except we have a check for >> just this case to disallow it [1], trying to force the user to write >> correct code. >> > > My position is that "str" is not the right type here. It's just like using > "float" where "long" would be appropriate. > Well, from the original posting it is stated that in Pyrex the above works nicely with text strings, which is what it is used for. > If you want "bytes", write "bytes". But I don't actually want bytes. Well, it's all bytes of course, but I want text strings. Something like Python-2.* 'str' or C++ std::string. The Python-3 semantics of 'bytes', in my opinion, deviates pretty much from what I consider a text string. See my previous email: code=b"ABC" print(code[1]) 66 I am pretty sure I don't want "text" strings like that. I rather have 'str' everywhere and be forced to convert them before passing them to C/C++ code="ABC" print(code[1]) 'B' is what I want and what I get in both Python 2 and 3 by using 'str'. And in both Python 2 and 3 I get type("") (well, almost). So I would expect that in Cython def foo(str bar=""): to be portable (which it is) and from thereon I need some conversion to get an ASCII representation of bar (provided that conversion is possible). We may have started talking about different things, though. In my original email I made the mistake of redefining 'str', which in Cython is builtin, and this caused an unexpected collision of redefined str and "". In pure Cython as well as in Pyrex with proper definition of str def foo(str bar=""): does and will hopefully always work. So the issue here is really reduced to getting pure-ASCII strings in out of 'str' objects, which as a matter of fact depends on the encoding of str that is different in Python 2 and 3. But that should be doable. Cheers, Joachim From greg.ewing at canterbury.ac.nz Fri Nov 27 00:03:06 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 27 Nov 2009 12:03:06 +1300 Subject: [Cython] str arguments In-Reply-To: <4B0EF8C7.3070306@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> <4B0E36BB.10504@behnel.de> <4B0EF8C7.3070306@gfz-potsdam.de> Message-ID: <4B0F092A.80708@canterbury.ac.nz> Joachim Saul wrote: > def foo(list L): > cdef array arr = L > > given that at Python level > > arr = array(L) > > is likely to work (depending on the *values* of L of course). It wouldn't make sense to me. Constructing new objects is not something you expect a plain assignment to do in Python. It certainly shouldn't happen if the objects in question are mutable. -- Greg From stefan_ml at behnel.de Fri Nov 27 07:46:08 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Nov 2009 07:46:08 +0100 Subject: [Cython] str arguments In-Reply-To: <4B0EF8C7.3070306@gfz-potsdam.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> <4B0E36BB.10504@behnel.de> <4B0EF8C7.3070306@gfz-potsdam.de> Message-ID: <4B0F75B0.20900@behnel.de> Joachim Saul, 26.11.2009 22:53: > It is not Cython's job to enforce correct (Cython) code. The > user (of the Cython-created modules) will normally not be aware of what > goes on inside. Oh, this is absolutely not about compiler internals. > To me, nevertheless, it would make a lot of sense to allow code like > > def foo(list L): > cdef array arr = L > > given that at Python level > > arr = array(L) > > is likely to work (depending on the *values* of L of course). -1. Constructing objects should be explicit. >> Lets look at the example that started this thread: >> cdef extern void c_foo(char*) >> >> def foo(str bar=""): >> c_foo(bar) >> >> Now I understand that your position that this is incorrect, broken, >> and fragile code just waiting to blow up in the users face, > > Under what conditions is this incorrect or fragile? Certainly not > because of the typing or the default vaule. Sure, also because of that. The 'str' type does not accept unicode values in Py2, so you will get a TypeError when users pass text as a unicode string. That may not sound like a bit deal ("just tell them not to do it"), but Py2 will actually convert strings to unicode on the fly (e.g. on string joining and concatenation), so you may not even be aware that you passed a unicode string. And you will most likely have a pretty hard time debugging your code to find out where it originated from. This problem doesn't usually appear in Python itself, because all Python 2.x APIs are designed to handle both str and unicode. But your code will fail for non-str input. Believe me, your problem is not with the oh-so-far-away Py3, where 'str' is really what you want. It's with Py2. > even though in Python 3 'str' is differently encoded. Unicode is not an encoding. > As a consequence, I need to > define a C function that takes care of the de/encoding from/to C strings > depending on whether this is Python 2 or 3. I am pretty sure doing this > at C level using #ifdef's for different Python versions should be > straightforward. The preprocessor #if isn't the problem. The encoding of the string is. But given that str->unicode coercion exists in Py2, maybe we should really give this a helping hand here by generating automatic Py2-only input coercion code for the 'str' type that encodes unicode input using the platform encoding (just like Py2 does). That's not correct in many cases, but at least you'd get exceptions for the same data that Py2 would have trouble with, and the data that happens to work in other places of Py2 would pass silently. > Manual type checking only if I expect bar to be anything other than a > 'str'. Which I don't :) Too bad for the users of the API you write, who will then have to find a work-around themselves. Putting the conversion code behind the API makes sure it only has to be written once. >> which is the most correct from a unicode standpoint, but may require >> substantial changes to an existing codebase (I know, it could be >> called bugfixing), is certainly less user-friendly, and could >> introduce a fair amount of overhead as well. > > You probably mean less developer-friendly ;) And more user friendly, although I expect the users of your library to be developers, too. > The changes to the existing codebase would probably only involve the > additional decoding step. The overhead is what I'm a bit concerned > about. See, often one wants to simply compare such codes, or perhaps > sorting a large list of objects with such strings as attributes; the > overhead can be expected to be quite substantial in such cases. That's the price you pay for a world that's not defined in ASCII. Stefan From arfrever.fta at gmail.com Fri Nov 27 17:29:46 2009 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Fri, 27 Nov 2009 17:29:46 +0100 Subject: [Cython] [PATCH] Fix exit status of runtests.py Message-ID: <200911271729.53725.Arfrever.FTA@gmail.com> When tests fail, runtests.py should exit with non-zero exit status. -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: cython-0.12-fix_exit_status_of_runtests.py.patch Type: text/x-patch Size: 569 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20091127/0c696047/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20091127/0c696047/attachment.pgp From robertwb at math.washington.edu Fri Nov 27 19:57:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 10:57:49 -0800 Subject: [Cython] [PATCH] Fix exit status of runtests.py In-Reply-To: <200911271729.53725.Arfrever.FTA@gmail.com> References: <200911271729.53725.Arfrever.FTA@gmail.com> Message-ID: Good call. Thanks. Pushed. On Nov 27, 2009, at 8:29 AM, Arfrever Frehtes Taifersar Arahesis wrote: > When tests fail, runtests.py should exit with non-zero exit status. > > -- > Arfrever Frehtes Taifersar Arahesis > fix_exit_status_of_runtests > .py.patch>_______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Fri Nov 27 21:20:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 12:20:15 -0800 Subject: [Cython] str arguments In-Reply-To: <4B0E4667.3070109@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <1C8BC47E-FBBF-4D9C-95A4-C4AA431038D6@math.washington.edu> <4B0CE205.9080003@behnel.de> <4B0D8DB0.3040208@canterbury.ac.nz> <4B0DA3ED.5060903@behnel.de> <4B0DB238.2080401@canterbury.ac.nz> <4B0E4667.3070109@behnel.de> Message-ID: On Nov 26, 2009, at 1:12 AM, Stefan Behnel wrote: > Greg Ewing, 25.11.2009 23:39: >> Stefan Behnel wrote: >>> Ah, right, that's due to the str/bytes/unicode separation in 0.12. >> >> Ah, I see. It's probably behaving as designed, then, >> because the user hasn't said that his str class is >> based on anything that would be compatible with a >> builtin str. >> >> I think the difference is that Cython has an internal >> notion of the builtin str class, whereas Pyrex >> doesn't -- to Pyrex, a Python string is just a >> generic Python object. > > Yes, I know. Cython as departed from this a little lately, > especially in 0.12. > > I'm a big fan of having the compiler fail early if it detects > something > that would lead to an error at runtime. Coercion between different > string > types is an area where I think this is particularly helpful, but I > think > the rules are a little too strict. Unknown/external/underdefined > types must > always be allowed in assignments as the compiler can't know for sure > if > they will work or not. http://trac.cython.org/cython_trac/ticket/258 > The only case where assignments between known distinct types work is > when > the value is None. But even then, I don't see a use case for > allowing this: > > cdef list l = None > cdef str s = l > > IMHO, if a user explicitly types a variable (which is absolutely > optional > for Python types), he/she must be aware that this restricts its > value set > and consequently its use. Disallowing the above (for cases where > automatic > coercion is not clearly obvious) is something I would want Cython to > do at > compile time, not runtime. Makes sense to me. - Robert From robertwb at math.washington.edu Fri Nov 27 22:07:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 13:07:15 -0800 Subject: [Cython] str arguments In-Reply-To: <4B0E7900.9030108@behnel.de> References: <4B0C3B20.4030007@gfz-potsdam.de> <4B0C42F5.1080303@behnel.de> <4B0C5305.90200@behnel.de> <4B0C626F.506@gfz-potsdam.de> <0E5E4FB7-9515-4007-BB2D-27D3744AB1EE@math.washington.edu> <4B0CFEF7.1020604@gfz-potsdam.de> <09DD06A5-00E8-4323-B434-6E921B188936@math.washington.edu> <4B0D914F.2080408@behnel.de> <4B0E36BB.10504@behnel.de> <4B0E7900.9030108@behnel.de> Message-ID: On Nov 26, 2009, at 4:48 AM, Stefan Behnel wrote: > Hi Robert, > > Robert Bradshaw, 26.11.2009 11:03: >> I think this boils down to a philosophical difference. I think we >> should encourage users to write correct code, but not require it. >> That's why there's warnings vs. errors. We should only raise errors >> for things that could never work. > > I'm fine with that in general. However, for the specific case of > strings, I > must note that Py3 is as much a reality as Py2, and that the > semantics of > strings in Cython dictate that certain things will not work in certain > Python runtimes. I admit that a warning allows users to actively say > "I > don't care, my code will never run in a different environment > anyway". So I > guess there is a case for making it a warning. > > >>> Please come up with a valid use case where a currently unsupported >>> automatic conversion between different string types clearly helps in >>> writing simple and correct code. >> >> Lets look at the example that started this thread: >> >> cdef extern void c_foo(char*) >> >> def foo(str bar=""): >> c_foo(bar) >> >> Now I understand that your position that this is incorrect, broken, >> and fragile code just waiting to blow up in the users face, but >> from a >> pragmatic point of view it's working code that's part of a larger >> project and compiles and runs fine with Python 2 and Pyrex. It would >> compile and work fine with Cython as well except we have a check for >> just this case to disallow it [1], trying to force the user to write >> correct code. > > My position is that "str" is not the right type here. It's just like > using > "float" where "long" would be appropriate. Sure. > If you want "bytes", write "bytes". Bytes is usually the wrong API. Encoding, if necessary, should usually happen inside the API, not be left to the users. > That will also give you an appropriate > runtime exception in both Py2 and Py3 when you pass a unicode > string. No > harm done to anyone. If you use 'str', the exception you get in Py3 > later > in the code will be much less beautiful. I think the error would actually be better--instead of TypeError when you call the function, it would be a TypeError (or it could even be an unspecified ending error) at the point conversion to a char* is attempted. Anther deficiency in my book is that it would reject rather than handle unicode in Py2. >> The other proposed option >> >> def foo(bytes bar=b""): >> c_foo(bar) >> >> doesn't maintain compatibility with Pyrex > > Note that the following works in both Cython and Pyrex: > > cdef bytes b = "abcdefg" It doesn't work with either, as the bytes type is not a builtin in Pyrex (and can't even be imported, as we fake it for Py2), and we require the b"" prefix in Cython. My other issue with the bytes object is that most people don't know about it (yes, they probably should) so it's non-obvious to try (and also has non-intuitive semantics if it leaks out). > This is actually specified in CEP 108, and it's supported for good > reason. > So the above could simply be written as > > def foo(bytes bar=""): > c_foo(bar) > > in a portable way. (If this doesn't work in Cython, I would consider > that a > bug w.r.t. the CEP). Looks like we decided to start out strict, but maybe this is good reason to loosen it: http://www.mail-archive.com/cython-dev at codespeak.net/msg07268.html >> and though I have no idea >> what foo actually does, I would guess that this is not the API one >> would want to actually use in Py3. > > At least the OP mentioned the intention to pass "text", so I'd > second that > (also in Py2, BTW). > > >> The most correct option is >> >> def foo(bar=""): >> [manual type checking and decoding on bar, perhaps via a helper >> function] >> c_foo(bar_c) >> >> which is the most correct from a unicode standpoint, but may require >> substantial changes to an existing codebase (I know, it could be >> called bugfixing), is certainly less user-friendly, and could >> introduce a fair amount of overhead as well. > > Ok, I understand that you understand that this is required anyway, > both in > Py2 and Py3, so there's not much to add here. > > But what about actually making this specific case easier? Not caring > too > much about syntax for now, we could support something like > > def foo(bytes[encoding='UTF-8'] bar=""): > c_foo(bar_c) > > and, respectively: > > def foo(unicode[encoding='UTF-8'] bar=u""): > assert isinstance(bar, unicode) > > def foo(str[encoding='UTF-8'] bar=""): > assert isinstance(bar, str) > > and let Cython generate the argument type checking and conversion code > internally, based on what kind of string is passed at runtime. Even > 1D char > buffers could be supported as argument here... > > OTOH, this is really only required in Py2.x, and Py3 is the pretty > immediate future. Adding syntactic sugar for a soon-to-be-dead > platform may > just drop legacy stuff into the Cython language. But it would > certainly > solve the above problem. I just got another idea for this (I'll start new thread). > > >> My modest proposal > > ;) Well, you know I can come up with far more outrageous ones when it comes to string handling... :) > > >> is to allow bytes <-> str, char* <-> str, and str <- >>> unicode *with* typechecking. > > So, you would basically allow this: > > cdef char* s = "abcdef" > cdef bytes bs = s > cdef str pys = bs > cdef unicode us = pys > > Not beautiful, if you ask me, and I still haven't seen a use case that > supports this. Of course the above would fail with a TypeError in both Py2 and Py3 (one either the last or second-to-last line). > What's so bad about an explicit cast if you really think you > need such an assignment? You can even use a safe "" cast to > tell > the compiler that you are not sure if this works. The most useful is the (Py2 only) str <-> char*, which currently fails in Cython. A cast is incompatible with Pyrex in two separate ways, as well as not being obvious. >> (Alternatively, we could omit an >> explicit error at C compile time if conversion is attempted between >> incompatible types. I'm not sure whether that would be better.) > > I thought about that, too, but I doubt that it would be more helpful > than a > runtime error. I've thought of another reason it would be bad--it would prevent a user from using the rest of a library if an illegal conversion is attempted in one function, so a runtime error it is. - Robert From robertwb at math.washington.edu Fri Nov 27 22:34:46 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 13:34:46 -0800 Subject: [Cython] Another string encoding idea Message-ID: Though I usually try to avoid the topic, I've been thinking a lot about string handling in Cython lately. I think we've taken a great step forward in terms of usability with CEP 108, especially for those who never deal with external libraries, but all this explicit encoding and decoding still seems too heavy (though I understand why it's necessary to deal with anything but pure ASCII). For an application like lxml that is all about string processing, the verbosity and explicitness isn't burdensome and the issue naturally comes up, but this is not true of many applications. (For example the last time I had to use strings, my character set was limited to [0-9Ee+-.].) On the other hand, it's clear letting users just ignore the encoding issue is unacceptable and undesirable. I had an epiphany when I realized that I find this burdensome not because the user needs to specify an encoding, but that they have to manually handle it every time they deal with a char*. So, my proposal is this: let the user specify via a compiler directive an encoding to use for all conversions. Cython could then transparently and efficiently handle all char* <-> str (a.k.a. unicode) encodings in Py3, and unicode -> char* in Py2. If no encoding is specified char* would still turn into bytes in Py3, and the conversions mentioned above would be disallowed. This might be a good compromise between explicitness, safety, and ease of use. Thoughts? - Robert From dagss at student.matnat.uio.no Fri Nov 27 23:23:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 27 Nov 2009 23:23:52 +0100 Subject: [Cython] Another string encoding idea In-Reply-To: References: Message-ID: <4B105178.1070807@student.matnat.uio.no> Robert Bradshaw wrote: > Though I usually try to avoid the topic, I've been thinking a lot > about string handling in Cython lately. I think we've taken a great > step forward in terms of usability with CEP 108, especially for those > who never deal with external libraries, but all this explicit encoding > and decoding still seems too heavy (though I understand why it's > necessary to deal with anything but pure ASCII). For an application > like lxml that is all about string processing, the verbosity and > explicitness isn't burdensome and the issue naturally comes up, but > this is not true of many applications. (For example the last time I > had to use strings, my character set was limited to [0-9Ee+-.].) On > the other hand, it's clear letting users just ignore the encoding > issue is unacceptable and undesirable. > > I had an epiphany when I realized that I find this burdensome not > because the user needs to specify an encoding, but that they have to > manually handle it every time they deal with a char*. So, my proposal > is this: let the user specify via a compiler directive an encoding to > use for all conversions. Cython could then transparently and > efficiently handle all char* <-> str (a.k.a. unicode) encodings in > Py3, and unicode -> char* in Py2. If no encoding is specified char* > would still turn into bytes in Py3, and the conversions mentioned > above would be disallowed. > > This might be a good compromise between explicitness, safety, and ease > of use. Thoughts? I'm somewhat sceptical/undecided about char* being coerced to unicode this way, i.e. char*->unicode. I don't have a problem with the idea for unicode->char* (as long as bytes->char* is still OK as well ). -- Dag Sverre From robertwb at math.washington.edu Fri Nov 27 23:30:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 14:30:58 -0800 Subject: [Cython] Another string encoding idea In-Reply-To: <4B105178.1070807@student.matnat.uio.no> References: <4B105178.1070807@student.matnat.uio.no> Message-ID: <3A188408-4961-47BE-99A3-876A622EA9EB@math.washington.edu> On Nov 27, 2009, at 2:23 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> Though I usually try to avoid the topic, I've been thinking a lot >> about string handling in Cython lately. I think we've taken a great >> step forward in terms of usability with CEP 108, especially for those >> who never deal with external libraries, but all this explicit >> encoding >> and decoding still seems too heavy (though I understand why it's >> necessary to deal with anything but pure ASCII). For an application >> like lxml that is all about string processing, the verbosity and >> explicitness isn't burdensome and the issue naturally comes up, but >> this is not true of many applications. (For example the last time I >> had to use strings, my character set was limited to [0-9Ee+-.].) On >> the other hand, it's clear letting users just ignore the encoding >> issue is unacceptable and undesirable. >> >> I had an epiphany when I realized that I find this burdensome not >> because the user needs to specify an encoding, but that they have to >> manually handle it every time they deal with a char*. So, my proposal >> is this: let the user specify via a compiler directive an encoding to >> use for all conversions. Cython could then transparently and >> efficiently handle all char* <-> str (a.k.a. unicode) encodings in >> Py3, and unicode -> char* in Py2. If no encoding is specified char* >> would still turn into bytes in Py3, and the conversions mentioned >> above would be disallowed. >> >> This might be a good compromise between explicitness, safety, and >> ease >> of use. Thoughts? > > I'm somewhat sceptical/undecided about char* being coerced to unicode > this way, i.e. char*->unicode. I don't have a problem with the idea > for > unicode->char* This would only be in Py3, as usually bytes is the wrong thing to return (and expose to the user). What the best thing to do in Py2 is still unclear. > (as long as bytes->char* is still OK as well ). Yes, for sure. - Robert From dalcinl at gmail.com Fri Nov 27 23:33:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Nov 2009 19:33:48 -0300 Subject: [Cython] Another string encoding idea In-Reply-To: <4B105178.1070807@student.matnat.uio.no> References: <4B105178.1070807@student.matnat.uio.no> Message-ID: On Fri, Nov 27, 2009 at 7:23 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> Though I usually try to avoid the topic, I've been thinking a lot >> about string handling in Cython lately. I think we've taken a great >> step forward in terms of usability with CEP 108, especially for those >> who never deal with external libraries, but all this explicit encoding >> and decoding still seems too heavy (though I understand why it's >> necessary to deal with anything but pure ASCII). For an application >> like lxml that is all about string processing, the verbosity and >> explicitness isn't burdensome and the issue naturally comes up, but >> this is not true of many applications. (For example the last time I >> had to use strings, my character set was limited to [0-9Ee+-.].) On >> the other hand, it's clear letting users just ignore the encoding >> issue is unacceptable and undesirable. >> >> I had an epiphany when I realized that I find this burdensome not >> because the user needs to specify an encoding, but that they have to >> manually handle it every time they deal with a char*. So, my proposal >> is this: let the user specify via a compiler directive an encoding to >> use for all conversions. Cython could then transparently and >> efficiently handle all char* <-> str (a.k.a. unicode) encodings in >> Py3, and unicode -> char* in Py2. If no encoding is specified char* >> would still turn into bytes in Py3, and the conversions mentioned >> above would be disallowed. >> >> This might be a good compromise between explicitness, safety, and ease >> of use. Thoughts? > > I'm somewhat sceptical/undecided about char* being coerced to unicode > this way, i.e. char*->unicode. I don't have a problem with the idea for > unicode->char* (as long as bytes->char* is still OK as well ). > I have the same feeling. However, I would accept to have two directives: one for unicode->char*, and another for char*->unicode. And of course, we will need a mechanism to override the default encoding by using explicit encode()/decode() method call. For example, if you have to deal with both text and filenames in a char*, you may need to special-handle filenames (hello, ext* filesystems). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Nov 27 23:57:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 14:57:53 -0800 Subject: [Cython] Another string encoding idea In-Reply-To: References: <4B105178.1070807@student.matnat.uio.no> Message-ID: <0FE9215C-E66F-4016-AF9D-BB8E2AAE2F9A@math.washington.edu> On Nov 27, 2009, at 2:33 PM, Lisandro Dalcin wrote: > On Fri, Nov 27, 2009 at 7:23 PM, Dag Sverre Seljebotn > wrote: >> Robert Bradshaw wrote: >>> Though I usually try to avoid the topic, I've been thinking a lot >>> about string handling in Cython lately. I think we've taken a great >>> step forward in terms of usability with CEP 108, especially for >>> those >>> who never deal with external libraries, but all this explicit >>> encoding >>> and decoding still seems too heavy (though I understand why it's >>> necessary to deal with anything but pure ASCII). For an application >>> like lxml that is all about string processing, the verbosity and >>> explicitness isn't burdensome and the issue naturally comes up, but >>> this is not true of many applications. (For example the last time I >>> had to use strings, my character set was limited to [0-9Ee+-.].) On >>> the other hand, it's clear letting users just ignore the encoding >>> issue is unacceptable and undesirable. >>> >>> I had an epiphany when I realized that I find this burdensome not >>> because the user needs to specify an encoding, but that they have to >>> manually handle it every time they deal with a char*. So, my >>> proposal >>> is this: let the user specify via a compiler directive an encoding >>> to >>> use for all conversions. Cython could then transparently and >>> efficiently handle all char* <-> str (a.k.a. unicode) encodings in >>> Py3, and unicode -> char* in Py2. If no encoding is specified char* >>> would still turn into bytes in Py3, and the conversions mentioned >>> above would be disallowed. >>> >>> This might be a good compromise between explicitness, safety, and >>> ease >>> of use. Thoughts? >> >> I'm somewhat sceptical/undecided about char* being coerced to unicode >> this way, i.e. char*->unicode. I don't have a problem with the idea >> for >> unicode->char* (as long as bytes->char* is still OK as well ). >> > > I have the same feeling. However, I would accept to have two > directives: one for unicode->char*, and another for char*->unicode. That might be a good idea. > And of course, we will need a mechanism to override the default > encoding by using explicit encode()/decode() method call. For example, > if you have to deal with both text and filenames in a char*, you may > need to special-handle filenames (hello, ext* filesystems). For sure. I'm imagining the mechanisms one uses now would still work, as would stuff like cdef char* s = ... print s - Robert From robertwb at math.washington.edu Sat Nov 28 00:02:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 15:02:51 -0800 Subject: [Cython] Two docs bugs In-Reply-To: <4B0C28C2.6010200@lophus.org> References: <4B0C28C2.6010200@lophus.org> Message-ID: On Nov 24, 2009, at 10:41 AM, jonas at lophus.org wrote: > Hi guys, > > first of all: nice work for 0.12! > > In the pxd docs source, use "files -- they" with spaces rather than > "files--they" without any in the first paragraph, so sphinx renders > this > correctly. Ah, I wondered how to get a good M-dash. Thanks. Fixed. > The second thing: pygments has experimental support for Cython source > code since some weeks' commit ago. Could you please make use of that > Cython highlighter? Cython code is very hard to read when not > highlighted. Sounds like you know a lot more about this kind of stuff than I--would you mind supplying a patch? The sources are all at http://hg.cython.org/cython-docs/ Thanks, - Robert From greg.ewing at canterbury.ac.nz Sat Nov 28 00:05:36 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 Nov 2009 12:05:36 +1300 Subject: [Cython] Another string encoding idea In-Reply-To: References: Message-ID: <4B105B40.4000702@canterbury.ac.nz> Robert Bradshaw wrote: > So, my proposal > is this: let the user specify via a compiler directive an encoding to > use for all conversions. Cython could then transparently and > efficiently handle all char* <-> str (a.k.a. unicode) encodings There's another kind of problem with implicit conversion from str to char * in Py3: for any encoding other than utf8, it requires allocating memory, and then there's the issue of who owns it and how it gets deallocated. -- Greg From dalcinl at gmail.com Sat Nov 28 00:35:22 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Nov 2009 20:35:22 -0300 Subject: [Cython] Another string encoding idea In-Reply-To: <4B105B40.4000702@canterbury.ac.nz> References: <4B105B40.4000702@canterbury.ac.nz> Message-ID: On Fri, Nov 27, 2009 at 8:05 PM, Greg Ewing wrote: > Robert Bradshaw wrote: >> So, my proposal >> is this: let the user specify via a compiler directive an encoding to >> use for all conversions. Cython could then transparently and >> efficiently handle all char* <-> str (a.k.a. unicode) encodings > > There's another kind of problem with implicit conversion from > str to char * in Py3: for any encoding other than utf8, it requires > allocating memory, and then there's the issue of who owns it > and how it gets deallocated. > Sorry, Greg, I got lost. It seems I still have things to learn in Py3. How do you avoid the memory allocation in the case of UTF-8 ? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Nov 28 00:32:34 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Nov 2009 15:32:34 -0800 Subject: [Cython] Another string encoding idea In-Reply-To: <4B105B40.4000702@canterbury.ac.nz> References: <4B105B40.4000702@canterbury.ac.nz> Message-ID: On Nov 27, 2009, at 3:05 PM, Greg Ewing wrote: > Robert Bradshaw wrote: >> So, my proposal >> is this: let the user specify via a compiler directive an encoding to >> use for all conversions. Cython could then transparently and >> efficiently handle all char* <-> str (a.k.a. unicode) encodings > > There's another kind of problem with implicit conversion from > str to char * in Py3: for any encoding other than utf8, it requires > allocating memory, and then there's the issue of who owns it > and how it gets deallocated. Yes, that's a good question, and I'm not sure how best this could/ should be handled. However, even just supporting utf8 would be simple first step. (Also ASCII only could be supported via the same defenc slot and a bit of manual checking.) - Robert From greg.ewing at canterbury.ac.nz Sat Nov 28 00:55:50 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 28 Nov 2009 12:55:50 +1300 Subject: [Cython] Another string encoding idea In-Reply-To: References: <4B105B40.4000702@canterbury.ac.nz> Message-ID: <4B106706.4050403@canterbury.ac.nz> Lisandro Dalcin wrote: > Sorry, Greg, I got lost. It seems I still have things to learn in Py3. > How do you avoid the memory allocation in the case of UTF-8 ? I haven't looked in detail, but I believe that the str object is able to hold a cached encoding of itself in utf8, so the conversion could just return a pointer to that. -- Greg From dalcinl at gmail.com Sat Nov 28 01:22:21 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Nov 2009 21:22:21 -0300 Subject: [Cython] Another string encoding idea In-Reply-To: <4B106706.4050403@canterbury.ac.nz> References: <4B105B40.4000702@canterbury.ac.nz> <4B106706.4050403@canterbury.ac.nz> Message-ID: On Fri, Nov 27, 2009 at 8:55 PM, Greg Ewing wrote: > Lisandro Dalcin wrote: > >> Sorry, Greg, I got lost. It seems I still have things to learn in Py3. >> How do you avoid the memory allocation in the case of UTF-8 ? > > I haven't looked in detail, but I believe that the str object > is able to hold a cached encoding of itself in utf8, so the > conversion could just return a pointer to that. > Yes, that's the way things worked in Py2 (with def.encodding ascii). In Py3, that stuff is still there, but these API's are commented deprecated in Include/unicodeobject.h. These comments also recommend to use PyUnicode_AsUTF8String(). http://svn.python.org/view/python/branches/py3k/Include/unicodeobject.h?view=markup -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Sat Nov 28 07:52:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 28 Nov 2009 07:52:00 +0100 Subject: [Cython] Another string encoding idea In-Reply-To: References: Message-ID: <4B10C890.6080106@behnel.de> Hi Robert, Robert Bradshaw, 27.11.2009 22:34: > I had an epiphany when I realized that I find this burdensome not > because the user needs to specify an encoding, but that they have to > manually handle it every time they deal with a char*. So, my proposal > is this: let the user specify via a compiler directive an encoding to > use for all conversions. Sounds better than defaulting to the Python system encoding (as Py2 does), which is unrelated to the encoding used by any C libraries etc. It's also explicit. On the downside, while being explicit, it can still lead to all sorts of unexpected behaviour for users because strings would pop up in non-obvious types in their code. Now the conversion from char* to bytes would have to be explicit, although it's certainly not uncommon when dealing with C code, and totally normal in Py2. > Cython could then transparently and > efficiently handle all char* <-> str (a.k.a. unicode) encodings in > Py3, and unicode -> char* in Py2. As Greg pointed out, going directly from unicode to char* isn't trivial to implement and the implications are certainly not obvious for most users and not controllable by user code, so you can't just free memory by setting a variable to None. I think that's straight out for not being explicit. Currently, coercion from char*/bytes to unicode is an explicit step that is easy to do via cdef char* s = ... u = s[:length].decode('UTF-8') in 0.12. See http://trac.cython.org/cython_trac/ticket/436 Your proposal would make that # cython: bytes-encoding=UTF-8 cdef char* s = ... cdef unicode u = s[:length] (well, I /hope/ you'd require the target to be typed, right?) or # cython: bytes-encoding=UTF-8 cdef char* s = ... cdef str py_s = s[:length] so you'd not really gain much in terms of typing and (IMO) loose readability. Note that many encodings (e.g. the Asian 2-byte encodings) naturally contain 0 bytes, so automatic conversion of char* can't even work in those cases, as only the user code would know the correct length of the string. I'm +0.3 on the opposite way in Py2 for the 'str' type, though, as I already mentioned. I think that would a) fit the intention of users, b) match the main use case of accepting both str and unicode as function arguments in Py2 (and only in Py2!), and c) be free of memory handling issues as the target would still be a Python object. So I think it makes sense to support this only in Py2, and only for Python objects, not for char*. BTW, you keep talking about supporting all sorts of encodings here, whereas the use cases you present seem to deal only with plain ASCII non-textual data. Maybe it would be enough to make ASCII the default encoding for unicode->str coercion of function arguments in Py2 then? Or (as my original proposal went) to use the platform encoding for this, as CPython does, and which is normally ASCII in Py2 anyway. Stefan From robertwb at math.washington.edu Sat Nov 28 12:02:26 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 28 Nov 2009 03:02:26 -0800 Subject: [Cython] Another string encoding idea In-Reply-To: <4B10C890.6080106@behnel.de> References: <4B10C890.6080106@behnel.de> Message-ID: On Nov 27, 2009, at 10:52 PM, Stefan Behnel wrote: > Hi Robert, > > Robert Bradshaw, 27.11.2009 22:34: >> I had an epiphany when I realized that I find this burdensome not >> because the user needs to specify an encoding, but that they have to >> manually handle it every time they deal with a char*. So, my proposal >> is this: let the user specify via a compiler directive an encoding to >> use for all conversions. > > Sounds better than defaulting to the Python system encoding (as Py2 > does), > which is unrelated to the encoding used by any C libraries etc. It's > also > explicit. > > On the downside, while being explicit, it can still lead to all > sorts of > unexpected behaviour for users because strings would pop up in non- > obvious > types in their code. I'm not following you here. > Now the conversion from char* to bytes would have to > be explicit, although it's certainly not uncommon when dealing with > C code, > and totally normal in Py2. Yes, though only when the directive is in place. > >> Cython could then transparently and >> efficiently handle all char* <-> str (a.k.a. unicode) encodings in >> Py3, and unicode -> char* in Py2. > > As Greg pointed out, going directly from unicode to char* isn't > trivial to > implement and the implications are certainly not obvious for most > users and > not controllable by user code, so you can't just free memory by > setting a > variable to None. I think that's straight out for not being explicit. We might have to limit ourselves to the system default encoding, as there is a slot for that in the unicode object for just this purpose. At least it's always UTF-8 in Py3. > Currently, coercion from char*/bytes to unicode is an explicit step > that is > easy to do via > > cdef char* s = ... > u = s[:length].decode('UTF-8') > > in 0.12. See > > http://trac.cython.org/cython_trac/ticket/436 That is an improvement, though still a lot more baggage than cdef char* s = ... u = s > Your proposal would make that > > # cython: bytes-encoding=UTF-8 > > cdef char* s = ... > cdef unicode u = s[:length] > > (well, I /hope/ you'd require the target to be typed, right?) > > or > > # cython: bytes-encoding=UTF-8 > > cdef char* s = ... > cdef str py_s = s[:length] > > so you'd not really gain much in terms of typing and (IMO) loose > readability. No, I wasn't thinking of requiring a typed target. That way you could do cdef extern from "fooey.h": char* foo_c(char*) def foo(char* s): return foo_c(s) That seems like a big savings in terms of both readability and typing. This especially holds if the encoding step is completely orthogonal to the issue at hand. (It's also obvious for a new user to try and maintains compatibility with Pyrex and behaves gracefully in Py3.) > Note that many encodings (e.g. the Asian 2-byte encodings) naturally > contain 0 bytes, so automatic conversion of char* can't even work in > those > cases, as only the user code would know the correct length of the > string. Sure. One would have to manually specify the length in that case, if we even allowed such encodings as defaults. This isn't useful for everyone, but a huge audience is scientific users whose "strings" are getting passed to legacy fortran and C libraries, stuff like algorithms or flags or sensor IDs that are not likely to be non-ASCII, and who aren't doing "text processing" in any meaningful way. > I'm +0.3 on the opposite way in Py2 for the 'str' type, though, as I > already mentioned. I think that would a) fit the intention of users, > b) > match the main use case of accepting both str and unicode as function > arguments in Py2 (and only in Py2!), and c) be free of memory handling > issues as the target would still be a Python object. > > So I think it makes sense to support this only in Py2, and only for > Python > objects, not for char*. So you're suggesting def foo(str s): char* ss = s would work fine in Py2 on unicode input, but not for Py3? And this is not a shortcut for def foo(char* ss): ... wh