From robert.kern at gmail.com Wed Dec 1 00:41:08 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 30 Nov 2010 17:41:08 -0600 Subject: [Cython] splitting up Python functions In-Reply-To: References: <4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de> <4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz> <4CF4CE11.5040707@student.matnat.uio.no> <4CF547E5.1080409@student.matnat.uio.no> <4CF550CA.3090106@student.matnat.uio.no> Message-ID: On 11/30/10 4:01 PM, Robert Bradshaw wrote: > On Tue, Nov 30, 2010 at 1:57 PM, Robert Kern wrote: >> On 11/30/10 3:31 PM, Robert Bradshaw wrote: >>> On Tue, Nov 30, 2010 at 1:04 PM, Vitja Makarov wrote: >>> >>>> I don't like the idea to add one more external dependency. But if that >>>> really helps its okay. >>> >>> It's not an external dependency if it ships with Python. Actually, >>> ctypes didn't ship with Python until 2.5, so that may be an issue. I'm >>> thinking we'd only use it where we couldn't do it otherwise. >> >> ctypes/libffi is deliberately considered an optional component of Python for >> several reasons, one of which is the lack of support on certain platforms. I >> would say that it exists in a somewhat unique middle ground between "ships with >> Python" and "external dependency". > > Thanks for the perspective. Are there any good alternatives > (especially that would be less of an "external" dependency)? I'm afraid not. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From vitja.makarov at gmail.com Wed Dec 1 06:10:40 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 1 Dec 2010 08:10:40 +0300 Subject: [Cython] splitting up Python functions In-Reply-To: References: <4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de> <4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz> <4CF4CE11.5040707@student.matnat.uio.no> <4CF547E5.1080409@student.matnat.uio.no> <4CF550CA.3090106@student.matnat.uio.no> Message-ID: 2010/12/1 Robert Kern : > On 11/30/10 4:01 PM, Robert Bradshaw wrote: >> On Tue, Nov 30, 2010 at 1:57 PM, Robert Kern ?wrote: >>> On 11/30/10 3:31 PM, Robert Bradshaw wrote: >>>> On Tue, Nov 30, 2010 at 1:04 PM, Vitja Makarov ? ?wrote: >>>> >>>>> I don't like the idea to add one more external dependency. But if that >>>>> really helps its okay. >>>> >>>> It's not an external dependency if it ships with Python. Actually, >>>> ctypes didn't ship with Python until 2.5, so that may be an issue. I'm >>>> thinking we'd only use it where we couldn't do it otherwise. >>> >>> ctypes/libffi is deliberately considered an optional component of Python for >>> several reasons, one of which is the lack of support on certain platforms. I >>> would say that it exists in a somewhat unique middle ground between "ships with >>> Python" and "external dependency". >> >> Thanks for the perspective. Are there any good alternatives >> (especially that would be less of an "external" dependency)? > > I'm afraid not. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > ?that is made terrible by our own mad attempt to interpret it as though it had > ?an underlying truth." > ? -- Umberto Eco > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Don't worry about bfin support, that was just an example of unsupported arch. About use cases, I see only two now (am I right?): - Function that returns inner function - Function is passed as callback to plain C (object methods or nested). (It's okay when nested function is used inside cdef, cython can handle it.) In the libffi example new function bound_puts() is assebled it know how to call original puts_binding(). The real problem is when newly created function is passed to C code, We don't know when to call ffi_closure_free(). -- vitja. From robertwb at math.washington.edu Wed Dec 1 18:38:57 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 1 Dec 2010 09:38:57 -0800 Subject: [Cython] splitting up Python functions In-Reply-To: References: <4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de> <4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz> <4CF4CE11.5040707@student.matnat.uio.no> <4CF547E5.1080409@student.matnat.uio.no> <4CF550CA.3090106@student.matnat.uio.no> Message-ID: On Tue, Nov 30, 2010 at 9:10 PM, Vitja Makarov wrote: > 2010/12/1 Robert Kern : >> On 11/30/10 4:01 PM, Robert Bradshaw wrote: >>> On Tue, Nov 30, 2010 at 1:57 PM, Robert Kern ?wrote: >>>> On 11/30/10 3:31 PM, Robert Bradshaw wrote: >>>>> On Tue, Nov 30, 2010 at 1:04 PM, Vitja Makarov ? ?wrote: >>>>> >>>>>> I don't like the idea to add one more external dependency. But if that >>>>>> really helps its okay. >>>>> >>>>> It's not an external dependency if it ships with Python. Actually, >>>>> ctypes didn't ship with Python until 2.5, so that may be an issue. I'm >>>>> thinking we'd only use it where we couldn't do it otherwise. >>>> >>>> ctypes/libffi is deliberately considered an optional component of Python for >>>> several reasons, one of which is the lack of support on certain platforms. I >>>> would say that it exists in a somewhat unique middle ground between "ships with >>>> Python" and "external dependency". >>> >>> Thanks for the perspective. Are there any good alternatives >>> (especially that would be less of an "external" dependency)? >> >> I'm afraid not. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless enigma >> ?that is made terrible by our own mad attempt to interpret it as though it had >> ?an underlying truth." >> ? -- Umberto Eco >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > Don't worry about bfin support, that was just an example of unsupported arch. > > About use cases, I see only two now (am I right?): > > ?- Function that returns inner function > ?- Function is passed as callback to plain C (object methods or nested). > (It's okay when nested function is used inside cdef, cython can handle it.) > > In the libffi example new function bound_puts() is assebled it know > how to call original puts_binding(). > The real problem is when newly created function is passed to C code, > We don't know when to call ffi_closure_free(). Yes, this is something we would have to figure out. Possibly we could tie the lifetime to another object (like strings vs. their internal char* buffers). - Robert From dagss at student.matnat.uio.no Thu Dec 2 10:08:12 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 02 Dec 2010 10:08:12 +0100 Subject: [Cython] [Numpy-discussion] A Cython apply_along_axis function In-Reply-To: References: <4CF6FC27.5040005@silveregg.co.jp> Message-ID: <4CF761FC.60804@student.matnat.uio.no> On 12/02/2010 08:17 AM, Robert Bradshaw wrote: > On Wed, Dec 1, 2010 at 6:09 PM, John Salvatier > wrote: > >> On Wed, Dec 1, 2010 at 6:07 PM, Keith Goodman wrote: >> >>> On Wed, Dec 1, 2010 at 5:53 PM, David wrote: >>> >>> >>>> On 12/02/2010 04:47 AM, Keith Goodman wrote: >>>> >>>>> It's hard to write Cython code that can handle all dtypes and >>>>> arbitrary number of dimensions. The former is typically dealt with >>>>> using templates, but what do people do about the latter? >>>>> >>>> The only way that I know to do that systematically is iterator. There is >>>> a relatively simple example in scipy/signal (lfilter.c.src). >>>> >>>> I wonder if it would be possible to add better support for numpy >>>> iterators in cython... >>>> >>> Thanks for the tip. I'm starting to think that for now I should just >>> template both dtype and ndim. >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >> I enthusiastically support better iterator support for cython >> > I enthusiastically welcome contributions along this line. > Me too :-) I guess we're moving into more Cython-list territory, so let's move any follow-ups there (posting this one both places). Just in case anybody is wondering what something like this could look like, here's a rough scetch complete with bugs. The idea would be to a) add some rudimentary support for using the yield keyword in Cython to make a generator function, b) inline the generator function if the generator is used directly in a for-loop. This should result in very efficient code, and would also be much easier to implement than a general purpose generator. @cython.inline cdef array_iter_double(np.ndarray a, int axis=-1): cdef np.flatiter it ita = np.PyArray_IterAllButAxis(a, &axis) cdef Py_ssize_t stride = a.strides[axis], length = a.shape[axis], i while np.PyArray_ITER_NOTDONE(ita): for i in range(length): yield (np.PyArray_ITER_DATA(it) + )[i * stride])[0] # TODO: Probably yield indices as well np.PyArray_ITER_NEXT(it) # TODO: add faster special-cases for stride == sizeof(double) # Use NumPy iterator API to sum all values of array with # arbitrary number of dimensions: cdef double s = 0, value for value in array_iter_double(myarray): s += value # at this point, the contents of the array_iter_double function is copied, # and "s += value" simply inserted everywhere "yield" occurs in the function Dag Sverre From robert.kern at gmail.com Thu Dec 2 17:00:24 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 02 Dec 2010 10:00:24 -0600 Subject: [Cython] [Numpy-discussion] A Cython apply_along_axis function In-Reply-To: <4CF761FC.60804@student.matnat.uio.no> References: <4CF6FC27.5040005@silveregg.co.jp> <4CF761FC.60804@student.matnat.uio.no> Message-ID: On 12/2/10 3:08 AM, Dag Sverre Seljebotn wrote: > Just in case anybody is wondering what something like this could look > like, here's a rough scetch complete with bugs. The idea would be to a) > add some rudimentary support for using the yield keyword in Cython to > make a generator function, b) inline the generator function if the > generator is used directly in a for-loop. This should result in very > efficient code, and would also be much easier to implement than a > general purpose generator. I expect that being able to inline a generator would be quite a trick. I would go so far as to say that it is a completely different feature from adding generators to Cython in general. It's tantamount to adding (hygienic?) macros to Cython. I like it. > @cython.inline > cdef array_iter_double(np.ndarray a, int axis=-1): > cdef np.flatiter it > ita = np.PyArray_IterAllButAxis(a,&axis) > cdef Py_ssize_t stride = a.strides[axis], length = a.shape[axis], i > while np.PyArray_ITER_NOTDONE(ita): > for i in range(length): > yield(np.PyArray_ITER_DATA(it) + )[i * stride])[0] > # TODO: Probably yield indices as well > np.PyArray_ITER_NEXT(it) > # TODO: add faster special-cases for stride == sizeof(double) > > > # Use NumPy iterator API to sum all values of array with > # arbitrary number of dimensions: > cdef double s = 0, value > for value in array_iter_double(myarray): > s += value > # at this point, the contents of the array_iter_double function is > copied, > # and "s += value" simply inserted everywhere "yield" occurs in the > function > > Dag Sverre > -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan_ml at behnel.de Thu Dec 2 17:44:43 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 02 Dec 2010 17:44:43 +0100 Subject: [Cython] [Numpy-discussion] A Cython apply_along_axis function In-Reply-To: References: <4CF6FC27.5040005@silveregg.co.jp> <4CF761FC.60804@student.matnat.uio.no> Message-ID: <4CF7CCFB.5050203@behnel.de> Robert Kern, 02.12.2010 17:00: > On 12/2/10 3:08 AM, Dag Sverre Seljebotn wrote: > >> Just in case anybody is wondering what something like this could look >> like, here's a rough scetch complete with bugs. The idea would be to a) >> add some rudimentary support for using the yield keyword in Cython to >> make a generator function, b) inline the generator function if the >> generator is used directly in a for-loop. I know that we discussed these things ages ago, but I can't find the link ATM. >> This should result in very >> efficient code, and would also be much easier to implement than a >> general purpose generator. > > I expect that being able to inline a generator would be quite a trick. I would > go so far as to say that it is a completely different feature from adding > generators to Cython in general. It's tantamount to adding (hygienic?) macros to > Cython. > > I like it. Note that we already have inlined generator expressions, i.e. things like sum(2*x for x in seq) get unfolded into a straight sum loop. Same for any, all and I forget what others. And they already use the yield keyword for that purpose, so much of what is needed is already there. However, the transformation from general 1-yield generator functions won't be trivial as the loop that runs over the generator function basically needs to get moved inside of the generator code (the other way round would be easier). This may involve some pretty hairy messing around with the scoping rules. It's worth a try, though. Stefan From stefan_ml at behnel.de Fri Dec 3 08:41:42 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 08:41:42 +0100 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: References: Message-ID: <4CF89F36.8030903@behnel.de> Vitja Makarov, 29.11.2010 21:18: > Attached patch tries to solve the issue. I added comments on the bug tracker. Stefan From vitja.makarov at gmail.com Fri Dec 3 08:56:23 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 3 Dec 2010 10:56:23 +0300 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: <4CF89F36.8030903@behnel.de> References: <4CF89F36.8030903@behnel.de> Message-ID: 2010/12/3 Stefan Behnel : > Vitja Makarov, 29.11.2010 21:18: >> Attached patch tries to solve the issue. > > I added comments on the bug tracker. > > Stefan Okay, I'll take a look. What's about #605? -- vitja. From stefan_ml at behnel.de Fri Dec 3 09:21:53 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 09:21:53 +0100 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: References: <4CF89F36.8030903@behnel.de> Message-ID: <4CF8A8A1.7070808@behnel.de> Vitja Makarov, 03.12.2010 08:56: > What's about #605? Comment posted there. Stefan From sturla at molden.no Fri Dec 3 13:56:22 2010 From: sturla at molden.no (Sturla Molden) Date: Fri, 3 Dec 2010 13:56:22 +0100 Subject: [Cython] [Numpy-discussion] A Cython apply_along_axis function In-Reply-To: References: <4CF6FC27.5040005@silveregg.co.jp> <4CF761FC.60804@student.matnat.uio.no> Message-ID: > On 12/2/10 3:08 AM, Dag Sverre Seljebotn wrote: > I expect that being able to inline a generator would be quite a trick. Except that applying a function along an axis is the kind of thing onw might want to workshare/multithread in a "nogil" block. Using a Python object like an iterator here kind of takes that advantage away, i.e. one has to hold the GIL to access the iterator. In additon to make C level threads compete for the GIL, it also prevents OpenMP's scheduler from balancing the work load. That is why I rather collect a temporary array of pointers instead. Sturla From vitja.makarov at gmail.com Fri Dec 3 14:27:06 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 3 Dec 2010 16:27:06 +0300 Subject: [Cython] Lamba support in class scope and cclass scope, #605 Message-ID: Stefan Behnel (from trac): > Why not have a method generate_lambda_definitions() somewhere up the AST hierarchy? Maybe in BlockNode? You could just pass in the relevant scope explicitly. I've updated patch. Now all lambdas are defined in BlockNode.generate_lambda_definitions() This patch (I tried with function-redefine applied) gives about ~+300 pyregr tests ;) -- vitja. From stefan_ml at behnel.de Fri Dec 3 14:48:34 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 14:48:34 +0100 Subject: [Cython] [Numpy-discussion] A Cython apply_along_axis function In-Reply-To: References: <4CF6FC27.5040005@silveregg.co.jp> <4CF761FC.60804@student.matnat.uio.no> Message-ID: <4CF8F532.7040304@behnel.de> Sturla Molden, 03.12.2010 13:56: >> On 12/2/10 3:08 AM, Dag Sverre Seljebotn wrote: > >> I expect that being able to inline a generator would be quite a trick. > > Except that applying a function along an axis is the kind of thing onw > might want to workshare/multithread in a "nogil" block. Using a Python > object like an iterator here kind of takes that advantage away, i.e. one > has to hold the GIL to access the iterator. In additon to make C level > threads compete for the GIL, it also prevents OpenMP's scheduler from > balancing the work load. That is why I rather collect a temporary array of > pointers instead. Actually, the idea was to inline the generator code *before* the C code gets generated. Stefan From dagss at student.matnat.uio.no Fri Dec 3 14:49:42 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 03 Dec 2010 14:49:42 +0100 Subject: [Cython] [Numpy-discussion] A Cython apply_along_axis function In-Reply-To: References: <4CF6FC27.5040005@silveregg.co.jp> <4CF761FC.60804@student.matnat.uio.no> Message-ID: <4CF8F576.8020705@student.matnat.uio.no> On 12/03/2010 01:56 PM, Sturla Molden wrote: >> On 12/2/10 3:08 AM, Dag Sverre Seljebotn wrote: >> > >> I expect that being able to inline a generator would be quite a trick. >> (That was actually said by Robert Kern.) > Except that applying a function along an axis is the kind of thing onw > might want to workshare/multithread in a "nogil" block. Using a Python > object like an iterator here kind of takes that advantage away, i.e. one > has to hold the GIL to access the iterator. In additon to make C level > threads compete for the GIL, it also prevents OpenMP's scheduler from > balancing the work load. That is why I rather collect a temporary array of > pointers instead. > There's no actual Python iterator involved here...As Robert Kern said, it's basically just a proposal for loop-macros that will be convenient to use for people used to the yield keyword. Still, one must release the gil somewhere. I.e. there must be a "with nogil" clause within the generator. Hmm, perhaps a bit confusing...perhaps we'd require a "with nogil" in the for-loop body as well in that case. It clearly needs more thought, but I don't think it is fundamentally broken, and as (AFAIK) nobody stands ready to implement it right now I don't think there's much point to discuss this further... Dag Sverre From stefan_ml at behnel.de Fri Dec 3 15:14:14 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 15:14:14 +0100 Subject: [Cython] Lamba support in class scope and cclass scope, #605 In-Reply-To: References: Message-ID: <4CF8FB36.8050200@behnel.de> Vitja Makarov, 03.12.2010 14:27: > Stefan Behnel (from trac): >> Why not have a method generate_lambda_definitions() somewhere up the AST hierarchy? Maybe in BlockNode? You could just pass in the relevant scope explicitly. > > I've updated patch. Now all lambdas are defined in > BlockNode.generate_lambda_definitions() Yes, that's much cleaner. Time to ask Hudson. http://hg.cython.org/cython-devel/rev/067bda3960e4 > This patch (I tried with function-redefine applied) gives about ~+300 > pyregr tests ;) Cool. :) Stefan From stefan_ml at behnel.de Fri Dec 3 15:22:31 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 15:22:31 +0100 Subject: [Cython] Lamba support in class scope and cclass scope, #605 In-Reply-To: <4CF8FB36.8050200@behnel.de> References: <4CF8FB36.8050200@behnel.de> Message-ID: <4CF8FD27.2020805@behnel.de> Stefan Behnel, 03.12.2010 15:14: > Vitja Makarov, 03.12.2010 14:27: >> Stefan Behnel (from trac): >>> Why not have a method generate_lambda_definitions() somewhere up the AST hierarchy? Maybe in BlockNode? You could just pass in the relevant scope explicitly. >> >> I've updated patch. Now all lambdas are defined in >> BlockNode.generate_lambda_definitions() > > Yes, that's much cleaner. Time to ask Hudson. > > http://hg.cython.org/cython-devel/rev/067bda3960e4 Hmm, looks like neither of us ran the test suite after the change: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py2-trunk-c/614/testReport/ Stefan From vitja.makarov at gmail.com Fri Dec 3 15:26:48 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 3 Dec 2010 17:26:48 +0300 Subject: [Cython] Lamba support in class scope and cclass scope, #605 In-Reply-To: <4CF8FD27.2020805@behnel.de> References: <4CF8FB36.8050200@behnel.de> <4CF8FD27.2020805@behnel.de> Message-ID: 2010/12/3 Stefan Behnel : > Stefan Behnel, 03.12.2010 15:14: >> Vitja Makarov, 03.12.2010 14:27: >>> Stefan Behnel (from trac): >>>> Why not have a method generate_lambda_definitions() somewhere up the AST hierarchy? Maybe in BlockNode? You could just pass in the relevant scope explicitly. >>> >>> I've updated patch. Now all lambdas are defined in >>> BlockNode.generate_lambda_definitions() >> >> Yes, that's much cleaner. Time to ask Hudson. >> >> http://hg.cython.org/cython-devel/rev/067bda3960e4 > > Hmm, looks like neither of us ran the test suite after the change: > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py2-trunk-c/614/testReport/ > > Stefan Sorry I ran it for previous version :( -- vitja. From vitja.makarov at gmail.com Fri Dec 3 15:37:27 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 3 Dec 2010 17:37:27 +0300 Subject: [Cython] Lamba support in class scope and cclass scope, #605 In-Reply-To: References: <4CF8FB36.8050200@behnel.de> <4CF8FD27.2020805@behnel.de> Message-ID: 2010/12/3 Vitja Makarov : > 2010/12/3 Stefan Behnel : >> Stefan Behnel, 03.12.2010 15:14: >>> Vitja Makarov, 03.12.2010 14:27: >>>> Stefan Behnel (from trac): >>>>> Why not have a method generate_lambda_definitions() somewhere up the AST hierarchy? Maybe in BlockNode? You could just pass in the relevant scope explicitly. >>>> >>>> I've updated patch. Now all lambdas are defined in >>>> BlockNode.generate_lambda_definitions() >>> >>> Yes, that's much cleaner. Time to ask Hudson. >>> >>> http://hg.cython.org/cython-devel/rev/067bda3960e4 >> >> Hmm, looks like neither of us ran the test suite after the change: >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py2-trunk-c/614/testReport/ >> >> Stefan > > Sorry I ran it for previous version :( > > -- > vitja. > Here is small patch. -- vitja. -------------- next part -------------- A non-text attachment was scrubbed... Name: lambda-class-scope-fix.patch Type: text/x-patch Size: 883 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20101203/716c6fb0/attachment.bin From stefan_ml at behnel.de Fri Dec 3 15:56:50 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 15:56:50 +0100 Subject: [Cython] Lamba support in class scope and cclass scope, #605 In-Reply-To: References: <4CF8FB36.8050200@behnel.de> <4CF8FD27.2020805@behnel.de> Message-ID: <4CF90532.5020108@behnel.de> Vitja Makarov, 03.12.2010 15:37: > Here is small patch. I applied a slightly different fix - if there's no body, there's no lambdas. All fine now. Stefan From stefan_ml at behnel.de Fri Dec 3 19:10:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 19:10:56 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? Message-ID: <4CF932B0.8010108@behnel.de> Hi, I just implemented C access to the fields of CPython's builtin types like this: cdef complex c = 1+2j c.real, c.imag = 10, 20 print (c.real, c.imag) and then noticed that "real" and "imag" are actually read-only in Python. So I wonder if we should disallow the above (i.e. raise a compiler error during type analysis) and instead only allow this: cdef complex c = 1+2j cval = &c.cval cval.real, cval.imag = 10, 20 print (c.real, c.imag) (which also works now, BTW). Opinions? Stefan From dalcinl at gmail.com Fri Dec 3 19:39:06 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 3 Dec 2010 15:39:06 -0300 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF932B0.8010108@behnel.de> References: <4CF932B0.8010108@behnel.de> Message-ID: On 3 December 2010 15:10, Stefan Behnel wrote: > Hi, > > I just implemented C access to the fields of CPython's builtin types like this: > > ? ? cdef complex c = 1+2j > ? ? c.real, c.imag = 10, 20 > ? ? print (c.real, c.imag) > > and then noticed that "real" and "imag" are actually read-only in Python. > So I wonder if we should disallow the above (i.e. raise a compiler error > during type analysis) and instead only allow this: > > ? ? cdef complex c = 1+2j > ? ? cval = &c.cval > ? ? cval.real, cval.imag = 10, 20 > ? ? print (c.real, c.imag) > > (which also works now, BTW). > > Opinions? > No! No! No! In Python, you cannot assign to real or imag simply because the complex is immutable (like Python ints and floats). For C types, immutability does not apply! -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Dec 3 19:53:31 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 19:53:31 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: References: <4CF932B0.8010108@behnel.de> Message-ID: <4CF93CAB.7070704@behnel.de> Lisandro Dalcin, 03.12.2010 19:39: > On 3 December 2010 15:10, Stefan Behnel wrote: >> I just implemented C access to the fields of CPython's builtin types like this: >> >> cdef complex c = 1+2j >> c.real, c.imag = 10, 20 >> print (c.real, c.imag) >> >> and then noticed that "real" and "imag" are actually read-only in Python. >> So I wonder if we should disallow the above (i.e. raise a compiler error >> during type analysis) and instead only allow this: >> >> cdef complex c = 1+2j >> cval =&c.cval >> cval.real, cval.imag = 10, 20 >> print (c.real, c.imag) >> >> (which also works now, BTW). >> >> Opinions? > > No! No! No! > > In Python, you cannot assign to real or imag simply because the > complex is immutable (like Python ints and floats). For C types, > immutability does not apply! I understand that. However, I hope that you are talking about the builtin Python complex type just like I am. Given that these two fields carry the same name as the Python visible attributes, this introduces behaviour that diverges from CPython's own behaviour. Making these fields mutable means that you will get an exception when you modify the field in CPython but a straight value update after you compiled your code. Would it be so bad to require c.cval.real = 3.3 instead of c.real = 3.3 if you *really* think you must change the value of an immutable complex object? BTW, the main reason why I implemented was the (mutable) slice type, not so much the complex type. Stefan From dagss at student.matnat.uio.no Fri Dec 3 19:56:52 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 03 Dec 2010 19:56:52 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93CAB.7070704@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> Message-ID: <4CF93D74.7050803@student.matnat.uio.no> On 12/03/2010 07:53 PM, Stefan Behnel wrote: > Lisandro Dalcin, 03.12.2010 19:39: > >> On 3 December 2010 15:10, Stefan Behnel wrote: >> >>> I just implemented C access to the fields of CPython's builtin types like this: >>> >>> cdef complex c = 1+2j >>> c.real, c.imag = 10, 20 >>> print (c.real, c.imag) >>> >>> and then noticed that "real" and "imag" are actually read-only in Python. >>> So I wonder if we should disallow the above (i.e. raise a compiler error >>> during type analysis) and instead only allow this: >>> >>> cdef complex c = 1+2j >>> cval =&c.cval >>> cval.real, cval.imag = 10, 20 >>> print (c.real, c.imag) >>> >>> (which also works now, BTW). >>> >>> Opinions? >>> >> No! No! No! >> >> In Python, you cannot assign to real or imag simply because the >> complex is immutable (like Python ints and floats). For C types, >> immutability does not apply! >> > I understand that. However, I hope that you are talking about the builtin > Python complex type just like I am. Given that these two fields carry the > No. If you do "cdef complex c", what you have is a C complex type. How would you treat "cdef float complex c"? Python has no object for single precision complex. Dag Sverre > same name as the Python visible attributes, this introduces behaviour that > diverges from CPython's own behaviour. Making these fields mutable means > that you will get an exception when you modify the field in CPython but a > straight value update after you compiled your code. > > Would it be so bad to require > > c.cval.real = 3.3 > > instead of > > c.real = 3.3 > > if you *really* think you must change the value of an immutable complex object? > > BTW, the main reason why I implemented was the (mutable) slice type, not so > much the complex type. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Fri Dec 3 19:57:56 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 3 Dec 2010 10:57:56 -0800 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93CAB.7070704@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> Message-ID: On Fri, Dec 3, 2010 at 10:53 AM, Stefan Behnel wrote: > Lisandro Dalcin, 03.12.2010 19:39: >> On 3 December 2010 15:10, Stefan Behnel wrote: >>> I just implemented C access to the fields of CPython's builtin types like this: >>> >>> ? ? ?cdef complex c = 1+2j >>> ? ? ?c.real, c.imag = 10, 20 >>> ? ? ?print (c.real, c.imag) >>> >>> and then noticed that "real" and "imag" are actually read-only in Python. >>> So I wonder if we should disallow the above (i.e. raise a compiler error >>> during type analysis) and instead only allow this: >>> >>> ? ? ?cdef complex c = 1+2j >>> ? ? ?cval =&c.cval >>> ? ? ?cval.real, cval.imag = 10, 20 >>> ? ? ?print (c.real, c.imag) >>> >>> (which also works now, BTW). >>> >>> Opinions? >> >> No! No! No! >> >> In Python, you cannot assign to real or imag simply because the >> complex is immutable (like Python ints and floats). For C types, >> immutability does not apply! > > I understand that. However, I hope that you are talking about the builtin > Python complex type just like I am. Given that these two fields carry the > same name as the Python visible attributes, this introduces behaviour that > diverges from CPython's own behaviour. Making these fields mutable means > that you will get an exception when you modify the field in CPython but a > straight value update after you compiled your code. > > Would it be so bad to require > > ? ? c.cval.real = 3.3 > > instead of > > ? ? c.real = 3.3 > > if you *really* think you must change the value of an immutable complex object? > > BTW, the main reason why I implemented was the (mutable) slice type, not so > much the complex type. Of course we have exactly the same asymmetry for the read-only attributes of typed cdef classes. - Robert From dagss at student.matnat.uio.no Fri Dec 3 20:04:43 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 03 Dec 2010 20:04:43 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93D74.7050803@student.matnat.uio.no> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> Message-ID: <4CF93F4B.4000403@student.matnat.uio.no> On 12/03/2010 07:56 PM, Dag Sverre Seljebotn wrote: > On 12/03/2010 07:53 PM, Stefan Behnel wrote: > >> Lisandro Dalcin, 03.12.2010 19:39: >> >> >>> On 3 December 2010 15:10, Stefan Behnel wrote: >>> >>> >>>> I just implemented C access to the fields of CPython's builtin types like this: >>>> >>>> cdef complex c = 1+2j >>>> c.real, c.imag = 10, 20 >>>> print (c.real, c.imag) >>>> >>>> and then noticed that "real" and "imag" are actually read-only in Python. >>>> So I wonder if we should disallow the above (i.e. raise a compiler error >>>> during type analysis) and instead only allow this: >>>> >>>> cdef complex c = 1+2j >>>> cval =&c.cval >>>> cval.real, cval.imag = 10, 20 >>>> print (c.real, c.imag) >>>> >>>> (which also works now, BTW). >>>> >>>> Opinions? >>>> >>>> >>> No! No! No! >>> >>> In Python, you cannot assign to real or imag simply because the >>> complex is immutable (like Python ints and floats). For C types, >>> immutability does not apply! >>> >>> >> I understand that. However, I hope that you are talking about the builtin >> Python complex type just like I am. Given that these two fields carry the >> >> > No. If you do "cdef complex c", what you have is a C complex type. > > How would you treat "cdef float complex c"? Python has no object for > single precision complex. > Ah...sorry, seems me and Lisandro skipped your first sentence. (And I was wrong, indeed "cdef complex c" is undefined in current Cython and available for this purpose.) YES, for access to something that is an object, I do support immutability. However, I do have a much bigger problem with the naming here. I find this very little user-friendly: cdef complex x # x is Python object cdef double complex x # same precision but in C cdef float complex x # single precision in C I really do feel that "int" has kind of set the precedent here, and that if only "complex" is made available it should be as "double complex" or similar. We should find a consistent solution for this that works the same for complex and int and bool... cdef __builtin__.int x cdef __builtin__.complex x or cdef object int x cdef object complex x or something like that. Dag Sverre From stefan_ml at behnel.de Fri Dec 3 20:05:17 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 20:05:17 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> Message-ID: <4CF93F6D.2070503@behnel.de> Robert Bradshaw, 03.12.2010 19:57: > On Fri, Dec 3, 2010 at 10:53 AM, Stefan Behnel wrote: >> Lisandro Dalcin, 03.12.2010 19:39: >>> On 3 December 2010 15:10, Stefan Behnel wrote: >>>> I just implemented C access to the fields of CPython's builtin types like this: >>>> >>>> cdef complex c = 1+2j >>>> c.real, c.imag = 10, 20 >>>> print (c.real, c.imag) >>>> >> Given that these two fields carry the >> same name as the Python visible attributes, this introduces behaviour that >> diverges from CPython's own behaviour. Making these fields mutable means >> that you will get an exception when you modify the field in CPython but a >> straight value update after you compiled your code. > > Of course we have exactly the same asymmetry for the read-only > attributes of typed cdef classes. Sure. But I think it's a difference if you allow users to change the values of their own types, or if you allow them to accidentally change the value of an internal type of the CPython runtime. Stefan From stefan_ml at behnel.de Fri Dec 3 20:07:15 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 20:07:15 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93D74.7050803@student.matnat.uio.no> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> Message-ID: <4CF93FE3.4040804@behnel.de> Dag Sverre Seljebotn, 03.12.2010 19:56: > No. If you do "cdef complex c", what you have is a C complex type. Hmm, not anymore, I guess. The above gives me a Python complex object. Why didn't this break any tests? > How would you treat "cdef float complex c"? Python has no object for > single precision complex. Still seems to work. The tests for the C89 complex numbers keeps giving the same results and the C code looks ok to me. Stefan From dagss at student.matnat.uio.no Fri Dec 3 20:07:25 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 03 Dec 2010 20:07:25 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93F6D.2070503@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93F6D.2070503@behnel.de> Message-ID: <4CF93FED.5060701@student.matnat.uio.no> On 12/03/2010 08:05 PM, Stefan Behnel wrote: > Robert Bradshaw, 03.12.2010 19:57: > >> On Fri, Dec 3, 2010 at 10:53 AM, Stefan Behnel wrote: >> >>> Lisandro Dalcin, 03.12.2010 19:39: >>> >>>> On 3 December 2010 15:10, Stefan Behnel wrote: >>>> >>>>> I just implemented C access to the fields of CPython's builtin types like this: >>>>> >>>>> cdef complex c = 1+2j >>>>> c.real, c.imag = 10, 20 >>>>> print (c.real, c.imag) >>>>> >>>>> >>> Given that these two fields carry the >>> same name as the Python visible attributes, this introduces behaviour that >>> diverges from CPython's own behaviour. Making these fields mutable means >>> that you will get an exception when you modify the field in CPython but a >>> straight value update after you compiled your code. >>> >> Of course we have exactly the same asymmetry for the read-only >> attributes of typed cdef classes. >> > Sure. But I think it's a difference if you allow users to change the values > of their own types, or if you allow them to accidentally change the value > of an internal type of the CPython runtime. > +1. If we end up with stuff like this it's going to be a major pain for users: >>> x = 1 + 2j >>> f(x) >>> print x (300 + 4j) Dag Sverre From robertwb at math.washington.edu Fri Dec 3 20:07:53 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 3 Dec 2010 11:07:53 -0800 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93F4B.4000403@student.matnat.uio.no> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> Message-ID: On Fri, Dec 3, 2010 at 11:04 AM, Dag Sverre Seljebotn wrote: > On 12/03/2010 07:56 PM, Dag Sverre Seljebotn wrote: >> On 12/03/2010 07:53 PM, Stefan Behnel wrote: >> >>> Lisandro Dalcin, 03.12.2010 19:39: >>> >>> >>>> On 3 December 2010 15:10, Stefan Behnel wrote: >>>> >>>> >>>>> I just implemented C access to the fields of CPython's builtin types like this: >>>>> >>>>> ? ? ? ?cdef complex c = 1+2j >>>>> ? ? ? ?c.real, c.imag = 10, 20 >>>>> ? ? ? ?print (c.real, c.imag) >>>>> >>>>> and then noticed that "real" and "imag" are actually read-only in Python. >>>>> So I wonder if we should disallow the above (i.e. raise a compiler error >>>>> during type analysis) and instead only allow this: >>>>> >>>>> ? ? ? ?cdef complex c = 1+2j >>>>> ? ? ? ?cval =&c.cval >>>>> ? ? ? ?cval.real, cval.imag = 10, 20 >>>>> ? ? ? ?print (c.real, c.imag) >>>>> >>>>> (which also works now, BTW). >>>>> >>>>> Opinions? >>>>> >>>>> >>>> No! No! No! >>>> >>>> In Python, you cannot assign to real or imag simply because the >>>> complex is immutable (like Python ints and floats). For C types, >>>> immutability does not apply! >>>> >>>> >>> I understand that. However, I hope that you are talking about the builtin >>> Python complex type just like I am. Given that these two fields carry the >>> >>> >> No. If you do "cdef complex c", what you have is a C complex type. >> >> How would you treat "cdef float complex c"? Python has no object for >> single precision complex. >> > > Ah...sorry, seems me and Lisandro skipped your first sentence. (And I > was wrong, indeed "cdef complex c" is undefined in current Cython and > available for this purpose.) > > YES, for access to something that is an object, I do support immutability. > > However, I do have a much bigger problem with the naming here. I find > this very little user-friendly: > > cdef complex x # x is Python object > cdef double complex x # same precision but in C > cdef float complex x # single precision in C > > I really do feel that "int" has kind of set the precedent here, and that > if only "complex" is made available it should be as "double complex" or > similar. > > We should find a consistent solution for this that works the same for > complex and int and bool... > > cdef __builtin__.int x > cdef __builtin__.complex x > > or > > cdef object int x > cdef object complex x > > or something like that. +1. We can already cimport these from cpython. object complex is a bit odd because it makes it feel like the real and imaginary components are objects. - Robert From stefan_ml at behnel.de Fri Dec 3 20:10:23 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 20:10:23 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93CAB.7070704@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> Message-ID: <4CF9409F.2010505@behnel.de> Stefan Behnel, 03.12.2010 19:53: > BTW, the main reason why I implemented was the (mutable) slice type, not so > much the complex type. Oops, sorry, actually, the slice type is immutable as well. That changes my proposal to actually making *all* inner fields of builtin types read-only. Stefan From robertwb at math.washington.edu Fri Dec 3 20:09:57 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 3 Dec 2010 11:09:57 -0800 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93FED.5060701@student.matnat.uio.no> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93F6D.2070503@behnel.de> <4CF93FED.5060701@student.matnat.uio.no> Message-ID: On Fri, Dec 3, 2010 at 11:07 AM, Dag Sverre Seljebotn wrote: > On 12/03/2010 08:05 PM, Stefan Behnel wrote: >> Robert Bradshaw, 03.12.2010 19:57: >> >>> On Fri, Dec 3, 2010 at 10:53 AM, Stefan Behnel wrote: >>> >>>> Lisandro Dalcin, 03.12.2010 19:39: >>>> >>>>> On 3 December 2010 15:10, Stefan Behnel wrote: >>>>> >>>>>> I just implemented C access to the fields of CPython's builtin types like this: >>>>>> >>>>>> ? ? ? ?cdef complex c = 1+2j >>>>>> ? ? ? ?c.real, c.imag = 10, 20 >>>>>> ? ? ? ?print (c.real, c.imag) >>>>>> >>>>>> >>>> Given that these two fields carry the >>>> same name as the Python visible attributes, this introduces behaviour that >>>> diverges from CPython's own behaviour. Making these fields mutable means >>>> that you will get an exception when you modify the field in CPython but a >>>> straight value update after you compiled your code. >>>> >>> Of course we have exactly the same asymmetry for the read-only >>> attributes of typed cdef classes. >>> >> Sure. But I think it's a difference if you allow users to change the values >> of their own types, or if you allow them to accidentally change the value >> of an internal type of the CPython runtime. >> > > +1. If we end up with stuff like this it's going to be a major pain for > users: > > ?>>> x = 1 + 2j > ?>>> f(x) > ?>>> print x > (300 + 4j) I can see a usecase for it--for example if you're evaluating a user-defined function at many different values and don't want to re-allocate the object, but in that case I'm willing to force the user to jump through some extra trickery. - Robert From robertwb at math.washington.edu Fri Dec 3 20:02:13 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 3 Dec 2010 11:02:13 -0800 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93D74.7050803@student.matnat.uio.no> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> Message-ID: On Fri, Dec 3, 2010 at 10:56 AM, Dag Sverre Seljebotn wrote: > On 12/03/2010 07:53 PM, Stefan Behnel wrote: >> Lisandro Dalcin, 03.12.2010 19:39: >> >>> On 3 December 2010 15:10, Stefan Behnel wrote: >>> >>>> I just implemented C access to the fields of CPython's builtin types like this: >>>> >>>> ? ? ? cdef complex c = 1+2j >>>> ? ? ? c.real, c.imag = 10, 20 >>>> ? ? ? print (c.real, c.imag) >>>> >>>> and then noticed that "real" and "imag" are actually read-only in Python. >>>> So I wonder if we should disallow the above (i.e. raise a compiler error >>>> during type analysis) and instead only allow this: >>>> >>>> ? ? ? cdef complex c = 1+2j >>>> ? ? ? cval =&c.cval >>>> ? ? ? cval.real, cval.imag = 10, 20 >>>> ? ? ? print (c.real, c.imag) >>>> >>>> (which also works now, BTW). >>>> >>>> Opinions? >>>> >>> No! No! No! >>> >>> In Python, you cannot assign to real or imag simply because the >>> complex is immutable (like Python ints and floats). For C types, >>> immutability does not apply! >>> >> I understand that. However, I hope that you are talking about the builtin >> Python complex type just like I am. Given that these two fields carry the >> > > No. If you do "cdef complex c", what you have is a C complex type. > > How would you treat "cdef float complex c"? Python has no object for > single precision complex. cdef complex c is (as of a couple of changesets ago?) the Python complex object. I'm not sure I like it--feels a bit like bool or int. - Robert From stefan_ml at behnel.de Fri Dec 3 20:13:33 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 20:13:33 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF93F4B.4000403@student.matnat.uio.no> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> Message-ID: <4CF9415D.2030400@behnel.de> Dag Sverre Seljebotn, 03.12.2010 20:04: > I do have a much bigger problem with the naming here. I find > this very little user-friendly: > > cdef complex x # x is Python object > cdef double complex x # same precision but in C > cdef float complex x # single precision in C > > I really do feel that "int" has kind of set the precedent here, and that > if only "complex" is made available it should be as "double complex" or > similar. I would argue that the precendent is actually "float". We already (safely) alias Python's "float" type to C's "double" type. Doing the same thing for "complex" won't hurt as long as it keeps the same interface, would it? Stefan From robertwb at math.washington.edu Fri Dec 3 20:20:43 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 3 Dec 2010 11:20:43 -0800 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF9415D.2030400@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> Message-ID: On Fri, Dec 3, 2010 at 11:13 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 03.12.2010 20:04: >> I do have a much bigger problem with the naming here. I find >> this very little user-friendly: >> >> cdef complex x # x is Python object >> cdef double complex x # same precision but in C >> cdef float complex x # single precision in C >> >> I really do feel that "int" has kind of set the precedent here, and that >> if only "complex" is made available it should be as "double complex" or >> similar. > > I would argue that the precendent is actually "float". We already (safely) > alias Python's "float" type to C's "double" type. Doing the same thing for > "complex" won't hurt as long as it keeps the same interface, would it? If I write cdef float x there's no Python float object in the picture at all. If we have complex as a builtin type name, I would rather it be a double complex, not a complex object. BTW, I find this bothersome: cdef float x x.foo() I'd much rather that raise a compile time error, but that's somewhat orthogonal. - Robert From stefan_ml at behnel.de Fri Dec 3 20:34:12 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 20:34:12 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> Message-ID: <4CF94634.2040801@behnel.de> Robert Bradshaw, 03.12.2010 20:20: > On Fri, Dec 3, 2010 at 11:13 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 03.12.2010 20:04: >>> I do have a much bigger problem with the naming here. I find >>> this very little user-friendly: >>> >>> cdef complex x # x is Python object >>> cdef double complex x # same precision but in C >>> cdef float complex x # single precision in C >>> >>> I really do feel that "int" has kind of set the precedent here, and that >>> if only "complex" is made available it should be as "double complex" or >>> similar. >> >> I would argue that the precendent is actually "float". We already (safely) >> alias Python's "float" type to C's "double" type. Doing the same thing for >> "complex" won't hurt as long as it keeps the same interface, would it? > > If I write > > cdef float x > > there's no Python float object in the picture at all. Come on, it's almost Christmas. Could we please all get back to understanding each other? ;) I meant: we alias floating point values to C doubles and infer "double" even if we know it's actually a Python float. When we do the same thing for complex values and alias them to "C complex", we won't loose anything. > If we have > complex as a builtin type name, I would rather it be a double complex, > not a complex object. +1 (*wink*) > BTW, I find this bothersome: > > cdef float x > x.foo() > > I'd much rather that raise a compile time error, but that's somewhat > orthogonal. This should keep working, though: >>> 1. .conjugate() 1.0 i.e. cdef double x = 1.0, y y = x.conjugate() should behave as in Python. Same for unbound methods, for example. But I wouldn't mind getting an error for unknown attributes, such as your "foo" above. After all, it's clear at compile time that this won't work at runtime (assuming that there won't be a new Python major release in the meantime ...). Stefan From dagss at student.matnat.uio.no Fri Dec 3 20:40:29 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 03 Dec 2010 20:40:29 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF94634.2040801@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> Message-ID: <4CF947AD.8080403@student.matnat.uio.no> On 12/03/2010 08:34 PM, Stefan Behnel wrote: > Robert Bradshaw, 03.12.2010 20:20: > >> On Fri, Dec 3, 2010 at 11:13 AM, Stefan Behnel wrote: >> >>> Dag Sverre Seljebotn, 03.12.2010 20:04: >>> >>>> I do have a much bigger problem with the naming here. I find >>>> this very little user-friendly: >>>> >>>> cdef complex x # x is Python object >>>> cdef double complex x # same precision but in C >>>> cdef float complex x # single precision in C >>>> >>>> I really do feel that "int" has kind of set the precedent here, and that >>>> if only "complex" is made available it should be as "double complex" or >>>> similar. >>>> >>> I would argue that the precendent is actually "float". We already (safely) >>> alias Python's "float" type to C's "double" type. Doing the same thing for >>> "complex" won't hurt as long as it keeps the same interface, would it? >>> >> If I write >> >> cdef float x >> >> there's no Python float object in the picture at all. >> > Come on, it's almost Christmas. Could we please all get back to > understanding each other? ;) > > I meant: we alias floating point values to C doubles and infer "double" > even if we know it's actually a Python float. When we do the same thing for > complex values and alias them to "C complex", we won't loose anything. > Well, you don't loose anything with cdef double complex x = obj either, and it just seems a bit confusing to let "x.real = ..." have subtly different meaning. How about this: When you write x.real = 4 it actually "means" x = 4 + x.imag * 1j similar to how += works with immutable objects. That is: We allow it to happen, but it should be understood that no Python object is actually modified, but that the behaviour is that the "object" is replaced.. Dag Sverre From dagss at student.matnat.uio.no Fri Dec 3 20:47:46 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 03 Dec 2010 20:47:46 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF94634.2040801@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> Message-ID: <4CF94962.90008@student.matnat.uio.no> On 12/03/2010 08:34 PM, Stefan Behnel wrote: > Robert Bradshaw, 03.12.2010 20:20: > >> On Fri, Dec 3, 2010 at 11:13 AM, Stefan Behnel wrote: >> >>> Dag Sverre Seljebotn, 03.12.2010 20:04: >>> >>>> I do have a much bigger problem with the naming here. I find >>>> this very little user-friendly: >>>> >>>> cdef complex x # x is Python object >>>> cdef double complex x # same precision but in C >>>> cdef float complex x # single precision in C >>>> >>>> I really do feel that "int" has kind of set the precedent here, and that >>>> if only "complex" is made available it should be as "double complex" or >>>> similar. >>>> >>> I would argue that the precendent is actually "float". We already (safely) >>> alias Python's "float" type to C's "double" type. Doing the same thing for >>> "complex" won't hurt as long as it keeps the same interface, would it? >>> >> If I write >> >> cdef float x >> >> there's no Python float object in the picture at all. >> > Come on, it's almost Christmas. Could we please all get back to > understanding each other? ;) > > I meant: we alias floating point values to C doubles and infer "double" > even if we know it's actually a Python float. When we do the same thing for > complex values and alias them to "C complex", we won't loose anything. > You do in fask risk loosing the id(obj). At least one library I know (pymc) does its caching by identity and not equality (although one can hope built-in scalars is an exception). (I really don't think we should design around such usecases, I'm just mentioning it :-) ) Dag Sverre From stefan_ml at behnel.de Fri Dec 3 20:48:59 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 20:48:59 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF947AD.8080403@student.matnat.uio.no> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> <4CF947AD.8080403@student.matnat.uio.no> Message-ID: <4CF949AB.3010006@behnel.de> Dag Sverre Seljebotn, 03.12.2010 20:40: > Well, you don't loose anything with > > cdef double complex x = obj > > either, and it just seems a bit confusing to let "x.real = ..." have > subtly different meaning. True. Maybe aliasing "complex" to "double complex" isn't a simple after all. > How about this: When you write > > x.real = 4 > > it actually "means" > > x = 4 + x.imag * 1j > > similar to how += works with immutable objects. That is: We allow it to > happen, but it should be understood that no Python object is actually > modified, but that the behaviour is that the "object" is replaced.. Yuck! That's evil. Stefan From robertwb at math.washington.edu Fri Dec 3 21:03:13 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 3 Dec 2010 12:03:13 -0800 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF949AB.3010006@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> <4CF947AD.8080403@student.matnat.uio.no> <4CF949AB.3010006@behnel.de> Message-ID: On Fri, Dec 3, 2010 at 11:48 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 03.12.2010 20:40: >> Well, you don't loose anything with >> >> cdef double complex x = obj >> >> either, and it just seems a bit confusing to let "x.real = ..." have >> subtly different meaning. > > True. Maybe aliasing "complex" to "double complex" isn't a simple after all. > > >> How about this: When you write >> >> x.real = 4 >> >> it actually "means" >> >> x = 4 + x.imag * 1j >> >> similar to how += works with immutable objects. That is: We allow it to >> happen, but it should be understood that no Python object is actually >> modified, but that the behaviour is that the "object" is replaced.. > > Yuck! That's evil. Yeah, I'm not a fan of that either. My inclination would be to have complex be a simple alias for complex double, and if one wants the Python object type one can cimport it from cpython. -1 to having complex be a Python object in the builtin namespace, or some hybrid object that's a complex double under the hood but has the (incomplete?) semantics of a complex object. In this respect the Python complex object type would be like the Python long, float, and int types. They're rarely what the user actually wants anyways. - Robert From stefan_ml at behnel.de Fri Dec 3 21:08:51 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 21:08:51 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: <4CF949AB.3010006@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> <4CF947AD.8080403@student.matnat.uio.no> <4CF949AB.3010006@behnel.de> Message-ID: <4CF94E53.7080009@behnel.de> Stefan Behnel, 03.12.2010 20:48: > Dag Sverre Seljebotn, 03.12.2010 20:40: >> Well, you don't loose anything with >> >> cdef double complex x = obj >> >> either, and it just seems a bit confusing to let "x.real = ..." have >> subtly different meaning. > > True. Maybe aliasing "complex" to "double complex" isn't a simple after all. ... except when it is. ;) Assuming that we always alias complex Python values to "double complex", the above doesn't actually hurt us. When users define a variable as type "complex", it's fine to make it mutable because he/she has just taken it out of the control of normal (non statically typed) Python behaviour. So only values for which Cython has no type information or which are explicitly defined as "object" will stay immutable because either Cython or Python will forbid assignments. Everything else is represented by a (mutable) C type anyway. Stefan From stefan_ml at behnel.de Fri Dec 3 21:10:31 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Dec 2010 21:10:31 +0100 Subject: [Cython] Should fields of builtin types mimic Python's read-only behaviour? In-Reply-To: References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> <4CF947AD.8080403@student.matnat.uio.no> <4CF949AB.3010006@behnel.de> Message-ID: <4CF94EB7.50008@behnel.de> Robert Bradshaw, 03.12.2010 21:03: > My inclination would be to have complex be a simple alias for complex > double, and if one wants the Python object type one can cimport it > from cpython. -1 to having complex be a Python object in the builtin > namespace, or some hybrid object that's a complex double under the > hood but has the (incomplete?) semantics of a complex object. In this > respect the Python complex object type would be like the Python long, > float, and int types. They're rarely what the user actually wants > anyways. Fine with me. Stefan From ondrej at certik.cz Sat Dec 4 03:39:50 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 3 Dec 2010 18:39:50 -0800 Subject: [Cython] libc.math implemented Message-ID: Hi, the pull request with the code is here: https://github.com/cython/cython/pull/2 Ondrej From ondrej at certik.cz Sat Dec 4 05:00:53 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 3 Dec 2010 20:00:53 -0800 Subject: [Cython] make cython stop on the first error Message-ID: Hi, I was missing the option to tell Cython to stop on the first error. So I have implemented it: https://github.com/cython/cython/pull/3 Ondrej From stefan_ml at behnel.de Sat Dec 4 11:24:56 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Dec 2010 11:24:56 +0100 Subject: [Cython] optimised "complex" type (was: Should fields of builtin types mimic Python's read-only behaviour?) In-Reply-To: References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> <4CF947AD.8080403@student.matnat.uio.no> <4CF949AB.3010006@behnel.de> Message-ID: <4CFA16F8.4040605@behnel.de> Robert Bradshaw, 03.12.2010 21:03: > My inclination would be to have complex be a simple alias for complex > double, and if one wants the Python object type one can cimport it > from cpython. -1 to having complex be a Python object in the builtin > namespace, or some hybrid object that's a complex double under the > hood but has the (incomplete?) semantics of a complex object. In this > respect the Python complex object type would be like the Python long, > float, and int types. They're rarely what the user actually wants > anyways. Changes are here: http://hg.cython.org/cython-devel/rev/e9e285544bf2 http://hg.cython.org/cython-devel/rev/06e29edc9f59 I'm not completely happy with the redeclaration of the type in cpython/complex.pxd. These things are always clumsy and also kill a couple of optimisations. Then again, the best optimisation is to not import the type from there in the first place... One thing to note about code like this: def dostuff(complex x): # ... Before, it simply checked the type of the input value and rejected any non-complex objects. Now, it converts the input to a C complex value before it extracts its real/imag parts. This means that it will accept more input types, basically everything that Python can coerce to complex. Likely not a drawback, but who knows what user code looks like... Stefan From stefan_ml at behnel.de Sat Dec 4 13:37:34 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Dec 2010 13:37:34 +0100 Subject: [Cython] Cython 0.13.1 In-Reply-To: References: Message-ID: <4CFA360E.5080407@behnel.de> Robert Bradshaw, 03.11.2010 18:18: > I think it's time we start thinking about rolling out another release. > [...] How does everyone feel about a target date of this time next month? Time for a release candidate? We don't have release notes yet. I think there's *tons* of new stuff that's worth listing. Anyone for a start? The last ones were here: http://wiki.cython.org/ReleaseNotes-0.13 I don't think trac is complete, but here's what it has to say about that milestone: http://trac.cython.org/cython_trac/query?status=closed&order=priority&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&milestone=0.13.1 Stefan From robertwb at math.washington.edu Sat Dec 4 18:25:46 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 4 Dec 2010 09:25:46 -0800 Subject: [Cython] Cython 0.13.1 In-Reply-To: <4CFA360E.5080407@behnel.de> References: <4CFA360E.5080407@behnel.de> Message-ID: On Sat, Dec 4, 2010 at 4:37 AM, Stefan Behnel wrote: > Robert Bradshaw, 03.11.2010 18:18: >> I think it's time we start thinking about rolling out another release. >> [...] How does everyone feel about a target date of this time next month? > > Time for a release candidate? I've been thinking about this too. Soon. Maybe even later today. I've got a tiny bit more on the build system that I want to get in, Mark's new gdb stuff, and a couple of other odds and ends. > We don't have release notes yet. I think there's *tons* of new stuff that's > worth listing. Anyone for a start? The last ones were here: > > http://wiki.cython.org/ReleaseNotes-0.13 I'm thinking there's so much new stuff that we should call this 0.14. http://wiki.cython.org/ReleaseNotes-0.14 > I don't think trac is complete, but here's what it has to say about that > milestone: > > http://trac.cython.org/cython_trac/query?status=closed&order=priority&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&milestone=0.13.1 I haven't been very good lately about adding my changes there, I should retroactively do that. - Robert From robertwb at math.washington.edu Sat Dec 4 18:27:32 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 4 Dec 2010 09:27:32 -0800 Subject: [Cython] optimised "complex" type (was: Should fields of builtin types mimic Python's read-only behaviour?) In-Reply-To: <4CFA16F8.4040605@behnel.de> References: <4CF932B0.8010108@behnel.de> <4CF93CAB.7070704@behnel.de> <4CF93D74.7050803@student.matnat.uio.no> <4CF93F4B.4000403@student.matnat.uio.no> <4CF9415D.2030400@behnel.de> <4CF94634.2040801@behnel.de> <4CF947AD.8080403@student.matnat.uio.no> <4CF949AB.3010006@behnel.de> <4CFA16F8.4040605@behnel.de> Message-ID: On Sat, Dec 4, 2010 at 2:24 AM, Stefan Behnel wrote: > Robert Bradshaw, 03.12.2010 21:03: >> My inclination would be to have complex be a simple alias for complex >> double, and if one wants the Python object type one can cimport it >> from cpython. -1 to having complex be a Python object in the builtin >> namespace, or some hybrid object that's a complex double under the >> hood but has the (incomplete?) semantics of a complex object. In this >> respect the Python complex object type would be like the Python long, >> float, and int types. They're rarely what the user actually wants >> anyways. > > Changes are here: > > http://hg.cython.org/cython-devel/rev/e9e285544bf2 > http://hg.cython.org/cython-devel/rev/06e29edc9f59 Thanks. > I'm not completely happy with the redeclaration of the type in > cpython/complex.pxd. These things are always clumsy and also kill a couple > of optimisations. Then again, the best optimisation is to not import the > type from there in the first place... Yep. > One thing to note about code like this: > > ? ? def dostuff(complex x): > ? ? ? ? # ... > > Before, it simply checked the type of the input value and rejected any > non-complex objects. Now, it converts the input to a C complex value before > it extracts its real/imag parts. This means that it will accept more input > types, basically everything that Python can coerce to complex. Likely not a > drawback, but who knows what user code looks like... I think it'll be pretty safe. - Robert From stefan_ml at behnel.de Sat Dec 4 19:03:29 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Dec 2010 19:03:29 +0100 Subject: [Cython] Cython 0.13.1 In-Reply-To: References: <4CFA360E.5080407@behnel.de> Message-ID: <4CFA8271.7080201@behnel.de> Robert Bradshaw, 04.12.2010 18:25: > On Sat, Dec 4, 2010 at 4:37 AM, Stefan Behnel wrote: >> Robert Bradshaw, 03.11.2010 18:18: >>> I think it's time we start thinking about rolling out another release. >>> [...] How does everyone feel about a target date of this time next month? >> >> Time for a release candidate? > > I've been thinking about this too. Soon. Maybe even later today. I've > got a tiny bit more on the build system that I want to get in, Mark's > new gdb stuff, and a couple of other odds and ends. Sure. > I'm thinking there's so much new stuff that we should call this 0.14. +1 >> I don't think trac is complete, but here's what it has to say about that >> milestone: >> >> http://trac.cython.org/cython_trac/query?status=closed&order=priority&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&milestone=0.13.1 > > I haven't been very good lately about adding my changes there, I > should retroactively do that. I'll take another look through the history as well. Stefan From vitja.makarov at gmail.com Sun Dec 5 00:38:50 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 5 Dec 2010 02:38:50 +0300 Subject: [Cython] make cython stop on the first error In-Reply-To: References: Message-ID: 2010/12/4 Ondrej Certik : > Hi, > > I was missing the option to tell Cython to stop on the first error. So > I have implemented it: > > https://github.com/cython/cython/pull/3 > > Ondrej > Hi! Is that a good idea to call sys.exit() in Errors.py maybe there is another way to stop compilation? -- vitja. From ondrej at certik.cz Sun Dec 5 01:27:29 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 4 Dec 2010 16:27:29 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: References: Message-ID: On Sat, Dec 4, 2010 at 3:38 PM, Vitja Makarov wrote: > 2010/12/4 Ondrej Certik : >> Hi, >> >> I was missing the option to tell Cython to stop on the first error. So >> I have implemented it: >> >> https://github.com/cython/cython/pull/3 >> >> Ondrej >> > > Hi! > > Is that a good idea to call sys.exit() in Errors.py maybe there is > another way to stop compilation? Yes, that is something that I was not sure when I coded it, as I am not too familiar with Cython internals. Ondrej From robertwb at math.washington.edu Sun Dec 5 05:18:14 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 4 Dec 2010 20:18:14 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: References: Message-ID: On Sat, Dec 4, 2010 at 4:27 PM, Ondrej Certik wrote: > On Sat, Dec 4, 2010 at 3:38 PM, Vitja Makarov wrote: >> 2010/12/4 Ondrej Certik : >>> Hi, >>> >>> I was missing the option to tell Cython to stop on the first error. So >>> I have implemented it: >>> >>> https://github.com/cython/cython/pull/3 >>> >>> Ondrej >>> >> >> Hi! >> >> Is that a good idea to call sys.exit() in Errors.py maybe there is >> another way to stop compilation? > > Yes, that is something that I was not sure when I coded it, as I am > not too familiar with Cython internals. Hmm... that's a good point. I wasn't thinking about that. Actually the cleanest would probably be to throw an exception and catch it higher up. This woud have the advantage of being able to do some cleanup (e.g. so you don't have half-generated C files). - Robert From dagss at student.matnat.uio.no Sun Dec 5 06:25:40 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 05 Dec 2010 06:25:40 +0100 Subject: [Cython] make cython stop on the first error In-Reply-To: References: Message-ID: <4CFB2254.80705@student.matnat.uio.no> On 12/05/2010 01:27 AM, Ondrej Certik wrote: > On Sat, Dec 4, 2010 at 3:38 PM, Vitja Makarov wrote: > >> 2010/12/4 Ondrej Certik: >> >>> Hi, >>> >>> I was missing the option to tell Cython to stop on the first error. So >>> I have implemented it: >>> >>> https://github.com/cython/cython/pull/3 >>> >>> Ondrej >>> >>> >> Hi! >> >> Is that a good idea to call sys.exit() in Errors.py maybe there is >> another way to stop compilation? >> > Yes, that is something that I was not sure when I coded it, as I am > not too familiar with Cython internals. > The usecases are more important than the internals: Imagine for a moment what this will to do pyximport, or build systems able to launch Cython in-process (SCons, waf). Dag Sverre From vitja.makarov at gmail.com Sun Dec 5 08:25:11 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 5 Dec 2010 10:25:11 +0300 Subject: [Cython] trailing spaces Message-ID: Hi! There are lots of trailing spaces be it is time to remove them? -- vitja. From robertwb at math.washington.edu Sun Dec 5 08:37:42 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 4 Dec 2010 23:37:42 -0800 Subject: [Cython] trailing spaces In-Reply-To: References: Message-ID: Looking at the amount of recent activity, I'd say not now. Maybe that'd be a good thing to do immediately before the actual release. - Robert On Sat, Dec 4, 2010 at 11:25 PM, Vitja Makarov wrote: > Hi! > > There are lots of trailing spaces be it is time to remove them? > > -- > vitja. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From vitja.makarov at gmail.com Sun Dec 5 08:41:25 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 5 Dec 2010 10:41:25 +0300 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: <4CF8A8A1.7070808@behnel.de> References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> Message-ID: 2010/12/3 Stefan Behnel : > Vitja Makarov, 03.12.2010 08:56: >> What's about #605? > > Comment posted there. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > I've updated patch. I removed pretty_name, bad idea should use name instead in pydeclare_lambda. `is_lambda` is now called `is_anonymous`, that means that entry itself doesn't know how it will be named inside scope, so its anonymous (both lambda and pyfunctions) -- vitja. From robertwb at math.washington.edu Sun Dec 5 08:49:33 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 4 Dec 2010 23:49:33 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: <4CFB2254.80705@student.matnat.uio.no> References: <4CFB2254.80705@student.matnat.uio.no> Message-ID: On Sat, Dec 4, 2010 at 9:25 PM, Dag Sverre Seljebotn wrote: > On 12/05/2010 01:27 AM, Ondrej Certik wrote: >> On Sat, Dec 4, 2010 at 3:38 PM, Vitja Makarov ?wrote: >> >>> 2010/12/4 Ondrej Certik: >>> >>>> Hi, >>>> >>>> I was missing the option to tell Cython to stop on the first error. So >>>> I have implemented it: >>>> >>>> https://github.com/cython/cython/pull/3 >>>> >>>> Ondrej >>>> >>>> >>> Hi! >>> >>> Is that a good idea to call sys.exit() in Errors.py maybe there is >>> another way to stop compilation? >>> >> Yes, that is something that I was not sure when I coded it, as I am >> not too familiar with Cython internals. >> > > The usecases are more important than the internals: Imagine for a moment > what this will to do pyximport, or build systems able to launch Cython > in-process (SCons, waf). I've made it raise the same error it does between phases to signal the compiler should abort. - Robert From vitja.makarov at gmail.com Sun Dec 5 08:53:18 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 5 Dec 2010 10:53:18 +0300 Subject: [Cython] trailing spaces In-Reply-To: References: Message-ID: 2010/12/5 Robert Bradshaw : > Looking at the amount of recent activity, I'd say not now. Maybe > that'd be a good thing to do immediately before the actual release. > > - Robert > > On Sat, Dec 4, 2010 at 11:25 PM, Vitja Makarov wrote: >> Hi! >> >> There are lots of trailing spaces be it is time to remove them? >> >> -- >> vitja. >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> Sure ) -- vitja. From robertwb at math.washington.edu Sun Dec 5 13:47:24 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 5 Dec 2010 04:47:24 -0800 Subject: [Cython] Cython 0.13.1 In-Reply-To: <4CFA8271.7080201@behnel.de> References: <4CFA360E.5080407@behnel.de> <4CFA8271.7080201@behnel.de> Message-ID: On Sat, Dec 4, 2010 at 10:03 AM, Stefan Behnel wrote: > Robert Bradshaw, 04.12.2010 18:25: >> On Sat, Dec 4, 2010 at 4:37 AM, Stefan Behnel wrote: >>> Robert Bradshaw, 03.11.2010 18:18: >>>> I think it's time we start thinking about rolling out another release. >>>> [...] How does everyone feel about a target date of this time next month? >>> >>> Time for a release candidate? >> >> I've been thinking about this too. Soon. Maybe even later today. I've >> got a tiny bit more on the build system that I want to get in, Mark's >> new gdb stuff, and a couple of other odds and ends. > > Sure. > > >> I'm thinking there's so much new stuff that we should call this 0.14. > > +1 > > >>> I don't think trac is complete, but here's what it has to say about that >>> milestone: >>> >>> http://trac.cython.org/cython_trac/query?status=closed&order=priority&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&milestone=0.13.1 >> >> I haven't been very good lately about adding my changes there, I >> should retroactively do that. > > I'll take another look through the history as well. Thanks for starting to get the release notes in shape. I've posted an alpha at http://cython.org/release/Cython-0.14.alpha0.tar.gz This is an alpha because I hope we can get that great gdb code in before the release (it looks good, but I'd like to have as many people test it out as possible, and there's an issue with side-effecting the tests to be too verbose). - Robert From stefan_ml at behnel.de Sun Dec 5 14:15:19 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 05 Dec 2010 14:15:19 +0100 Subject: [Cython] Cython 0.13.1 In-Reply-To: References: <4CFA360E.5080407@behnel.de> <4CFA8271.7080201@behnel.de> Message-ID: <4CFB9067.9020506@behnel.de> Robert Bradshaw, 05.12.2010 13:47: > alpha at http://cython.org/release/Cython-0.14.alpha0.tar.gz This is > an alpha because I hope we can get that great gdb code in before the > release (it looks good, but I'd like to have as many people test it > out as possible, and there's an issue with side-effecting the tests to > be too verbose). I haven't tested it yet, mainly because I don't have the latest gdb version installed. When I tried building lxml with that branch, I ran into a couple of compiler crashes that Mark resolved, but I can't tell if there aren't any left. I also didn't review the code in any way. AFAIC, I'm not sure I want this right in 0.14, but that's more because I can't really comment on it. I certainly agree that it needs more testing. Stefan From markflorisson88 at gmail.com Sun Dec 5 15:11:08 2010 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 5 Dec 2010 15:11:08 +0100 Subject: [Cython] Cython 0.13.1 In-Reply-To: <4CFB9067.9020506@behnel.de> References: <4CFA360E.5080407@behnel.de> <4CFA8271.7080201@behnel.de> <4CFB9067.9020506@behnel.de> Message-ID: On 5 December 2010 14:15, Stefan Behnel wrote: > Robert Bradshaw, 05.12.2010 13:47: > > alpha at http://cython.org/release/Cython-0.14.alpha0.tar.gz This is > > an alpha because I hope we can get that great gdb code in before the > > release (it looks good, but I'd like to have as many people test it > > out as possible, and there's an issue with side-effecting the tests to > > be too verbose). > The verbose-tests issue should be solved now. > I haven't tested it yet, mainly because I don't have the latest gdb version > installed. When I tried building lxml with that branch, I ran into a couple > of compiler crashes that Mark resolved, but I can't tell if there aren't > any left. I also didn't review the code in any way. > > AFAIC, I'm not sure I want this right in 0.14, but that's more because I > can't really comment on it. I certainly agree that it needs more testing. I agree that it would be nice if more people would test it, however I don't think there should be any compiler crashes anymore. In any case, even if there would still be compiler crashes, users could just not enable the --gdb flag (or --pyrex-gdb to build_ext in case setup.py is used). But I agree that it should be tested more widely before merging it. The documentation (currently only on hudson but I presume it will make it to docs.cython.org with the new release) already contains a link to my git branch so users that are interested could still use it. > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Kind regards, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101205/be2f1d5b/attachment.htm From stefan_ml at behnel.de Sun Dec 5 16:55:11 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 05 Dec 2010 16:55:11 +0100 Subject: [Cython] Cython 0.13.1 In-Reply-To: References: <4CFA360E.5080407@behnel.de> <4CFA8271.7080201@behnel.de> <4CFB9067.9020506@behnel.de> Message-ID: <4CFBB5DF.6010205@behnel.de> mark florisson, 05.12.2010 15:11: > On 5 December 2010 14:15, Stefan Behnel wrote: >> I haven't tested it yet, mainly because I don't have the latest gdb version >> installed. When I tried building lxml with that branch, I ran into a couple >> of compiler crashes that Mark resolved, but I can't tell if there aren't >> any left. I also didn't review the code in any way. >> >> AFAIC, I'm not sure I want this right in 0.14, but that's more because I >> can't really comment on it. I certainly agree that it needs more testing. > [...] > But I agree that it should be tested more widely before merging it. The > documentation (currently only on hudson but I presume it will make it > to docs.cython.org with the new release) already contains a link to my > git branch so users that are interested could still use it. Then I would suggest we keep the docs as they are now to make your gdb support visible, but with a big note that this hasn't been merged yet and keep your branch alive for a bit longer after the release. I wouldn't mind merging it for 0.14.1 when it seems mature enough, and I hope that release isn't too far off. 0.14 is a release with many, many new features, so it's quite likely that people find bugs quickly once it's out in the wild. Stefan From stefan_ml at behnel.de Sun Dec 5 19:16:49 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 05 Dec 2010 19:16:49 +0100 Subject: [Cython] General generator expressions Message-ID: <4CFBD711.5060206@behnel.de> Hi, I just noticed that Vitja's closure refactoring brought us closer to supporting generator expressions (before supporting generators and coroutines). Here's what's missing: 1) transform a GeneratorExpressionNode into a DefNode or a subclass (obviously excluding inlined generator expressions) 2) generate the resume code at the start of the function body 3) store temps in closures. For simplicity, this can be done with a tuple (which supports NULL values to a certain extent). That way, we do not need to declare each temp separately as a closure field. The remaining tricky bit is then to figure out the maximum number of temps that are alive at a YieldExprNode (ok, that's actually easy) and to pass that number on to the closure class that needs to allocate a sufficiently large tuple. That doesn't sound too hard. The generators CEP has the details for each step. http://wiki.cython.org/enhancements/generators Going for generator expressions before implementing generators has the advantage that it gives us most of the infrastructure without requiring the additional steps of passing initial arguments into the generator and implementing the complete coroutine protocol (the yield nodes cannot return values, for example). Anyone with a little time to spare? Stefan From markflorisson88 at gmail.com Sun Dec 5 19:55:23 2010 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 5 Dec 2010 19:55:23 +0100 Subject: [Cython] Cython 0.13.1 In-Reply-To: <4CFBB5DF.6010205@behnel.de> References: <4CFA360E.5080407@behnel.de> <4CFA8271.7080201@behnel.de> <4CFB9067.9020506@behnel.de> <4CFBB5DF.6010205@behnel.de> Message-ID: On 5 December 2010 16:55, Stefan Behnel wrote: > mark florisson, 05.12.2010 15:11: > > On 5 December 2010 14:15, Stefan Behnel wrote: > >> I haven't tested it yet, mainly because I don't have the latest gdb > version > >> installed. When I tried building lxml with that branch, I ran into a > couple > >> of compiler crashes that Mark resolved, but I can't tell if there aren't > >> any left. I also didn't review the code in any way. > >> > >> AFAIC, I'm not sure I want this right in 0.14, but that's more because I > >> can't really comment on it. I certainly agree that it needs more > testing. > > [...] > > But I agree that it should be tested more widely before merging it. The > > documentation (currently only on hudson but I presume it will make it > > to docs.cython.org with the new release) already contains a link to my > > git branch so users that are interested could still use it. > > Then I would suggest we keep the docs as they are now to make your gdb > support visible, but with a big note that this hasn't been merged yet and > keep your branch alive for a bit longer after the release. I wouldn't mind > merging it for 0.14.1 when it seems mature enough, and I hope that release > isn't too far off. 0.14 is a release with many, many new features, so it's > quite likely that people find bugs quickly once it's out in the wild. Sounds good, I updated the docs accordingly. Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101205/eb6dd8d5/attachment.htm From vitja.makarov at gmail.com Sun Dec 5 20:12:30 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 5 Dec 2010 22:12:30 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4CFBD711.5060206@behnel.de> References: <4CFBD711.5060206@behnel.de> Message-ID: 2010/12/5 Stefan Behnel : > Hi, > > I just noticed that Vitja's closure refactoring brought us closer to > supporting generator expressions (before supporting generators and > coroutines). Here's what's missing: > > 1) transform a GeneratorExpressionNode into a DefNode or a subclass > (obviously excluding inlined generator expressions) > > 2) generate the resume code at the start of the function body > > 3) store temps in closures. For simplicity, this can be done with a tuple > (which supports NULL values to a certain extent). That way, we do not need > to declare each temp separately as a closure field. The remaining tricky > bit is then to figure out the maximum number of temps that are alive at a > YieldExprNode (ok, that's actually easy) and to pass that number on to the > closure class that needs to allocate a sufficiently large tuple. > > That doesn't sound too hard. > > The generators CEP has the details for each step. > > http://wiki.cython.org/enhancements/generators > > Going for generator expressions before implementing generators has the > advantage that it gives us most of the infrastructure without requiring the > additional steps of passing initial arguments into the generator and > implementing the complete coroutine protocol (the yield nodes cannot return > values, for example). > > Anyone with a little time to spare? > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Hi! I've started woring on generators on friday. Now it can generate some code, but it doesn't work yet, what is done: - switch() for resuming and argument parsing - yield stuff - closure temp allocation I'm going to put all the generator stuff into closure Generator function: Creates closure with all temps, args, variables, and all the internal stuff: Also closure have all the required generator methods __iter__(), __nextiter__() and so on. def my_generator(self, args, kwargs): closure = my_generator_closure_new() closure.is_running = 0 closure.resume_label = 0 closure.args = args closure.kwargs = kwargs closure.outer_scope = self closure.generator_body = my_generator_body return closure generator body: def my_generator_body(closure, value, is_exception): # switch case switch (closure.resume_lable): case 0: goto parse_arguments; case 1: goto resume_from_yield1; parse_arguments: # function body here About temps: now temps are allocated in closure using declare_var() and it works fine. I'm not sure about manage_ref flag I guess that temps with manage_ref=False shouldn't be allocated on closure (am I right?) I'm not sure that tuple should be better for temps. TODO: - create C function that returns closure - generator utility code I have problem with closures and generators. Now I want to implement closure structure this way: struct abstract_generator { PY_OBJECT_HEAD /* generator internal vars */ PyObject *(*body)(struct closure *, PyObject *, int); int resume_label; } struct closure { /* abstract_generator */ PY_OBJECT_HEAD /* generator internal vars */ PyObject *(*body)(struct closure *, PyObject *, int); int resume_label; ,,,, /* Closure stuff */ } ; and generator utility function will work like this: PyObject *abstract_generator_next(PyObject *self, PyObject *args) { struct abstract_generator *generator = (struct abstract_generator *) self; PyObject *retval; // check running retval = generator->body(self, None, 0); // unlock running return retval; } But this way should be much better, don't know if that is possible btw. struct closure { struct abstract_generator generator; /* closure stuff here */ ... } ; I add MarkGenerator visitor, that sets is_generator flag on DefNode and checks for returns/yields. Then CreateClosureClasses will create correct closure class for generator. -- vitja. From stefan_ml at behnel.de Sun Dec 5 22:13:03 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 05 Dec 2010 22:13:03 +0100 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> Message-ID: <4CFC005F.2070103@behnel.de> Vitja Makarov, 05.12.2010 08:41: > I've updated patch. > > I removed pretty_name, bad idea should use name instead in pydeclare_lambda. > `is_lambda` is now called `is_anonymous`, that means that entry itself > doesn't know how it will be named inside scope, so its anonymous (both > lambda and pyfunctions) That's ok and the patch looks good to me now. However, it lacks error tests and the modified code actually fails to detect redefinitions of cdef functions as Python functions and vice versa, which is not supported (and IMHO doesn't make sense either). Similarly, redefinitions in cdef classes aren't currently supported but are not detected either. I've attached a patch that currently breaks cpdef functions, but that handles (and tests) at least some of the error cases. I'm not sure what to do to fix the cpdef functions, though. BTW, note that there's a typo in the test name. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: pyfunc_redef_tests.patch Type: text/x-patch Size: 3898 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20101205/acc942ea/attachment.bin From ondrej at certik.cz Mon Dec 6 02:42:11 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 5 Dec 2010 17:42:11 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: References: <4CFB2254.80705@student.matnat.uio.no> Message-ID: On Sat, Dec 4, 2010 at 11:49 PM, Robert Bradshaw wrote: > On Sat, Dec 4, 2010 at 9:25 PM, Dag Sverre Seljebotn > wrote: >> On 12/05/2010 01:27 AM, Ondrej Certik wrote: >>> On Sat, Dec 4, 2010 at 3:38 PM, Vitja Makarov ?wrote: >>> >>>> 2010/12/4 Ondrej Certik: >>>> >>>>> Hi, >>>>> >>>>> I was missing the option to tell Cython to stop on the first error. So >>>>> I have implemented it: >>>>> >>>>> https://github.com/cython/cython/pull/3 >>>>> >>>>> Ondrej >>>>> >>>>> >>>> Hi! >>>> >>>> Is that a good idea to call sys.exit() in Errors.py maybe there is >>>> another way to stop compilation? >>>> >>> Yes, that is something that I was not sure when I coded it, as I am >>> not too familiar with Cython internals. >>> >> >> The usecases are more important than the internals: Imagine for a moment >> what this will to do pyximport, or build systems able to launch Cython >> in-process (SCons, waf). > > I've made it raise the same error it does between phases to signal the > compiler should abort. After this "abort" change, here is how it looks like: ondrej at crow:~/repos/hermes/hermes1d/hermes1d(fix)$ make [ 4%] Built target cython_utils [ 8%] Built target numpy_utils [ 93%] Built target hermes1d-debug [ 95%] Cythonizing h1d_wrapper.pyx Error compiling Cython file: ------------------------------------------------------------ ... prj_type = hermes1d.H1D_H1_ortho_global else: raise ValueError("Unknown projection type") global _A _A = f hermes1d.assemble_projection_matrix_rhs(mesh.thisptr, A.thisptr, ^ ------------------------------------------------------------ /home/ondrej/repos/hermes/hermes1d/hermes1d/h1d_wrapper/h1d_wrapper.pyx:369:48: Cannot convert 'Space *' to Python object Error compiling Cython file: ------------------------------------------------------------ ... prj_type = hermes1d.H1D_H1_ortho_global else: raise ValueError("Unknown projection type") global _A _A = f hermes1d.assemble_projection_matrix_rhs(mesh.thisptr, A.thisptr, ^ ------------------------------------------------------------ /home/ondrej/repos/hermes/hermes1d/hermes1d/h1d_wrapper/h1d_wrapper.pyx:369:48: Compiler crash in AnalyseExpressionsTransform ModuleNode.body = StatListNode(h1d_wrapper.pyx:6:0) StatListNode.stats[19] = StatListNode(h1d_wrapper.pyx:358:0) StatListNode.stats[0] = DefNode(h1d_wrapper.pyx:358:0, modifiers = [...]/0, name = u'assemble_projection_matrix_rhs', num_required_args = 4, reqd_kw_flags_cname = '0') File 'Nodes.py', line 351, in analyse_expressions: StatListNode(h1d_wrapper.pyx:360:4) File 'Nodes.py', line 3346, in analyse_expressions: ExprStatNode(h1d_wrapper.pyx:369:43) File 'ExprNodes.py', line 322, in analyse_expressions: SimpleCallNode(h1d_wrapper.pyx:369:43, analysed = True, use_managed_ref = True) File 'ExprNodes.py', line 2865, in analyse_types: SimpleCallNode(h1d_wrapper.pyx:369:43, analysed = True, use_managed_ref = True) File 'ExprNodes.py', line 3947, in analyse_types: TupleNode(h1d_wrapper.pyx:369:43, is_sequence_constructor = 1, use_managed_ref = True) File 'ExprNodes.py', line 3742, in analyse_types: TupleNode(h1d_wrapper.pyx:369:43, is_sequence_constructor = 1, use_managed_ref = True) File 'ExprNodes.py', line 614, in coerce_to_pyobject: AttributeNode(h1d_wrapper.pyx:369:48, attribute = u'thisptr', is_attribute = 1, member = u'thisptr', needs_none_check = True, op = '->', use_managed_ref = True) File 'ExprNodes.py', line 3317, in coerce_to: AttributeNode(h1d_wrapper.pyx:369:48, attribute = u'thisptr', is_attribute = 1, member = u'thisptr', needs_none_check = True, op = '->', use_managed_ref = True) File 'ExprNodes.py', line 580, in coerce_to: AttributeNode(h1d_wrapper.pyx:369:48, attribute = u'thisptr', is_attribute = 1, member = u'thisptr', needs_none_check = True, op = '->', use_managed_ref = True) File 'ExprNodes.py', line 7076, in __init__: CoerceToPyTypeNode(h1d_wrapper.pyx:369:48, is_temp = 1, use_managed_ref = True) Compiler crash traceback from this point on: File "/home/ondrej/usr/lib/python/Cython/Compiler/ExprNodes.py", line 7076, in __init__ "Cannot convert '%s' to Python object" % arg.type) File "/home/ondrej/usr/lib/python/Cython/Compiler/Errors.py", line 151, in error report_error(err) File "/home/ondrej/usr/lib/python/Cython/Compiler/Errors.py", line 143, in report_error raise InternalError, "abort" InternalError: Internal compiler error: abort Traceback (most recent call last): File "/home/ondrej/usr/bin/cython", line 8, in main(command_line = 1) File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line 773, in main result = compile(sources, options) File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line 748, in compile return compile_multiple(source, options) File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line 720, in compile_multiple result = run_pipeline(source, options) File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line 585, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line 231, in run_pipeline Errors.report_error(err) File "/home/ondrej/usr/lib/python/Cython/Compiler/Errors.py", line 143, in report_error raise InternalError, "abort" Cython.Compiler.Errors.InternalError: Internal compiler error: abort make[2]: *** [hermes1d/h1d_wrapper/h1d_wrapper.cpp] Error 1 make[1]: *** [hermes1d/h1d_wrapper/CMakeFiles/h1d_wrapper.dir/all] Error 2 make: *** [all] Error 2 Now compare to the previous situation: ondrej at crow:~/repos/hermes/hermes1d/hermes1d(fix)$ make [ 4%] Built target cython_utils [ 8%] Built target numpy_utils [ 93%] Built target hermes1d-debug [ 95%] Cythonizing h1d_wrapper.pyx Error compiling Cython file: ------------------------------------------------------------ ... prj_type = hermes1d.H1D_H1_ortho_global else: raise ValueError("Unknown projection type") global _A _A = f hermes1d.assemble_projection_matrix_rhs(mesh.thisptr, A.thisptr, ^ ------------------------------------------------------------ /home/ondrej/repos/hermes/hermes1d/hermes1d/h1d_wrapper/h1d_wrapper.pyx:369:48: Cannot convert 'Space *' to Python object make[2]: *** [hermes1d/h1d_wrapper/h1d_wrapper.cpp] Error 1 make[1]: *** [hermes1d/h1d_wrapper/CMakeFiles/h1d_wrapper.dir/all] Error 2 make: *** [all] Error 2 So I believe that the latter is simpler. So where should we catch the InternalError exception? Maybe we should create a StopOnFirstError exception, and then catch it and print "exiting the compilation due to --fatal-errors command line switch"? Ondrej From stefan_ml at behnel.de Mon Dec 6 06:47:03 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 06:47:03 +0100 Subject: [Cython] make cython stop on the first error In-Reply-To: References: <4CFB2254.80705@student.matnat.uio.no> Message-ID: <4CFC78D7.7080109@behnel.de> Ondrej Certik, 06.12.2010 02:42: > Maybe we should create a StopOnFirstError > exception, and then catch it and print "exiting the compilation due to > --fatal-errors command line switch"? +1, although I'd name it AbortCompilation or something like that, maybe SilentlyAbortCompilation. Stefan From ondrej at certik.cz Mon Dec 6 06:54:40 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 5 Dec 2010 21:54:40 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: <4CFC78D7.7080109@behnel.de> References: <4CFB2254.80705@student.matnat.uio.no> <4CFC78D7.7080109@behnel.de> Message-ID: On Sun, Dec 5, 2010 at 9:47 PM, Stefan Behnel wrote: > Ondrej Certik, 06.12.2010 02:42: >> Maybe we should create a StopOnFirstError >> exception, and then catch it and print "exiting the compilation due to >> --fatal-errors command line switch"? > > +1, although I'd name it AbortCompilation or something like that, maybe > SilentlyAbortCompilation. Ok. Where should the cleanup code go? Into Cython/Compiler/Main.py? Ondrej From vitja.makarov at gmail.com Mon Dec 6 07:05:22 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 6 Dec 2010 09:05:22 +0300 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: <4CFC005F.2070103@behnel.de> References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> <4CFC005F.2070103@behnel.de> Message-ID: 2010/12/6 Stefan Behnel : > Vitja Makarov, 05.12.2010 08:41: >> >> I've updated patch. >> >> I removed pretty_name, bad idea should use name instead in >> pydeclare_lambda. >> `is_lambda` is now called `is_anonymous`, that means that entry itself >> doesn't know how it will be named inside scope, so its anonymous (both >> lambda and pyfunctions) > > That's ok and the patch looks good to me now. > > However, it lacks error tests and the modified code actually fails to detect > redefinitions of cdef functions as Python functions and vice versa, which is > not supported (and IMHO doesn't make sense either). Similarly, redefinitions > in cdef classes aren't currently supported but are not detected either. > > I've attached a patch that currently breaks cpdef functions, but that > handles (and tests) at least some of the error cases. I'm not sure what to > do to fix the cpdef functions, though. > > BTW, note that there's a typo in the test name. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > I can't say it breaks cpdefs, this is "classic" cython behavior. May by we need some flag like is_hardcoded? -- vitja. From stefan_ml at behnel.de Mon Dec 6 07:52:37 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 07:52:37 +0100 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> <4CFC005F.2070103@behnel.de> Message-ID: <4CFC8835.1010500@behnel.de> Vitja Makarov, 06.12.2010 07:05: > 2010/12/6 Stefan Behnel: >> Vitja Makarov, 05.12.2010 08:41: >>> >>> I've updated patch. >>> >>> I removed pretty_name, bad idea should use name instead in >>> pydeclare_lambda. >>> `is_lambda` is now called `is_anonymous`, that means that entry itself >>> doesn't know how it will be named inside scope, so its anonymous (both >>> lambda and pyfunctions) >> >> That's ok and the patch looks good to me now. >> >> However, it lacks error tests and the modified code actually fails to detect >> redefinitions of cdef functions as Python functions and vice versa, which is >> not supported (and IMHO doesn't make sense either). Similarly, redefinitions >> in cdef classes aren't currently supported but are not detected either. >> >> I've attached a patch that currently breaks cpdef functions, but that >> handles (and tests) at least some of the error cases. I'm not sure what to >> do to fix the cpdef functions, though. > > I can't say it breaks cpdefs, this is "classic" cython behavior. Did you try running the test suite with it? > May by we need some flag like is_hardcoded? Which would mean what exactly? Stefan From vitja.makarov at gmail.com Mon Dec 6 08:33:19 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 6 Dec 2010 10:33:19 +0300 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: <4CFC8835.1010500@behnel.de> References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> <4CFC005F.2070103@behnel.de> <4CFC8835.1010500@behnel.de> Message-ID: 2010/12/6 Stefan Behnel : > Vitja Makarov, 06.12.2010 07:05: >> 2010/12/6 Stefan Behnel: >>> Vitja Makarov, 05.12.2010 08:41: >>>> >>>> I've updated patch. >>>> >>>> I removed pretty_name, bad idea should use name instead in >>>> pydeclare_lambda. >>>> `is_lambda` is now called `is_anonymous`, that means that entry itself >>>> doesn't know how it will be named inside scope, so its anonymous (both >>>> lambda and pyfunctions) >>> >>> That's ok and the patch looks good to me now. >>> >>> However, it lacks error tests and the modified code actually fails to detect >>> redefinitions of cdef functions as Python functions and vice versa, which is >>> not supported (and IMHO doesn't make sense either). Similarly, redefinitions >>> in cdef classes aren't currently supported but are not detected either. >>> >>> I've attached a patch that currently breaks cpdef functions, but that >>> handles (and tests) at least some of the error cases. I'm not sure what to >>> do to fix the cpdef functions, though. >> >> I can't say it breaks cpdefs, this is "classic" cython behavior. > > Did you try running the test suite with it? > > I mean that cython wo redefinition patches will copile code like this but gcc will not: cpdef foo(): pass cpdef foo(): pass xxx.c:444: error: redefinition of ?__pyx_f_3xxx_foo? >> May by we need some flag like is_hardcoded? > > Which would mean what exactly? > Means that symbol is defined in method table, bad name for flag. So as I understand when function is cpdef it first declare cfunction, then pycfunction. So entry is always of type CFunction so maybe this flag should be called has_pyfunction? and error message should be raised in _declare_pyfunction() if has_pyfunction flag is set? -- vitja. From stefan_ml at behnel.de Mon Dec 6 08:33:58 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 08:33:58 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> Message-ID: <4CFC91E6.8020604@behnel.de> Vitja Makarov, 05.12.2010 20:12: > I've started woring on generators on friday. Cool. This sounds like a larger code change, though. Try not to submit it as a single patch as that would likely be hard to review. Instead, put up a repo somewhere that others can look at, or submit bundles (as hg calls them) of separate commits. > Now it can generate some code, but it doesn't work yet, what is done: > - switch() for resuming and argument parsing > - yield stuff > - closure temp allocation > > I'm going to put all the generator stuff into closure > > Generator function: > > Creates closure with all temps, args, variables, and all the internal stuff: > Also closure have all the required generator methods __iter__(), > __nextiter__() and so on. Ok, why not. > def my_generator(self, args, kwargs): > closure = my_generator_closure_new() > closure.is_running = 0 > closure.resume_label = 0 > closure.args = args > closure.kwargs = kwargs > closure.outer_scope = self > closure.generator_body = my_generator_body > return closure > > generator body: > > def my_generator_body(closure, value, is_exception): Shouldn't that be a cdef function? > # switch case > switch (closure.resume_lable): > case 0: > goto parse_arguments; > case 1: > goto resume_from_yield1; > parse_arguments: > # function body here Ok, sure, as in the CEP. > About temps: > now temps are allocated in closure using declare_var() and it works fine. I think that could seriously slow down things inside of the generator, though. It means that it needs to jump through the closure indirection for every little value that it keeps around temporarily, including lots of intermediate C results. Also, I would expect that a lot more temps are used inside of the generator body than what actually needs to be kept alive during yields, so a single yield inside of a lengthy generator body could have a tremendous impact on the overall runtime and closure size. If you only store away the live temp values before a yield, you don't even need to do any reference counting for them. It's just a quick bunch of C assignments on yield and resume. That's about the smallest impact you can get. BTW, if you change the object struct of the generator closure anyway, maybe a dynamically sized PyVarObject would be a way to incorporate the temps? > I'm not sure about manage_ref flag I guess that temps with > manage_ref=False shouldn't be allocated on closure (am I right?) "manage_ref=False" (for a Python object value) just means that the reference is either borrowed or will be stolen at some point. I'm not sure that stealing the reference may not happen *after* a yield, but I guess that would produce other problems if the reference isn't kept alive in another way. Nothing to worry about now, carefully crafted tests will eventually catch that. In any case, all live temps (Python refs, borrowed refs and C values) may be needed after the yield, so they must *all* end up in the closure. > TODO: > - create C function that returns closure > - generator utility code At least for generator expressions, the latter wouldn't be needed for now. > I have problem with closures and generators. > > Now I want to implement closure structure this way: > struct abstract_generator { > PY_OBJECT_HEAD > /* generator internal vars */ > PyObject *(*body)(struct closure *, PyObject *, int); > int resume_label; > } I hope you're not trying to do it manually at that level, are you? Declaring the fields on the closure class should just work. > struct closure { > /* abstract_generator */ > PY_OBJECT_HEAD > /* generator internal vars */ > PyObject *(*body)(struct closure *, PyObject *, int); > int resume_label; > ,,,, > /* Closure stuff */ > } ; > > and generator utility function will work like this: > > PyObject *abstract_generator_next(PyObject *self, PyObject *args) > { > struct abstract_generator *generator = (struct abstract_generator *) self; > PyObject *retval; > // check running > retval = generator->body(self, None, 0); > // unlock running > return retval; > } > > But this way should be much better, don't know if that is possible btw. > > struct closure { > struct abstract_generator generator; > /* closure stuff here */ > ... > } ; That's called inheritance in Cython. Just declare a subtype of the specific closure class. > I add MarkGenerator visitor, that sets is_generator flag on DefNode > and checks for returns/yields. Why do you need a new class here? Can't the MarkClosureVisitor handle this? It's not like transform runs over the whole AST come for free... Stefan From stefan_ml at behnel.de Mon Dec 6 08:47:06 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 08:47:06 +0100 Subject: [Cython] General generator expressions In-Reply-To: <4CFC91E6.8020604@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: <4CFC94FA.7020804@behnel.de> Stefan Behnel, 06.12.2010 08:33: > Vitja Makarov, 05.12.2010 20:12: >> I have problem with closures and generators. >> >> Now I want to implement closure structure this way: >> struct abstract_generator { >> PY_OBJECT_HEAD >> /* generator internal vars */ >> PyObject *(*body)(struct closure *, PyObject *, int); >> int resume_label; >> } > > I hope you're not trying to do it manually at that level, are you? > Declaring the fields on the closure class should just work. > > >> struct closure { >> /* abstract_generator */ >> PY_OBJECT_HEAD >> /* generator internal vars */ >> PyObject *(*body)(struct closure *, PyObject *, int); >> int resume_label; >> ,,,, >> /* Closure stuff */ >> } ; >> >> and generator utility function will work like this: >> >> PyObject *abstract_generator_next(PyObject *self, PyObject *args) >> { >> struct abstract_generator *generator = (struct abstract_generator *) self; >> PyObject *retval; >> // check running >> retval = generator->body(self, None, 0); >> // unlock running >> return retval; >> } >> >> But this way should be much better, don't know if that is possible btw. >> >> struct closure { >> struct abstract_generator generator; >> /* closure stuff here */ >> ... >> } ; > > That's called inheritance in Cython. Just declare a subtype of the specific > closure class. Erm, sorry, the other way round. Yes, I think it's a good idea to let the closure class inherit from the abstract generator. Stefan From vitja.makarov at gmail.com Mon Dec 6 09:56:55 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 6 Dec 2010 11:56:55 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4CFC91E6.8020604@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: 2010/12/6 Stefan Behnel : > Vitja Makarov, 05.12.2010 20:12: >> I've started woring on generators on friday. > > Cool. This sounds like a larger code change, though. Try not to submit it > as a single patch as that would likely be hard to review. Instead, put up a > repo somewhere that others can look at, or submit bundles (as hg calls > them) of separate commits. > > Ok, I'll try :) >> Now it can generate some code, but it doesn't work yet, what is done: >> ? - switch() for resuming and argument parsing >> ? - yield stuff >> ? - closure temp allocation >> >> I'm going to put all the generator stuff into closure >> >> Generator function: >> >> Creates closure with all temps, args, variables, and all the internal stuff: >> Also closure have all the required generator methods __iter__(), >> __nextiter__() and so on. > > Ok, why not. > > >> def my_generator(self, args, kwargs): >> ? ? closure ?= my_generator_closure_new() >> ? ? closure.is_running = 0 >> ? ? closure.resume_label ?= 0 >> ? ? closure.args = args >> ? ? closure.kwargs = kwargs >> ? ? closure.outer_scope = self >> ? ? closure.generator_body = my_generator_body >> ? ? return closure >> >> generator body: >> >> def my_generator_body(closure, value, is_exception): > > Shouldn't that be a cdef function? Guess no. It has very special signature and should be used only in generator closure. It can't be cdef as cdef doesn't parse (args, kwargs) and generator does this on first run. > > >> ? ? ? # switch case >> ? ? ? switch (closure.resume_lable): >> ? ? ? ? ? case 0: >> ? ? ? ? ? ? ? ? ?goto parse_arguments; >> ? ? ? ? ? case 1: >> ? ? ? ? ? ? ? ? ?goto resume_from_yield1; >> ? ? ? parse_arguments: >> ? ? ? # function body here > > Ok, sure, as in the CEP. > > >> About temps: >> now temps are allocated in closure using declare_var() and it works fine. > > I think that could seriously slow down things inside of the generator, > though. It means that it needs to jump through the closure indirection for > every little value that it keeps around temporarily, including lots of > intermediate C results. Also, I would expect that a lot more temps are used > inside of the generator body than what actually needs to be kept alive > during yields, so a single yield inside of a lengthy generator body could > have a tremendous impact on the overall runtime and closure size. > > If you only store away the live temp values before a yield, you don't even > need to do any reference counting for them. It's just a quick bunch of C > assignments on yield and resume. That's about the smallest impact you can get. > > BTW, if you change the object struct of the generator closure anyway, maybe > a dynamically sized PyVarObject would be a way to incorporate the temps? > > I don't understand why do you think about performance problems? It seems to me that if temps are stored in tuple (or another kind of object) you have to manully save/restore them, and this code will really slow down generator. On the other hand when all temps are declared in closure you just have to say "closure->pyx_temp_v1" Do you think that closure-> will be much slower then stack operations? >> I'm not sure about manage_ref flag I guess that temps with >> manage_ref=False shouldn't be allocated on closure (am I right?) > > "manage_ref=False" (for a Python object value) just means that the > reference is either borrowed or will be stolen at some point. I'm not sure > that stealing the reference may not happen *after* a yield, but I guess > that would produce other problems if the reference isn't kept alive in > another way. Nothing to worry about now, carefully crafted tests will > eventually catch that. > > In any case, all live temps (Python refs, borrowed refs and C values) may > be needed after the yield, so they must *all* end up in the closure. > > >> TODO: >> ? - create C function that returns closure >> ? - generator utility code > > At least for generator expressions, the latter wouldn't be needed for now. > > >> I have problem with closures and generators. >> >> Now I want to implement closure structure this way: >> struct abstract_generator { >> ? ? ?PY_OBJECT_HEAD >> ? ? ?/* generator internal vars */ >> ? ? ?PyObject *(*body)(struct closure *, PyObject *, int); >> ? ? ?int resume_label; >> } > > I hope you're not trying to do it manually at that level, are you? > Declaring the fields on the closure class should just work. > No, I declare things using scope_class.declare_var ;) > >> struct closure { >> ? ? ?/* abstract_generator */ >> ? ? ?PY_OBJECT_HEAD >> ? ? ?/* generator internal vars */ >> ? ? ?PyObject *(*body)(struct closure *, PyObject *, int); >> ? ? ?int resume_label; >> ? ? ?,,,, >> ? ? ?/* Closure stuff */ >> } ; >> >> and generator utility function will work like this: >> >> PyObject *abstract_generator_next(PyObject *self, PyObject *args) >> { >> ? ? ?struct abstract_generator *generator = (struct abstract_generator *) self; >> ? ? ?PyObject *retval; >> ? ? ?// check running >> ? ? ?retval = ?generator->body(self, None, 0); >> ? ? ?// unlock running >> ? ? ?return retval; >> } >> >> But this way should be much better, don't know if that is possible btw. >> >> struct closure { >> ? ? ?struct abstract_generator generator; >> ? ? ?/* closure stuff here */ >> ? ? ?... >> } ; > > That's called inheritance in Cython. Just declare a subtype of the specific > closure class. > That's not inheritance exactly I don't want to call all cython & python stuff for base class (why not btw?) Just want to have some fields (now I think that is_running and body is enough) on top of closure stuff, and I don't want to declare things twice. > >> I add MarkGenerator visitor, that sets is_generator flag on DefNode >> and checks for returns/yields. > > Why do you need a new class here? Can't the MarkClosureVisitor handle this? > It's not like transform runs over the whole AST come for free... > Now I check for yield/returns inside MarkGenerator, and set is_generator flag or raise error if both yield and return was found. Guess that's not right place for this. -- vitja. From stefan_ml at behnel.de Mon Dec 6 10:15:48 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 10:15:48 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: <4CFCA9C4.5080101@behnel.de> Vitja Makarov, 06.12.2010 09:56: > 2010/12/6 Stefan Behnel: >> Vitja Makarov, 05.12.2010 20:12: >>> def my_generator(self, args, kwargs): >>> closure = my_generator_closure_new() >>> closure.is_running = 0 >>> closure.resume_label = 0 >>> closure.args = args >>> closure.kwargs = kwargs >>> closure.outer_scope = self >>> closure.generator_body = my_generator_body >>> return closure >>> >>> generator body: >>> >>> def my_generator_body(closure, value, is_exception): >> >> Shouldn't that be a cdef function? > > Guess no. It has very special signature and should be used only in > generator closure. > > It can't be cdef as cdef doesn't parse (args, kwargs) and generator > does this on first run. Generators, yes. Generator expressions, no. I've written in the CEP that it would be nice to start with splitting the way Python functions work. I don't ask you to do this (certainly makes a nice optimisation when the functionality itself is done), but it would easily enable a cdef function here, which would hugely reduce the burden when advancing a generator. >>> About temps: >>> now temps are allocated in closure using declare_var() and it works fine. >> >> I think that could seriously slow down things inside of the generator, >> though. It means that it needs to jump through the closure indirection for >> every little value that it keeps around temporarily, including lots of >> intermediate C results. Also, I would expect that a lot more temps are used >> inside of the generator body than what actually needs to be kept alive >> during yields, so a single yield inside of a lengthy generator body could >> have a tremendous impact on the overall runtime and closure size. >> >> If you only store away the live temp values before a yield, you don't even >> need to do any reference counting for them. It's just a quick bunch of C >> assignments on yield and resume. That's about the smallest impact you can get. >> >> BTW, if you change the object struct of the generator closure anyway, maybe >> a dynamically sized PyVarObject would be a way to incorporate the temps? > > I don't understand why do you think about performance problems? > > It seems to me that if temps are stored in tuple (or another kind of > object) you have to manully save/restore them, and this code will > really slow down generator. But it only has to be done when suspending/resuming, not every time a temp is used. Given that there is quite some overhead involved in executing a yield/resume cycle, it doesn't matter if we add a couple of fast C assignments to that. > On the other hand when all temps are declared in closure you just have > to say "closure->pyx_temp_v1" > > Do you think that closure-> will be much slower then stack operations? Yes, *much* slower. Temps are not meant to live on the stack. Most of the time, they will just live in a CPU register, especially on x86_64 and other architectures with plenty of registers. When you put them into the closure, the C compiler must assume that it's an external operation when accessing them and can't know that the next time it reads the variable it still has the same value that it just wrote. That's kills tons of possible C optimisations. >>> struct closure { >>> /* abstract_generator */ >>> PY_OBJECT_HEAD >>> /* generator internal vars */ >>> PyObject *(*body)(struct closure *, PyObject *, int); >>> int resume_label; >>> ,,,, >>> /* Closure stuff */ >>> } ; >>> >>> and generator utility function will work like this: >>> >>> PyObject *abstract_generator_next(PyObject *self, PyObject *args) >>> { >>> struct abstract_generator *generator = (struct abstract_generator *) self; >>> PyObject *retval; >>> // check running >>> retval = generator->body(self, None, 0); >>> // unlock running >>> return retval; >>> } >>> >>> But this way should be much better, don't know if that is possible btw. >>> >>> struct closure { >>> struct abstract_generator generator; >>> /* closure stuff here */ >>> ... >>> } ; >> >> That's called inheritance in Cython. Just declare a subtype of the specific >> closure class. > > That's not inheritance exactly I don't want to call all cython& > python stuff for base class (why not btw?) Right, why not? Closures are instantiated by calling tp_new() directly, so even the overhead of creating them as a subtype is pretty low. Plus, it only has to be done once when instantiating the generator, not every time it advances. So I consider it totally negligible over the overall lifetime. > Just want to have some fields (now I think that is_running and body is > enough) on top of closure stuff, and I don't want to declare things > twice. Then inheritance is your friend. It will create a vtable, though, if you use methods. >>> I add MarkGenerator visitor, that sets is_generator flag on DefNode >>> and checks for returns/yields. >> >> Why do you need a new class here? Can't the MarkClosureVisitor handle this? >> It's not like transform runs over the whole AST come for free... > > Now I check for yield/returns inside MarkGenerator, and set > is_generator flag or raise error if both yield and return was found. > Guess that's not right place for this. It's fine, but I think it's better to merge that with the MarkClosureVisitor. It's a very related purpose. Having lots of transforms for tiny things is just too large a burden for the compiler. Stefan From vitja.makarov at gmail.com Mon Dec 6 10:56:02 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 6 Dec 2010 12:56:02 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4CFCA9C4.5080101@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4CFCA9C4.5080101@behnel.de> Message-ID: 2010/12/6 Stefan Behnel : > Vitja Makarov, 06.12.2010 09:56: >> 2010/12/6 Stefan Behnel: >>> Vitja Makarov, 05.12.2010 20:12: >>>> def my_generator(self, args, kwargs): >>>> ? ? ?closure ?= my_generator_closure_new() >>>> ? ? ?closure.is_running = 0 >>>> ? ? ?closure.resume_label ?= 0 >>>> ? ? ?closure.args = args >>>> ? ? ?closure.kwargs = kwargs >>>> ? ? ?closure.outer_scope = self >>>> ? ? ?closure.generator_body = my_generator_body >>>> ? ? ?return closure >>>> >>>> generator body: >>>> >>>> def my_generator_body(closure, value, is_exception): >>> >>> Shouldn't that be a cdef function? >> >> Guess no. It has very special signature and should be used only in >> generator closure. >> >> It can't be cdef as cdef doesn't parse (args, kwargs) and generator >> does this on first run. > > Generators, yes. Generator expressions, no. I've written in the CEP that it > would be nice to start with splitting the way Python functions work. I > don't ask you to do this (certainly makes a nice optimisation when the > functionality itself is done), but it would easily enable a cdef function > here, which would hugely reduce the burden when advancing a generator. > Yeah. Splitting would be nice. But I don't know yet how to do it the best way. So it could wait a little. > >>>> About temps: >>>> now temps are allocated in closure using declare_var() and it works fine. >>> >>> I think that could seriously slow down things inside of the generator, >>> though. It means that it needs to jump through the closure indirection for >>> every little value that it keeps around temporarily, including lots of >>> intermediate C results. Also, I would expect that a lot more temps are used >>> inside of the generator body than what actually needs to be kept alive >>> during yields, so a single yield inside of a lengthy generator body could >>> have a tremendous impact on the overall runtime and closure size. >>> >>> If you only store away the live temp values before a yield, you don't even >>> need to do any reference counting for them. It's just a quick bunch of C >>> assignments on yield and resume. That's about the smallest impact you can get. >>> >>> BTW, if you change the object struct of the generator closure anyway, maybe >>> a dynamically sized PyVarObject would be a way to incorporate the temps? >> >> I don't understand why do you think about performance problems? >> >> It seems to me that if temps are stored in tuple (or another kind of >> object) you have to manully save/restore them, and this code will >> really slow down generator. > > But it only has to be done when suspending/resuming, not every time a temp > is used. Given that there is quite some overhead involved in executing a > yield/resume cycle, it doesn't matter if we add a couple of fast C > assignments to that. > > >> On the other hand when all temps are declared in closure you just have >> to say "closure->pyx_temp_v1" >> >> Do you think that closure-> ?will be much slower then stack operations? > > Yes, *much* slower. Temps are not meant to live on the stack. Most of the > time, they will just live in a CPU register, especially on x86_64 and other > architectures with plenty of registers. When you put them into the closure, > the C compiler must assume that it's an external operation when accessing > them and can't know that the next time it reads the variable it still has > the same value that it just wrote. That's kills tons of possible C > optimisations. > > Hmm. It is not really necessery to check each for value each time, it's not defined as volatile. Compiler will try to put it in registers and write back when reqired. Btw it's better to try both ways and choose the fastest one. How should save/restore code look like? PyObject *all_temps[NUMBER_OF_PYOBJECT_TEMPS]; // save memcpy(closure->all_temps, all_temps, sizeof(all_temps));? Or: for (int i=0; i < NUMBER_OF_PYOBJECT_TEMPS; i++) { if (all_temps[i]) { // this will create tuple (n, all_temps[i]) and add it to closure->temps_tuple add_temp_to_closure(closure, i, all_temps[i]); } } How should it handle native C types? >>>> struct closure { >>>> ? ? ? /* abstract_generator */ >>>> ? ? ? PY_OBJECT_HEAD >>>> ? ? ? /* generator internal vars */ >>>> ? ? ? PyObject *(*body)(struct closure *, PyObject *, int); >>>> ? ? ? int resume_label; >>>> ? ? ? ,,,, >>>> ? ? ? /* Closure stuff */ >>>> } ; >>>> >>>> and generator utility function will work like this: >>>> >>>> PyObject *abstract_generator_next(PyObject *self, PyObject *args) >>>> { >>>> ? ? ? struct abstract_generator *generator = (struct abstract_generator *) self; >>>> ? ? ? PyObject *retval; >>>> ? ? ? // check running >>>> ? ? ? retval = ?generator->body(self, None, 0); >>>> ? ? ? // unlock running >>>> ? ? ? return retval; >>>> } >>>> >>>> But this way should be much better, don't know if that is possible btw. >>>> >>>> struct closure { >>>> ? ? ? struct abstract_generator generator; >>>> ? ? ? /* closure stuff here */ >>>> ? ? ? ... >>>> } ; >>> >>> That's called inheritance in Cython. Just declare a subtype of the specific >>> closure class. >> >> That's not inheritance exactly I don't want to call all cython& >> python stuff for base class (why not btw?) > > Right, why not? Closures are instantiated by calling tp_new() directly, so > even the overhead of creating them as a subtype is pretty low. Plus, it > only has to be done once when instantiating the generator, not every time > it advances. So I consider it totally negligible over the overall lifetime. > > >> Just want to have some fields (now I think that is_running and body is >> enough) on top of closure stuff, and I don't want to declare things >> twice. > > Then inheritance is your friend. It will create a vtable, though, if you > use methods. > > Ok. >>>> I add MarkGenerator visitor, that sets is_generator flag on DefNode >>>> and checks for returns/yields. >>> >>> Why do you need a new class here? Can't the MarkClosureVisitor handle this? >>> It's not like transform runs over the whole AST come for free... >> >> Now I check for yield/returns inside MarkGenerator, and set >> is_generator flag or raise error if both yield and return was found. >> Guess that's not right place for this. > > It's fine, but I think it's better to merge that with the > MarkClosureVisitor. It's a very related purpose. Having lots of transforms > for tiny things is just too large a burden for the compiler. > Ok. I'll try. -- vitja. From stefan_ml at behnel.de Mon Dec 6 11:35:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 11:35:05 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4CFCA9C4.5080101@behnel.de> Message-ID: <4CFCBC59.9040302@behnel.de> Vitja Makarov, 06.12.2010 10:56: > 2010/12/6 Stefan Behnel: >> Vitja Makarov, 06.12.2010 09:56: >>> 2010/12/6 Stefan Behnel: >>>> Vitja Makarov, 05.12.2010 20:12: >>>>> About temps: >>>>> now temps are allocated in closure using declare_var() and it works fine. >>>> >>>> I think that could seriously slow down things inside of the generator, >>>> though. It means that it needs to jump through the closure indirection for >>>> every little value that it keeps around temporarily, including lots of >>>> intermediate C results. Also, I would expect that a lot more temps are used >>>> inside of the generator body than what actually needs to be kept alive >>>> during yields, so a single yield inside of a lengthy generator body could >>>> have a tremendous impact on the overall runtime and closure size. >>>> >>>> If you only store away the live temp values before a yield, you don't even >>>> need to do any reference counting for them. It's just a quick bunch of C >>>> assignments on yield and resume. That's about the smallest impact you can get. >>>> >>>> BTW, if you change the object struct of the generator closure anyway, maybe >>>> a dynamically sized PyVarObject would be a way to incorporate the temps? >>> >>> I don't understand why do you think about performance problems? >>> >>> It seems to me that if temps are stored in tuple (or another kind of >>> object) you have to manully save/restore them, and this code will >>> really slow down generator. >> >> But it only has to be done when suspending/resuming, not every time a temp >> is used. Given that there is quite some overhead involved in executing a >> yield/resume cycle, it doesn't matter if we add a couple of fast C >> assignments to that. >> >>> On the other hand when all temps are declared in closure you just have >>> to say "closure->pyx_temp_v1" >>> >>> Do you think that closure-> will be much slower then stack operations? >> >> Yes, *much* slower. Temps are not meant to live on the stack. Most of the >> time, they will just live in a CPU register, especially on x86_64 and other >> architectures with plenty of registers. When you put them into the closure, >> the C compiler must assume that it's an external operation when accessing >> them and can't know that the next time it reads the variable it still has >> the same value that it just wrote. That's kills tons of possible C >> optimisations. > > Hmm. It is not really necessery to check each for value each time, > it's not defined as volatile. My knowledge of C is admittedly limited. Why should the C compiler be allowed to assume that a value stored somewhere in a foreign memory location doesn't change its value between separate store and read operations? > Compiler will try to put it in registers and write back when reqired. The difference is, when it's not in the closure, the function body (or current C scope) owns it, so the C compiler doesn't have to generate code to write it anywhere in the first place. Most temps are really just there for convenience when generating the C code, not because there really is something to store away. The C compiler is expected to drop almost all of them. > Btw it's better to try both ways and choose the fastest one. I know, predicting performance is hard at best, but in this case, it's clear enough to me that letting all temps live in the closure suffers from two serious drawbacks: space *and* time. > How should save/restore code look like? > > PyObject *all_temps[NUMBER_OF_PYOBJECT_TEMPS]; > > // save > memcpy(closure->all_temps, all_temps, sizeof(all_temps));? > > Or: > > for (int i=0; i< NUMBER_OF_PYOBJECT_TEMPS; i++) { > if (all_temps[i]) { > // this will create tuple (n, all_temps[i]) and add it to > closure->temps_tuple > add_temp_to_closure(closure, i, all_temps[i]); > } > } > > How should it handle native C types? The CEP has some comments on this. I think it would be best to generate the straight assignment code from within of the YieldExprNode: closure->place_for_int_temp[0] = __pyx_t_2 closure->place_for_int_temp[1] = __pyx_t_5 closure->place_for_object_temp[0] = __pyx_t_9 return result resume: __pyx_t_2 = closure->place_for_int_temp[0] ... For starters, I wouldn't mind if the closure fields were all "size_t" and the assignments just did an ugly cast on the RHS. Might be easier than properly allocating separate arrays of correctly sized fields. BTW, I just noticed that this won't work with exception handling yet. In try-except and try-finally statements, we keep the current exception state in block local variables, those would have to get stored away as well. However, the problem is that this can be in nested block scopes, so the outer variables may not even be available to the C code. Another thing to deal with once this works... Stefan From dagss at student.matnat.uio.no Mon Dec 6 11:55:46 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 06 Dec 2010 11:55:46 +0100 Subject: [Cython] General generator expressions In-Reply-To: <4CFCBC59.9040302@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4CFCA9C4.5080101@behnel.de> <4CFCBC59.9040302@behnel.de> Message-ID: <4CFCC132.5070803@student.matnat.uio.no> On 12/06/2010 11:35 AM, Stefan Behnel wrote: > Vitja Makarov, 06.12.2010 10:56: > >> 2010/12/6 Stefan Behnel: >> >>> Vitja Makarov, 06.12.2010 09:56: >>> >>>> 2010/12/6 Stefan Behnel: >>>> >>>>> Vitja Makarov, 05.12.2010 20:12: >>>>> >>>>>> About temps: >>>>>> now temps are allocated in closure using declare_var() and it works fine. >>>>>> >>>>> I think that could seriously slow down things inside of the generator, >>>>> though. It means that it needs to jump through the closure indirection for >>>>> every little value that it keeps around temporarily, including lots of >>>>> intermediate C results. Also, I would expect that a lot more temps are used >>>>> inside of the generator body than what actually needs to be kept alive >>>>> during yields, so a single yield inside of a lengthy generator body could >>>>> have a tremendous impact on the overall runtime and closure size. >>>>> >>>>> If you only store away the live temp values before a yield, you don't even >>>>> need to do any reference counting for them. It's just a quick bunch of C >>>>> assignments on yield and resume. That's about the smallest impact you can get. >>>>> >>>>> BTW, if you change the object struct of the generator closure anyway, maybe >>>>> a dynamically sized PyVarObject would be a way to incorporate the temps? >>>>> >>>> I don't understand why do you think about performance problems? >>>> >>>> It seems to me that if temps are stored in tuple (or another kind of >>>> object) you have to manully save/restore them, and this code will >>>> really slow down generator. >>>> >>> But it only has to be done when suspending/resuming, not every time a temp >>> is used. Given that there is quite some overhead involved in executing a >>> yield/resume cycle, it doesn't matter if we add a couple of fast C >>> assignments to that. >>> >>> >>>> On the other hand when all temps are declared in closure you just have >>>> to say "closure->pyx_temp_v1" >>>> >>>> Do you think that closure-> will be much slower then stack operations? >>>> >>> Yes, *much* slower. Temps are not meant to live on the stack. Most of the >>> time, they will just live in a CPU register, especially on x86_64 and other >>> architectures with plenty of registers. When you put them into the closure, >>> the C compiler must assume that it's an external operation when accessing >>> them and can't know that the next time it reads the variable it still has >>> the same value that it just wrote. That's kills tons of possible C >>> optimisations. >>> >> Hmm. It is not really necessery to check each for value each time, >> it's not defined as volatile. >> > My knowledge of C is admittedly limited. Why should the C compiler be > allowed to assume that a value stored somewhere in a foreign memory > location doesn't change its value between separate store and read operations? > Unless you call another function, the C compiler will know just from the actions of the executing thread. (You can't use a variable/struct field to e.g. communicate between threads...unless perhaps if you mark it volatile, but one should use locking instead). But, of course, the closure may well call other functions in libraries where the C compiler has no access. So I believe Stefan is right. It corresponds to providing hints to the compiler that "no external code will modify this memory". Dag Sverre From stefan_ml at behnel.de Mon Dec 6 14:42:47 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 14:42:47 +0100 Subject: [Cython] General generator expressions In-Reply-To: <4CFCBC59.9040302@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4CFCA9C4.5080101@behnel.de> <4CFCBC59.9040302@behnel.de> Message-ID: <4CFCE857.6000404@behnel.de> Stefan Behnel, 06.12.2010 11:35: > I think it would be best to generate the straight assignment code from > within of the YieldExprNode: > > closure->place_for_int_temp[0] = __pyx_t_2 > closure->place_for_int_temp[1] = __pyx_t_5 > closure->place_for_object_temp[0] = __pyx_t_9 > return result > resume: > __pyx_t_2 = closure->place_for_int_temp[0] > ... > > For starters, I wouldn't mind if the closure fields were all "size_t" and > the assignments just did an ugly cast on the RHS. Might be easier than > properly allocating separate arrays of correctly sized fields. ... although, of course, not every C value fits into size_t, either. There may be structs being passed around by value, for example. Tricky... Hmm, maybe it's really ok to start with temps in the closure for now, just like we accepted that closures were huge until they were cut down to what was actually needed. I think the feature is more important than the optimisations around it. Stefan From stefan_ml at behnel.de Mon Dec 6 18:49:20 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 18:49:20 +0100 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> <4CFC005F.2070103@behnel.de> <4CFC8835.1010500@behnel.de> Message-ID: <4CFD2220.5060600@behnel.de> Vitja Makarov, 06.12.2010 08:33: > 2010/12/6 Stefan Behnel: >> Vitja Makarov, 06.12.2010 07:05: >>> 2010/12/6 Stefan Behnel: >>>> Vitja Makarov, 05.12.2010 08:41: >>>>> >>>>> I've updated patch. >>>>> >>>>> I removed pretty_name, bad idea should use name instead in >>>>> pydeclare_lambda. >>>>> `is_lambda` is now called `is_anonymous`, that means that entry itself >>>>> doesn't know how it will be named inside scope, so its anonymous (both >>>>> lambda and pyfunctions) >>>> >>>> That's ok and the patch looks good to me now. >>>> >>>> However, it lacks error tests and the modified code actually fails to detect >>>> redefinitions of cdef functions as Python functions and vice versa, which is >>>> not supported (and IMHO doesn't make sense either). Similarly, redefinitions >>>> in cdef classes aren't currently supported but are not detected either. >>>> >>>> I've attached a patch that currently breaks cpdef functions, but that >>>> handles (and tests) at least some of the error cases. I'm not sure what to >>>> do to fix the cpdef functions, though. >>> >>> I can't say it breaks cpdefs, this is "classic" cython behavior. >> >> Did you try running the test suite with it? > > I mean that cython wo redefinition patches will copile code like this > but gcc will not: > > cpdef foo(): pass > cpdef foo(): pass > > xxx.c:444: error: redefinition of ?__pyx_f_3xxx_foo? Right. That needs fixing, but I agree that it's not related to your patch. >>> May by we need some flag like is_hardcoded? >> >> Which would mean what exactly? > > Means that symbol is defined in method table, bad name for flag. > > So as I understand when function is cpdef it first declare cfunction, > then pycfunction. > > So entry is always of type CFunction so maybe this flag should be > called has_pyfunction? and error message should be raised in > _declare_pyfunction() if has_pyfunction flag is set? Something like that. There should be a way to detect a cpdef function and to prevent overriding cdef functions otherwise. Stefan From vitja.makarov at gmail.com Mon Dec 6 19:09:21 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 6 Dec 2010 21:09:21 +0300 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: <4CFD2220.5060600@behnel.de> References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> <4CFC005F.2070103@behnel.de> <4CFC8835.1010500@behnel.de> <4CFD2220.5060600@behnel.de> Message-ID: 2010/12/6 Stefan Behnel : > Vitja Makarov, 06.12.2010 08:33: >> 2010/12/6 Stefan Behnel: >>> Vitja Makarov, 06.12.2010 07:05: >>>> 2010/12/6 Stefan Behnel: >>>>> Vitja Makarov, 05.12.2010 08:41: >>>>>> >>>>>> I've updated patch. >>>>>> >>>>>> I removed pretty_name, bad idea should use name instead in >>>>>> pydeclare_lambda. >>>>>> `is_lambda` is now called `is_anonymous`, that means that entry itself >>>>>> doesn't know how it will be named inside scope, so its anonymous (both >>>>>> lambda and pyfunctions) >>>>> >>>>> That's ok and the patch looks good to me now. >>>>> >>>>> However, it lacks error tests and the modified code actually fails to detect >>>>> redefinitions of cdef functions as Python functions and vice versa, which is >>>>> not supported (and IMHO doesn't make sense either). Similarly, redefinitions >>>>> in cdef classes aren't currently supported but are not detected either. >>>>> >>>>> I've attached a patch that currently breaks cpdef functions, but that >>>>> handles (and tests) at least some of the error cases. I'm not sure what to >>>>> do to fix the cpdef functions, though. >>>> >>>> I can't say it breaks cpdefs, this is "classic" cython behavior. >>> >>> Did you try running the test suite with it? >> >> I mean that cython wo redefinition patches will copile code like this >> but gcc will not: >> >> cpdef foo(): pass >> cpdef foo(): pass >> >> xxx.c:444: error: redefinition of ?__pyx_f_3xxx_foo? > > Right. That needs fixing, but I agree that it's not related to your patch. > > >>>> May by we need some flag like is_hardcoded? >>> >>> Which would mean what exactly? >> >> Means that symbol is defined in method table, bad name for flag. >> >> So as I understand when function is cpdef it first declare cfunction, >> then pycfunction. >> >> So entry is always of type CFunction so maybe this flag should be >> called has_pyfunction? and error message should be raised in >> _declare_pyfunction() if has_pyfunction flag is set? > > Something like that. There should be a way to detect a cpdef function and > to prevent overriding cdef functions otherwise. > > Stefan I've update patch (just fixed typo, and added "first declared here" error) I think we should create ticket for cpdef issue. What the problem is: cpdef foo(): pass cpdef foo(): pass 1. declare pyfunction `foo` with allow_redefine=False 2. declare cfunction `foo`, overwriting entry 3. declare pyfunction `foo` with allow_redefine=False 3. declare cfunction `foo`, already overwrited So the problem actually is in declare_cfunction() -- vitja. From stefan_ml at behnel.de Mon Dec 6 19:15:30 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 19:15:30 +0100 Subject: [Cython] Redefinition of Python functions (ticket #489) In-Reply-To: References: <4CF89F36.8030903@behnel.de> <4CF8A8A1.7070808@behnel.de> <4CFC005F.2070103@behnel.de> <4CFC8835.1010500@behnel.de> <4CFD2220.5060600@behnel.de> Message-ID: <4CFD2842.4070901@behnel.de> Vitja Makarov, 06.12.2010 19:09: > I've update patch (just fixed typo, and added "first declared here" error) Thanks. I'll just apply it for now. > I think we should create ticket for cpdef issue. > > What the problem is: > > cpdef foo(): pass > cpdef foo(): pass > > 1. declare pyfunction `foo` with allow_redefine=False > 2. declare cfunction `foo`, overwriting entry > 3. declare pyfunction `foo` with allow_redefine=False > 3. declare cfunction `foo`, already overwrited > > So the problem actually is in declare_cfunction() Yes, I think so, too. That's a different ticket. Stefan From robertwb at math.washington.edu Mon Dec 6 19:37:50 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 6 Dec 2010 10:37:50 -0800 Subject: [Cython] Cython 0.13.1 In-Reply-To: <4CFBB5DF.6010205@behnel.de> References: <4CFA360E.5080407@behnel.de> <4CFA8271.7080201@behnel.de> <4CFB9067.9020506@behnel.de> <4CFBB5DF.6010205@behnel.de> Message-ID: On Sun, Dec 5, 2010 at 7:55 AM, Stefan Behnel wrote: > mark florisson, 05.12.2010 15:11: >> On 5 December 2010 14:15, Stefan Behnel wrote: >>> I haven't tested it yet, mainly because I don't have the latest gdb version >>> installed. When I tried building lxml with that branch, I ran into a couple >>> of compiler crashes that Mark resolved, but I can't tell if there aren't >>> any left. I also didn't review the code in any way. >>> >>> AFAIC, I'm not sure I want this right in 0.14, but that's more because I >>> can't really comment on it. I certainly agree that it needs more testing. >> [...] >> But I agree that it should be tested more widely before merging it. The >> documentation (currently only on hudson but I presume it will make it >> to docs.cython.org with the new release) already contains a link to my >> git branch so users that are interested could still use it. > > Then I would suggest we keep the docs as they are now to make your gdb > support visible, but with a big note that this hasn't been merged yet and > keep your branch alive for a bit longer after the release. I wouldn't mind > merging it for 0.14.1 when it seems mature enough, and I hope that release > isn't too far off. 0.14 is a release with many, many new features, so it's > quite likely that people find bugs quickly once it's out in the wild. OTOH, when the flag is off, the changes made are quite noninvasive, so I would be fine merging this as an experimental feature to get more widespread testing and for what utility it already has, even if it's not fully polished yet (not that I have any bugs to point at...). - Robert From robertwb at math.washington.edu Mon Dec 6 19:50:33 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 6 Dec 2010 10:50:33 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: <4CFC78D7.7080109@behnel.de> References: <4CFB2254.80705@student.matnat.uio.no> <4CFC78D7.7080109@behnel.de> Message-ID: On Sun, Dec 5, 2010 at 9:47 PM, Stefan Behnel wrote: > Ondrej Certik, 06.12.2010 02:42: >> Maybe we should create a StopOnFirstError >> exception, and then catch it and print "exiting the compilation due to >> --fatal-errors command line switch"? > > +1, although I'd name it AbortCompilation or something like that, maybe > SilentlyAbortCompilation. We should trigger the exact same kind of abort that happens between (some) pipeline phases if there are any errors. This will trigger the necessary cleanup (e.g. a castrated C file), etc. I thought that's what I did, but I guess I only tested running cython manually (which didn't give a nasty traceback). While I'm thinking about this, I'd rather the flag be named --fast-fail, as errors are always fatal, it's just a question of timing. - Robert From stefan_ml at behnel.de Mon Dec 6 20:15:32 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Dec 2010 20:15:32 +0100 Subject: [Cython] Cython 0.13.1 In-Reply-To: References: <4CFA360E.5080407@behnel.de> <4CFA8271.7080201@behnel.de> <4CFB9067.9020506@behnel.de> <4CFBB5DF.6010205@behnel.de> Message-ID: <4CFD3654.4000603@behnel.de> Robert Bradshaw, 06.12.2010 19:37: > On Sun, Dec 5, 2010 at 7:55 AM, Stefan Behnel wrote: >> mark florisson, 05.12.2010 15:11: >>> On 5 December 2010 14:15, Stefan Behnel wrote: >>>> I haven't tested it yet, mainly because I don't have the latest gdb version >>>> installed. When I tried building lxml with that branch, I ran into a couple >>>> of compiler crashes that Mark resolved, but I can't tell if there aren't >>>> any left. I also didn't review the code in any way. >>>> >>>> AFAIC, I'm not sure I want this right in 0.14, but that's more because I >>>> can't really comment on it. I certainly agree that it needs more testing. >>> [...] >>> But I agree that it should be tested more widely before merging it. The >>> documentation (currently only on hudson but I presume it will make it >>> to docs.cython.org with the new release) already contains a link to my >>> git branch so users that are interested could still use it. >> >> Then I would suggest we keep the docs as they are now to make your gdb >> support visible, but with a big note that this hasn't been merged yet and >> keep your branch alive for a bit longer after the release. I wouldn't mind >> merging it for 0.14.1 when it seems mature enough, and I hope that release >> isn't too far off. 0.14 is a release with many, many new features, so it's >> quite likely that people find bugs quickly once it's out in the wild. > > OTOH, when the flag is off, the changes made are quite noninvasive, so > I would be fine merging this as an experimental feature to get more > widespread testing and for what utility it already has, even if it's > not fully polished yet (not that I have any bugs to point at...). Ok, whatever. If it doesn't hurt by default, just merge it in. It's a nice feature that we can just keep growing as people start using it. Stefan From ondrej at certik.cz Mon Dec 6 22:29:53 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 6 Dec 2010 13:29:53 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: References: <4CFB2254.80705@student.matnat.uio.no> <4CFC78D7.7080109@behnel.de> Message-ID: On Mon, Dec 6, 2010 at 10:50 AM, Robert Bradshaw wrote: > On Sun, Dec 5, 2010 at 9:47 PM, Stefan Behnel wrote: >> Ondrej Certik, 06.12.2010 02:42: >>> Maybe we should create a StopOnFirstError >>> exception, and then catch it and print "exiting the compilation due to >>> --fatal-errors command line switch"? >> >> +1, although I'd name it AbortCompilation or something like that, maybe >> SilentlyAbortCompilation. > > We should trigger the exact same kind of abort that happens between > (some) pipeline phases if there are any errors. This will trigger the > necessary cleanup (e.g. a castrated C file), etc. I thought that's > what I did, but I guess I only tested running cython manually (which > didn't give a nasty traceback). > > While I'm thinking about this, I'd rather the flag be named > --fast-fail, as errors are always fatal, it's just a question of > timing. Sure. Can you make this change before the release? So that people don't start using the other option. I just followed the gcc naming convention, to stay compatible. Gcc uses -Wfatal-errors. Ondrej From robertwb at math.washington.edu Tue Dec 7 00:18:22 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 6 Dec 2010 15:18:22 -0800 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: First off, great to hear you're working on this. I've wanted to do it for a while (well, essentially ever since I got closures done), but don't know when I'll find the time. On Mon, Dec 6, 2010 at 12:56 AM, Vitja Makarov wrote: >>> def my_generator_body(closure, value, is_exception): >> >> Shouldn't that be a cdef function? > > Guess no. It has very special signature and should be used only in > generator closure. > > It can't be cdef as cdef doesn't parse (args, kwargs) and generator > does this on first run. Eventually the argument parsing code should be abstracted out so that we don't have this restriction. >>> ? ? ? # switch case >>> ? ? ? switch (closure.resume_lable): >>> ? ? ? ? ? case 0: >>> ? ? ? ? ? ? ? ? ?goto parse_arguments; >>> ? ? ? ? ? case 1: >>> ? ? ? ? ? ? ? ? ?goto resume_from_yield1; >>> ? ? ? parse_arguments: >>> ? ? ? # function body here >> >> Ok, sure, as in the CEP. >> >> >>> About temps: >>> now temps are allocated in closure using declare_var() and it works fine. >> >> I think that could seriously slow down things inside of the generator, >> though. It means that it needs to jump through the closure indirection for >> every little value that it keeps around temporarily, including lots of >> intermediate C results. Also, I would expect that a lot more temps are used >> inside of the generator body than what actually needs to be kept alive >> during yields, so a single yield inside of a lengthy generator body could >> have a tremendous impact on the overall runtime and closure size. >> >> If you only store away the live temp values before a yield, you don't even >> need to do any reference counting for them. It's just a quick bunch of C >> assignments on yield and resume. That's about the smallest impact you can get. >> >> BTW, if you change the object struct of the generator closure anyway, maybe >> a dynamically sized PyVarObject would be a way to incorporate the temps? >> >> > > I don't understand why do you think about performance problems? > > It seems to me that if temps are stored in tuple (or another kind of > object) you have to manully save/restore them, and this code will > really slow down generator. -1 to storing them all in a tuple. That wouldn't handle C values very well, for one thing. I wouldn't use arrays though--either declare each temp individually or create a custom struct that has the correct slot for each temp. Copy to locals on enter, copy out on yield/exception/return. > On the other hand when all temps are declared in closure you just have > to say "closure->pyx_temp_v1" > > Do you think that closure-> will be much slower then stack operations? Yes, the performance difference can be dramatic. This is one of the reasons good numpy support is tricky to get right--the compiler can optimize stuff on the stack like crazy, but the same can't be said of indirect references. I'm fine with sticking all temps in the closure for now if that gets us up and running faster, and optimizing things later. >>> I'm not sure about manage_ref flag I guess that temps with >>> manage_ref=False shouldn't be allocated on closure (am I right?) >> >> "manage_ref=False" (for a Python object value) just means that the >> reference is either borrowed or will be stolen at some point. I'm not sure >> that stealing the reference may not happen *after* a yield, but I guess >> that would produce other problems if the reference isn't kept alive in >> another way. Nothing to worry about now, carefully crafted tests will >> eventually catch that. >> >> In any case, all live temps (Python refs, borrowed refs and C values) may >> be needed after the yield, so they must *all* end up in the closure. And of course we need to make all live reference are properly deallocated when the closure is GC'd as well. - Robert From vitja.makarov at gmail.com Tue Dec 7 06:15:27 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 7 Dec 2010 08:15:27 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: 2010/12/7 Robert Bradshaw : > First off, great to hear you're working on this. I've wanted to do it > for a while (well, essentially ever since I got closures done), but > don't know when I'll find the time. > > > On Mon, Dec 6, 2010 at 12:56 AM, Vitja Makarov wrote: > >>>> def my_generator_body(closure, value, is_exception): >>> >>> Shouldn't that be a cdef function? >> >> Guess no. It has very special signature and should be used only in >> generator closure. >> >> It can't be cdef as cdef doesn't parse (args, kwargs) and generator >> does this on first run. > > Eventually the argument parsing code should be abstracted out so that > we don't have this restriction. > >>>> ? ? ? # switch case >>>> ? ? ? switch (closure.resume_lable): >>>> ? ? ? ? ? case 0: >>>> ? ? ? ? ? ? ? ? ?goto parse_arguments; >>>> ? ? ? ? ? case 1: >>>> ? ? ? ? ? ? ? ? ?goto resume_from_yield1; >>>> ? ? ? parse_arguments: >>>> ? ? ? # function body here >>> >>> Ok, sure, as in the CEP. >>> >>> >>>> About temps: >>>> now temps are allocated in closure using declare_var() and it works fine. >>> >>> I think that could seriously slow down things inside of the generator, >>> though. It means that it needs to jump through the closure indirection for >>> every little value that it keeps around temporarily, including lots of >>> intermediate C results. Also, I would expect that a lot more temps are used >>> inside of the generator body than what actually needs to be kept alive >>> during yields, so a single yield inside of a lengthy generator body could >>> have a tremendous impact on the overall runtime and closure size. >>> >>> If you only store away the live temp values before a yield, you don't even >>> need to do any reference counting for them. It's just a quick bunch of C >>> assignments on yield and resume. That's about the smallest impact you can get. >>> >>> BTW, if you change the object struct of the generator closure anyway, maybe >>> a dynamically sized PyVarObject would be a way to incorporate the temps? >>> >>> >> >> I don't understand why do you think about performance problems? >> >> It seems to me that if temps are stored in tuple (or another kind of >> object) you have to manully save/restore them, and this code will >> really slow down generator. > > -1 to storing them all in a tuple. That wouldn't handle C values very > well, for one thing. I wouldn't use arrays though--either declare each > temp individually or create a custom struct that has the correct slot > for each temp. Copy to locals on enter, copy out on > yield/exception/return. > >> On the other hand when all temps are declared in closure you just have >> to say "closure->pyx_temp_v1" >> >> Do you think that closure-> will be much slower then stack operations? > > Yes, the performance difference can be dramatic. This is one of the > reasons good numpy support is tricky to get right--the compiler can > optimize stuff on the stack like crazy, but the same can't be said of > indirect references. I'm fine with sticking all temps in the closure > for now if that gets us up and running faster, and optimizing things > later. > >>>> I'm not sure about manage_ref flag I guess that temps with >>>> manage_ref=False shouldn't be allocated on closure (am I right?) >>> >>> "manage_ref=False" (for a Python object value) just means that the >>> reference is either borrowed or will be stolen at some point. I'm not sure >>> that stealing the reference may not happen *after* a yield, but I guess >>> that would produce other problems if the reference isn't kept alive in >>> another way. Nothing to worry about now, carefully crafted tests will >>> eventually catch that. >>> >>> In any case, all live temps (Python refs, borrowed refs and C values) may >>> be needed after the yield, so they must *all* end up in the closure. > > And of course we need to make all live reference are properly > deallocated when the closure is GC'd as well. > > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > YieldExprNode can check what temps are currently used and save/restore, and declare them on demand in closure. Sounds not bad too. This is not a problem. I created now I create AbstractGeneratorClass and make it base for closure. I want to bind tp_iter directly to my C functions, I found this way, but it's a little bit trickey... entry = klass.declare_var('__iter__', PyrexTypes.py_object_type, pos, visibility='extern') entry.func_cname = 'PyObject_SelfIter' I'm now stuck with generator body and function that returns genertor: DefNode should be translated into generator body: first I wanted to make it PyObject (*)(PyObject *closure, PyObject *value, int is_exception) But it seems to be hard now. And I'm thinking to put value and is_exception into AbstractGenerator class. And let generator body C-function accept original PyFunction arguments (PyObject *args, PyObject *kwargs) Also GeneratorWrapperNode should create C-function that creates closure and returns it. That's not very clean should that be attribute of DefNode, or should it subclass DefNode. So I'm thinking to try it in a dirty way. -- vitja. From vitja.makarov at gmail.com Tue Dec 7 06:22:28 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 7 Dec 2010 08:22:28 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: 2010/12/7 Vitja Makarov : > 2010/12/7 Robert Bradshaw : >> First off, great to hear you're working on this. I've wanted to do it >> for a while (well, essentially ever since I got closures done), but >> don't know when I'll find the time. >> >> >> On Mon, Dec 6, 2010 at 12:56 AM, Vitja Makarov wrote: >> >>>>> def my_generator_body(closure, value, is_exception): >>>> >>>> Shouldn't that be a cdef function? >>> >>> Guess no. It has very special signature and should be used only in >>> generator closure. >>> >>> It can't be cdef as cdef doesn't parse (args, kwargs) and generator >>> does this on first run. >> >> Eventually the argument parsing code should be abstracted out so that >> we don't have this restriction. >> >>>>> ? ? ? # switch case >>>>> ? ? ? switch (closure.resume_lable): >>>>> ? ? ? ? ? case 0: >>>>> ? ? ? ? ? ? ? ? ?goto parse_arguments; >>>>> ? ? ? ? ? case 1: >>>>> ? ? ? ? ? ? ? ? ?goto resume_from_yield1; >>>>> ? ? ? parse_arguments: >>>>> ? ? ? # function body here >>>> >>>> Ok, sure, as in the CEP. >>>> >>>> >>>>> About temps: >>>>> now temps are allocated in closure using declare_var() and it works fine. >>>> >>>> I think that could seriously slow down things inside of the generator, >>>> though. It means that it needs to jump through the closure indirection for >>>> every little value that it keeps around temporarily, including lots of >>>> intermediate C results. Also, I would expect that a lot more temps are used >>>> inside of the generator body than what actually needs to be kept alive >>>> during yields, so a single yield inside of a lengthy generator body could >>>> have a tremendous impact on the overall runtime and closure size. >>>> >>>> If you only store away the live temp values before a yield, you don't even >>>> need to do any reference counting for them. It's just a quick bunch of C >>>> assignments on yield and resume. That's about the smallest impact you can get. >>>> >>>> BTW, if you change the object struct of the generator closure anyway, maybe >>>> a dynamically sized PyVarObject would be a way to incorporate the temps? >>>> >>>> >>> >>> I don't understand why do you think about performance problems? >>> >>> It seems to me that if temps are stored in tuple (or another kind of >>> object) you have to manully save/restore them, and this code will >>> really slow down generator. >> >> -1 to storing them all in a tuple. That wouldn't handle C values very >> well, for one thing. I wouldn't use arrays though--either declare each >> temp individually or create a custom struct that has the correct slot >> for each temp. Copy to locals on enter, copy out on >> yield/exception/return. >> >>> On the other hand when all temps are declared in closure you just have >>> to say "closure->pyx_temp_v1" >>> >>> Do you think that closure-> will be much slower then stack operations? >> >> Yes, the performance difference can be dramatic. This is one of the >> reasons good numpy support is tricky to get right--the compiler can >> optimize stuff on the stack like crazy, but the same can't be said of >> indirect references. I'm fine with sticking all temps in the closure >> for now if that gets us up and running faster, and optimizing things >> later. >> >>>>> I'm not sure about manage_ref flag I guess that temps with >>>>> manage_ref=False shouldn't be allocated on closure (am I right?) >>>> >>>> "manage_ref=False" (for a Python object value) just means that the >>>> reference is either borrowed or will be stolen at some point. I'm not sure >>>> that stealing the reference may not happen *after* a yield, but I guess >>>> that would produce other problems if the reference isn't kept alive in >>>> another way. Nothing to worry about now, carefully crafted tests will >>>> eventually catch that. >>>> >>>> In any case, all live temps (Python refs, borrowed refs and C values) may >>>> be needed after the yield, so they must *all* end up in the closure. >> >> And of course we need to make all live reference are properly >> deallocated when the closure is GC'd as well. >> >> - Robert >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > YieldExprNode can check what temps are currently used and > save/restore, and declare them on demand in closure. > Sounds not bad too. This is not a problem. > > I created now I create AbstractGeneratorClass and make it base for closure. > I ?want to bind tp_iter directly to my C functions, I found this way, > but it's a little bit trickey... > > entry = klass.declare_var('__iter__', PyrexTypes.py_object_type, pos, > visibility='extern') > entry.func_cname = 'PyObject_SelfIter' > > > I'm now stuck with generator body and function that returns genertor: > > DefNode should be translated into generator body: > > first I wanted to make it PyObject (*)(PyObject *closure, PyObject > *value, int is_exception) > But it seems to be hard now. And I'm thinking to put value and > is_exception into AbstractGenerator class. And let generator body > C-function accept original PyFunction arguments (PyObject *args, > PyObject *kwargs) > > Also GeneratorWrapperNode should create C-function that creates > closure and returns it. That's not very clean should that be attribute > of DefNode, or should it subclass DefNode. So I'm thinking to try it > in a dirty way. > > -- > vitja. > Btw it would be very nice to split DefNode into smaller parts. Not sure how. -- vitja. From vitja.makarov at gmail.com Wed Dec 8 19:51:54 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 8 Dec 2010 21:51:54 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: 2010/12/7 Vitja Makarov : > 2010/12/7 Vitja Makarov : >> 2010/12/7 Robert Bradshaw : >>> First off, great to hear you're working on this. I've wanted to do it >>> for a while (well, essentially ever since I got closures done), but >>> don't know when I'll find the time. >>> >>> >>> On Mon, Dec 6, 2010 at 12:56 AM, Vitja Makarov wrote: >>> >>>>>> def my_generator_body(closure, value, is_exception): >>>>> >>>>> Shouldn't that be a cdef function? >>>> >>>> Guess no. It has very special signature and should be used only in >>>> generator closure. >>>> >>>> It can't be cdef as cdef doesn't parse (args, kwargs) and generator >>>> does this on first run. >>> >>> Eventually the argument parsing code should be abstracted out so that >>> we don't have this restriction. >>> >>>>>> ? ? ? # switch case >>>>>> ? ? ? switch (closure.resume_lable): >>>>>> ? ? ? ? ? case 0: >>>>>> ? ? ? ? ? ? ? ? ?goto parse_arguments; >>>>>> ? ? ? ? ? case 1: >>>>>> ? ? ? ? ? ? ? ? ?goto resume_from_yield1; >>>>>> ? ? ? parse_arguments: >>>>>> ? ? ? # function body here >>>>> >>>>> Ok, sure, as in the CEP. >>>>> >>>>> >>>>>> About temps: >>>>>> now temps are allocated in closure using declare_var() and it works fine. >>>>> >>>>> I think that could seriously slow down things inside of the generator, >>>>> though. It means that it needs to jump through the closure indirection for >>>>> every little value that it keeps around temporarily, including lots of >>>>> intermediate C results. Also, I would expect that a lot more temps are used >>>>> inside of the generator body than what actually needs to be kept alive >>>>> during yields, so a single yield inside of a lengthy generator body could >>>>> have a tremendous impact on the overall runtime and closure size. >>>>> >>>>> If you only store away the live temp values before a yield, you don't even >>>>> need to do any reference counting for them. It's just a quick bunch of C >>>>> assignments on yield and resume. That's about the smallest impact you can get. >>>>> >>>>> BTW, if you change the object struct of the generator closure anyway, maybe >>>>> a dynamically sized PyVarObject would be a way to incorporate the temps? >>>>> >>>>> >>>> >>>> I don't understand why do you think about performance problems? >>>> >>>> It seems to me that if temps are stored in tuple (or another kind of >>>> object) you have to manully save/restore them, and this code will >>>> really slow down generator. >>> >>> -1 to storing them all in a tuple. That wouldn't handle C values very >>> well, for one thing. I wouldn't use arrays though--either declare each >>> temp individually or create a custom struct that has the correct slot >>> for each temp. Copy to locals on enter, copy out on >>> yield/exception/return. >>> >>>> On the other hand when all temps are declared in closure you just have >>>> to say "closure->pyx_temp_v1" >>>> >>>> Do you think that closure-> will be much slower then stack operations? >>> >>> Yes, the performance difference can be dramatic. This is one of the >>> reasons good numpy support is tricky to get right--the compiler can >>> optimize stuff on the stack like crazy, but the same can't be said of >>> indirect references. I'm fine with sticking all temps in the closure >>> for now if that gets us up and running faster, and optimizing things >>> later. >>> >>>>>> I'm not sure about manage_ref flag I guess that temps with >>>>>> manage_ref=False shouldn't be allocated on closure (am I right?) >>>>> >>>>> "manage_ref=False" (for a Python object value) just means that the >>>>> reference is either borrowed or will be stolen at some point. I'm not sure >>>>> that stealing the reference may not happen *after* a yield, but I guess >>>>> that would produce other problems if the reference isn't kept alive in >>>>> another way. Nothing to worry about now, carefully crafted tests will >>>>> eventually catch that. >>>>> >>>>> In any case, all live temps (Python refs, borrowed refs and C values) may >>>>> be needed after the yield, so they must *all* end up in the closure. >>> >>> And of course we need to make all live reference are properly >>> deallocated when the closure is GC'd as well. >>> >>> - Robert >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >> YieldExprNode can check what temps are currently used and >> save/restore, and declare them on demand in closure. >> Sounds not bad too. This is not a problem. >> >> I created now I create AbstractGeneratorClass and make it base for closure. >> I ?want to bind tp_iter directly to my C functions, I found this way, >> but it's a little bit trickey... >> >> entry = klass.declare_var('__iter__', PyrexTypes.py_object_type, pos, >> visibility='extern') >> entry.func_cname = 'PyObject_SelfIter' >> >> >> I'm now stuck with generator body and function that returns genertor: >> >> DefNode should be translated into generator body: >> >> first I wanted to make it PyObject (*)(PyObject *closure, PyObject >> *value, int is_exception) >> But it seems to be hard now. And I'm thinking to put value and >> is_exception into AbstractGenerator class. And let generator body >> C-function accept original PyFunction arguments (PyObject *args, >> PyObject *kwargs) >> >> Also GeneratorWrapperNode should create C-function that creates >> closure and returns it. That's not very clean should that be attribute >> of DefNode, or should it subclass DefNode. So I'm thinking to try it >> in a dirty way. >> >> -- >> vitja. >> > > Btw it would be very nice to split DefNode into smaller parts. Not sure how. > > -- > vitja. > First success vitja at vitja-laptop:~/tmp/cython-devel-hg/zzz$ python Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import generator >>> x = generator.foo([1,2,3]) >>> next(x) 1 >>> next(x) 2 >>> next(x) 3 >>> next(x) Traceback (most recent call last): File "", line 1, in File "generator.pyx", line 1, in generator.foo (generator.c:518) def foo(x): StopIteration >>> -- vitja. From vitja.makarov at gmail.com Wed Dec 8 22:45:13 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 9 Dec 2010 00:45:13 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: 2010/12/8 Vitja Makarov : > 2010/12/7 Vitja Makarov : >> 2010/12/7 Vitja Makarov : >>> 2010/12/7 Robert Bradshaw : >>>> First off, great to hear you're working on this. I've wanted to do it >>>> for a while (well, essentially ever since I got closures done), but >>>> don't know when I'll find the time. >>>> >>>> >>>> On Mon, Dec 6, 2010 at 12:56 AM, Vitja Makarov wrote: >>>> >>>>>>> def my_generator_body(closure, value, is_exception): >>>>>> >>>>>> Shouldn't that be a cdef function? >>>>> >>>>> Guess no. It has very special signature and should be used only in >>>>> generator closure. >>>>> >>>>> It can't be cdef as cdef doesn't parse (args, kwargs) and generator >>>>> does this on first run. >>>> >>>> Eventually the argument parsing code should be abstracted out so that >>>> we don't have this restriction. >>>> >>>>>>> ? ? ? # switch case >>>>>>> ? ? ? switch (closure.resume_lable): >>>>>>> ? ? ? ? ? case 0: >>>>>>> ? ? ? ? ? ? ? ? ?goto parse_arguments; >>>>>>> ? ? ? ? ? case 1: >>>>>>> ? ? ? ? ? ? ? ? ?goto resume_from_yield1; >>>>>>> ? ? ? parse_arguments: >>>>>>> ? ? ? # function body here >>>>>> >>>>>> Ok, sure, as in the CEP. >>>>>> >>>>>> >>>>>>> About temps: >>>>>>> now temps are allocated in closure using declare_var() and it works fine. >>>>>> >>>>>> I think that could seriously slow down things inside of the generator, >>>>>> though. It means that it needs to jump through the closure indirection for >>>>>> every little value that it keeps around temporarily, including lots of >>>>>> intermediate C results. Also, I would expect that a lot more temps are used >>>>>> inside of the generator body than what actually needs to be kept alive >>>>>> during yields, so a single yield inside of a lengthy generator body could >>>>>> have a tremendous impact on the overall runtime and closure size. >>>>>> >>>>>> If you only store away the live temp values before a yield, you don't even >>>>>> need to do any reference counting for them. It's just a quick bunch of C >>>>>> assignments on yield and resume. That's about the smallest impact you can get. >>>>>> >>>>>> BTW, if you change the object struct of the generator closure anyway, maybe >>>>>> a dynamically sized PyVarObject would be a way to incorporate the temps? >>>>>> >>>>>> >>>>> >>>>> I don't understand why do you think about performance problems? >>>>> >>>>> It seems to me that if temps are stored in tuple (or another kind of >>>>> object) you have to manully save/restore them, and this code will >>>>> really slow down generator. >>>> >>>> -1 to storing them all in a tuple. That wouldn't handle C values very >>>> well, for one thing. I wouldn't use arrays though--either declare each >>>> temp individually or create a custom struct that has the correct slot >>>> for each temp. Copy to locals on enter, copy out on >>>> yield/exception/return. >>>> >>>>> On the other hand when all temps are declared in closure you just have >>>>> to say "closure->pyx_temp_v1" >>>>> >>>>> Do you think that closure-> will be much slower then stack operations? >>>> >>>> Yes, the performance difference can be dramatic. This is one of the >>>> reasons good numpy support is tricky to get right--the compiler can >>>> optimize stuff on the stack like crazy, but the same can't be said of >>>> indirect references. I'm fine with sticking all temps in the closure >>>> for now if that gets us up and running faster, and optimizing things >>>> later. >>>> >>>>>>> I'm not sure about manage_ref flag I guess that temps with >>>>>>> manage_ref=False shouldn't be allocated on closure (am I right?) >>>>>> >>>>>> "manage_ref=False" (for a Python object value) just means that the >>>>>> reference is either borrowed or will be stolen at some point. I'm not sure >>>>>> that stealing the reference may not happen *after* a yield, but I guess >>>>>> that would produce other problems if the reference isn't kept alive in >>>>>> another way. Nothing to worry about now, carefully crafted tests will >>>>>> eventually catch that. >>>>>> >>>>>> In any case, all live temps (Python refs, borrowed refs and C values) may >>>>>> be needed after the yield, so they must *all* end up in the closure. >>>> >>>> And of course we need to make all live reference are properly >>>> deallocated when the closure is GC'd as well. >>>> >>>> - Robert >>>> _______________________________________________ >>>> Cython-dev mailing list >>>> Cython-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/cython-dev >>>> >>> >>> YieldExprNode can check what temps are currently used and >>> save/restore, and declare them on demand in closure. >>> Sounds not bad too. This is not a problem. >>> >>> I created now I create AbstractGeneratorClass and make it base for closure. >>> I ?want to bind tp_iter directly to my C functions, I found this way, >>> but it's a little bit trickey... >>> >>> entry = klass.declare_var('__iter__', PyrexTypes.py_object_type, pos, >>> visibility='extern') >>> entry.func_cname = 'PyObject_SelfIter' >>> >>> >>> I'm now stuck with generator body and function that returns genertor: >>> >>> DefNode should be translated into generator body: >>> >>> first I wanted to make it PyObject (*)(PyObject *closure, PyObject >>> *value, int is_exception) >>> But it seems to be hard now. And I'm thinking to put value and >>> is_exception into AbstractGenerator class. And let generator body >>> C-function accept original PyFunction arguments (PyObject *args, >>> PyObject *kwargs) >>> >>> Also GeneratorWrapperNode should create C-function that creates >>> closure and returns it. That's not very clean should that be attribute >>> of DefNode, or should it subclass DefNode. So I'm thinking to try it >>> in a dirty way. >>> >>> -- >>> vitja. >>> >> >> Btw it would be very nice to split DefNode into smaller parts. Not sure how. >> >> -- >> vitja. >> > > First success > > vitja at vitja-laptop:~/tmp/cython-devel-hg/zzz$ python > Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) > [GCC 4.4.5] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import generator >>>> x = generator.foo([1,2,3]) >>>> next(x) > 1 >>>> next(x) > 2 >>>> next(x) > 3 >>>> next(x) > Traceback (most recent call last): > ?File "", line 1, in > ?File "generator.pyx", line 1, in generator.foo (generator.c:518) > ? ?def foo(x): > StopIteration >>>> > > > > -- > vitja. > Please review this patch. It's not yet finished, and mostly doesn't work. But you can take a look at patch and generated code. - Temps are saved/restored/allocated inside YieldExprNode using helper ClosureTempAllocator - PyxGenerator methods are defined via declare_var - Send, __next__(), __iter__() are implemented while close(), throw() aren't - YieldExprNode doesn't handle exceptions - I don't know how to make refnanny work, should refnanny context be stored in closure? - Objects can leak now, wouldn't check until refnanny work - GeneratorWrapperNode isn't node and is mostly a hack - Comprehensions are now handled by OldYieldExprNode - regr tests are broken -- vitja. -------------- next part -------------- A non-text attachment was scrubbed... Name: generators-support.patch.gz Type: application/x-gzip Size: 6107 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20101209/a0ac5479/attachment-0001.bin From stefan_ml at behnel.de Thu Dec 9 10:46:46 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Dec 2010 10:46:46 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> Message-ID: <4D00A586.8060801@behnel.de> Vitja Makarov, 08.12.2010 22:45: > Please review this patch. It's not yet finished, and mostly doesn't work. > But you can take a look at patch and generated code. *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send in a new patch when you have it. If you use hg, you can use "hg bundle" to collect commit series. Don't know about git. BTW, why not set up a github branch for this? > - Temps are saved/restored/allocated inside YieldExprNode using > helper ClosureTempAllocator > - PyxGenerator methods are defined via declare_var Should we call that "CyGenerator" ? :) > - Send, __next__(), __iter__() are implemented while close(), throw() aren't > - YieldExprNode doesn't handle exceptions Ok, sure, that's why close() and throw() don't work yet. I think that's fine for now. The first three are the most interesting ones. However, adding exception support shouldn't be hard at all, AFAICT, but they may benefit from a cdef function for the generator body in order to properly pass in the exception propagation trigger. > - I don't know how to make refnanny work, should refnanny context be > stored in closure? Good call. That's a tricky question. What about generating a GIVEREF for every Python temp you store in the closure, a GOTREF for everything you take back out, and finish/restart the refnanny context around the yield? > - Comprehensions are now handled by OldYieldExprNode ... another one of those bad names. ;) Any reason you couldn't reuse it for both? Stefan From robertwb at math.washington.edu Thu Dec 9 11:00:54 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 02:00:54 -0800 Subject: [Cython] General generator expressions In-Reply-To: <4D00A586.8060801@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> Message-ID: On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: > Vitja Makarov, 08.12.2010 22:45: >> Please review this patch. It's not yet finished, and mostly doesn't work. >> But you can take a look at patch and generated code. > > *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send > in a new patch when you have it. If you use hg, you can use "hg bundle" to > collect commit series. Don't know about git. > > BTW, why not set up a github branch for this? That's exactly what I was thinking. It'll make it easier to review as well, and for others to contribute without disturbing the main line. >> ? - Temps are saved/restored/allocated inside YieldExprNode using >> helper ClosureTempAllocator >> ? - PyxGenerator methods are defined via declare_var > > Should we call that "CyGenerator" ? :) > > >> ? - Send, ?__next__(), __iter__() are implemented while close(), throw() aren't >> ? - YieldExprNode doesn't handle exceptions > > Ok, sure, that's why close() and throw() don't work yet. I think that's > fine for now. The first three are the most interesting ones. However, > adding exception support shouldn't be hard at all, AFAICT, but they may > benefit from a cdef function for the generator body in order to properly > pass in the exception propagation trigger. > > >> ? - I don't know how to make refnanny work, should refnanny context be >> stored in closure? > > Good call. That's a tricky question. > > What about generating a GIVEREF for every Python temp you store in the > closure, a GOTREF for everything you take back out, and finish/restart the > refnanny context around the yield? That's what I would do as well. - Robert From stefan_ml at behnel.de Thu Dec 9 11:10:38 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Dec 2010 11:10:38 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> Message-ID: <4D00AB1E.5020706@behnel.de> Robert Bradshaw, 09.12.2010 11:00: > On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: >> Vitja Makarov, 08.12.2010 22:45: >>> Please review this patch. It's not yet finished, and mostly doesn't work. >>> But you can take a look at patch and generated code. >> >> *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send >> in a new patch when you have it. If you use hg, you can use "hg bundle" to >> collect commit series. Don't know about git. >> >> BTW, why not set up a github branch for this? > > That's exactly what I was thinking. It'll make it easier to review as > well, and for others to contribute without disturbing the main line. BTW, when this goes in, the next Cython version to release is basically 1.0, right? There are some minor Python compatibility issues left, but that'll be the last major feature, the way I count it. Stefan From robertwb at math.washington.edu Thu Dec 9 11:34:35 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 02:34:35 -0800 Subject: [Cython] General generator expressions In-Reply-To: <4D00AB1E.5020706@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D00AB1E.5020706@behnel.de> Message-ID: On Thu, Dec 9, 2010 at 2:10 AM, Stefan Behnel wrote: > Robert Bradshaw, 09.12.2010 11:00: >> On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: >>> Vitja Makarov, 08.12.2010 22:45: >>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>> But you can take a look at patch and generated code. >>> >>> *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send >>> in a new patch when you have it. If you use hg, you can use "hg bundle" to >>> collect commit series. Don't know about git. >>> >>> BTW, why not set up a github branch for this? >> >> That's exactly what I was thinking. It'll make it easier to review as >> well, and for others to contribute without disturbing the main line. > > BTW, when this goes in, the next Cython version to release is basically > 1.0, right? There are some minor Python compatibility issues left, but > that'll be the last major feature, the way I count it. This will be the last major feature before 1.0, but I don't know that the first release this is in should be the 1.0 release. I also think we should set the compatibility bar a bit higher for 1.0 as well--I want a concrete list of all the differences between Python X.y and Cython (for un-annotated code) and to be able to say that, if you don't count on the behavior in this list (e.g. identity vs equality for floating point literals, maybe stack frames) then you should be able to compile your code and have it just work, but hopefully faster. At the very least, until we at least attempt to make such a list, we won't know where we stand (the regression tests presumably being somewhat of a starting point, though I'd like to get all of them passing (unless they deal with implementation details...)). - Robert From robertwb at math.washington.edu Thu Dec 9 11:48:46 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 02:48:46 -0800 Subject: [Cython] make cython stop on the first error In-Reply-To: References: <4CFB2254.80705@student.matnat.uio.no> Message-ID: On Sun, Dec 5, 2010 at 5:42 PM, Ondrej Certik wrote: > On Sat, Dec 4, 2010 at 11:49 PM, Robert Bradshaw > wrote: >> On Sat, Dec 4, 2010 at 9:25 PM, Dag Sverre Seljebotn >> wrote: >>> On 12/05/2010 01:27 AM, Ondrej Certik wrote: >>>> On Sat, Dec 4, 2010 at 3:38 PM, Vitja Makarov ?wrote: >>>> >>>>> 2010/12/4 Ondrej Certik: >>>>> >>>>>> Hi, >>>>>> >>>>>> I was missing the option to tell Cython to stop on the first error. So >>>>>> I have implemented it: >>>>>> >>>>>> https://github.com/cython/cython/pull/3 >>>>>> >>>>>> Ondrej >>>>>> >>>>>> >>>>> Hi! >>>>> >>>>> Is that a good idea to call sys.exit() in Errors.py maybe there is >>>>> another way to stop compilation? >>>>> >>>> Yes, that is something that I was not sure when I coded it, as I am >>>> not too familiar with Cython internals. >>>> >>> >>> The usecases are more important than the internals: Imagine for a moment >>> what this will to do pyximport, or build systems able to launch Cython >>> in-process (SCons, waf). >> >> I've made it raise the same error it does between phases to signal the >> compiler should abort. > > After this "abort" change, here is how it looks like: > > > ondrej at crow:~/repos/hermes/hermes1d/hermes1d(fix)$ make > [ ?4%] Built target cython_utils > [ ?8%] Built target numpy_utils > [ 93%] Built target hermes1d-debug > [ 95%] Cythonizing h1d_wrapper.pyx > > Error compiling Cython file: > ------------------------------------------------------------ > ... > ? ? ? ?prj_type = hermes1d.H1D_H1_ortho_global > ? ?else: > ? ? ? ?raise ValueError("Unknown projection type") > ? ?global _A > ? ?_A = f > ? ?hermes1d.assemble_projection_matrix_rhs(mesh.thisptr, A.thisptr, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ > ------------------------------------------------------------ > > /home/ondrej/repos/hermes/hermes1d/hermes1d/h1d_wrapper/h1d_wrapper.pyx:369:48: > Cannot convert 'Space *' to Python object > > Error compiling Cython file: > ------------------------------------------------------------ > ... > ? ? ? ?prj_type = hermes1d.H1D_H1_ortho_global > ? ?else: > ? ? ? ?raise ValueError("Unknown projection type") > ? ?global _A > ? ?_A = f > ? ?hermes1d.assemble_projection_matrix_rhs(mesh.thisptr, A.thisptr, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ > ------------------------------------------------------------ > > /home/ondrej/repos/hermes/hermes1d/hermes1d/h1d_wrapper/h1d_wrapper.pyx:369:48: > Compiler crash in AnalyseExpressionsTransform > > ModuleNode.body = StatListNode(h1d_wrapper.pyx:6:0) > StatListNode.stats[19] = StatListNode(h1d_wrapper.pyx:358:0) > StatListNode.stats[0] = DefNode(h1d_wrapper.pyx:358:0, > ? ?modifiers = [...]/0, > ? ?name = u'assemble_projection_matrix_rhs', > ? ?num_required_args = 4, > ? ?reqd_kw_flags_cname = '0') > File 'Nodes.py', line 351, in analyse_expressions: > StatListNode(h1d_wrapper.pyx:360:4) > File 'Nodes.py', line 3346, in analyse_expressions: > ExprStatNode(h1d_wrapper.pyx:369:43) > File 'ExprNodes.py', line 322, in analyse_expressions: > SimpleCallNode(h1d_wrapper.pyx:369:43, > ? ?analysed = True, > ? ?use_managed_ref = True) > File 'ExprNodes.py', line 2865, in analyse_types: > SimpleCallNode(h1d_wrapper.pyx:369:43, > ? ?analysed = True, > ? ?use_managed_ref = True) > File 'ExprNodes.py', line 3947, in analyse_types: > TupleNode(h1d_wrapper.pyx:369:43, > ? ?is_sequence_constructor = 1, > ? ?use_managed_ref = True) > File 'ExprNodes.py', line 3742, in analyse_types: > TupleNode(h1d_wrapper.pyx:369:43, > ? ?is_sequence_constructor = 1, > ? ?use_managed_ref = True) > File 'ExprNodes.py', line 614, in coerce_to_pyobject: > AttributeNode(h1d_wrapper.pyx:369:48, > ? ?attribute = u'thisptr', > ? ?is_attribute = 1, > ? ?member = u'thisptr', > ? ?needs_none_check = True, > ? ?op = '->', > ? ?use_managed_ref = True) > File 'ExprNodes.py', line 3317, in coerce_to: > AttributeNode(h1d_wrapper.pyx:369:48, > ? ?attribute = u'thisptr', > ? ?is_attribute = 1, > ? ?member = u'thisptr', > ? ?needs_none_check = True, > ? ?op = '->', > ? ?use_managed_ref = True) > File 'ExprNodes.py', line 580, in coerce_to: > AttributeNode(h1d_wrapper.pyx:369:48, > ? ?attribute = u'thisptr', > ? ?is_attribute = 1, > ? ?member = u'thisptr', > ? ?needs_none_check = True, > ? ?op = '->', > ? ?use_managed_ref = True) > File 'ExprNodes.py', line 7076, in __init__: > CoerceToPyTypeNode(h1d_wrapper.pyx:369:48, > ? ?is_temp = 1, > ? ?use_managed_ref = True) > > Compiler crash traceback from this point on: > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/ExprNodes.py", > line 7076, in __init__ > ? ?"Cannot convert '%s' to Python object" % arg.type) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Errors.py", line > 151, in error > ? ?report_error(err) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Errors.py", line > 143, in report_error > ? ?raise InternalError, "abort" > InternalError: Internal compiler error: abort > Traceback (most recent call last): > ?File "/home/ondrej/usr/bin/cython", line 8, in > ? ?main(command_line = 1) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line 773, in main > ? ?result = compile(sources, options) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line > 748, in compile > ? ?return compile_multiple(source, options) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line > 720, in compile_multiple > ? ?result = run_pipeline(source, options) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line > 585, in run_pipeline > ? ?err, enddata = context.run_pipeline(pipeline, source) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Main.py", line > 231, in run_pipeline > ? ?Errors.report_error(err) > ?File "/home/ondrej/usr/lib/python/Cython/Compiler/Errors.py", line > 143, in report_error > ? ?raise InternalError, "abort" > Cython.Compiler.Errors.InternalError: Internal compiler error: abort > make[2]: *** [hermes1d/h1d_wrapper/h1d_wrapper.cpp] Error 1 > make[1]: *** [hermes1d/h1d_wrapper/CMakeFiles/h1d_wrapper.dir/all] Error 2 > make: *** [all] Error 2 > > > > > Now compare to the previous situation: > > > ondrej at crow:~/repos/hermes/hermes1d/hermes1d(fix)$ make > [ ?4%] Built target cython_utils > [ ?8%] Built target numpy_utils > [ 93%] Built target hermes1d-debug > [ 95%] Cythonizing h1d_wrapper.pyx > > Error compiling Cython file: > ------------------------------------------------------------ > ... > ? ? ? ?prj_type = hermes1d.H1D_H1_ortho_global > ? ?else: > ? ? ? ?raise ValueError("Unknown projection type") > ? ?global _A > ? ?_A = f > ? ?hermes1d.assemble_projection_matrix_rhs(mesh.thisptr, A.thisptr, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ > ------------------------------------------------------------ > > /home/ondrej/repos/hermes/hermes1d/hermes1d/h1d_wrapper/h1d_wrapper.pyx:369:48: > Cannot convert 'Space *' to Python object > make[2]: *** [hermes1d/h1d_wrapper/h1d_wrapper.cpp] Error 1 > make[1]: *** [hermes1d/h1d_wrapper/CMakeFiles/h1d_wrapper.dir/all] Error 2 > make: *** [all] Error 2 > > > > > So I believe that the latter is simpler. So where should we catch the > InternalError exception? Maybe we should create a StopOnFirstError > exception, and then catch it and print "exiting the compilation due to > --fatal-errors command line switch"? Interestingly enough, this all goes away when I tried to get a better handle on it by enabling debug_no_exception_intercept. The problem is that it crashes while trying to report the error, which leads to the traceback. I'll push a fix. - Robert From vitja.makarov at gmail.com Thu Dec 9 11:50:05 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 9 Dec 2010 13:50:05 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D00AB1E.5020706@behnel.de> Message-ID: 2010/12/9 Robert Bradshaw : > On Thu, Dec 9, 2010 at 2:10 AM, Stefan Behnel wrote: >> Robert Bradshaw, 09.12.2010 11:00: >>> On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: >>>> Vitja Makarov, 08.12.2010 22:45: >>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>> But you can take a look at patch and generated code. >>>> >>>> *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send >>>> in a new patch when you have it. If you use hg, you can use "hg bundle" to >>>> collect commit series. Don't know about git. >>>> >>>> BTW, why not set up a github branch for this? >>> >>> That's exactly what I was thinking. It'll make it easier to review as >>> well, and for others to contribute without disturbing the main line. >> >> BTW, when this goes in, the next Cython version to release is basically >> 1.0, right? There are some minor Python compatibility issues left, but >> that'll be the last major feature, the way I count it. > > This will be the last major feature before 1.0, but I don't know that > the first release this is in should be the 1.0 release. > > I also think we should set the compatibility bar a bit higher for 1.0 > as well--I want a concrete list of all the differences between Python > X.y and Cython (for un-annotated code) and to be able to say that, if > you don't count on the behavior in this list (e.g. identity vs > equality for floating point literals, maybe stack frames) then you > should be able to compile your code and have it just work, but > hopefully faster. At the very least, until we at least attempt to make > such a list, we won't know where we stand (the regression tests > presumably being somewhat of a starting point, though I'd like to get > all of them passing (unless they deal with implementation > details...)). > > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > >> - Comprehensions are now handled by OldYieldExprNode > >... another one of those bad names. ;) > >Any reason you couldn't reuse it for both? Didn't yet implemented ;) GenexpNode should be replaced with DefFunc one that's easy but I want to fix refnanny first. Then fix refcount leaks. >> - I don't know how to make refnanny work, should refnanny context be >> stored in closure? > > Good call. That's a tricky question. > > What about generating a GIVEREF for every Python temp you store in the > closure, a GOTREF for everything you take back out, and finish/restart the > refnanny context around the yield? Yes, that would help. But the main problem is refnanny context setupcontext defines local variable __pyx_refnanny, we should have one more macro refnanny_reinitalize_context() >> BTW, why not set up a github branch for this? > > That's exactly what I was thinking. It'll make it easier to review as > well, and for others to contribute without disturbing the main line. Should I fork cython from github? Is github insync with hg repo? What is main repo now? ps: I think we should horry with 1.0 release -- vitja. From robertwb at math.washington.edu Thu Dec 9 12:22:58 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 03:22:58 -0800 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D00AB1E.5020706@behnel.de> Message-ID: On Thu, Dec 9, 2010 at 2:50 AM, Vitja Makarov wrote: > 2010/12/9 Robert Bradshaw : >> On Thu, Dec 9, 2010 at 2:10 AM, Stefan Behnel wrote: >>> Robert Bradshaw, 09.12.2010 11:00: >>>> On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: >>>>> Vitja Makarov, 08.12.2010 22:45: >>>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>>> But you can take a look at patch and generated code. >>>>> >>>>> *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send >>>>> in a new patch when you have it. If you use hg, you can use "hg bundle" to >>>>> collect commit series. Don't know about git. >>>>> >>>>> BTW, why not set up a github branch for this? >>>> >>>> That's exactly what I was thinking. It'll make it easier to review as >>>> well, and for others to contribute without disturbing the main line. >>> >>> BTW, when this goes in, the next Cython version to release is basically >>> 1.0, right? There are some minor Python compatibility issues left, but >>> that'll be the last major feature, the way I count it. >> >> This will be the last major feature before 1.0, but I don't know that >> the first release this is in should be the 1.0 release. >> >> I also think we should set the compatibility bar a bit higher for 1.0 >> as well--I want a concrete list of all the differences between Python >> X.y and Cython (for un-annotated code) and to be able to say that, if >> you don't count on the behavior in this list (e.g. identity vs >> equality for floating point literals, maybe stack frames) then you >> should be able to compile your code and have it just work, but >> hopefully faster. At the very least, until we at least attempt to make >> such a list, we won't know where we stand (the regression tests >> presumably being somewhat of a starting point, though I'd like to get >> all of them passing (unless they deal with implementation >> details...)). >> >> - Robert >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > >>> ? - Comprehensions are now handled by OldYieldExprNode >> >>... another one of those bad names. ;) >> >>Any reason you couldn't reuse it for both? > > Didn't yet implemented ;) GenexpNode should be replaced with DefFunc > one that's easy but I want to fix refnanny first. Then fix refcount > leaks. > >>> ? - I don't know how to make refnanny work, should refnanny context be >>> stored in closure? >> >> Good call. That's a tricky question. >> >> What about generating a GIVEREF for every Python temp you store in the >> closure, a GOTREF for everything you take back out, and finish/restart the >> refnanny context around the yield? > > Yes, that would help. But the main problem is refnanny context > setupcontext defines local variable __pyx_refnanny, we should have one > more macro refnanny_reinitalize_context() Yes, we'll need to split this up into two macros. This has actually bitten be before, when I was writing the profiler. Did you know that ; all by itself is considered a statement, hence one can't declare variables after it in some strict compilers? >>> BTW, why not set up a github branch for this? >> >> That's exactly what I was thinking. It'll make it easier to review as >> well, and for others to contribute without disturbing the main line. > > Should I fork cython from github? Yes, please do. > Is github insync with hg repo? I've been manually keeping the two in sync--just pushed now. > What is main repo now? It's a DVCS :). Some people have been developing on the git side of things, and some on the hg side of things, and I've been manually syncing the two now and then. Obviously this isn't a long term solution, and I think setting up an automated synchronization both ways would cause more confusion than benefit, so as soon as the buildbot starts looking at github I'll make the hg ones read-only. (Probably worth waiting until after the release to do this.) > ps: I think we should hurry with 1.0 release Yes, I'd like to hurry too, and with all the stuff you're doing it should be soon :). I just think it's important to be able to say "compiles all Python code" rather than "compiles most Python code" and have a (hopefully small) concrete lits of well-delineated caveats. - Robert From robertwb at math.washington.edu Thu Dec 9 13:27:42 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 04:27:42 -0800 Subject: [Cython] Cython 0.14 In-Reply-To: References: Message-ID: A beta is up at http://cython.org/release/Cython-0.14.beta1.tar.gz . I'm planning on rolling a release candidate in the next day or two, and if there are no issues, that will become the release. - Robert From dagss at student.matnat.uio.no Thu Dec 9 14:58:59 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 09 Dec 2010 14:58:59 +0100 Subject: [Cython] Complex to float coercion Message-ID: <4D00E0A3.2060607@student.matnat.uio.no> In the ongoing fwrapification of SciPy [1], I'm getting regression test breakage because of this: def func(float x): return x >>> func(1 + 0j) Traceback (most recent call last): ... TypeError: can't convert complex to float I'm wondering: Would people be OK to Cython so that when the imaginary part is *exactly* 0, it is OK to convert to native float? Or is failing hard the preferred behaviour? (For all I know, SciPy may prefer to change their behaviour; so I'm just doing a poll for now.) I guess the way to do it would be to try float conversion first, and if that fails, try complex conversion and see if the imaginary part ends up as 0. That is, it would mean that __complex__ would be called on passed in objects that do not support __float__ in the example above. Dag Sverre [1] http://fortrancython.wordpress.com/2010/11/03/fwrap-to-be-used-by-scipy/ From robertwb at math.washington.edu Thu Dec 9 15:00:33 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 06:00:33 -0800 Subject: [Cython] Cython 0.14 In-Reply-To: References: Message-ID: On Thu, Dec 9, 2010 at 4:27 AM, Robert Bradshaw wrote: > A beta is up at http://cython.org/release/Cython-0.14.beta1.tar.gz . > I'm planning on rolling a release candidate in the next day or two, > and if there are no issues, that will become the release. Make that http://cython.org/release/Cython-0.14.beta2.tar.gz From robertwb at math.washington.edu Thu Dec 9 15:08:10 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 06:08:10 -0800 Subject: [Cython] Complex to float coercion In-Reply-To: <4D00E0A3.2060607@student.matnat.uio.no> References: <4D00E0A3.2060607@student.matnat.uio.no> Message-ID: On Thu, Dec 9, 2010 at 5:58 AM, Dag Sverre Seljebotn wrote: > In the ongoing fwrapification of SciPy [1], I'm getting regression test > breakage because of this: > > def func(float x): > ? ? return x > > ?>>> func(1 + 0j) > ? ? Traceback (most recent call last): > ? ? ... > ? ? TypeError: can't convert complex to float > > > I'm wondering: Would people be OK to Cython so that when the imaginary > part is *exactly* 0, it is OK to convert to native float? Or is failing > hard the preferred behaviour? > > (For all I know, SciPy may prefer to change their behaviour; so I'm just > doing a poll for now.) > > I guess the way to do it would be to try float conversion first, and if > that fails, try complex conversion and see if the imaginary part ends up > as 0. That is, it would mean that __complex__ would be called on passed > in objects that do not support __float__ in the example above. I'm -1 on making this change. If the author of the user class defined a __complex__ but not a __float__, we should respect that. Not to mention the added verbosity to the object -> double path. - Robert From dagss at student.matnat.uio.no Thu Dec 9 15:21:20 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 09 Dec 2010 15:21:20 +0100 Subject: [Cython] Complex to float coercion In-Reply-To: References: <4D00E0A3.2060607@student.matnat.uio.no> Message-ID: <4D00E5E0.3040201@student.matnat.uio.no> On 12/09/2010 03:08 PM, Robert Bradshaw wrote: > On Thu, Dec 9, 2010 at 5:58 AM, Dag Sverre Seljebotn > wrote: > >> In the ongoing fwrapification of SciPy [1], I'm getting regression test >> breakage because of this: >> >> def func(float x): >> return x >> >> >>> func(1 + 0j) >> Traceback (most recent call last): >> ... >> TypeError: can't convert complex to float >> >> >> I'm wondering: Would people be OK to Cython so that when the imaginary >> part is *exactly* 0, it is OK to convert to native float? Or is failing >> hard the preferred behaviour? >> >> (For all I know, SciPy may prefer to change their behaviour; so I'm just >> doing a poll for now.) >> >> I guess the way to do it would be to try float conversion first, and if >> that fails, try complex conversion and see if the imaginary part ends up >> as 0. That is, it would mean that __complex__ would be called on passed >> in objects that do not support __float__ in the example above. >> > I'm -1 on making this change. If the author of the user class defined > a __complex__ but not a __float__, we should respect that. Not to > mention the added verbosity to the object -> double path. > How about a special case for the Python complex type? Still -1? Dag Sverre From ndbecker2 at gmail.com Thu Dec 9 15:21:35 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 09 Dec 2010 09:21:35 -0500 Subject: [Cython] Complex to float coercion References: <4D00E0A3.2060607@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > In the ongoing fwrapification of SciPy [1], I'm getting regression test > breakage because of this: > > def func(float x): > return x > > >>> func(1 + 0j) > Traceback (most recent call last): > ... > TypeError: can't convert complex to float > > > I'm wondering: Would people be OK to Cython so that when the imaginary > part is *exactly* 0, it is OK to convert to native float? Or is failing > hard the preferred behaviour? > > (For all I know, SciPy may prefer to change their behaviour; so I'm just > doing a poll for now.) > > I guess the way to do it would be to try float conversion first, and if > that fails, try complex conversion and see if the imaginary part ends up > as 0. That is, it would mean that __complex__ would be called on passed > in objects that do not support __float__ in the example above. > > Dag Sverre > > > [1] > http://fortrancython.wordpress.com/2010/11/03/fwrap-to-be-used-by-scipy/ I prefer failing hard. I believe that allowing complex to be used in place of float is a wart in numpy that leads to unintentional hidden mistakes. If I want to take the real part I would say so. The proposal to allow when im part is exactly zero is a horrible kluge. From ndbecker2 at gmail.com Thu Dec 9 15:46:24 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 09 Dec 2010 09:46:24 -0500 Subject: [Cython] Cython 0.14 References: Message-ID: Robert Bradshaw wrote: > A beta is up at http://cython.org/release/Cython-0.14.beta1.tar.gz . > I'm planning on rolling a release candidate in the next day or two, > and if there are no issues, that will become the release. > > - Robert fedora 14 x86_64 python runtests.py -x numpy Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] Running tests against Cython 0.14.beta2 import_star.cpp: In function ?int __Pyx_import_all_from(PyObject*, PyObject*)?: import_star.cpp:658:60: warning: deprecated conversion from string constant to ?char*? gcc: /usr/lib64/libpython2.7.a: No such file or directory make: *** [embedded] Error 1 ====================================================================== FAIL: test_embed (__main__.EmbedTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 829, in test_embed "make PYTHON='%s' LIBDIR1='%s' test > make.output" % (sys.executable, libdir)) == 0) AssertionError: False is not True ---------------------------------------------------------------------- Ran 4782 tests in 807.203s FAILED (failures=1) ALL DONE From Paul.Connell at disney.com Thu Dec 9 15:54:11 2010 From: Paul.Connell at disney.com (Connell, Paul) Date: Thu, 9 Dec 2010 14:54:11 +0000 Subject: [Cython] Cython 0.14 In-Reply-To: References: Message-ID: Hi, I can't install the 0.14 Beta2 using Windows 7 x64, Python 2.7 x64. One of the generated modules fails to compile. building 'Cython.Compiler.Parsing' extension C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python27\include -Ic:\Python27\PC /TcCython\Compiler\Parsing.c /Fobuild\temp.win-amd64-2.7\Release\Cython\Compiler\Parsing.obj Parsing.c Cython\Compiler\Parsing.c(8519) : error C2275: 'PyObject' : illegal use of this type as an expression c:\python27\include\object.h(108) : see declaration of 'PyObject' Cython\Compiler\Parsing.c(8519) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8520) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8520) : warning C4047: '=' : 'int' differs in levels of indirection from 'PyObject *' Cython\Compiler\Parsing.c(8560) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8560) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8560) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8561) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8561) : warning C4047: '=' : 'int' differs in levels of indirection from 'PyObject *' Cython\Compiler\Parsing.c(8576) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8576) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8576) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8577) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8577) : warning C4047: '=' : 'int' differs in levels of indirection from 'PyObject *' Cython\Compiler\Parsing.c(8597) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8597) : warning C4047: 'function' : 'PyObject *' differs in levels of indirection from 'int' Cython\Compiler\Parsing.c(8597) : warning C4024: 'PyDict_SetItem' : different types for formal and actual parameter 3 Cython\Compiler\Parsing.c(8607) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8607) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8607) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8611) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8611) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(8611) : error C2065: '__pyx_8genexpr0__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31797) : error C2275: 'PyObject' : illegal use of this type as an expression c:\python27\include\object.h(108) : see declaration of 'PyObject' Cython\Compiler\Parsing.c(31797) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31798) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31798) : warning C4047: '=' : 'int' differs in levels of indirection from 'PyObject *' Cython\Compiler\Parsing.c(31854) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31854) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31854) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31855) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31855) : warning C4047: '=' : 'int' differs in levels of indirection from 'PyObject *' Cython\Compiler\Parsing.c(31870) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31870) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31870) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31871) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31871) : warning C4047: '=' : 'int' differs in levels of indirection from 'PyObject *' Cython\Compiler\Parsing.c(31891) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31891) : warning C4047: 'function' : 'PyObject *' differs in levels of indirection from 'int' Cython\Compiler\Parsing.c(31891) : warning C4024: 'PyDict_SetItem' : different types for formal and actual parameter 3 Cython\Compiler\Parsing.c(31901) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31901) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31901) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31905) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31905) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier Cython\Compiler\Parsing.c(31905) : error C2065: '__pyx_8genexpr2__pyx_v_value' : undeclared identifier error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe"' failed with exit status 2 Robert Bradshaw wrote: > A beta is up at http://cython.org/release/Cython-0.14.beta1.tar.gz . > I'm planning on rolling a release candidate in the next day or two, > and if there are no issues, that will become the release. > > - Robert fedora 14 x86_64 python runtests.py -x numpy Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] From stefan_ml at behnel.de Thu Dec 9 16:36:43 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Dec 2010 16:36:43 +0100 Subject: [Cython] Complex to float coercion In-Reply-To: References: <4D00E0A3.2060607@student.matnat.uio.no> Message-ID: <4D00F78B.9090401@behnel.de> Neal Becker, 09.12.2010 15:21: > I prefer failing hard. I believe that allowing complex to be used in place > of float is a wart in numpy that leads to unintentional hidden mistakes. If > I want to take the real part I would say so. +1 Stefan From stefan_ml at behnel.de Thu Dec 9 16:59:28 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Dec 2010 16:59:28 +0100 Subject: [Cython] Cython 0.14 In-Reply-To: References: Message-ID: <4D00FCE0.9000708@behnel.de> Connell, Paul, 09.12.2010 15:54: > I can't install the 0.14 Beta2 using Windows 7 x64, Python 2.7 x64. One of the generated modules fails to compile. > > building 'Cython.Compiler.Parsing' extension > C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python27\include > -Ic:\Python27\PC /TcCython\Compiler\Parsing.c /Fobuild\temp.win-amd64-2.7\Release\Cython\Compiler\Parsing.obj > Parsing.c > Cython\Compiler\Parsing.c(8519) : error C2275: 'PyObject' : illegal use of this type as an expression Thanks for catching this. I pushed the following patch to cython-devel. Could you apply it and retry? Stefan diff -r 0ede6da42c99 Cython/Compiler/ExprNodes.py --- a/Cython/Compiler/ExprNodes.py Thu Dec 09 05:41:59 2010 -0800 +++ b/Cython/Compiler/ExprNodes.py Thu Dec 09 16:55:14 2010 +0100 @@ -4181,12 +4181,13 @@ code.put_var_declaration(entry) if entry.type.is_pyobject and entry.used: py_entries.append(entry) - code.put_init_var_to_py_none(entry) if not py_entries: # no local Python references => no cleanup required generate_inner_evaluation_code(code) code.putln('} /* exit inner scope */') return + for entry in py_entries: + code.put_init_var_to_py_none(entry) # must free all local Python references at each exit point old_loop_labels = tuple(code.new_loop_labels()) From Paul.Connell at disney.com Thu Dec 9 17:27:44 2010 From: Paul.Connell at disney.com (Connell, Paul) Date: Thu, 9 Dec 2010 16:27:44 +0000 Subject: [Cython] Cython 0.14 In-Reply-To: <4D00FCE0.9000708@behnel.de> References: <4D00FCE0.9000708@behnel.de> Message-ID: Yes, that's worked. Thanks! FYI I have also now checked, and all my Cython code here compiles and works fine with it too (using --cplus), no regression problems discovered yet. Paul -----Original Message----- From: cython-dev-bounces at codespeak.net [mailto:cython-dev-bounces at codespeak.net] On Behalf Of Stefan Behnel >> I can't install the 0.14 Beta2 using Windows 7 x64, Python 2.7 x64. One of the generated modules fails to compile. >Thanks for catching this. I pushed the following patch to cython-devel. >Could you apply it and retry? >Stefan From vitja.makarov at gmail.com Thu Dec 9 18:31:41 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 9 Dec 2010 20:31:41 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D00AB1E.5020706@behnel.de> Message-ID: 2010/12/9 Robert Bradshaw : > On Thu, Dec 9, 2010 at 2:50 AM, Vitja Makarov wrote: >> 2010/12/9 Robert Bradshaw : >>> On Thu, Dec 9, 2010 at 2:10 AM, Stefan Behnel wrote: >>>> Robert Bradshaw, 09.12.2010 11:00: >>>>> On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: >>>>>> Vitja Makarov, 08.12.2010 22:45: >>>>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>>>> But you can take a look at patch and generated code. >>>>>> >>>>>> *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send >>>>>> in a new patch when you have it. If you use hg, you can use "hg bundle" to >>>>>> collect commit series. Don't know about git. >>>>>> >>>>>> BTW, why not set up a github branch for this? >>>>> >>>>> That's exactly what I was thinking. It'll make it easier to review as >>>>> well, and for others to contribute without disturbing the main line. >>>> >>>> BTW, when this goes in, the next Cython version to release is basically >>>> 1.0, right? There are some minor Python compatibility issues left, but >>>> that'll be the last major feature, the way I count it. >>> >>> This will be the last major feature before 1.0, but I don't know that >>> the first release this is in should be the 1.0 release. >>> >>> I also think we should set the compatibility bar a bit higher for 1.0 >>> as well--I want a concrete list of all the differences between Python >>> X.y and Cython (for un-annotated code) and to be able to say that, if >>> you don't count on the behavior in this list (e.g. identity vs >>> equality for floating point literals, maybe stack frames) then you >>> should be able to compile your code and have it just work, but >>> hopefully faster. At the very least, until we at least attempt to make >>> such a list, we won't know where we stand (the regression tests >>> presumably being somewhat of a starting point, though I'd like to get >>> all of them passing (unless they deal with implementation >>> details...)). >>> >>> - Robert >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >>>> ? - Comprehensions are now handled by OldYieldExprNode >>> >>>... another one of those bad names. ;) >>> >>>Any reason you couldn't reuse it for both? >> >> Didn't yet implemented ;) GenexpNode should be replaced with DefFunc >> one that's easy but I want to fix refnanny first. Then fix refcount >> leaks. >> >>>> ? - I don't know how to make refnanny work, should refnanny context be >>>> stored in closure? >>> >>> Good call. That's a tricky question. >>> >>> What about generating a GIVEREF for every Python temp you store in the >>> closure, a GOTREF for everything you take back out, and finish/restart the >>> refnanny context around the yield? >> >> Yes, that would help. But the main problem is refnanny context >> setupcontext defines local variable __pyx_refnanny, we should have one >> more macro refnanny_reinitalize_context() > > Yes, we'll need to split this up into two macros. This has actually > bitten be before, when I was writing the profiler. Did you know that ; > all by itself is considered a statement, hence one can't declare > variables after it in some strict compilers? > >>>> BTW, why not set up a github branch for this? >>> >>> That's exactly what I was thinking. It'll make it easier to review as >>> well, and for others to contribute without disturbing the main line. >> >> Should I fork cython from github? > > Yes, please do. > >> Is github insync with hg repo? > > I've been manually keeping the two in sync--just pushed now. > >> What is main repo now? > > It's a DVCS :). Some people have been developing on the git side of > things, and some on the hg side of things, and I've been manually > syncing the two now and then. Obviously this isn't a long term > solution, and I think setting up an automated synchronization both > ways would cause more confusion than benefit, so as soon as the > buildbot starts looking at github I'll make the hg ones read-only. > (Probably worth waiting until after the release to do this.) > >> ps: I think we should hurry with 1.0 release > > Yes, I'd like to hurry too, and with all the stuff you're doing it > should be soon :). I just think it's important to be able to say > "compiles all Python code" rather than "compiles most Python code" and > have a (hopefully small) concrete lits of well-delineated caveats. > > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > I've created github repo https://github.com/vitek/cython/ And changed __Pyx_Generator to __CyGenerator... -- vitja. From vitja.makarov at gmail.com Thu Dec 9 21:45:03 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 9 Dec 2010 23:45:03 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D00AB1E.5020706@behnel.de> Message-ID: 2010/12/9 Vitja Makarov : > 2010/12/9 Robert Bradshaw : >> On Thu, Dec 9, 2010 at 2:50 AM, Vitja Makarov wrote: >>> 2010/12/9 Robert Bradshaw : >>>> On Thu, Dec 9, 2010 at 2:10 AM, Stefan Behnel wrote: >>>>> Robert Bradshaw, 09.12.2010 11:00: >>>>>> On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: >>>>>>> Vitja Makarov, 08.12.2010 22:45: >>>>>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>>>>> But you can take a look at patch and generated code. >>>>>>> >>>>>>> *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send >>>>>>> in a new patch when you have it. If you use hg, you can use "hg bundle" to >>>>>>> collect commit series. Don't know about git. >>>>>>> >>>>>>> BTW, why not set up a github branch for this? >>>>>> >>>>>> That's exactly what I was thinking. It'll make it easier to review as >>>>>> well, and for others to contribute without disturbing the main line. >>>>> >>>>> BTW, when this goes in, the next Cython version to release is basically >>>>> 1.0, right? There are some minor Python compatibility issues left, but >>>>> that'll be the last major feature, the way I count it. >>>> >>>> This will be the last major feature before 1.0, but I don't know that >>>> the first release this is in should be the 1.0 release. >>>> >>>> I also think we should set the compatibility bar a bit higher for 1.0 >>>> as well--I want a concrete list of all the differences between Python >>>> X.y and Cython (for un-annotated code) and to be able to say that, if >>>> you don't count on the behavior in this list (e.g. identity vs >>>> equality for floating point literals, maybe stack frames) then you >>>> should be able to compile your code and have it just work, but >>>> hopefully faster. At the very least, until we at least attempt to make >>>> such a list, we won't know where we stand (the regression tests >>>> presumably being somewhat of a starting point, though I'd like to get >>>> all of them passing (unless they deal with implementation >>>> details...)). >>>> >>>> - Robert >>>> _______________________________________________ >>>> Cython-dev mailing list >>>> Cython-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/cython-dev >>>> >>> >>>>> ? - Comprehensions are now handled by OldYieldExprNode >>>> >>>>... another one of those bad names. ;) >>>> >>>>Any reason you couldn't reuse it for both? >>> >>> Didn't yet implemented ;) GenexpNode should be replaced with DefFunc >>> one that's easy but I want to fix refnanny first. Then fix refcount >>> leaks. >>> >>>>> ? - I don't know how to make refnanny work, should refnanny context be >>>>> stored in closure? >>>> >>>> Good call. That's a tricky question. >>>> >>>> What about generating a GIVEREF for every Python temp you store in the >>>> closure, a GOTREF for everything you take back out, and finish/restart the >>>> refnanny context around the yield? >>> >>> Yes, that would help. But the main problem is refnanny context >>> setupcontext defines local variable __pyx_refnanny, we should have one >>> more macro refnanny_reinitalize_context() >> >> Yes, we'll need to split this up into two macros. This has actually >> bitten be before, when I was writing the profiler. Did you know that ; >> all by itself is considered a statement, hence one can't declare >> variables after it in some strict compilers? >> >>>>> BTW, why not set up a github branch for this? >>>> >>>> That's exactly what I was thinking. It'll make it easier to review as >>>> well, and for others to contribute without disturbing the main line. >>> >>> Should I fork cython from github? >> >> Yes, please do. >> >>> Is github insync with hg repo? >> >> I've been manually keeping the two in sync--just pushed now. >> >>> What is main repo now? >> >> It's a DVCS :). Some people have been developing on the git side of >> things, and some on the hg side of things, and I've been manually >> syncing the two now and then. Obviously this isn't a long term >> solution, and I think setting up an automated synchronization both >> ways would cause more confusion than benefit, so as soon as the >> buildbot starts looking at github I'll make the hg ones read-only. >> (Probably worth waiting until after the release to do this.) >> >>> ps: I think we should hurry with 1.0 release >> >> Yes, I'd like to hurry too, and with all the stuff you're doing it >> should be soon :). I just think it's important to be able to say >> "compiles all Python code" rather than "compiles most Python code" and >> have a (hopefully small) concrete lits of well-delineated caveats. >> >> - Robert >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > I've created github repo https://github.com/vitek/cython/ > > And changed __Pyx_Generator to __CyGenerator... > > -- > vitja. > I've fixed refnanny. Now regr are okay: ---------------------------------------------------------------------- Ran 4058 tests in 1520.368s FAILED (failures=18, errors=611, skipped=4) Not that much, genexpr aren't still supported. Git isn't so bad btw ;) -- vitja. From greg.ewing at canterbury.ac.nz Thu Dec 9 23:14:45 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 10 Dec 2010 11:14:45 +1300 Subject: [Cython] Complex to float coercion In-Reply-To: References: <4D00E0A3.2060607@student.matnat.uio.no> Message-ID: <4D0154D5.4090803@canterbury.ac.nz> Neal Becker wrote: > The proposal to allow when im part is exactly zero is a horrible kluge. Not to mention that "exactly zero" is often an ill-defined concept in numerical algorithms. You have no way of knowing whether it really is zero or just something small that happened to get rounded down to zero. -- Greg From dagss at student.matnat.uio.no Fri Dec 10 07:26:56 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 10 Dec 2010 07:26:56 +0100 Subject: [Cython] Complex to float coercion In-Reply-To: <4D0154D5.4090803@canterbury.ac.nz> References: <4D00E0A3.2060607@student.matnat.uio.no> <4D0154D5.4090803@canterbury.ac.nz> Message-ID: <4D01C830.8080901@student.matnat.uio.no> Greg Ewing wrote: > Neal Becker wrote: > > >> The proposal to allow when im part is exactly zero is a horrible kluge. >> > > Not to mention that "exactly zero" is often an ill-defined > concept in numerical algorithms. You have no way of knowing > whether it really is zero or just something small that > happened to get rounded down to zero. > Well, as it turns out when I looked at this in more detail, f2py in fact happily discards *any* imaginary part, not just 0. So we may probably as well make a departure from f2py behaviour anyway. (And I've come to agree with all of you.) Dag Sverre From robertwb at math.washington.edu Fri Dec 10 08:00:13 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 9 Dec 2010 23:00:13 -0800 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D00AB1E.5020706@behnel.de> Message-ID: On Thu, Dec 9, 2010 at 12:45 PM, Vitja Makarov wrote: > 2010/12/9 Vitja Makarov : >> 2010/12/9 Robert Bradshaw : >>> On Thu, Dec 9, 2010 at 2:50 AM, Vitja Makarov wrote: >>>> 2010/12/9 Robert Bradshaw : >>>>> On Thu, Dec 9, 2010 at 2:10 AM, Stefan Behnel wrote: >>>>>> Robert Bradshaw, 09.12.2010 11:00: >>>>>>> On Thu, Dec 9, 2010 at 1:46 AM, Stefan Behnel wrote: >>>>>>>> Vitja Makarov, 08.12.2010 22:45: >>>>>>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>>>>>> But you can take a look at patch and generated code. >>>>>>>> >>>>>>>> *Way* cool, thanks! I'll take a close look ASAP. Please feel free to send >>>>>>>> in a new patch when you have it. If you use hg, you can use "hg bundle" to >>>>>>>> collect commit series. Don't know about git. >>>>>>>> >>>>>>>> BTW, why not set up a github branch for this? >>>>>>> >>>>>>> That's exactly what I was thinking. It'll make it easier to review as >>>>>>> well, and for others to contribute without disturbing the main line. >>>>>> >>>>>> BTW, when this goes in, the next Cython version to release is basically >>>>>> 1.0, right? There are some minor Python compatibility issues left, but >>>>>> that'll be the last major feature, the way I count it. >>>>> >>>>> This will be the last major feature before 1.0, but I don't know that >>>>> the first release this is in should be the 1.0 release. >>>>> >>>>> I also think we should set the compatibility bar a bit higher for 1.0 >>>>> as well--I want a concrete list of all the differences between Python >>>>> X.y and Cython (for un-annotated code) and to be able to say that, if >>>>> you don't count on the behavior in this list (e.g. identity vs >>>>> equality for floating point literals, maybe stack frames) then you >>>>> should be able to compile your code and have it just work, but >>>>> hopefully faster. At the very least, until we at least attempt to make >>>>> such a list, we won't know where we stand (the regression tests >>>>> presumably being somewhat of a starting point, though I'd like to get >>>>> all of them passing (unless they deal with implementation >>>>> details...)). >>>>> >>>>> - Robert >>>>> _______________________________________________ >>>>> Cython-dev mailing list >>>>> Cython-dev at codespeak.net >>>>> http://codespeak.net/mailman/listinfo/cython-dev >>>>> >>>> >>>>>> ? - Comprehensions are now handled by OldYieldExprNode >>>>> >>>>>... another one of those bad names. ;) >>>>> >>>>>Any reason you couldn't reuse it for both? >>>> >>>> Didn't yet implemented ;) GenexpNode should be replaced with DefFunc >>>> one that's easy but I want to fix refnanny first. Then fix refcount >>>> leaks. >>>> >>>>>> ? - I don't know how to make refnanny work, should refnanny context be >>>>>> stored in closure? >>>>> >>>>> Good call. That's a tricky question. >>>>> >>>>> What about generating a GIVEREF for every Python temp you store in the >>>>> closure, a GOTREF for everything you take back out, and finish/restart the >>>>> refnanny context around the yield? >>>> >>>> Yes, that would help. But the main problem is refnanny context >>>> setupcontext defines local variable __pyx_refnanny, we should have one >>>> more macro refnanny_reinitalize_context() >>> >>> Yes, we'll need to split this up into two macros. This has actually >>> bitten be before, when I was writing the profiler. Did you know that ; >>> all by itself is considered a statement, hence one can't declare >>> variables after it in some strict compilers? >>> >>>>>> BTW, why not set up a github branch for this? >>>>> >>>>> That's exactly what I was thinking. It'll make it easier to review as >>>>> well, and for others to contribute without disturbing the main line. >>>> >>>> Should I fork cython from github? >>> >>> Yes, please do. >>> >>>> Is github insync with hg repo? >>> >>> I've been manually keeping the two in sync--just pushed now. >>> >>>> What is main repo now? >>> >>> It's a DVCS :). Some people have been developing on the git side of >>> things, and some on the hg side of things, and I've been manually >>> syncing the two now and then. Obviously this isn't a long term >>> solution, and I think setting up an automated synchronization both >>> ways would cause more confusion than benefit, so as soon as the >>> buildbot starts looking at github I'll make the hg ones read-only. >>> (Probably worth waiting until after the release to do this.) >>> >>>> ps: I think we should hurry with 1.0 release >>> >>> Yes, I'd like to hurry too, and with all the stuff you're doing it >>> should be soon :). I just think it's important to be able to say >>> "compiles all Python code" rather than "compiles most Python code" and >>> have a (hopefully small) concrete lits of well-delineated caveats. >>> >>> - Robert >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >> I've created github repo https://github.com/vitek/cython/ >> >> And changed __Pyx_Generator to __CyGenerator... >> >> -- >> vitja. >> > > I've fixed refnanny. Now regr are okay: > > ---------------------------------------------------------------------- > Ran 4058 tests in 1520.368s > > FAILED (failures=18, errors=611, skipped=4) Cool. From stefan_ml at behnel.de Fri Dec 10 08:34:07 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Dec 2010 08:34:07 +0100 Subject: [Cython] General generator expressions In-Reply-To: <4D00A586.8060801@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> Message-ID: <4D01D7EF.6060906@behnel.de> Stefan Behnel, 09.12.2010 10:46: > Vitja Makarov, 08.12.2010 22:45: >> Please review this patch. It's not yet finished, and mostly doesn't work. >> But you can take a look at patch and generated code. > > I'll take a close look ASAP. > >> - Temps are saved/restored/allocated inside YieldExprNode using >> helper ClosureTempAllocator Ok, that currently lives in ParseTreeTransforms, which is totally the wrong place for anything that generates code. It also mixes functionality that normally lives in Symtab.py with functionality that lives in Code.py. Could you give me ("scoder", or better all Cython devs) commit rights to your repo so that I can start refactoring the code and fix problems I find? I could also add Hudson jobs for it. Robert, would it be ok to add them for all Python versions? >> - PyxGenerator methods are defined via declare_var > > Should we call that "CyGenerator" ? :) I actually meant to call it "CyGenerator" at the Python level, but I now see that it's actually a purely internal thing. That makes __Pyx_Generator a better choice, given that "__pyx_" is the globally reserved prefix in Cython code. Eventually, we may(!) want to reuse the superclass across modules, but I currently have no idea how that could be made to work. So it's just another thing that we don't need to worry about now. Stefan From vitja.makarov at gmail.com Fri Dec 10 09:22:03 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 10 Dec 2010 11:22:03 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D01D7EF.6060906@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> Message-ID: 2010/12/10 Stefan Behnel : > Stefan Behnel, 09.12.2010 10:46: >> Vitja Makarov, 08.12.2010 22:45: >>> Please review this patch. It's not yet finished, and mostly doesn't work. >>> But you can take a look at patch and generated code. >> >> I'll take a close look ASAP. >> >>> ? ?- Temps are saved/restored/allocated inside YieldExprNode using >>> helper ClosureTempAllocator > > Ok, that currently lives in ParseTreeTransforms, which is totally the wrong > place for anything that generates code. It also mixes functionality that > normally lives in Symtab.py with functionality that lives in Code.py. > Yes, I know. That was placed there to give it a try (actually I'm not sure what is the best place for it). ClosureTempAllocator now is attribute of defnode and all yieldnodes, that's not clean. ExprNode.StopIterationNode isn't expression should it be moved to Nodes.py? create_abstract_generator() is rather dirty I don't know how to bind C functions to CClass. I think that generator body could have PyObject *body(PyObject *self, PyObject *value) signature With value=Null meaning exception. > Could you give me ("scoder", or better all Cython devs) commit rights to > your repo so that I can start refactoring the code and fix problems I find? > I've added you, but I don't know how to add all devs together. > I could also add Hudson jobs for it. Robert, would it be ok to add them for > all Python versions? > > Yeah, please that would be nice. >>> ? ?- PyxGenerator methods are defined via declare_var >> >> Should we call that "CyGenerator" ? :) > > I actually meant to call it "CyGenerator" at the Python level, but I now > see that it's actually a purely internal thing. That makes __Pyx_Generator > a better choice, given that "__pyx_" is the globally reserved prefix in > Cython code. > Should rename it back ;) > Eventually, we may(!) want to reuse the superclass across modules, but I > currently have no idea how that could be made to work. So it's just another > thing that we don't need to worry about now. > Okay. How can I declare function entry with METH_VARGS signature? Using TypeSlots I can add METH_VARGS|METH_KWARGS entries but not single METH_VARGS That's preferd for "throw" method. -- vitja. From robertwb at math.washington.edu Fri Dec 10 19:15:10 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 10 Dec 2010 10:15:10 -0800 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> Message-ID: On Fri, Dec 10, 2010 at 12:22 AM, Vitja Makarov wrote: > 2010/12/10 Stefan Behnel : >> Stefan Behnel, 09.12.2010 10:46: >>> Vitja Makarov, 08.12.2010 22:45: >>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>> But you can take a look at patch and generated code. >>> >>> I'll take a close look ASAP. >>> >>>> ? ?- Temps are saved/restored/allocated inside YieldExprNode using >>>> helper ClosureTempAllocator >> >> Ok, that currently lives in ParseTreeTransforms, which is totally the wrong >> place for anything that generates code. It also mixes functionality that >> normally lives in Symtab.py with functionality that lives in Code.py. >> > > Yes, I know. That was placed there to give it a try (actually I'm not > sure what is the best place for it). ClosureTempAllocator now is > attribute of defnode and all yieldnodes, that's not clean. > > ExprNode.StopIterationNode isn't expression should it be moved to Nodes.py? > > create_abstract_generator() is rather dirty > > I don't know how to bind C functions to CClass. > > I think that generator body could have PyObject *body(PyObject *self, > PyObject *value) signature > With value=Null meaning exception. > >> Could you give me ("scoder", or better all Cython devs) commit rights to >> your repo so that I can start refactoring the code and fix problems I find? >> > > I've added you, but I don't know how to add all devs together. I don't think there's a way to do that, just add whoever you want help participating on this branch. My username is robertwb, and though I haven't rally had time to read any code yet, I'm very interested and would like to help out if I can. >> I could also add Hudson jobs for it. Robert, would it be ok to add them for >> all Python versions? I think that would be excessive. Lets just do 2.4, 2.7, and a 3.x. If there's a strange corner case that only manifests in, say, 2.5 then we'll deal with it later. > Yeah, please that would be nice. > >>>> ? ?- PyxGenerator methods are defined via declare_var >>> >>> Should we call that "CyGenerator" ? :) >> >> I actually meant to call it "CyGenerator" at the Python level, but I now >> see that it's actually a purely internal thing. That makes __Pyx_Generator >> a better choice, given that "__pyx_" is the globally reserved prefix in >> Cython code. >> > > Should rename it back ;) > >> Eventually, we may(!) want to reuse the superclass across modules, but I >> currently have no idea how that could be made to work. So it's just another >> thing that we don't need to worry about now. >> > > Okay. > > How can I declare function entry with METH_VARGS signature? > Using TypeSlots I can add METH_VARGS|METH_KWARGS entries but not > single METH_VARGS > > That's preferd for "throw" method. Just let it be METH_VARGS|METH_KWARGS for now. That doesn't mean it can actually take arbitrary keyword arguments, just that the arguments that are needed can be passed in as keywords. - Robert From vitja.makarov at gmail.com Fri Dec 10 20:15:00 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 10 Dec 2010 22:15:00 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> Message-ID: 2010/12/10 Robert Bradshaw : > On Fri, Dec 10, 2010 at 12:22 AM, Vitja Makarov wrote: >> 2010/12/10 Stefan Behnel : >>> Stefan Behnel, 09.12.2010 10:46: >>>> Vitja Makarov, 08.12.2010 22:45: >>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>> But you can take a look at patch and generated code. >>>> >>>> I'll take a close look ASAP. >>>> >>>>> ? ?- Temps are saved/restored/allocated inside YieldExprNode using >>>>> helper ClosureTempAllocator >>> >>> Ok, that currently lives in ParseTreeTransforms, which is totally the wrong >>> place for anything that generates code. It also mixes functionality that >>> normally lives in Symtab.py with functionality that lives in Code.py. >>> >> >> Yes, I know. That was placed there to give it a try (actually I'm not >> sure what is the best place for it). ClosureTempAllocator now is >> attribute of defnode and all yieldnodes, that's not clean. >> >> ExprNode.StopIterationNode isn't expression should it be moved to Nodes.py? >> >> create_abstract_generator() is rather dirty >> >> I don't know how to bind C functions to CClass. >> >> I think that generator body could have PyObject *body(PyObject *self, >> PyObject *value) signature >> With value=Null meaning exception. >> >>> Could you give me ("scoder", or better all Cython devs) commit rights to >>> your repo so that I can start refactoring the code and fix problems I find? >>> >> >> I've added you, but I don't know how to add all devs together. > > I don't think there's a way to do that, just add whoever you want help > participating on this branch. My username is robertwb, and though I > haven't rally had time to read any code yet, I'm very interested and > would like to help out if I can. > I've added you too. >>> I could also add Hudson jobs for it. Robert, would it be ok to add them for >>> all Python versions? > > I think that would be excessive. Lets just do 2.4, 2.7, and a 3.x. If > there's a strange corner case that only manifests in, say, 2.5 then > we'll deal with it later. > Can you please add pyregr for 2.7 also? >> Yeah, please that would be nice. >> >>>>> ? ?- PyxGenerator methods are defined via declare_var >>>> >>>> Should we call that "CyGenerator" ? :) >>> >>> I actually meant to call it "CyGenerator" at the Python level, but I now >>> see that it's actually a purely internal thing. That makes __Pyx_Generator >>> a better choice, given that "__pyx_" is the globally reserved prefix in >>> Cython code. >>> >> >> Should rename it back ;) >> >>> Eventually, we may(!) want to reuse the superclass across modules, but I >>> currently have no idea how that could be made to work. So it's just another >>> thing that we don't need to worry about now. >>> >> >> Okay. >> >> How can I declare function entry with METH_VARGS signature? >> Using TypeSlots I can add METH_VARGS|METH_KWARGS entries but not >> single METH_VARGS >> >> That's preferd for "throw" method. > > Just let it be METH_VARGS|METH_KWARGS for now. That doesn't mean it > can actually take arbitrary keyword arguments, just that the arguments > that are needed can be passed in as keywords. > Ok -- vitja. From robertwb at math.washington.edu Fri Dec 10 20:35:58 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 10 Dec 2010 11:35:58 -0800 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> Message-ID: On Fri, Dec 10, 2010 at 11:15 AM, Vitja Makarov wrote: > 2010/12/10 Robert Bradshaw : >> On Fri, Dec 10, 2010 at 12:22 AM, Vitja Makarov wrote: >>> 2010/12/10 Stefan Behnel : >>>> Stefan Behnel, 09.12.2010 10:46: >>>>> Vitja Makarov, 08.12.2010 22:45: >>>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>>> But you can take a look at patch and generated code. >>>>> >>>>> I'll take a close look ASAP. >>>>> >>>>>> ? ?- Temps are saved/restored/allocated inside YieldExprNode using >>>>>> helper ClosureTempAllocator >>>> >>>> Ok, that currently lives in ParseTreeTransforms, which is totally the wrong >>>> place for anything that generates code. It also mixes functionality that >>>> normally lives in Symtab.py with functionality that lives in Code.py. >>>> >>> >>> Yes, I know. That was placed there to give it a try (actually I'm not >>> sure what is the best place for it). ClosureTempAllocator now is >>> attribute of defnode and all yieldnodes, that's not clean. >>> >>> ExprNode.StopIterationNode isn't expression should it be moved to Nodes.py? >>> >>> create_abstract_generator() is rather dirty >>> >>> I don't know how to bind C functions to CClass. >>> >>> I think that generator body could have PyObject *body(PyObject *self, >>> PyObject *value) signature >>> With value=Null meaning exception. >>> >>>> Could you give me ("scoder", or better all Cython devs) commit rights to >>>> your repo so that I can start refactoring the code and fix problems I find? >>>> >>> >>> I've added you, but I don't know how to add all devs together. >> >> I don't think there's a way to do that, just add whoever you want help >> participating on this branch. My username is robertwb, and though I >> haven't rally had time to read any code yet, I'm very interested and >> would like to help out if I can. >> > > I've added you too. Thanks. >>>> I could also add Hudson jobs for it. Robert, would it be ok to add them for >>>> all Python versions? >> >> I think that would be excessive. Lets just do 2.4, (I meant 2.3 here...) >> 2.7, and a 3.x. If >> there's a strange corner case that only manifests in, say, 2.5 then >> we'll deal with it later. >> > > Can you please add pyregr for 2.7 also? Sure, that makes sense as well. - Robert From stefan_ml at behnel.de Fri Dec 10 20:51:39 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Dec 2010 20:51:39 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> Message-ID: <4D0284CB.9060905@behnel.de> Robert Bradshaw, 10.12.2010 20:35: > On Fri, Dec 10, 2010 at 11:15 AM, Vitja Makarov wrote: >> 2010/12/10 Robert Bradshaw: >>> On Fri, Dec 10, 2010 at 12:22 AM, Vitja Makarov wrote: >>>> 2010/12/10 Stefan Behnel: >>>>> I could also add Hudson jobs for it. Robert, would it be ok to add them for >>>>> all Python versions? >>> >>> I think that would be excessive. Lets just do 2.4, > > (I meant 2.3 here...) The 2.3 tests do not run the doctests. I have no idea why (likely a bug or missing feature in the old doctest or unittest module), and I didn't care much since 2.3 is long dead anyway. In any case, 2.4 is a better choice. >>> 2.7, and a 3.x. If >>> there's a strange corner case that only manifests in, say, 2.5 then >>> we'll deal with it later. >>> >> >> Can you please add pyregr for 2.7 also? > > Sure, that makes sense as well. Done. Stefan From robertwb at math.washington.edu Fri Dec 10 20:53:18 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 10 Dec 2010 11:53:18 -0800 Subject: [Cython] General generator expressions In-Reply-To: <4D0284CB.9060905@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> Message-ID: On Fri, Dec 10, 2010 at 11:51 AM, Stefan Behnel wrote: > Robert Bradshaw, 10.12.2010 20:35: >> On Fri, Dec 10, 2010 at 11:15 AM, Vitja Makarov wrote: >>> 2010/12/10 Robert Bradshaw: >>>> On Fri, Dec 10, 2010 at 12:22 AM, Vitja Makarov wrote: >>>>> 2010/12/10 Stefan Behnel: >>>>>> I could also add Hudson jobs for it. Robert, would it be ok to add them for >>>>>> all Python versions? >>>> >>>> I think that would be excessive. Lets just do 2.4, >> >> (I meant 2.3 here...) > > The 2.3 tests do not run the doctests. I have no idea why (likely a bug or > missing feature in the old doctest or unittest module), and I didn't care > much since 2.3 is long dead anyway. In any case, 2.4 is a better choice. Ah, OK, I thought it was an oversight and just went through trying to change it (which you probably noticed). Sure, 2.4 is fine. - Robert From vitja.makarov at gmail.com Fri Dec 10 22:01:05 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 00:01:05 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> Message-ID: 2010/12/10 Robert Bradshaw : > On Fri, Dec 10, 2010 at 11:51 AM, Stefan Behnel wrote: >> Robert Bradshaw, 10.12.2010 20:35: >>> On Fri, Dec 10, 2010 at 11:15 AM, Vitja Makarov wrote: >>>> 2010/12/10 Robert Bradshaw: >>>>> On Fri, Dec 10, 2010 at 12:22 AM, Vitja Makarov wrote: >>>>>> 2010/12/10 Stefan Behnel: >>>>>>> I could also add Hudson jobs for it. Robert, would it be ok to add them for >>>>>>> all Python versions? >>>>> >>>>> I think that would be excessive. Lets just do 2.4, >>> >>> (I meant 2.3 here...) >> >> The 2.3 tests do not run the doctests. I have no idea why (likely a bug or >> missing feature in the old doctest or unittest module), and I didn't care >> much since 2.3 is long dead anyway. In any case, 2.4 is a better choice. > > Ah, OK, I thought it was an oversight and just went through trying to > change it (which you probably noticed). Sure, 2.4 is fine. > > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > I've pushed commit that moves exc_save_vars into temps (now it's declared with type=py_object_type and manage_refs=False) Is that ok? That's required for propper generator support to store exc_save_vars somewhere. -- vitja. From stefan_ml at behnel.de Fri Dec 10 23:38:45 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Dec 2010 23:38:45 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> Message-ID: <4D02ABF5.4010504@behnel.de> Vitja Makarov, 10.12.2010 22:01: > I've pushed commit that moves exc_save_vars into temps Yes, I think that makes good sense and should simplify things a lot. > (now it's declared with type=py_object_type and manage_refs=False) Sounds right. Stefan From vitja.makarov at gmail.com Fri Dec 10 23:42:00 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 01:42:00 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D02ABF5.4010504@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> <4D02ABF5.4010504@behnel.de> Message-ID: 2010/12/11 Stefan Behnel : > Vitja Makarov, 10.12.2010 22:01: >> I've pushed commit that moves exc_save_vars into temps > > Yes, I think that makes good sense and should simplify things a lot. > Here is commit for it https://github.com/vitek/cython/commit/403fe34aa5ca0a9e512755b7d403cd9965a8d830 > >> (now it's declared with type=py_object_type and manage_refs=False) > > Sounds right. > > Stefan Pyregr tests works now on hudson but there is still problem with heapq don't understand is that related to generators or not -- vitja. From robertwb at math.washington.edu Fri Dec 10 23:41:54 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 10 Dec 2010 14:41:54 -0800 Subject: [Cython] General generator expressions In-Reply-To: <4D02ABF5.4010504@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> <4D02ABF5.4010504@behnel.de> Message-ID: On Fri, Dec 10, 2010 at 2:38 PM, Stefan Behnel wrote: > Vitja Makarov, 10.12.2010 22:01: >> I've pushed commit that moves exc_save_vars into temps > > Yes, I think that makes good sense and should simplify things a lot. > >> (now it's declared with type=py_object_type and manage_refs=False) > > Sounds right. Can't their be multiple nestings of exc_save_vars (if try statements are nested)? - Robert From vitja.makarov at gmail.com Fri Dec 10 23:43:38 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 01:43:38 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> <4D02ABF5.4010504@behnel.de> Message-ID: 2010/12/11 Robert Bradshaw : > On Fri, Dec 10, 2010 at 2:38 PM, Stefan Behnel wrote: >> Vitja Makarov, 10.12.2010 22:01: >>> I've pushed commit that moves exc_save_vars into temps >> >> Yes, I think that makes good sense and should simplify things a lot. >> >>> (now it's declared with type=py_object_type and manage_refs=False) >> >> Sounds right. > > Can't their be multiple nestings of exc_save_vars (if try statements > are nested)? > > - Robert That should be ok, temps are allocated dynamicly. Take a look at commit please. -- vitja. From robertwb at math.washington.edu Fri Dec 10 23:52:18 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 10 Dec 2010 14:52:18 -0800 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> <4D02ABF5.4010504@behnel.de> Message-ID: On Fri, Dec 10, 2010 at 2:43 PM, Vitja Makarov wrote: > 2010/12/11 Robert Bradshaw : >> On Fri, Dec 10, 2010 at 2:38 PM, Stefan Behnel wrote: >>> Vitja Makarov, 10.12.2010 22:01: >>>> I've pushed commit that moves exc_save_vars into temps >>> >>> Yes, I think that makes good sense and should simplify things a lot. >>> >>>> (now it's declared with type=py_object_type and manage_refs=False) >>> >>> Sounds right. >> >> Can't their be multiple nestings of exc_save_vars (if try statements >> are nested)? >> >> - Robert > > That should be ok, temps are allocated dynamicly. Take a look at commit please. +1, that should be fine. (I'd actually been meaning to take a good look at your code when I had some time and then respond with any comments, including this concern were it was relevant, but just wanted to confirm this was considered when I saw someone else answered.) - Robert From stefan_ml at behnel.de Sat Dec 11 00:43:14 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Dec 2010 00:43:14 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> Message-ID: <4D02BB12.7060306@behnel.de> Vitja Makarov, 10.12.2010 09:22: > 2010/12/10 Stefan Behnel: >> Stefan Behnel, 09.12.2010 10:46: >>> Vitja Makarov, 08.12.2010 22:45: >>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>> But you can take a look at patch and generated code. >>> >>> I'll take a close look ASAP. >>> >>>> - Temps are saved/restored/allocated inside YieldExprNode using >>>> helper ClosureTempAllocator >> >> Ok, that currently lives in ParseTreeTransforms, which is totally the wrong >> place for anything that generates code. It also mixes functionality that >> normally lives in Symtab.py with functionality that lives in Code.py. >> > > Yes, I know. That was placed there to give it a try (actually I'm not > sure what is the best place for it). ClosureTempAllocator now is > attribute of defnode and all yieldnodes, that's not clean. FunctionState in Code.py seems like a good place to hold a reference, at least for the code generation part of it. > ExprNode.StopIterationNode isn't expression should it be moved to Nodes.py? Why not drop it completely and generate the code in the function node instead? ISTM that the only reason it's there is to drop one line of code at the end of the function body. That can be done much easier. > create_abstract_generator() is rather dirty I've seen worse. ;) > I don't know how to bind C functions to CClass. Such as? > I think that generator body could have PyObject *body(PyObject *self, > PyObject *value) signature > With value=Null meaning exception. See the CEP. ;) >>>> - PyxGenerator methods are defined via declare_var >>> >>> Should we call that "CyGenerator" ? :) >> >> I actually meant to call it "CyGenerator" at the Python level, but I now >> see that it's actually a purely internal thing. That makes __Pyx_Generator >> a better choice, given that "__pyx_" is the globally reserved prefix in >> Cython code. > > Should rename it back ;) Yep. > How can I declare function entry with METH_VARGS signature? > Using TypeSlots I can add METH_VARGS|METH_KWARGS entries but not > single METH_VARGS > > That's preferd for "throw" method. I see what you mean. It's a minor issue, though. Just ignore it for now, or raise an exception if kwargs are passed in. After all, .throw() is rarely used and not performance critical at all. Stefan From stefan_ml at behnel.de Sat Dec 11 00:46:08 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Dec 2010 00:46:08 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> <4D02ABF5.4010504@behnel.de> Message-ID: <4D02BBC0.9090800@behnel.de> Vitja Makarov, 10.12.2010 23:42: > Pyregr tests works now on hudson but there is still problem with heapq > don't understand is that related to generators or not Not sure either. It works for me in Py3 but not in Py2.7. Stefan From stefan_ml at behnel.de Sat Dec 11 00:59:19 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Dec 2010 00:59:19 +0100 Subject: [Cython] General generator expressions In-Reply-To: <4D02BBC0.9090800@behnel.de> References: <4CFBD711.5060206@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D0284CB.9060905@behnel.de> <4D02ABF5.4010504@behnel.de> <4D02BBC0.9090800@behnel.de> Message-ID: <4D02BED7.2090802@behnel.de> Stefan Behnel, 11.12.2010 00:46: > Vitja Makarov, 10.12.2010 23:42: >> Pyregr tests works now on hudson but there is still problem with heapq >> don't understand is that related to generators or not > > Not sure either. It works for me in Py3 but not in Py2.7. It seems to be generator related and it looks like there's a bug in the exception handling. It crashes when checking for exceptions after running through the heapiter() generator method. These things often point at a dangling reference in Python's exception state. Stefan From stefan_ml at behnel.de Sat Dec 11 03:00:34 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Dec 2010 03:00:34 +0100 Subject: [Cython] preliminary generator benchmark Message-ID: <4D02DB42.6000108@behnel.de> Hi, here's a little generator benchmark: ====================== def cgen(a,b): while 1: yield a+b d = {} exec ''' def pygen(a,b): while 1: yield a+b ''' in d pygen = d['pygen'] def repeat_gen(gen, int count): cdef int i for i in range(count): next(gen) try: gen.throw(GeneratorExit) except GeneratorExit: pass ====================== timeit gives me this in Py3.2: $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' 'repeat_gen(pygen(1,2), 10000)' 1000 loops, best of 3: 827 usec per loop $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' 'repeat_gen(cgen(1,2), 10000)' 1000 loops, best of 3: 271 usec per loop And here's another one where we add Cython's looping advantage: ====================== def cgen(a,b): cdef int i for i in range(a): yield b d = {} exec ''' def pygen(a,b): for i in range(a): yield b ''' in d pygen = d['pygen'] def repeat_gen(gen): for _ in gen: pass ====================== $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' 'repeat_gen(pygen(10000, 2))' 1000 loops, best of 3: 881 usec per loop $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' 'repeat_gen(cgen(10000, 2))' 10000 loops, best of 3: 116 usec per loop That's not too bad for a start. Stefan From robertwb at math.washington.edu Sat Dec 11 03:41:53 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 10 Dec 2010 18:41:53 -0800 Subject: [Cython] Release candidate/feature freeze Message-ID: I'm planning on posting a release candidate tonight, at which point I would no longer be pulling from the devel branch. If you know of any features/bugfixes that should go in, let me know. If all goes well, we'll release early next week. We also need to finish fleshing out the release notes and documentation (myself included). - Robert From vitja.makarov at gmail.com Sat Dec 11 07:03:53 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 09:03:53 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D02BB12.7060306@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> Message-ID: 2010/12/11 Stefan Behnel : > Vitja Makarov, 10.12.2010 09:22: >> 2010/12/10 Stefan Behnel: >>> Stefan Behnel, 09.12.2010 10:46: >>>> Vitja Makarov, 08.12.2010 22:45: >>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>> But you can take a look at patch and generated code. >>>> >>>> I'll take a close look ASAP. >>>> >>>>> ? ? - Temps are saved/restored/allocated inside YieldExprNode using >>>>> helper ClosureTempAllocator >>> >>> Ok, that currently lives in ParseTreeTransforms, which is totally the wrong >>> place for anything that generates code. It also mixes functionality that >>> normally lives in Symtab.py with functionality that lives in Code.py. >>> >> >> Yes, I know. That was placed there to give it a try (actually I'm not >> sure what is the best place for it). ClosureTempAllocator now is >> attribute of defnode and all yieldnodes, that's not clean. > > FunctionState in Code.py seems like a good place to hold a reference, at > least for the code generation part of it. > At some point we can call code.set_closure_class() that would create code.temp_allocator and then refer to it as code.temp_allocator. temp_allocator isn't good name here maybe closure_temps? > >> ExprNode.StopIterationNode isn't expression should it be moved to Nodes.py? > > Why not drop it completely and generate the code in the function node > instead? ISTM that the only reason it's there is to drop one line of code > at the end of the function body. That can be done much easier. > > Sure. You've already done this, thanks! >> create_abstract_generator() is rather dirty > > I've seen worse. ;) > > >> I don't know how to bind C functions to CClass. > > Such as? > > e = declare() e.func_cname = ... e.signature = ... I think that there should be a short way >> I think that generator body could have PyObject *body(PyObject *self, >> PyObject *value) signature >> With value=Null meaning exception. > > See the CEP. ;) > Yeah! Forget about that, that's already done. I was looking into python implementation and it has `i_exc` arg. > >>>>> ? ? - PyxGenerator methods are defined via declare_var >>>> >>>> Should we call that "CyGenerator" ? :) >>> >>> I actually meant to call it "CyGenerator" at the Python level, but I now >>> see that it's actually a purely internal thing. That makes __Pyx_Generator >>> a better choice, given that "__pyx_" is the globally reserved prefix in >>> Cython code. >> >> Should rename it back ;) > > Yep. > __pyx_CyGenerator_(Send|Close)() <-- or lover case suffix is better, should look at pep7? class __pyx_CyGenerator type __pyx_CyGeneratorType is that ok? > >> How can I declare function entry with METH_VARGS signature? >> Using TypeSlots I can add METH_VARGS|METH_KWARGS entries but not >> single METH_VARGS >> >> That's preferd for "throw" method. > > I see what you mean. It's a minor issue, though. Just ignore it for now, or > raise an exception if kwargs are passed in. After all, .throw() is rarely > used and not performance critical at all. > ok >> Pyregr tests works now on hudson but there is still problem with heapq >> don't understand is that related to generators or not > > Not sure either. It works for me in Py3 but not in Py2.7. I see you've fixed this issue. Cool. > disable generators test in Py2.4 - depends on GeneratorExit and potentially other stuff May be it's better to generate another exception (StopIteration) in 2.4 and control this with #ifdefs ... #endif What do you think about generate_preamble? Think it's not correct name. In GeneratorWrapper() buffer cleanup code isn't implemented when closure is created. What is the place for YieldCollector? About genexp: now it should be easy to implement just like lambda function -- vitja. From vitja.makarov at gmail.com Sat Dec 11 09:59:41 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 11:59:41 +0300 Subject: [Cython] pyregr test runner Message-ID: I would like to fix test runner for pyregr. Original python regrtest.py import module and run its test_main() if any. It's not a problem to do this in our test runner but I don't know yet how to get result back to . I think test_support.run_unittest and run_doctestcan should be patched. -- vitja. From robertwb at math.washington.edu Sat Dec 11 12:01:11 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 11 Dec 2010 03:01:11 -0800 Subject: [Cython] Cython 0.14 release candidate Message-ID: Release candidate up at http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something bad is discovered, this will be the release sometime early next week. - Robert From vitja.makarov at gmail.com Sat Dec 11 14:42:18 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 16:42:18 +0300 Subject: [Cython] pyregr test runner In-Reply-To: References: Message-ID: 2010/12/11 Vitja Makarov : > I would like to fix test runner for pyregr. > > Original python regrtest.py import module and run its test_main() if any. > It's not a problem to do this in our test runner but I don't know yet > how to get result back to . > I think test_support.run_unittest and run_doctestcan should be patched. > > -- > vitja. > I've pushed support for this stuff here: https://github.com/vitek/cython/commit/1bcf08a87901265ad7dfe16e952c872c20869cea Here are results: https://sage.math.washington.edu:8091/hudson/view/cython-generators/job/cython-vitek-tests-pyregr-py27-c/ 5675/283 was 3865/636 I think that doctests for pyregr should be disabled... they mostly test python itself not cython -- vitja. From stefan_ml at behnel.de Sat Dec 11 15:01:34 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Dec 2010 15:01:34 +0100 Subject: [Cython] pyregr test runner In-Reply-To: References: Message-ID: <4D03843E.40002@behnel.de> Vitja Makarov, 11.12.2010 14:42: > 2010/12/11 Vitja Makarov: >> I would like to fix test runner for pyregr. >> >> Original python regrtest.py import module and run its test_main() if any. >> It's not a problem to do this in our test runner but I don't know yet >> how to get result back to . >> I think test_support.run_unittest and run_doctestcan should be patched. > > I've pushed support for this stuff here: > > https://github.com/vitek/cython/commit/1bcf08a87901265ad7dfe16e952c872c20869cea > > Here are results: > https://sage.math.washington.edu:8091/hudson/view/cython-generators/job/cython-vitek-tests-pyregr-py27-c/ > > 5675/283 was 3865/636 That's cool. Patch looks fine to me. > I think that doctests for pyregr should be disabled... they mostly > test python itself not cython *All* pyregr tests test mostly Python. I don't see why we should disable all doctests, some of them may still test compiled stuff. Stefan From vitja.makarov at gmail.com Sat Dec 11 15:04:36 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 17:04:36 +0300 Subject: [Cython] pyregr test runner In-Reply-To: <4D03843E.40002@behnel.de> References: <4D03843E.40002@behnel.de> Message-ID: 2010/12/11 Stefan Behnel : > Vitja Makarov, 11.12.2010 14:42: >> 2010/12/11 Vitja Makarov: >>> I would like to fix test runner for pyregr. >>> >>> Original python regrtest.py import module and run its test_main() if any. >>> It's not a problem to do this in our test runner but I don't know yet >>> how to get result back to . >>> I think test_support.run_unittest and run_doctestcan should be patched. >> >> I've pushed support for this stuff here: >> >> https://github.com/vitek/cython/commit/1bcf08a87901265ad7dfe16e952c872c20869cea >> >> Here are results: >> https://sage.math.washington.edu:8091/hudson/view/cython-generators/job/cython-vitek-tests-pyregr-py27-c/ >> >> 5675/283 was 3865/636 > > That's cool. Patch looks fine to me. > > >> I think that doctests for pyregr should be disabled... they mostly >> test python itself not cython > > *All* pyregr tests test mostly Python. I don't see why we should disable > all doctests, some of them may still test compiled stuff. > Ok. I'm now going to handle SkipTest and ResourceDenied exceptions -- vitja. From dagss at student.matnat.uio.no Sat Dec 11 15:16:46 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 11 Dec 2010 15:16:46 +0100 Subject: [Cython] preliminary generator benchmark In-Reply-To: <4D02DB42.6000108@behnel.de> References: <4D02DB42.6000108@behnel.de> Message-ID: <4D0387CE.5010405@student.matnat.uio.no> On 12/11/2010 03:00 AM, Stefan Behnel wrote: > Hi, > > here's a little generator benchmark: > > ====================== > def cgen(a,b): > while 1: > yield a+b > > d = {} > exec ''' > def pygen(a,b): > while 1: > yield a+b > ''' in d > > pygen = d['pygen'] > > > def repeat_gen(gen, int count): > cdef int i > for i in range(count): > next(gen) > try: > gen.throw(GeneratorExit) > except GeneratorExit: > pass > ====================== > > timeit gives me this in Py3.2: > > $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' > 'repeat_gen(pygen(1,2), 10000)' > 1000 loops, best of 3: 827 usec per loop > > $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' > 'repeat_gen(cgen(1,2), 10000)' > 1000 loops, best of 3: 271 usec per loop > > > And here's another one where we add Cython's looping advantage: > > ====================== > def cgen(a,b): > cdef int i > for i in range(a): > yield b > > d = {} > exec ''' > def pygen(a,b): > for i in range(a): > yield b > ''' in d > > pygen = d['pygen'] > > > def repeat_gen(gen): > for _ in gen: > pass > ====================== > > $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' > 'repeat_gen(pygen(10000, 2))' > 1000 loops, best of 3: 881 usec per loop > $ python3 -m timeit -s 'from genbench import repeat_gen, cgen, pygen' > 'repeat_gen(cgen(10000, 2))' > 10000 loops, best of 3: 116 usec per loop > > > That's not too bad for a start. > This is all very cool! When it comes to performance, I want to point out that there seem to be quite a lot of libraries (of the "works only on some platforms" type) implementing real coroutines in C. So for really pushing generator performance, perhaps we could use one of those. Keeping multiple C stacks around is probably always going to be faster than unpacking/repacking variables. As long as a "fast enough" implementation is available on all platforms, perhaps it will suffice to only support the most common platforms "lightning fast". Dag Sverre From vitja.makarov at gmail.com Sat Dec 11 19:19:35 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 11 Dec 2010 21:19:35 +0300 Subject: [Cython] pyregr test runner In-Reply-To: References: <4D03843E.40002@behnel.de> Message-ID: 2010/12/11 Vitja Makarov : > 2010/12/11 Stefan Behnel : >> Vitja Makarov, 11.12.2010 14:42: >>> 2010/12/11 Vitja Makarov: >>>> I would like to fix test runner for pyregr. >>>> >>>> Original python regrtest.py import module and run its test_main() if any. >>>> It's not a problem to do this in our test runner but I don't know yet >>>> how to get result back to . >>>> I think test_support.run_unittest and run_doctestcan should be patched. >>> >>> I've pushed support for this stuff here: >>> >>> https://github.com/vitek/cython/commit/1bcf08a87901265ad7dfe16e952c872c20869cea >>> >>> Here are results: >>> https://sage.math.washington.edu:8091/hudson/view/cython-generators/job/cython-vitek-tests-pyregr-py27-c/ >>> >>> 5675/283 was 3865/636 >> >> That's cool. Patch looks fine to me. >> >> >>> I think that doctests for pyregr should be disabled... they mostly >>> test python itself not cython >> >> *All* pyregr tests test mostly Python. I don't see why we should disable >> all doctests, some of them may still test compiled stuff. >> > > Ok. I'm now going to handle SkipTest and ResourceDenied exceptions > Added. That was hard to make it work with both py2 and py3... -- vitja. From arfrever.fta at gmail.com Sat Dec 11 22:44:41 2010 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sat, 11 Dec 2010 22:44:41 +0100 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: Message-ID: <201012112245.14866.Arfrever.FTA@gmail.com> 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): > Release candidate up at > http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something > bad is discovered, this will be the release sometime early next week. Some tests fail. Test failures with Python 2.6 (the same tests fail with Python 2.7): ====================================================================== FAIL: Doctest: numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for numpy_test File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 25, in numpy_test ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 114, in numpy_test Failed example: test_c_contig(f_arr) Expected: Traceback (most recent call last): ... ValueError: ndarray is not C contiguous Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_c_contig(f_arr) File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.c:2619) ValueError: ndarray is not C-contiguous ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 122, in numpy_test Failed example: test_c_contig(c_arr[::2,::2]) Expected: Traceback (most recent call last): ... ValueError: ndarray is not C contiguous Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_c_contig(c_arr[::2,::2]) File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.c:2619) ValueError: ndarray is not C-contiguous ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 161, in numpy_test Failed example: test_dtype('%si' % other_endian, inc1_int) Expected: Traceback (most recent call last): ... ValueError: Non-native byte order not supported Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_dtype('%si' % other_endian, inc1_int) File "numpy_test.pyx", line 339, in numpy_test.test_dtype (numpy_test.c:5311) File "numpy_test.pyx", line 280, in numpy_test.inc1_int (numpy_test.c:3129) ValueError: Big-endian buffer not supported on little-endian compiler ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 168, in numpy_test Failed example: test_recordarray() Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_recordarray() File "numpy_test.pyx", line 347, in numpy_test.test_recordarray (numpy_test.c:5474) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 170, in numpy_test Failed example: print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,i')), ('b', np.dtype('i,i')) ])))) Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,i')), ('b', np.dtype('i,i')) ])))) File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.c:5792) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 174, in numpy_test Failed example: print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,f')), ('b', np.dtype('i,i')) ])))) Expected: Traceback (most recent call last): ... ValueError: Buffer dtype mismatch, expected 'int' but got 'float' in 'DoubleInt.y' Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,f')), ('b', np.dtype('i,i')) ])))) File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.c:5792) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 179, in numpy_test Failed example: print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.c:6296) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 182, in numpy_test Failed example: print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.c:6427) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 186, in numpy_test Failed example: print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) Expected: Traceback (most recent call last): ... ValueError: Buffer dtype mismatch; next field is at offset 4 but 1 expected Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.c:6296) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 191, in numpy_test Failed example: print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) Expected: Traceback (most recent call last): ... ValueError: Buffer dtype mismatch; next field is at offset 1 but 4 expected Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.c:6427) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 209, in numpy_test Failed example: test_point_record() Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_point_record() File "numpy_test.pyx", line 429, in numpy_test.test_point_record (numpy_test.c:6871) ValueError: Does not understand character buffer dtype format string (':') ====================================================================== FAIL: Doctest: numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for numpy_test File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 11, in numpy_test ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 100, in numpy_test Failed example: test_c_contig(f_arr) Expected: Traceback (most recent call last): ... ValueError: ndarray is not C contiguous Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_c_contig(f_arr) File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.cpp:2619) ValueError: ndarray is not C-contiguous ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 108, in numpy_test Failed example: test_c_contig(c_arr[::2,::2]) Expected: Traceback (most recent call last): ... ValueError: ndarray is not C contiguous Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_c_contig(c_arr[::2,::2]) File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.cpp:2619) ValueError: ndarray is not C-contiguous ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 147, in numpy_test Failed example: test_dtype('%si' % other_endian, inc1_int) Expected: Traceback (most recent call last): ... ValueError: Non-native byte order not supported Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_dtype('%si' % other_endian, inc1_int) File "numpy_test.pyx", line 339, in numpy_test.test_dtype (numpy_test.cpp:5311) File "numpy_test.pyx", line 280, in numpy_test.inc1_int (numpy_test.cpp:3129) ValueError: Big-endian buffer not supported on little-endian compiler ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 154, in numpy_test Failed example: test_recordarray() Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_recordarray() File "numpy_test.pyx", line 347, in numpy_test.test_recordarray (numpy_test.cpp:5474) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 156, in numpy_test Failed example: print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,i')), ('b', np.dtype('i,i')) ])))) Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,i')), ('b', np.dtype('i,i')) ])))) File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.cpp:5792) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 160, in numpy_test Failed example: print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,f')), ('b', np.dtype('i,i')) ])))) Expected: Traceback (most recent call last): ... ValueError: Buffer dtype mismatch, expected 'int' but got 'float' in 'DoubleInt.y' Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ('a', np.dtype('i,f')), ('b', np.dtype('i,i')) ])))) File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.cpp:5792) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 165, in numpy_test Failed example: print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.cpp:6296) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 168, in numpy_test Failed example: print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.cpp:6427) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 172, in numpy_test Failed example: print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) Expected: Traceback (most recent call last): ... ValueError: Buffer dtype mismatch; next field is at offset 4 but 1 expected Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.cpp:6296) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 177, in numpy_test Failed example: print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) Expected: Traceback (most recent call last): ... ValueError: Buffer dtype mismatch; next field is at offset 1 but 4 expected Got: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.cpp:6427) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 195, in numpy_test Failed example: test_point_record() Exception raised: Traceback (most recent call last): File "/usr/lib64/python2.6/doctest.py", line 1253, in __run compileflags, 1) in test.globs File "", line 1, in test_point_record() File "numpy_test.pyx", line 429, in numpy_test.test_point_record (numpy_test.cpp:6871) ValueError: Does not understand character buffer dtype format string (':') ---------------------------------------------------------------------- Ran 4812 tests in 1194.085s FAILED (failures=2) Test failures with Python 3.1 (the same tests fail with Python 3.2): ====================================================================== ERROR: test_globals (Cython.Build.Tests.TestInline.TestStripLiterals) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 32, in test_globals self.assertEquals(inline("return global_value + 1", **test_kwds), global_value + 1) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols code = code.decode('ascii') AttributeError: 'str' object has no attribute 'decode' ====================================================================== ERROR: test_locals (Cython.Build.Tests.TestInline.TestStripLiterals) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 29, in test_locals self.assertEquals(inline("return a+b", **test_kwds), 3) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols code = code.decode('ascii') AttributeError: 'str' object has no attribute 'decode' ====================================================================== ERROR: test_numpy (Cython.Build.Tests.TestInline.TestStripLiterals) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 41, in test_numpy self.assertEquals(inline("return a[0,0]", a=a, **test_kwds), 10.0) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols code = code.decode('ascii') AttributeError: 'str' object has no attribute 'decode' ====================================================================== ERROR: test_simple (Cython.Build.Tests.TestInline.TestStripLiterals) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 18, in test_simple self.assertEquals(inline("return 1+2", **test_kwds), 3) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols code = code.decode('ascii') AttributeError: 'str' object has no attribute 'decode' ====================================================================== ERROR: test_types (Cython.Build.Tests.TestInline.TestStripLiterals) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 24, in test_types """, a=1.0, b=[], **test_kwds), ('double', 'list object')) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols code = code.decode('ascii') AttributeError: 'str' object has no attribute 'decode' ====================================================================== ERROR: compiling (c) and running numpy_bufacc_T155 ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != "10:9: 'ndarray' is not a type identifier" ====================================================================== ERROR: compiling (cpp) and running numpy_bufacc_T155 ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != "10:9: 'ndarray' is not a type identifier" ====================================================================== ERROR: compiling (c) and running numpy_cimport ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ====================================================================== ERROR: compiling (cpp) and running numpy_cimport ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ====================================================================== ERROR: compiling (c) and running numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ====================================================================== ERROR: compiling (cpp) and running numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ---------------------------------------------------------------------- Ran 4773 tests in 1128.064s FAILED (errors=11) === Expected errors: === === Got errors: === 10:9: 'ndarray' is not a type identifier 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 10:9: 'ndarray' is not a type identifier 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option 253:18: 'ndarray' is not a type identifier 258:22: 'ndarray' is not a type identifier 265:18: 'ndarray' is not a type identifier 270:18: 'ndarray' is not a type identifier 276:14: 'ndarray' is not a type identifier 277:15: 'ndarray' is not a type identifier 278:15: 'ndarray' is not a type identifier 279:16: 'ndarray' is not a type identifier 280:13: 'ndarray' is not a type identifier 281:14: 'ndarray' is not a type identifier 282:14: 'ndarray' is not a type identifier 283:15: 'ndarray' is not a type identifier 284:18: 'ndarray' is not a type identifier 285:19: 'ndarray' is not a type identifier 287:15: 'ndarray' is not a type identifier 288:16: 'ndarray' is not a type identifier 289:20: 'ndarray' is not a type identifier 291:16: 'ndarray' is not a type identifier 292:17: 'ndarray' is not a type identifier 293:21: 'ndarray' is not a type identifier 295:23: 'ndarray' is not a type identifier 299:24: 'ndarray' is not a type identifier 303:28: 'ndarray' is not a type identifier 309:16: 'ndarray' is not a type identifier 315:15: 'ndarray' is not a type identifier 316:16: 'ndarray' is not a type identifier 317:17: 'ndarray' is not a type identifier 318:18: 'ndarray' is not a type identifier 319:22: 'ndarray' is not a type identifier 320:16: 'ndarray' is not a type identifier 321:17: 'ndarray' is not a type identifier 324:17: 'ndarray' is not a type identifier 325:19: 'ndarray' is not a type identifier 387:9: 'ndarray' is not a type identifier 393:9: 'ndarray' is not a type identifier 403:22: 'ndarray' is not a type identifier 408:24: 'ndarray' is not a type identifier 414:9: 'complex64_t' is not a type identifier 415:9: 'complex128_t' is not a type identifier 424:4: 'float64_t' is not a type identifier 427:9: 'ndarray' is not a type identifier === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option 253:18: 'ndarray' is not a type identifier 258:22: 'ndarray' is not a type identifier 265:18: 'ndarray' is not a type identifier 270:18: 'ndarray' is not a type identifier 276:14: 'ndarray' is not a type identifier 277:15: 'ndarray' is not a type identifier 278:15: 'ndarray' is not a type identifier 279:16: 'ndarray' is not a type identifier 280:13: 'ndarray' is not a type identifier 281:14: 'ndarray' is not a type identifier 282:14: 'ndarray' is not a type identifier 283:15: 'ndarray' is not a type identifier 284:18: 'ndarray' is not a type identifier 285:19: 'ndarray' is not a type identifier 287:15: 'ndarray' is not a type identifier 288:16: 'ndarray' is not a type identifier 289:20: 'ndarray' is not a type identifier 291:16: 'ndarray' is not a type identifier 292:17: 'ndarray' is not a type identifier 293:21: 'ndarray' is not a type identifier 295:23: 'ndarray' is not a type identifier 299:24: 'ndarray' is not a type identifier 303:28: 'ndarray' is not a type identifier 309:16: 'ndarray' is not a type identifier 315:15: 'ndarray' is not a type identifier 316:16: 'ndarray' is not a type identifier 317:17: 'ndarray' is not a type identifier 318:18: 'ndarray' is not a type identifier 319:22: 'ndarray' is not a type identifier 320:16: 'ndarray' is not a type identifier 321:17: 'ndarray' is not a type identifier 324:17: 'ndarray' is not a type identifier 325:19: 'ndarray' is not a type identifier 387:9: 'ndarray' is not a type identifier 393:9: 'ndarray' is not a type identifier 403:22: 'ndarray' is not a type identifier 408:24: 'ndarray' is not a type identifier 414:9: 'complex64_t' is not a type identifier 415:9: 'complex128_t' is not a type identifier 424:4: 'float64_t' is not a type identifier 427:9: 'ndarray' is not a type identifier -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20101211/98434bab/attachment.pgp From robertwb at math.washington.edu Sat Dec 11 23:06:56 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 11 Dec 2010 14:06:56 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: <201012112245.14866.Arfrever.FTA@gmail.com> References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis wrote: > 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >> Release candidate up at >> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >> bad is discovered, this will be the release sometime early next week. > > Some tests fail. :( > Test failures with Python 2.6 (the same tests fail with Python 2.7): What OS and NumPy version? > ====================================================================== > FAIL: Doctest: numpy_test > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for numpy_test > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 25, in numpy_test > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 114, in numpy_test > Failed example: > ? ?test_c_contig(f_arr) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ... > ? ?ValueError: ndarray is not C contiguous > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_c_contig(f_arr) > ? ? ?File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.c:2619) > ? ?ValueError: ndarray is not C-contiguous > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 122, in numpy_test > Failed example: > ? ?test_c_contig(c_arr[::2,::2]) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ... > ? ?ValueError: ndarray is not C contiguous > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_c_contig(c_arr[::2,::2]) > ? ? ?File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.c:2619) > ? ?ValueError: ndarray is not C-contiguous > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 161, in numpy_test > Failed example: > ? ?test_dtype('%si' % other_endian, inc1_int) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ... > ? ?ValueError: Non-native byte order not supported > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_dtype('%si' % other_endian, inc1_int) > ? ? ?File "numpy_test.pyx", line 339, in numpy_test.test_dtype (numpy_test.c:5311) > ? ? ?File "numpy_test.pyx", line 280, in numpy_test.inc1_int (numpy_test.c:3129) > ? ?ValueError: Big-endian buffer not supported on little-endian compiler > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 168, in numpy_test > Failed example: > ? ?test_recordarray() > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_recordarray() > ? ? ?File "numpy_test.pyx", line 347, in numpy_test.test_recordarray (numpy_test.c:5474) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 170, in numpy_test > Failed example: > ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,i')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,i')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > ? ? ?File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.c:5792) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 174, in numpy_test > Failed example: > ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,f')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?ValueError: Buffer dtype mismatch, expected 'int' but got 'float' in 'DoubleInt.y' > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,f')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > ? ? ?File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.c:5792) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 179, in numpy_test > Failed example: > ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > ? ? ?File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.c:6296) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 182, in numpy_test > Failed example: > ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > ? ? ?File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.c:6427) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 186, in numpy_test > Failed example: > ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?ValueError: Buffer dtype mismatch; next field is at offset 4 but 1 expected > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > ? ? ?File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.c:6296) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 191, in numpy_test > Failed example: > ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?ValueError: Buffer dtype mismatch; next field is at offset 1 but 4 expected > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > ? ? ?File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.c:6427) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/c/numpy_test.so", line 209, in numpy_test > Failed example: > ? ?test_point_record() > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_point_record() > ? ? ?File "numpy_test.pyx", line 429, in numpy_test.test_point_record (numpy_test.c:6871) > ? ?ValueError: Does not understand character buffer dtype format string (':') > > > ====================================================================== > FAIL: Doctest: numpy_test > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for numpy_test > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 11, in numpy_test > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 100, in numpy_test > Failed example: > ? ?test_c_contig(f_arr) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ... > ? ?ValueError: ndarray is not C contiguous > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_c_contig(f_arr) > ? ? ?File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.cpp:2619) > ? ?ValueError: ndarray is not C-contiguous > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 108, in numpy_test > Failed example: > ? ?test_c_contig(c_arr[::2,::2]) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ... > ? ?ValueError: ndarray is not C contiguous > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_c_contig(c_arr[::2,::2]) > ? ? ?File "numpy_test.pyx", line 265, in numpy_test.test_c_contig (numpy_test.cpp:2619) > ? ?ValueError: ndarray is not C-contiguous > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 147, in numpy_test > Failed example: > ? ?test_dtype('%si' % other_endian, inc1_int) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ... > ? ?ValueError: Non-native byte order not supported > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_dtype('%si' % other_endian, inc1_int) > ? ? ?File "numpy_test.pyx", line 339, in numpy_test.test_dtype (numpy_test.cpp:5311) > ? ? ?File "numpy_test.pyx", line 280, in numpy_test.inc1_int (numpy_test.cpp:3129) > ? ?ValueError: Big-endian buffer not supported on little-endian compiler > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 154, in numpy_test > Failed example: > ? ?test_recordarray() > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_recordarray() > ? ? ?File "numpy_test.pyx", line 347, in numpy_test.test_recordarray (numpy_test.cpp:5474) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 156, in numpy_test > Failed example: > ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,i')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,i')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > ? ? ?File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.cpp:5792) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 160, in numpy_test > Failed example: > ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,f')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?ValueError: Buffer dtype mismatch, expected 'int' but got 'float' in 'DoubleInt.y' > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_nested_dtypes(np.zeros((3,), dtype=np.dtype([ ? ? ? ? ? ?('a', np.dtype('i,f')), ? ? ? ? ? ?('b', np.dtype('i,i')) ? ? ? ?])))) > ? ? ?File "numpy_test.pyx", line 374, in numpy_test.test_nested_dtypes (numpy_test.cpp:5792) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 165, in numpy_test > Failed example: > ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > ? ? ?File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.cpp:6296) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 168, in numpy_test > Failed example: > ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > ? ? ?File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.cpp:6427) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 172, in numpy_test > Failed example: > ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?ValueError: Buffer dtype mismatch; next field is at offset 4 but 1 expected > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=True)))) > ? ? ?File "numpy_test.pyx", line 403, in numpy_test.test_packed_align (numpy_test.cpp:6296) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 177, in numpy_test > Failed example: > ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?ValueError: Buffer dtype mismatch; next field is at offset 1 but 4 expected > Got: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?print(test_unpacked_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > ? ? ?File "numpy_test.pyx", line 408, in numpy_test.test_unpacked_align (numpy_test.cpp:6427) > ? ?ValueError: Does not understand character buffer dtype format string (':') > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/run/cpp/numpy_test.so", line 195, in numpy_test > Failed example: > ? ?test_point_record() > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/usr/lib64/python2.6/doctest.py", line 1253, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", line 1, in > ? ? ? ?test_point_record() > ? ? ?File "numpy_test.pyx", line 429, in numpy_test.test_point_record (numpy_test.cpp:6871) > ? ?ValueError: Does not understand character buffer dtype format string (':') > > > ---------------------------------------------------------------------- > Ran 4812 tests in 1194.085s > > FAILED (failures=2) > > > > Test failures with Python 3.1 (the same tests fail with Python 3.2): > > ====================================================================== > ERROR: test_globals (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 32, in test_globals > ? ?self.assertEquals(inline("return global_value + 1", **test_kwds), global_value + 1) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_locals (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 29, in test_locals > ? ?self.assertEquals(inline("return a+b", **test_kwds), 3) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_numpy (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 41, in test_numpy > ? ?self.assertEquals(inline("return a[0,0]", a=a, **test_kwds), 10.0) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_simple (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 18, in test_simple > ? ?self.assertEquals(inline("return 1+2", **test_kwds), 3) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_types (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 24, in test_types > ? ?""", a=1.0, b=[], **test_kwds), ('double', 'list object')) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: compiling (c) and running numpy_bufacc_T155 > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "runtests.py", line 464, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 299, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 441, in compile > ? ?self.assertEquals(None, unexpected_error) > AssertionError: None != "10:9: 'ndarray' is not a type identifier" > > ====================================================================== > ERROR: compiling (cpp) and running numpy_bufacc_T155 > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "runtests.py", line 464, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 299, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 441, in compile > ? ?self.assertEquals(None, unexpected_error) > AssertionError: None != "10:9: 'ndarray' is not a type identifier" > > ====================================================================== > ERROR: compiling (c) and running numpy_cimport > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "runtests.py", line 464, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 299, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 441, in compile > ? ?self.assertEquals(None, unexpected_error) > AssertionError: None != '173:49: "mode" is not a buffer option' > > ====================================================================== > ERROR: compiling (cpp) and running numpy_cimport > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "runtests.py", line 464, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 299, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 441, in compile > ? ?self.assertEquals(None, unexpected_error) > AssertionError: None != '173:49: "mode" is not a buffer option' > > ====================================================================== > ERROR: compiling (c) and running numpy_test > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "runtests.py", line 464, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 299, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 441, in compile > ? ?self.assertEquals(None, unexpected_error) > AssertionError: None != '173:49: "mode" is not a buffer option' > > ====================================================================== > ERROR: compiling (cpp) and running numpy_test > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "runtests.py", line 464, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 299, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 441, in compile > ? ?self.assertEquals(None, unexpected_error) > AssertionError: None != '173:49: "mode" is not a buffer option' > > ---------------------------------------------------------------------- > Ran 4773 tests in 1128.064s > > FAILED (errors=11) > > === Expected errors: === > > > > === Got errors: === > 10:9: 'ndarray' is not a type identifier > 173:49: "mode" is not a buffer option > > > > === Expected errors: === > > > > === Got errors: === > 10:9: 'ndarray' is not a type identifier > 173:49: "mode" is not a buffer option > > > > === Expected errors: === > > > > === Got errors: === > 173:49: "mode" is not a buffer option > > > > === Expected errors: === > > > > === Got errors: === > 173:49: "mode" is not a buffer option > > > > === Expected errors: === > > > > === Got errors: === > 173:49: "mode" is not a buffer option > 253:18: 'ndarray' is not a type identifier > 258:22: 'ndarray' is not a type identifier > 265:18: 'ndarray' is not a type identifier > 270:18: 'ndarray' is not a type identifier > 276:14: 'ndarray' is not a type identifier > 277:15: 'ndarray' is not a type identifier > 278:15: 'ndarray' is not a type identifier > 279:16: 'ndarray' is not a type identifier > 280:13: 'ndarray' is not a type identifier > 281:14: 'ndarray' is not a type identifier > 282:14: 'ndarray' is not a type identifier > 283:15: 'ndarray' is not a type identifier > 284:18: 'ndarray' is not a type identifier > 285:19: 'ndarray' is not a type identifier > 287:15: 'ndarray' is not a type identifier > 288:16: 'ndarray' is not a type identifier > 289:20: 'ndarray' is not a type identifier > 291:16: 'ndarray' is not a type identifier > 292:17: 'ndarray' is not a type identifier > 293:21: 'ndarray' is not a type identifier > 295:23: 'ndarray' is not a type identifier > 299:24: 'ndarray' is not a type identifier > 303:28: 'ndarray' is not a type identifier > 309:16: 'ndarray' is not a type identifier > 315:15: 'ndarray' is not a type identifier > 316:16: 'ndarray' is not a type identifier > 317:17: 'ndarray' is not a type identifier > 318:18: 'ndarray' is not a type identifier > 319:22: 'ndarray' is not a type identifier > 320:16: 'ndarray' is not a type identifier > 321:17: 'ndarray' is not a type identifier > 324:17: 'ndarray' is not a type identifier > 325:19: 'ndarray' is not a type identifier > 387:9: 'ndarray' is not a type identifier > 393:9: 'ndarray' is not a type identifier > 403:22: 'ndarray' is not a type identifier > 408:24: 'ndarray' is not a type identifier > 414:9: 'complex64_t' is not a type identifier > 415:9: 'complex128_t' is not a type identifier > 424:4: 'float64_t' is not a type identifier > 427:9: 'ndarray' is not a type identifier > > > > === Expected errors: === > > > > === Got errors: === > 173:49: "mode" is not a buffer option > 253:18: 'ndarray' is not a type identifier > 258:22: 'ndarray' is not a type identifier > 265:18: 'ndarray' is not a type identifier > 270:18: 'ndarray' is not a type identifier > 276:14: 'ndarray' is not a type identifier > 277:15: 'ndarray' is not a type identifier > 278:15: 'ndarray' is not a type identifier > 279:16: 'ndarray' is not a type identifier > 280:13: 'ndarray' is not a type identifier > 281:14: 'ndarray' is not a type identifier > 282:14: 'ndarray' is not a type identifier > 283:15: 'ndarray' is not a type identifier > 284:18: 'ndarray' is not a type identifier > 285:19: 'ndarray' is not a type identifier > 287:15: 'ndarray' is not a type identifier > 288:16: 'ndarray' is not a type identifier > 289:20: 'ndarray' is not a type identifier > 291:16: 'ndarray' is not a type identifier > 292:17: 'ndarray' is not a type identifier > 293:21: 'ndarray' is not a type identifier > 295:23: 'ndarray' is not a type identifier > 299:24: 'ndarray' is not a type identifier > 303:28: 'ndarray' is not a type identifier > 309:16: 'ndarray' is not a type identifier > 315:15: 'ndarray' is not a type identifier > 316:16: 'ndarray' is not a type identifier > 317:17: 'ndarray' is not a type identifier > 318:18: 'ndarray' is not a type identifier > 319:22: 'ndarray' is not a type identifier > 320:16: 'ndarray' is not a type identifier > 321:17: 'ndarray' is not a type identifier > 324:17: 'ndarray' is not a type identifier > 325:19: 'ndarray' is not a type identifier > 387:9: 'ndarray' is not a type identifier > 393:9: 'ndarray' is not a type identifier > 403:22: 'ndarray' is not a type identifier > 408:24: 'ndarray' is not a type identifier > 414:9: 'complex64_t' is not a type identifier > 415:9: 'complex128_t' is not a type identifier > 424:4: 'float64_t' is not a type identifier > 427:9: 'ndarray' is not a type identifier > > -- > Arfrever Frehtes Taifersar Arahesis > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > From arfrever.fta at gmail.com Sat Dec 11 23:11:28 2010 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sat, 11 Dec 2010 23:11:28 +0100 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: <201012112311.28700.Arfrever.FTA@gmail.com> 2010-12-11 23:06:56 Robert Bradshaw napisa?(a): > On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis > wrote: > > 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): > >> Release candidate up at > >> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something > >> bad is discovered, this will be the release sometime early next week. > > > > Some tests fail. > > :( > > > Test failures with Python 2.6 (the same tests fail with Python 2.7): > > What OS and NumPy version? Gentoo Linux, NumPy 1.5.1. -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20101211/582d1f13/attachment.pgp From robertwb at math.washington.edu Sun Dec 12 00:09:48 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 11 Dec 2010 15:09:48 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: <201012112245.14866.Arfrever.FTA@gmail.com> References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis wrote: > 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >> Release candidate up at >> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >> bad is discovered, this will be the release sometime early next week. > > Some tests fail. > > Test failures with Python 2.6 (the same tests fail with Python 2.7): [...] Not sue what's up with all these numpy errors, but many (not all) of them look like a change in formatting of error messages. The basic functionality still seems to be passing--does anyone know if this is a regression? > Test failures with Python 3.1 (the same tests fail with Python 3.2): > > ====================================================================== > ERROR: test_globals (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 32, in test_globals > ? ?self.assertEquals(inline("return global_value + 1", **test_kwds), global_value + 1) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_locals (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 29, in test_locals > ? ?self.assertEquals(inline("return a+b", **test_kwds), 3) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_numpy (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 41, in test_numpy > ? ?self.assertEquals(inline("return a[0,0]", a=a, **test_kwds), 10.0) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_simple (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 18, in test_simple > ? ?self.assertEquals(inline("return 1+2", **test_kwds), 3) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' > > ====================================================================== > ERROR: test_types (Cython.Build.Tests.TestInline.TestStripLiterals) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 24, in test_types > ? ?""", a=1.0, b=[], **test_kwds), ('double', 'list object')) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Shadow.py", line 16, in inline > ? ?return cython_inline(f, *args, **kwds) > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 101, in cython_inline > ? ?for symbol in unbound_symbols(code): > ?File "/var/tmp/portage/dev-python/cython-0.14_rc0/work/Cython-0.14.rc0/BUILD/Cy3/Cython/Build/Inline.py", line 35, in unbound_symbols > ? ?code = code.decode('ascii') > AttributeError: 'str' object has no attribute 'decode' OK, I've fixed these ones. Can you try out the latest tip? > ====================================================================== > ERROR: compiling (c) and running numpy_bufacc_T155 > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "runtests.py", line 464, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 299, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 441, in compile > ? ?self.assertEquals(None, unexpected_error) > AssertionError: None != "10:9: 'ndarray' is not a type identifier" [...] Looks like somehow it's not finding numpy.pxid? - Robert From robert.kern at gmail.com Sun Dec 12 02:52:40 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 11 Dec 2010 19:52:40 -0600 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: On 2010-12-11 17:09 , Robert Bradshaw wrote: > On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis > wrote: >> 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >>> Release candidate up at >>> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >>> bad is discovered, this will be the release sometime early next week. >> >> Some tests fail. >> >> Test failures with Python 2.6 (the same tests fail with Python 2.7): > > [...] > > Not sue what's up with all these numpy errors, but many (not all) of > them look like a change in formatting of error messages. The basic > functionality still seems to be passing--does anyone know if this is a > regression? I think so. I think these are real regressions: ValueError: Does not understand character buffer dtype format string (':') From regular Python, I've tried a few of the dtype() calls that you make, and they seem to work fine. I don't see a problem in the generated code, though. The rest are, as you say, just changes to the contents of the exception messages. That's the problem with over-specific doctests. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robertwb at math.washington.edu Sun Dec 12 04:07:30 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 11 Dec 2010 19:07:30 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: On Sat, Dec 11, 2010 at 5:52 PM, Robert Kern wrote: > On 2010-12-11 17:09 , Robert Bradshaw wrote: >> On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis >> ?wrote: >>> 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >>>> Release candidate up at >>>> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >>>> bad is discovered, this will be the release sometime early next week. >>> >>> Some tests fail. >>> >>> Test failures with Python 2.6 (the same tests fail with Python 2.7): >> >> [...] >> >> Not sue what's up with all these numpy errors, but many (not all) of >> them look like a change in formatting of error messages. The basic >> functionality still seems to be passing--does anyone know if this is a >> regression? > > I think so. I think these are real regressions: > > ? ValueError: Does not understand character buffer dtype format string (':') > > ?From regular Python, I've tried a few of the dtype() calls that you make, and > they seem to work fine. I don't see a problem in the generated code, though. They all work for me too, but maybe that's because I'm using an older version of NumPy (the one that ships with Sage). Has anyone tried Numpy 1.5 with Cython 0.13? > The rest are, as you say, just changes to the contents of the exception > messages. That's the problem with over-specific doctests. :-) Very true (though in this case we are verifying that the right error is raised). - Robert From robertwb at math.washington.edu Sun Dec 12 05:29:04 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 11 Dec 2010 20:29:04 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: On Sat, Dec 11, 2010 at 7:07 PM, Robert Bradshaw wrote: > On Sat, Dec 11, 2010 at 5:52 PM, Robert Kern wrote: >> On 2010-12-11 17:09 , Robert Bradshaw wrote: >>> On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis >>> ?wrote: >>>> 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >>>>> Release candidate up at >>>>> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >>>>> bad is discovered, this will be the release sometime early next week. >>>> >>>> Some tests fail. >>>> >>>> Test failures with Python 2.6 (the same tests fail with Python 2.7): >>> >>> [...] >>> >>> Not sue what's up with all these numpy errors, but many (not all) of >>> them look like a change in formatting of error messages. The basic >>> functionality still seems to be passing--does anyone know if this is a >>> regression? >> >> I think so. I think these are real regressions: >> >> ? ValueError: Does not understand character buffer dtype format string (':') >> >> ?From regular Python, I've tried a few of the dtype() calls that you make, and >> they seem to work fine. I don't see a problem in the generated code, though. > > They all work for me too, but maybe that's because I'm using an older > version of NumPy (the one that ships with Sage). Has anyone tried > Numpy 1.5 with Cython 0.13? OK, I just install Numpy 1.5.1, and all the tests pass for me with both 0.13 and 0.14.rc0 (Python 2.6, OS X). Anyone else able to reproduce these errors? Arfrever, can you try Cython 0.13 in your setup? - Robert From dagss at student.matnat.uio.no Sun Dec 12 06:45:42 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 12 Dec 2010 06:45:42 +0100 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: <4D046186.2060501@student.matnat.uio.no> On 12/12/2010 05:29 AM, Robert Bradshaw wrote: > On Sat, Dec 11, 2010 at 7:07 PM, Robert Bradshaw > wrote: > >> On Sat, Dec 11, 2010 at 5:52 PM, Robert Kern wrote: >> >>> On 2010-12-11 17:09 , Robert Bradshaw wrote: >>> >>>> On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis >>>> wrote: >>>> >>>>> 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >>>>> >>>>>> Release candidate up at >>>>>> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >>>>>> bad is discovered, this will be the release sometime early next week. >>>>>> >>>>> Some tests fail. >>>>> >>>>> Test failures with Python 2.6 (the same tests fail with Python 2.7): >>>>> >>>> [...] >>>> >>>> Not sue what's up with all these numpy errors, but many (not all) of >>>> them look like a change in formatting of error messages. The basic >>>> functionality still seems to be passing--does anyone know if this is a >>>> regression? >>>> >>> I think so. I think these are real regressions: >>> >>> ValueError: Does not understand character buffer dtype format string (':') >>> >>> From regular Python, I've tried a few of the dtype() calls that you make, and >>> they seem to work fine. I don't see a problem in the generated code, though. >>> >> They all work for me too, but maybe that's because I'm using an older >> version of NumPy (the one that ships with Sage). Has anyone tried >> Numpy 1.5 with Cython 0.13? >> > OK, I just install Numpy 1.5.1, and all the tests pass for me with > both 0.13 and 0.14.rc0 (Python 2.6, OS X). Anyone else able to > reproduce these errors? Arfrever, can you try Cython 0.13 in your > setup? > The thing that has happened here is that the new NumPy supports the PEP 3118 protocol, and so the code in numpy.pxd is not used, but instead the other code. Probably new tests should be written, perhaps some changes must be done to buffer format string parsing, and so on. But it is unlikely to be a regression in Cython, nothing changed here w.r.t Cython 0.13.x. Dag Sverre From robertwb at math.washington.edu Sun Dec 12 06:56:41 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 11 Dec 2010 21:56:41 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: <4D046186.2060501@student.matnat.uio.no> References: <201012112245.14866.Arfrever.FTA@gmail.com> <4D046186.2060501@student.matnat.uio.no> Message-ID: On Sat, Dec 11, 2010 at 9:45 PM, Dag Sverre Seljebotn wrote: > On 12/12/2010 05:29 AM, Robert Bradshaw wrote: >> On Sat, Dec 11, 2010 at 7:07 PM, Robert Bradshaw >> ?wrote: >> >>> On Sat, Dec 11, 2010 at 5:52 PM, Robert Kern ?wrote: >>> >>>> On 2010-12-11 17:09 , Robert Bradshaw wrote: >>>> >>>>> On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis >>>>> ? ?wrote: >>>>> >>>>>> 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >>>>>> >>>>>>> Release candidate up at >>>>>>> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >>>>>>> bad is discovered, this will be the release sometime early next week. >>>>>>> >>>>>> Some tests fail. >>>>>> >>>>>> Test failures with Python 2.6 (the same tests fail with Python 2.7): >>>>>> >>>>> [...] >>>>> >>>>> Not sue what's up with all these numpy errors, but many (not all) of >>>>> them look like a change in formatting of error messages. The basic >>>>> functionality still seems to be passing--does anyone know if this is a >>>>> regression? >>>>> >>>> I think so. I think these are real regressions: >>>> >>>> ? ?ValueError: Does not understand character buffer dtype format string (':') >>>> >>>> ? From regular Python, I've tried a few of the dtype() calls that you make, and >>>> they seem to work fine. I don't see a problem in the generated code, though. >>>> >>> They all work for me too, but maybe that's because I'm using an older >>> version of NumPy (the one that ships with Sage). Has anyone tried >>> Numpy 1.5 with Cython 0.13? >>> >> OK, I just install Numpy 1.5.1, and all the tests pass for me with >> both 0.13 and 0.14.rc0 (Python 2.6, OS X). Anyone else able to >> reproduce these errors? Arfrever, can you try Cython 0.13 in your >> setup? >> > > The thing that has happened here is that the new NumPy supports the PEP > 3118 protocol, and so the code in numpy.pxd is not used, but instead the > other code. Probably new tests should be written, perhaps some changes > must be done to buffer format string parsing, and so on. But it is > unlikely to be a regression in Cython, nothing changed here w.r.t > Cython ?0.13.x. That would make sense, thanks. I didn't think much had changed with numpy support on our side, but didn't know for sure. - Robert From robert.kern at gmail.com Sun Dec 12 07:32:13 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 12 Dec 2010 00:32:13 -0600 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: On 2010-12-11 22:29 , Robert Bradshaw wrote: > OK, I just install Numpy 1.5.1, and all the tests pass for me with > both 0.13 and 0.14.rc0 (Python 2.6, OS X). Anyone else able to > reproduce these errors? Arfrever, can you try Cython 0.13 in your > setup? I can reproduce these errors using numpy 1.5.1 and Cython 0.14rc0 on Python 2.6 and OS X. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From vitja.makarov at gmail.com Sun Dec 12 14:51:55 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 12 Dec 2010 16:51:55 +0300 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: 2010/12/12 Robert Kern : > On 2010-12-11 22:29 , Robert Bradshaw wrote: > >> OK, I just install Numpy 1.5.1, and all the tests pass for me with >> both 0.13 and 0.14.rc0 (Python 2.6, OS X). Anyone else able to >> reproduce these errors? Arfrever, can you try Cython 0.13 in your >> setup? > > I can reproduce these errors using numpy 1.5.1 and Cython 0.14rc0 on Python 2.6 > and OS X. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > ?that is made terrible by our own mad attempt to interpret it as though it had > ?an underlying truth." > ? -- Umberto Eco > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Cython-tests are now broken... -- vitja. From stefan_ml at behnel.de Sun Dec 12 17:10:22 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 12 Dec 2010 17:10:22 +0100 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> Message-ID: <4D04F3EE.30001@behnel.de> [remainings of a completely unrelated thread deleted] Vitja Makarov, 12.12.2010 14:51: > Cython-tests are now broken... Yep, the new class scope for the conjugate changes broke three tests. Unless it's an easy fix, I'd vote for reverting this post-RC change. Stefan From vitja.makarov at gmail.com Sun Dec 12 20:23:27 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 12 Dec 2010 22:23:27 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> Message-ID: 2010/12/11 Vitja Makarov : > 2010/12/11 Stefan Behnel : >> Vitja Makarov, 10.12.2010 09:22: >>> 2010/12/10 Stefan Behnel: >>>> Stefan Behnel, 09.12.2010 10:46: >>>>> Vitja Makarov, 08.12.2010 22:45: >>>>>> Please review this patch. It's not yet finished, and mostly doesn't work. >>>>>> But you can take a look at patch and generated code. >>>>> >>>>> I'll take a close look ASAP. >>>>> >>>>>> ? ? - Temps are saved/restored/allocated inside YieldExprNode using >>>>>> helper ClosureTempAllocator >>>> >>>> Ok, that currently lives in ParseTreeTransforms, which is totally the wrong >>>> place for anything that generates code. It also mixes functionality that >>>> normally lives in Symtab.py with functionality that lives in Code.py. >>>> >>> >>> Yes, I know. That was placed there to give it a try (actually I'm not >>> sure what is the best place for it). ClosureTempAllocator now is >>> attribute of defnode and all yieldnodes, that's not clean. >> >> FunctionState in Code.py seems like a good place to hold a reference, at >> least for the code generation part of it. >> > > At some point we can call code.set_closure_class() > that would create code.temp_allocator and then refer to it as > code.temp_allocator. > temp_allocator isn't good name here maybe closure_temps? > >> >>> ExprNode.StopIterationNode isn't expression should it be moved to Nodes.py? >> >> Why not drop it completely and generate the code in the function node >> instead? ISTM that the only reason it's there is to drop one line of code >> at the end of the function body. That can be done much easier. >> >> > > Sure. You've already done this, thanks! > >>> create_abstract_generator() is rather dirty >> >> I've seen worse. ;) >> >> >>> I don't know how to bind C functions to CClass. >> >> Such as? >> >> > > e = declare() > e.func_cname = ... > e.signature = ... > I think that there should be a short way > >>> I think that generator body could have PyObject *body(PyObject *self, >>> PyObject *value) signature >>> With value=Null meaning exception. >> >> See the CEP. ;) >> > > Yeah! Forget about that, that's already done. I was looking into > python implementation and it has `i_exc` arg. > >> >>>>>> ? ? - PyxGenerator methods are defined via declare_var >>>>> >>>>> Should we call that "CyGenerator" ? :) >>>> >>>> I actually meant to call it "CyGenerator" at the Python level, but I now >>>> see that it's actually a purely internal thing. That makes __Pyx_Generator >>>> a better choice, given that "__pyx_" is the globally reserved prefix in >>>> Cython code. >>> >>> Should rename it back ;) >> >> Yep. >> > > __pyx_CyGenerator_(Send|Close)() <-- or lover case suffix is better, > should look at pep7? > class __pyx_CyGenerator > type __pyx_CyGeneratorType > > is that ok? > >> >>> How can I declare function entry with METH_VARGS signature? >>> Using TypeSlots I can add METH_VARGS|METH_KWARGS entries but not >>> single METH_VARGS >>> >>> That's preferd for "throw" method. >> >> I see what you mean. It's a minor issue, though. Just ignore it for now, or >> raise an exception if kwargs are passed in. After all, .throw() is rarely >> used and not performance critical at all. >> > > ok > >>> Pyregr tests works now on hudson but there is still problem with heapq >>> don't understand is that related to generators or not >> >> Not sure either. It works for me in Py3 but not in Py2.7. > > I see you've fixed this issue. Cool. > >> disable generators test in Py2.4 - depends on GeneratorExit and potentially other stuff > > May be it's better to generate another exception (StopIteration) in > 2.4 and control this with #ifdefs ... #endif > > What do you think about generate_preamble? Think it's not correct name. > > In GeneratorWrapper() buffer cleanup code isn't implemented when > closure is created. > > What is the place for YieldCollector? > > About genexp: now it should be easy to implement just like lambda function > > -- > vitja. > I've cleaned generators stuff a little: * Turn YieldNodeCollector into TreeVisitor * Merge MarkClosureVisitor and MarkGeneratorVisitor * Rename stuff from CyGenerator to __Pyx_Generator * Move ClosureTempAllocator to Code.py It seems that there is no major issues with generators, refleaks and so on. So now is time to add support for genexps and code cleanup. What do you think about GeneratorWrapperNode? -- vitja. From stefan_ml at behnel.de Mon Dec 13 06:37:26 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 13 Dec 2010 06:37:26 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> Message-ID: <4D05B116.2020900@behnel.de> Vitja Makarov, 12.12.2010 20:23: > I've cleaned generators stuff a little: > > * Turn YieldNodeCollector into TreeVisitor I think that's ok. It's a bit redundant with the tree traversal that happens anyway, but it also keeps the code more compact and makes it more understandable. BTW, please start using error test cases. That would easily have caught the obvious reporting bug in visit_ReturnStatNode. > * Merge MarkClosureVisitor and MarkGeneratorVisitor That looks much better. > * Rename stuff from CyGenerator to __Pyx_Generator Thanks. However, the following doesn't seem to have an effect, does it? - entry.cname = 'CyGenerator' + entry.cname = 'Generator' If that's the only place where that cname is used, I'd guess that it should better be dropped. > * Move ClosureTempAllocator to Code.py I still think that the reference should live in FunctionState so that it goes out of scope automatically. It shouldn't live longer than the function. > It seems that there is no major issues with generators, refleaks and so on. I'm not so sure about the ref-counting yet. I'll have to think it through a bit more, but it currently feels a bit funny. (That doesn't mean it's wrong, just that I'm not sure it's the way it should be) > So now is time to add support for genexps and code cleanup. Absolutely. :) > What do you think about GeneratorWrapperNode? It seems ok. I'll give it a deeper look ASAP. Stefan From vitja.makarov at gmail.com Mon Dec 13 07:49:32 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 13 Dec 2010 09:49:32 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D05B116.2020900@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> Message-ID: 2010/12/13 Stefan Behnel : > Vitja Makarov, 12.12.2010 20:23: >> I've cleaned generators stuff a little: >> >> ? * Turn YieldNodeCollector into TreeVisitor > > I think that's ok. It's a bit redundant with the tree traversal that > happens anyway, but it also keeps the code more compact and makes it more > understandable. > > BTW, please start using error test cases. That would easily have caught the > obvious reporting bug in visit_ReturnStatNode. > > I was thinking about that. I also found that 'return' statment is allowed inside generators. That should raise StopIteration and close generator. Don't know how to go here replace ReturnStatNode with another one or set attribute of return node: node.return_from_generator = True? >> ? * Merge MarkClosureVisitor and MarkGeneratorVisitor > > That looks much better. > > >> ? * Rename stuff from CyGenerator to __Pyx_Generator > > Thanks. However, the following doesn't seem to have an effect, does it? > > - ? ? ? ?entry.cname = 'CyGenerator' > + ? ? ? ?entry.cname = 'Generator' > > If that's the only place where that cname is used, I'd guess that it should > better be dropped. > Sure. Obsolete code ) > >> ? * Move ClosureTempAllocator to Code.py > > I still think that the reference should live in FunctionState so that it > goes out of scope automatically. It shouldn't live longer than the function. > Ok, I'll put reference to code.funcstate, or maybe add method to funcstate set_closure_class() That will create tempallocator? > >> It seems that there is no major issues with generators, refleaks and so on. > > I'm not so sure about the ref-counting yet. I'll have to think it through a > bit more, but it currently feels a bit funny. (That doesn't mean it's > wrong, just that I'm not sure it's the way it should be) > > >> So now is time to add support for genexps and code cleanup. > > Absolutely. :) > > >> What do you think about GeneratorWrapperNode? > > It seems ok. I'll give it a deeper look ASAP. > ok. -- vitja. From stefan_ml at behnel.de Mon Dec 13 09:02:08 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 13 Dec 2010 09:02:08 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> Message-ID: <4D05D300.2080506@behnel.de> Vitja Makarov, 12.12.2010 20:23: > It seems that there is no major issues with generators, refleaks and so on. I know what the refcounting problem is now. It's a bug that has already been (mostly) fixed in Haoyu's branch. So far, we didn't have "nonlocal", so there was no way to test how closure variables behaved on assignment. Since local variables in generator functions now live in the closure and can happily get assigned to, they suffer from the same problem that nonlocal has. The place to fix this is in the assignment code in NameNode. For values in closures, it must generate a GOTREF() before DECREF-ing the lhs in the assignment and a GIVREF() after the assignment. I fixed that and also ripped out the "make refnanny happy" code sections. Doing that, I also noticed a couple of other related refnanny problems for argument handling. I didn't care to port them back to mainline since I'd expect that the current 0.14 release can live with them. Let's assume from the current state of the generators branch that 0.15 is not too far off. Stefan From stefan_ml at behnel.de Mon Dec 13 13:40:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 13 Dec 2010 13:40:05 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> Message-ID: <4D061425.30908@behnel.de> Vitja Makarov, 13.12.2010 07:49: > 2010/12/13 Stefan Behnel: >> Vitja Makarov, 12.12.2010 20:23: >>> I've cleaned generators stuff a little: >>> >>> * Turn YieldNodeCollector into TreeVisitor >> >> I think that's ok. It's a bit redundant with the tree traversal that >> happens anyway, but it also keeps the code more compact and makes it more >> understandable. >> >> BTW, please start using error test cases. That would easily have caught the >> obvious reporting bug in visit_ReturnStatNode. > > I was thinking about that. I also found that 'return' statment is > allowed inside generators. > That should raise StopIteration and close generator. > Don't know how to go here replace ReturnStatNode with another one You can replace it by the equivalent of "raise StopIteration" during the MarkClosureVisitor transform, e.g. inside of the YieldNodeCollector (which would then have to be a transform again). Either emit a compiler error if the return statement has an argument (IIRC that's the forbidden case) or replace the node if it has none. >>> * Move ClosureTempAllocator to Code.py >> >> I still think that the reference should live in FunctionState so that it >> goes out of scope automatically. It shouldn't live longer than the function. >> > > Ok, I'll put reference to code.funcstate, or maybe add method to > funcstate set_closure_class() > That will create tempallocator? Hmm, not sure. What about adding an "init_generator_temps(scope)" method to it that creates the CTA and attaches it to the current funcstate? Stefan From vitja.makarov at gmail.com Mon Dec 13 13:54:04 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 13 Dec 2010 15:54:04 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D05D300.2080506@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05D300.2080506@behnel.de> Message-ID: 2010/12/13 Stefan Behnel : > Vitja Makarov, 12.12.2010 20:23: >> It seems that there is no major issues with generators, refleaks and so on. > > I know what the refcounting problem is now. It's a bug that has already > been (mostly) fixed in Haoyu's branch. So far, we didn't have "nonlocal", > so there was no way to test how closure variables behaved on assignment. > Since local variables in generator functions now live in the closure and > can happily get assigned to, they suffer from the same problem that > nonlocal has. > > The place to fix this is in the assignment code in NameNode. For values in > closures, it must generate a GOTREF() before DECREF-ing the lhs in the > assignment and a GIVREF() after the assignment. I fixed that and also > ripped out the "make refnanny happy" code sections. > > Doing that, I also noticed a couple of other related refnanny problems for > argument handling. I didn't care to port them back to mainline since I'd > expect that the current 0.14 release can live with them. Let's assume from > the current state of the generators branch that 0.15 is not too far off. > > Stefan Did you already fixed this bug? -- vitja. From vitja.makarov at gmail.com Mon Dec 13 13:56:24 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 13 Dec 2010 15:56:24 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D061425.30908@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D061425.30908@behnel.de> Message-ID: 2010/12/13 Stefan Behnel : > Vitja Makarov, 13.12.2010 07:49: >> 2010/12/13 Stefan Behnel: >>> Vitja Makarov, 12.12.2010 20:23: >>>> I've cleaned generators stuff a little: >>>> >>>> ? ?* Turn YieldNodeCollector into TreeVisitor >>> >>> I think that's ok. It's a bit redundant with the tree traversal that >>> happens anyway, but it also keeps the code more compact and makes it more >>> understandable. >>> >>> BTW, please start using error test cases. That would easily have caught the >>> obvious reporting bug in visit_ReturnStatNode. >> >> I was thinking about that. I also found that 'return' statment is >> allowed inside generators. >> That should raise StopIteration and close generator. >> Don't know how to go here replace ReturnStatNode with another one > > You can replace it by the equivalent of "raise StopIteration" during the > MarkClosureVisitor transform, e.g. inside of the YieldNodeCollector (which > would then have to be a transform again). Either emit a compiler error if > the return statement has an argument (IIRC that's the forbidden case) or > replace the node if it has none. > > That shold be much better to jump to the place where StopIteration is raised. Couse StopIteration should not be catched here and finallize should be called. >>>> ? ?* Move ClosureTempAllocator to Code.py >>> >>> I still think that the reference should live in FunctionState so that it >>> goes out of scope automatically. It shouldn't live longer than the function. >>> >> >> Ok, I'll put reference to code.funcstate, or maybe add method to >> funcstate set_closure_class() >> That will create tempallocator? > > Hmm, not sure. What about adding an "init_generator_temps(scope)" method to > it that creates the CTA and attaches it to the current funcstate? > Ok > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- vitja. From stefan_ml at behnel.de Mon Dec 13 14:36:38 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 13 Dec 2010 14:36:38 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05D300.2080506@behnel.de> Message-ID: <4D062166.305@behnel.de> Vitja Makarov, 13.12.2010 13:54: > 2010/12/13 Stefan Behnel: >> Vitja Makarov, 12.12.2010 20:23: >>> It seems that there is no major issues with generators, refleaks and so on. >> >> I know what the refcounting problem is now. It's a bug that has already >> been (mostly) fixed in Haoyu's branch. So far, we didn't have "nonlocal", >> so there was no way to test how closure variables behaved on assignment. >> Since local variables in generator functions now live in the closure and >> can happily get assigned to, they suffer from the same problem that >> nonlocal has. >> >> The place to fix this is in the assignment code in NameNode. For values in >> closures, it must generate a GOTREF() before DECREF-ing the lhs in the >> assignment and a GIVREF() after the assignment. I fixed that and also >> ripped out the "make refnanny happy" code sections. >> >> Doing that, I also noticed a couple of other related refnanny problems for >> argument handling. I didn't care to port them back to mainline since I'd >> expect that the current 0.14 release can live with them. Let's assume from >> the current state of the generators branch that 0.15 is not too far off. > > Did you already fixed this bug? Yes, and *now* it's even up on github. :) Stefan From vitja.makarov at gmail.com Mon Dec 13 14:39:28 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 13 Dec 2010 16:39:28 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D062166.305@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05D300.2080506@behnel.de> <4D062166.305@behnel.de> Message-ID: 2010/12/13 Stefan Behnel : > Vitja Makarov, 13.12.2010 13:54: >> 2010/12/13 Stefan Behnel: >>> Vitja Makarov, 12.12.2010 20:23: >>>> It seems that there is no major issues with generators, refleaks and so on. >>> >>> I know what the refcounting problem is now. It's a bug that has already >>> been (mostly) fixed in Haoyu's branch. So far, we didn't have "nonlocal", >>> so there was no way to test how closure variables behaved on assignment. >>> Since local variables in generator functions now live in the closure and >>> can happily get assigned to, they suffer from the same problem that >>> nonlocal has. >>> >>> The place to fix this is in the assignment code in NameNode. For values in >>> closures, it must generate a GOTREF() before DECREF-ing the lhs in the >>> assignment and a GIVREF() after the assignment. I fixed that and also >>> ripped out the "make refnanny happy" code sections. >>> >>> Doing that, I also noticed a couple of other related refnanny problems for >>> argument handling. I didn't care to port them back to mainline since I'd >>> expect that the current 0.14 release can live with them. Let's assume from >>> the current state of the generators branch that 0.15 is not too far off. >> >> Did you already fixed this bug? > > Yes, and *now* it's even up on github. :) > > Stefan Cool :)) -- vitja. From robertwb at math.washington.edu Mon Dec 13 18:28:36 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 13 Dec 2010 09:28:36 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: <4D04F3EE.30001@behnel.de> References: <201012112245.14866.Arfrever.FTA@gmail.com> <4D04F3EE.30001@behnel.de> Message-ID: On Sun, Dec 12, 2010 at 8:10 AM, Stefan Behnel wrote: > [remainings of a completely unrelated thread deleted] > > Vitja Makarov, 12.12.2010 14:51: >> Cython-tests are now broken... > > Yep, the new class scope for the conjugate changes broke three tests. > Unless it's an easy fix, I'd vote for reverting this post-RC change. Oops. I'll fix that. At this point everything pushed to cython-devel is post-release, unless something really serious comes up. - Robert From vitja.makarov at gmail.com Mon Dec 13 18:32:41 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 13 Dec 2010 20:32:41 +0300 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> <4D04F3EE.30001@behnel.de> Message-ID: 2010/12/13 Robert Bradshaw : > On Sun, Dec 12, 2010 at 8:10 AM, Stefan Behnel wrote: >> [remainings of a completely unrelated thread deleted] >> >> Vitja Makarov, 12.12.2010 14:51: >>> Cython-tests are now broken... >> >> Yep, the new class scope for the conjugate changes broke three tests. >> Unless it's an easy fix, I'd vote for reverting this post-RC change. > > Oops. I'll fix that. > > At this point everything pushed to cython-devel is post-release, > unless something really serious comes up. > > - Robert What's about trailing spaces? ;) -- vitja. From robertwb at math.washington.edu Mon Dec 13 22:41:26 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 13 Dec 2010 13:41:26 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: References: <201012112245.14866.Arfrever.FTA@gmail.com> <4D04F3EE.30001@behnel.de> Message-ID: On Mon, Dec 13, 2010 at 9:32 AM, Vitja Makarov wrote: > 2010/12/13 Robert Bradshaw : >> On Sun, Dec 12, 2010 at 8:10 AM, Stefan Behnel wrote: >>> [remainings of a completely unrelated thread deleted] >>> >>> Vitja Makarov, 12.12.2010 14:51: >>>> Cython-tests are now broken... >>> >>> Yep, the new class scope for the conjugate changes broke three tests. >>> Unless it's an easy fix, I'd vote for reverting this post-RC change. >> >> Oops. I'll fix that. >> >> At this point everything pushed to cython-devel is post-release, >> unless something really serious comes up. >> >> - Robert > > What's about trailing spaces? ;) OK, fair enough. Done. That's not a big code change though. I haven't yet pulled into the generators branch, but I think git has a nice ignore-trailing-whitespace-conflicts merge option. - Robert From dalcinl at gmail.com Tue Dec 14 00:37:00 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 13 Dec 2010 20:37:00 -0300 Subject: [Cython] failure with MSVC Message-ID: builtin_type_inheritance_T608.c(371) : error C2016: C requires that a struct or union has at least one member builtin_type_inheritance_T608.c(384) : error C2016: C requires that a struct or union has at least one member builtin_type_inheritance_T608.c(397) : error C2016: C requires that a struct or union has at least one member I'm not sure what's the best way to fix this... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Tue Dec 14 00:46:03 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 13 Dec 2010 20:46:03 -0300 Subject: [Cython] failing test: py_unicode_type Message-ID: No clue about what's going on here... $ python runtests.py py_unicode_type Python 2.6.4 (r264:75706, Jun 4 2010, 18:20:16) [GCC 4.4.4 20100503 (Red Hat 4.4.4-2)] Running tests against Cython 0.14.rc0 py_unicode_type.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation. py_unicode_type.cpp:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation. ====================================================================== ERROR: compiling (c) and running py_unicode_type ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 451, in compile self.run_distutils(test_directory, module, workdir, incdir) File "runtests.py", line 411, in run_distutils build_extension.run() File "/usr/lib/python2.6/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/usr/lib/python2.6/distutils/command/build_ext.py", line 449, in build_extensions self.build_extension(ext) File "runtests.py", line 95, in build_extension _build_ext.build_extension(self, ext) File "/usr/lib/python2.6/distutils/command/build_ext.py", line 499, in build_extension depends=ext.depends) File "/usr/lib/python2.6/distutils/ccompiler.py", line 697, in compile self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) File "/usr/lib/python2.6/distutils/unixccompiler.py", line 190, in _compile raise CompileError, msg CompileError: command 'gcc' failed with exit status 1 ====================================================================== ERROR: compiling (cpp) and running py_unicode_type ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 451, in compile self.run_distutils(test_directory, module, workdir, incdir) File "runtests.py", line 411, in run_distutils build_extension.run() File "/usr/lib/python2.6/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/usr/lib/python2.6/distutils/command/build_ext.py", line 449, in build_extensions self.build_extension(ext) File "runtests.py", line 95, in build_extension _build_ext.build_extension(self, ext) File "/usr/lib/python2.6/distutils/command/build_ext.py", line 499, in build_extension depends=ext.depends) File "/usr/lib/python2.6/distutils/ccompiler.py", line 697, in compile self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) File "/usr/lib/python2.6/distutils/unixccompiler.py", line 190, in _compile raise CompileError, msg CompileError: command 'gcc' failed with exit status 1 ---------------------------------------------------------------------- Ran 3 tests in 0.851s FAILED (errors=2) ALL DONE -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Tue Dec 14 06:41:20 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 13 Dec 2010 21:41:20 -0800 Subject: [Cython] Sage broken Message-ID: It appears that something broke circular cimports in Cython, and now Sage won't start up (as of https://sage.math.washington.edu:8091/hudson/job/sage-tests/388/ ) Unfortunately, the tests reported 0 failures for not starting up (fixed now), so it looked like all was well. It looks like the offending patch is http://hg.cython.org/cython-devel/rev/6c79b26286ce (or somewhere close by). Any ideas? I'm looking into this now. - Robert From robertwb at math.washington.edu Tue Dec 14 06:45:38 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 13 Dec 2010 21:45:38 -0800 Subject: [Cython] Sage broken In-Reply-To: References: Message-ID: On Mon, Dec 13, 2010 at 9:41 PM, Robert Bradshaw wrote: > It appears that something broke circular cimports in Cython, and now > Sage won't start up (as of > https://sage.math.washington.edu:8091/hudson/job/sage-tests/388/ ) > Unfortunately, the tests reported 0 failures for not starting up > (fixed now), so it looked like all was well. > > It looks like the offending patch is > http://hg.cython.org/cython-devel/rev/6c79b26286ce (or somewhere close > by). Any ideas? I'm looking into this now. Specifically, it's somewhere between http://hg.cython.org/cython-devel/rev/06a35b532a49 and http://hg.cython.org/cython-devel/rev/c48f814a8481 - Robert From robertwb at math.washington.edu Tue Dec 14 08:32:11 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 13 Dec 2010 23:32:11 -0800 Subject: [Cython] Sage broken In-Reply-To: References: Message-ID: On Mon, Dec 13, 2010 at 9:45 PM, Robert Bradshaw wrote: > On Mon, Dec 13, 2010 at 9:41 PM, Robert Bradshaw > wrote: >> It appears that something broke circular cimports in Cython, and now >> Sage won't start up (as of >> https://sage.math.washington.edu:8091/hudson/job/sage-tests/388/ ) >> Unfortunately, the tests reported 0 failures for not starting up >> (fixed now), so it looked like all was well. >> >> It looks like the offending patch is >> http://hg.cython.org/cython-devel/rev/6c79b26286ce (or somewhere close >> by). Any ideas? I'm looking into this now. > > Specifically, it's somewhere between > http://hg.cython.org/cython-devel/rev/06a35b532a49 and > http://hg.cython.org/cython-devel/rev/c48f814a8481 FYI, I've figured out what the problem was. With the new function re-declaration code, def functions are created and added to the module as it is executed, rather than initialized as part of the module creation from the method_def struct. While this is arguably the more correct behavior, it broke Sage which was depending on the fact that functions were defined before type imports, and manifested itself because the import was circular. I've added a --disable-function-redefinition option for legacy code like this (which should in the long run get cleaned up, but I don't think we should break outright break it now). Of course, if set, this disables module-level lambdas, etc., but none of those are in use for old code. I'm not sure exactly what to do about this in the long term, but I'm wondering (but not sure) if we should have import, cimport, cdef class, etc. all happen as if they were executed in sequence too, especially in the face of circular imports. - Robert From stefan_ml at behnel.de Mon Dec 13 22:41:31 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 13 Dec 2010 22:41:31 +0100 Subject: [Cython] General generator expressions In-Reply-To: <4D05B116.2020900@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> Message-ID: <4D06930B.2090205@behnel.de> Stefan Behnel, 13.12.2010 06:37: > Vitja Makarov, 12.12.2010 20:23: >> What do you think about GeneratorWrapperNode? > > It seems ok. I'll give it a deeper look ASAP. Actually, after looking through it, I think it's an ugly hack. It tries to hide the fact that the DefNode does the wrong thing for the two faces of generators, and that the current implementation does not build a separate node at transform time that does the right thing instead. IMHO, two nodes are needed. A CFuncDefNode (or a subtype) that writes out the generator C function with the right signature and the original body, and the original DefNode (with the original signature) that has its body replaced by the necessary generator setup code. That's basically why the original CEP proposed to split the DefNode implementation *before* going for generators. Would have made this easier. Stefan From stefan_ml at behnel.de Tue Dec 14 09:10:49 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 14 Dec 2010 09:10:49 +0100 Subject: [Cython] failing test: py_unicode_type In-Reply-To: References: Message-ID: <4D072689.1040807@behnel.de> Lisandro Dalcin, 14.12.2010 00:46: > No clue about what's going on here... > > $ python runtests.py py_unicode_type > Python 2.6.4 (r264:75706, Jun 4 2010, 18:20:16) > [GCC 4.4.4 20100503 (Red Hat 4.4.4-2)] > > Running tests against Cython 0.14.rc0 > > py_unicode_type.c:1:2: error: #error Do not use this file, it is the > result of a failed Cython compilation. That was one of the three broken tests I meantioned earlier. Seems to be fixed now. Stefan From robertwb at math.washington.edu Tue Dec 14 09:15:59 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 14 Dec 2010 00:15:59 -0800 Subject: [Cython] failing test: py_unicode_type In-Reply-To: <4D072689.1040807@behnel.de> References: <4D072689.1040807@behnel.de> Message-ID: On Tue, Dec 14, 2010 at 12:10 AM, Stefan Behnel wrote: > Lisandro Dalcin, 14.12.2010 00:46: >> No clue about what's going on here... >> >> $ python runtests.py py_unicode_type >> Python 2.6.4 (r264:75706, Jun ?4 2010, 18:20:16) >> [GCC 4.4.4 20100503 (Red Hat 4.4.4-2)] >> >> Running tests against Cython 0.14.rc0 >> >> py_unicode_type.c:1:2: error: #error Do not use this file, it is the >> result of a failed Cython compilation. > > That was one of the three broken tests I meantioned earlier. Seems to be > fixed now. Yep, I fixed that. It was a missing None check that was getting triggered. - Robert From stefan_ml at behnel.de Tue Dec 14 09:43:42 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 14 Dec 2010 09:43:42 +0100 Subject: [Cython] failure with MSVC In-Reply-To: References: Message-ID: <4D072E3E.4000200@behnel.de> Lisandro Dalcin, 14.12.2010 00:37: > builtin_type_inheritance_T608.c(371) : error C2016: C requires that a > struct or union has at least one member > builtin_type_inheritance_T608.c(384) : error C2016: C requires that a > struct or union has at least one member > builtin_type_inheritance_T608.c(397) : error C2016: C requires that a > struct or union has at least one member > > I'm not sure what's the best way to fix this... That must seriously be the first time MSVC actually discovered a bug. ;-) The problem is that the types in this test inherit from builtin types that have builtin methods declared on them. These methods are considered a reason to generate a vtable for the type, even though the type itself does not define any methods, and the builtin methods do not need a vtable. I'll check if this can be special cased somehow. Stefan From stefan_ml at behnel.de Tue Dec 14 11:03:29 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 14 Dec 2010 11:03:29 +0100 Subject: [Cython] failure with MSVC In-Reply-To: <4D072E3E.4000200@behnel.de> References: <4D072E3E.4000200@behnel.de> Message-ID: <4D0740F1.4080001@behnel.de> Stefan Behnel, 14.12.2010 09:43: > Lisandro Dalcin, 14.12.2010 00:37: >> builtin_type_inheritance_T608.c(371) : error C2016: C requires that a >> struct or union has at least one member >> builtin_type_inheritance_T608.c(384) : error C2016: C requires that a >> struct or union has at least one member >> builtin_type_inheritance_T608.c(397) : error C2016: C requires that a >> struct or union has at least one member >> >> I'm not sure what's the best way to fix this... > > That must seriously be the first time MSVC actually discovered a bug. ;-) > > The problem is that the types in this test inherit from builtin types that > have builtin methods declared on them. These methods are considered a > reason to generate a vtable for the type, even though the type itself does > not define any methods, and the builtin methods do not need a vtable. > > I'll check if this can be special cased somehow. Fix is up in cython-devel: http://hg.cython.org/cython-devel/rev/5a033978dd11 Stefan From dalcinl at gmail.com Tue Dec 14 13:46:52 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 14 Dec 2010 09:46:52 -0300 Subject: [Cython] failure with MSVC In-Reply-To: <4D0740F1.4080001@behnel.de> References: <4D072E3E.4000200@behnel.de> <4D0740F1.4080001@behnel.de> Message-ID: On 14 December 2010 07:03, Stefan Behnel wrote: > Stefan Behnel, 14.12.2010 09:43: >> >> That must seriously be the first time MSVC actually discovered a bug. ;-) >> Indeed! > > Fix is up in cython-devel: > > http://hg.cython.org/cython-devel/rev/5a033978dd11 > Many Thanks! I'll have a chance to test it at night. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From vitja.makarov at gmail.com Tue Dec 14 14:45:24 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 14 Dec 2010 16:45:24 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D06930B.2090205@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> Message-ID: 2010/12/14 Stefan Behnel : > Stefan Behnel, 13.12.2010 06:37: >> Vitja Makarov, 12.12.2010 20:23: >>> What do you think about GeneratorWrapperNode? >> >> It seems ok. I'll give it a deeper look ASAP. > > Actually, after looking through it, I think it's an ugly hack. It tries to > hide the fact that the DefNode does the wrong thing for the two faces of > generators, and that the current implementation does not build a separate > node at transform time that does the right thing instead. IMHO, two nodes > are needed. A CFuncDefNode (or a subtype) that writes out the generator C > function with the right signature and the original body, and the original > DefNode (with the original signature) that has its body replaced by the > necessary generator setup code. That's basically why the original CEP > proposed to split the DefNode implementation *before* going for generators. > Would have made this easier. > Are you sure that CFuncDefNode for generator body is a good idea? Nested function and classes, lambda, generators are not supported inside CFuncDefNode. Btw. It would be nice to make DefNode base for GeneratorNode, all the generator creation stuff could be moved into generate_function_body() and make generator body childern of GeneratorNode. I'm thinking of genexprs they are very similar to lambdas and could be implemented reusing lambda stuff. -- vitja. From stefan_ml at behnel.de Tue Dec 14 15:18:13 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 14 Dec 2010 15:18:13 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> Message-ID: <4D077CA5.4010103@behnel.de> Vitja Makarov, 14.12.2010 14:45: > 2010/12/14 Stefan Behnel: >> Stefan Behnel, 13.12.2010 06:37: >>> Vitja Makarov, 12.12.2010 20:23: >>>> What do you think about GeneratorWrapperNode? >>> >>> It seems ok. I'll give it a deeper look ASAP. >> >> Actually, after looking through it, I think it's an ugly hack. It tries to >> hide the fact that the DefNode does the wrong thing for the two faces of >> generators, and that the current implementation does not build a separate >> node at transform time that does the right thing instead. IMHO, two nodes >> are needed. A CFuncDefNode (or a subtype) that writes out the generator C >> function with the right signature and the original body, and the original >> DefNode (with the original signature) that has its body replaced by the >> necessary generator setup code. That's basically why the original CEP >> proposed to split the DefNode implementation *before* going for generators. >> Would have made this easier. > > Are you sure that CFuncDefNode for generator body is a good idea? > > Nested function and classes, lambda, generators are not supported > inside CFuncDefNode. Hmm, right, that's a missing feature that could become a problem. Second try. Create a new subtype of DefNode that nails the signature to METH_O and additionally implements all the generator specific stuff, potentially through sensible hooks in DefNode's code generation. > Btw. It would be nice to make DefNode base for GeneratorNode, all the > generator creation stuff could be moved into generate_function_body() > and make generator body childern of GeneratorNode. I'm having trouble parsing the last part of this sentence, but it seems we are agreeing that a subtype would be a good idea. > I'm thinking of genexprs they are very similar to lambdas and could be > implemented reusing lambda stuff. Lambda is basically implemented as a little transform that creates a DefNode. I agree, it should be enough to just wrap generator expressions in a new DefNode and let the normal generator machinery do its work. However, note that this needs to interact with the EarlyReplaceBuiltinCalls transform in some way, which inlines generator expressions if possible. Looks like it's going to be tricky to arrange the related transformations according to the interdependencies, e.g. to the AnalyseDeclarationsTransform. I guess subtyping DefNode in two levels (DefNode<-GeneratorNode<-GeneratorExpressionNode) could be a step to fix this, as the inlining transform could then be changed to properly unpack that node instead of the current plain expression. Stefan From vitja.makarov at gmail.com Tue Dec 14 15:49:57 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 14 Dec 2010 17:49:57 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D077CA5.4010103@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> <4D077CA5.4010103@behnel.de> Message-ID: 2010/12/14 Stefan Behnel : > Vitja Makarov, 14.12.2010 14:45: >> 2010/12/14 Stefan Behnel: >>> Stefan Behnel, 13.12.2010 06:37: >>>> Vitja Makarov, 12.12.2010 20:23: >>>>> What do you think about GeneratorWrapperNode? >>>> >>>> It seems ok. I'll give it a deeper look ASAP. >>> >>> Actually, after looking through it, I think it's an ugly hack. It tries to >>> hide the fact that the DefNode does the wrong thing for the two faces of >>> generators, and that the current implementation does not build a separate >>> node at transform time that does the right thing instead. IMHO, two nodes >>> are needed. A CFuncDefNode (or a subtype) that writes out the generator C >>> function with the right signature and the original body, and the original >>> DefNode (with the original signature) that has its body replaced by the >>> necessary generator setup code. That's basically why the original CEP >>> proposed to split the DefNode implementation *before* going for generators. >>> Would have made this easier. >> >> Are you sure that CFuncDefNode for generator body is a good idea? >> >> Nested function and classes, lambda, generators are not supported >> inside CFuncDefNode. > > Hmm, right, that's a missing feature that could become a problem. > > Second try. Create a new subtype of DefNode that nails the signature to > METH_O and additionally implements all the generator specific stuff, > potentially through sensible hooks in DefNode's code generation. > What is the best place to create node for generator body? In analyse_...? > >> Btw. It would be nice to make DefNode base for GeneratorNode, all the >> generator creation stuff could be moved into generate_function_body() >> and make generator body childern of GeneratorNode. > > I'm having trouble parsing the last part of this sentence, but it seems we > are agreeing that a subtype would be a good idea. > oops -- vitja. From stefan_ml at behnel.de Tue Dec 14 16:02:47 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 14 Dec 2010 16:02:47 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> <4D077CA5.4010103@behnel.de> Message-ID: <4D078717.4050105@behnel.de> Vitja Makarov, 14.12.2010 15:49: > 2010/12/14 Stefan Behnel: >> Vitja Makarov, 14.12.2010 14:45: >>> 2010/12/14 Stefan Behnel: >>>> Stefan Behnel, 13.12.2010 06:37: >>>>> Vitja Makarov, 12.12.2010 20:23: >>>>>> What do you think about GeneratorWrapperNode? >>>>> >>>>> It seems ok. I'll give it a deeper look ASAP. >>>> >>>> Actually, after looking through it, I think it's an ugly hack. It tries to >>>> hide the fact that the DefNode does the wrong thing for the two faces of >>>> generators, and that the current implementation does not build a separate >>>> node at transform time that does the right thing instead. IMHO, two nodes >>>> are needed. A CFuncDefNode (or a subtype) that writes out the generator C >>>> function with the right signature and the original body, and the original >>>> DefNode (with the original signature) that has its body replaced by the >>>> necessary generator setup code. That's basically why the original CEP >>>> proposed to split the DefNode implementation *before* going for generators. >>>> Would have made this easier. >>> >>> Are you sure that CFuncDefNode for generator body is a good idea? >>> >>> Nested function and classes, lambda, generators are not supported >>> inside CFuncDefNode. >> >> Hmm, right, that's a missing feature that could become a problem. >> >> Second try. Create a new subtype of DefNode that nails the signature to >> METH_O and additionally implements all the generator specific stuff, >> potentially through sensible hooks in DefNode's code generation. >> > > What is the best place to create node for generator body? "Create" as in "implement"? In Nodes.py. "Create" as in "instantiate"? Depends. The canonical answer is "in a transform", but sometimes it's easier to do it inside of a node. If the latter, the canonical answer would become analyse_types(). If that's too late, well, back to a transform. :) Stefan From vitja.makarov at gmail.com Tue Dec 14 16:35:21 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 14 Dec 2010 18:35:21 +0300 Subject: [Cython] General generator expressions In-Reply-To: <4D078717.4050105@behnel.de> References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> <4D077CA5.4010103@behnel.de> <4D078717.4050105@behnel.de> Message-ID: 2010/12/14 Stefan Behnel : > Vitja Makarov, 14.12.2010 15:49: >> 2010/12/14 Stefan Behnel: >>> Vitja Makarov, 14.12.2010 14:45: >>>> 2010/12/14 Stefan Behnel: >>>>> Stefan Behnel, 13.12.2010 06:37: >>>>>> Vitja Makarov, 12.12.2010 20:23: >>>>>>> What do you think about GeneratorWrapperNode? >>>>>> >>>>>> It seems ok. I'll give it a deeper look ASAP. >>>>> >>>>> Actually, after looking through it, I think it's an ugly hack. It tries to >>>>> hide the fact that the DefNode does the wrong thing for the two faces of >>>>> generators, and that the current implementation does not build a separate >>>>> node at transform time that does the right thing instead. IMHO, two nodes >>>>> are needed. A CFuncDefNode (or a subtype) that writes out the generator C >>>>> function with the right signature and the original body, and the original >>>>> DefNode (with the original signature) that has its body replaced by the >>>>> necessary generator setup code. That's basically why the original CEP >>>>> proposed to split the DefNode implementation *before* going for generators. >>>>> Would have made this easier. >>>> >>>> Are you sure that CFuncDefNode for generator body is a good idea? >>>> >>>> Nested function and classes, lambda, generators are not supported >>>> inside CFuncDefNode. >>> >>> Hmm, right, that's a missing feature that could become a problem. >>> >>> Second try. Create a new subtype of DefNode that nails the signature to >>> METH_O and additionally implements all the generator specific stuff, >>> potentially through sensible hooks in DefNode's code generation. >>> >> >> What is the best place to create node for generator body? > > "Create" as in "implement"? In Nodes.py. > > "Create" as in "instantiate"? Depends. The canonical answer is "in a > transform", but sometimes it's easier to do it inside of a node. If the > latter, the canonical answer would become analyse_types(). If that's too > late, well, back to a transform. :) > I think analyse_types() should be ok, and it wouldn't break things -- vitja. From robertwb at math.washington.edu Tue Dec 14 22:40:05 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 14 Dec 2010 13:40:05 -0800 Subject: [Cython] ANN: Cython 0.14 released Message-ID: The Cython project is happy to announce the release of Cython 0.14. This release fixes several bugs in the previous 0.13 release, improved debugging and build support, and heavily improves Python compatibility in terms of supported features. It can be downloaded from http://cython.org/ or http://pypi.python.org/pypi/Cython/0.14 Release notes: http://wiki.cython.org/ReleaseNotes-0.14 == New Features == * Python classes can now be nested and receive a proper closure at definition time. * Redefinition is supported for Python functions, even within the same scope. * Lambda expressions are supported in class bodies and at the module level. * Metaclasses are supported for Python classes, both in Python 2 and Python 3 syntax. The Python 3 syntax (using a keyword argument in the type declaration) is preferred and optimised at compile time. * "final" extension classes prevent inheritance in Python space. This feature is available through the new "cython.final" decorator. In the future, these classes may receive further optimisations. * "internal" extension classes do not show up in the module dictionary. This feature is available through the new "cython.internal" decorator. * Extension type inheritance from builtin types, such as "cdef class MyUnicode(unicode)", now works without further external type redeclarations (which are also strongly discouraged now and continue to issue a warning). * GDB support. http://docs.cython.org/src/userguide/debugging.html * A new build system with support for inline distutils directives, correct dependency tracking, and parallel compilation. http://wiki.cython.org/enhancements/distutils_preprocessing * Support for dynamic compilation at runtime via the new cython.inline function and cython.compile decorator. http://wiki.cython.org/enhancements/inline == General improvements and bug fixes == * In parallel assignments, the right side was evaluated in reverse order in 0.13. This could result in errors if it had side effects (e.g. function calls). * In some cases, methods of builtin types would raise a SystemError instead of an AttributeError when called on None. * Constant tuples are now cached over the lifetime of an extension module, just like CPython does. Constant argument tuples of Python function calls are also cached. * Closures have tightened to include exactly the names used in the inner functions and classes. Previously, they held the complete locals of the defining function. * "nogil" blocks are supported when compiling pure Python code by writing "with cython.nogil". * The builtin "next()" function in Python 2.6 and later is now implemented internally and therefore available in all Python versions. This makes it the preferred and portable way of manually advancing an iterator. * In addition to the previously supported inlined generator expressions in 0.13, "sorted(genexpr)" can now be used as well. Typing issues were fixed in "sum(genexpr)" that could lead to invalid C code being generated. Other known issues with inlined generator expressions were also fixed that make upgrading to 0.14 a strong recommendation for code that uses them. Note that general generators and generator expressions continue to be not supported. * Iterating over arbitrary pointer types is now supported, as is an optimized version of the in operator, e.g. x in ptr[a:b]. * Inplace arithmetic operators now respect the cdivision directive and are supported for complex types. == Incompatible changes == * Typing a variable as type "complex" previously gave it the Python object type. It now uses the appropriate C/C++ {{{double complex}}} type. A side-effect is that assignments and typed function parameters now accept anything that Python can coerce to a complex, including integers and floats, and not only complex instances. * Large integer literals pass through the compiler in a safer way. To prevent truncation in C code, non 32-bit literals are turned into Python objects if not used in a C context. This context can either be given by a clear C literal suffix such as "UL" or "LL" (or "L" in Python 3 code), or it can be an assignment to a typed variable or a typed function argument, in which case it is up to the user to take care of a sufficiently large value space of the target. * Python functions are declared in the order they appear in the file, rather than all being created at module creation time. This is consistent with Python and needed to support, for example, conditional or repeated declarations of functions. In the face of circular imports this may cause code to break, so a new {{{--disable-function-redefinition}}} flag was added to revert to the old behavior. This flag will be removed in a future release, so should only be used as a stopgap until old code can be fixed. == Contributors == Many people contributed to this release, including: * Haoyu Bai * Stefan Behnel * Robert Bradshaw * Ondrej Certik * Lisandro Dalcin * Mark Florisson * Eric Huss * Vitja Makarov * Corbin Simpson * Kurt Smith From robertwb at math.washington.edu Wed Dec 15 00:00:08 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 14 Dec 2010 15:00:08 -0800 Subject: [Cython] Cython 0.14 release candidate In-Reply-To: <4D046186.2060501@student.matnat.uio.no> References: <201012112245.14866.Arfrever.FTA@gmail.com> <4D046186.2060501@student.matnat.uio.no> Message-ID: On Sat, Dec 11, 2010 at 9:45 PM, Dag Sverre Seljebotn wrote: > On 12/12/2010 05:29 AM, Robert Bradshaw wrote: >> On Sat, Dec 11, 2010 at 7:07 PM, Robert Bradshaw >> ?wrote: >> >>> On Sat, Dec 11, 2010 at 5:52 PM, Robert Kern ?wrote: >>> >>>> On 2010-12-11 17:09 , Robert Bradshaw wrote: >>>> >>>>> On Sat, Dec 11, 2010 at 1:44 PM, Arfrever Frehtes Taifersar Arahesis >>>>> ? ?wrote: >>>>> >>>>>> 2010-12-11 12:01:11 Robert Bradshaw napisa?(a): >>>>>> >>>>>>> Release candidate up at >>>>>>> http://cython.org/release/Cython-0.14.rc0.tar.gz . Unless something >>>>>>> bad is discovered, this will be the release sometime early next week. >>>>>>> >>>>>> Some tests fail. >>>>>> >>>>>> Test failures with Python 2.6 (the same tests fail with Python 2.7): >>>>>> >>>>> [...] >>>>> >>>>> Not sue what's up with all these numpy errors, but many (not all) of >>>>> them look like a change in formatting of error messages. The basic >>>>> functionality still seems to be passing--does anyone know if this is a >>>>> regression? >>>>> >>>> I think so. I think these are real regressions: >>>> >>>> ? ?ValueError: Does not understand character buffer dtype format string (':') >>>> >>>> ? From regular Python, I've tried a few of the dtype() calls that you make, and >>>> they seem to work fine. I don't see a problem in the generated code, though. >>>> >>> They all work for me too, but maybe that's because I'm using an older >>> version of NumPy (the one that ships with Sage). Has anyone tried >>> Numpy 1.5 with Cython 0.13? >>> >> OK, I just install Numpy 1.5.1, and all the tests pass for me with >> both 0.13 and 0.14.rc0 (Python 2.6, OS X). Anyone else able to >> reproduce these errors? Arfrever, can you try Cython 0.13 in your >> setup? >> > > The thing that has happened here is that the new NumPy supports the PEP > 3118 protocol, and so the code in numpy.pxd is not used, but instead the > other code. Probably new tests should be written, perhaps some changes > must be done to buffer format string parsing, and so on. But it is > unlikely to be a regression in Cython, nothing changed here w.r.t > Cython ?0.13.x. I've created http://trac.cython.org/cython_trac/ticket/630 , hopefully someone will look into this. - Robert From arfrever.fta at gmail.com Wed Dec 15 13:02:31 2010 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Wed, 15 Dec 2010 13:02:31 +0100 Subject: [Cython] ANN: Cython 0.14 released In-Reply-To: References: Message-ID: <201012151302.59250.Arfrever.FTA@gmail.com> The following test failures still occur with Python 3.1: ====================================================================== ERROR: test_globals (Cython.Build.Tests.TestInline.TestInline) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 32, in test_globals self.assertEquals(inline("return global_value + 1", **test_kwds), global_value + 1) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 111, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 57, in unbound_symbols if not tree.scope.lookup(name) and not hasattr(__builtin__, name): NameError: global name '__builtin__' is not defined ====================================================================== ERROR: test_locals (Cython.Build.Tests.TestInline.TestInline) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 29, in test_locals self.assertEquals(inline("return a+b", **test_kwds), 3) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 111, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 57, in unbound_symbols if not tree.scope.lookup(name) and not hasattr(__builtin__, name): NameError: global name '__builtin__' is not defined ====================================================================== ERROR: test_numpy (Cython.Build.Tests.TestInline.TestInline) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 41, in test_numpy self.assertEquals(inline("return a[0,0]", a=a, **test_kwds), 10.0) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 111, in cython_inline for symbol in unbound_symbols(code): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 57, in unbound_symbols if not tree.scope.lookup(name) and not hasattr(__builtin__, name): NameError: global name '__builtin__' is not defined ====================================================================== ERROR: test_simple (Cython.Build.Tests.TestInline.TestInline) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 18, in test_simple self.assertEquals(inline("return 1+2", **test_kwds), 3) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 128, in cython_inline module_name = "_cython_inline_" + hashlib.md5(str(key)).hexdigest() TypeError: Unicode-objects must be encoded before hashing ====================================================================== ERROR: test_types (Cython.Build.Tests.TestInline.TestInline) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Tests/TestInline.py", line 24, in test_types """, a=1.0, b=[], **test_kwds), ('double', 'list object')) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Shadow.py", line 16, in inline return cython_inline(f, *args, **kwds) File "/var/tmp/portage/dev-python/cython-0.14/work/Cython-0.14/BUILD/Cy3/Cython/Build/Inline.py", line 128, in cython_inline module_name = "_cython_inline_" + hashlib.md5(str(key)).hexdigest() TypeError: Unicode-objects must be encoded before hashing ====================================================================== ERROR: compiling (c) and running numpy_bufacc_T155 ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != "10:9: 'ndarray' is not a type identifier" ====================================================================== ERROR: compiling (cpp) and running numpy_bufacc_T155 ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != "10:9: 'ndarray' is not a type identifier" ====================================================================== ERROR: compiling (c) and running numpy_cimport ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ====================================================================== ERROR: compiling (cpp) and running numpy_cimport ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ====================================================================== ERROR: compiling (c) and running numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ====================================================================== ERROR: compiling (cpp) and running numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 464, in run self.runCompileTest() File "runtests.py", line 299, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 441, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '173:49: "mode" is not a buffer option' ---------------------------------------------------------------------- Ran 4769 tests in 1415.911s FAILED (errors=11) === Expected errors: === === Got errors: === 10:9: 'ndarray' is not a type identifier 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 10:9: 'ndarray' is not a type identifier 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option 253:18: 'ndarray' is not a type identifier 258:22: 'ndarray' is not a type identifier 265:18: 'ndarray' is not a type identifier 270:18: 'ndarray' is not a type identifier 276:14: 'ndarray' is not a type identifier 277:15: 'ndarray' is not a type identifier 278:15: 'ndarray' is not a type identifier 279:16: 'ndarray' is not a type identifier 280:13: 'ndarray' is not a type identifier 281:14: 'ndarray' is not a type identifier 282:14: 'ndarray' is not a type identifier 283:15: 'ndarray' is not a type identifier 284:18: 'ndarray' is not a type identifier 285:19: 'ndarray' is not a type identifier 287:15: 'ndarray' is not a type identifier 288:16: 'ndarray' is not a type identifier 289:20: 'ndarray' is not a type identifier 291:16: 'ndarray' is not a type identifier 292:17: 'ndarray' is not a type identifier 293:21: 'ndarray' is not a type identifier 295:23: 'ndarray' is not a type identifier 299:24: 'ndarray' is not a type identifier 303:28: 'ndarray' is not a type identifier 309:16: 'ndarray' is not a type identifier 315:15: 'ndarray' is not a type identifier 316:16: 'ndarray' is not a type identifier 317:17: 'ndarray' is not a type identifier 318:18: 'ndarray' is not a type identifier 319:22: 'ndarray' is not a type identifier 320:16: 'ndarray' is not a type identifier 321:17: 'ndarray' is not a type identifier 324:17: 'ndarray' is not a type identifier 325:19: 'ndarray' is not a type identifier 387:9: 'ndarray' is not a type identifier 393:9: 'ndarray' is not a type identifier 403:22: 'ndarray' is not a type identifier 408:24: 'ndarray' is not a type identifier 414:9: 'complex64_t' is not a type identifier 415:9: 'complex128_t' is not a type identifier 424:4: 'float64_t' is not a type identifier 427:9: 'ndarray' is not a type identifier === Expected errors: === === Got errors: === 173:49: "mode" is not a buffer option 253:18: 'ndarray' is not a type identifier 258:22: 'ndarray' is not a type identifier 265:18: 'ndarray' is not a type identifier 270:18: 'ndarray' is not a type identifier 276:14: 'ndarray' is not a type identifier 277:15: 'ndarray' is not a type identifier 278:15: 'ndarray' is not a type identifier 279:16: 'ndarray' is not a type identifier 280:13: 'ndarray' is not a type identifier 281:14: 'ndarray' is not a type identifier 282:14: 'ndarray' is not a type identifier 283:15: 'ndarray' is not a type identifier 284:18: 'ndarray' is not a type identifier 285:19: 'ndarray' is not a type identifier 287:15: 'ndarray' is not a type identifier 288:16: 'ndarray' is not a type identifier 289:20: 'ndarray' is not a type identifier 291:16: 'ndarray' is not a type identifier 292:17: 'ndarray' is not a type identifier 293:21: 'ndarray' is not a type identifier 295:23: 'ndarray' is not a type identifier 299:24: 'ndarray' is not a type identifier 303:28: 'ndarray' is not a type identifier 309:16: 'ndarray' is not a type identifier 315:15: 'ndarray' is not a type identifier 316:16: 'ndarray' is not a type identifier 317:17: 'ndarray' is not a type identifier 318:18: 'ndarray' is not a type identifier 319:22: 'ndarray' is not a type identifier 320:16: 'ndarray' is not a type identifier 321:17: 'ndarray' is not a type identifier 324:17: 'ndarray' is not a type identifier 325:19: 'ndarray' is not a type identifier 387:9: 'ndarray' is not a type identifier 393:9: 'ndarray' is not a type identifier 403:22: 'ndarray' is not a type identifier 408:24: 'ndarray' is not a type identifier 414:9: 'complex64_t' is not a type identifier 415:9: 'complex128_t' is not a type identifier 424:4: 'float64_t' is not a type identifier 427:9: 'ndarray' is not a type identifier ALL DONE -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20101215/03b2b35f/attachment.pgp From dalcinl at gmail.com Wed Dec 15 20:38:33 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 15 Dec 2010 16:38:33 -0300 Subject: [Cython] ANN: Cython 0.14 released In-Reply-To: <201012151302.59250.Arfrever.FTA@gmail.com> References: <201012151302.59250.Arfrever.FTA@gmail.com> Message-ID: On 15 December 2010 09:02, Arfrever Frehtes Taifersar Arahesis wrote: > The following test failures still occur with Python 3.1: > > ====================================================================== > ERROR: test_globals (Cython.Build.Tests.TestInline.TestInline) > NameError: global name '__builtin__' is not defined > > ====================================================================== > ERROR: test_locals (Cython.Build.Tests.TestInline.TestInline) > ---------------------------------------------------------------------- > TypeError: Unicode-objects must be encoded before hashing > These two should be fixed here: http://hg.cython.org/cython-devel/rev/b1e030b07cd7 > > ====================================================================== > ERROR: compiling (c) and running numpy_bufacc_T155 > ---------------------------------------------------------------------- > FAILED (errors=11) > > > === Got errors: === > 10:9: 'ndarray' is not a type identifier > 173:49: "mode" is not a buffer option > The problem here is that the dictionary keywords of __cythonbufferdefaults__ are parsed as BytesLiteral, there is code with "if not mode in buffer_options" at Buffer.py that fails because "mode" is 'bytes' and "buffer_options" do have 'str' keys... Stefan, could you take a look at this? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From markflorisson88 at gmail.com Wed Dec 15 21:50:48 2010 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 15 Dec 2010 21:50:48 +0100 Subject: [Cython] Cython debugging wiki Message-ID: With the addition of cygdb ( http://docs.cython.org/src/userguide/debugging.html), should we now remove the debugging wiki (http://wiki.cython.org/DebuggingTechniques)? It seems kind of obsolete now. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101215/0093a651/attachment.htm From stefan_ml at behnel.de Thu Dec 16 06:03:13 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 16 Dec 2010 06:03:13 +0100 Subject: [Cython] ANN: Cython 0.14 released In-Reply-To: References: <201012151302.59250.Arfrever.FTA@gmail.com> Message-ID: <4D099D91.2010408@behnel.de> Lisandro Dalcin, 15.12.2010 20:38: > On 15 December 2010 09:02, Arfrever Frehtes Taifersar Arahesis >> ====================================================================== >> ERROR: compiling (c) and running numpy_bufacc_T155 >> ---------------------------------------------------------------------- >> FAILED (errors=11) >> >> === Got errors: === >> 10:9: 'ndarray' is not a type identifier >> 173:49: "mode" is not a buffer option > > The problem here is that the dictionary keywords of > __cythonbufferdefaults__ are parsed as BytesLiteral, there is code > with "if not mode in buffer_options" at Buffer.py that fails because > "mode" is 'bytes' and "buffer_options" do have 'str' keys... > > Stefan, could you take a look at this? Yes, I know. IIRC, NumPy 1.4 *requires* bytes values here, though. I don't have any NumPy's installed, BTW, especially not in a Py3 build. I think the right way to parse the dict keys here is as identifiers. I usually do d = dict(a=1, b=5) to get this behaviour. Cython transforms this to a literal dict with identifier keywords internally. Not sure if this works in a .pxd... We could also override the type of the dict keys as identifiers explicitly when handling __cythonbufferdefaults__. Stefan From ndbecker2 at gmail.com Thu Dec 16 14:38:31 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 16 Dec 2010 08:38:31 -0500 Subject: [Cython] numpy - accel slicing? Message-ID: I noticed in http://docs.cython.org/src/tutorial/numpy.html that slices are not accelerated. To me, this is unfortunate. Is it planned to 'fix' this? I'd like to be able to write code that will accept arbitrary numpy arrays, as all other numpy code does, and that includes slices. From dalcinl at gmail.com Thu Dec 16 17:56:41 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 16 Dec 2010 13:56:41 -0300 Subject: [Cython] ANN: Cython 0.14 released In-Reply-To: <4D099D91.2010408@behnel.de> References: <201012151302.59250.Arfrever.FTA@gmail.com> <4D099D91.2010408@behnel.de> Message-ID: On 16 December 2010 02:03, Stefan Behnel wrote: > Lisandro Dalcin, 15.12.2010 20:38: >> On 15 December 2010 09:02, Arfrever Frehtes Taifersar Arahesis >>> ====================================================================== >>> ERROR: compiling (c) and running numpy_bufacc_T155 >>> ---------------------------------------------------------------------- >>> FAILED (errors=11) >>> >>> === Got errors: === >>> 10:9: 'ndarray' is not a type identifier >>> 173:49: "mode" is not a buffer option >> >> The problem here is that the dictionary keywords of >> __cythonbufferdefaults__ are parsed as BytesLiteral, there is code >> with "if not mode in buffer_options" ?at Buffer.py that fails because >> "mode" is 'bytes' and "buffer_options" do have 'str' keys... >> >> Stefan, could you take a look at this? > > Yes, I know. IIRC, NumPy 1.4 *requires* bytes values here, though. Sorry, what do you mean? > I don't > have any NumPy's installed, BTW, especially not in a Py3 build. > > I think the right way to parse the dict keys here is as identifiers. I > usually do > > ? ? d = dict(a=1, b=5) > > to get this behaviour. Cython transforms this to a literal dict with > identifier keywords internally. Not sure if this works in a .pxd... > Unfortunately, this does not work in a .pxd.. And we still have the issue of string values, like "strided" > We could also override the type of the dict keys as identifiers explicitly > when handling __cythonbufferdefaults__. > I thing this is the best option. And I believe that the type of dict values also needs to be fixed. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dagss at student.matnat.uio.no Thu Dec 16 18:37:49 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 16 Dec 2010 18:37:49 +0100 Subject: [Cython] ANN: Cython 0.14 released In-Reply-To: References: <201012151302.59250.Arfrever.FTA@gmail.com> <4D099D91.2010408@behnel.de> Message-ID: <4D0A4E6D.7040501@student.matnat.uio.no> On 12/16/2010 05:56 PM, Lisandro Dalcin wrote: > On 16 December 2010 02:03, Stefan Behnel wrote: > >> Lisandro Dalcin, 15.12.2010 20:38: >> >>> On 15 December 2010 09:02, Arfrever Frehtes Taifersar Arahesis >>> >>>> ====================================================================== >>>> ERROR: compiling (c) and running numpy_bufacc_T155 >>>> ---------------------------------------------------------------------- >>>> FAILED (errors=11) >>>> >>>> === Got errors: === >>>> 10:9: 'ndarray' is not a type identifier >>>> 173:49: "mode" is not a buffer option >>>> >>> The problem here is that the dictionary keywords of >>> __cythonbufferdefaults__ are parsed as BytesLiteral, there is code >>> with "if not mode in buffer_options" at Buffer.py that fails because >>> "mode" is 'bytes' and "buffer_options" do have 'str' keys... >>> >>> Stefan, could you take a look at this? >>> >> Yes, I know. IIRC, NumPy 1.4 *requires* bytes values here, though. >> > Sorry, what do you mean? > __cythonbufferdefaults__ have no dependency on NumPy (tests/run/bufaccess.pyx exercises it without NumPy). It's purely used as a mechanism to get settings from an arbitrary pxd file to Cython/Compiler/Buffer.py (as a way of getting "custom syntax" through without having to modify the parser). Sorry that I don't have time to fix all the buffer/NumPy issues cropping up myself at the moment... I do still consider them "my" responsibility and hope I'll have more time for this in spring, but not right now... Dag Sverre From dagss at student.matnat.uio.no Thu Dec 16 18:43:37 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 16 Dec 2010 18:43:37 +0100 Subject: [Cython] ANN: Cython 0.14 released In-Reply-To: <4D0A4E6D.7040501@student.matnat.uio.no> References: <201012151302.59250.Arfrever.FTA@gmail.com> <4D099D91.2010408@behnel.de> <4D0A4E6D.7040501@student.matnat.uio.no> Message-ID: <4D0A4FC9.4000602@student.matnat.uio.no> On 12/16/2010 06:37 PM, Dag Sverre Seljebotn wrote: > On 12/16/2010 05:56 PM, Lisandro Dalcin wrote: > >> On 16 December 2010 02:03, Stefan Behnel wrote: >> >> >>> Lisandro Dalcin, 15.12.2010 20:38: >>> >>> >>>> On 15 December 2010 09:02, Arfrever Frehtes Taifersar Arahesis >>>> >>>> >>>>> ====================================================================== >>>>> ERROR: compiling (c) and running numpy_bufacc_T155 >>>>> ---------------------------------------------------------------------- >>>>> FAILED (errors=11) >>>>> >>>>> === Got errors: === >>>>> 10:9: 'ndarray' is not a type identifier >>>>> 173:49: "mode" is not a buffer option >>>>> >>>>> >>>> The problem here is that the dictionary keywords of >>>> __cythonbufferdefaults__ are parsed as BytesLiteral, there is code >>>> with "if not mode in buffer_options" at Buffer.py that fails because >>>> "mode" is 'bytes' and "buffer_options" do have 'str' keys... >>>> >>>> Stefan, could you take a look at this? >>>> >>>> >>> Yes, I know. IIRC, NumPy 1.4 *requires* bytes values here, though. >>> >>> >> Sorry, what do you mean? >> >> > __cythonbufferdefaults__ have no dependency on NumPy > (tests/run/bufaccess.pyx exercises it without NumPy). It's purely used > as a mechanism to get settings from an arbitrary pxd file to > Cython/Compiler/Buffer.py (as a way of getting "custom syntax" through > without having to modify the parser). > To be clear: The if-tests in Buffer.py is the *only* place where __cythonbufferdefaults__ is used. So just turning any BytesLiteral's into unicode by brute force/ASCII decoding should work well. Probably one should handle both BytesLiteral and UnicodeLiteral and convert them both to str. Dag Sverre From robertwb at math.washington.edu Thu Dec 16 19:01:27 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 16 Dec 2010 10:01:27 -0800 Subject: [Cython] Cython debugging wiki In-Reply-To: References: Message-ID: Yes, I'd say so. Perhaps we should replace the entire page with a link to the docs, in case it's being linked to elsewhere. On Wed, Dec 15, 2010 at 12:50 PM, mark florisson wrote: > With the addition of cygdb > (http://docs.cython.org/src/userguide/debugging.html), ?should we now remove > the debugging wiki (http://wiki.cython.org/DebuggingTechniques)? It seems > kind of obsolete now. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > From stefan_ml at behnel.de Thu Dec 16 23:11:40 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 16 Dec 2010 23:11:40 +0100 Subject: [Cython] provide 'builtins' builtin module also for -2 mode In-Reply-To: References: <4D0A8A40.2010102@behnel.de> Message-ID: <4D0A8E9C.8010008@behnel.de> [CC-ing cython-dev] Lisandro Dalcin, 16.12.2010 23:03: > On 16 December 2010 18:53, Stefan Behnel wrote: >> Lisandro Dalcin, 16.12.2010 22:33: >>> >>> Just a few comments about this changeset: >>> >>> changeset: 4244:e0508ac59b35 >>> user: Stefan Behnel >>> date: Thu Dec 16 20:11:29 2010 +0100 >>> summary: provide 'builtins' builtin module in -3 mode >>> >>> >>> IIUC, under -3 we can use both "__builtin__" and "builtins", but under >>> -2 only "__builtin__". Why not to also provide "builtins" under -2 >>> mode? We are already supporting "builtins" in some parts, for example >>> at ModuleNode.py, method "generate_type_import_call()" ... >> >> Well, as long as it actually means that users can import any of the two in >> their code and have it work at runtime in both Py2 and Py3? > > Yes, of course... > >> But that would >> require that Cython changes the user imported module name depending on the C >> compile time environment. I'm not sure that's a good idea. > > Well, the code at generate_type_import_call() work along these lines, > right? I mean, it translates __builtin__ and builtins to the right > name in the C runtime. But that's only when it actually runs in Py3, which is still in "eventual migration" state for many users. As long as you keep your code in a Py2 environment and only "import __builtin__", Cython won't do any magic behind your back. >> The problem I see is that users may have a "builtins" module lying around >> somewhere in their Py2 code base that Cython would misinterpret and wouldn't >> be able to import. "builtins" isn't that a bad name for a module, e.g. in a >> plugin environment... > > Yes, you have a good point. However the import would need to be > absolute for Cython to misinterpret it, right? I would argue that such > code smells... It didn't smell all that much before Py3 renamed the module. Anyway, I'm +0 on changing this. Let's hear what the others think. Stefan From vinjvinj at gmail.com Sat Dec 18 17:38:55 2010 From: vinjvinj at gmail.com (Vineet Jain) Date: Sat, 18 Dec 2010 11:38:55 -0500 Subject: [Cython] Compiler crash with pyrex_gdb=True Message-ID: Fails to compile when I add pyrex_gdb=True to my setup.py file. def test(param): def test2(): x = param.x rror compiling Cython file: ------------------------------------------------------------ ... def test(param): def test2(): ^ ------------------------------------------------------------ container.pyx:2:4: Compiler crash in DebugTransform -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101218/595cd767/attachment.htm From markflorisson88 at gmail.com Sat Dec 18 18:13:29 2010 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 18 Dec 2010 18:13:29 +0100 Subject: [Cython] Compiler crash with pyrex_gdb=True In-Reply-To: References: Message-ID: On 18 December 2010 17:38, Vineet Jain wrote: > Fails to compile when I add > > pyrex_gdb=True > > to my setup.py file. > > > def test(param): > def test2(): > x = param.x > > rror compiling Cython file: > ------------------------------------------------------------ > ... > def test(param): > def test2(): > ^ > ------------------------------------------------------------ > > container.pyx:2:4: Compiler crash in DebugTransform > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > Dear Vineet, Could you paste the line in the setup script and tell us the python version you're using? Also, it that all the output Cython gave? If there's more, paste everything. Kind regards, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101218/267d66dd/attachment.htm From vinjvinj at gmail.com Sat Dec 18 18:19:03 2010 From: vinjvinj at gmail.com (Vineet Jain) Date: Sat, 18 Dec 2010 12:19:03 -0500 Subject: [Cython] Compiler crash with pyrex_gdb=True In-Reply-To: References: Message-ID: The error can be produced by directly using cython as well. See below: Python version: Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. vinj at host61:/host/trading/pytrade$ python-dbg Python 2.6.6 (r266:84292, Sep 15 2010, 17:09:03) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. cython --gdb container.pyx Error compiling Cython file: ------------------------------------------------------------ ... def test(param): def test2(): ^ ------------------------------------------------------------ container.pyx:2:4: Compiler crash in DebugTransform ModuleNode.body = StatListNode(container.pyx:1:0) StatListNode.stats[0] = DefNode(container.pyx:1:0, modifiers = [...]/0, name = u'test', needs_closure = True, num_required_args = 1, reqd_kw_flags_cname = '0') DefNode.body = StatListNode(container.pyx:2:4) StatListNode.stats[0] = DefNode(container.pyx:2:4, modifiers = [...]/0, name = u'test2', needs_outer_scope = True, reqd_kw_flags_cname = '0') Compiler crash traceback from this point on: File "Visitor.py", line 173, in Cython.Compiler.Visitor.TreeVisitor._visitchild (/root/Cython-0.14/Cython/Compiler/Visitor.c:3444) File "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", line 1646, in visit_FuncDefNode self.serialize_local_variables(node.local_scope.entries) File "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", line 1703, in serialize_local_variables qualified_name=entry.qualified_name, AttributeError: 'Entry' object has no attribute 'qualified_name' [126744 refs] On Sat, Dec 18, 2010 at 11:38 AM, Vineet Jain wrote: > Fails to compile when I add > > pyrex_gdb=True > > to my setup.py file. > > > def test(param): > def test2(): > x = param.x > > rror compiling Cython file: > ------------------------------------------------------------ > ... > def test(param): > def test2(): > ^ > ------------------------------------------------------------ > > container.pyx:2:4: Compiler crash in DebugTransform > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101218/3fea635a/attachment.htm From markflorisson88 at gmail.com Sat Dec 18 18:28:41 2010 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 18 Dec 2010 18:28:41 +0100 Subject: [Cython] Compiler crash with pyrex_gdb=True In-Reply-To: References: Message-ID: On 18 December 2010 18:19, Vineet Jain wrote: > The error can be produced by directly using cython as well. See below: > > Python version: > > Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) > [GCC 4.4.5] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > > vinj at host61:/host/trading/pytrade$ python-dbg > Python 2.6.6 (r266:84292, Sep 15 2010, 17:09:03) > [GCC 4.4.5] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > > cython --gdb container.pyx > > Error compiling Cython file: > ------------------------------------------------------------ > ... > def test(param): > def test2(): > ^ > ------------------------------------------------------------ > > container.pyx:2:4: Compiler crash in DebugTransform > > ModuleNode.body = StatListNode(container.pyx:1:0) > StatListNode.stats[0] = DefNode(container.pyx:1:0, > modifiers = [...]/0, > name = u'test', > needs_closure = True, > num_required_args = 1, > reqd_kw_flags_cname = '0') > DefNode.body = StatListNode(container.pyx:2:4) > StatListNode.stats[0] = DefNode(container.pyx:2:4, > modifiers = [...]/0, > name = u'test2', > needs_outer_scope = True, > reqd_kw_flags_cname = '0') > > Compiler crash traceback from this point on: > File "Visitor.py", line 173, in > Cython.Compiler.Visitor.TreeVisitor._visitchild > (/root/Cython-0.14/Cython/Compiler/Visitor.c:3444) > File > "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", > line 1646, in visit_FuncDefNode > self.serialize_local_variables(node.local_scope.entries) > File > "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", > line 1703, in serialize_local_variables > qualified_name=entry.qualified_name, > AttributeError: 'Entry' object has no attribute 'qualified_name' > [126744 refs] > > > > On Sat, Dec 18, 2010 at 11:38 AM, Vineet Jain wrote: > >> Fails to compile when I add >> >> pyrex_gdb=True >> >> to my setup.py file. >> >> >> def test(param): >> def test2(): >> x = param.x >> >> rror compiling Cython file: >> ------------------------------------------------------------ >> ... >> def test(param): >> def test2(): >> ^ >> ------------------------------------------------------------ >> >> container.pyx:2:4: Compiler crash in DebugTransform >> >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > Ah yes, at the time it was developed the closure support wasn't in yet, so that wasn't tested. I'll have a look at it, thanks for the feedback! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101218/ed3128e7/attachment.htm From markflorisson88 at gmail.com Sat Dec 18 19:56:28 2010 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 18 Dec 2010 19:56:28 +0100 Subject: [Cython] Compiler crash with pyrex_gdb=True In-Reply-To: References: Message-ID: On 18 December 2010 18:19, Vineet Jain wrote: > The error can be produced by directly using cython as well. See below: > > Python version: > > Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) > [GCC 4.4.5] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > > vinj at host61:/host/trading/pytrade$ python-dbg > Python 2.6.6 (r266:84292, Sep 15 2010, 17:09:03) > [GCC 4.4.5] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > > cython --gdb container.pyx > > Error compiling Cython file: > ------------------------------------------------------------ > ... > def test(param): > def test2(): > ^ > ------------------------------------------------------------ > > container.pyx:2:4: Compiler crash in DebugTransform > > ModuleNode.body = StatListNode(container.pyx:1:0) > StatListNode.stats[0] = DefNode(container.pyx:1:0, > modifiers = [...]/0, > name = u'test', > needs_closure = True, > num_required_args = 1, > reqd_kw_flags_cname = '0') > DefNode.body = StatListNode(container.pyx:2:4) > StatListNode.stats[0] = DefNode(container.pyx:2:4, > modifiers = [...]/0, > name = u'test2', > needs_outer_scope = True, > reqd_kw_flags_cname = '0') > > Compiler crash traceback from this point on: > File "Visitor.py", line 173, in > Cython.Compiler.Visitor.TreeVisitor._visitchild > (/root/Cython-0.14/Cython/Compiler/Visitor.c:3444) > File > "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", > line 1646, in visit_FuncDefNode > self.serialize_local_variables(node.local_scope.entries) > File > "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", > line 1703, in serialize_local_variables > qualified_name=entry.qualified_name, > AttributeError: 'Entry' object has no attribute 'qualified_name' > [126744 refs] > > > > On Sat, Dec 18, 2010 at 11:38 AM, Vineet Jain wrote: > >> Fails to compile when I add >> >> pyrex_gdb=True >> >> to my setup.py file. >> >> >> def test(param): >> def test2(): >> x = param.x >> >> rror compiling Cython file: >> ------------------------------------------------------------ >> ... >> def test(param): >> def test2(): >> ^ >> ------------------------------------------------------------ >> >> container.pyx:2:4: Compiler crash in DebugTransform >> >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > I fixed the issue and pushed to https://github.com/markflorisson88/cython , but I was actually not able to reproduce the crash. For me it generated some extra XML that would simply be ignored by the debugger (which would mean you wouldn't be able to set a Cython breakpoint for the inner function). This was even the case with a fresh 0.14 non-development branch copy. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101218/4232ee25/attachment.htm From cornail at gmail.com Sun Dec 19 10:31:59 2010 From: cornail at gmail.com (=?ISO-8859-1?Q?Korn=E9l_JAHN?=) Date: Sun, 19 Dec 2010 10:31:59 +0100 Subject: [Cython] bug report, operator overloading for cdef classes Message-ID: Hello, I have been experiencing strange behavior regarding overloaded operators for cdef classes, using Cython 0.14. I think it can be regarded as a bug. See the example code below: # File: opoverload.pyx class Test1: def __rmul__(self, other): print "Calling Test1.__rmul__() with", type(self), type(other) def __mul__(self, other): print "Calling Test1.__mul__() with", type(self), type(other) cdef class Test2: def __rmul__(self, other): print "Calling Test2.__rmul__() with", type(self), type(other) def __mul__(self, other): print "Calling Test2.__mul__() with", type(self), type(other) Python console: >>> from opoverload import * >>> Test1() * 2.0 Calling Test1.__mul__() with >>> 2.0 * Test1() Calling Test1.__rmul__() with >>> Test2() * 2.0 Calling Test2.__mul__() with *>>> 2.0 * Test2()* *Calling Test2.__mul__() with * Test2.__rmul__() should be called and type(self) should not become float! Can you please suggest a temporary workaround? Thank you very much! Kornel Jahn -- Korn?l JAHN engineering physicist, PhD student Budapest University of Technology and Economics, Hungary -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101219/0328fb11/attachment.htm From robertwb at math.washington.edu Sun Dec 19 10:40:21 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 19 Dec 2010 01:40:21 -0800 Subject: [Cython] bug report, operator overloading for cdef classes In-Reply-To: References: Message-ID: On Sun, Dec 19, 2010 at 1:31 AM, Korn?l JAHN wrote: > Hello, > I have been experiencing strange behavior regarding overloaded operators for > cdef classes, using Cython 0.14. I think it can be regarded as a bug. See > the example code below: > # File: opoverload.pyx > class Test1: > ?? ?def __rmul__(self, other): > ?? ? ? ?print "Calling Test1.__rmul__() with", type(self), type(other) > ?? ?def __mul__(self, other): > ?? ? ? ?print "Calling Test1.__mul__() with", type(self), type(other) > cdef class Test2: > ?? ?def __rmul__(self, other): > ?? ? ? ?print "Calling Test2.__rmul__() with", type(self), type(other) > ?? ?def __mul__(self, other): > ?? ? ? ?print "Calling Test2.__mul__() with", type(self), type(other) > Python console: >>>> from opoverload import * >>>> Test1() * 2.0 > Calling Test1.__mul__() with >>>> 2.0 * Test1() > Calling Test1.__rmul__() with >>>> Test2() * 2.0 > Calling Test2.__mul__() with >>>> 2.0 * Test2() > Calling Test2.__mul__() with > Test2.__rmul__() should be called and type(self) should not become float! Have you read http://docs.cython.org/src/userguide/special_methods.html ? > Can you please suggest a temporary workaround? Manually check the types in your arithmetic functions. (Yes, I know that's not as nice as it could be, it's just a reflection of the underlying C API that we haven't glossed over.) - Robert From cornail at gmail.com Sun Dec 19 10:47:28 2010 From: cornail at gmail.com (=?ISO-8859-1?Q?Korn=E9l_JAHN?=) Date: Sun, 19 Dec 2010 10:47:28 +0100 Subject: [Cython] bug report, operator overloading for cdef classes In-Reply-To: References: Message-ID: Thank you for your quick reply, you are right, I ought to have read it. Sorry for bothering. Kornel On Sun, Dec 19, 2010 at 10:40 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Sun, Dec 19, 2010 at 1:31 AM, Korn?l JAHN wrote: > > Hello, > > I have been experiencing strange behavior regarding overloaded operators > for > > cdef classes, using Cython 0.14. I think it can be regarded as a bug. See > > the example code below: > > # File: opoverload.pyx > > class Test1: > > def __rmul__(self, other): > > print "Calling Test1.__rmul__() with", type(self), type(other) > > def __mul__(self, other): > > print "Calling Test1.__mul__() with", type(self), type(other) > > cdef class Test2: > > def __rmul__(self, other): > > print "Calling Test2.__rmul__() with", type(self), type(other) > > def __mul__(self, other): > > print "Calling Test2.__mul__() with", type(self), type(other) > > Python console: > >>>> from opoverload import * > >>>> Test1() * 2.0 > > Calling Test1.__mul__() with > >>>> 2.0 * Test1() > > Calling Test1.__rmul__() with > >>>> Test2() * 2.0 > > Calling Test2.__mul__() with > >>>> 2.0 * Test2() > > Calling Test2.__mul__() with > > Test2.__rmul__() should be called and type(self) should not become float! > > Have you read http://docs.cython.org/src/userguide/special_methods.html ? > > > Can you please suggest a temporary workaround? > > Manually check the types in your arithmetic functions. (Yes, I know > that's not as nice as it could be, it's just a reflection of the > underlying C API that we haven't glossed over.) > > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Korn?l JAHN engineering physicist, PhD student Budapest University of Technology and Economics, Hungary -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101219/0448937b/attachment-0001.htm From robertwb at math.washington.edu Sun Dec 19 10:51:48 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 19 Dec 2010 01:51:48 -0800 Subject: [Cython] bug report, operator overloading for cdef classes In-Reply-To: References: Message-ID: On Sun, Dec 19, 2010 at 1:47 AM, Korn?l JAHN wrote: > Thank you for your quick reply, you are right, I ought to have read it. > Sorry for bothering. No problem, you're not the first to be bitten by that. It's an unfortunate wart in the language, but fortunately one of the few. > On Sun, Dec 19, 2010 at 10:40 AM, Robert Bradshaw > wrote: >> >> On Sun, Dec 19, 2010 at 1:31 AM, Korn?l JAHN wrote: >> > Hello, >> > I have been experiencing strange behavior regarding overloaded operators >> > for >> > cdef classes, using Cython 0.14. I think it can be regarded as a bug. >> > See >> > the example code below: >> > # File: opoverload.pyx >> > class Test1: >> > ?? ?def __rmul__(self, other): >> > ?? ? ? ?print "Calling Test1.__rmul__() with", type(self), type(other) >> > ?? ?def __mul__(self, other): >> > ?? ? ? ?print "Calling Test1.__mul__() with", type(self), type(other) >> > cdef class Test2: >> > ?? ?def __rmul__(self, other): >> > ?? ? ? ?print "Calling Test2.__rmul__() with", type(self), type(other) >> > ?? ?def __mul__(self, other): >> > ?? ? ? ?print "Calling Test2.__mul__() with", type(self), type(other) >> > Python console: >> >>>> from opoverload import * >> >>>> Test1() * 2.0 >> > Calling Test1.__mul__() with >> >>>> 2.0 * Test1() >> > Calling Test1.__rmul__() with >> >>>> Test2() * 2.0 >> > Calling Test2.__mul__() with >> >>>> 2.0 * Test2() >> > Calling Test2.__mul__() with >> > Test2.__rmul__() should be called and type(self) should not become >> > float! >> >> Have you read http://docs.cython.org/src/userguide/special_methods.html ? >> >> > Can you please suggest a temporary workaround? >> >> Manually check the types in your arithmetic functions. (Yes, I know >> that's not as nice as it could be, it's just a reflection of the >> underlying C API that we haven't glossed over.) >> >> - Robert >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > > > -- > Korn?l JAHN > engineering physicist, PhD student > Budapest University of Technology and Economics, Hungary > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > From wking at drexel.edu Sun Dec 19 14:26:25 2010 From: wking at drexel.edu (W. Trevor King) Date: Sun, 19 Dec 2010 08:26:25 -0500 Subject: [Cython] [PATCH] fix broken PyBuffer_Release declaration Message-ID: <20101219132624.GA17426@tyr.home.net> Here's a patch bringing the ByBuffer_Release declaration up to speed with current Python headers. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- # HG changeset patch # User W. Trevor King # Date 1292760469 18000 # Node ID 703ccf60e7e2da6608bd33adbcf9d5d869015af8 # Parent 3acb88570aa589bc92c45ac1ada9fd4ff55a07e2 Update PyBuffer_Release() declaration to match new buffer API. See http://docs.python.org/c-api/buffer.html#PyBuffer_Release for the authoritative definition. diff -r 3acb88570aa5 -r 703ccf60e7e2 Cython/Includes/cpython/buffer.pxd --- a/Cython/Includes/cpython/buffer.pxd Thu Dec 16 22:40:25 2010 -0300 +++ b/Cython/Includes/cpython/buffer.pxd Sun Dec 19 07:07:49 2010 -0500 @@ -49,9 +49,9 @@ # 0 is returned on success and -1 on error. - void PyBuffer_Release(object obj, object view) - # Release the buffer view over obj. This should be called when the - # buffer is no longer being used as it may free memory from it. + void PyBuffer_Release(Py_buffer *view) + # Release the buffer view. This should be called when the buffer + # is no longer being used as it may free memory from it. void* PyBuffer_GetPointer(Py_buffer *view, Py_ssize_t *indices) # ?? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20101219/52eb68ad/attachment.pgp From arfrever.fta at gmail.com Sun Dec 19 15:39:39 2010 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sun, 19 Dec 2010 15:39:39 +0100 Subject: [Cython] Cython 0.14: KeyError during compilation of Yasm Message-ID: <201012191540.24378.Arfrever.FTA@gmail.com> $ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz ... $ tar -xzf yasm-1.1.0.tar.gz $ cd yasm-1.1.0 $ ./configure --enable-python --enable-python-bindings ... $ make ... /usr/bin/python -c "from Cython.Compiler.Main import main; main(command_line=1)" \ -o yasm_python.c yasm.pyx Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py", line 790, in main result = compile(sources, options) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py", line 765, in compile return compile_multiple(source, options) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py", line 737, in compile_multiple result = run_pipeline(source, options) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py", line 601, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py", line 240, in run_pipeline data = phase(data) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1025, in __call__ return super(AnalyseDeclarationsTransform, self).__call__(root) File "Visitor.py", line 272, in Cython.Compiler.Visitor.CythonTransform.__call__ (Cython/Compiler/Visitor.c:4936) File "Visitor.py", line 255, in Cython.Compiler.Visitor.VisitorTransform.__call__ (Cython/Compiler/Visitor.c:4696) File "Visitor.py", line 163, in Cython.Compiler.Visitor.TreeVisitor._visit (Cython/Compiler/Visitor.c:3254) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1033, in visit_ModuleNode node.analyse_declarations(self.env_stack[-1]) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/ModuleNode.py", line 64, in analyse_declarations self.body.analyse_declarations(env) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 346, in analyse_declarations stat.analyse_declarations(env) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 346, in analyse_declarations stat.analyse_declarations(env) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 3249, in analyse_declarations self.body.analyse_declarations(scope) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 346, in analyse_declarations stat.analyse_declarations(env) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 1999, in analyse_declarations self.analyse_signature(env) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 2097, in analyse_signature arg.hdr_type = sig.fixed_arg_type(i) File "/usr/lib64/python2.7/site-packages/Cython/Compiler/TypeSlots.py", line 100, in fixed_arg_type return self.format_map[self.fixed_arg_format[i]] KeyError: 'T' This problem doesn't occur with Cython 0.13. Is it a bug in Cython 0.14 or in Yasm? -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20101219/7c69f5b5/attachment.pgp From stefan_ml at behnel.de Sun Dec 19 19:28:32 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 19 Dec 2010 19:28:32 +0100 Subject: [Cython] [PATCH] fix broken PyBuffer_Release declaration In-Reply-To: <20101219132624.GA17426@tyr.home.net> References: <20101219132624.GA17426@tyr.home.net> Message-ID: <4D0E4ED0.4040906@behnel.de> W. Trevor King, 19.12.2010 14:26: > Here's a patch bringing the ByBuffer_Release declaration up to speed > with current Python headers. Applied, thanks. The buffer PEP and its implementation were still in flux even shortly before the release of Py3. That was the time when the buffer.pxd was written, and I guess it wasn't fixed since because most people just use the builtin buffer support in Cython. Stefan From stefan_ml at behnel.de Sun Dec 19 20:20:35 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 19 Dec 2010 20:20:35 +0100 Subject: [Cython] Cython 0.14: KeyError during compilation of Yasm In-Reply-To: <201012191540.24378.Arfrever.FTA@gmail.com> References: <201012191540.24378.Arfrever.FTA@gmail.com> Message-ID: <4D0E5B03.2090100@behnel.de> Arfrever Frehtes Taifersar Arahesis, 19.12.2010 15:39: > $ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz > ... > $ tar -xzf yasm-1.1.0.tar.gz > $ cd yasm-1.1.0 > $ ./configure --enable-python --enable-python-bindings > ... > $ make > ... > /usr/bin/python -c "from Cython.Compiler.Main import main; main(command_line=1)" \ > -o yasm_python.c yasm.pyx > Traceback (most recent call last): > [...] > File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 3249, in analyse_declarations > self.body.analyse_declarations(scope) > File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 346, in analyse_declarations > stat.analyse_declarations(env) > File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 1999, in analyse_declarations > self.analyse_signature(env) > File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 2097, in analyse_signature > arg.hdr_type = sig.fixed_arg_type(i) > File "/usr/lib64/python2.7/site-packages/Cython/Compiler/TypeSlots.py", line 100, in fixed_arg_type > return self.format_map[self.fixed_arg_format[i]] > KeyError: 'T' > > This problem doesn't occur with Cython 0.13. > Is it a bug in Cython 0.14 or in Yasm? Both, I'd say. The crash is in line 100 of yasm.pyx, where it says: cdef class __assoc_data_callback: cdef yasm_assoc_data_callback *cb def __new__(self, destroy, print_): # <=== HERE self.cb = \ malloc(sizeof(yasm_assoc_data_callback)) self.cb.destroy = PyCObject_AsVoidPtr(destroy) #self.cb.print_ = \ PyCObject_AsVoidPtr(print_) The "__new__" method must be called "__cinit__". Without closer investigation, I suspect that the relevant change in Cython 0.14 is that it knows about "__new__" being a staticmethod. However, for cdef classes is silently translates "__new__" into "__cinit__" internally. It seems like both don't work together. Stefan From vitja.makarov at gmail.com Mon Dec 20 20:26:48 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 20 Dec 2010 22:26:48 +0300 Subject: [Cython] pyregr test runner In-Reply-To: References: <4D03843E.40002@behnel.de> Message-ID: 2010/12/11 Vitja Makarov : > 2010/12/11 Vitja Makarov : >> 2010/12/11 Stefan Behnel : >>> Vitja Makarov, 11.12.2010 14:42: >>>> 2010/12/11 Vitja Makarov: >>>>> I would like to fix test runner for pyregr. >>>>> >>>>> Original python regrtest.py import module and run its test_main() if any. >>>>> It's not a problem to do this in our test runner but I don't know yet >>>>> how to get result back to . >>>>> I think test_support.run_unittest and run_doctestcan should be patched. >>>> >>>> I've pushed support for this stuff here: >>>> >>>> https://github.com/vitek/cython/commit/1bcf08a87901265ad7dfe16e952c872c20869cea >>>> >>>> Here are results: >>>> https://sage.math.washington.edu:8091/hudson/view/cython-generators/job/cython-vitek-tests-pyregr-py27-c/ >>>> >>>> 5675/283 was 3865/636 >>> >>> That's cool. Patch looks fine to me. >>> >>> >>>> I think that doctests for pyregr should be disabled... they mostly >>>> test python itself not cython >>> >>> *All* pyregr tests test mostly Python. I don't see why we should disable >>> all doctests, some of them may still test compiled stuff. >>> >> >> Ok. I'm now going to handle SkipTest and ResourceDenied exceptions >> > > Added. That was hard to make it work with both py2 and py3... > > -- > vitja. > Wouldn't you like to merge this change into cython-devel? That would be nice to compare results ;) -- vitja. From robertwb at math.washington.edu Mon Dec 20 20:35:53 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 20 Dec 2010 11:35:53 -0800 Subject: [Cython] Cython 0.14: KeyError during compilation of Yasm In-Reply-To: <4D0E5B03.2090100@behnel.de> References: <201012191540.24378.Arfrever.FTA@gmail.com> <4D0E5B03.2090100@behnel.de> Message-ID: On Sun, Dec 19, 2010 at 11:20 AM, Stefan Behnel wrote: > Arfrever Frehtes Taifersar Arahesis, 19.12.2010 15:39: >> $ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz >> ... >> $ tar -xzf yasm-1.1.0.tar.gz >> $ cd yasm-1.1.0 >> $ ./configure --enable-python --enable-python-bindings >> ... >> $ make >> ... >> /usr/bin/python -c "from Cython.Compiler.Main import main; main(command_line=1)" \ >> ? ? ? ? ?-o yasm_python.c yasm.pyx >> Traceback (most recent call last): >> [...] >> ? ?File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 3249, in analyse_declarations >> ? ? ?self.body.analyse_declarations(scope) >> ? ?File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 346, in analyse_declarations >> ? ? ?stat.analyse_declarations(env) >> ? ?File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 1999, in analyse_declarations >> ? ? ?self.analyse_signature(env) >> ? ?File "/usr/lib64/python2.7/site-packages/Cython/Compiler/Nodes.py", line 2097, in analyse_signature >> ? ? ?arg.hdr_type = sig.fixed_arg_type(i) >> ? ?File "/usr/lib64/python2.7/site-packages/Cython/Compiler/TypeSlots.py", line 100, in fixed_arg_type >> ? ? ?return self.format_map[self.fixed_arg_format[i]] >> KeyError: 'T' >> >> This problem doesn't occur with Cython 0.13. >> Is it a bug in Cython 0.14 or in Yasm? > > Both, I'd say. The crash is in line 100 of yasm.pyx, where it says: > > cdef class __assoc_data_callback: > ? ? cdef yasm_assoc_data_callback *cb > ? ? def __new__(self, destroy, print_): ? ? ? ? ?# <=== HERE > ? ? ? ? self.cb = ?\ > ? ? ? ? ? ? ? ? ? malloc(sizeof(yasm_assoc_data_callback)) > ? ? ? ? self.cb.destroy = PyCObject_AsVoidPtr(destroy) > ? ? ? ? #self.cb.print_ = \ > ? ? ? ? ? ? ? ? ? ? ? ? ? ?PyCObject_AsVoidPtr(print_) > > > The "__new__" method must be called "__cinit__". > > Without closer investigation, I suspect that the relevant change in Cython > 0.14 is that it knows about "__new__" being a staticmethod. However, for > cdef classes is silently translates "__new__" into "__cinit__" internally. > It seems like both don't work together. Using __new__ instead of __cinit__ has been deprecated for years now, how about turning it into a clean error (which would fix this bug). - Robert From robertwb at math.washington.edu Mon Dec 20 20:42:17 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 20 Dec 2010 11:42:17 -0800 Subject: [Cython] Compiler crash with pyrex_gdb=True In-Reply-To: References: Message-ID: On Sat, Dec 18, 2010 at 10:56 AM, mark florisson wrote: > > > On 18 December 2010 18:19, Vineet Jain wrote: >> >> The error can be produced by directly using cython as well. See below: >> Python version: >> >> Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) >> [GCC 4.4.5] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >> vinj at host61:/host/trading/pytrade$ python-dbg >> Python 2.6.6 (r266:84292, Sep 15 2010, 17:09:03) >> [GCC 4.4.5] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >> cython --gdb container.pyx >> Error compiling Cython file: >> ------------------------------------------------------------ >> ... >> def test(param): >> ?? ?def test2(): >> ?? ^ >> ------------------------------------------------------------ >> container.pyx:2:4: Compiler crash in DebugTransform >> ModuleNode.body = StatListNode(container.pyx:1:0) >> StatListNode.stats[0] = DefNode(container.pyx:1:0, >> ?? ?modifiers = [...]/0, >> ?? ?name = u'test', >> ?? ?needs_closure = True, >> ?? ?num_required_args = 1, >> ?? ?reqd_kw_flags_cname = '0') >> DefNode.body = StatListNode(container.pyx:2:4) >> StatListNode.stats[0] = DefNode(container.pyx:2:4, >> ?? ?modifiers = [...]/0, >> ?? ?name = u'test2', >> ?? ?needs_outer_scope = True, >> ?? ?reqd_kw_flags_cname = '0') >> Compiler crash traceback from this point on: >> ??File "Visitor.py", line 173, in >> Cython.Compiler.Visitor.TreeVisitor._visitchild >> (/root/Cython-0.14/Cython/Compiler/Visitor.c:3444) >> ??File >> "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", >> line 1646, in visit_FuncDefNode >> ?? ?self.serialize_local_variables(node.local_scope.entries) >> ??File >> "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", >> line 1703, in serialize_local_variables >> ?? ?qualified_name=entry.qualified_name, >> AttributeError: 'Entry' object has no attribute 'qualified_name' >> [126744 refs] >> >> >> On Sat, Dec 18, 2010 at 11:38 AM, Vineet Jain wrote: >>> >>> Fails to compile when I add >>> ?? ? ? ?pyrex_gdb=True >>> to my setup.py file. >>> >>> def test(param): >>> ?? ?def test2(): >>> ?? ? ? ?x = param.x >>> rror compiling Cython file: >>> ------------------------------------------------------------ >>> ... >>> def test(param): >>> ?? ?def test2(): >>> ?? ^ >>> ------------------------------------------------------------ >>> container.pyx:2:4: Compiler crash in DebugTransform >> >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > I fixed the issue and pushed to?https://github.com/markflorisson88/cython?, > but I was actually not able to reproduce the crash. For me it generated some > extra XML that would simply be ignored by the debugger (which would mean you > wouldn't be able to set a Cython breakpoint for the inner function). This > was even the case with a fresh 0.14 non-development branch copy. I was never able to reproduce the crash either, but +1 to closure support. Is your branch ready to pull from? (The fix looks pretty clean.) - Robert From robertwb at math.washington.edu Mon Dec 20 20:57:31 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 20 Dec 2010 11:57:31 -0800 Subject: [Cython] provide 'builtins' builtin module also for -2 mode In-Reply-To: <4D0A8E9C.8010008@behnel.de> References: <4D0A8A40.2010102@behnel.de> <4D0A8E9C.8010008@behnel.de> Message-ID: On Thu, Dec 16, 2010 at 2:11 PM, Stefan Behnel wrote: > [CC-ing cython-dev] > > Lisandro Dalcin, 16.12.2010 23:03: >> On 16 December 2010 18:53, Stefan Behnel wrote: >>> Lisandro Dalcin, 16.12.2010 22:33: >>>> >>>> Just a few comments about this changeset: >>>> >>>> changeset: ? 4244:e0508ac59b35 >>>> user: ? ? ? ?Stefan Behnel >>>> date: ? ? ? ?Thu Dec 16 20:11:29 2010 +0100 >>>> summary: ? ? provide 'builtins' builtin module in -3 mode >>>> >>>> >>>> IIUC, under -3 we can use both "__builtin__" and "builtins", but under >>>> -2 only "__builtin__". Why not to also provide "builtins" under -2 >>>> mode? We are already supporting "builtins" in some parts, for example >>>> at ModuleNode.py, method "generate_type_import_call()" ... >>> >>> Well, as long as it actually means that users can import any of the two in >>> their code and have it work at runtime in both Py2 and Py3? >> >> Yes, of course... >> >>> But that would >>> require that Cython changes the user imported module name depending on the C >>> compile time environment. I'm not sure that's a good idea. >> >> Well, the code at generate_type_import_call() work along these lines, >> right? I mean, it translates __builtin__ and builtins to the right >> name in the C runtime. > > But that's only when it actually runs in Py3, which is still in "eventual > migration" state for many users. As long as you keep your code in a Py2 > environment and only "import __builtin__", Cython won't do any magic behind > your back. > > >>> The problem I see is that users may have a "builtins" module lying around >>> somewhere in their Py2 code base that Cython would misinterpret and wouldn't >>> be able to import. "builtins" isn't that a bad name for a module, e.g. in a >>> plugin environment... >> >> Yes, you have a good point. However the import would need to be >> absolute for Cython to misinterpret it, right? I would argue that such >> code smells... > > It didn't smell all that much before Py3 renamed the module. > > Anyway, I'm +0 on changing this. Let's hear what the others think. I'm +0 as well. - Robert From robertwb at math.washington.edu Tue Dec 21 07:12:40 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 20 Dec 2010 22:12:40 -0800 Subject: [Cython] Hg to git Message-ID: Anyone object to making http://hg.cython.org/ read-only, and making https://github.com/cython/cython/ (and its forks) the live devel branch? - Robert From stefan_ml at behnel.de Tue Dec 21 10:00:02 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 21 Dec 2010 10:00:02 +0100 Subject: [Cython] Cython 0.14: KeyError during compilation of Yasm In-Reply-To: References: <201012191540.24378.Arfrever.FTA@gmail.com> <4D0E5B03.2090100@behnel.de> Message-ID: <4D106C92.1050709@behnel.de> Robert Bradshaw, 20.12.2010 20:35: > On Sun, Dec 19, 2010 at 11:20 AM, Stefan Behnel wrote: >> The "__new__" method must be called "__cinit__". >> >> Without closer investigation, I suspect that the relevant change in Cython >> 0.14 is that it knows about "__new__" being a staticmethod. However, for >> cdef classes is silently translates "__new__" into "__cinit__" internally. >> It seems like both don't work together. > > Using __new__ instead of __cinit__ has been deprecated for years now, > how about turning it into a clean error (which would fix this bug). +1, that's what I thought, too. Stefan From robertwb at math.washington.edu Tue Dec 21 11:12:22 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 21 Dec 2010 02:12:22 -0800 Subject: [Cython] Cython 0.14: KeyError during compilation of Yasm In-Reply-To: <4D106C92.1050709@behnel.de> References: <201012191540.24378.Arfrever.FTA@gmail.com> <4D0E5B03.2090100@behnel.de> <4D106C92.1050709@behnel.de> Message-ID: On Tue, Dec 21, 2010 at 1:00 AM, Stefan Behnel wrote: > Robert Bradshaw, 20.12.2010 20:35: >> On Sun, Dec 19, 2010 at 11:20 AM, Stefan Behnel wrote: >>> The "__new__" method must be called "__cinit__". >>> >>> Without closer investigation, I suspect that the relevant change in Cython >>> 0.14 is that it knows about "__new__" being a staticmethod. However, for >>> cdef classes is silently translates "__new__" into "__cinit__" internally. >>> It seems like both don't work together. >> >> Using __new__ instead of __cinit__ has been deprecated for years now, >> how about turning it into a clean error (which would fix this bug). > > +1, that's what I thought, too. Done: https://github.com/cython/cython/commit/e3c9a78686b7a7f0d36da8e6189b1e1fb4037c73 From stefan_ml at behnel.de Tue Dec 21 12:10:50 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 21 Dec 2010 12:10:50 +0100 Subject: [Cython] Hg to git In-Reply-To: References: Message-ID: <4D108B3A.7090901@behnel.de> Robert Bradshaw, 21.12.2010 07:12: > Anyone object to making http://hg.cython.org/ read-only, and making > https://github.com/cython/cython/ (and its forks) the live devel > branch? I think it's a good time to do this now that 0.14 is out. Stefan From wking at drexel.edu Tue Dec 21 14:56:17 2010 From: wking at drexel.edu (W. Trevor King) Date: Tue, 21 Dec 2010 08:56:17 -0500 Subject: [Cython] [PATCH] fix broken PyBuffer_Release declaration In-Reply-To: <4D0E4ED0.4040906@behnel.de> References: <20101219132624.GA17426@tyr.home.net> <4D0E4ED0.4040906@behnel.de> Message-ID: <20101221135616.GA12103@tyr.home.net> On Sun, Dec 19, 2010 at 07:28:32PM +0100, Stefan Behnel wrote: > W. Trevor King, 19.12.2010 14:26: > > Here's a patch bringing the ByBuffer_Release declaration up to speed > > with current Python headers. > > Applied, thanks. The buffer PEP and its implementation were still in flux > even shortly before the release of Py3. Actually, the declaration in PEP 3118 has not been updated [1]. Is that a PEP bug? > That was the time when the buffer.pxd was written, and I guess it > wasn't fixed since because most people just use the builtin buffer > support in Cython. Hmm, perhaps I'm doing things the wrong way then ;). I'm calling an external library that needs: fn(unsigned sort *array, int len) Is there another way to get that pointer? [1]: http://www.python.org/dev/peps/pep-3118/#releasing-the-buffer -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20101221/17af8544/attachment.pgp From stefan_ml at behnel.de Tue Dec 21 15:49:08 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 21 Dec 2010 15:49:08 +0100 Subject: [Cython] [PATCH] fix broken PyBuffer_Release declaration In-Reply-To: <20101221135616.GA12103@tyr.home.net> References: <20101219132624.GA17426@tyr.home.net> <4D0E4ED0.4040906@behnel.de> <20101221135616.GA12103@tyr.home.net> Message-ID: <4D10BE64.7090503@behnel.de> W. Trevor King, 21.12.2010 14:56: > On Sun, Dec 19, 2010 at 07:28:32PM +0100, Stefan Behnel wrote: >> W. Trevor King, 19.12.2010 14:26: >>> Here's a patch bringing the ByBuffer_Release declaration up to speed >>> with current Python headers. >> >> Applied, thanks. The buffer PEP and its implementation were still in flux >> even shortly before the release of Py3. > > Actually, the declaration in PEP 3118 has not been updated [1]. Is > that a PEP bug? Looks like it (and wouldn't be the first time ;). >> That was the time when the buffer.pxd was written, and I guess it >> wasn't fixed since because most people just use the builtin buffer >> support in Cython. > > Hmm, perhaps I'm doing things the wrong way then ;). I'm calling an > external library that needs: > fn(unsigned sort *array, int len) > Is there another way to get that pointer? Admittedly, it's a bit hidden in the docs if you don't use NumPy. Does this help? http://docs.cython.org/src/tutorial/numpy.html http://wiki.cython.org/enhancements/buffer I'm surprised I didn't find any more extensive documentation of this feature in the core documentation. Stefan From robertwb at math.washington.edu Tue Dec 21 18:32:31 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 21 Dec 2010 09:32:31 -0800 Subject: [Cython] Hg to git In-Reply-To: <4D108B3A.7090901@behnel.de> References: <4D108B3A.7090901@behnel.de> Message-ID: On Tue, Dec 21, 2010 at 3:10 AM, Stefan Behnel wrote: > Robert Bradshaw, 21.12.2010 07:12: >> Anyone object to making http://hg.cython.org/ read-only, and making >> https://github.com/cython/cython/ (and its forks) the live devel >> branch? > > I think it's a good time to do this now that 0.14 is out. Yep. Done. - Robert From snaury at gmail.com Tue Dec 21 23:19:02 2010 From: snaury at gmail.com (Alexey Borzenkov) Date: Wed, 22 Dec 2010 01:19:02 +0300 Subject: [Cython] Broken .pyxdep files Message-ID: Hello, I've recently started using Cython in one of my projects because I needed pyximport and automatic building of extensions and found that adding .pyxdep files makes Cython raise NameError because _test_files is not defined. Now oddly the comment says that _test_files is used for tests, but since _test_files is not assigned anywhere it makes me wonder how would tests even pass when it's supposed to generate NameError in tests as well? Also it's quite strange that I looked as far back as 0.11.3 and can't find _test_files assignment, but Google shows that in 0.10.2 it seems it was there (http://cython.sourcearchive.com/documentation/0.10.2/pyximport_8py-source.html). It seems really strange to me how it was broken for that long! :-/ I just checked with Cython 0.14 and it seems that pyximport tests indeed don't pass: $ PYTHONPATH=. python pyximport/test/test_pyximport.py 1. pyximport.install is missing reload_support=True 2. NameError is raised during build because of _test_files 3. If _test_files = [] is added as was in 0.10.2 then AttributeError is still raised because incorrect pyximport is imported. Now if I fix both problems and run tests like this: $ PYTHONPATH=pyximport:. python pyximport/test/test_pyximport.py Then it finally seems to work. So pyximport and tests are obviously broken. Also, there's another problem with pyximport that makes it unsafe to use it without build_in_temp=False. For example, if I have package1/module.pyx and package2/module.pyx then only one of those will build/import correctly, because the same .c filename is used for both. This is not a pyximport problem however, because pyxbuild merely uses --pyrex-c-in-temp, and it causes build_ext.py to use hard-coded "pyrex" temp subdirectory. I hope someone can fix this. Best regards, Alexey Borzenkov. From snaury at gmail.com Wed Dec 22 08:49:14 2010 From: snaury at gmail.com (Alexey Borzenkov) Date: Wed, 22 Dec 2010 10:49:14 +0300 Subject: [Cython] Broken .pyxdep files In-Reply-To: References: Message-ID: On Wed, Dec 22, 2010 at 1:19 AM, Alexey Borzenkov wrote: > I've recently started using Cython in one of my projects because I > needed pyximport and automatic building of extensions and found that > adding .pyxdep files makes Cython raise NameError because _test_files > is not defined. [...] > Also, there's another problem with pyximport that makes it unsafe to > use it without build_in_temp=False. For example, if I have > package1/module.pyx and package2/module.pyx then only one of those > will build/import correctly, Btw, I just realized that Cython is finally using git and github, so if anyone is interested here are my patches: https://github.com/snaury/cython/commit/b43f68280919e98767bd4d163cb1eaf88c4c857e https://github.com/snaury/cython/commit/6f73b4280958f5d3f5fa77c3a4ed19b277784140 Best regards, Alexey Borzenkov. From robertwb at math.washington.edu Wed Dec 22 09:48:47 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 22 Dec 2010 00:48:47 -0800 Subject: [Cython] Broken .pyxdep files In-Reply-To: References: Message-ID: On Tue, Dec 21, 2010 at 11:49 PM, Alexey Borzenkov wrote: > On Wed, Dec 22, 2010 at 1:19 AM, Alexey Borzenkov wrote: >> I've recently started using Cython in one of my projects because I >> needed pyximport and automatic building of extensions and found that >> adding .pyxdep files makes Cython raise NameError because _test_files >> is not defined. > [...] >> Also, there's another problem with pyximport that makes it unsafe to >> use it without build_in_temp=False. For example, if I have >> package1/module.pyx and package2/module.pyx then only one of those >> will build/import correctly, > > Btw, I just realized that Cython is finally using git and github, so > if anyone is interested here are my patches: > > https://github.com/snaury/cython/commit/b43f68280919e98767bd4d163cb1eaf88c4c857e > https://github.com/snaury/cython/commit/6f73b4280958f5d3f5fa77c3a4ed19b277784140 Thanks. pyximport is really cool, but no one's been maintaining it, and it looks like we haven't been testing it either. (Code is either new, tested, or broken...) At first glance the patches look fine, I'll try to take a deeper look over the holidays (if no one beats me to it...) - Robert From arfrever.fta at gmail.com Thu Dec 23 14:16:52 2010 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Thu, 23 Dec 2010 14:16:52 +0100 Subject: [Cython] Hg to git In-Reply-To: References: Message-ID: <201012231418.00966.Arfrever.FTA@gmail.com> 2010-12-21 07:12:40 Robert Bradshaw napisa?(a): > Anyone object to making http://hg.cython.org/ read-only, and making > https://github.com/cython/cython/ (and its forks) the live devel > branch? http://hg.cython.org/cython-devel/rev/b816b03ff502 contains http://hg.cython.org/cython-devel/raw-rev/b816b03ff502 URL, which allows to easily download a patch. Does https://github.com/cython/cython/commit/e3c9a78686b7a7f0d36da8e6189b1e1fb4037c73 provide similar functionality? -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20101223/f24cc69a/attachment.pgp From markflorisson88 at gmail.com Thu Dec 23 19:06:24 2010 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 23 Dec 2010 19:06:24 +0100 Subject: [Cython] Compiler crash with pyrex_gdb=True In-Reply-To: References: Message-ID: On 20 December 2010 20:42, Robert Bradshaw wrote: > On Sat, Dec 18, 2010 at 10:56 AM, mark florisson > wrote: > > > > > > On 18 December 2010 18:19, Vineet Jain wrote: > >> > >> The error can be produced by directly using cython as well. See below: > >> Python version: > >> > >> Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) > >> [GCC 4.4.5] on linux2 > >> Type "help", "copyright", "credits" or "license" for more information. > >> > >> vinj at host61:/host/trading/pytrade$ python-dbg > >> Python 2.6.6 (r266:84292, Sep 15 2010, 17:09:03) > >> [GCC 4.4.5] on linux2 > >> Type "help", "copyright", "credits" or "license" for more information. > >> > >> cython --gdb container.pyx > >> Error compiling Cython file: > >> ------------------------------------------------------------ > >> ... > >> def test(param): > >> def test2(): > >> ^ > >> ------------------------------------------------------------ > >> container.pyx:2:4: Compiler crash in DebugTransform > >> ModuleNode.body = StatListNode(container.pyx:1:0) > >> StatListNode.stats[0] = DefNode(container.pyx:1:0, > >> modifiers = [...]/0, > >> name = u'test', > >> needs_closure = True, > >> num_required_args = 1, > >> reqd_kw_flags_cname = '0') > >> DefNode.body = StatListNode(container.pyx:2:4) > >> StatListNode.stats[0] = DefNode(container.pyx:2:4, > >> modifiers = [...]/0, > >> name = u'test2', > >> needs_outer_scope = True, > >> reqd_kw_flags_cname = '0') > >> Compiler crash traceback from this point on: > >> File "Visitor.py", line 173, in > >> Cython.Compiler.Visitor.TreeVisitor._visitchild > >> (/root/Cython-0.14/Cython/Compiler/Visitor.c:3444) > >> File > >> > "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", > >> line 1646, in visit_FuncDefNode > >> self.serialize_local_variables(node.local_scope.entries) > >> File > >> > "/usr/local/lib/python2.6/dist-packages/Cython/Compiler/ParseTreeTransforms.py", > >> line 1703, in serialize_local_variables > >> qualified_name=entry.qualified_name, > >> AttributeError: 'Entry' object has no attribute 'qualified_name' > >> [126744 refs] > >> > >> > >> On Sat, Dec 18, 2010 at 11:38 AM, Vineet Jain > wrote: > >>> > >>> Fails to compile when I add > >>> pyrex_gdb=True > >>> to my setup.py file. > >>> > >>> def test(param): > >>> def test2(): > >>> x = param.x > >>> rror compiling Cython file: > >>> ------------------------------------------------------------ > >>> ... > >>> def test(param): > >>> def test2(): > >>> ^ > >>> ------------------------------------------------------------ > >>> container.pyx:2:4: Compiler crash in DebugTransform > >> > >> > >> _______________________________________________ > >> Cython-dev mailing list > >> Cython-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/cython-dev > >> > > > > I fixed the issue and pushed to > https://github.com/markflorisson88/cython , > > but I was actually not able to reproduce the crash. For me it generated > some > > extra XML that would simply be ignored by the debugger (which would mean > you > > wouldn't be able to set a Cython breakpoint for the inner function). This > > was even the case with a fresh 0.14 non-development branch copy. > > I was never able to reproduce the crash either, but +1 to closure > support. Is your branch ready to pull from? (The fix looks pretty > clean.) > > I think so, it worked for me, but I haven't had time to write any unit tests (I'm on vacation now). > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20101223/8edd6541/attachment.htm From robertwb at math.washington.edu Thu Dec 23 19:28:13 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 23 Dec 2010 10:28:13 -0800 Subject: [Cython] Hg to git In-Reply-To: <201012231418.00966.Arfrever.FTA@gmail.com> References: <201012231418.00966.Arfrever.FTA@gmail.com> Message-ID: On Thu, Dec 23, 2010 at 5:16 AM, Arfrever Frehtes Taifersar Arahesis wrote: > 2010-12-21 07:12:40 Robert Bradshaw napisa?(a): >> Anyone object to making http://hg.cython.org/ read-only, and making >> https://github.com/cython/cython/ (and its forks) the live devel >> branch? > > http://hg.cython.org/cython-devel/rev/b816b03ff502 contains http://hg.cython.org/cython-devel/raw-rev/b816b03ff502 URL, > which allows to easily download a patch. Does https://github.com/cython/cython/commit/e3c9a78686b7a7f0d36da8e6189b1e1fb4037c73 > provide similar functionality? Apparently it works like this: http://support.github.com/discussions/feature-requests/429-download-patch-files - Robert From wesmckinn at gmail.com Thu Dec 23 22:37:50 2010 From: wesmckinn at gmail.com (Wes McKinney) Date: Thu, 23 Dec 2010 22:37:50 +0100 Subject: [Cython] Compilation error with gcc 4.5.2 on Windows 64 Message-ID: I'm getting the following error trying to build a Cython 0.14-generated extension: gcc -g -DDEBUG -DMS_WIN64 -mno-cygwin -O0 -Wall -Wstrict-prototypes -IC:\Python2 6\lib\site-packages\numpy\core\include -IC:\Python26\lib\site-packages\numpy\cor e\include -IC:\Python26\include -IC:\Python26\PC -c pandas\lib\src\tseries.c -o build\temp.win-amd64-2.6\Release\pandas\lib\src\tseries.o Found executable C:\Python26\Scripts\gcc.exe pandas\lib\src\tseries.c: In function '__Pyx_RaiseArgtupleInvalid': pandas\lib\src\tseries.c:18397:9: warning: unknown conversion type character 'z' in format pandas\lib\src\tseries.c:18397:9: warning: format '%s' expects type 'char *', bu t argument 5 has type 'Py_ssize_t' pandas\lib\src\tseries.c:18397:9: warning: unknown conversion type character 'z' in format pandas\lib\src\tseries.c:18397:9: warning: too many arguments for format pandas\lib\src\tseries.c: In function '__Pyx_RaiseNeedMoreValuesError': pandas\lib\src\tseries.c:18523:18: warning: unknown conversion type character 'z ' in format pandas\lib\src\tseries.c:18523:18: warning: format '%s' expects type 'char *', b ut argument 3 has type 'Py_ssize_t' pandas\lib\src\tseries.c:18523:18: warning: too many arguments for format pandas\lib\src\tseries.c: In function '__Pyx_RaiseTooManyValuesError': pandas\lib\src\tseries.c:18531:13: warning: unknown conversion type character 'z ' in format pandas\lib\src\tseries.c:18531:13: warning: too many arguments for format pandas\lib\src\tseries.c: In function '__Pyx_c_absf': pandas\lib\src\tseries.c:18906:25: error: #if with no expression pandas\lib\src\tseries.c: In function '__Pyx_c_abs': pandas\lib\src\tseries.c:19026:25: error: #if with no expression pandas\lib\src\tseries.c:19028:17: warning: extra tokens at end of #else directi ve pandas\lib\src\tseries.c: At top level: C:\Python26\lib\site-packages\numpy\core\include/numpy/__ufunc_api.h:196:1: warn ing: '_import_umath' defined but not used pandas\lib\src\tseries.c:1589:33: warning: '__pyx_f_7tseries_get_int16_ptr' defi ned but not used pandas\lib\src\tseries.c:1626:33: warning: '__pyx_f_7tseries_get_int32_ptr' defi ned but not used pandas\lib\src\tseries.c:1663:33: warning: '__pyx_f_7tseries_get_int64_ptr' defi ned but not used pandas\lib\src\tseries.c:1700:34: warning: '__pyx_f_7tseries_get_double_ptr' def ined but not used error: Command "gcc -g -DDEBUG -DMS_WIN64 -mno-cygwin -O0 -Wall -Wstrict-prototy pes -IC:\Python26\lib\site-packages\numpy\core\include -IC:\Python26\lib\site-pa ckages\numpy\core\include -IC:\Python26\include -IC:\Python26\PC -c pandas\lib\s rc\tseries.c -o build\temp.win-amd64-2.6\Release\pandas\lib\src\tseries.o" faile d with exit status 1 You can see the file here: https://github.com/wesm/pandas/raw/master/pandas/lib/src/tseries.c Seems it might be related to this: http://gcc.gnu.org/gcc-4.4/porting_to.html I understand you might not be able to fix this immediately, but being a non expert C programmer could anyone tell me what changes are necessary to get this to compile? Or other advice compiling on Windows 64 (I'm using the gcc/g++ shipped with latest EPD 6.3-2)? Cheers, Wes From dalcinl at gmail.com Fri Dec 24 02:53:05 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 23 Dec 2010 22:53:05 -0300 Subject: [Cython] Compilation error with gcc 4.5.2 on Windows 64 In-Reply-To: References: Message-ID: On 23 December 2010 18:37, Wes McKinney wrote: > I'm getting the following error trying to build a Cython > 0.14-generated extension: > I bet you have old code around. I fixed this isue a few commits before release 0.14 https://github.com/cython/cython/commit/29efcd449d933cab1405b98530e6272dea683987 http://hg.cython.org/cython-devel/rev/d8d4e037d690 Could you "rm -rf" your Cython installation and next reinstall 0.14? > gcc -g -DDEBUG -DMS_WIN64 -mno-cygwin -O0 -Wall -Wstrict-prototypes -IC:\Python2 > 6\lib\site-packages\numpy\core\include -IC:\Python26\lib\site-packages\numpy\cor > e\include -IC:\Python26\include -IC:\Python26\PC -c pandas\lib\src\tseries.c -o > build\temp.win-amd64-2.6\Release\pandas\lib\src\tseries.o > Found executable C:\Python26\Scripts\gcc.exe > pandas\lib\src\tseries.c: In function '__Pyx_RaiseArgtupleInvalid': > pandas\lib\src\tseries.c:18397:9: warning: unknown conversion type character 'z' > ?in format > pandas\lib\src\tseries.c:18397:9: warning: format '%s' expects type 'char *', bu > t argument 5 has type 'Py_ssize_t' > pandas\lib\src\tseries.c:18397:9: warning: unknown conversion type character 'z' > ?in format > pandas\lib\src\tseries.c:18397:9: warning: too many arguments for format > pandas\lib\src\tseries.c: In function '__Pyx_RaiseNeedMoreValuesError': > pandas\lib\src\tseries.c:18523:18: warning: unknown conversion type character 'z > ' in format > pandas\lib\src\tseries.c:18523:18: warning: format '%s' expects type 'char *', b > ut argument 3 has type 'Py_ssize_t' > pandas\lib\src\tseries.c:18523:18: warning: too many arguments for format > pandas\lib\src\tseries.c: In function '__Pyx_RaiseTooManyValuesError': > pandas\lib\src\tseries.c:18531:13: warning: unknown conversion type character 'z > ' in format > pandas\lib\src\tseries.c:18531:13: warning: too many arguments for format > pandas\lib\src\tseries.c: In function '__Pyx_c_absf': > pandas\lib\src\tseries.c:18906:25: error: #if with no expression > pandas\lib\src\tseries.c: In function '__Pyx_c_abs': > pandas\lib\src\tseries.c:19026:25: error: #if with no expression > pandas\lib\src\tseries.c:19028:17: warning: extra tokens at end of #else directi > ve > pandas\lib\src\tseries.c: At top level: > C:\Python26\lib\site-packages\numpy\core\include/numpy/__ufunc_api.h:196:1: warn > ing: '_import_umath' defined but not used > pandas\lib\src\tseries.c:1589:33: warning: '__pyx_f_7tseries_get_int16_ptr' defi > ned but not used > pandas\lib\src\tseries.c:1626:33: warning: '__pyx_f_7tseries_get_int32_ptr' defi > ned but not used > pandas\lib\src\tseries.c:1663:33: warning: '__pyx_f_7tseries_get_int64_ptr' defi > ned but not used > pandas\lib\src\tseries.c:1700:34: warning: '__pyx_f_7tseries_get_double_ptr' def > ined but not used > error: Command "gcc -g -DDEBUG -DMS_WIN64 -mno-cygwin -O0 -Wall -Wstrict-prototy > pes -IC:\Python26\lib\site-packages\numpy\core\include -IC:\Python26\lib\site-pa > ckages\numpy\core\include -IC:\Python26\include -IC:\Python26\PC -c pandas\lib\s > rc\tseries.c -o build\temp.win-amd64-2.6\Release\pandas\lib\src\tseries.o" faile > d with exit status 1 > > You can see the file here: > > https://github.com/wesm/pandas/raw/master/pandas/lib/src/tseries.c > > Seems it might be related to this: http://gcc.gnu.org/gcc-4.4/porting_to.html > > I understand you might not be able to fix this immediately, but being > a non expert C programmer could anyone tell me what changes are > necessary to get this to compile? Or other advice compiling on Windows > 64 (I'm using the gcc/g++ shipped with latest EPD 6.3-2)? > > Cheers, > Wes > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From wesmckinn at gmail.com Fri Dec 24 22:22:40 2010 From: wesmckinn at gmail.com (Wes McKinney) Date: Fri, 24 Dec 2010 16:22:40 -0500 Subject: [Cython] Compilation error with gcc 4.5.2 on Windows 64 In-Reply-To: References: Message-ID: On Thu, Dec 23, 2010 at 8:53 PM, Lisandro Dalcin wrote: > On 23 December 2010 18:37, Wes McKinney wrote: >> I'm getting the following error trying to build a Cython >> 0.14-generated extension: >> > > I bet you have old code around. I fixed this isue a few commits before > release 0.14 > > https://github.com/cython/cython/commit/29efcd449d933cab1405b98530e6272dea683987 > > http://hg.cython.org/cython-devel/rev/d8d4e037d690 > > Could you "rm -rf" your Cython installation and next reinstall 0.14? > > >> gcc -g -DDEBUG -DMS_WIN64 -mno-cygwin -O0 -Wall -Wstrict-prototypes -IC:\Python2 >> 6\lib\site-packages\numpy\core\include -IC:\Python26\lib\site-packages\numpy\cor >> e\include -IC:\Python26\include -IC:\Python26\PC -c pandas\lib\src\tseries.c -o >> build\temp.win-amd64-2.6\Release\pandas\lib\src\tseries.o >> Found executable C:\Python26\Scripts\gcc.exe >> pandas\lib\src\tseries.c: In function '__Pyx_RaiseArgtupleInvalid': >> pandas\lib\src\tseries.c:18397:9: warning: unknown conversion type character 'z' >> ?in format >> pandas\lib\src\tseries.c:18397:9: warning: format '%s' expects type 'char *', bu >> t argument 5 has type 'Py_ssize_t' >> pandas\lib\src\tseries.c:18397:9: warning: unknown conversion type character 'z' >> ?in format >> pandas\lib\src\tseries.c:18397:9: warning: too many arguments for format >> pandas\lib\src\tseries.c: In function '__Pyx_RaiseNeedMoreValuesError': >> pandas\lib\src\tseries.c:18523:18: warning: unknown conversion type character 'z >> ' in format >> pandas\lib\src\tseries.c:18523:18: warning: format '%s' expects type 'char *', b >> ut argument 3 has type 'Py_ssize_t' >> pandas\lib\src\tseries.c:18523:18: warning: too many arguments for format >> pandas\lib\src\tseries.c: In function '__Pyx_RaiseTooManyValuesError': >> pandas\lib\src\tseries.c:18531:13: warning: unknown conversion type character 'z >> ' in format >> pandas\lib\src\tseries.c:18531:13: warning: too many arguments for format >> pandas\lib\src\tseries.c: In function '__Pyx_c_absf': >> pandas\lib\src\tseries.c:18906:25: error: #if with no expression >> pandas\lib\src\tseries.c: In function '__Pyx_c_abs': >> pandas\lib\src\tseries.c:19026:25: error: #if with no expression >> pandas\lib\src\tseries.c:19028:17: warning: extra tokens at end of #else directi >> ve >> pandas\lib\src\tseries.c: At top level: >> C:\Python26\lib\site-packages\numpy\core\include/numpy/__ufunc_api.h:196:1: warn >> ing: '_import_umath' defined but not used >> pandas\lib\src\tseries.c:1589:33: warning: '__pyx_f_7tseries_get_int16_ptr' defi >> ned but not used >> pandas\lib\src\tseries.c:1626:33: warning: '__pyx_f_7tseries_get_int32_ptr' defi >> ned but not used >> pandas\lib\src\tseries.c:1663:33: warning: '__pyx_f_7tseries_get_int64_ptr' defi >> ned but not used >> pandas\lib\src\tseries.c:1700:34: warning: '__pyx_f_7tseries_get_double_ptr' def >> ined but not used >> error: Command "gcc -g -DDEBUG -DMS_WIN64 -mno-cygwin -O0 -Wall -Wstrict-prototy >> pes -IC:\Python26\lib\site-packages\numpy\core\include -IC:\Python26\lib\site-pa >> ckages\numpy\core\include -IC:\Python26\include -IC:\Python26\PC -c pandas\lib\s >> rc\tseries.c -o build\temp.win-amd64-2.6\Release\pandas\lib\src\tseries.o" faile >> d with exit status 1 >> >> You can see the file here: >> >> https://github.com/wesm/pandas/raw/master/pandas/lib/src/tseries.c >> >> Seems it might be related to this: http://gcc.gnu.org/gcc-4.4/porting_to.html >> >> I understand you might not be able to fix this immediately, but being >> a non expert C programmer could anyone tell me what changes are >> necessary to get this to compile? Or other advice compiling on Windows >> 64 (I'm using the gcc/g++ shipped with latest EPD 6.3-2)? >> >> Cheers, >> Wes >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Tried this and no luck. I'd say it's definitely a 0.14 bug (I nuked everything tried 0.14 and it failed, then nuked everything again, installed 0.13 and things are back to working). Let me know how I can help debug. Best, Wes From dalcinl at gmail.com Fri Dec 24 23:39:13 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 24 Dec 2010 19:39:13 -0300 Subject: [Cython] Compilation error with gcc 4.5.2 on Windows 64 In-Reply-To: References: Message-ID: On 24 December 2010 18:22, Wes McKinney wrote: > > Tried this and no luck. I'd say it's definitely a 0.14 bug (I nuked > everything tried 0.14 and it failed, then nuked everything again, > installed 0.13 and things are back to working). > > Let me know how I can help debug. > OK, now I understand what's going on. I seems Cython 0.14 do not actually have the fix I was talking about. Any chance you could install cython from github repo and give another try? Alternatively, you could apply the patch below to your copy of 0.14 and rebuild+install Cython. $ git show 29efcd449d933cab1405b98530e6272dea683987 commit 29efcd449d933cab1405b98530e6272dea683987 Author: Lisandro Dalcin Date: Mon Dec 13 20:10:55 2010 -0300 MSVC: fixes for complex types diff --git a/Cython/Compiler/PyrexTypes.py b/Cython/Compiler/PyrexTypes.py index 38df20d..bde7356 100755 --- a/Cython/Compiler/PyrexTypes.py +++ b/Cython/Compiler/PyrexTypes.py @@ -1371,10 +1371,10 @@ impl=""" } #if %(is_float)s static CYTHON_INLINE %(real_type)s __Pyx_c_abs%(m)s(%(type)s z) { - #if HAVE_HYPOT - return hypot%(m)s(z.real, z.imag); - #else + #if !defined(HAVE_HYPOT) || defined(_MSC_VER) return sqrt%(m)s(z.real*z.real + z.imag*z.imag); + #else + return hypot%(m)s(z.real, z.imag); #endif } static CYTHON_INLINE %(type)s __Pyx_c_pow%(m)s(%(type)s a, %(type)s b) { diff --git a/tests/run/complex_numbers_T305.pyx b/tests/run/complex_numbers_T305.pyx index f743330..9bcafe7 100644 --- a/tests/run/complex_numbers_T305.pyx +++ b/tests/run/complex_numbers_T305.pyx @@ -28,8 +28,8 @@ def test_pow(double complex z, double complex w, tol=None): Various implementations produce slightly different results... >>> a = complex(3, 1) - >>> test_pow(a, 1) - (3+1j) + >>> test_pow(a, 1, 1e-15) + True >>> test_pow(a, 2, 1e-15) True >>> test_pow(a, a, 1e-15) @@ -48,7 +48,7 @@ def test_int_pow(double complex z, int n, tol=None): [True, True, True, True, True, True, True, True, True] >>> [test_int_pow(complex(0, 2), k, 1e-15) for k in range(-4, 5)] [True, True, True, True, True, True, True, True, True] - >>> [test_int_pow(complex(2, 0.5), k, 1e-15) for k in range(0, 10)] + >>> [test_int_pow(complex(2, 0.5), k, 1e-14) for k in range(0, 10)] [True, True, True, True, True, True, True, True, True, True] """ if tol is None: -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From wesmckinn at gmail.com Sat Dec 25 18:03:23 2010 From: wesmckinn at gmail.com (Wes McKinney) Date: Sat, 25 Dec 2010 12:03:23 -0500 Subject: [Cython] Compilation error with gcc 4.5.2 on Windows 64 In-Reply-To: References: Message-ID: On Fri, Dec 24, 2010 at 5:39 PM, Lisandro Dalcin wrote: > On 24 December 2010 18:22, Wes McKinney wrote: >> >> Tried this and no luck. I'd say it's definitely a 0.14 bug (I nuked >> everything tried 0.14 and it failed, then nuked everything again, >> installed 0.13 and things are back to working). >> >> Let me know how I can help debug. >> > > OK, now I understand what's going on. I seems Cython 0.14 do not > actually have the fix I was talking about. Any chance you could > install cython from github repo and give another try? > > Alternatively, you could apply the patch below to your copy of ?0.14 > and rebuild+install Cython. > > > $ git show 29efcd449d933cab1405b98530e6272dea683987 > commit 29efcd449d933cab1405b98530e6272dea683987 > Author: Lisandro Dalcin > Date: ? Mon Dec 13 20:10:55 2010 -0300 > > ? ?MSVC: fixes for complex types > > diff --git a/Cython/Compiler/PyrexTypes.py b/Cython/Compiler/PyrexTypes.py > index 38df20d..bde7356 100755 > --- a/Cython/Compiler/PyrexTypes.py > +++ b/Cython/Compiler/PyrexTypes.py > @@ -1371,10 +1371,10 @@ impl=""" > ? ? } > ? ? #if %(is_float)s > ? ? ? ? static CYTHON_INLINE %(real_type)s __Pyx_c_abs%(m)s(%(type)s z) { > - ? ? ? ? ?#if HAVE_HYPOT > - ? ? ? ? ? ?return hypot%(m)s(z.real, z.imag); > - ? ? ? ? ?#else > + ? ? ? ? ?#if !defined(HAVE_HYPOT) || defined(_MSC_VER) > ? ? ? ? ? ? return sqrt%(m)s(z.real*z.real + z.imag*z.imag); > + ? ? ? ? ?#else > + ? ? ? ? ? ?return hypot%(m)s(z.real, z.imag); > ? ? ? ? ? #endif > ? ? ? ? } > ? ? ? ? static CYTHON_INLINE %(type)s __Pyx_c_pow%(m)s(%(type)s a, > %(type)s b) { > diff --git a/tests/run/complex_numbers_T305.pyx > b/tests/run/complex_numbers_T305.pyx > index f743330..9bcafe7 100644 > --- a/tests/run/complex_numbers_T305.pyx > +++ b/tests/run/complex_numbers_T305.pyx > @@ -28,8 +28,8 @@ def test_pow(double complex z, double complex w, tol=None): > ? ? Various implementations produce slightly different results... > > ? ? >>> a = complex(3, 1) > - ? ?>>> test_pow(a, 1) > - ? ?(3+1j) > + ? ?>>> test_pow(a, 1, 1e-15) > + ? ?True > ? ? >>> test_pow(a, 2, 1e-15) > ? ? True > ? ? >>> test_pow(a, a, 1e-15) > @@ -48,7 +48,7 @@ def test_int_pow(double complex z, int n, tol=None): > ? ? [True, True, True, True, True, True, True, True, True] > ? ? >>> [test_int_pow(complex(0, 2), k, 1e-15) for k in range(-4, 5)] > ? ? [True, True, True, True, True, True, True, True, True] > - ? ?>>> [test_int_pow(complex(2, 0.5), k, 1e-15) for k in range(0, 10)] > + ? ?>>> [test_int_pow(complex(2, 0.5), k, 1e-14) for k in range(0, 10)] > ? ? [True, True, True, True, True, True, True, True, True, True] > ? ? """ > ? ? if tol is None: > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Works fine for me with the git HEAD-- maybe time for a bugfix release? :) I suspect that quite a few Windows users will be affected by this. From stefan_ml at behnel.de Sun Dec 26 09:46:48 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 26 Dec 2010 09:46:48 +0100 Subject: [Cython] Cython 0.14.1 Message-ID: <4D1700F8.6070605@behnel.de> Hi, regarding the next release, I think we should fix the ref-counting bug that Martijn Meijers found (ticket #633) and then release 0.14.1 ASAP from the current git tip, which already has a couple of other bug fixes. I'll see if I can free some time to work on a fix next week, but wouldn't mind if someone beat me to it. Stefan From arfrever.fta at gmail.com Sun Dec 26 12:58:07 2010 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sun, 26 Dec 2010 12:58:07 +0100 Subject: [Cython] Cython 0.14.1 In-Reply-To: <4D1700F8.6070605@behnel.de> References: <4D1700F8.6070605@behnel.de> Message-ID: <201012261258.42840.Arfrever.FTA@gmail.com> 2010-12-26 09:46:48 Stefan Behnel napisa?(a): > regarding the next release, I think we should fix the ref-counting bug that > Martijn Meijers found (ticket #633) and then release 0.14.1 ASAP from the > current git tip, which already has a couple of other bug fixes. It would be nice if compatibility with Numpy 1.5 was fixed (ticket #630). -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/cython-dev/attachments/20101226/43fc02ed/attachment.pgp From stefan_ml at behnel.de Mon Dec 27 09:07:30 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Dec 2010 09:07:30 +0100 Subject: [Cython] Cython 0.14.1 In-Reply-To: <201012261258.42840.Arfrever.FTA@gmail.com> References: <4D1700F8.6070605@behnel.de> <201012261258.42840.Arfrever.FTA@gmail.com> Message-ID: <4D184942.4010405@behnel.de> Arfrever Frehtes Taifersar Arahesis, 26.12.2010 12:58: > 2010-12-26 09:46:48 Stefan Behnel napisa?(a): >> regarding the next release, I think we should fix the ref-counting bug that >> Martijn Meijers found (ticket #633) and then release 0.14.1 ASAP from the >> current git tip, which already has a couple of other bug fixes. > > It would be nice if compatibility with Numpy 1.5 was fixed (ticket #630). IMHO, that's not the same level of criticality as a serious crash bug, so unless someone provides a ready-to-review-and-merge patch for #630, I'd prefer getting 0.14.1 out without waiting for a fix. Stefan From stefan_ml at behnel.de Mon Dec 27 11:31:37 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Dec 2010 11:31:37 +0100 Subject: [Cython] Hg to git In-Reply-To: References: <4D108B3A.7090901@behnel.de> Message-ID: <4D186B09.7030600@behnel.de> Robert Bradshaw, 21.12.2010 18:32: > On Tue, Dec 21, 2010 at 3:10 AM, Stefan Behnel wrote: >> Robert Bradshaw, 21.12.2010 07:12: >>> Anyone object to making http://hg.cython.org/ read-only, and making >>> https://github.com/cython/cython/ (and its forks) the live devel >>> branch? >> >> I think it's a good time to do this now that 0.14 is out. > > Yep. Done. Hudson also builds from the git repo now. Stefan From robertwb at math.washington.edu Mon Dec 27 17:32:58 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 27 Dec 2010 08:32:58 -0800 Subject: [Cython] Hg to git In-Reply-To: <4D186B09.7030600@behnel.de> References: <4D108B3A.7090901@behnel.de> <4D186B09.7030600@behnel.de> Message-ID: On Mon, Dec 27, 2010 at 2:31 AM, Stefan Behnel wrote: > Robert Bradshaw, 21.12.2010 18:32: >> On Tue, Dec 21, 2010 at 3:10 AM, Stefan Behnel ?wrote: >>> Robert Bradshaw, 21.12.2010 07:12: >>>> Anyone object to making http://hg.cython.org/ read-only, and making >>>> https://github.com/cython/cython/ (and its forks) the live devel >>>> branch? >>> >>> I think it's a good time to do this now that 0.14 is out. >> >> Yep. Done. > > Hudson also builds from the git repo now. Excellent, thanks. Ironically, I was just about to do this myself this morning. - Robert From robertwb at math.washington.edu Mon Dec 27 17:36:47 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 27 Dec 2010 08:36:47 -0800 Subject: [Cython] Cython 0.14.1 In-Reply-To: <4D184942.4010405@behnel.de> References: <4D1700F8.6070605@behnel.de> <201012261258.42840.Arfrever.FTA@gmail.com> <4D184942.4010405@behnel.de> Message-ID: On Mon, Dec 27, 2010 at 12:07 AM, Stefan Behnel wrote: > Arfrever Frehtes Taifersar Arahesis, 26.12.2010 12:58: >> 2010-12-26 09:46:48 Stefan Behnel napisa?(a): >>> regarding the next release, I think we should fix the ref-counting bug that >>> Martijn Meijers found (ticket #633) and then release 0.14.1 ASAP from the >>> current git tip, which already has a couple of other bug fixes. My thoughts exactly. >> It would be nice if compatibility with Numpy 1.5 was fixed (ticket #630). > > IMHO, that's not the same level of criticality as a serious crash bug, so > unless someone provides a ready-to-review-and-merge patch for #630, I'd > prefer getting 0.14.1 out without waiting for a fix. I agree. There's a fair number of fixes in the tree already, so I was actually thinking about doing a 0.14.1 release soon anyways. Of course something big like this necessitates a quicker release, but how about we see what fixes we can get in this next release and release first thing in January? - Robert From robertwb at math.washington.edu Mon Dec 27 23:54:33 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 27 Dec 2010 14:54:33 -0800 Subject: [Cython] Broken .pyxdep files In-Reply-To: References: Message-ID: On Wed, Dec 22, 2010 at 12:48 AM, Robert Bradshaw wrote: > On Tue, Dec 21, 2010 at 11:49 PM, Alexey Borzenkov wrote: >> On Wed, Dec 22, 2010 at 1:19 AM, Alexey Borzenkov wrote: >>> I've recently started using Cython in one of my projects because I >>> needed pyximport and automatic building of extensions and found that >>> adding .pyxdep files makes Cython raise NameError because _test_files >>> is not defined. >> [...] >>> Also, there's another problem with pyximport that makes it unsafe to >>> use it without build_in_temp=False. For example, if I have >>> package1/module.pyx and package2/module.pyx then only one of those >>> will build/import correctly, >> >> Btw, I just realized that Cython is finally using git and github, so >> if anyone is interested here are my patches: >> >> https://github.com/snaury/cython/commit/b43f68280919e98767bd4d163cb1eaf88c4c857e >> https://github.com/snaury/cython/commit/6f73b4280958f5d3f5fa77c3a4ed19b277784140 > > Thanks. pyximport is really cool, but no one's been maintaining it, > and it looks like we haven't been testing it either. (Code is either > new, tested, or broken...) At first glance the patches look fine, I'll > try to take a deeper look over the holidays (if no one beats me to > it...) Merged. Thanks! - Robert From stefan_ml at behnel.de Tue Dec 28 08:25:17 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 28 Dec 2010 08:25:17 +0100 Subject: [Cython] Cython 0.14.1 In-Reply-To: References: <4D1700F8.6070605@behnel.de> <201012261258.42840.Arfrever.FTA@gmail.com> <4D184942.4010405@behnel.de> Message-ID: <4D1990DD.6030705@behnel.de> Robert Bradshaw, 27.12.2010 17:36: > On Mon, Dec 27, 2010 at 12:07 AM, Stefan Behnel wrote: >> Arfrever Frehtes Taifersar Arahesis, 26.12.2010 12:58: >>> 2010-12-26 09:46:48 Stefan Behnel napisa?(a): >>>> regarding the next release, I think we should fix the ref-counting bug that >>>> Martijn Meijers found (ticket #633) and then release 0.14.1 ASAP from the >>>> current git tip, which already has a couple of other bug fixes. > > My thoughts exactly. > >>> It would be nice if compatibility with Numpy 1.5 was fixed (ticket #630). >> >> IMHO, that's not the same level of criticality as a serious crash bug, so >> unless someone provides a ready-to-review-and-merge patch for #630, I'd >> prefer getting 0.14.1 out without waiting for a fix. > > I agree. > > There's a fair number of fixes in the tree already, so I was actually > thinking about doing a 0.14.1 release soon anyways. Of course > something big like this necessitates a quicker release, but how about > we see what fixes we can get in this next release and release first > thing in January? +1, that should be close enough. Stefan From vitja.makarov at gmail.com Thu Dec 30 18:53:17 2010 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 30 Dec 2010 20:53:17 +0300 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4CFC91E6.8020604@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> <4D077CA5.4010103@behnel.de> <4D078717.4050105@behnel.de> Message-ID: 2010/12/14 Vitja Makarov : > 2010/12/14 Stefan Behnel : >> Vitja Makarov, 14.12.2010 15:49: >>> 2010/12/14 Stefan Behnel: >>>> Vitja Makarov, 14.12.2010 14:45: >>>>> 2010/12/14 Stefan Behnel: >>>>>> Stefan Behnel, 13.12.2010 06:37: >>>>>>> Vitja Makarov, 12.12.2010 20:23: >>>>>>>> What do you think about GeneratorWrapperNode? >>>>>>> >>>>>>> It seems ok. I'll give it a deeper look ASAP. >>>>>> >>>>>> Actually, after looking through it, I think it's an ugly hack. It tries to >>>>>> hide the fact that the DefNode does the wrong thing for the two faces of >>>>>> generators, and that the current implementation does not build a separate >>>>>> node at transform time that does the right thing instead. IMHO, two nodes >>>>>> are needed. A CFuncDefNode (or a subtype) that writes out the generator C >>>>>> function with the right signature and the original body, and the original >>>>>> DefNode (with the original signature) that has its body replaced by the >>>>>> necessary generator setup code. That's basically why the original CEP >>>>>> proposed to split the DefNode implementation *before* going for generators. >>>>>> Would have made this easier. >>>>> >>>>> Are you sure that CFuncDefNode for generator body is a good idea? >>>>> >>>>> Nested function and classes, lambda, generators are not supported >>>>> inside CFuncDefNode. >>>> >>>> Hmm, right, that's a missing feature that could become a problem. >>>> >>>> Second try. Create a new subtype of DefNode that nails the signature to >>>> METH_O and additionally implements all the generator specific stuff, >>>> potentially through sensible hooks in DefNode's code generation. >>>> >>> >>> What is the best place to create node for generator body? >> >> "Create" as in "implement"? In Nodes.py. >> >> "Create" as in "instantiate"? Depends. The canonical answer is "in a >> transform", but sometimes it's easier to do it inside of a node. If the >> latter, the canonical answer would become analyse_types(). If that's too >> late, well, back to a transform. :) >> > > I think analyse_types() should be ok, and it wouldn't break things > Why did you reverted decorators stuff? https://github.com/vitek/cython/commit/e0d366d9409680849e6f429992ac9724e2ad1016 -- vitja. From stefan_ml at behnel.de Thu Dec 30 19:05:40 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 30 Dec 2010 19:05:40 +0100 Subject: [Cython] General generator expressions In-Reply-To: References: <4CFBD711.5060206@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> <4D077CA5.4010103@behnel.de> <4D078717.4050105@behnel.de> Message-ID: <4D1CC9F4.60408@behnel.de> Vitja Makarov, 30.12.2010 18:53: > Why did you reverted decorators stuff? > > https://github.com/vitek/cython/commit/e0d366d9409680849e6f429992ac9724e2ad1016 Because I didn't get it finished, it broke the Sage build and cython-devel should stay cleanly working before the release. I added a link to the build failure log to the trac ticket but didn't even manage to come up with a suitable test case before I had to stop working on it. What I implemented so far works in most cases but there are issues with functions at the module scope. I didn't check the history but I think I recall from the code comments that Robert worked in this area (method binding) a while ago, and the comments hint at an unfinished refactoring that would help with this change. Stefan From robertwb at math.washington.edu Fri Dec 31 08:39:35 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 30 Dec 2010 23:39:35 -0800 Subject: [Cython] General generator expressions In-Reply-To: <4D1CC9F4.60408@behnel.de> References: <4CFBD711.5060206@behnel.de> <4D00A586.8060801@behnel.de> <4D01D7EF.6060906@behnel.de> <4D02BB12.7060306@behnel.de> <4D05B116.2020900@behnel.de> <4D06930B.2090205@behnel.de> <4D077CA5.4010103@behnel.de> <4D078717.4050105@behnel.de> <4D1CC9F4.60408@behnel.de> Message-ID: On Thu, Dec 30, 2010 at 10:05 AM, Stefan Behnel wrote: > Vitja Makarov, 30.12.2010 18:53: >> Why did you reverted decorators stuff? >> >> https://github.com/vitek/cython/commit/e0d366d9409680849e6f429992ac9724e2ad1016 > > Because I didn't get it finished, it broke the Sage build and cython-devel > should stay cleanly working before the release. +1 > I added a link to the build > failure log to the trac ticket but didn't even manage to come up with a > suitable test case before I had to stop working on it. > > What I implemented so far works in most cases but there are issues with > functions at the module scope. I didn't check the history but I think I > recall from the code comments that Robert worked in this area (method > binding) a while ago, and the comments hint at an unfinished refactoring > that would help with this change. I'm not working on anything in the area at the moment though (or, contrary to my expectations, much of anything Cython-wise... having too much fun hanging out with the family :). - Robert