From strombrg at gmail.com Tue Jun 1 06:48:47 2010 From: strombrg at gmail.com (Dan Stromberg) Date: Mon, 31 May 2010 21:48:47 -0700 Subject: [Cython] Exceptions? Message-ID: <4C04912F.8000408@gmail.com> I'm attempting to get a Cython module to raise exceptions that'll be visible to the calling CPython. The Cython code in question looks like: def add_from_fileno(self, fileno, length_to_add): exception_string = self.add_from_fileno_c(fileno, length_to_add) if exception_string.startswith("Buffer error"): raise exceptions.BufferError, exception_string elif exception_string == '': pass else: raise exceptions.AssertionError, exception_string cdef add_from_fileno_c(self, fileno, length_to_add): if self.would_overflow(length_to_add): return "Buffer error: Would overflow" # FIXME: We need to do something about EINTR on these 3 read's! if self.would_add_wrap(length_to_add): # do the read in two pieces first_length, second_length = self.split_to_two_add(length_to_add) length_added = read(fileno, &self.buffer[self.end], first_length) if length_added != first_length: return "length_added != first_length" length_added = read(fileno, self.buffer, second_length) if length_added != second_length: return "length_added != second_length" self.end = second_length else: # do the read in one piece length_added = read(fileno, &self.buffer[self.end], length_to_add) if length_added != length_to_add: return "length_added != length_to_add" self.end += length_to_add return '' ...and in the caller (CPython code, inheriting from unittest): def test_fileno_from_file_overflow(self): file_ = open('input-file', 'w') file_.write('abc' * 15) file_.close() rb = ring_buffer_mod.Ring_buffer(buffer_size = 10) input_fileno = os.open('input-file', os.O_RDONLY) for i in xrange(10): rb.add_from_fileno(input_fileno, 1) #self.assertRaises(exceptions.BufferError, rb.add_from_fileno, input_fileno, 1) try: rb.add_from_fileno(input_fileno, 1) except exceptions.BufferError: pass All seems fine, except for the exceptions. When I raise an exception in Cython, I see a message on my terminal (I believe there should be none), and the calling CPython code doesn't appear to realize that an exception has been raised. I added the add_from_fileno_c/add_from_file distinction, because I was hoping that a def would be able to raise an exception, after finding that cdef's and cpdef's seemed to have problems with it - but it appears that there's some sort of exception barrier between Cython and CPython. I've googled quite a bit, but haven't found much on the topic that didn't seem kind of hand wavey. What do I need to do, to raise an exception in Cython, that CPython code will be able to see? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100531/cbee1be8/attachment-0001.htm From robertwb at math.washington.edu Tue Jun 1 07:21:16 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 31 May 2010 22:21:16 -0700 Subject: [Cython] Exceptions? In-Reply-To: <4C04912F.8000408@gmail.com> References: <4C04912F.8000408@gmail.com> Message-ID: On May 31, 2010, at 9:48 PM, Dan Stromberg wrote: > > I'm attempting to get a Cython module to raise exceptions that'll be > visible to the calling CPython. > > The Cython code in question looks like: > def add_from_fileno(self, fileno, length_to_add): > exception_string = self.add_from_fileno_c(fileno, length_to_add) > if exception_string.startswith("Buffer error"): > raise exceptions.BufferError, exception_string > elif exception_string == '': > pass > else: > raise exceptions.AssertionError, exception_string > > cdef add_from_fileno_c(self, fileno, length_to_add): > if self.would_overflow(length_to_add): > return "Buffer error: Would overflow" > # FIXME: We need to do something about EINTR on these 3 read's! > if self.would_add_wrap(length_to_add): > # do the read in two pieces > first_length, second_length = > self.split_to_two_add(length_to_add) > length_added = read(fileno, &self.buffer[self.end], > first_length) > if length_added != first_length: > return "length_added != first_length" > length_added = read(fileno, self.buffer, second_length) > if length_added != second_length: > return "length_added != second_length" > self.end = second_length > else: > # do the read in one piece > length_added = read(fileno, &self.buffer[self.end], > length_to_add) > if length_added != length_to_add: > return "length_added != length_to_add" > self.end += length_to_add > return '' > ...and in the caller (CPython code, inheriting from unittest): > def test_fileno_from_file_overflow(self): > file_ = open('input-file', 'w') > file_.write('abc' * 15) > file_.close() > > rb = ring_buffer_mod.Ring_buffer(buffer_size = 10) > input_fileno = os.open('input-file', os.O_RDONLY) > for i in xrange(10): > rb.add_from_fileno(input_fileno, 1) > #self.assertRaises(exceptions.BufferError, rb.add_from_fileno, > input_fileno, 1) > try: > rb.add_from_fileno(input_fileno, 1) > except exceptions.BufferError: > pass > > All seems fine, except for the exceptions. When I raise an > exception in Cython, I see a message on my terminal (I believe there > should be none), and the calling CPython code doesn't appear to > realize that an exception has been raised. I added the > add_from_fileno_c/add_from_file distinction, because I was hoping > that a def would be able to raise an exception, after finding that > cdef's and cpdef's seemed to have problems with it - but it appears > that there's some sort of exception barrier between Cython and > CPython. > > I've googled quite a bit, but haven't found much on the topic that > didn't seem kind of hand wavey. > > What do I need to do, to raise an exception in Cython, that CPython > code will be able to see? This should work just fine for def functions, as well as c(p)def functions not declaring a c return type (in which case you should see [1]). Have you tried calling this directly from the command line (eliminating the possibility that it's something with the unittest framework?) What if you do cdef class A: def spam(self): raise TypeError cpdef eggs(self): raise ValueError Does that work for you? (It does for me.) - Robert [1] http://docs.cython.org/src/userguide/language_basics.html#error-return-values From stefan_ml at behnel.de Tue Jun 1 08:12:18 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 01 Jun 2010 08:12:18 +0200 Subject: [Cython] Exceptions? In-Reply-To: <4C04912F.8000408@gmail.com> References: <4C04912F.8000408@gmail.com> Message-ID: <4C04A4C2.7050401@behnel.de> Dan Stromberg, 01.06.2010 06:48: > All seems fine, except for the exceptions. When I raise an exception in > Cython, I see a message on my terminal (I believe there should be none), I wonder why you didn't think of presenting that "message on your terminal" here, so that we know what it says and can /tell/ you if there really "should be none". > I added the add_from_fileno_c/add_from_file > distinction, because I was hoping that a def would be able to raise an > exception, after finding that cdef's and cpdef's seemed to have problems > with it - but it appears that there's some sort of exception barrier > between Cython and CPython. No, there isn't. There may be one in your code if you use cdef functions, as those can only propagate exceptions when a) their return type implicitly allows it or b) they are explicitly declared to raise an exception. You should read the Cython docs on exceptions. Also note that this question would have been more appropriate for the cython-users mailing list. Stefan From stefan_ml at behnel.de Tue Jun 1 13:30:59 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 01 Jun 2010 13:30:59 +0200 Subject: [Cython] Hudson jobs for Haoyu's GSoC Message-ID: <4C04EF73.5060708@behnel.de> Hi, I set up the Hudson jobs for the gsoc branch: https://sage.math.washington.edu:8091/hudson/view/cython-haoyu Happy coding, Stefan From strombrg at gmail.com Wed Jun 2 05:43:20 2010 From: strombrg at gmail.com (Dan Stromberg) Date: Tue, 01 Jun 2010 20:43:20 -0700 Subject: [Cython] Exceptions? In-Reply-To: References: <4C04912F.8000408@gmail.com> Message-ID: <4C05D358.5070406@gmail.com> On 05/31/2010 10:21 PM, Robert Bradshaw wrote: > On May 31, 2010, at 9:48 PM, Dan Stromberg wrote: > > >> I'm attempting to get a Cython module to raise exceptions that'll be >> visible to the calling CPython. >> >> The Cython code in question looks like: >> def add_from_fileno(self, fileno, length_to_add): >> exception_string = self.add_from_fileno_c(fileno, length_to_add) >> if exception_string.startswith("Buffer error"): >> raise exceptions.BufferError, exception_string >> elif exception_string == '': >> pass >> else: >> raise exceptions.AssertionError, exception_string >> >> cdef add_from_fileno_c(self, fileno, length_to_add): >> if self.would_overflow(length_to_add): >> return "Buffer error: Would overflow" >> # FIXME: We need to do something about EINTR on these 3 read's! >> if self.would_add_wrap(length_to_add): >> # do the read in two pieces >> first_length, second_length = >> self.split_to_two_add(length_to_add) >> length_added = read(fileno,&self.buffer[self.end], >> first_length) >> if length_added != first_length: >> return "length_added != first_length" >> length_added = read(fileno, self.buffer, second_length) >> if length_added != second_length: >> return "length_added != second_length" >> self.end = second_length >> else: >> # do the read in one piece >> length_added = read(fileno,&self.buffer[self.end], >> length_to_add) >> if length_added != length_to_add: >> return "length_added != length_to_add" >> self.end += length_to_add >> return '' >> ...and in the caller (CPython code, inheriting from unittest): >> def test_fileno_from_file_overflow(self): >> file_ = open('input-file', 'w') >> file_.write('abc' * 15) >> file_.close() >> >> rb = ring_buffer_mod.Ring_buffer(buffer_size = 10) >> input_fileno = os.open('input-file', os.O_RDONLY) >> for i in xrange(10): >> rb.add_from_fileno(input_fileno, 1) >> #self.assertRaises(exceptions.BufferError, rb.add_from_fileno, >> input_fileno, 1) >> try: >> rb.add_from_fileno(input_fileno, 1) >> except exceptions.BufferError: >> pass >> >> All seems fine, except for the exceptions. When I raise an >> exception in Cython, I see a message on my terminal (I believe there >> should be none), and the calling CPython code doesn't appear to >> realize that an exception has been raised. I added the >> add_from_fileno_c/add_from_file distinction, because I was hoping >> that a def would be able to raise an exception, after finding that >> cdef's and cpdef's seemed to have problems with it - but it appears >> that there's some sort of exception barrier between Cython and >> CPython. >> >> I've googled quite a bit, but haven't found much on the topic that >> didn't seem kind of hand wavey. >> >> What do I need to do, to raise an exception in Cython, that CPython >> code will be able to see? >> > This should work just fine for def functions, as well as c(p)def > functions not declaring a c return type (in which case you should see > [1]). Have you tried calling this directly from the command line > (eliminating the possibility that it's something with the unittest > framework?) What if you do > > cdef class A: > def spam(self): > raise TypeError > cpdef eggs(self): > raise ValueError > > Does that work for you? (It does for me.) > > - Robert > > [1] http://docs.cython.org/src/userguide/language_basics.html#error-return-values > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > You're right - it must be something about the unittest module - for some reason, it's printing the exception I'm trying to catch. It works as expected interactively. Apologies for not going to the cython-users list - I wasn't aware of it when I sent my message, and had subscribed to cython-dev (only) long ago. Thanks for the cool software and assistance! PS: I might suggest adding something to the FAQ about cdef functions not declaring a return type doing this well. It's kind of there, but kind of not - and in a FAQ, it's perhaps best to have it fully there. From dagss at student.matnat.uio.no Wed Jun 2 08:34:08 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Jun 2010 08:34:08 +0200 Subject: [Cython] Exceptions? In-Reply-To: <4C05D358.5070406@gmail.com> References: <4C04912F.8000408@gmail.com> <4C05D358.5070406@gmail.com> Message-ID: <4C05FB60.8070607@student.matnat.uio.no> Dan Stromberg wrote: > On 05/31/2010 10:21 PM, Robert Bradshaw wrote: > >> On May 31, 2010, at 9:48 PM, Dan Stromberg wrote: >> >> >> >>> I'm attempting to get a Cython module to raise exceptions that'll be >>> visible to the calling CPython. >>> >>> The Cython code in question looks like: >>> def add_from_fileno(self, fileno, length_to_add): >>> exception_string = self.add_from_fileno_c(fileno, length_to_add) >>> if exception_string.startswith("Buffer error"): >>> raise exceptions.BufferError, exception_string >>> elif exception_string == '': >>> pass >>> else: >>> raise exceptions.AssertionError, exception_string >>> >>> cdef add_from_fileno_c(self, fileno, length_to_add): >>> if self.would_overflow(length_to_add): >>> return "Buffer error: Would overflow" >>> # FIXME: We need to do something about EINTR on these 3 read's! >>> if self.would_add_wrap(length_to_add): >>> # do the read in two pieces >>> first_length, second_length = >>> self.split_to_two_add(length_to_add) >>> length_added = read(fileno,&self.buffer[self.end], >>> first_length) >>> if length_added != first_length: >>> return "length_added != first_length" >>> length_added = read(fileno, self.buffer, second_length) >>> if length_added != second_length: >>> return "length_added != second_length" >>> self.end = second_length >>> else: >>> # do the read in one piece >>> length_added = read(fileno,&self.buffer[self.end], >>> length_to_add) >>> if length_added != length_to_add: >>> return "length_added != length_to_add" >>> self.end += length_to_add >>> return '' >>> ...and in the caller (CPython code, inheriting from unittest): >>> def test_fileno_from_file_overflow(self): >>> file_ = open('input-file', 'w') >>> file_.write('abc' * 15) >>> file_.close() >>> >>> rb = ring_buffer_mod.Ring_buffer(buffer_size = 10) >>> input_fileno = os.open('input-file', os.O_RDONLY) >>> for i in xrange(10): >>> rb.add_from_fileno(input_fileno, 1) >>> #self.assertRaises(exceptions.BufferError, rb.add_from_fileno, >>> input_fileno, 1) >>> try: >>> rb.add_from_fileno(input_fileno, 1) >>> except exceptions.BufferError: >>> pass >>> >>> All seems fine, except for the exceptions. When I raise an >>> exception in Cython, I see a message on my terminal (I believe there >>> should be none), and the calling CPython code doesn't appear to >>> realize that an exception has been raised. I added the >>> add_from_fileno_c/add_from_file distinction, because I was hoping >>> that a def would be able to raise an exception, after finding that >>> cdef's and cpdef's seemed to have problems with it - but it appears >>> that there's some sort of exception barrier between Cython and >>> CPython. >>> >>> I've googled quite a bit, but haven't found much on the topic that >>> didn't seem kind of hand wavey. >>> >>> What do I need to do, to raise an exception in Cython, that CPython >>> code will be able to see? >>> >>> >> This should work just fine for def functions, as well as c(p)def >> functions not declaring a c return type (in which case you should see >> [1]). Have you tried calling this directly from the command line >> (eliminating the possibility that it's something with the unittest >> framework?) What if you do >> >> cdef class A: >> def spam(self): >> raise TypeError >> cpdef eggs(self): >> raise ValueError >> >> Does that work for you? (It does for me.) >> >> - Robert >> >> [1] http://docs.cython.org/src/userguide/language_basics.html#error-return-values >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> >> > You're right - it must be something about the unittest module - for some > reason, it's printing the exception I'm trying to catch. It works as > expected interactively. > > Apologies for not going to the cython-users list - I wasn't aware of it > when I sent my message, and had subscribed to cython-dev (only) long ago. > > Thanks for the cool software and assistance! > > PS: I might suggest adding something to the FAQ about cdef functions not > declaring a return type doing this well. It's kind of there, but kind > of not - and in a FAQ, it's perhaps best to have it fully there. > Please go ahead! Dag From cb at pdos.csail.mit.edu Tue Jun 8 15:23:45 2010 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Tue, 8 Jun 2010 09:23:45 -0400 Subject: [Cython] line-directives support for Distutils build_ext Message-ID: <20100608132345.GA20423@pdos.lcs.mit.edu> Hey, guys. I did this a long time ago, but it patches cleanly against the cython-devel tip (at least yesterday's tip). There was some discussion of what the default should be for the #line/#file directives Robert added. This is a hopefully non-contentious addition to support pushing that decision to higher levels of the build system. All this patch does is add pyrex_line_directives to support that. It's an analogue of the extra attributes .pyrex_include_dirs going with -I, or .pyrex_cplus going with the --cplus command option, etc. This allows you to add .pyrex_line_directives to Extension() objects in your setup.py. The appropriate command option is then added to the invocations of cython for those extensions, as it is for the -I/--cplus/.. options. -------------------------------------------------------------------------- diff -r dd29215de4a0 Cython/Distutils/build_ext.py --- a/Cython/Distutils/build_ext.py Mon Jun 07 11:06:45 2010 -0700 +++ b/Cython/Distutils/build_ext.py Mon Jun 07 16:45:43 2010 -0400 @@ -36,6 +36,8 @@ "generate C++ source files"), ('pyrex-create-listing', None, "write errors to a listing file"), + ('pyrex-line-directives', None, + "emit source line directives"), ('pyrex-include-dirs=', None, "path to the Cython include files" + sep_by), ('pyrex-c-in-temp', None, @@ -45,13 +47,14 @@ ]) boolean_options.extend([ - 'pyrex-cplus', 'pyrex-create-listing', 'pyrex-c-in-temp' + 'pyrex-cplus', 'pyrex-create-listing', 'pyrex-line-directives', 'pyrex-c-in-temp' ]) def initialize_options(self): _build_ext.build_ext.initialize_options(self) self.pyrex_cplus = 0 self.pyrex_create_listing = 0 + self.pyrex_line_directives = 0 self.pyrex_include_dirs = None self.pyrex_c_in_temp = 0 self.pyrex_gen_pxi = 0 @@ -114,6 +117,8 @@ create_listing = self.pyrex_create_listing or \ getattr(extension, 'pyrex_create_listing', 0) + line_directives = self.pyrex_line_directives or \ + getattr(extension, 'pyrex_line_directives', 0) cplus = self.pyrex_cplus or getattr(extension, 'pyrex_cplus', 0) or \ (extension.language and extension.language.lower() == 'c++') pyrex_gen_pxi = self.pyrex_gen_pxi or getattr(extension, 'pyrex_gen_pxi', 0) @@ -186,6 +191,7 @@ include_path = includes, output_file = target, cplus = cplus, + emit_linenums = line_directives, generate_pxi = pyrex_gen_pxi) result = cython_compile(source, options=options, full_module_name=module_name) diff -r dd29215de4a0 Cython/Distutils/extension.py --- a/Cython/Distutils/extension.py Mon Jun 07 11:06:45 2010 -0700 +++ b/Cython/Distutils/extension.py Mon Jun 07 16:45:43 2010 -0400 @@ -21,6 +21,8 @@ Unix form for portability) pyrex_create_listing_file : boolean write pyrex error messages to a listing (.lis) file. + pyrex_line_directivess : boolean + emit pyx line numbers for debugging/profiling pyrex_cplus : boolean use the C++ compiler for compiling and linking. pyrex_c_in_temp : boolean @@ -70,6 +72,7 @@ self.pyrex_include_dirs = pyrex_include_dirs or [] self.pyrex_create_listing = pyrex_create_listing + self.pyrex_line_directives = pyrex_line_directives self.pyrex_cplus = pyrex_cplus self.pyrex_c_in_temp = pyrex_c_in_temp self.pyrex_gen_pxi = pyrex_gen_pxi From cb at pdos.csail.mit.edu Tue Jun 8 19:57:22 2010 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Tue, 8 Jun 2010 13:57:22 -0400 Subject: [Cython] Buffer API Code Generation Bug Message-ID: <20100608175722.GA15160@pdos.lcs.mit.edu> Say you want to implement an extension in Cython exporting a Buffer API. This is a great canonical use-case for cython. So, you write something (stripped to its bare essentials) like cdef class Foo: def __getreadbuffer__ (Foo f, long s, void **A): pass def __getwritebuffer__(Foo f, long s, void **A): pass def __getsegcount__ (Foo f, int *C): if C: C[0] = 1 return 1 def __getbuffer__(Foo f, Py_buffer *b, int flags): pass Currently, Cython generates a PyBufferProcs initializer like static PyBufferProcs __pyx_tp_as_buffer_Foo = { #if PY_MAJOR_VERSION < 3 __pyx_pf_3int_3Foo___getreadbuffer__, /*bf_getreadbuffer*/ #endif with appropriate method definitions like static int __pyx_pf_3int_3Foo___getsegcount__(PyObject *__pyx_v_f, int *__pyx_v_C) { That is fine for Python 2.3/2.4 and Python 3.0. However, for Python 2.5 and 2.6, object.h changed the type of the function arguments for segment counts (and a lot of other things) to Py_ssize_t which can be 64-bits. For the read/write/charbuffer methods this isn't that big a deal if you have less than 4 giga-segments since the arguments are values and they should be converted, though you do get a C compiler warning. However, for the [get]segcountproc method (the typedef name dropped the "get" for 2.5 and added it back in for 2.6), it does actually matter to get the type right. Otherwise, Cython defines a function taking an int* but calls are made through a function pointer the C compiler thinks is an Py_ssize_t*. So, assuming the call site really uses the address of a Py_ssize_t variable, the generated assembly from the assignment through the int* will only alter 4 of the 8 bytes at the region passed to it. On little-endian 64-bit architectures, it still works out all right. On big-endian 64-bit architectures (e.g. PPC/IBM's POWER), it will erroneously convert some small number of segments like "1" into some crazy giant number of giga-segments (which will likely translate into some kind of segmentation violation downstream). So, it's a rare and kind of subtle bug. Anyway, one reason to pay attention to certain C compiler warnings for the generated code. So, we really want to instead generate a signature like static Py_ssize_t __pyx_pf_7ssize_t_3Foo___getsegcount__(PyObject *__pyx_v_f, Py_ssize_t *__pyx_v_C) { The following patch does exactly this. Note that the generated code is correct for all Python 2,3 versions because of Cython generating typedefs of Py_ssize_t to "int" for backward compatibility. So for those versions the Py_ssize_t will be effectively int like it is in the current generation interface, though it will be spelled differently in emitted code. We also fix the signatures for other PyBufferProcs entries to be complete. The only real issue I had was that it may err too much on the side of backward compatibility within Cython itself instead of being more internally consistent. The Z/z choice reverses the format_map convention of "more capitialization = more indirection" as used by p/P, i/I, s/S. Capital Z was already taken. On the other hand, Z is only used by about 6 signatures that I see anywhere in all of Cython and format_map itself seems isolated. So, it is cleaner to go with the more consistent approach and flip the case of the Z in those 6 sigs and make z=Py_ssize_t and Z=Py_ssize_t*. If you want, I'm happy to prepare a patch that does z/Z the way i/I are. It's pretty trivial text editing, though, to search for Signature.*Z and make the change in TypeSlots.py. For what it's worth, I've tested this patch and the code it generates on Py 2.4, 2.5, 2.6, and 3.1 and it seems fine. 3.0 was harder as my Gentoo isn't currently letting me install that simultaneously with 3.1, but I don't think there is any issue. Thanks! -------------------------------------------------------------------------- diff -r dd29215de4a0 Cython/Compiler/TypeSlots.py --- a/Cython/Compiler/TypeSlots.py Mon Jun 07 11:06:45 2010 -0700 +++ b/Cython/Compiler/TypeSlots.py Mon Jun 07 16:59:49 2010 -0400 @@ -49,6 +49,7 @@ 'I': PyrexTypes.c_int_ptr_type, 'l': PyrexTypes.c_long_type, 'Z': PyrexTypes.c_py_ssize_t_type, + 'z': PyrexTypes.c_py_ssize_t_ptr_type, # Z here z above would be more conventional but invasive 's': PyrexTypes.c_char_ptr_type, 'S': PyrexTypes.c_char_ptr_ptr_type, 'r': PyrexTypes.c_returncode_type, @@ -506,7 +507,7 @@ getcharbufferproc = Signature("TiS", 'i') # typedef int (*getcharbufferproc)(PyObject *, int, const char **); readbufferproc = Signature("TZP", "Z") # typedef Py_ssize_t (*readbufferproc)(PyObject *, Py_ssize_t, void **); writebufferproc = Signature("TZP", "Z") # typedef Py_ssize_t (*writebufferproc)(PyObject *, Py_ssize_t, void **); -segcountproc = Signature("TZ", "Z") # typedef Py_ssize_t (*segcountproc)(PyObject *, Py_ssize_t *); +segcountproc = Signature("Tz", "Z") # typedef Py_ssize_t (*segcountproc)(PyObject *, Py_ssize_t *); charbufferproc = Signature("TZS", "Z") # typedef Py_ssize_t (*charbufferproc)(PyObject *, Py_ssize_t, char **); objargproc = Signature("TO", 'r') # typedef int (*objobjproc)(PyObject *, PyObject *); # typedef int (*visitproc)(PyObject *, void *); @@ -625,10 +626,10 @@ ) PyBufferProcs = ( - MethodSlot(getreadbufferproc, "bf_getreadbuffer", "__getreadbuffer__", py3 = False), - MethodSlot(getwritebufferproc, "bf_getwritebuffer", "__getwritebuffer__", py3 = False), - MethodSlot(getsegcountproc, "bf_getsegcount", "__getsegcount__", py3 = False), - MethodSlot(getcharbufferproc, "bf_getcharbuffer", "__getcharbuffer__", py3 = False), + MethodSlot(readbufferproc, "bf_getreadbuffer", "__getreadbuffer__", py3 = False), + MethodSlot(writebufferproc, "bf_getwritebuffer", "__getwritebuffer__", py3 = False), + MethodSlot(segcountproc, "bf_getsegcount", "__getsegcount__", py3 = False), + MethodSlot(charbufferproc, "bf_getcharbuffer", "__getcharbuffer__", py3 = False), MethodSlot(getbufferproc, "bf_getbuffer", "__getbuffer__", ifdef = "PY_VERSION_HEX >= 0x02060000"), MethodSlot(releasebufferproc, "bf_releasebuffer", "__releasebuffer__", ifdef = "PY_VERSION_HEX >= 0x02060000") From dalcinl at gmail.com Wed Jun 9 02:54:33 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 8 Jun 2010 21:54:33 -0300 Subject: [Cython] Buffer API Code Generation Bug In-Reply-To: <20100608175722.GA15160@pdos.lcs.mit.edu> References: <20100608175722.GA15160@pdos.lcs.mit.edu> Message-ID: On 8 June 2010 14:57, Chuck Blake wrote: > Say you want to implement an extension in Cython exporting a Buffer API. > This is a great canonical use-case for cython. ?So, you write something > (stripped to its bare essentials) like > > ? ?cdef class Foo: > ? ? ? ?def __getreadbuffer__ (Foo f, long s, void **A): pass > ? ? ? ?def __getwritebuffer__(Foo f, long s, void **A): pass > ? ? ? ?def __getsegcount__ ? (Foo f, int *C): > ? ? ? ? ? ?if C: C[0] = 1 > ? ? ? ? ? ?return 1 > ? ? ? ?def __getbuffer__(Foo f, Py_buffer *b, int flags): pass > > Currently, Cython generates a PyBufferProcs initializer like > > ? ?static PyBufferProcs __pyx_tp_as_buffer_Foo = { > ? ? ? ?#if PY_MAJOR_VERSION < 3 > ? ? ? ?__pyx_pf_3int_3Foo___getreadbuffer__, /*bf_getreadbuffer*/ > ? ? ? ?#endif > > with appropriate method definitions like > > ? ?static int __pyx_pf_3int_3Foo___getsegcount__(PyObject *__pyx_v_f, int *__pyx_v_C) { > > That is fine for Python 2.3/2.4 and Python 3.0. > > > However, for Python 2.5 and 2.6, object.h changed the type of the function > arguments for segment counts (and a lot of other things) to Py_ssize_t > which can be 64-bits. > > For the read/write/charbuffer methods this isn't that big a deal if you > have less than 4 giga-segments since the arguments are values and they > should be converted, though you do get a C compiler warning. > > However, for the [get]segcountproc method (the typedef name dropped the > "get" for 2.5 and added it back in for 2.6), it does actually matter to > get the type right. ?Otherwise, Cython defines a function taking an int* > but calls are made through a function pointer the C compiler thinks is > an Py_ssize_t*. ?So, assuming the call site really uses the address of > a Py_ssize_t variable, the generated assembly from the assignment through > the int* will only alter 4 of the 8 bytes at the region passed to it. > > On little-endian 64-bit architectures, it still works out all right. > On big-endian 64-bit architectures (e.g. PPC/IBM's POWER), it will > erroneously convert some small number of segments like "1" into some > crazy giant number of giga-segments (which will likely translate into > some kind of segmentation violation downstream). ?So, it's a rare and > kind of subtle bug. ?Anyway, one reason to pay attention to certain > C compiler warnings for the generated code. > > So, we really want to instead generate a signature like > > ?static Py_ssize_t __pyx_pf_7ssize_t_3Foo___getsegcount__(PyObject *__pyx_v_f, Py_ssize_t *__pyx_v_C) { > > > The following patch does exactly this. ?Note that the generated code is > correct for all Python 2,3 versions because of Cython generating typedefs > of Py_ssize_t to "int" for backward compatibility. ?So for those versions > the Py_ssize_t will be effectively int like it is in the current generation > interface, though it will be spelled differently in emitted code. > > We also fix the signatures for other PyBufferProcs entries to be complete. > > The only real issue I had was that it may err too much on the side of > backward compatibility within Cython itself instead of being more > internally consistent. ?The Z/z choice reverses the format_map convention > of "more capitialization = more indirection" as used by p/P, i/I, s/S. > Capital Z was already taken. > > On the other hand, Z is only used by about 6 signatures that I see > anywhere in all of Cython and format_map itself seems isolated. > So, it is cleaner to go with the more consistent approach and flip > the case of the Z in those 6 sigs and make z=Py_ssize_t and Z=Py_ssize_t*. > > If you want, I'm happy to prepare a patch that does z/Z the way i/I are. > It's pretty trivial text editing, though, to search for Signature.*Z and > make the change in TypeSlots.py. ?For what it's worth, I've tested this > patch and the code it generates on Py 2.4, 2.5, 2.6, and 3.1 and it > seems fine. ?3.0 was harder as my Gentoo isn't currently letting me > install that simultaneously with 3.1, but I don't think there is any > issue. > > Thanks! > > -------------------------------------------------------------------------- > diff -r dd29215de4a0 Cython/Compiler/TypeSlots.py > --- a/Cython/Compiler/TypeSlots.py ? ? ?Mon Jun 07 11:06:45 2010 -0700 > +++ b/Cython/Compiler/TypeSlots.py ? ? ?Mon Jun 07 16:59:49 2010 -0400 > @@ -49,6 +49,7 @@ > ? ? ? ? 'I': PyrexTypes.c_int_ptr_type, > ? ? ? ? 'l': PyrexTypes.c_long_type, > ? ? ? ? 'Z': PyrexTypes.c_py_ssize_t_type, > + ? ? ? ?'z': PyrexTypes.c_py_ssize_t_ptr_type, # Z here z above would be more conventional but invasive > ? ? ? ? 's': PyrexTypes.c_char_ptr_type, > ? ? ? ? 'S': PyrexTypes.c_char_ptr_ptr_type, > ? ? ? ? 'r': PyrexTypes.c_returncode_type, > @@ -506,7 +507,7 @@ > ?getcharbufferproc = Signature("TiS", 'i') ?# typedef int (*getcharbufferproc)(PyObject *, int, const char **); > ?readbufferproc = Signature("TZP", "Z") ? ? # typedef Py_ssize_t (*readbufferproc)(PyObject *, Py_ssize_t, void **); > ?writebufferproc = Signature("TZP", "Z") ? ?# typedef Py_ssize_t (*writebufferproc)(PyObject *, Py_ssize_t, void **); > -segcountproc = Signature("TZ", "Z") ? ? ? ?# typedef Py_ssize_t (*segcountproc)(PyObject *, Py_ssize_t *); > +segcountproc = Signature("Tz", "Z") ? ? ? ?# typedef Py_ssize_t (*segcountproc)(PyObject *, Py_ssize_t *); > ?charbufferproc = Signature("TZS", "Z") ? ? # typedef Py_ssize_t (*charbufferproc)(PyObject *, Py_ssize_t, char **); > ?objargproc = Signature("TO", 'r') ? ? ? ? ?# typedef int (*objobjproc)(PyObject *, PyObject *); > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?# typedef int (*visitproc)(PyObject *, void *); > @@ -625,10 +626,10 @@ > ?) > > ?PyBufferProcs = ( > - ? ?MethodSlot(getreadbufferproc, "bf_getreadbuffer", "__getreadbuffer__", py3 = False), > - ? ?MethodSlot(getwritebufferproc, "bf_getwritebuffer", "__getwritebuffer__", py3 = False), > - ? ?MethodSlot(getsegcountproc, "bf_getsegcount", "__getsegcount__", py3 = False), > - ? ?MethodSlot(getcharbufferproc, "bf_getcharbuffer", "__getcharbuffer__", py3 = False), > + ? ?MethodSlot(readbufferproc, "bf_getreadbuffer", "__getreadbuffer__", py3 = False), > + ? ?MethodSlot(writebufferproc, "bf_getwritebuffer", "__getwritebuffer__", py3 = False), > + ? ?MethodSlot(segcountproc, "bf_getsegcount", "__getsegcount__", py3 = False), > + ? ?MethodSlot(charbufferproc, "bf_getcharbuffer", "__getcharbuffer__", py3 = False), > > ? ? MethodSlot(getbufferproc, "bf_getbuffer", "__getbuffer__", ifdef = "PY_VERSION_HEX >= 0x02060000"), > ? ? MethodSlot(releasebufferproc, "bf_releasebuffer", "__releasebuffer__", ifdef = "PY_VERSION_HEX >= 0x02060000") > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Looks fine, but I think you should follow the i/I rules. Any other opinion? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed Jun 9 02:56:43 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 8 Jun 2010 21:56:43 -0300 Subject: [Cython] line-directives support for Distutils build_ext In-Reply-To: <20100608132345.GA20423@pdos.lcs.mit.edu> References: <20100608132345.GA20423@pdos.lcs.mit.edu> Message-ID: On 8 June 2010 10:23, Chuck Blake wrote: > Hey, guys. ?I did this a long time ago, but it patches cleanly against the > cython-devel tip (at least yesterday's tip). > > There was some discussion of what the default should be for the #line/#file > directives Robert added. ?This is a hopefully non-contentious addition to > support pushing that decision to higher levels of the build system. > > All this patch does is add pyrex_line_directives to support that. ?It's > an analogue of the extra attributes .pyrex_include_dirs going with -I, > or .pyrex_cplus going with the --cplus command option, etc. > > This allows you to add .pyrex_line_directives to Extension() objects in > your setup.py. ?The appropriate command option is then added to the > invocations of cython for those extensions, as it is for the -I/--cplus/.. > options. > > -------------------------------------------------------------------------- > diff -r dd29215de4a0 Cython/Distutils/build_ext.py > --- a/Cython/Distutils/build_ext.py ? ? Mon Jun 07 11:06:45 2010 -0700 > +++ b/Cython/Distutils/build_ext.py ? ? Mon Jun 07 16:45:43 2010 -0400 > @@ -36,6 +36,8 @@ > ? ? ? ? ?"generate C++ source files"), > ? ? ? ? ('pyrex-create-listing', None, > ? ? ? ? ?"write errors to a listing file"), > + ? ? ? ?('pyrex-line-directives', None, > + ? ? ? ? "emit source line directives"), > ? ? ? ? ('pyrex-include-dirs=', None, > ? ? ? ? ?"path to the Cython include files" + sep_by), > ? ? ? ? ('pyrex-c-in-temp', None, > @@ -45,13 +47,14 @@ > ? ? ? ? ]) > > ? ? boolean_options.extend([ > - ? ? ? ?'pyrex-cplus', 'pyrex-create-listing', 'pyrex-c-in-temp' > + ? ? ? ?'pyrex-cplus', 'pyrex-create-listing', 'pyrex-line-directives', 'pyrex-c-in-temp' > ? ? ]) > > ? ? def initialize_options(self): > ? ? ? ? _build_ext.build_ext.initialize_options(self) > ? ? ? ? self.pyrex_cplus = 0 > ? ? ? ? self.pyrex_create_listing = 0 > + ? ? ? ?self.pyrex_line_directives = 0 > ? ? ? ? self.pyrex_include_dirs = None > ? ? ? ? self.pyrex_c_in_temp = 0 > ? ? ? ? self.pyrex_gen_pxi = 0 > @@ -114,6 +117,8 @@ > > ? ? ? ? create_listing = self.pyrex_create_listing or \ > ? ? ? ? ? ? getattr(extension, 'pyrex_create_listing', 0) > + ? ? ? ?line_directives = self.pyrex_line_directives or \ > + ? ? ? ? ? ?getattr(extension, 'pyrex_line_directives', 0) > ? ? ? ? cplus = self.pyrex_cplus or getattr(extension, 'pyrex_cplus', 0) or \ > ? ? ? ? ? ? ? ? (extension.language and extension.language.lower() == 'c++') > ? ? ? ? pyrex_gen_pxi = self.pyrex_gen_pxi or getattr(extension, 'pyrex_gen_pxi', 0) > @@ -186,6 +191,7 @@ > ? ? ? ? ? ? ? ? ? ? include_path = includes, > ? ? ? ? ? ? ? ? ? ? output_file = target, > ? ? ? ? ? ? ? ? ? ? cplus = cplus, > + ? ? ? ? ? ? ? ? ? ?emit_linenums = line_directives, > ? ? ? ? ? ? ? ? ? ? generate_pxi = pyrex_gen_pxi) > ? ? ? ? ? ? ? ? result = cython_compile(source, options=options, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? full_module_name=module_name) > diff -r dd29215de4a0 Cython/Distutils/extension.py > --- a/Cython/Distutils/extension.py ? ? Mon Jun 07 11:06:45 2010 -0700 > +++ b/Cython/Distutils/extension.py ? ? Mon Jun 07 16:45:43 2010 -0400 > @@ -21,6 +21,8 @@ > ? ? ? ? Unix form for portability) > ? ? pyrex_create_listing_file : boolean > ? ? ? ? write pyrex error messages to a listing (.lis) file. > + ? ?pyrex_line_directivess : boolean > + ? ? ? ?emit pyx line numbers for debugging/profiling > ? ? pyrex_cplus : boolean > ? ? ? ? use the C++ compiler for compiling and linking. > ? ? pyrex_c_in_temp : boolean > @@ -70,6 +72,7 @@ > > ? ? ? ? self.pyrex_include_dirs = pyrex_include_dirs or [] > ? ? ? ? self.pyrex_create_listing = pyrex_create_listing > + ? ? ? ?self.pyrex_line_directives = pyrex_line_directives > ? ? ? ? self.pyrex_cplus = pyrex_cplus > ? ? ? ? self.pyrex_c_in_temp = pyrex_c_in_temp > ? ? ? ? self.pyrex_gen_pxi = pyrex_gen_pxi > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Looks fine. Could you generate the patch with hg export and send it attached? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From cb at pdos.csail.mit.edu Wed Jun 9 18:37:54 2010 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Wed, 9 Jun 2010 12:37:54 -0400 Subject: [Cython] Buffer API Code Generation Bug Message-ID: <20100609163754.GA9015@pdos.lcs.mit.edu> Lisandro Dalcin wrote: >Looks fine, but I think you should follow the i/I rules. Any other opinion? I hesitated to speculate why it might have been capital Z in the first place. Someone might have been thinking of using lowercase z for the unsigned version Py_size_t, but that isn't in use anywhere [ or else the letter slot would be used :) ]. Given that there is more of an existing indirection case convention, I agree that seems clearer. There are also a couple (more implicit) Signature uses in Builtin.py as well as TypeSlots.py. So, if you prefer (as I and Lisandro do) the z/Z scalar/pointer version of all this, I've gone ahead and attached that version of this fix as well, in hg export format this time. This is more invasive and harder to guarantee to be non-impacting of other code. There are no other uses of Signature() objects that I can find besides Bultin and Typeslots, though, at least by searching for '\('. This one also deletes getreadbufferproc and the other get*buffer* Signature variables as they are no longer relevant. (The C signatures are all the same with the Cython compatibility typedef making them actually different. So, the upstream Python typename shufflng around with gets coming and going no longer matters). I also tested this version against 2.4, 2.5, 2.6, and 3.1. -------------- next part -------------- # HG changeset patch # User cb # Date 1276099380 14400 # Node ID 425f43eeb32a15e2904b6f33284618f1b8dee931 # Parent fe6066e3be002cb8523ef3b356730115b03fa240 Fix Py_ssize_t/Py_ssize_t* issues in buffer API signatures by renaming format_map['Z'] => format_map['z'] to make room for a new Z=>Py_ssize_t*. Also update existing uses of Z in TypeSlots.py and Builtin.py and delete no longer needed "get"-prefixed versions of readbufferproc & friends. diff -r fe6066e3be00 -r 425f43eeb32a Cython/Compiler/Builtin.py --- a/Cython/Compiler/Builtin.py Wed Jun 09 11:15:22 2010 -0400 +++ b/Cython/Compiler/Builtin.py Wed Jun 09 12:03:00 2010 -0400 @@ -32,7 +32,7 @@ ('isinstance', "OO", "b", "PyObject_IsInstance"), ('issubclass', "OO", "b", "PyObject_IsSubclass"), #('iter', "O", "O", "PyObject_GetIter"), # optimised later on - ('len', "O", "Z", "PyObject_Length"), + ('len', "O", "z", "PyObject_Length"), ('locals', "", "O", "__pyx_locals"), #('map', "", "", ""), #('max', "", "", ""), @@ -103,7 +103,7 @@ ("tuple", "PyTuple_Type", []), - ("list", "PyList_Type", [("insert", "OZO", "i", "PyList_Insert")]), + ("list", "PyList_Type", [("insert", "OzO", "i", "PyList_Insert")]), ("dict", "PyDict_Type", [("items", "O", "O", "PyDict_Items"), ("keys", "O", "O", "PyDict_Keys"), diff -r fe6066e3be00 -r 425f43eeb32a Cython/Compiler/TypeSlots.py --- a/Cython/Compiler/TypeSlots.py Wed Jun 09 11:15:22 2010 -0400 +++ b/Cython/Compiler/TypeSlots.py Wed Jun 09 12:03:00 2010 -0400 @@ -29,7 +29,8 @@ # 'b' bint # 'I' int * # 'l' long - # 'Z' Py_ssize_t + # 'z' Py_ssize_t + # 'Z' Py_ssize_t * # 's' char * # 'S' char ** # 'r' int used only to signal exception @@ -48,7 +49,8 @@ 'b': PyrexTypes.c_bint_type, 'I': PyrexTypes.c_int_ptr_type, 'l': PyrexTypes.c_long_type, - 'Z': PyrexTypes.c_py_ssize_t_type, + 'z': PyrexTypes.c_py_ssize_t_type, + 'Z': PyrexTypes.c_py_ssize_t_ptr_type, 's': PyrexTypes.c_char_ptr_type, 'S': PyrexTypes.c_char_ptr_ptr_type, 'r': PyrexTypes.c_returncode_type, @@ -63,7 +65,7 @@ 'b': "-1", 'l': "-1", 'r': "-1", - 'Z': "-1", + 'z': "-1", } def __init__(self, arg_format, ret_format): @@ -484,30 +486,26 @@ iternaryfunc = Signature("TOO", "O") # typedef PyObject * (*ternaryfunc)(PyObject *, PyObject *, PyObject *); callfunc = Signature("T*", "O") # typedef PyObject * (*ternaryfunc)(PyObject *, PyObject *, PyObject *); inquiry = Signature("T", "i") # typedef int (*inquiry)(PyObject *); -lenfunc = Signature("T", "Z") # typedef Py_ssize_t (*lenfunc)(PyObject *); +lenfunc = Signature("T", "z") # typedef Py_ssize_t (*lenfunc)(PyObject *); # typedef int (*coercion)(PyObject **, PyObject **); intargfunc = Signature("Ti", "O") # typedef PyObject *(*intargfunc)(PyObject *, int); -ssizeargfunc = Signature("TZ", "O") # typedef PyObject *(*ssizeargfunc)(PyObject *, Py_ssize_t); +ssizeargfunc = Signature("Tz", "O") # typedef PyObject *(*ssizeargfunc)(PyObject *, Py_ssize_t); intintargfunc = Signature("Tii", "O") # typedef PyObject *(*intintargfunc)(PyObject *, int, int); -ssizessizeargfunc = Signature("TZZ", "O") # typedef PyObject *(*ssizessizeargfunc)(PyObject *, Py_ssize_t, Py_ssize_t); +ssizessizeargfunc = Signature("Tzz", "O") # typedef PyObject *(*ssizessizeargfunc)(PyObject *, Py_ssize_t, Py_ssize_t); intobjargproc = Signature("TiO", 'r') # typedef int(*intobjargproc)(PyObject *, int, PyObject *); -ssizeobjargproc = Signature("TZO", 'r') # typedef int(*ssizeobjargproc)(PyObject *, Py_ssize_t, PyObject *); +ssizeobjargproc = Signature("TzO", 'r') # typedef int(*ssizeobjargproc)(PyObject *, Py_ssize_t, PyObject *); intintobjargproc = Signature("TiiO", 'r') # typedef int(*intintobjargproc)(PyObject *, int, int, PyObject *); -ssizessizeobjargproc = Signature("TZZO", 'r') # typedef int(*ssizessizeobjargproc)(PyObject *, Py_ssize_t, Py_ssize_t, PyObject *); +ssizessizeobjargproc = Signature("TzzO", 'r') # typedef int(*ssizessizeobjargproc)(PyObject *, Py_ssize_t, Py_ssize_t, PyObject *); intintargproc = Signature("Tii", 'r') -ssizessizeargproc = Signature("TZZ", 'r') +ssizessizeargproc = Signature("Tzz", 'r') objargfunc = Signature("TO", "O") objobjargproc = Signature("TOO", 'r') # typedef int (*objobjargproc)(PyObject *, PyObject *, PyObject *); -getreadbufferproc = Signature("TiP", 'i') # typedef int (*getreadbufferproc)(PyObject *, int, void **); -getwritebufferproc = Signature("TiP", 'i') # typedef int (*getwritebufferproc)(PyObject *, int, void **); -getsegcountproc = Signature("TI", 'i') # typedef int (*getsegcountproc)(PyObject *, int *); -getcharbufferproc = Signature("TiS", 'i') # typedef int (*getcharbufferproc)(PyObject *, int, const char **); -readbufferproc = Signature("TZP", "Z") # typedef Py_ssize_t (*readbufferproc)(PyObject *, Py_ssize_t, void **); -writebufferproc = Signature("TZP", "Z") # typedef Py_ssize_t (*writebufferproc)(PyObject *, Py_ssize_t, void **); -segcountproc = Signature("TZ", "Z") # typedef Py_ssize_t (*segcountproc)(PyObject *, Py_ssize_t *); -charbufferproc = Signature("TZS", "Z") # typedef Py_ssize_t (*charbufferproc)(PyObject *, Py_ssize_t, char **); +readbufferproc = Signature("TzP", "z") # typedef Py_ssize_t (*readbufferproc)(PyObject *, Py_ssize_t, void **); +writebufferproc = Signature("TzP", "z") # typedef Py_ssize_t (*writebufferproc)(PyObject *, Py_ssize_t, void **); +segcountproc = Signature("TZ", "z") # typedef Py_ssize_t (*segcountproc)(PyObject *, Py_ssize_t *); +charbufferproc = Signature("TzS", "z") # typedef Py_ssize_t (*charbufferproc)(PyObject *, Py_ssize_t, char **); objargproc = Signature("TO", 'r') # typedef int (*objobjproc)(PyObject *, PyObject *); # typedef int (*visitproc)(PyObject *, void *); # typedef int (*traverseproc)(PyObject *, visitproc, void *); @@ -625,10 +623,10 @@ ) PyBufferProcs = ( - MethodSlot(getreadbufferproc, "bf_getreadbuffer", "__getreadbuffer__", py3 = False), - MethodSlot(getwritebufferproc, "bf_getwritebuffer", "__getwritebuffer__", py3 = False), - MethodSlot(getsegcountproc, "bf_getsegcount", "__getsegcount__", py3 = False), - MethodSlot(getcharbufferproc, "bf_getcharbuffer", "__getcharbuffer__", py3 = False), + MethodSlot(readbufferproc, "bf_getreadbuffer", "__getreadbuffer__", py3 = False), + MethodSlot(writebufferproc, "bf_getwritebuffer", "__getwritebuffer__", py3 = False), + MethodSlot(segcountproc, "bf_getsegcount", "__getsegcount__", py3 = False), + MethodSlot(charbufferproc, "bf_getcharbuffer", "__getcharbuffer__", py3 = False), MethodSlot(getbufferproc, "bf_getbuffer", "__getbuffer__", ifdef = "PY_VERSION_HEX >= 0x02060000"), MethodSlot(releasebufferproc, "bf_releasebuffer", "__releasebuffer__", ifdef = "PY_VERSION_HEX >= 0x02060000") From dalcinl at gmail.com Wed Jun 9 20:02:34 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 9 Jun 2010 15:02:34 -0300 Subject: [Cython] Buffer API Code Generation Bug In-Reply-To: <20100609163754.GA9015@pdos.lcs.mit.edu> References: <20100609163754.GA9015@pdos.lcs.mit.edu> Message-ID: On 9 June 2010 13:37, Chuck Blake wrote: > Lisandro Dalcin wrote: >>Looks fine, but I think you should follow the i/I rules. Any other opinion? > > I hesitated to speculate why it might have been capital Z in the first > place. ?Someone might have been thinking of using lowercase z for the > unsigned version Py_size_t, but that isn't in use anywhere [ or else the > letter slot would be used :) ]. ?Given that there is more of an existing > indirection case convention, I agree that seems clearer. > > There are also a couple (more implicit) Signature uses in Builtin.py > as well as TypeSlots.py. > > So, if you prefer (as I and Lisandro do) the z/Z scalar/pointer version > of all this, I've gone ahead and attached that version of this fix as > well, in hg export format this time. ?This is more invasive and harder > to guarantee to be non-impacting of other code. ?There are no other uses > of Signature() objects that I can find besides Bultin and Typeslots, > though, at least by searching for '\('. > > This one also deletes getreadbufferproc and the other get*buffer* > Signature variables as they are no longer relevant. ?(The C signatures > are all the same with the Cython compatibility typedef making them > actually different. ?So, the upstream Python typename shufflng around > with gets coming and going no longer matters). > > I also tested this version against 2.4, 2.5, 2.6, and 3.1. > The patch looks good. I'll give it a more closer look very soon. If no one objects, I'll push it by tomorrow. Chuck, could I use "Chuck Blake " as the author commit? A "cb" author name is not descriptive enough we want to give proper credits to all contributions. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From cb at pdos.csail.mit.edu Thu Jun 10 02:07:02 2010 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Wed, 9 Jun 2010 20:07:02 -0400 Subject: [Cython] line-directives support for Distutils build_ext In-Reply-To: Message-ID: <20100610000702.GA27158@pdos.lcs.mit.edu> Lisandro Dalcin wrote: >Looks fine. Could you generate the patch with hg export and send it attached? Done. It's attached. I sent this message earlier for you, but it does not appear to have made it through the system. Attribution as "Chuck Blake" is fine, Lisandro. -------------- next part -------------- # HG changeset patch # User cb # Date 1276096522 14400 # Node ID fe6066e3be002cb8523ef3b356730115b03fa240 # Parent dd29215de4a0930a7bb8e15d8dd293eedc82f5ba Add control of --line-directives to Distutils build_ext diff -r dd29215de4a0 -r fe6066e3be00 Cython/Distutils/build_ext.py --- a/Cython/Distutils/build_ext.py Mon Jun 07 11:06:45 2010 -0700 +++ b/Cython/Distutils/build_ext.py Wed Jun 09 11:15:22 2010 -0400 @@ -36,6 +36,8 @@ "generate C++ source files"), ('pyrex-create-listing', None, "write errors to a listing file"), + ('pyrex-line-directives', None, + "emit source line directives"), ('pyrex-include-dirs=', None, "path to the Cython include files" + sep_by), ('pyrex-c-in-temp', None, @@ -45,13 +47,14 @@ ]) boolean_options.extend([ - 'pyrex-cplus', 'pyrex-create-listing', 'pyrex-c-in-temp' + 'pyrex-cplus', 'pyrex-create-listing', 'pyrex-line-directives', 'pyrex-c-in-temp' ]) def initialize_options(self): _build_ext.build_ext.initialize_options(self) self.pyrex_cplus = 0 self.pyrex_create_listing = 0 + self.pyrex_line_directives = 0 self.pyrex_include_dirs = None self.pyrex_c_in_temp = 0 self.pyrex_gen_pxi = 0 @@ -114,6 +117,8 @@ create_listing = self.pyrex_create_listing or \ getattr(extension, 'pyrex_create_listing', 0) + line_directives = self.pyrex_line_directives or \ + getattr(extension, 'pyrex_line_directives', 0) cplus = self.pyrex_cplus or getattr(extension, 'pyrex_cplus', 0) or \ (extension.language and extension.language.lower() == 'c++') pyrex_gen_pxi = self.pyrex_gen_pxi or getattr(extension, 'pyrex_gen_pxi', 0) @@ -186,6 +191,7 @@ include_path = includes, output_file = target, cplus = cplus, + emit_linenums = line_directives, generate_pxi = pyrex_gen_pxi) result = cython_compile(source, options=options, full_module_name=module_name) diff -r dd29215de4a0 -r fe6066e3be00 Cython/Distutils/extension.py --- a/Cython/Distutils/extension.py Mon Jun 07 11:06:45 2010 -0700 +++ b/Cython/Distutils/extension.py Wed Jun 09 11:15:22 2010 -0400 @@ -21,6 +21,8 @@ Unix form for portability) pyrex_create_listing_file : boolean write pyrex error messages to a listing (.lis) file. + pyrex_line_directivess : boolean + emit pyx line numbers for debugging/profiling pyrex_cplus : boolean use the C++ compiler for compiling and linking. pyrex_c_in_temp : boolean @@ -70,6 +72,7 @@ self.pyrex_include_dirs = pyrex_include_dirs or [] self.pyrex_create_listing = pyrex_create_listing + self.pyrex_line_directives = pyrex_line_directives self.pyrex_cplus = pyrex_cplus self.pyrex_c_in_temp = pyrex_c_in_temp self.pyrex_gen_pxi = pyrex_gen_pxi From dalcinl at gmail.com Thu Jun 10 16:44:37 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 10 Jun 2010 11:44:37 -0300 Subject: [Cython] line-directives support for Distutils build_ext In-Reply-To: <20100610000702.GA27158@pdos.lcs.mit.edu> References: <20100610000702.GA27158@pdos.lcs.mit.edu> Message-ID: On 9 June 2010 21:07, Chuck Blake wrote: > Lisandro Dalcin wrote: >>Looks fine. Could you generate the patch with hg export and send it attached? > > Done. ?It's attached. > > I sent this message earlier for you, but it does not appear to have made it > through the system. > > Attribution as "Chuck Blake" is fine, Lisandro. > Could you double check the attached hg export patch? I noticed your original patch did not have the addition of 'pyrex_line_directives' to the argument list of the 'Extension' __init__ method... Could you please actually check the attached patch? It is almost all yours, plus my fix adding the arg to Extension.__init__ -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 -------------- next part -------------- A non-text attachment was scrubbed... Name: linedirs.diff Type: text/x-patch Size: 3393 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100610/eecfec50/attachment.bin From dalcinl at gmail.com Thu Jun 10 16:50:00 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 10 Jun 2010 11:50:00 -0300 Subject: [Cython] cpdef (unbound) method inheritance broken Message-ID: Please try the attached test case. If you switch the comment in the first two lines, you will get the failure. So far, I could not figure out what's going on. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 -------------- next part -------------- A non-text attachment was scrubbed... Name: methinherit.pyx Type: application/octet-stream Size: 620 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100610/a4f22253/attachment.obj From cb at pdos.csail.mit.edu Thu Jun 10 18:44:56 2010 From: cb at pdos.csail.mit.edu (Chuck Blake) Date: Thu, 10 Jun 2010 12:44:56 -0400 Subject: [Cython] line-directives support for Distutils build_ext In-Reply-To: Message-ID: <20100610164456.GA30698@pdos.lcs.mit.edu> Whoops. Somehow that one line add to build_ext.py got dropped in the conversion of my regular unidiff patch to the export. I'm not sure exactly how that happened, but I did check the export you attached. That seems to work fine to me. So, I think that's fine to push. Thanks, Lisandro. From dalcinl at gmail.com Thu Jun 10 18:54:36 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 10 Jun 2010 13:54:36 -0300 Subject: [Cython] line-directives support for Distutils build_ext In-Reply-To: <20100610164456.GA30698@pdos.lcs.mit.edu> References: <20100610164456.GA30698@pdos.lcs.mit.edu> Message-ID: On 10 June 2010 13:44, Chuck Blake wrote: >?That seems to work fine to me. ?So, I > think that's fine to push. Pushed: http://hg.cython.org/cython-devel/rev/16bbddac0bf8 > > ?Thanks, Lisandro. > Thank you for the patch and testing. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Fri Jun 11 18:12:45 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 11 Jun 2010 13:12:45 -0300 Subject: [Cython] Buffer API Code Generation Bug In-Reply-To: References: <20100609163754.GA9015@pdos.lcs.mit.edu> Message-ID: On 9 June 2010 15:02, Lisandro Dalcin wrote: > On 9 June 2010 13:37, Chuck Blake wrote: >> Lisandro Dalcin wrote: >>>Looks fine, but I think you should follow the i/I rules. Any other opinion? >> >> I hesitated to speculate why it might have been capital Z in the first >> place. ?Someone might have been thinking of using lowercase z for the >> unsigned version Py_size_t, but that isn't in use anywhere [ or else the >> letter slot would be used :) ]. ?Given that there is more of an existing >> indirection case convention, I agree that seems clearer. >> >> There are also a couple (more implicit) Signature uses in Builtin.py >> as well as TypeSlots.py. >> >> So, if you prefer (as I and Lisandro do) the z/Z scalar/pointer version >> of all this, I've gone ahead and attached that version of this fix as >> well, in hg export format this time. ?This is more invasive and harder >> to guarantee to be non-impacting of other code. ?There are no other uses >> of Signature() objects that I can find besides Bultin and Typeslots, >> though, at least by searching for '\('. >> >> This one also deletes getreadbufferproc and the other get*buffer* >> Signature variables as they are no longer relevant. ?(The C signatures >> are all the same with the Cython compatibility typedef making them >> actually different. ?So, the upstream Python typename shufflng around >> with gets coming and going no longer matters). >> >> I also tested this version against 2.4, 2.5, 2.6, and 3.1. >> > > The patch looks good. I'll give it a more closer look very soon. If no > one objects, I'll push it by tomorrow. > Pushed: http://hg.cython.org/cython-devel/rev/266a77df8eec -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Fri Jun 11 21:55:50 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 11 Jun 2010 16:55:50 -0300 Subject: [Cython] legacy pxd's in Cython/Includes In-Reply-To: <621B3AAC-F69A-4BED-83EA-5016A7F57EDC@math.washington.edu> References: <621B3AAC-F69A-4BED-83EA-5016A7F57EDC@math.washington.edu> Message-ID: On 8 May 2010 16:39, Robert Bradshaw wrote: > On May 5, 2010, at 8:23 AM, Lisandro Dalcin wrote: > >> How are we going to manage these old, deprecated pxd's? I would Like >> to move all them to other place (let say Cython/IncludesOld) and add >> that dir to the default search path for a few releases. > > Or maybe even Cython/Includes/deprecated (where one would still be > able to look them up as if they were top level with a warning 'till > their gone). > >> No idea how to generate a deprecation warning, though. > > Maybe in the lookup module code, where it tries to locate the pxd, if > all else fails it could try that directory (hard coded in) and raise a > deprecation warning before failing. Somehow it'd be nice for the > message to tell it where to actually look. > Pushed: http://hg.cython.org/cython-devel/rev/45e628719226 -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Sat Jun 12 01:39:12 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 11 Jun 2010 16:39:12 -0700 Subject: [Cython] legacy pxd's in Cython/Includes In-Reply-To: References: <621B3AAC-F69A-4BED-83EA-5016A7F57EDC@math.washington.edu> Message-ID: <0AC70EEF-9A18-4B80-B792-E69D3D836FF3@math.washington.edu> On Jun 11, 2010, at 12:55 PM, Lisandro Dalcin wrote: > On 8 May 2010 16:39, Robert Bradshaw > wrote: >> On May 5, 2010, at 8:23 AM, Lisandro Dalcin wrote: >> >>> How are we going to manage these old, deprecated pxd's? I would Like >>> to move all them to other place (let say Cython/IncludesOld) and add >>> that dir to the default search path for a few releases. >> >> Or maybe even Cython/Includes/deprecated (where one would still be >> able to look them up as if they were top level with a warning 'till >> their gone). >> >>> No idea how to generate a deprecation warning, though. >> >> Maybe in the lookup module code, where it tries to locate the pxd, if >> all else fails it could try that directory (hard coded in) and >> raise a >> deprecation warning before failing. Somehow it'd be nice for the >> message to tell it where to actually look. >> > > Pushed: http://hg.cython.org/cython-devel/rev/45e628719226 Nice. Thanks. - Robert From craigcitro at gmail.com Sat Jun 12 09:21:09 2010 From: craigcitro at gmail.com (Craig Citro) Date: Sat, 12 Jun 2010 00:21:09 -0700 Subject: [Cython] Cython 0.13 status: coming along nicely. Message-ID: Hi all, So I hope you're sitting down for this one: I just ran a complete build and test of Sage with a recent cython-devel + some extra patches, and everything builds and passes. Huzzah! I just need to add some tests to my patches tomorrow and I'll push those, along with posting a few patches to the Sage library. (Robert, you're going to have a few easy reviews coming up! :P) Here's my current plan: - Saturday: add tests, push some patches. (I'm flying to CA tomorrow, which is a perfect time for this.) - Sunday: cut cython-0.13.alpha0, *finally*. After that, I'm going to pull the new patches into cython-closures, and test Sage with that. Here are some famous last words: I don't expect to run into too much trouble, because nothing is *using* the new features. If that goes well, I'll cut alpha1 shortly thereafter. (There's some other code here and there, but at least I'm familiar with parts of it already.) If not, we'll see how bad the fallout is and make a decision. My goal is to get alpha1 out and give people, say, a week or so to try it out, see what trouble they run into, and then plan on doing an actual release shortly thereafter. Thoughts? Comments? What's the schedule usually like? For the curious: there was nothing terribly bad about this particular release -- most of the holdup was due to other things (like buying a house, moving, designing a new course this past quarter, baby illnesses, that kind of stuff). There was one significant change from 0.12.1, though -- "safe" type inference is turned on by default. As you might expect, this causes *lots* of small changes, especially in a codebase like Sage. So there were a bunch of bugs to track down -- most of which weren't good for hunting 30 minutes at a time. ;) -cc From robertwb at math.washington.edu Sat Jun 12 17:54:27 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 12 Jun 2010 08:54:27 -0700 Subject: [Cython] Cython 0.13 status: coming along nicely. In-Reply-To: References: Message-ID: <958B1895-4576-4C98-8B25-6C54D3EF4E7D@math.washington.edu> On Jun 12, 2010, at 12:21 AM, Craig Citro wrote: > Hi all, > > So I hope you're sitting down for this one: I just ran a complete > build and test of Sage with a recent cython-devel + some extra > patches, and everything builds and passes. Huzzah! I just need to add > some tests to my patches tomorrow and I'll push those, along with > posting a few patches to the Sage library. (Robert, you're going to > have a few easy reviews coming up! :P) Yay! My availability will be low next week and high the two next weeks, but if we can get the patches out there for everyone else to try this next week I'll take care of the Sage reviews the week after and be ready to help with any fallout. > Here's my current plan: > > - Saturday: add tests, push some patches. (I'm flying to CA tomorrow, > which is a perfect time for this.) > - Sunday: cut cython-0.13.alpha0, *finally*. After that, I'm going to > pull the new patches into cython-closures, and test Sage with that. > Here are some famous last words: I don't expect to run into too much > trouble, because nothing is *using* the new features. If that goes > well, I'll cut alpha1 shortly thereafter. (There's some other code > here and there, but at least I'm familiar with parts of it already.) > If not, we'll see how bad the fallout is and make a decision. > > My goal is to get alpha1 out and give people, say, a week or so to try > it out, see what trouble they run into, and then plan on doing an > actual release shortly thereafter. Thoughts? Comments? What's the > schedule usually like? I'd say that sounds great. So we'd be looking at the last week or so of June then. > For the curious: there was nothing terribly bad about this particular > release -- most of the holdup was due to other things (like buying a > house, moving, designing a new course this past quarter, baby > illnesses, that kind of stuff). There was one significant change from > 0.12.1, though -- "safe" type inference is turned on by default. As > you might expect, this causes *lots* of small changes, especially in a > codebase like Sage. So there were a bunch of bugs to track down -- > most of which weren't good for hunting 30 minutes at a time. ;) It would be interesting to know what they were, so other people could prepare. So far, the ones I know of were: - Some expressions were typed as str vs. bytes (not really due to type inference). - Badly resolved imports were left as undefined types. Was there anything else? - Robert From dalcinl at gmail.com Thu Jun 17 18:31:32 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 17 Jun 2010 13:31:32 -0300 Subject: [Cython] adding support for __dict__ in extension types Message-ID: If we special case a __dict__ attribute in extension types, i.e: cdef class Foo: cdef dict __dict__ and fill type->tp_dictoffset, then we can support __dict__ in extension types. What do you think? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Sun Jun 20 14:49:04 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Jun 2010 14:49:04 +0200 Subject: [Cython] ref-count handling in closures In-Reply-To: References: <4C160E27.5050800@behnel.de> <4C1B40A8.5060203@behnel.de> <4C1DC715.9090708@behnel.de> Message-ID: <4C1E0E40.6090204@behnel.de> [taking this to cython-dev, as it's worth going into the archive] Haoyu Bai, 20.06.2010 14:22: >> Haoyu Bai, 20.06.2010 04:48: >>> compiling (cpp) and running knuth_man_or_boy_test ... Doctest: >>> knuth_man_or_boy_test.__test__.a (line 44) ... python: >>> Modules/gcmodule.c:276: visit_decref: Assertion `gc->gc.gc_refs != 0' >>> failed. >>> >>> This seems related to refcounting problem due to my work on nonlocal. >>> I attempted to fix it but no success yet. In knuth_man_or_boy_test, >>> the assertion failed on a binding_PyCFunction object (the function >>> object b). > > I solved the problem. This is due to GC collecting may start inside a > refnanny Pyx_INCREF or Pyx_DECREF. Sure, DECREF can run arbitrary code. INCREF can't, though. Even in PyDebug builds, it's pretty restricted. > Thus a inconsistent state could be exposed to the GC. > > For example I was writing code like this: > > scope->v1 = arg1; > scope->v2 = arg2; > scope->v3 = arg3; > Pyx_INCREF(scope->v1); Pyx_GIVEREF(scope->v1); > Pyx_INCREF(scope->v2); Pyx_GIVEREF(scope->v2); > Pyx_INCREF(scope->v3); Pyx_GIVEREF(scope->v3); Don't ever do that! The correct way to do this is to do the INCREF first, before the assignment, i.e. in this order: Pyx_INCREF(arg1); Pyx_DECREF(scope->v1); (if required) scope->v1 = arg1; Pyx_GIVEREF(scope->v1); (or GIVEREF(arg1)) Actually, it would be even better to do this instead: Pyx_INCREF(arg1); temp = scope->v1; scope->v1 = arg1; Pyx_DECREF(temp); Pyx_GIVEREF(arg1); as DECREF(scope->v1) can run arbitrary code that may involve traversing the references stored in "scope", including the reference in "scope->v1". Read up on the Py_CLEAR() macro. Stefan From robertwb at math.washington.edu Mon Jun 21 20:41:15 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 21 Jun 2010 11:41:15 -0700 Subject: [Cython] adding support for __dict__ in extension types In-Reply-To: References: Message-ID: <5E596740-6D0C-4AB9-9149-A28698DA1789@math.washington.edu> On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: > If we special case a __dict__ attribute in extension types, i.e: > > cdef class Foo: > cdef dict __dict__ > > and fill type->tp_dictoffset, then we can support __dict__ in > extension types. > > What do you think? Sounds like a good idea to me. Note that we check tp_dictoffset for fast dispatching for cpdef methods (which would be correct as a dict lookup *would* be needed if __dict__ is available). - Robert From dalcinl at gmail.com Tue Jun 22 16:01:25 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 22 Jun 2010 11:01:25 -0300 Subject: [Cython] ref-count handling in closures In-Reply-To: <4C1E0E40.6090204@behnel.de> References: <4C160E27.5050800@behnel.de> <4C1B40A8.5060203@behnel.de> <4C1DC715.9090708@behnel.de> <4C1E0E40.6090204@behnel.de> Message-ID: On 20 June 2010 09:49, Stefan Behnel wrote: > > Actually, it would be even better to do this instead: > > ? ? Pyx_INCREF(arg1); > ? ? temp = scope->v1; > ? ? scope->v1 = arg1; > ? ? Pyx_DECREF(temp); > ? ? Pyx_GIVEREF(arg1); > > as DECREF(scope->v1) can run arbitrary code that may involve traversing the > references stored in "scope", including the reference in "scope->v1". > BTW, Shouldn't we use Py_CLEAR that at ExtType_tp_free() slots? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From baihaoyu at gmail.com Tue Jun 22 18:29:08 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Wed, 23 Jun 2010 00:29:08 +0800 Subject: [Cython] ref-count handling in closures In-Reply-To: <4C1E0E40.6090204@behnel.de> References: <4C160E27.5050800@behnel.de> <4C1B40A8.5060203@behnel.de> <4C1DC715.9090708@behnel.de> <4C1E0E40.6090204@behnel.de> Message-ID: On Sun, Jun 20, 2010 at 8:49 PM, Stefan Behnel wrote: > [taking this to cython-dev, as it's worth going into the archive] > > Haoyu Bai, 20.06.2010 14:22: >>> Haoyu Bai, 20.06.2010 04:48: >>>> compiling (cpp) and running knuth_man_or_boy_test ... ? Doctest: >>>> knuth_man_or_boy_test.__test__.a (line 44) ... python: >>>> Modules/gcmodule.c:276: visit_decref: Assertion `gc->gc.gc_refs != 0' >>>> failed. >>>> >>>> This seems related to refcounting problem due to my work on nonlocal. >>>> I attempted to fix it but no success yet. In knuth_man_or_boy_test, >>>> the assertion failed on a binding_PyCFunction object (the function >>>> object b). >> >> >> I solved the problem. This is due to GC collecting may start inside a >> refnanny Pyx_INCREF or Pyx_DECREF. > > Sure, DECREF can run arbitrary code. INCREF can't, though. Even in PyDebug > builds, it's pretty restricted. > > However, with refnanny, INCREF may also run arbitrary code (at least trigger GC). That may cause strange things. We should be aware. >> Thus a inconsistent state could be exposed to the GC. >> >> For example I was writing code like this: >> >> scope->v1 = arg1; >> scope->v2 = arg2; >> scope->v3 = arg3; >> Pyx_INCREF(scope->v1); Pyx_GIVEREF(scope->v1); >> Pyx_INCREF(scope->v2); Pyx_GIVEREF(scope->v2); >> Pyx_INCREF(scope->v3); Pyx_GIVEREF(scope->v3); > > Don't ever do that! > > The correct way to do this is to do the INCREF first, before the assignment, > i.e. in this order: > > ? ?Pyx_INCREF(arg1); > ? ?Pyx_DECREF(scope->v1); ?(if required) > ? ?scope->v1 = arg1; > ? ?Pyx_GIVEREF(scope->v1); ?(or GIVEREF(arg1)) > > Actually, it would be even better to do this instead: > > ? ?Pyx_INCREF(arg1); > ? ?temp = scope->v1; > ? ?scope->v1 = arg1; > ? ?Pyx_DECREF(temp); > ? ?Pyx_GIVEREF(arg1); > > as DECREF(scope->v1) can run arbitrary code that may involve traversing the > references stored in "scope", including the reference in "scope->v1". > > Read up on the Py_CLEAR() macro. > > Stefan > Thanks for the comments. I fixed my branch with the correct way. Now the build status on Hudson all turns blue or yellow. Indeed, there are some other places that we should use Py_CLEAR() insted of DECREF. I'll try to go through and fix them. -- Haoyu BAI School of Computing, National University of Singapore. From stefan_ml at behnel.de Tue Jun 22 18:46:04 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Jun 2010 18:46:04 +0200 Subject: [Cython] ref-count handling in closures In-Reply-To: References: <4C160E27.5050800@behnel.de> <4C1B40A8.5060203@behnel.de> <4C1DC715.9090708@behnel.de> <4C1E0E40.6090204@behnel.de> Message-ID: <4C20E8CC.3040405@behnel.de> Haoyu Bai, 22.06.2010 18:29: > Indeed, there are > some other places that we should use Py_CLEAR() insted of DECREF. I'll > try to go through and fix them. Be careful, though, it's not always the right thing to do. Py_CLEAR() is nice for hand written code, but Cython can sometimes do better or has to do other things in order to generate correct code. I don't remember where, but I already ran into this trap once. Whatever you change in this regard, please call for a review before you invest too much time into it. Stefan From baihaoyu at gmail.com Wed Jun 23 17:11:50 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Wed, 23 Jun 2010 23:11:50 +0800 Subject: [Cython] ref-count handling in closures In-Reply-To: <4C20E8CC.3040405@behnel.de> References: <4C160E27.5050800@behnel.de> <4C1B40A8.5060203@behnel.de> <4C1DC715.9090708@behnel.de> <4C1E0E40.6090204@behnel.de> <4C20E8CC.3040405@behnel.de> Message-ID: On Wed, Jun 23, 2010 at 12:46 AM, Stefan Behnel wrote: > Haoyu Bai, 22.06.2010 18:29: >> >> Indeed, there are >> some other places that we should use Py_CLEAR() insted of DECREF. I'll >> try to go through and fix them. > > Be careful, though, it's not always the right thing to do. Py_CLEAR() is > nice for hand written code, but Cython can sometimes do better or has to do > other things in order to generate correct code. I don't remember where, but > I already ran into this trap once. > > Whatever you change in this regard, please call for a review before you > invest too much time into it. > > Stefan > In tp_dealloc we are generating code like this: XDECREF(scope->var1); XDECREF(scope->var2); XDECREF(scope->var3); scope->tp_free(scope); Should we use Py_CLEAR here? In DefNode.generate_argument_parsing_code(), we are calling put_var_xdecref_clear() to clean up closure when error occurs. And the decref_clear() generates code like DECREF(scope->var); scope->var=0; Should this also be replaced by Py_CLEAR? -- Haoyu BAI School of Computing, National University of Singapore. From dagss at student.matnat.uio.no Thu Jun 24 19:09:55 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 24 Jun 2010 19:09:55 +0200 Subject: [Cython] .cy extension? Message-ID: <4C239163.2020504@student.matnat.uio.no> Cython files are called .pyx files for historical reasons: For a long time Cython was simply smaller patches on top of Pyrex. This is not longer true. I don't believe we would honestly recommend to anyone that they maintain codebases that should be buildable on both compilers. (That it is easy to convert a Pyrex project to a Cython project is another story.) While the .pyx extension will obviously stick with us for many years to come, I'm proposing that we add support for a .cy extension as well, and change the documentation to use this. They'll be completely interchangeable, almost nothing in the Cython codebase should know the difference (and I imagine that Sage and lxml would stick to pyx for a long time). This is simply to be a good citizen. Build systems rely on extensions to invoke the proper tool, and it would then be possible for SCons, waf, CMake etc. to include default builders for both Cython and Pyrex without any conflicts (if you want to use such tools, you simply use the .cy extension -- if you already use .pyx you obviously already have a build system set up for that). It would even be possible to mix Cython and Pyrex in the same project (e.g. with distutils...). Not that that is a likely situation, but it seems less arrogant with respect to Pyrex to use our own suffix rather than patching away Pyrex support in the build tools. (setuptools already hardwires .pyx to Pyrex, for instance). -- Dag Sverre From kwmsmith at gmail.com Thu Jun 24 19:52:47 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Thu, 24 Jun 2010 12:52:47 -0500 Subject: [Cython] .cy extension? In-Reply-To: <4C239163.2020504@student.matnat.uio.no> References: <4C239163.2020504@student.matnat.uio.no> Message-ID: On Thu, Jun 24, 2010 at 12:09 PM, Dag Sverre Seljebotn wrote: > Cython files are called .pyx files for historical reasons: For a long > time Cython was simply smaller patches on top of Pyrex. This is not > longer true. I don't believe we would honestly recommend to anyone that > they maintain codebases that should be buildable on both compilers. > (That it is easy to convert a Pyrex project to a Cython project is > another story.) > > While the .pyx extension will obviously stick with us for many years to > come, I'm proposing that we add support for a .cy extension as well, and > change the documentation to use this. They'll be completely > interchangeable, almost nothing in the Cython codebase should know the > difference (and I imagine that Sage and lxml would stick to pyx for a > long time). +1 (FWIW). This would definitely make things easier working with distutils since it assumes '.pyx' means Pyrex, as you mention. Would .pxd map to .cyd? .pxi -> .cyi? As an internal issue, not that important: would it be preferable to change the name-mangling prefix from '__pyx' to '__cy'? Since annotations (or whatever the '-a' option gives you) make the generated code visible to the end user, this might help give a sense of overall project consistency. > > This is simply to be a good citizen. Build systems rely on extensions to > invoke the proper tool, and it would then be possible for SCons, waf, > CMake etc. to include default builders for both Cython and Pyrex without > any conflicts (if you want to use such tools, you simply use the .cy > extension -- if you already use .pyx you obviously already have a build > system set up for that). It would even be possible to mix Cython and > Pyrex in the same project (e.g. with distutils...). Not that that is a > likely situation, but it seems less arrogant with respect to Pyrex to > use our own suffix rather than patching away Pyrex support in the build > tools. > > (setuptools already hardwires .pyx to Pyrex, for instance). > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Thu Jun 24 20:29:25 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 24 Jun 2010 20:29:25 +0200 Subject: [Cython] .cy extension? In-Reply-To: References: <4C239163.2020504@student.matnat.uio.no> Message-ID: <4C23A405.2040600@student.matnat.uio.no> Kurt Smith wrote: > On Thu, Jun 24, 2010 at 12:09 PM, Dag Sverre Seljebotn > wrote: >> Cython files are called .pyx files for historical reasons: For a long >> time Cython was simply smaller patches on top of Pyrex. This is not >> longer true. I don't believe we would honestly recommend to anyone that >> they maintain codebases that should be buildable on both compilers. >> (That it is easy to convert a Pyrex project to a Cython project is >> another story.) >> >> While the .pyx extension will obviously stick with us for many years to >> come, I'm proposing that we add support for a .cy extension as well, and >> change the documentation to use this. They'll be completely >> interchangeable, almost nothing in the Cython codebase should know the >> difference (and I imagine that Sage and lxml would stick to pyx for a >> long time). > > +1 (FWIW). This would definitely make things easier working with > distutils since it assumes '.pyx' means Pyrex, as you mention. > > Would .pxd map to .cyd? .pxi -> .cyi? I think this is much less important as they don't need to be entry points for build tools. But I guess we could just as well support "cyd" (or "cyh"? which seems to be more explanative to me). pxi I think we've pretty much deprecated, but why not.. -- Dag Sverre From dalcinl at gmail.com Thu Jun 24 20:47:10 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 24 Jun 2010 15:47:10 -0300 Subject: [Cython] .cy extension? In-Reply-To: <4C23A405.2040600@student.matnat.uio.no> References: <4C239163.2020504@student.matnat.uio.no> <4C23A405.2040600@student.matnat.uio.no> Message-ID: My first reaction to Dag's request was "No, please! Yet another extension...", but then I reached the rationale (build tools and chains), and my mind suddenly changed to a +1. On 24 June 2010 15:29, Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> >> +1 (FWIW). ?This would definitely make things easier working with >> distutils ?since it assumes '.pyx' means Pyrex, as you mention. >> >> Would .pxd map to .cyd? ?.pxi -> .cyi? > > I think this is much less important as they don't need to be entry > points for build tools. But I guess we could just as well support "cyd" > (or "cyh"? which seems to be more explanative to me). > Either would do. BTW, an "h" instead of "d" in not so much explanatory IMO. > > pxi I think we've pretty much deprecated, > Indeed. > > but why not.. > -1. Why not? Well, because they are deprecated :-). Additionally, they are not entry points to build tools. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From greg.ewing at canterbury.ac.nz Fri Jun 25 02:55:11 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 25 Jun 2010 12:55:11 +1200 Subject: [Cython] .cy extension? In-Reply-To: <4C23A405.2040600@student.matnat.uio.no> References: <4C239163.2020504@student.matnat.uio.no> <4C23A405.2040600@student.matnat.uio.no> Message-ID: <4C23FE6F.7050706@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > pxi I think we've pretty much deprecated, but why not.. The .pxi extension was only ever a user convention anyway. I don't think there's anything in the Pyrex compiler that recognises it as being special. -- Greg From stefan_ml at behnel.de Fri Jun 25 07:33:55 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 25 Jun 2010 07:33:55 +0200 Subject: [Cython] .cy extension? In-Reply-To: <4C23A405.2040600@student.matnat.uio.no> References: <4C239163.2020504@student.matnat.uio.no> <4C23A405.2040600@student.matnat.uio.no> Message-ID: <4C243FC3.9050803@behnel.de> Dag Sverre Seljebotn, 24.06.2010 20:29: > pxi I think we've pretty much deprecated Objection to anything that deprecates .pxi. Inclusion is a very convenient way to split a single extension module up into separate files that depend on each other and form a common API. As long as we do not support linking together multiple .pyx modules in a similarly simple way, I can't see it being deprecated any time soon. Stefan From dalcinl at gmail.com Fri Jun 25 15:48:39 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 25 Jun 2010 10:48:39 -0300 Subject: [Cython] .cy extension? In-Reply-To: <4C243FC3.9050803@behnel.de> References: <4C239163.2020504@student.matnat.uio.no> <4C23A405.2040600@student.matnat.uio.no> <4C243FC3.9050803@behnel.de> Message-ID: On 25 June 2010 02:33, Stefan Behnel wrote: > Dag Sverre Seljebotn, 24.06.2010 20:29: >> pxi I think we've pretty much deprecated > > Objection to anything that deprecates .pxi. Inclusion is a very convenient > way to split a single extension module up into separate files that depend > on each other and form a common API. As long as we do not support linking > together multiple .pyx modules in a similarly simple way, I can't see it > being deprecated any time soon. > I think Dag was talking about generation of *.pxi files with API declarations. Regarding using *.pxi for splitting a module in separate files (as I do in my projects), the pxi extension is just a convention (as Greg pointed out). We could use any extension we want, right? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From greg.ewing at canterbury.ac.nz Sat Jun 26 00:53:29 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Jun 2010 10:53:29 +1200 Subject: [Cython] .cy extension? In-Reply-To: References: <4C239163.2020504@student.matnat.uio.no> <4C23A405.2040600@student.matnat.uio.no> <4C243FC3.9050803@behnel.de> Message-ID: <4C253369.8050702@canterbury.ac.nz> Lisandro Dalcin wrote: > I think Dag was talking about generation of *.pxi files with API > declarations. I'd forgotten about that. It probably wouldn't do any harm to drop the .pxi generation feature, because the purpose it was designed for (allowing one module access to extension types defined in another) has been subsumed by the cimport mechanism. The 'include' statement is still useful for other purposes, though, so I would recommend keeping it. -- Greg From robertwb at math.washington.edu Sat Jun 26 09:22:33 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 26 Jun 2010 00:22:33 -0700 Subject: [Cython] .cy extension? In-Reply-To: <4C253369.8050702@canterbury.ac.nz> References: <4C239163.2020504@student.matnat.uio.no> <4C23A405.2040600@student.matnat.uio.no> <4C243FC3.9050803@behnel.de> <4C253369.8050702@canterbury.ac.nz> Message-ID: <1D5F6019-4A86-4282-A585-43370F4A7D9C@math.washington.edu> On Jun 25, 2010, at 3:53 PM, Greg Ewing wrote: > Lisandro Dalcin wrote: > >> I think Dag was talking about generation of *.pxi files with API >> declarations. > > I'd forgotten about that. It probably wouldn't do any harm > to drop the .pxi generation feature, because the purpose it > was designed for (allowing one module access to extension > types defined in another) has been subsumed by the cimport > mechanism. +1 > The 'include' statement is still useful for other purposes, > though, so I would recommend keeping it. For sure. - Robert From robertwb at math.washington.edu Sat Jun 26 09:31:13 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 26 Jun 2010 00:31:13 -0700 Subject: [Cython] .cy extension? In-Reply-To: <4C239163.2020504@student.matnat.uio.no> References: <4C239163.2020504@student.matnat.uio.no> Message-ID: On Jun 24, 2010, at 10:09 AM, Dag Sverre Seljebotn wrote: > Cython files are called .pyx files for historical reasons: For a long > time Cython was simply smaller patches on top of Pyrex. This is not > longer true. I don't believe we would honestly recommend to anyone > that > they maintain codebases that should be buildable on both compilers. > (That it is easy to convert a Pyrex project to a Cython project is > another story.) I've always thought of pyx as meaning "python extension" more than Pyrex. > While the .pyx extension will obviously stick with us for many years > to > come, I'm proposing that we add support for a .cy extension as well, > and > change the documentation to use this. They'll be completely > interchangeable, almost nothing in the Cython codebase should know the > difference (and I imagine that Sage and lxml would stick to pyx for a > long time). > > This is simply to be a good citizen. Build systems rely on > extensions to > invoke the proper tool, and it would then be possible for SCons, waf, > CMake etc. to include default builders for both Cython and Pyrex > without > any conflicts (if you want to use such tools, you simply use the .cy > extension -- if you already use .pyx you obviously already have a > build > system set up for that). It would even be possible to mix Cython and > Pyrex in the same project (e.g. with distutils...). Not that that is a > likely situation, but it seems less arrogant with respect to Pyrex to > use our own suffix rather than patching away Pyrex support in the > build > tools. > > (setuptools already hardwires .pyx to Pyrex, for instance). If we do change, it should probably be to .cy, .cyd, and .cyi (the latter simply being a convention). However, I'm not convinced (or unconvinced) the gain would be worth the resulting confusion. Gracefully handling build tools is a good argument, but in terms of that specifically I would like to see us moving towards handling the Cython -> C step ourselves and letting distutils only worry about the C -> shared object library step [1] (and a robust pyximport or even inline/decorated functions in pure Python files). - Robert [1] http://wiki.cython.org/enhancements/distutils_preprocessing From robert.kern at gmail.com Sat Jun 26 09:47:02 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 26 Jun 2010 02:47:02 -0500 Subject: [Cython] .cy extension? In-Reply-To: References: <4C239163.2020504@student.matnat.uio.no> Message-ID: On 2010-06-26 02:31 , Robert Bradshaw wrote: > On Jun 24, 2010, at 10:09 AM, Dag Sverre Seljebotn wrote: > >> Cython files are called .pyx files for historical reasons: For a long >> time Cython was simply smaller patches on top of Pyrex. This is not >> longer true. I don't believe we would honestly recommend to anyone >> that >> they maintain codebases that should be buildable on both compilers. >> (That it is easy to convert a Pyrex project to a Cython project is >> another story.) > > I've always thought of pyx as meaning "python extension" more than > Pyrex. > >> While the .pyx extension will obviously stick with us for many years >> to >> come, I'm proposing that we add support for a .cy extension as well, >> and >> change the documentation to use this. They'll be completely >> interchangeable, almost nothing in the Cython codebase should know the >> difference (and I imagine that Sage and lxml would stick to pyx for a >> long time). >> >> This is simply to be a good citizen. Build systems rely on >> extensions to >> invoke the proper tool, and it would then be possible for SCons, waf, >> CMake etc. to include default builders for both Cython and Pyrex >> without >> any conflicts (if you want to use such tools, you simply use the .cy >> extension -- if you already use .pyx you obviously already have a >> build >> system set up for that). It would even be possible to mix Cython and >> Pyrex in the same project (e.g. with distutils...). Not that that is a >> likely situation, but it seems less arrogant with respect to Pyrex to >> use our own suffix rather than patching away Pyrex support in the >> build >> tools. >> >> (setuptools already hardwires .pyx to Pyrex, for instance). > > If we do change, it should probably be to .cy, .cyd, and .cyi (the > latter simply being a convention). However, I'm not convinced (or > unconvinced) the gain would be worth the resulting confusion. > Gracefully handling build tools is a good argument, but in terms of > that specifically I would like to see us moving towards handling the > Cython -> C step ourselves and letting distutils only worry about the > C -> shared object library step [1] (and a robust pyximport or even > inline/decorated functions in pure Python files). I'm afraid that doesn't help other build tools, like numpy.distutils. We need to know whether a .pyx file is a Pyrex file or a Cython file before we can import Pyrex or Cython to let them do the transformations. A .cy extension would be a huge help. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robertwb at math.washington.edu Sat Jun 26 09:51:03 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 26 Jun 2010 00:51:03 -0700 Subject: [Cython] .cy extension? In-Reply-To: References: <4C239163.2020504@student.matnat.uio.no> Message-ID: On Jun 26, 2010, at 12:47 AM, Robert Kern wrote: > On 2010-06-26 02:31 , Robert Bradshaw wrote: >> On Jun 24, 2010, at 10:09 AM, Dag Sverre Seljebotn wrote: >> >>> Cython files are called .pyx files for historical reasons: For a >>> long >>> time Cython was simply smaller patches on top of Pyrex. This is not >>> longer true. I don't believe we would honestly recommend to anyone >>> that >>> they maintain codebases that should be buildable on both compilers. >>> (That it is easy to convert a Pyrex project to a Cython project is >>> another story.) >> >> I've always thought of pyx as meaning "python extension" more than >> Pyrex. >> >>> While the .pyx extension will obviously stick with us for many years >>> to >>> come, I'm proposing that we add support for a .cy extension as well, >>> and >>> change the documentation to use this. They'll be completely >>> interchangeable, almost nothing in the Cython codebase should know >>> the >>> difference (and I imagine that Sage and lxml would stick to pyx >>> for a >>> long time). >>> >>> This is simply to be a good citizen. Build systems rely on >>> extensions to >>> invoke the proper tool, and it would then be possible for SCons, >>> waf, >>> CMake etc. to include default builders for both Cython and Pyrex >>> without >>> any conflicts (if you want to use such tools, you simply use the .cy >>> extension -- if you already use .pyx you obviously already have a >>> build >>> system set up for that). It would even be possible to mix Cython and >>> Pyrex in the same project (e.g. with distutils...). Not that that >>> is a >>> likely situation, but it seems less arrogant with respect to Pyrex >>> to >>> use our own suffix rather than patching away Pyrex support in the >>> build >>> tools. >>> >>> (setuptools already hardwires .pyx to Pyrex, for instance). >> >> If we do change, it should probably be to .cy, .cyd, and .cyi (the >> latter simply being a convention). However, I'm not convinced (or >> unconvinced) the gain would be worth the resulting confusion. >> Gracefully handling build tools is a good argument, but in terms of >> that specifically I would like to see us moving towards handling the >> Cython -> C step ourselves and letting distutils only worry about >> the >> C -> shared object library step [1] (and a robust pyximport or even >> inline/decorated functions in pure Python files). > > I'm afraid that doesn't help other build tools, like > numpy.distutils. We need to > know whether a .pyx file is a Pyrex file or a Cython file before we > can import > Pyrex or Cython to let them do the transformations. A .cy extension > would be a > huge help. The point is that numpy.distutils wouldn't have to know about Cython at all--it would just be handed a bunch of .c files to make extensions out of. This way we wouldn't have to make any of the many distutils variants Cython-aware. - Robert From robert.kern at gmail.com Sat Jun 26 19:26:32 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 26 Jun 2010 12:26:32 -0500 Subject: [Cython] .cy extension? In-Reply-To: References: <4C239163.2020504@student.matnat.uio.no> Message-ID: On 2010-06-26 02:51 , Robert Bradshaw wrote: > On Jun 26, 2010, at 12:47 AM, Robert Kern wrote: > >> On 2010-06-26 02:31 , Robert Bradshaw wrote: >>> If we do change, it should probably be to .cy, .cyd, and .cyi (the >>> latter simply being a convention). However, I'm not convinced (or >>> unconvinced) the gain would be worth the resulting confusion. >>> Gracefully handling build tools is a good argument, but in terms of >>> that specifically I would like to see us moving towards handling the >>> Cython -> C step ourselves and letting distutils only worry about >>> the >>> C -> shared object library step [1] (and a robust pyximport or even >>> inline/decorated functions in pure Python files). >> >> I'm afraid that doesn't help other build tools, like >> numpy.distutils. We need to >> know whether a .pyx file is a Pyrex file or a Cython file before we >> can import >> Pyrex or Cython to let them do the transformations. A .cy extension >> would be a >> huge help. > > The point is that numpy.distutils wouldn't have to know about Cython > at all--it would just be handed a bunch of .c files to make extensions > out of. This way we wouldn't have to make any of the many distutils > variants Cython-aware. cythonize() wouldn't work with a numpy.distutils-using setup.py. We don't pass the list of Extensions explicitly. There are also other non-distutils build tools, like SCons, waf, and yaku, that would be able to use the .cy extension to distinguish Pyrex files from Cython files. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Sun Jun 27 04:45:27 2010 From: cournape at gmail.com (David Cournapeau) Date: Sun, 27 Jun 2010 11:45:27 +0900 Subject: [Cython] .cy extension? In-Reply-To: References: <4C239163.2020504@student.matnat.uio.no> Message-ID: On Sun, Jun 27, 2010 at 2:26 AM, Robert Kern wrote: > On 2010-06-26 02:51 , Robert Bradshaw wrote: >> On Jun 26, 2010, at 12:47 AM, Robert Kern wrote: >> >>> On 2010-06-26 02:31 , Robert Bradshaw wrote: > >>>> If we do change, it should probably be to .cy, .cyd, and .cyi (the >>>> latter simply being a convention). However, I'm not convinced (or >>>> unconvinced) the gain would be worth the resulting confusion. >>>> Gracefully handling build tools is a good argument, but in terms of >>>> that specifically I would like to see us moving towards handling the >>>> Cython -> ? C step ourselves and letting distutils only worry about >>>> the >>>> C -> ? shared object library step [1] (and a robust pyximport or even >>>> inline/decorated functions in pure Python files). >>> >>> I'm afraid that doesn't help other build tools, like >>> numpy.distutils. We need to >>> know whether a .pyx file is a Pyrex file or a Cython file before we >>> can import >>> Pyrex or Cython to let them do the transformations. A .cy extension >>> would be a >>> huge help. >> >> The point is that numpy.distutils wouldn't have to know about Cython >> at all--it would just be handed a bunch of .c files to make extensions >> out of. This way we wouldn't have to make any of the many distutils >> variants Cython-aware. > > cythonize() wouldn't work with a numpy.distutils-using setup.py. We don't pass > the list of Extensions explicitly. > > There are also other non-distutils build tools, like SCons, waf, and yaku, that > would be able to use the .cy extension to distinguish Pyrex files from Cython files. To be fair, having a different extension is only useful if there are both pyrex and cython files in the same project, which is pretty rare. I myself see cython more like a compiler for a more recent dialect that pyrex, a bit like requiring a C99 compiler. Requesting cython instead of pyrex to compile .pyx should not be difficult, except for distutils of course. David From njs at pobox.com Sun Jun 27 08:53:59 2010 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 26 Jun 2010 23:53:59 -0700 Subject: [Cython] .cy extension? In-Reply-To: References: <4C239163.2020504@student.matnat.uio.no> Message-ID: On Sat, Jun 26, 2010 at 7:45 PM, David Cournapeau wrote: > To be fair, having a different extension is only useful if there are > both pyrex and cython files in the same project, which is pretty rare. It's only *necessary* if both are in a project, but it's definitely *useful* to avoid having to specify what's going on to the build tool and to newbies to the code. IMHO. -- Nathaniel From robertwb at math.washington.edu Sun Jun 27 10:54:56 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 27 Jun 2010 01:54:56 -0700 Subject: [Cython] [sage-devel] OpenSUSE source build failure In-Reply-To: References: Message-ID: <2228BF59-CF4D-46A0-8FA3-65FBD0839E1C@math.washington.edu> On Jun 27, 2010, at 1:40 AM, caleb miles wrote: > Hello, > > I am attempting to compile SAGE 4.4.4 on an OpenSUSE cluster for > local use, however, the build process fails while installing Cython, > and I have attached the portion of install.log corresponding to the > build process. Thanks. The key lines seem to be /nano/scratch/camiles/binaries/sage-4.4.4/spkg/build/cython-0.12.1/src/ Cython/Compiler/Parsing.c: In function ?__pyx_f_6Cython_8Compiler_7Parsing_flatten_parallel_assignments?: /nano/scratch/camiles/binaries/sage-4.4.4/spkg/build/cython-0.12.1/src/ Cython/Compiler/Parsing.c:17225: internal compiler error: in merge_alias_info, at tree-ssa-copy.c:235 Please submit a full bug report, with preprocessed source if appropriate. See for instructions. error: command 'gcc' failed with exit status 1 i.e. a compiler error. Can you do gcc -v? > The cluster has four processors, on the administration node I am > working on of the type: > > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 15 > model name : Intel(R) Xeon(R) CPU 5140 @ 2.33GHz > stepping : 6 > cpu MHz : 2327.554 > cache size : 4096 KB > physical id : 0 > siblings : 2 > core id : 0 > cpu cores : 2 > fdiv_bug : no > hlt_bug : no > f00f_bug : no > coma_bug : no > fpu : yes > fpu_exception : yes > cpuid level : 10 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr > pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm > pbe nx lm constant_tsc pni monitor ds_cpl v > mx est tm2 cx16 xtpr lahf_lm > bogomips : 4658.82 > > further information about the computer > > Linux version 2.6.16.27-0.9-chpc (geeko at buildhost) (gcc version > 4.1.0 (SUSE Linux)) #1 SMP Tue Mar 27 15:24:37 MDT 2007 > > I was able to compile SAGE version 3.4.3 on another similar cluster > which is working wonderfully, so I was rather surprised by the > failure in this case. Any help would be greatly appreciated. > > Sincerely, > > Caleb Miles > -- > Cahn's Axiom: > When all else fails, read the instructions. > > > > > > > -- > To post to this group, send an email to sage-devel at googlegroups.com > To unsubscribe from this group, send an email to sage-devel+unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/group/sage-devel > URL: http://www.sagemath.org > From strombrg at gmail.com Mon Jun 28 01:09:38 2010 From: strombrg at gmail.com (Dan Stromberg) Date: Sun, 27 Jun 2010 16:09:38 -0700 Subject: [Cython] Performance difference? Message-ID: What's the size of the performance difference between Cython and a C Extension Module? Obviously this depends on how well each is written, and how many concessions to maintainability are made, but just as rough estimates? If it's not a teensy difference, what is the main thing that accounts for that difference? Is it perhaps the implicit memory management? -- Dan Stromberg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100627/dd088e62/attachment.htm From jsalvati at u.washington.edu Mon Jun 28 04:57:21 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Sun, 27 Jun 2010 19:57:21 -0700 Subject: [Cython] Error when trying to cast a void* to a function pointer Message-ID: Hello Cython list, I am trying to create a generic loop function for building UFuncs, and when I try to compile the following code, I get an error: from numpy cimport * cdef void PyUFunc_i_d(char *args, npy_intp *dimensions, npy_intp *steps, void *extra): cdef npy_intp i cdef npy_intp is1 = steps[0], cdef npy_intp os = steps[1], n = dimensions[0] cdef char *i1 = args[0], *op = args[1] for i in range(n): *(op) = (*extra ) ( *( i1) ) i1 += is1 op += os Error Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef npy_intp os = steps[1], n = dimensions[0] cdef char *i1 = args[0], *op = args[1] for i in range(n): *(op) = (*extra ) ( *( i1) ) ^ ------------------------------------------------------------ ...pymc_numexprtest/src/test/test3.pyx:12:37: Expected an identifier or literal building 'base_functions' extension Am I doing something wrong or is this just not supported? John -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100627/bef5b737/attachment.htm From craigcitro at gmail.com Mon Jun 28 06:37:31 2010 From: craigcitro at gmail.com (Craig Citro) Date: Sun, 27 Jun 2010 21:37:31 -0700 Subject: [Cython] Performance difference? In-Reply-To: References: Message-ID: > What's the size of the performance difference between Cython and a C > Extension Module? ?Obviously this depends on how well each is written, and > how many concessions to maintainability are made, but just as rough > estimates? > If it's not a teensy difference, what is the main thing that accounts for > that difference? ?Is it perhaps the implicit memory management? > I don't think this is something you could give a clear-cut answer to without knowing what you were doing in your C extension module, but I'd say the answer should be something along the lines of "probably negligible, unless you were going to perform crazy black magic in your C extension." And even if you were going to do something crazy -- you could always create a first pass with Cython, and then tweak the parts where the speed was that essential. For instance, if you're planning on interfacing with a C library, and creating/manipulating some C/C++ data structures, then you should get basically the same performance (unless you got tricked into converting back and forth between C and Python, but avoiding that should fall under the "how well each is written" bit in your question). If you're planning on mostly working with Python objects in a fairly safe way, then the answer is probably "there's still a small gap, but we're working on closing it fast." There are definitely situations where we have enough information to do things like avoid typechecks and generate tighter code, but I don't think we always take advantage of everything that's available. We also do a lot of things you probably wouldn't take the time to do yourself -- for instance, we generate custom argument parsing code, which some people have definitely seen some performance improvements from. However, there's one class of things you could do directly in a C extension that we don't, which is "cheat:" you could write a function that takes an arbitrary Python object, but decide that you "know" it will only ever get called with a dict, and directly dispatch to various PyDict_* macros, or access parts of the struct directly. Or you could decide some object doesn't get reused, and just side-effect it instead of allocating a new object to store a result, that kind of thing. You could definitely create some faster, more fragile code this way -- you'd start hitting lots of segfaults if your assumptions were off. (This was the "black magic" I referred to above.) Maybe that's not directed/helpful enough -- if you give us an idea of what kind of C extension you were thinking about writing, we could probably give a much better answer about why Cython is the right tool for the job. ;) -cc From stefan_ml at behnel.de Mon Jun 28 09:06:48 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 28 Jun 2010 09:06:48 +0200 Subject: [Cython] Performance difference? In-Reply-To: References: Message-ID: <4C284A08.3070303@behnel.de> Craig Citro, 28.06.2010 06:37: > For instance, if you're planning on interfacing with a C library, and > creating/manipulating some C/C++ data structures, then you should get > basically the same performance If you do a lot of calling between Python and library code, Cython code is usually faster than the obvious C code. The difference shrinks when you start doing a lot of heavy stuff in C, so that the call overhead gets negligible. In that case, it only depends on how well the computational code is tuned, which usually means that you have to do a lot more work in C than what you'd have to do in Cython to get similar ("close enough") results. Obviously, Cython code cannot be faster than C code for computational code, but in most cases, it's not hard to get it so close that a rewrite in C is no longer worth the effort. Stefan From greg.ewing at canterbury.ac.nz Mon Jun 28 10:12:15 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 28 Jun 2010 20:12:15 +1200 Subject: [Cython] Performance difference? In-Reply-To: References: Message-ID: <4C28595F.5030903@canterbury.ac.nz> Dan Stromberg wrote: > > What's the size of the performance difference between Cython and a C > Extension Module? Obviously this depends on how well each is written, > and how many concessions to maintainability are made, but just as rough > estimates? If you do the meat of the processing using C data types and C operations wherever possible, there should be very little difference. It all depends on how careful and diligent you want to be. So, rough estimates would be: in the best case, no difference; in the worst case, not much better than pure Python. It's hard to say much more than that. -- Greg From greg.ewing at canterbury.ac.nz Mon Jun 28 10:41:12 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 28 Jun 2010 20:41:12 +1200 Subject: [Cython] Error when trying to cast a void* to a function pointer In-Reply-To: References: Message-ID: <4C286028.7020004@canterbury.ac.nz> John Salvatier wrote: > *(op) = (*extra ) ( *( i1) ) You can't use * for dereferencing in Pyrex/Cython. Instead of *p you need to write p[0] But I'm having trouble parsing this part myself: (*extra ) Unless this is some weird Cython construct I'm not aware of, it looks like nonsense. Are you trying to cast 'extra' to a function pointer? If that's the case, the expression you want is something like (op)[0] = (extra)((i1)[0]) -- Greg From greg.ewing at canterbury.ac.nz Mon Jun 28 11:12:25 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 28 Jun 2010 21:12:25 +1200 Subject: [Cython] Performance difference? In-Reply-To: References: Message-ID: <4C286779.60600@canterbury.ac.nz> Craig Citro wrote: > you could write a function > that takes an arbitrary Python object, but decide that you "know" it > will only ever get called with a dict, and directly dispatch to > various PyDict_* macros Pyrex will do this if you declare the object as being of type 'dict' (and it will check the type on the boundary between Python and extension code, so it's still type-safe). Similarly for 'list' and a few other built-in types that have dedicated C APIs. -- Greg From stefan_ml at behnel.de Mon Jun 28 11:30:28 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 28 Jun 2010 11:30:28 +0200 Subject: [Cython] Performance difference? In-Reply-To: <4C286779.60600@canterbury.ac.nz> References: <4C286779.60600@canterbury.ac.nz> Message-ID: <4C286BB4.2050402@behnel.de> Greg Ewing, 28.06.2010 11:12: > Craig Citro wrote: >> you could write a function >> that takes an arbitrary Python object, but decide that you "know" it >> will only ever get called with a dict, and directly dispatch to >> various PyDict_* macros > > Pyrex will do this if you declare the object as being of > type 'dict' (and it will check the type on the boundary > between Python and extension code, so it's still type-safe). > Similarly for 'list' and a few other built-in types that > have dedicated C APIs. Right, this is totally not "black magic". There are some cases, though, where you can easily write C-API code that outperforms Cython code. A well known example is a call to PyDict_GetItem() in C and a subsequent check for a NULL return value. Cython code needs to read the value into a variable first and then test it, a common idiom being result = d.get(key) if result is not None: ... So the compiler would need to know that the None return value is no longer used later on, i.e. can safely be allowed to be NULL instead of None, so that it can generate simpler code without additional ref-counting overhead, and the C compiler can see more directly what happens. It's even worse for the exception catching idiom: try: result = d[key] except KeyError: pass else: ... which is a relatively efficient (and therefore common) way to do it in Python, but could really collapse into a single call to PyDict_GetItem() plus a NULL check in the generated C code. Cython isn't that smart yet. But in general, this is really chasing microseconds in places where it rarely matters. Stefan From jsalvati at u.washington.edu Mon Jun 28 16:01:40 2010 From: jsalvati at u.washington.edu (John Salvatier) Date: Mon, 28 Jun 2010 07:01:40 -0700 Subject: [Cython] Error when trying to cast a void* to a function pointer In-Reply-To: <4C286028.7020004@canterbury.ac.nz> References: <4C286028.7020004@canterbury.ac.nz> Message-ID: Ok, thank you. On Mon, Jun 28, 2010 at 1:41 AM, Greg Ewing wrote: > John Salvatier wrote: > > > *(op) = (*extra ) ( *( i1) ) > > You can't use * for dereferencing in Pyrex/Cython. Instead of > > *p > > you need to write > > p[0] > > But I'm having trouble parsing this part myself: > > (*extra ) > > Unless this is some weird Cython construct I'm not aware of, > it looks like nonsense. Are you trying to cast 'extra' to > a function pointer? > > If that's the case, the expression you want is something like > > (op)[0] = (extra)((i1)[0]) > > -- > Greg > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/1f0872d2/attachment.htm From robertwb at math.washington.edu Mon Jun 28 16:32:17 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Jun 2010 07:32:17 -0700 Subject: [Cython] Performance difference? In-Reply-To: References: Message-ID: On Jun 27, 2010, at 4:09 PM, Dan Stromberg wrote: > What's the size of the performance difference between Cython and a C > Extension Module? Obviously this depends on how well each is > written, and how many concessions to maintainability are made, but > just as rough estimates? > > If it's not a teensy difference, what is the main thing that > accounts for that difference? Is it perhaps the implicit memory > management? The first thing to note is that Cython compiles down to C extension modules, so what you're asking is how Cython-generated modules compare to hand written ones. An analogy can be made with other compilers-- what is the performance of C vs. hand generated assembly? Obviously, this depends on the compiler and the human, but I would say in our case that, especially for sufficiently typed code, Cython usually emits the same code you would write in C, and the boilerplate code is more efficient than what a human would usually bother with. As with C vs. assembly, it is possible cut corners and beat the compiler in some cases, but this usually involves a high level of expertise, sacrificing portability/versatility, more easily shooting yourself in the foot, and a whole lot of timings of cases that the compilers never thought of. (Granted, Cython is not as mature as something like GCC, but it's a simpler problem and constantly improving.) Even then, most people I know of who use assembly start with C compiler output and tweak it from there (if necessary), which is worth doing with Cython. (Personally, I would never bother hand generating C until at least looking at what Cython gave me.) The average user, however, isn't going to do much better hand coding stuff in the vast majority of cases though. - Robert From jared at jaredforsyth.com Mon Jun 28 16:45:09 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 08:45:09 -0600 Subject: [Cython] Bitbucket? Message-ID: Any chance of cython moving to bitbucket (or some similar hosted service)? I'd love to be able to fork/pull-request with reckless abandon.... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/468b6f6d/attachment.htm From jared at jaredforsyth.com Mon Jun 28 16:43:23 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 08:43:23 -0600 Subject: [Cython] "can't find pxd" Message-ID: Apparently, if your .pyx is inside a "package" -- nested inside a few directories -- cython's searching protocol changes -- without telling you. It *won't look in the current directory*, instead opting for the base package dir. I've fixed it to first look in the package base, and then in the current directory. diff -r 8bff3332e34f Cython/Compiler/Main.py --- a/Cython/Compiler/Main.py Tue Feb 02 02:10:32 2010 -0800 +++ b/Cython/Compiler/Main.py Mon Jun 28 08:42:42 2010 -0600 @@ -337,7 +337,7 @@ if include: dirs = [os.path.dirname(file_desc.filename)] + dirs else: - dirs = [self.find_root_package_dir(file_desc.filename)] + dirs + dirs = [self.find_root_package_dir(file_desc.filename), os.path.dirname(file_desc.filename)] + dirs dotted_filename = qualified_name if suffix: -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/5f097b5d/attachment.htm From jared at jaredforsyth.com Mon Jun 28 17:04:27 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 09:04:27 -0600 Subject: [Cython] Loop conversion Message-ID: In the manual, it states that "for i in range(...)" gets automatically converted to "for i from..." for performance benefits. But it doesn't... I've tried cdef int a=5 for i in range(a):pass and for i in range(10):pass and neither are converted. Is the documentation wrong? [it seems like at least the second one should be optimized] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/0074216d/attachment.htm From kwmsmith at gmail.com Mon Jun 28 17:22:36 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 28 Jun 2010 10:22:36 -0500 Subject: [Cython] Loop conversion In-Reply-To: References: Message-ID: On Mon, Jun 28, 2010 at 10:04 AM, Jared Forsyth wrote: > In the manual, it states that "for i in range(...)" gets automatically > converted to "for i from..." for performance benefits. > But it doesn't... I've tried > cdef int a=5 > for i in range(a):pass > and > for i in range(10):pass If you don't type the 'i' in the above examples as some kind of integer, it will fallback to Python code. Try: cdef int a=5 cdef int i for i in range(a): pass > and neither are converted. Is the documentation wrong? [it seems like at > least the second one should be optimized] > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > From stefan_ml at behnel.de Mon Jun 28 17:29:35 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 28 Jun 2010 17:29:35 +0200 Subject: [Cython] Loop conversion In-Reply-To: References: Message-ID: <4C28BFDF.2060308@behnel.de> Jared Forsyth, 28.06.2010 17:04: > In the manual, it states that "for i in range(...)" gets automatically > converted to "for i from..." for performance benefits. Sounds like you didn't read on from the point where it said that. If that's really all it says, please provide a link to the docs, so that we can fix it. > But it doesn't... I've tried > > cdef int a=5 > for i in range(a):pass > > and > > for i in range(10):pass > > and neither are converted. Is the documentation wrong? [it seems like at > least the second one should be optimized] You didn't state which Cython version you are using, so I assume it's 0.12.1, which doesn't automatically optimise these things yet. You have to explicitly tell it that it's safe to restrict 'i' to a C integer in order to let it break the normal Python semantics of your code. Cython 0.13 will recognise the second case, though, and I think even the first one. Then, the C compiler should be able to drop the above loops as dead code, as expected. Stefan From jared at jaredforsyth.com Mon Jun 28 17:57:16 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 09:57:16 -0600 Subject: [Cython] Loop conversion In-Reply-To: <4C28BFDF.2060308@behnel.de> References: <4C28BFDF.2060308@behnel.de> Message-ID: Sorry for no link -- I was using the PDF...which looks like it's not actually the newest version of the docs (says for cython 0.11.2). Here's the link in the online docs. And no, it doesn't really say much... This will convert statements of the form for i in range(...) to for i from > ... when i is any cdef?d integer type, and the direction (i.e. sign of step) can be determined. On Mon, Jun 28, 2010 at 9:29 AM, Stefan Behnel wrote: > Jared Forsyth, 28.06.2010 17:04: > > In the manual, it states that "for i in range(...)" gets automatically > > converted to "for i from..." for performance benefits. > > Sounds like you didn't read on from the point where it said that. If that's > really all it says, please provide a link to the docs, so that we can fix > it. > > > > But it doesn't... I've tried > > > > cdef int a=5 > > for i in range(a):pass > > > > and > > > > for i in range(10):pass > > > > and neither are converted. Is the documentation wrong? [it seems like at > > least the second one should be optimized] > > You didn't state which Cython version you are using, so I assume it's > 0.12.1, which doesn't automatically optimise these things yet. You have to > explicitly tell it that it's safe to restrict 'i' to a C integer in order > to let it break the normal Python semantics of your code. > > Cython 0.13 will recognise the second case, though, and I think even the > first one. Then, the C compiler should be able to drop the above loops as > dead code, as expected. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/71e69e95/attachment.htm From craigcitro at gmail.com Mon Jun 28 19:34:30 2010 From: craigcitro at gmail.com (Craig Citro) Date: Mon, 28 Jun 2010 10:34:30 -0700 Subject: [Cython] Performance difference? In-Reply-To: <4C286779.60600@canterbury.ac.nz> References: <4C286779.60600@canterbury.ac.nz> Message-ID: > Pyrex will do this if you declare the object as being of > type 'dict' (and it will check the type on the boundary > between Python and extension code, so it's still type-safe). > Similarly for 'list' and a few other built-in types that > have dedicated C APIs. > Sure, Cython will also (usually) do this if you *explicitly declare* the type -- any places where it doesn't qualify as TODOs in my head, anyway. In fact, now that we have some type inferencing going on, Cython will even do this in some cases where you don't declare the type, but we can deduce it. In the example above, though, notice I said "a function that takes an arbitrary Python object" -- I'm talking about the case where there *is* no explicit type information, but you still want to avoid typechecks and dispatching into generic code. I'm talking about a snippet like this: cdef void foo(object x): if isinstance(x, dict): x['spam'] = 5 Is Pyrex smart enough to infer that x is in fact a dict, and emit a call to PyDict_GetItemString or the like? I don't think Cython is right now, but I sure would like to make it smart enough to handle that case. Similarly, if someone made a call to a library function that always returned a dict, it'd be nice to tag the resulting object as a dict for the same kind of things. I think there are at least three key ingredients to handle things like the above: (1) type inference, (2) control-flow analysis, and (3) potentially shadowing variables (if x is used for several different Python types in a block). These are all in the vein of "we don't do them yet, but we want to." That said, the real "black magic" I was referring to is beyond that example: I'm talking about code that *doesn't* have explicit type declarations, and *doesn't* do any type testing -- just blindly takes an arbitrary PyObject * and calls off to a PyDict_* method, or manually reaches into the underlying PyDictObject struct and does something crazy. Of course, this isn't something you can do from Python -- but it *is* something you could do from C. These are generally horribly unsafe things to do, which is what I was saying about segfaults above -- however, there are times that speed is more important than any sort of safety, and that's a barrier you can't cross in Pyrex/Cython without directly using the Python/C API yourself, I think. -cc From craigcitro at gmail.com Mon Jun 28 19:42:08 2010 From: craigcitro at gmail.com (Craig Citro) Date: Mon, 28 Jun 2010 10:42:08 -0700 Subject: [Cython] Performance difference? In-Reply-To: <4C286BB4.2050402@behnel.de> References: <4C286779.60600@canterbury.ac.nz> <4C286BB4.2050402@behnel.de> Message-ID: >> Pyrex will do this if you declare the object as being of >> type 'dict' (and it will check the type on the boundary >> between Python and extension code, so it's still type-safe). >> Similarly for 'list' and a few other built-in types that >> have dedicated C APIs. > > Right, this is totally not "black magic". > Of course not -- but as I mentioned above, this isn't what I was referring to. I was referring to cases where either (1) there's no *explicit* type information, which I think we can handle many of as Cython gets smarter, and (2) cases where it's not *possible* to infer the type information, but you have some sort of implicit contract (because, say, you're writing the code that calls this C extension). > There are some cases, though, where you can easily write C-API code that > outperforms Cython code. A well known example is a call to PyDict_GetItem() > in C and a subsequent check for a NULL return value. Cython code needs to > read the value into a variable first and then test it, a common idiom being > > ? ? result = d.get(key) > ? ? if result is not None: > ? ? ? ?... > Maybe I'm being too optimistic, but there are some cases like this that we *could* teach the compiler to figure out, right? It would probably involve *some* amount of Python-specific analysis, and we surely couldn't catch *all* of the cases, but we could probably get the lion's share of the idiomatic ones ... > But in general, this is really chasing microseconds in places where it > rarely matters. > Sure. But that said, if you *really* do care about those last few microseconds -- because maybe you're obsessed with irrelevant microbenchmarks like I've often been? ;) -- there is a little extra you can squeeze with direct use of the Python/C API. -cc From dalcinl at gmail.com Mon Jun 28 22:52:18 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 28 Jun 2010 17:52:18 -0300 Subject: [Cython] Py3: __truediv__ and __div__ Message-ID: While porting petsc4py to work in Python 3, I noticed that __div__ special method is ignored. Should __truediv__ have __div__ as a fallback ? PS: the bytes/unicode thing was again a major pain. I had to insert a call to a helper function at EVERY point the str -> bytes -> char* conversion was needed. Stefan, any chance you have implemented some sugar I'm not aware of? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From jared at jaredforsyth.com Mon Jun 28 23:18:00 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 15:18:00 -0600 Subject: [Cython] "can't find pxd" In-Reply-To: References: Message-ID: So, ignore that patch. I realize now that package level importing is required for c modules -- why?? -- anyway, the real inconsistency is that relative importing is required in the pxd files... I would rather all one way or all the other. On Mon, Jun 28, 2010 at 8:43 AM, Jared Forsyth wrote: > Apparently, if your .pyx is inside a "package" -- nested inside a few > directories -- cython's searching protocol changes -- without telling you. > It *won't look in the current directory*, instead opting for the base > package dir. > I've fixed it to first look in the package base, and then in the current > directory. > > diff -r 8bff3332e34f Cython/Compiler/Main.py > --- a/Cython/Compiler/Main.py Tue Feb 02 02:10:32 2010 -0800 > +++ b/Cython/Compiler/Main.py Mon Jun 28 08:42:42 2010 -0600 > @@ -337,7 +337,7 @@ > if include: > dirs = [os.path.dirname(file_desc.filename)] + dirs > else: > - dirs = [self.find_root_package_dir(file_desc.filename)] + > dirs > + dirs = [self.find_root_package_dir(file_desc.filename), > os.path.dirname(file_desc.filename)] + dirs > > dotted_filename = qualified_name > if suffix: > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/0e113a31/attachment.htm From dalcinl at gmail.com Mon Jun 28 23:30:56 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 28 Jun 2010 18:30:56 -0300 Subject: [Cython] "can't find pxd" In-Reply-To: References: Message-ID: On 28 June 2010 18:18, Jared Forsyth wrote: > So, ignore that patch. I realize now that package level importing is > required for c modules -- why?? -- anyway, the real inconsistency is that > relative importing is required in the pxd files... I would rather all one > way or all the other. > Sorry, could you elaborate a bit more about your complains? Could you provide a few little examples? > On Mon, Jun 28, 2010 at 8:43 AM, Jared Forsyth > wrote: >> >> Apparently, if your .pyx is inside a "package" -- nested inside a few >> directories -- cython's searching protocol changes -- without telling you. >> It *won't look in the current directory*, instead opting for the base >> package dir. >> I've fixed it to first look in the package base, and then in the current >> directory. >> diff -r 8bff3332e34f Cython/Compiler/Main.py >> --- a/Cython/Compiler/Main.py ? Tue Feb 02 02:10:32 2010 -0800 >> +++ b/Cython/Compiler/Main.py ? Mon Jun 28 08:42:42 2010 -0600 >> @@ -337,7 +337,7 @@ >> ?? ? ? ? ? ? if include: >> ?? ? ? ? ? ? ? ? dirs = [os.path.dirname(file_desc.filename)] + dirs >> ?? ? ? ? ? ? else: >> - ? ? ? ? ? ? ? ?dirs = [self.find_root_package_dir(file_desc.filename)] + >> dirs >> + ? ? ? ? ? ? ? ?dirs = [self.find_root_package_dir(file_desc.filename), >> os.path.dirname(file_desc.filename)] + dirs >> >> ?? ? ? ? dotted_filename = qualified_name >> ?? ? ? ? if suffix: > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From greg.ewing at canterbury.ac.nz Tue Jun 29 02:38:02 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 29 Jun 2010 12:38:02 +1200 Subject: [Cython] Performance difference? In-Reply-To: References: <4C286779.60600@canterbury.ac.nz> Message-ID: <4C29406A.5050109@canterbury.ac.nz> Craig Citro wrote: > I'm talking about a > snippet like this: > > cdef void foo(object x): > if isinstance(x, dict): > x['spam'] = 5 > > Is Pyrex smart enough to infer that x is in fact a dict, and emit a > call to PyDict_GetItemString or the like? No, it's not, but you can get the same effect by casting the object to a dict: cdef void foo(object x): cdef dict d if isinstance(x, dict): d = x d['spam'] = 5 Or if you don't care about type safety, you can skip the type test and just do cdef void foo(object x): cdef dict d d = x d['spam'] = 5 -- Greg From jared at jaredforsyth.com Tue Jun 29 03:16:26 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 19:16:26 -0600 Subject: [Cython] "can't find pxd" In-Reply-To: References: Message-ID: Here's an example directory structure: one/ one/__init__.py one/two/ one/two/__init__.py one/two/foo.pyx one/two/bar.pxd one/two/bar.pyx one/two/baz.pxd foo *cimport*s bar and baz. baz defined some *struct*s, and so has to be *cimport*ed by bar.pxd In order to do this, in *foo.pyx *and *bar.pyx* > from one.two.baz cimport * whereas in *bar.pxd* > from baz cimport * I believe it should be consistent -- require the full package namespacing in pxd files. ** On Mon, Jun 28, 2010 at 3:30 PM, Lisandro Dalcin wrote: > On 28 June 2010 18:18, Jared Forsyth wrote: > > So, ignore that patch. I realize now that package level importing is > > required for c modules -- why?? -- anyway, the real inconsistency is that > > relative importing is required in the pxd files... I would rather all one > > way or all the other. > > > > Sorry, could you elaborate a bit more about your complains? Could you > provide a few little examples? > > > > On Mon, Jun 28, 2010 at 8:43 AM, Jared Forsyth > > wrote: > >> > >> Apparently, if your .pyx is inside a "package" -- nested inside a few > >> directories -- cython's searching protocol changes -- without telling > you. > >> It *won't look in the current directory*, instead opting for the base > >> package dir. > >> I've fixed it to first look in the package base, and then in the current > >> directory. > >> diff -r 8bff3332e34f Cython/Compiler/Main.py > >> --- a/Cython/Compiler/Main.py Tue Feb 02 02:10:32 2010 -0800 > >> +++ b/Cython/Compiler/Main.py Mon Jun 28 08:42:42 2010 -0600 > >> @@ -337,7 +337,7 @@ > >> if include: > >> dirs = [os.path.dirname(file_desc.filename)] + dirs > >> else: > >> - dirs = [self.find_root_package_dir(file_desc.filename)] > + > >> dirs > >> + dirs = [self.find_root_package_dir(file_desc.filename), > >> os.path.dirname(file_desc.filename)] + dirs > >> > >> dotted_filename = qualified_name > >> if suffix: > > > > > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/cython-dev > > > > > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/e1c3eaa1/attachment.htm From robertwb at math.washington.edu Tue Jun 29 06:26:53 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Jun 2010 21:26:53 -0700 Subject: [Cython] Loop conversion In-Reply-To: References: <4C28BFDF.2060308@behnel.de> Message-ID: <5889F0A9-4DD5-454D-8C51-A323E7F6DFF1@math.washington.edu> On Jun 28, 2010, at 8:57 AM, Jared Forsyth wrote: > Sorry for no link -- I was using the PDF...which looks like it's not > actually the newest version of the docs (says for cython 0.11.2). > Here's the link in the online docs. > And no, it doesn't really say much... > > This will convert statements of the form for i in range(...) to for > i from ... when i is any cdef?d integer type, and the direction > (i.e. sign of step) can be determined. The issue is that you have to type i (the index variable itself). > > > On Mon, Jun 28, 2010 at 9:29 AM, Stefan Behnel > wrote: > Jared Forsyth, 28.06.2010 17:04: > > In the manual, it states that "for i in range(...)" gets > automatically > > converted to "for i from..." for performance benefits. > > Sounds like you didn't read on from the point where it said that. If > that's > really all it says, please provide a link to the docs, so that we > can fix it. > > > > But it doesn't... I've tried > > > > cdef int a=5 > > for i in range(a):pass > > > > and > > > > for i in range(10):pass > > > > and neither are converted. Is the documentation wrong? [it seems > like at > > least the second one should be optimized] > > You didn't state which Cython version you are using, so I assume it's > 0.12.1, which doesn't automatically optimise these things yet. You > have to > explicitly tell it that it's safe to restrict 'i' to a C integer in > order > to let it break the normal Python semantics of your code. > > Cython 0.13 will recognise the second case, though, and I think even > the > first one. Then, the C compiler should be able to drop the above > loops as > dead code, as expected. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Tue Jun 29 06:44:54 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Jun 2010 21:44:54 -0700 Subject: [Cython] Bitbucket? In-Reply-To: References: Message-ID: On Jun 28, 2010, at 7:45 AM, Jared Forsyth wrote: > Any chance of cython moving to bitbucket (or some similar hosted > service)? I'd love to be able to fork/pull-request with reckless > abandon.... I'm not seeing why we would move--Mercurial is a DVCS, so you can fork/ pull with reckless abandon already (and publish your changes, if you so desire, up on bitbucket--I believe there's already a couple of personal Cython repositories up there already, and we're totally OK with people doing that). - Robert From jared at jaredforsyth.com Tue Jun 29 06:46:36 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 22:46:36 -0600 Subject: [Cython] stupid race conditions Arggghh Message-ID: Ok, so I just spent ~an hour hunting this down...and it is *really weird*. I've pared down the code as much as possible while preserving the anomaly. cdef one(): print two() # should be "33"...but its actually "stdout" cdef char* two(): cdef char* three = four() print three ## will be "33" return three cdef char* four(): cdef char* a = '33' cdef char* b = '33' c = '' python_vbl = b # looks like a no-op, but if you take it out, the anomaly disappears c = a # is not "33" return c # returing "33" one() and...the output is >>> import strange 33 stdout VERSION: HEAD -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/075546cd/attachment.htm From jared at jaredforsyth.com Tue Jun 29 06:54:27 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Mon, 28 Jun 2010 22:54:27 -0600 Subject: [Cython] Bitbucket? In-Reply-To: References: Message-ID: The advantages that I see are 1) Issue tracking 2) better/clearer communication between core committers and us peons. On Mon, Jun 28, 2010 at 10:44 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Jun 28, 2010, at 7:45 AM, Jared Forsyth wrote: > > > Any chance of cython moving to bitbucket (or some similar hosted > > service)? I'd love to be able to fork/pull-request with reckless > > abandon.... > > I'm not seeing why we would move--Mercurial is a DVCS, so you can fork/ > pull with reckless abandon already (and publish your changes, if you > so desire, up on bitbucket--I believe there's already a couple of > personal Cython repositories up there already, and we're totally OK > with people doing that). > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100628/acfa0f98/attachment.htm From robertwb at math.washington.edu Tue Jun 29 07:03:52 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Jun 2010 22:03:52 -0700 Subject: [Cython] stupid race conditions Arggghh In-Reply-To: References: Message-ID: <2FBE5C58-F8B8-43BE-8EC4-C4B6DD3EF1BE@math.washington.edu> On Jun 28, 2010, at 9:46 PM, Jared Forsyth wrote: > Ok, so I just spent ~an hour hunting this down...and it is really > weird. I've pared down the code as much as possible while preserving > the anomaly. > > cdef one(): > print two() # should be "33"...but its actually "stdout" > > cdef char* two(): > cdef char* three = four() > print three ## will be "33" > return three > > cdef char* four(): > cdef char* a = '33' > cdef char* b = '33' > c = '' > python_vbl = b # looks like a no-op, but if you take it out, > the anomaly disappears > c = a # is not "33" > return c # returing "33" > > one() > > and...the output is > > >>> import strange > 33 > stdout The problem is that c is a Python string, which is getting cast into a char*, which is no longer valid once c goes out of scope (though it's non-deterministic exactly when the memory in question will get overwritten). We try to detect and prevent abuse of automatic conversion of str/bytes to char*, but aren't catching it in this case. - Robert From robertwb at math.washington.edu Tue Jun 29 07:05:55 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Jun 2010 22:05:55 -0700 Subject: [Cython] Bitbucket? In-Reply-To: References: Message-ID: On Jun 28, 2010, at 9:54 PM, Jared Forsyth wrote: > The advantages that I see are > 1) Issue tracking That's what we use http://trac.cython.org/cython_trac for. > 2) better/clearer communication between core committers and us peons. Perhaps I'm not understanding you, could you clarify how hosting the repositories on a 3rd party site would help with this? - Robert > > On Mon, Jun 28, 2010 at 10:44 PM, Robert Bradshaw > wrote: > On Jun 28, 2010, at 7:45 AM, Jared Forsyth wrote: > > > Any chance of cython moving to bitbucket (or some similar hosted > > service)? I'd love to be able to fork/pull-request with reckless > > abandon.... > > I'm not seeing why we would move--Mercurial is a DVCS, so you can > fork/ > pull with reckless abandon already (and publish your changes, if you > so desire, up on bitbucket--I believe there's already a couple of > personal Cython repositories up there already, and we're totally OK > with people doing that). > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Tue Jun 29 08:48:47 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jun 2010 08:48:47 +0200 Subject: [Cython] Py3: __truediv__ and __div__ In-Reply-To: References: Message-ID: <4C29974F.4060609@behnel.de> Lisandro Dalcin, 28.06.2010 22:52: > While porting petsc4py to work in Python 3, I noticed that __div__ > special method is ignored. Should __truediv__ have __div__ as a > fallback ? -1, they're different things: >>> class test(object): ... def __div__(a,b): return b ... >>> t = test() >>> t / 2 2 >>> t // 2 Traceback (most recent call last): TypeError: unsupported operand type(s) for //: 'test' and 'int' And I can't find a fixer in 2to3 either. What about a warning instead when __div__ is defined but __truediv__ is not? > PS: the bytes/unicode thing was again a major pain. I had to insert a > call to a helper function at EVERY point the str -> bytes -> char* > conversion was needed. Stefan, any chance you have implemented some > sugar I'm not aware of? No. You need a helper function here, as you need to check if the str input is valid input for your code. You didn't say what kind of input it can accept, but assuming it's a subset of ASCII, you will have to make sure it's really a suitable string, both in Py2 and Py3. Decoding a unicode string here is not substantially more work, neither in Py2 nor in Py3 (assuming that you accept unicode strings in Py2 as well, which I hope you do). Encoding is really just a small part of the work, but remember that the problem here is Py2, not Py3. Input checking is much easier in Py3, where you only need to call .encode(target_encoding) on the input string, and bytes input as well as incompatible unicode input will raise an exception for free. Then all you have to do is check the data for invalid bytes afterwards, such as NULL bytes or other constrains. In Py2, you have to check the type of the string, then either encode it (unicode input) or check the byte data for incompatible input data, and only then can you take the result of either of the two and check it for invalid bytes. That's a lot more complicated. However, in both cases, a helper function is the right way to do it, as it builds a single point of truth that holds your input constrains. Stefan From stefan_ml at behnel.de Tue Jun 29 08:50:51 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jun 2010 08:50:51 +0200 Subject: [Cython] stupid race conditions Arggghh In-Reply-To: References: Message-ID: <4C2997CB.8040805@behnel.de> Jared Forsyth, 29.06.2010 06:46: > VERSION: HEAD Erm, which "HEAD" from where? Stefan From jared at jaredforsyth.com Tue Jun 29 09:11:22 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Tue, 29 Jun 2010 01:11:22 -0600 Subject: [Cython] stupid race conditions Arggghh In-Reply-To: <4C2997CB.8040805@behnel.de> References: <4C2997CB.8040805@behnel.de> Message-ID: the hg repo http://hg.cython.org/cython On Tue, Jun 29, 2010 at 12:50 AM, Stefan Behnel wrote: > Jared Forsyth, 29.06.2010 06:46: > > VERSION: HEAD > > Erm, which "HEAD" from where? > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100629/35496de5/attachment-0001.htm From stefan_ml at behnel.de Tue Jun 29 09:21:51 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jun 2010 09:21:51 +0200 Subject: [Cython] stupid race conditions Arggghh In-Reply-To: References: <4C2997CB.8040805@behnel.de> Message-ID: <4C299F0F.4050403@behnel.de> Jared Forsyth, 29.06.2010 09:11: > On Tue, Jun 29, 2010 at 12:50 AM, Stefan Behnel wrote: > >> Jared Forsyth, 29.06.2010 06:46: >>> VERSION: HEAD >> >> Erm, which "HEAD" from where? >> > the hg repo http://hg.cython.org/cython Ah, that's the "old" release version. You may want to try http://hg.cython.org/cython-devel or http://hg.cython.org/cython-closures instead. They're what is about to become Cython 0.13. Stefan From dalcinl at gmail.com Tue Jun 29 17:35:23 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 29 Jun 2010 12:35:23 -0300 Subject: [Cython] Py3: __truediv__ and __div__ In-Reply-To: <4C29974F.4060609@behnel.de> References: <4C29974F.4060609@behnel.de> Message-ID: On 29 June 2010 03:48, Stefan Behnel wrote: > Lisandro Dalcin, 28.06.2010 22:52: >> While porting petsc4py to work in Python 3, I noticed that __div__ >> special method is ignored. Should __truediv__ have __div__ as a >> fallback ? > > -1, they're different things: > > ? >>> class test(object): > ? ... ? def __div__(a,b): return b > ? ... > ? >>> t = test() > ? >>> t / 2 > ? 2 > ? >>> t // 2 > ? Traceback (most recent call last): > ? TypeError: unsupported operand type(s) for //: 'test' and 'int' > > And I can't find a fixer in 2to3 either. > > What about a warning instead when __div__ is defined but __truediv__ is not? > OK, a warning would work. However, in Python 2, __truediv__ without __div__ should also generate a warning; try your example with "from __future__ import division", it will fail with TypeError. This is a mess, still not sure what to do. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Tue Jun 29 17:43:48 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 29 Jun 2010 12:43:48 -0300 Subject: [Cython] "can't find pxd" In-Reply-To: References: Message-ID: On 28 June 2010 22:16, Jared Forsyth wrote: > Here's an example directory structure: > one/ > one/__init__.py > one/two/ > one/two/__init__.py > one/two/foo.pyx > one/two/bar.pxd > one/two/bar.pyx > one/two/baz.pxd > foo cimports bar and baz. > baz defined some structs, and so has to be cimported by bar.pxd > In order to do this, in foo.pyx and bar.pyx >> >> from one.two.baz cimport * > > whereas in bar.pxd >> >> from baz cimport * > > I believe it should be consistent -- require the full package namespacing in > pxd files. Are you using cython-devel repo? If you can reproduce this with cython-devel, could you send a tarball/zip with the self-contained example (including a shell script or setup.py file showing how you cythonize the *.pyx sources)? I'm really bussy, but if you help me a bit, I can help you to get this fixed. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Tue Jun 29 17:43:46 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jun 2010 17:43:46 +0200 Subject: [Cython] Py3: __truediv__ and __div__ In-Reply-To: References: <4C29974F.4060609@behnel.de> Message-ID: <4C2A14B2.4010104@behnel.de> Lisandro Dalcin, 29.06.2010 17:35: > On 29 June 2010 03:48, Stefan Behnel wrote: >> Lisandro Dalcin, 28.06.2010 22:52: >>> While porting petsc4py to work in Python 3, I noticed that __div__ >>> special method is ignored. Should __truediv__ have __div__ as a >>> fallback ? >> >> -1, they're different things: >> >> >>> class test(object): >> ... def __div__(a,b): return b >> ... >> >>> t = test() >> >>> t / 2 >> 2 >> >>> t // 2 >> Traceback (most recent call last): >> TypeError: unsupported operand type(s) for //: 'test' and 'int' >> >> And I can't find a fixer in 2to3 either. >> >> What about a warning instead when __div__ is defined but __truediv__ is not? > > OK, a warning would work. However, in Python 2, __truediv__ without > __div__ should also generate a warning; try your example with "from > __future__ import division", it will fail with TypeError. This is a > mess, still not sure what to do. ... certainly not the first time that I regret that Cython is following Python 2. Stefan From jared at jaredforsyth.com Tue Jun 29 17:46:55 2010 From: jared at jaredforsyth.com (Jared Forsyth) Date: Tue, 29 Jun 2010 09:46:55 -0600 Subject: [Cython] stupid race conditions Arggghh In-Reply-To: <4C299F0F.4050403@behnel.de> References: <4C2997CB.8040805@behnel.de> <4C299F0F.4050403@behnel.de> Message-ID: which do you recommend? On Tue, Jun 29, 2010 at 1:21 AM, Stefan Behnel wrote: > Jared Forsyth, 29.06.2010 09:11: > > On Tue, Jun 29, 2010 at 12:50 AM, Stefan Behnel wrote: > > > >> Jared Forsyth, 29.06.2010 06:46: > >>> VERSION: HEAD > >> > >> Erm, which "HEAD" from where? > >> > > the hg repo http://hg.cython.org/cython > > Ah, that's the "old" release version. You may want to try > > http://hg.cython.org/cython-devel > > or > > http://hg.cython.org/cython-closures > > instead. They're what is about to become Cython 0.13. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100629/f03e0d40/attachment.htm From stefan_ml at behnel.de Tue Jun 29 18:46:42 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Jun 2010 18:46:42 +0200 Subject: [Cython] stupid race conditions Arggghh In-Reply-To: References: <4C2997CB.8040805@behnel.de> <4C299F0F.4050403@behnel.de> Message-ID: <4C2A2372.7080407@behnel.de> [moving your reply where it belongs] Jared Forsyth, 29.06.2010 17:46: > > On Tue, Jun 29, 2010 at 1:21 AM, Stefan Behnel wrote: > >> Jared Forsyth, 29.06.2010 09:11: >>> On Tue, Jun 29, 2010 at 12:50 AM, Stefan Behnel wrote: >>> >>>> Jared Forsyth, 29.06.2010 06:46: >>>>> VERSION: HEAD >>>> >>>> Erm, which "HEAD" from where? >>>> >>> the hg repo http://hg.cython.org/cython >> >> Ah, that's the "old" release version. You may want to try >> >> http://hg.cython.org/cython-devel >> >> or >> >> http://hg.cython.org/cython-closures >> >> instead. They're what is about to become Cython 0.13. > > which do you recommend? The latter, it tends to be a superset of the first. Stefan From kwmsmith at gmail.com Tue Jun 29 23:26:57 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 29 Jun 2010 16:26:57 -0500 Subject: [Cython] Quick cython status update at SciPy 2010 In-Reply-To: <4C2A5C04.1070106@behnel.de> References: <4C2A5C04.1070106@behnel.de> Message-ID: On Tue, Jun 29, 2010 at 3:48 PM, Stefan Behnel wrote: > Kurt Smith, 29.06.2010 21:40: >> >> Jarrod asked me to give a quick no-slide status update on Cython (2-5 >> minutes) tomorrow. ?I've been mildly following the mailing list and >> noticed much work going on with automatic type infererence, closures, >> generators and genexps. ?What else should I include on the bullet >> list? ?If there's a wiki page with it all collated just point me >> there. > > Check the (unfinished) 0.13 release notes in the Wiki. Thanks. Is it possible to give a rough estimate 0.13's release date, for those who might ask? End of summer, end of year? I'll mention some things from Hayou's GSoC project and Danilo's C++ integration from last year. Kurt From robertwb at math.washington.edu Tue Jun 29 23:32:54 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 29 Jun 2010 14:32:54 -0700 (PDT) Subject: [Cython] Quick cython status update at SciPy 2010 In-Reply-To: References: <4C2A5C04.1070106@behnel.de> Message-ID: On Tue, 29 Jun 2010, Kurt Smith wrote: > On Tue, Jun 29, 2010 at 3:48 PM, Stefan Behnel wrote: >> Kurt Smith, 29.06.2010 21:40: >>> >>> Jarrod asked me to give a quick no-slide status update on Cython (2-5 >>> minutes) tomorrow. ?I've been mildly following the mailing list and >>> noticed much work going on with automatic type infererence, closures, >>> generators and genexps. ?What else should I include on the bullet >>> list? ?If there's a wiki page with it all collated just point me >>> there. >> >> Check the (unfinished) 0.13 release notes in the Wiki. > > Thanks. > > Is it possible to give a rough estimate 0.13's release date, for those > who might ask? End of summer, end of year? > > I'll mention some things from Hayou's GSoC project and Danilo's C++ > integration from last year. Really soon, I hope (weeks at most). Unlike the last several months, I'll actually be able to work on it again, and we're way overdue. - Robert From luiji at users.sourceforge.net Wed Jun 30 06:55:39 2010 From: luiji at users.sourceforge.net (Luiji Maryo) Date: Wed, 30 Jun 2010 00:55:39 -0400 Subject: [Cython] (no subject) Message-ID: Hello, For a project I am working on I am creating a super-optimized Python distrobution of which I hope will reach heights of 10,000 times faster code. ?What I am planning to do is bundle Psyco with a refactor of the Standard Python Library. ?The refactor is to be a modified version where I use optimization abilities from Cython like CDEFs to increase code speed, after which I will compile it all into native machine code modules. My inquisitions are: - Has this already been done? - Do people have any suggestions (such as using something other then Psyco). Thank you, - Luiji Maryo -- - Luiji Maryo (a.k.a. Brain Boy) Visit me at http://brainboyblogger.blogspot.com/. From stefan_ml at behnel.de Wed Jun 30 08:55:39 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Jun 2010 08:55:39 +0200 Subject: [Cython] Optimised Python runtime using Cython (added subject to match content) In-Reply-To: References: Message-ID: <4C2AEA6B.5080808@behnel.de> Luiji Maryo, 30.06.2010 06:55: > For a project I am working on I am creating a super-optimized Python > distrobution of which I hope will reach heights of 10,000 times faster > code. Good luck. I don't mean to discourage you, but this is a pretty ambitious goal. Could you tell us more about the project background? > What I am planning to do is bundle Psyco with a refactor of the > Standard Python Library. The refactor is to be a modified version > where I use optimization abilities from Cython like CDEFs to increase > code speed, after which I will compile it all into native machine code > modules. > > My inquisitions are: > - Has this already been done? Certainly not. Even with the soon-to-be-released Cython 0.13 (cython-closures branch), most stdlib modules won't compile without modifications. That being said, I'd appreciate seeing pure Python typing additions to a couple of stdlib modules to speed them up. This is something that can often be done with little source code impact, so that you can keep your maintenance overhead low. > - Do people have any suggestions (such as using something other then Psyco). You should also look at the recent developments around CPython, especially Unladen Swallow. Besides a couple of other improvements, they're building a JIT compiled backend that is currently scheduled for integration into Python 3.3. However, I'm sure there are modules in the stdlib that can benefit a lot more from Cython compilation than from the current state of JIT compilation in CPython. Stefan From dagss at student.matnat.uio.no Wed Jun 30 13:30:21 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 30 Jun 2010 13:30:21 +0200 Subject: [Cython] (no subject) In-Reply-To: References: Message-ID: <4C2B2ACD.5050905@student.matnat.uio.no> Luiji Maryo wrote: > Hello, > > For a project I am working on I am creating a super-optimized Python > distrobution of which I hope will reach heights of 10,000 times faster > code. What I am planning to do is bundle Psyco with a refactor of the > I hope you'll adjust that figure. Take for instance matrix muliplication -- that's probably one of the benchmarks you'll find where Python performs worst. But even comparing that to specifically CPU-tuned assembly code using SSE instructions, Python is only about 4000-5000 times slower :-) For more "normal" OOP code, we're talking between 2x and 100x I think. Dag Sverre