From tjhnson at gmail.com Mon Sep 1 01:41:08 2008 From: tjhnson at gmail.com (T J) Date: Sun, 31 Aug 2008 16:41:08 -0700 Subject: [Cython] Lists? Message-ID: Hi, New to cython, and I found the sphinx documentation very helpful. However, I didn't find much about dictionaries and lists. How does cython handle these....for example, if I were to have a list of strings....is there a way to inform cython about this so that it could be compiled? I'm assuming non-homogeneous lists are not candidates for optimizations (correct?). In general, I guess I am looking for more discussion/help about such topics. Any pointers would be appreciated. Thanks. From greg.ewing at canterbury.ac.nz Mon Sep 1 03:23:09 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 01 Sep 2008 13:23:09 +1200 Subject: [Cython] 0.9.8.1.1 and .pxd files In-Reply-To: <48BA6509.6080905@behnel.de> References: <20080827224353.f641a024.simon@arrowtheory.com> <48BA6509.6080905@behnel.de> Message-ID: <48BB43FD.7000907@canterbury.ac.nz> Stefan Behnel wrote: > To clarify a bit, the only difference is that Cython knows both, but Python > only recognises __init__.py. The idea of allowing __init__.pyx was so that the main code of a package could be written in Pyrex. However, I've never actually tested whether Python recognises an __init__.so file as a package main file, so I'm not sure if this works. -- Greg From stefan_ml at behnel.de Mon Sep 1 07:42:55 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 01 Sep 2008 07:42:55 +0200 Subject: [Cython] Lists? In-Reply-To: References: Message-ID: <48BB80DF.4070505@behnel.de> Hi, T J wrote: > New to cython, and I found the sphinx documentation very helpful. > However, I didn't find much about dictionaries and lists. How does > cython handle these.... It uses Python's list/dict type. > for example, if I were to have a list of > strings....is there a way to inform cython about this so that it could > be compiled? The creation of such a list runs in compiled code. But you cannot specify the type of the list content. And I don't see a use case for that. > I'm assuming non-homogeneous lists are not candidates > for optimizations (correct?). There is no difference between a list with homogeneous content and one with mixed content. What would you like to see 'optimised'? > In general, I guess I am looking for > more discussion/help about such topics. Any pointers would be > appreciated. Maybe you could tell us what you want to achieve? That would make it easier to answer your question. Stefan From hoytak at cs.ubc.ca Mon Sep 1 19:41:51 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 1 Sep 2008 20:41:51 +0300 Subject: [Cython] Lists? In-Reply-To: <48BB80DF.4070505@behnel.de> References: <48BB80DF.4070505@behnel.de> Message-ID: <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> Hi, This is actually something I am wondering about as well, so I hope it's okay to join the discussion. My understanding is that when you declare a variable to have a list, tuple, or dict type, then cython optimizes it by using faster direct ways to accessing the elements when you use them. However, since these containers always store the elements as PyObject* pointers which hold any type, it's too difficult to use element type information in a robust way. Is that correct? What is the most "cythonic" way to mimic the list functionality when you do have homogeneous list content? I have run into cases in numerical calculations where I need to be append an arbitrary number of numbers to a list (e.g. accumulating data statistics while processing large amounts of data). In such contexts, using a list is too slow. One thing I've done is use a numpy array with a significant amount of extra space, keep a separate counter to denote the end, and access it through the nice buffer interface. This works pretty well, but seems quite clunky compared to how natural it is in python. Doing slicing and concatenation stuff would also be nice. Personally, what I think would be really nice is a unified wrapper to the C++ STL containers and some of the libraries with easy conversion functions between the parallel python containers. I know that wrapping an STL vector is one of the examples given, so this shouldn't be too hard. I don't have time for at least a few weeks, and there are several other important things on my to-do list, but I'll try to look into it. That said, I am pretty new still to cython, so please correct me if my perspective on things is off or I'm just plain wrong. Thanks, --Hoyt On Mon, Sep 1, 2008 at 8:42 AM, Stefan Behnel wrote: > Hi, > > T J wrote: >> New to cython, and I found the sphinx documentation very helpful. >> However, I didn't find much about dictionaries and lists. How does >> cython handle these.... > > It uses Python's list/dict type. > > >> for example, if I were to have a list of >> strings....is there a way to inform cython about this so that it could >> be compiled? > > The creation of such a list runs in compiled code. But you cannot specify the > type of the list content. And I don't see a use case for that. > > >> I'm assuming non-homogeneous lists are not candidates >> for optimizations (correct?). > > There is no difference between a list with homogeneous content and one with > mixed content. What would you like to see 'optimised'? > > >> In general, I guess I am looking for >> more discussion/help about such topics. Any pointers would be >> appreciated. > > Maybe you could tell us what you want to achieve? That would make it easier to > answer your question. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From robertwb at math.washington.edu Mon Sep 1 20:27:36 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Sep 2008 11:27:36 -0700 Subject: [Cython] Lists? In-Reply-To: <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> References: <48BB80DF.4070505@behnel.de> <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> Message-ID: On Sep 1, 2008, at 10:41 AM, Hoyt Koepke wrote: > Hi, > > This is actually something I am wondering about as well, so I hope > it's okay to join the discussion. My understanding is that when you > declare a variable to have a list, tuple, or dict type, then cython > optimizes it by using faster direct ways to accessing the elements > when you use them. However, since these containers always store the > elements as PyObject* pointers which hold any type, it's too difficult > to use element type information in a robust way. Is that correct? Exactly. > What is the most "cythonic" way to mimic the list functionality when > you do have homogeneous list content? I have run into cases in > numerical calculations where I need to be append an arbitrary number > of numbers to a list (e.g. accumulating data statistics while > processing large amounts of data). In such contexts, using a list is > too slow. One thing I've done is use a numpy array with a significant > amount of extra space, keep a separate counter to denote the end, and > access it through the nice buffer interface. This works pretty well, > but seems quite clunky compared to how natural it is in python. Doing > slicing and concatenation stuff would also be nice. The traditional way to do this is the same as in C, with malloc, realloc, etc. Then one can create actual int*, double*, etc. as in C. Of course then one has to worry about memory leaks and/or segfaults... The Numpy approach is a good one too, and there is a resize() method. > Personally, what I think would be really nice is a unified wrapper to > the C++ STL containers and some of the libraries with easy conversion > functions between the parallel python containers. I know that > wrapping an STL vector is one of the examples given, so this shouldn't > be too hard. I don't have time for at least a few weeks, and there > are several other important things on my to-do list, but I'll try to > look into it. That would be excellent! It would necessitate using C++, which is not going to be the default Cython output but would be very hand for people wanting to use it. > That said, I am pretty new still to cython, so please correct me if my > perspective on things is off or I'm just plain wrong. Your assessment is entirely correct. Another option would be something like http://wiki.cython.org/ enhancements/arraytypes, though this is probably further off in the future than wrapping several C++ containers. - Robert From chris at axique.de Mon Sep 1 20:51:00 2008 From: chris at axique.de (=?ISO-8859-15?Q?Christoph_W=FCrstle?=) Date: Mon, 01 Sep 2008 20:51:00 +0200 Subject: [Cython] Cython and Gtk signals Message-ID: <48BC3994.5050204@axique.de> Hi, I'm new to cython and try to optimize a gtk program. So far I'm really impressed how easy it is. So far I ask me two things: First, does anyone has experience in the speedup of gtk apps? And second, how can I translate my own signals to cython code? In Python I write: class myClass(gtk.VBox): __gsignals__ = { 'mysignal' : (gobject.SIGNAL_RUN_LAST, gobject.TYPE_NONE,(gobject.TYPE_STRING,gobject.TYPE_STRING)), } and Cython does not complain when 'compiling', but I can't connect to it: TypeError: : unknown signal name: mysignal Thanks a lot for your help. Chris From hoytak at cs.ubc.ca Mon Sep 1 20:56:13 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 1 Sep 2008 21:56:13 +0300 Subject: [Cython] Lists? In-Reply-To: References: <48BB80DF.4070505@behnel.de> <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> Message-ID: <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> Hi, > That would be excellent! It would necessitate using C++, which is not > going to be the default Cython output but would be very hand for > people wanting to use it. Okay, encouragement is a good motivator. I'll try to get to it. One of my main questions when I initially thought about it is how to handle the operator methods in c++. When I define __getitem__, I have to define it as a python function using def, so is it compiled directly into c code? To ask it another way, is there any way to create a class A such that I can write a = A(...) a[i] += 1 and have the a[i] += 1 be entirely c(++) code? --Hoyt > Another option would be something like http://wiki.cython.org/ > enhancements/arraytypes, though this is probably further off in the > future than wrapping several C++ containers. Okay, I'll look into it. Thanks. -- Hoyt P.S. Robert -- I see you're at UW. I'm starting a PhD in stats there in a few weeks. Just down the hall from the math department, IIRC. I'll drop by some time and say hi. +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Mon Sep 1 21:13:11 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 01 Sep 2008 21:13:11 +0200 Subject: [Cython] Lists? In-Reply-To: <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> References: <48BB80DF.4070505@behnel.de> <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> Message-ID: <48BC3EC7.6010300@behnel.de> Hi, Hoyt Koepke wrote: > One of my main questions when I initially thought about it is how to > handle the operator methods in c++. When I define __getitem__, I have > to define it as a python function using def, so is it compiled > directly into c code? To ask it another way, is there any way to > create a class A such that I can write > > a = A(...) > a[i] += 1 > > and have the a[i] += 1 be entirely c(++) code? Yes, but within the limits of the Python getitem protocol. This means that the item itself will be a Python object, and thus the "+= 1" will be executed on two Python objects. Stefan From robertwb at math.washington.edu Mon Sep 1 21:18:30 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Sep 2008 12:18:30 -0700 Subject: [Cython] Lists? In-Reply-To: <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> References: <48BB80DF.4070505@behnel.de> <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> Message-ID: On Sep 1, 2008, at 11:56 AM, Hoyt Koepke wrote: > Hi, > >> That would be excellent! It would necessitate using C++, which is not >> going to be the default Cython output but would be very hand for >> people wanting to use it. > > Okay, encouragement is a good motivator. I'll try to get to it. > > One of my main questions when I initially thought about it is how to > handle the operator methods in c++. When I define __getitem__, I have > to define it as a python function using def, so is it compiled > directly into c code? Yes. To clarify, the bodies of def methods are compiled into C code just as with cdef methods, and special methods are special, meaning the get called as C functions via the Python/C API, so they are not really "def" methods but nor are they really "cdef" methods. > To ask it another way, is there any way to > create a class A such that I can write > > a = A(...) > a[i] += 1 > > and have the a[i] += 1 be entirely c(++) code? Almost. One can use the buffer interface for the class A, then use the "operator overloading" trick at the bottom of http:// wiki.cython.org/WrappingCPlusPlus. (Of course this still has a level of indirection). Being able to do this directly without a wrapper would be very nice...and probably not too hard either. >> Another option would be something like http://wiki.cython.org/ >> enhancements/arraytypes, though this is probably further off in the >> future than wrapping several C++ containers. > > Okay, I'll look into it. Thanks. On this note, it would be good to come up with better support for wrapping templated libraries. > > -- Hoyt > > P.S. Robert -- I see you're at UW. I'm starting a PhD in stats there > in a few weeks. Just down the hall from the math department, IIRC. > I'll drop by some time and say hi. Neat. There's a lot of other Cython (Sage) folks around the math department there too. - Robert From stefan_ml at behnel.de Mon Sep 1 21:13:11 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 01 Sep 2008 21:13:11 +0200 Subject: [Cython] Lists? In-Reply-To: <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> References: <48BB80DF.4070505@behnel.de> <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> Message-ID: <48BC3EC7.6010300@behnel.de> Hi, Hoyt Koepke wrote: > One of my main questions when I initially thought about it is how to > handle the operator methods in c++. When I define __getitem__, I have > to define it as a python function using def, so is it compiled > directly into c code? To ask it another way, is there any way to > create a class A such that I can write > > a = A(...) > a[i] += 1 > > and have the a[i] += 1 be entirely c(++) code? Yes, but within the limits of the Python getitem protocol. This means that the item itself will be a Python object, and thus the "+= 1" will be executed on two Python objects. Stefan From robertwb at math.washington.edu Mon Sep 1 21:53:57 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Sep 2008 12:53:57 -0700 Subject: [Cython] Lists? In-Reply-To: <48BC3EC7.6010300@behnel.de> References: <48BB80DF.4070505@behnel.de> <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> <48BC3EC7.6010300@behnel.de> Message-ID: On Sep 1, 2008, at 12:13 PM, Stefan Behnel wrote: > Hi, > > Hoyt Koepke wrote: >> One of my main questions when I initially thought about it is how to >> handle the operator methods in c++. When I define __getitem__, I >> have >> to define it as a python function using def, so is it compiled >> directly into c code? To ask it another way, is there any way to >> create a class A such that I can write >> >> a = A(...) >> a[i] += 1 >> >> and have the a[i] += 1 be entirely c(++) code? > > Yes, but within the limits of the Python getitem protocol. This > means that the > item itself will be a Python object, and thus the "+= 1" will be > executed on > two Python objects. One *can* get around this with the buffer interface however. - Robert From greg.ewing at canterbury.ac.nz Tue Sep 2 03:26:59 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 02 Sep 2008 13:26:59 +1200 Subject: [Cython] Lists? In-Reply-To: References: <48BB80DF.4070505@behnel.de> <4db580fd0809011041o6f2c8f7dq9bc6bd39558bb067@mail.gmail.com> <4db580fd0809011156md64f93bv612c7307275cce66@mail.gmail.com> Message-ID: <48BC9663.8050604@canterbury.ac.nz> Robert Bradshaw wrote: > Yes. To clarify, the bodies of def methods are compiled into C code > just as with cdef methods, and special methods are special, meaning > the get called as C functions via the Python/C API, so they are not > really "def" methods but nor are they really "cdef" methods. In case anyone is wondering, the reason they're defined with "def" is that they're invoked (albeit indirectly) from Python code, and their headers are more like those of Python functions than C functions -- they can have * and ** arguments, default values, etc., at least in some cases. In Pyrex, the machinery for handling all that exists in the parse tree nodes for "def" functions but not "cdef" ones. -- Greg From marcin.cieslik at gmail.com Tue Sep 2 05:14:36 2008 From: marcin.cieslik at gmail.com (=?UTF-8?Q?Marcin_Cie=C5=9Blik?=) Date: Mon, 1 Sep 2008 23:14:36 -0400 Subject: [Cython] signed vs. unsigned range Message-ID: <3f9ed1a70809012014p7489b206o4f068f26c3b7806e@mail.gmail.com> Hello, I just wanted to make shure that this how it is intended to be: My intention was to code this C-loop: for (i=j-1;i>=0;i--) def pycount(n): for j in range(1, n): print j for i in range(j-1, -1, -1): print i def cycount_unsigned(unsigned int n): cdef unsigned int i, j for j in range(1, n): print j for i in range(j-1, -1, -1): print i def cycount_signed(int n): cdef int i, j for j in range(1, n): print j for i in range(j-1, -1, -1): print i In [2]: sign.pycount(3) 1 0 2 1 0 In [3]: sign.cycount_unsigned(3) 1 2 In [4]: sign.cycount_signed(3) 1 0 2 1 0 Yours, Marcin From robertwb at math.washington.edu Tue Sep 2 06:48:43 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 1 Sep 2008 21:48:43 -0700 Subject: [Cython] Cython and Gtk signals In-Reply-To: <48BC3994.5050204@axique.de> References: <48BC3994.5050204@axique.de> Message-ID: On Sep 1, 2008, at 11:51 AM, Christoph W?rstle wrote: > Hi, > I'm new to cython and try to optimize a gtk program. So far I'm really > impressed how easy it is. > So far I ask me two things: > First, does anyone has experience in the speedup of gtk apps? > And second, how can I translate my own signals to cython code? > > In Python I write: > > class myClass(gtk.VBox): > __gsignals__ = { > 'mysignal' : (gobject.SIGNAL_RUN_LAST, > gobject.TYPE_NONE,(gobject.TYPE_STRING,gobject.TYPE_STRING)), > } > > and Cython does not complain when 'compiling', but I can't connect > to it: > > TypeError: 0x8588100)>: unknown signal name: mysignal > > Thanks a lot for your help. > Chris I'm not sure--this code works in Python but not in Cython? - Robert From tjhnson at gmail.com Tue Sep 2 08:05:50 2008 From: tjhnson at gmail.com (T J) Date: Mon, 1 Sep 2008 23:05:50 -0700 Subject: [Cython] Question on Efficient Indexing Message-ID: I was reading: http://wiki.cython.org/tutorials/numpy and I am confused on the difference between: A) cimport numpy as np cdef np.ndarray h = np.zeros([xmax, ymax], dtype=DTYPE) and B) cimport numpy as np cdef np.ndarray[DTYPE_t, ndim=2] h =np.zeros([xmax, ymax], dtype=DTYPE) In particular, I guess I don't understand what is gained by using A. I had thought that the cimport of numpy and the cdef h would be enough to give significant benefits. From the article, I understand what B improves upon over A, but what does A give us in the first place? From simon at arrowtheory.com Tue Sep 2 09:10:37 2008 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 2 Sep 2008 03:10:37 -0400 Subject: [Cython] Cython and Gtk signals In-Reply-To: <48BC3994.5050204@axique.de> References: <48BC3994.5050204@axique.de> Message-ID: <20080902031037.bdd35615.simon@arrowtheory.com> On Mon, 01 Sep 2008 20:51:00 +0200 Christoph W?rstle wrote: > Hi, > I'm new to cython and try to optimize a gtk program. So far I'm really > impressed how easy it is. > So far I ask me two things: > First, does anyone has experience in the speedup of gtk apps? > And second, how can I translate my own signals to cython code? > > In Python I write: > > class myClass(gtk.VBox): > __gsignals__ = { > 'mysignal' : (gobject.SIGNAL_RUN_LAST, > gobject.TYPE_NONE,(gobject.TYPE_STRING,gobject.TYPE_STRING)), > } > > and Cython does not complain when 'compiling', but I can't connect to it: > > TypeError: 0x8588100)>: unknown signal name: mysignal > I'm not familiar with gtk, but this looks like VBox has a special metaclass.. Not sure if cython supports this. You could try digging in deeper and see what __gsignals__ invokes and instead doing it manually.... Simon. From chris at axique.de Tue Sep 2 09:53:15 2008 From: chris at axique.de (=?ISO-8859-1?Q?Christoph_W=FCrstle?=) Date: Tue, 02 Sep 2008 09:53:15 +0200 Subject: [Cython] Cython and Gtk signals In-Reply-To: References: <48BC3994.5050204@axique.de> Message-ID: <48BCF0EB.60108@axique.de> Robert Bradshaw schrieb: > On Sep 1, 2008, at 11:51 AM, Christoph W?rstle wrote: > >> In Python I write: >> >> class myClass(gtk.VBox): >> __gsignals__ = { >> 'mysignal' : (gobject.SIGNAL_RUN_LAST, >> gobject.TYPE_NONE,(gobject.TYPE_STRING,gobject.TYPE_STRING)), >> } >> >> and Cython does not complain when 'compiling', but I can't connect >> to it: >> >> TypeError: > 0x8588100)>: unknown signal name: mysignal >> >> Thanks a lot for your help. >> Chris >> > > I'm not sure--this code works in Python but not in Cython? > > - Robert > Yes, in Python it works without a problem (and this is the method explained at: http://www.sicem.biz/personal/lgs/docs/gobject-python/gobject-tutorial.html#d0e570). But after compiling the modul does not have the signal to connect with. It works, when I add the signal with: gobject.type_register(myClass) gobject.signal_new('mysignal',myClass, gobject.SIGNAL_RUN_LAST, gobject.TYPE_NONE,(gobject.TYPE_STRING,gobject.TYPE_STRING)) Thanks for your answer Robert. -Chris From dagss at student.matnat.uio.no Tue Sep 2 10:11:26 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 02 Sep 2008 10:11:26 +0200 Subject: [Cython] Question on Efficient Indexing In-Reply-To: References: Message-ID: <48BCF52E.5010408@student.matnat.uio.no> T J wrote: > I was reading: http://wiki.cython.org/tutorials/numpy > > and I am confused on the difference between: > > A) > > cimport numpy as np > cdef np.ndarray h = np.zeros([xmax, ymax], dtype=DTYPE) > > and > > B) > > cimport numpy as np > cdef np.ndarray[DTYPE_t, ndim=2] h =np.zeros([xmax, ymax], dtype=DTYPE) > > > In particular, I guess I don't understand what is gained by using A. > I had thought that the cimport of numpy and the cdef h would be enough > to give significant benefits. From the article, I understand what B > improves upon over A, but what does A give us in the first place? > I'll have to admit that this was not very well motivated in the tutorial (feel free to improve it). First off, cimport of numpy and "cdef h" (whatever you mean by that?) will not give you any improvements at all. All the cimport statement really does is make A) and B) available, but it still has to be used. If something is still unclear here then please ask again. The reason for introducing A) first was primarily so that things would be done one step at the time. However A) gives efficient access to the parts of "h" which does not access the subitems, specifically lookups of "h.shape[0]" is much more efficient. (This is only used for f and g in the example (and even then it doesn't really matter as it is a constant overhead). I felt that being inconsistent would be more confusing, but in the specific example nothing is actually gained using A) for h). Another reason for A) is that it makes it possible for the code to get benefits from future improvements to the Cython/NumPy interface. For instance operations like "h += g" or "np.asum(h)" have potential for optimizations if h is typed to ndarray that doesn't depend on knowing the dtype or ndim (but at present such things are only hypothetical). Dag Sverre From hoytak at cs.ubc.ca Tue Sep 2 10:21:03 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Tue, 2 Sep 2008 11:21:03 +0300 Subject: [Cython] Question on Efficient Indexing In-Reply-To: <48BCF52E.5010408@student.matnat.uio.no> References: <48BCF52E.5010408@student.matnat.uio.no> Message-ID: <4db580fd0809020121i52ce4523uc2910ea3c985b3cd@mail.gmail.com> >> I was reading: http://wiki.cython.org/tutorials/numpy >> >> and I am confused on the difference between: >> >> A) >> >> cimport numpy as np >> cdef np.ndarray h = np.zeros([xmax, ymax], dtype=DTYPE) >> >> and >> >> B) >> >> cimport numpy as np >> cdef np.ndarray[DTYPE_t, ndim=2] h =np.zeros([xmax, ymax], dtype=DTYPE) A quick clarification for my own understanding. Does B provide everything that A does? Or are some operations slower? Thanks! --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From robertwb at math.washington.edu Tue Sep 2 10:23:08 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 2 Sep 2008 01:23:08 -0700 Subject: [Cython] Question on Efficient Indexing In-Reply-To: <4db580fd0809020121i52ce4523uc2910ea3c985b3cd@mail.gmail.com> References: <48BCF52E.5010408@student.matnat.uio.no> <4db580fd0809020121i52ce4523uc2910ea3c985b3cd@mail.gmail.com> Message-ID: <42B9B6F7-0D6A-4254-B6C9-32606F4B8EE8@math.washington.edu> On Sep 2, 2008, at 1:21 AM, Hoyt Koepke wrote: >>> I was reading: http://wiki.cython.org/tutorials/numpy >>> >>> and I am confused on the difference between: >>> >>> A) >>> >>> cimport numpy as np >>> cdef np.ndarray h = np.zeros([xmax, ymax], dtype=DTYPE) >>> >>> and >>> >>> B) >>> >>> cimport numpy as np >>> cdef np.ndarray[DTYPE_t, ndim=2] h =np.zeros([xmax, ymax], >>> dtype=DTYPE) > > A quick clarification for my own understanding. Does B provide > everything that A does? Or are some operations slower? B is a superset of A. - Robert From tjhnson at gmail.com Tue Sep 2 11:26:46 2008 From: tjhnson at gmail.com (T J) Date: Tue, 2 Sep 2008 02:26:46 -0700 Subject: [Cython] setup.py and cmdclass Message-ID: [Sorry for all the posts...but hopefully, it gives the devs a sense of the varied user backgrounds] I've figured out how to get my cython extensions to compile via distutils. This site: http://wiki.cython.org/WrappingCPlusPlus says that I can pass the cmdclass keyword to the Extension() object. However, when I do this, I get (for example): error: unknown file type '.pyx' (from 'primes.pyx') Only if I move the cmdclass declaration to the setup() will it compile properly. That is fine, but now I wonder how I can handle extensions if they are not Cython extensions. That is, suppose I have: from Cython.Distutils import build_ext setup(.... ext_modules=[Extension('primes', 'primes.pyx'), Extension('noncython', 'noncython.c')] cmdclass={'build_ext': build_ext} ) Would this work? My first reaction is that I don't want the cython build_ext for a non-cython extension. Or is the Cython build_ext a superset of distutils.command.build_ext? Thanks again. From stefan_ml at behnel.de Tue Sep 2 12:21:46 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 2 Sep 2008 12:21:46 +0200 (CEST) Subject: [Cython] setup.py and cmdclass In-Reply-To: References: Message-ID: <47708.213.61.181.86.1220350906.squirrel@groupware.dvs.informatik.tu-darmstadt.de> T J wrote: > [Sorry for all the posts...but hopefully, it gives the devs a sense of > the varied user backgrounds] > > I've figured out how to get my cython extensions to compile via > distutils. This site: > > http://wiki.cython.org/WrappingCPlusPlus > > says that I can pass the cmdclass keyword to the Extension() object. > However, when I do this, I get (for example): > > error: unknown file type '.pyx' (from 'primes.pyx') > > Only if I move the cmdclass declaration to the setup() will it compile > properly. That is fine, but now I wonder how I can handle extensions > if they are not Cython extensions. That is, suppose I have: > > from Cython.Distutils import build_ext > setup(.... > ext_modules=[Extension('primes', 'primes.pyx'), > Extension('noncython', 'noncython.c')] > cmdclass={'build_ext': build_ext} > ) > > Would this work? My first reaction is that I don't want the cython > build_ext for a non-cython extension. Or is the Cython build_ext a > superset of distutils.command.build_ext? Apparently, you forgot to try it before posting. Reading the source would also have answered your question. Cython's own build_ext only special-cases .pyx files and passes on their .c/.cpp result, so replacing it once and for all shouldn't hurt. Stefan From dalcinl at gmail.com Tue Sep 2 16:54:44 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 2 Sep 2008 11:54:44 -0300 Subject: [Cython] autodoc function/method signatures Message-ID: Would it be possible to tell Cython/Pyrex to add a line in docstings with function/method signatures? Or should this wait until Cython/Pyrex supports PEP 3107 (Function Annotations)? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From tjhnson at gmail.com Tue Sep 2 17:28:43 2008 From: tjhnson at gmail.com (T J) Date: Tue, 2 Sep 2008 08:28:43 -0700 Subject: [Cython] setup.py and cmdclass In-Reply-To: <47708.213.61.181.86.1220350906.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <47708.213.61.181.86.1220350906.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Tue, Sep 2, 2008 at 3:21 AM, Stefan Behnel wrote: > > Apparently, you forgot to try it before posting. Reading the source would > also have answered your question. Yes I should have tried, but incorrect documentation is still a boon to new users. And I will state the obvious and mention that pointing users to the source code is a not a technique that scales particularly well. For sure, it would have been easy in this scenario. > > Cython's own build_ext only special-cases .pyx files and passes on their > .c/.cpp result, so replacing it once and for all shouldn't hurt. Good to know. I feel like my general question still stands---but it is not specific to Cython. Likely, I just need to learn more about distutils. From stefan_ml at behnel.de Tue Sep 2 18:25:17 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 02 Sep 2008 18:25:17 +0200 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: Message-ID: <48BD68ED.6000503@behnel.de> Hi, Lisandro Dalcin wrote: > Would it be possible to tell Cython/Pyrex to add a line in docstings > with function/method signatures? I bet you're asking because of epydoc & friends. I thought about that, too, a while ago, but never got around to actually doing it. Mainly because it wasn't obvious to me at the time how to recover the signature as a string that is parsable by epydoc. A compiler option to enable this would be nice. Stefan From dalcinl at gmail.com Tue Sep 2 21:47:59 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 2 Sep 2008 16:47:59 -0300 Subject: [Cython] autodoc function/method signatures In-Reply-To: <48BD68ED.6000503@behnel.de> References: <48BD68ED.6000503@behnel.de> Message-ID: On Tue, Sep 2, 2008 at 1:25 PM, Stefan Behnel wrote: > Hi, > > Lisandro Dalcin wrote: >> Would it be possible to tell Cython/Pyrex to add a line in docstings >> with function/method signatures? > > I bet you're asking because of epydoc & friends. Not only those!. If you use IPython, or even built-in help(), you cannot get the signature. > I thought about that, too, a > while ago, but never got around to actually doing it. Mainly because it wasn't > obvious to me at the time how to recover the signature as a string that is > parsable by epydoc. A compiler option to enable this would be nice. Yes, really nice, but I cannot help here :-( > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Sep 3 17:10:50 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 03 Sep 2008 17:10:50 +0200 Subject: [Cython] Initializing var to struct? Message-ID: <48BEA8FA.4040903@student.matnat.uio.no> Was this ever supposed to work, if so, what did it do? cdef struct A: int a = 4 I have code to disallow it ready (raise compiler error) but then I thought that perhaps it had a purpose (how else did Hoyt run into it...) http://trac.cython.org/cython_trac/ticket/64 -- Dag Sverre From dalcinl at gmail.com Wed Sep 3 17:38:00 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Sep 2008 12:38:00 -0300 Subject: [Cython] Initializing var to struct? In-Reply-To: <48BEA8FA.4040903@student.matnat.uio.no> References: <48BEA8FA.4040903@student.matnat.uio.no> Message-ID: Well, if that could be supported, perhaps it would be a good feature ;-). On Wed, Sep 3, 2008 at 12:10 PM, Dag Sverre Seljebotn wrote: > Was this ever supposed to work, if so, what did it do? > > cdef struct A: > int a = 4 > > I have code to disallow it ready (raise compiler error) but then I > thought that perhaps it had a purpose (how else did Hoyt run into it...) > > http://trac.cython.org/cython_trac/ticket/64 > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From hoytak at cs.ubc.ca Wed Sep 3 17:45:36 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Wed, 3 Sep 2008 18:45:36 +0300 Subject: [Cython] Initializing var to struct? In-Reply-To: References: <48BEA8FA.4040903@student.matnat.uio.no> Message-ID: <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> Hi, Here was my use case: I wanted to have a single structure that holds a bunch of option parameters for my algorithm. This structure would get passed around between some of the functions, and some of the particular options are in the inner loop of things and so I wanted access to be quick. I wanted to put the defaults in the structure definition and then allow the user to change them if needed after creating the structure. I ended up using an extension class w/ constructor to hold the options, which I think is a better solution, but perhaps there are other use cases. --Hoyt On Wed, Sep 3, 2008 at 6:38 PM, Lisandro Dalcin wrote: > Well, if that could be supported, perhaps it would be a good feature ;-). > > On Wed, Sep 3, 2008 at 12:10 PM, Dag Sverre Seljebotn > wrote: >> Was this ever supposed to work, if so, what did it do? >> >> cdef struct A: >> int a = 4 >> >> I have code to disallow it ready (raise compiler error) but then I >> thought that perhaps it had a purpose (how else did Hoyt run into it...) >> >> http://trac.cython.org/cython_trac/ticket/64 >> >> -- >> Dag Sverre >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From dalcinl at gmail.com Wed Sep 3 17:55:57 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Sep 2008 12:55:57 -0300 Subject: [Cython] setup.py and cmdclass In-Reply-To: References: <47708.213.61.181.86.1220350906.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Dear TJ, > Yes I should have tried, but incorrect documentation is still a boon > to new users. I hope you understands that the documentation you are refering is in fact a wiki page, and that means that you cannot assume that core Cython developers have the time to review wiki docs. I hope you have a bit of free time to update the wiki page and fix the wrong docs ;-). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Sep 3 18:03:15 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 03 Sep 2008 18:03:15 +0200 Subject: [Cython] Initializing var to struct? In-Reply-To: <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> References: <48BEA8FA.4040903@student.matnat.uio.no> <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> Message-ID: <48BEB543.4050602@student.matnat.uio.no> Hoyt Koepke wrote: > Hi, > > Here was my use case: > > I wanted to have a single structure that holds a bunch of option > parameters for my algorithm. This structure would get passed around > between some of the functions, and some of the particular options are > in the inner loop of things and so I wanted access to be quick. I > wanted to put the defaults in the structure definition and then allow > the user to change them if needed after creating the structure. I > ended up using an extension class w/ constructor to hold the options, > which I think is a better solution, but perhaps there are other use > cases. To be clear, you say "as of r1117", does that mean that this used to work prior to r1117? I.e. did the feature use to work and then got broken, or was this never allowed? -- Dag Sverre From hoytak at cs.ubc.ca Wed Sep 3 17:59:21 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Wed, 3 Sep 2008 18:59:21 +0300 Subject: [Cython] Initializing var to struct? In-Reply-To: <48BEB543.4050602@student.matnat.uio.no> References: <48BEA8FA.4040903@student.matnat.uio.no> <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> <48BEB543.4050602@student.matnat.uio.no> Message-ID: <4db580fd0809030859y1ca4c46yc9649fc19166624f@mail.gmail.com> > To be clear, you say "as of r1117", does that mean that this used to > work prior to r1117? > > I.e. did the feature use to work and then got broken, or was this never > allowed? No, it was my first time trying it, so I have no evidence it ever worked. --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From dalcinl at gmail.com Wed Sep 3 18:00:24 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Sep 2008 13:00:24 -0300 Subject: [Cython] Initializing var to struct? In-Reply-To: <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> References: <48BEA8FA.4040903@student.matnat.uio.no> <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> Message-ID: On Wed, Sep 3, 2008 at 12:45 PM, Hoyt Koepke wrote: > I wanted to have a single structure that holds a bunch of option > parameters for my algorithm. This structure would get passed around > between some of the functions, and some of the particular options are > in the inner loop of things and so I wanted access to be quick. I > wanted to put the defaults in the structure definition and then allow > the user to change them if needed after creating the structure. That's a very valid and useful use case. > I ended up using an extension class w/ constructor to hold the options, > which I think is a better solution, Well, I'm not sure if that is a better solution. It seems suboptimal to have to create a new python instance just for the purposes of storing a bunch of paramenters. > > On Wed, Sep 3, 2008 at 6:38 PM, Lisandro Dalcin wrote: >> Well, if that could be supported, perhaps it would be a good feature ;-). >> >> On Wed, Sep 3, 2008 at 12:10 PM, Dag Sverre Seljebotn >> wrote: >>> Was this ever supposed to work, if so, what did it do? >>> >>> cdef struct A: >>> int a = 4 >>> >>> I have code to disallow it ready (raise compiler error) but then I >>> thought that perhaps it had a purpose (how else did Hoyt run into it...) >>> >>> http://trac.cython.org/cython_trac/ticket/64 >>> >>> -- >>> Dag Sverre >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >> >> >> -- >> Lisandro Dalc?n >> --------------- >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > +++++++++++++++++++++++++++++++++++ > Hoyt Koepke > UBC Department of Computer Science > http://www.cs.ubc.ca/~hoytak/ > hoytak at gmail.com > +++++++++++++++++++++++++++++++++++ > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed Sep 3 18:14:47 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Sep 2008 13:14:47 -0300 Subject: [Cython] Cython and Gtk signals In-Reply-To: <20080902031037.bdd35615.simon@arrowtheory.com> References: <48BC3994.5050204@axique.de> <20080902031037.bdd35615.simon@arrowtheory.com> Message-ID: On Tue, Sep 2, 2008 at 4:10 AM, Simon Burton wrote: > I'm not familiar with gtk, but this looks like VBox has a special > metaclass.. Not sure if cython supports this. You could try digging in > deeper and see what __gsignals__ invokes and instead doing it manually.... > I believe Cython cannot support metaclases. Why? Because Cython first creates and empty class, and next add stuff to it; metaclases do not work in such scenario. However, I believe that this could be fixed, and create new classes following what Python does, that is, calling 'type(name, bases, namespace)' -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Sep 3 18:24:03 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 03 Sep 2008 18:24:03 +0200 Subject: [Cython] Initializing var to struct? In-Reply-To: References: <48BEA8FA.4040903@student.matnat.uio.no> <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> Message-ID: <48BEBA23.9020401@student.matnat.uio.no> Lisandro Dalcin wrote: > On Wed, Sep 3, 2008 at 12:45 PM, Hoyt Koepke wrote: >> I wanted to have a single structure that holds a bunch of option >> parameters for my algorithm. This structure would get passed around >> between some of the functions, and some of the particular options are >> in the inner loop of things and so I wanted access to be quick. I >> wanted to put the defaults in the structure definition and then allow >> the user to change them if needed after creating the structure. > > That's a very valid and useful use case. Yes, but there's no lack of those :-) And for me personally stuff like accessing complex numbers in a NumPy array, or inner functions and "yield" definitely has a higher priority. So I'll just mark it as a syntax error; then someone else can come along and provide a better solution later... Hoyt: Thanks for the report! -- Dag Sverre From hoytak at cs.ubc.ca Wed Sep 3 18:23:40 2008 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Wed, 3 Sep 2008 19:23:40 +0300 Subject: [Cython] Initializing var to struct? In-Reply-To: <48BEBA23.9020401@student.matnat.uio.no> References: <48BEA8FA.4040903@student.matnat.uio.no> <4db580fd0809030845q330d1f94vd6bf74be77ba2411@mail.gmail.com> <48BEBA23.9020401@student.matnat.uio.no> Message-ID: <4db580fd0809030923g7e2522c6p35b955e2fa43ee31@mail.gmail.com> > Yes, but there's no lack of those :-) And for me personally stuff like > accessing complex numbers in a NumPy array, or inner functions and > "yield" definitely has a higher priority. So I'll just mark it as a > syntax error; then someone else can come along and provide a better > solution later... +1. I agree. --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak at gmail.com +++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Wed Sep 3 18:34:23 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 03 Sep 2008 18:34:23 +0200 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: <48BD68ED.6000503@behnel.de> Message-ID: <48BEBC8F.8050808@student.matnat.uio.no> Lisandro Dalcin wrote: > On Tue, Sep 2, 2008 at 1:25 PM, Stefan Behnel wrote: >> Hi, >> >> Lisandro Dalcin wrote: >>> Would it be possible to tell Cython/Pyrex to add a line in docstings >>> with function/method signatures? >> I bet you're asking because of epydoc & friends. > > Not only those!. If you use IPython, or even built-in help(), you > cannot get the signature. Well, if anybody wants to help with this one, here's how to do it: (0. Create a feature request ticket :-) ) 1. Create a new file TypedTreeTransforms.py 2. Write a transform that intercepts the functions (have a look at ParseTreeTransforms.py for examples). For a FuncDefNode the docstring is available in "node.doc" (and you can simply change it!) and information about the arguments are available in "node.local_scope.arg_entries". Note that node.doc must probably not be a regular string but an instance of Utils.EncodedString. 3. Write code in the pipeline creation in Main.py (search for pipeline) to insert an instance of your transform if and only if a command line argument (CmdLine.py, options at bottom of Main.py) was passed. (You can insert None without harm, that makes the code cleaner.) It you opt for making it a compiler directive (so that it can be turned on/off for individual functions) then have a look at Options.py and the "options" attribute of all nodes. Finally, when writing a transform, "print node.dump()" is your friend. -- Dag Sverre From dfdeshom at gmail.com Thu Sep 4 14:25:11 2008 From: dfdeshom at gmail.com (didier deshommes) Date: Thu, 4 Sep 2008 08:25:11 -0400 Subject: [Cython] Fwd: forcing uninitialized variables to be declared In-Reply-To: <4db014670809040522i3d5b5584wbc3c8f0b72bd086b@mail.gmail.com> References: <4db014670809040522i3d5b5584wbc3c8f0b72bd086b@mail.gmail.com> Message-ID: <4db014670809040525o6b4bfeeck841899b1df09dea5@mail.gmail.com> Hi guys, I have some C++ code that I'm wrapping which relies on a variable to be declared, but *not* initialized. When the variable is declared, everything is fine, but if the variable is not declared, the library crashes. Cython discards uninitialized variables, which is causing trouble for me. Is there a way to tell Cython to not touch uninitialized variables (or just one variable)? I could always find run a script that patches the generated .cpp file, but that's cumbersome. didier From stefan_ml at behnel.de Thu Sep 4 15:21:49 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 4 Sep 2008 15:21:49 +0200 (CEST) Subject: [Cython] Fwd: forcing uninitialized variables to be declared In-Reply-To: <4db014670809040525o6b4bfeeck841899b1df09dea5@mail.gmail.com> References: <4db014670809040522i3d5b5584wbc3c8f0b72bd086b@mail.gmail.com> <4db014670809040525o6b4bfeeck841899b1df09dea5@mail.gmail.com> Message-ID: <53826.213.61.181.86.1220534509.squirrel@groupware.dvs.informatik.tu-darmstadt.de> didier deshommes wrote: > Hi guys, note that non-male subscribers to this list might find this greating inappropriate. > I have some C++ code that I'm wrapping which relies on a variable to > be declared, but *not* initialized. When the variable is declared, > everything is fine, but if the variable is not declared, the library > crashes. Cython discards uninitialized variables, No, Cython does no such thing. Any variable you declare in your Cython source will appear in the generated C(++) file (although that may not be enough for your library to see it according to C/C++ linking rules). Would you have a code example that shows your problem? Stefan From dfdeshom at gmail.com Thu Sep 4 16:01:27 2008 From: dfdeshom at gmail.com (didier deshommes) Date: Thu, 4 Sep 2008 10:01:27 -0400 Subject: [Cython] Fwd: forcing uninitialized variables to be declared In-Reply-To: <53826.213.61.181.86.1220534509.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <4db014670809040522i3d5b5584wbc3c8f0b72bd086b@mail.gmail.com> <4db014670809040525o6b4bfeeck841899b1df09dea5@mail.gmail.com> <53826.213.61.181.86.1220534509.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4db014670809040701u7bc8c992tf9c72f58a59c7c4f@mail.gmail.com> On Thu, Sep 4, 2008 at 9:21 AM, Stefan Behnel wrote: > didier deshommes wrote: > Would you have a code example that shows your problem? Here's the simplest example I could compile: {{{ cdef class T: cdef int i #cdef c_struct def __cinit__(self): cdef char char_name = '2' cdef int int_name cdef bint bint_name print 1 }}} Looking at the generated cpp file, around the corresponding __cinit__, I see char_name, (mangled as pyx_v_char_name), but no sign of mangled versions of int_name or bint_name. The cpp file also shows which lines are being executed and seems to be skipping lines where variables are declared. didier > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Thu Sep 4 16:39:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 04 Sep 2008 16:39:44 +0200 Subject: [Cython] Fwd: forcing uninitialized variables to be declared In-Reply-To: <4db014670809040701u7bc8c992tf9c72f58a59c7c4f@mail.gmail.com> References: <4db014670809040522i3d5b5584wbc3c8f0b72bd086b@mail.gmail.com> <4db014670809040525o6b4bfeeck841899b1df09dea5@mail.gmail.com> <53826.213.61.181.86.1220534509.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db014670809040701u7bc8c992tf9c72f58a59c7c4f@mail.gmail.com> Message-ID: <48BFF330.2060206@student.matnat.uio.no> didier deshommes wrote: > On Thu, Sep 4, 2008 at 9:21 AM, Stefan Behnel wrote: >> didier deshommes wrote: >> Would you have a code example that shows your problem? > > Here's the simplest example I could compile: > {{{ > cdef class T: > cdef int i > #cdef c_struct > def __cinit__(self): > cdef char char_name = '2' > cdef int int_name > cdef bint bint_name > print 1 > > }}} > > Looking at the generated cpp file, around the corresponding __cinit__, > I see char_name, (mangled as pyx_v_char_name), but no sign of mangled > versions of int_name or bint_name. The cpp file also shows which lines > are being executed and seems to be skipping lines where variables are > declared. Right.. this is resource acquisition on initialization? Unfortunately the C++ support of Cython is probably too weak for that (I'm assuming that using a global cdef variable won't really work either for your usecase, otherwise just do that). What you can do is create a header file like this: #define setup_func_context() MyInitializerClass __var i.e. turn it into a call using a macro, and then call that function from Cython: cdef extern from "myheader.h": void setup_func_context() cdef class T: def __cinit__(self): setup_func_context() .... do stuff ... Of course it is rather obscure as that function call makes something happen when you exit the function as well, but it should work. You can always make modifications (pass in the variable name to use, create accessor macros to get hold of the variable etc.) to make it work for you. -- Dag Sverre From dfdeshom at gmail.com Thu Sep 4 18:35:41 2008 From: dfdeshom at gmail.com (didier deshommes) Date: Thu, 4 Sep 2008 12:35:41 -0400 Subject: [Cython] Fwd: forcing uninitialized variables to be declared In-Reply-To: <48BFF330.2060206@student.matnat.uio.no> References: <4db014670809040522i3d5b5584wbc3c8f0b72bd086b@mail.gmail.com> <4db014670809040525o6b4bfeeck841899b1df09dea5@mail.gmail.com> <53826.213.61.181.86.1220534509.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db014670809040701u7bc8c992tf9c72f58a59c7c4f@mail.gmail.com> <48BFF330.2060206@student.matnat.uio.no> Message-ID: <4db014670809040935t4b67131y6b3bdb2414ce6355@mail.gmail.com> On Thu, Sep 4, 2008 at 10:39 AM, Dag Sverre Seljebotn wrote: > What you can do is create a header file like this: > > #define setup_func_context() MyInitializerClass __var > > i.e. turn it into a call using a macro, and then call that function from > Cython: > > cdef extern from "myheader.h": > void setup_func_context() > > cdef class T: > def __cinit__(self): > setup_func_context() > .... do stuff ... Thanks! Works perfectly. didier > > Of course it is rather obscure as that function call makes something > happen when you exit the function as well, but it should work. You can > always make modifications (pass in the variable name to use, create > accessor macros to get hold of the variable etc.) to make it work for you. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From ivanov.maxim at gmail.com Thu Sep 4 22:13:01 2008 From: ivanov.maxim at gmail.com (Max Ivanov) Date: Fri, 5 Sep 2008 00:13:01 +0400 Subject: [Cython] dict set value Message-ID: Hi all! I'm new to Cython, but I'm really impressed by this product, thx to all developers it look really great. My first newbie question following. Do I do something wrong here: --------- cdef dict a a = {} a['a'] = 1 ---------- it still generates inefficient code via PyObject, not via PyDict_SetItem: PyObject_SetItem(((PyObject *)__pyx_v_1a_a), __pyx_kp_1, __pyx_int_1) From robertwb at math.washington.edu Thu Sep 4 23:39:21 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 4 Sep 2008 14:39:21 -0700 (PDT) Subject: [Cython] dict set value In-Reply-To: References: Message-ID: On Fri, 5 Sep 2008, Max Ivanov wrote: > Hi all! > > I'm new to Cython, but I'm really impressed by this product, thx to > all developers it look really great. Thanks. > My first newbie question following. > > Do I do something wrong here: > --------- > cdef dict a > a = {} > a['a'] = 1 > ---------- > > it still generates inefficient code via PyObject, not via PyDict_SetItem: > > PyObject_SetItem(((PyObject *)__pyx_v_1a_a), __pyx_kp_1, __pyx_int_1) No, this simply hasn't been implemented yet. In this specific case there isn't going to be a huge speedup (though there will be some). - Robert From greg.ewing at canterbury.ac.nz Fri Sep 5 06:07:55 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 05 Sep 2008 16:07:55 +1200 Subject: [Cython] dict set value In-Reply-To: References: Message-ID: <48C0B09B.2050308@canterbury.ac.nz> Robert Bradshaw wrote: >> it still generates inefficient code via PyObject, not via PyDict_SetItem: >> >> PyObject_SetItem(((PyObject *)__pyx_v_1a_a), __pyx_kp_1, __pyx_int_1) Keep in mind that the difference in efficiency is very small, since there is a direct path from PyObject_SetItem via the object's mp_setitem slot to the relevant C function. There isn't even any type test involved. The type-dependent optimisations which are implemented are more significant, since they avoid type tests, method lookups and/or conversion between C and Python data. -- Greg From stefan_ml at behnel.de Fri Sep 5 10:27:27 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 5 Sep 2008 10:27:27 +0200 (CEST) Subject: [Cython] autodoc function/method signatures In-Reply-To: References: Message-ID: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > Would it be possible to tell Cython/Pyrex to add a line in docstings > with function/method signatures? Or should this wait until > Cython/Pyrex supports PEP 3107 (Function Annotations)? Just for the archives, this is actually already a trac ticket. http://trac.cython.org/cython_trac/ticket/2 Stefan From ondrej at certik.cz Fri Sep 5 21:41:32 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 5 Sep 2008 21:41:32 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning Message-ID: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> Hi, please apply the attached patch that fixes one g++ warning, or if it doesn't apply cleanly due to whitespace, pull it from here: http://freehg.org/u/certik/cython-devel/ Thanks, Ondrej # HG changeset patch # User Ondrej Certik # Date 1220643365 -7200 # Node ID ac7feebf9be4947f0daec201d63b2d5ce3c3b43b # Parent 22c6ce692a3a8837df4f1202293d1447a1f9c34b Make __Pyx_ArgTypeTest() use "const char*" to suppress warning. diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py --- a/Cython/Compiler/Nodes.py +++ b/Cython/Compiler/Nodes.py @@ -4364,9 +4364,9 @@ static void __Pyx_ReRaise(void) { arg_type_test_utility_code = [ """ -static int __Pyx_ArgTypeTest(PyObject *obj, PyTypeObject *type, int none_allowed, char *name, int exact); /*proto*/ -""",""" -static int __Pyx_ArgTypeTest(PyObject *obj, PyTypeObject *type, int none_allowed, char *name, int exact) { +static int __Pyx_ArgTypeTest(PyObject *obj, PyTypeObject *type, int none_allowed, const char *name, int exact); /*proto*/ +""",""" +static int __Pyx_ArgTypeTest(PyObject *obj, PyTypeObject *type, int none_allowed, const char *name, int exact) { if (!type) { PyErr_Format(PyExc_SystemError, "Missing type object"); return 0; From stefan_ml at behnel.de Sat Sep 6 21:59:45 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Sep 2008 21:59:45 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> Message-ID: <48C2E131.4030600@behnel.de> Hi, Ondrej Certik wrote: > please apply the attached patch that fixes one g++ warning Done, thanks! Stefan From ondrej at certik.cz Sun Sep 7 14:53:54 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 7 Sep 2008 14:53:54 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <48C2E131.4030600@behnel.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> Message-ID: <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> On Sat, Sep 6, 2008 at 9:59 PM, Stefan Behnel wrote: > Hi, > > Ondrej Certik wrote: >> please apply the attached patch that fixes one g++ warning > > Done, thanks! Thanks, please pull from here another const fix. http://freehg.org/u/certik/cython-devel/ Ondrej From ondrej at certik.cz Sun Sep 7 14:59:08 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 7 Sep 2008 14:59:08 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> Message-ID: <85b5c3130809070559i4618af7dpcbf8579034d43b5c@mail.gmail.com> On Sun, Sep 7, 2008 at 2:53 PM, Ondrej Certik wrote: > On Sat, Sep 6, 2008 at 9:59 PM, Stefan Behnel wrote: >> Hi, >> >> Ondrej Certik wrote: >>> please apply the attached patch that fixes one g++ warning >> >> Done, thanks! > > Thanks, please pull from here another const fix. > > http://freehg.org/u/certik/cython-devel/ I just pushed another patch in there (there are now total 2 new patches). Now my C++ project works with Cython without a single warning. Ondrej From stefan_ml at behnel.de Sun Sep 7 15:28:21 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 07 Sep 2008 15:28:21 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> Message-ID: <48C3D6F5.9020005@behnel.de> Hi, Ondrej Certik wrote: > please pull from here another const fix. > > http://freehg.org/u/certik/cython-devel/ Sorry, the const change in __Pyx_ImportType() breaks building under Py2.3, so it can't go in. Stefan From ondrej at certik.cz Sun Sep 7 15:58:45 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 7 Sep 2008 15:58:45 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <48C3D6F5.9020005@behnel.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> Message-ID: <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> On Sun, Sep 7, 2008 at 3:28 PM, Stefan Behnel wrote: > Hi, > > Ondrej Certik wrote: >> please pull from here another const fix. >> >> http://freehg.org/u/certik/cython-devel/ > > Sorry, the const change in __Pyx_ImportType() breaks building under Py2.3, so > it can't go in. I see -- if you have the error log around, could you please post it, so that I can see if it can be fixed? Ondrej From stefan_ml at behnel.de Sun Sep 7 16:16:38 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 07 Sep 2008 16:16:38 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> Message-ID: <48C3E246.5060201@behnel.de> Ondrej Certik wrote: > On Sun, Sep 7, 2008 at 3:28 PM, Stefan Behnel wrote: >> Ondrej Certik wrote: >>> please pull from here another const fix. >>> >>> http://freehg.org/u/certik/cython-devel/ >> Sorry, the const change in __Pyx_ImportType() breaks building under Py2.3, so >> it can't go in. > > I see -- if you have the error log around, could you please post it, > so that I can see if it can be fixed? No, lost it already, so I don't know which of the two consts was the problem. But you can run runtests.py under Py2.3 to see for yourself. Stefan From dalcinl at gmail.com Mon Sep 8 16:41:43 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 8 Sep 2008 11:41:43 -0300 Subject: [Cython] autodoc function/method signatures In-Reply-To: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I have some working code following Dag advices (BTW, that transform stuff its a real win!). Now I would like to know your opinions about how the signatures should be rendered. Robert, if you see this, please enter the discussion. On Fri, Sep 5, 2008 at 5:27 AM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Would it be possible to tell Cython/Pyrex to add a line in docstings >> with function/method signatures? Or should this wait until >> Cython/Pyrex supports PEP 3107 (Function Annotations)? > > Just for the archives, this is actually already a trac ticket. > > http://trac.cython.org/cython_trac/ticket/2 > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Sep 8 17:12:22 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 8 Sep 2008 12:12:22 -0300 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <48C3E246.5060201@behnel.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> Message-ID: In Python 2.3 and 2.4 we have this: PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, char *); In Python 2.5 and above, we have this: PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, const char *); So pehaps we need the a define of inline function __Pyx_PyObject_GetAttr() that do the special-case according to the Python version. On Sun, Sep 7, 2008 at 11:16 AM, Stefan Behnel wrote: > > Ondrej Certik wrote: >> On Sun, Sep 7, 2008 at 3:28 PM, Stefan Behnel wrote: >>> Ondrej Certik wrote: >>>> please pull from here another const fix. >>>> >>>> http://freehg.org/u/certik/cython-devel/ >>> Sorry, the const change in __Pyx_ImportType() breaks building under Py2.3, so >>> it can't go in. >> >> I see -- if you have the error log around, could you please post it, >> so that I can see if it can be fixed? > > No, lost it already, so I don't know which of the two consts was the problem. > But you can run runtests.py under Py2.3 to see for yourself. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Mon Sep 8 17:17:57 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 8 Sep 2008 17:17:57 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> Message-ID: <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> On Mon, Sep 8, 2008 at 5:12 PM, Lisandro Dalcin wrote: > In Python 2.3 and 2.4 we have this: > > PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, char *); > > In Python 2.5 and above, we have this: > > PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, const char *); > > > So pehaps we need the a define of inline function > __Pyx_PyObject_GetAttr() that do the special-case according to the > Python version. Either way is fine. Clearly the current state, that it prints the warnings is a bug, so it should be fixed (either way). Ondrej From dagss at student.matnat.uio.no Mon Sep 8 17:20:36 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 08 Sep 2008 17:20:36 +0200 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48C542C4.3020601@student.matnat.uio.no> Lisandro Dalcin wrote: > I have some working code following Dag advices (BTW, that transform > stuff its a real win!). Now I would like to know your opinions about > how the signatures should be rendered. Robert, if you see this, please > enter the discussion. > > Great! Here's my opinions: For coordination purposes I have now assigned this to you: http://trac.cython.org/cython_trac/ticket/2 (If you haven't got a login there just send a .htpasswd file to Robert) As for signatures, I think that whatever you do, it should be parseable by epydoc if possible. See the links Stefan put up under the ticket. This pretty much defines the format for non-typed code I think (i.e. if the type is "object" then don't specify the type, and Python-only syntax will look like Python..) As for typed function signatures, I think just make it look like the Cython syntax, and whenever we actually support the "f(x: int)"-notation in Cython we can put in an option to support it in docstrings as well *shrug*. But an alternative is dropping the types I suppose (or making that part configurable; or even insert multiple lines with the first Python-only and the second explaining the types in prose). (If you look at the top of runtests.py, you can plug in epydoc as an optional testing library, and make a regression test named "tests/run/epydoc_docstrings.pyx" which only runs if epydoc is installed on the system and call parse_function_signature in the testcase directly). Dag Sverre From robertwb at math.washington.edu Mon Sep 8 18:13:02 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 8 Sep 2008 09:13:02 -0700 Subject: [Cython] [Pyrex] Using pyrex to wrap C++ classes with lots of operators: lots of work? In-Reply-To: <200809081244.39378.paul.leopardi@iinet.net.au> References: <200809081244.39378.paul.leopardi@iinet.net.au> Message-ID: On Sep 7, 2008, at 7:44 PM, Paul C. Leopardi wrote: > Hi, > I'll be attending Sage Days 10 in Nancy next month and was > wondering how much > work it would take to wrap my GluCat library for Sage. > http://wiki.sagemath.org/days10 > > GluCat ( http://glucat.sf.net ) is a C++ template library with > classes which > contain many unary and binary operators. Here is just a taste: > > #define _GLUCAT_CLIFFORD_ALGEBRA_OPERATIONS \ > bool operator== (const multivector_t& val) > const; \ > bool operator== (const Scalar_T& scr) > const; \ > multivector_t& operator+= (const multivector_t& > rhs); \ > multivector_t& operator+= (const Scalar_T& > scr); \ > multivector_t& operator-= (const multivector_t& > rhs); \ > const multivector_t operator- () > const; \ > multivector_t& operator*= (const Scalar_T& > scr); \ > multivector_t& operator*= (const multivector_t& > rhs); \ > multivector_t& operator%= (const multivector_t& > rhs); \ > multivector_t& operator&= (const multivector_t& > rhs); \ > multivector_t& operator^= (const multivector_t& > rhs); \ > multivector_t& operator/= (const Scalar_T& > scr); \ > multivector_t& operator/= (const multivector_t& > rhs); \ > const multivector_t inv () > const; \ > const multivector_t pow (int m) > const; \ > const multivector_t outer_pow (int m) > const; \ > Scalar_T operator[] (const index_set_t ist) > const; \ > Scalar_T scalar() > const; \ > const multivector_t operator() (index_t grade) > const; \ > const multivector_t even() > const; \ > const multivector_t odd() > const; \ > const vector_t vector_part() > const; \ > const multivector_t involute() > const; \ > const multivector_t reverse() > const; \ > const multivector_t conj() > const; \ > Scalar_T norm() > const; \ > Scalar_T quad() > const; \ > const index_set_t frame() > const; \ > Scalar_T max_abs() > const; \ > const multivector_t > truncated \ > (const Scalar_T& limit = Scalar_T(DEFAULT_TRUNCATION)) > const;\ > bool isnan () > const; \ > void write (const std::string& msg="") > const;\ > void write (std::ofstream& ofile, const > std::string& > msg="") const; > > Has anyone yet tried to wrap C++ classes with lots of operators? > How much > effort is involved? It's not too much work, basically one has to list each operator as a method. It would be nice to have notation that allows one to specify that certain types support the infix format. > Would *anyone* be able to complete the warpping of such > classes in the context of a 3 day coding sprint? I think this could *easily* be done. - Robert From robertwb at math.washington.edu Mon Sep 8 18:18:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 8 Sep 2008 09:18:42 -0700 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> On Sep 8, 2008, at 7:41 AM, Lisandro Dalcin wrote: > I have some working code following Dag advices (BTW, that transform > stuff its a real win!). Yes, it certainly is! One thing that worries me is that adding a bunch of tree traversals starts to slow things down--at the very least we should have a transform that doesn't descend into function bodies for speed reasons for stuff like this. > Now I would like to know your opinions about > how the signatures should be rendered. Robert, if you see this, please > enter the discussion. I'd render them the natural way, e.g. [return-type] name([type] arg, [type] arg, ...) - Robert > > > On Fri, Sep 5, 2008 at 5:27 AM, Stefan Behnel > wrote: >> Lisandro Dalcin wrote: >>> Would it be possible to tell Cython/Pyrex to add a line in docstings >>> with function/method signatures? Or should this wait until >>> Cython/Pyrex supports PEP 3107 (Function Annotations)? >> >> Just for the archives, this is actually already a trac ticket. >> >> http://trac.cython.org/cython_trac/ticket/2 >> >> Stefan >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Mon Sep 8 18:39:14 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 8 Sep 2008 13:39:14 -0300 Subject: [Cython] autodoc function/method signatures In-Reply-To: <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> Message-ID: On Mon, Sep 8, 2008 at 1:18 PM, Robert Bradshaw wrote: > On Sep 8, 2008, at 7:41 AM, Lisandro Dalcin wrote: > > Yes, it certainly is! One thing that worries me is that adding a > bunch of tree traversals starts to slow things down--at the very > least we should have a transform that doesn't descend into function > bodies for speed reasons for stuff like this. Well, I believe my current code do not descend into functions body, but perhaps I'm wrong. >> Now I would like to know your opinions about >> how the signatures should be rendered. Robert, if you see this, please >> enter the discussion. > > I'd render them the natural way, e.g. [return-type] name([type] arg, > [type] arg, ...) > That's the natural way from a C function. Docstrings do not apply for 'cdef' functions, but for 'cdef' of 'cpdef'. Additionally, for 'def' functions, I have no idea how to provide a good 'return-type'. We should handle default aguments; I'm currently using 'arg.default.compile_time_value(None)' (I'm passing 'None' to the call because not sure at this point what to pass). If this fails, I just use ''. Other stuff I'm not sure how to render is __get__/__set__/__del__ on properties. Should I generate docstrings for them? >> >> -- >> Lisandro Dalc?n >> --------------- >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Sep 8 18:46:01 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 08 Sep 2008 18:46:01 +0200 Subject: [Cython] transform efficiency In-Reply-To: <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> Message-ID: <48C556C9.7050608@student.matnat.uio.no> Robert Bradshaw wrote: > On Sep 8, 2008, at 7:41 AM, Lisandro Dalcin wrote: > > >> I have some working code following Dag advices (BTW, that transform >> stuff its a real win!). >> > > Yes, it certainly is! One thing that worries me is that adding a > bunch of tree traversals starts to slow things down--at the very > least we should have a transform that doesn't descend into function > bodies for speed reasons for stuff like this. > Lisandro simply shouldn't call self.visitchildren in visit_FuncDefNode, no new transform type is needed. For the record (re: transforms): 1) If they enable us to more quickly develop to a point where Cython can self-compile, then that likely win backs what we temporarily loose (more tree-traversals is only a constant-time overhead after all, and quick function dispatches during traversals will likely win back what we loose?). (A compiler slowdown for about a year is perhaps not "temporary" enough though...) 2) The transform framework could be extended to be able to merge transforms. For instance: pipeline = [PostParseTransform(...), ..., MergeNonConflicting(A, B, C, D), ...] where you know that A, B, C and D are not interfering with one another and can be executed "simultaneously" in one sense (the restriction here is that a B for a given node only depends on the previous siblings and ancestors with respect to what is done by A). So MergeNonConflicting simply takes the result of visit_FuncDefNode of A and feeds it directly to visit_FuncDefNode of B. Many nodes are leaves and the tree is shallow so perhaps it doesn't win that much; but the visitor dispatch only needs to be used once (i.e. the types of the nodes are looked at less times) so perhaps there is some potential here *shrug*. For my own uses, gcc still uses *a lot* more running-time than Cython, so I don't have an itch to scratch here -- how are things in SAGE? Dag Sverre From robertwb at math.washington.edu Mon Sep 8 18:46:48 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 8 Sep 2008 09:46:48 -0700 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> Message-ID: On Sep 8, 2008, at 9:39 AM, Lisandro Dalcin wrote: > On Mon, Sep 8, 2008 at 1:18 PM, Robert Bradshaw > wrote: >> On Sep 8, 2008, at 7:41 AM, Lisandro Dalcin wrote: >> >> Yes, it certainly is! One thing that worries me is that adding a >> bunch of tree traversals starts to slow things down--at the very >> least we should have a transform that doesn't descend into function >> bodies for speed reasons for stuff like this. > > Well, I believe my current code do not descend into functions body, > but perhaps I'm wrong. Good. It's just that the default does descend all the way down. >>> Now I would like to know your opinions about >>> how the signatures should be rendered. Robert, if you see this, >>> please >>> enter the discussion. >> >> I'd render them the natural way, e.g. [return-type] name([type] arg, >> [type] arg, ...) >> > > That's the natural way from a C function. Docstrings do not apply for > 'cdef' functions, but for 'cdef' of 'cpdef'. > > Additionally, for 'def' functions, I have no idea how to provide a > good 'return-type'. Not sure I'm understanding your question... for def functions the return type is always object, which is implied (i.e. one doesn't have to explicitly write "object"). > We should handle default aguments; I'm currently using > 'arg.default.compile_time_value(None)' (I'm passing 'None' to the call > because not sure at this point what to pass). If this fails, I just > use ''. That sounds like a good first pass to me. I think a context object goes here, right? > Other stuff I'm not sure how to render is __get__/__set__/__del__ on > properties. Should I generate docstrings for them? Maybe, though those are lower priorities. - Robert From robertwb at math.washington.edu Mon Sep 8 18:53:06 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 8 Sep 2008 09:53:06 -0700 Subject: [Cython] transform efficiency In-Reply-To: <48C556C9.7050608@student.matnat.uio.no> References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> <48C556C9.7050608@student.matnat.uio.no> Message-ID: <216EE0BC-60E0-467F-9071-F17DD48371B0@math.washington.edu> On Sep 8, 2008, at 9:46 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Sep 8, 2008, at 7:41 AM, Lisandro Dalcin wrote: >> >> >>> I have some working code following Dag advices (BTW, that transform >>> stuff its a real win!). >>> >> >> Yes, it certainly is! One thing that worries me is that adding a >> bunch of tree traversals starts to slow things down--at the very >> least we should have a transform that doesn't descend into function >> bodies for speed reasons for stuff like this. >> > Lisandro simply shouldn't call self.visitchildren in > visit_FuncDefNode, > no new transform type is needed. > > For the record (re: transforms): > 1) If they enable us to more quickly develop to a point where > Cython can > self-compile, then that likely win backs what we temporarily loose > (more > tree-traversals is only a constant-time overhead after all, and quick > function dispatches during traversals will likely win back what we > loose?). (A compiler slowdown for about a year is perhaps not > "temporary" enough though...) > > 2) The transform framework could be extended to be able to merge > transforms. For instance: > > pipeline = [PostParseTransform(...), ..., MergeNonConflicting(A, B, C, > D), ...] > > where you know that A, B, C and D are not interfering with one another > and can be executed "simultaneously" in one sense (the restriction > here > is that a B for a given node only depends on the previous siblings and > ancestors with respect to what is done by A). So MergeNonConflicting > simply takes the result of visit_FuncDefNode of A and feeds it > directly > to visit_FuncDefNode of B. Yep, this might be a savings if we have lots of little stuff. > Many nodes are leaves and the tree is shallow so perhaps it doesn't > win > that much; but the visitor dispatch only needs to be used once > (i.e. the > types of the nodes are looked at less times) so perhaps there is some > potential here *shrug*. > > For my own uses, gcc still uses *a lot* more running-time than Cython, > so I don't have an itch to scratch here -- how are things in SAGE? The opposite, Cython takes longer than gcc. Some of this might be due to lots of re-parsing of .pxd files, which I hope to be able to resolve in a better way (someday...). On profiling visitChildren is up there at the top. I am bringing this up as food for though on how to make things faster, but for the record I think that transforms are awesome and well worth the additional (so far minor) cost in speed. - Robert From dalcinl at gmail.com Mon Sep 8 19:02:39 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 8 Sep 2008 14:02:39 -0300 Subject: [Cython] transform efficiency In-Reply-To: <48C556C9.7050608@student.matnat.uio.no> References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> <48C556C9.7050608@student.matnat.uio.no> Message-ID: On Mon, Sep 8, 2008 at 1:46 PM, Dag Sverre Seljebotn wrote: >> > Lisandro simply shouldn't call self.visitchildren in visit_FuncDefNode, > no new transform type is needed. > OK, I'm not visiting children. However, what should I do if I want to 'push/pop' the class name where a method is defined? > > For my own uses, gcc still uses *a lot* more running-time than Cython, > so I don't have an itch to scratch here -- how are things in SAGE? That's the situation for me, too. As a side note, I never experienced any speed-up from using a compiled 'Scanners.so' in my projects. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Sep 8 19:11:03 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 8 Sep 2008 14:11:03 -0300 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> Message-ID: On Mon, Sep 8, 2008 at 1:46 PM, Robert Bradshaw wrote: >> Additionally, for 'def' functions, I have no idea how to provide a >> good 'return-type'. > > Not sure I'm understanding your question... for def functions the > return type is always object, which is implied (i.e. one doesn't have > to explicitly write "object"). Well, I believe saying a function returns 'object' is equivalent to say nothing :-). Perhaps this should wait until functions annotations are supported. >> We should handle default aguments; I'm currently using >> 'arg.default.compile_time_value(None)' (I'm passing 'None' to the call >> because not sure at this point what to pass). If this fails, I just >> use ''. > > That sounds like a good first pass to me. I think a context object > goes here, right? Yes, but I'm not sure where to get it. Dag, please help me!! > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Mon Sep 8 22:17:59 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 08 Sep 2008 22:17:59 +0200 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> Message-ID: <48C58877.6010209@behnel.de> Hi, Lisandro Dalcin wrote: > On Mon, Sep 8, 2008 at 1:18 PM, Robert Bradshaw >> I'd render them the natural way, e.g. [return-type] name([type] arg, >> [type] arg, ...) > > That's the natural way from a C function. Docstrings do not apply for > 'cdef' functions, but for 'cdef' of 'cpdef'. > > Additionally, for 'def' functions, I have no idea how to provide a > good 'return-type'. Also note that e.g. epydoc is not particularly smart about what it understands as a signature. One might consider that a bug in epydoc, but as long as we can generate a simple-enough signature that it can parse and that users can read, I don't think there's much gained by additionally providing type information, for example. > Other stuff I'm not sure how to render is __get__/__set__/__del__ on > properties. Should I generate docstrings for them? There's only one visible docstring for them, which is the docstring of the property itself (i.e. of the descriptor). So augmenting the docstring doesn't make much sense. Stefan From stefan_ml at behnel.de Mon Sep 8 22:27:53 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 08 Sep 2008 22:27:53 +0200 Subject: [Cython] transform efficiency In-Reply-To: References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> <48C556C9.7050608@student.matnat.uio.no> Message-ID: <48C58AC9.1020308@behnel.de> Hi, Lisandro Dalcin wrote: > On Mon, Sep 8, 2008 at 1:46 PM, Dag Sverre Seljebotn >> For my own uses, gcc still uses *a lot* more running-time than Cython, Same for lxml. Cython is still several times faster than gcc, at least with a high optimisation level. It has become slower during the summer, though, so I think the transformations take their CPU cycles. But they also enabled a lot of great features and optimisations that really pay for the additional run-time. > That's the situation for me, too. As a side note, I never experienced > any speed-up from using a compiled 'Scanners.so' in my projects. The gain is not large, but it does make a difference if you have a lot of Cython code to parse. Stefan From dalcinl at gmail.com Mon Sep 8 23:27:05 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 8 Sep 2008 18:27:05 -0300 Subject: [Cython] autodoc function/method signatures In-Reply-To: <48C58877.6010209@behnel.de> References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> <48C58877.6010209@behnel.de> Message-ID: On Mon, Sep 8, 2008 at 5:17 PM, Stefan Behnel wrote: > Also note that e.g. epydoc is not particularly smart about what it understands > as a signature. One might consider that a bug in epydoc, but as long as we can > generate a simple-enough signature that it can parse and that users can read, > I don't think there's much gained by additionally providing type information, > for example. > But what about IPython? Type information is valuable when typing, for example In [1]: from petsc4py import PETSc In [2]: PETSc.Mat.mult? Type: method_descriptor Base Class: String Form: Namespace: Interactive Docstring: mult(self, Vec x, Vec y) > >> Other stuff I'm not sure how to render is __get__/__set__/__del__ on >> properties. Should I generate docstrings for them? > > There's only one visible docstring for them, which is the docstring of the > property itself (i.e. of the descriptor). So augmenting the docstring doesn't > make much sense. OK, so getting this (at least inside IPython) is just fine, right? In [3]: PETSc.Vec.size Out[3]: In [4]: PETSc.Mat.size Out[4]: > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Sep 9 00:01:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 09 Sep 2008 00:01:00 +0200 Subject: [Cython] transform efficiency Message-ID: <3303763279.1145048@smtp.netcom.no> Speaking of that, visitchildren (which I unfortunately named contrary to conventions) is actually a bit (or possibly very) inefficient at the moment. One could try to clone and modify it into a fast an a simple version, where the simple drops the attribute exclude filters. The advanced functionality is used in so few places that one can just change those. As I don't have profiling etc. set up I'll leave it as an idea for now...an extra list lookup per node per transform...don't know what of a difference it would make... (as for gcc vs. Cython running time it appears to be a simple function of how much .h vs .pxd you are using. Makes sense.) Dag Sverre Seljebotn -----Original Message----- From: Robert Bradshaw Date: Monday, Sep 8, 2008 6:53 pm Subject: Re: [Cython] transform efficiency To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Sep 8, 2008, at 9:46 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >> On Sep 8, 2008, at 7:41 AM, Lisandro Dalcin wrote: >> >> >>> I have some working code following Dag advices (BTW, that transform >>> stuff its a real win!). >>> >> >> Yes, it certainly is! One thing that worries me is that adding a >> bunch of tree traversals starts to slow things down--at the very >> least we should have a transform that doesn't descend into function >> bodies for speed reasons for stuff like this. >> > Lisandro simply shouldn't call self.visitchildren in > visit_FuncDefNode, > no new transform type is needed. > >> For the record (re: transforms): > 1) If they enable us to more quickly develop to a point where > Cython can > self-compile, then that likely win backs what we temporarily loose > (more > tree-traversals is only a constant-time overhead after all, and quick > function dispatches during traversals will likely win back what we > loose?). (A compiler slowdown for about a year is perhaps not > "temporary" enough though...) > >> 2) The transform framework could be extended to be able to merge > transforms. For instance: > >> pipeline = [PostParseTransform(...), ..., MergeNonConflicting(A, B, C, > D), ...] > >> where you know that A, B, C and D are not interfering with one another > and can be executed "simultaneously" in one sense (the restriction > here > is that a B for a given node only depends on the previous siblings and > ancestors with respect to what is done by A). So MergeNonConflicting > simply takes the result of visit_FuncDefNode of A and feeds it > directly > to visit_FuncDefNode of B. > >Yep, this might be a savings if we have lots of little stuff. > >> Many nodes are leaves and the tree is shallow so perhaps it doesn't > win > that much; but the visitor dispatch only needs to be used once > (i.e. the > types of the nodes are looked at less times) so perhaps there is some > potential here *shrug*. > >> For my own uses, gcc still uses *a lot* more running-time than Cython, > so I don't have an itch to scratch here -- how are things in SAGE? > >The opposite, Cython takes longer than gcc. Some of this might be due >to lots of re-parsing of .pxd files, which I hope to be able to >resolve in a better way (someday...). On profiling visitChildren is >up there at the top. > >I am bringing this up as food for though on how to make things >faster, but for the record I think that transforms are awesome and >well worth the additional (so far minor) cost in speed. > >- Robert > > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Tue Sep 9 00:06:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 09 Sep 2008 00:06:00 +0200 Subject: [Cython] autodoc function/method signatures Message-ID: <3303763580.1175155@smtp.netcom.no> Help for Lisandro: Use Interpreter.py (which calls get...value in the right way). Yes, no way for you to know (in fact I wrote it in summer and never told anyone about it...). If I understand correctly, the context is only used by the parser (it holds symbols defined by DEF I think, but those are expanded during parsing and so there are no such symbols left when you get to it). Anyway it sets up and passes an empty context. Dag Sverre Seljebotn -----Original Message----- From: "Lisandro Dalcin" Date: Monday, Sep 8, 2008 7:11 pm Subject: Re: [Cython] autodoc function/method signatures To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Mon, Sep 8, 2008 at 1:46 PM, Robert Bradshaw > wrote: >>> Additionally, for 'def' functions, I have no idea how to provide a >>> good 'return-type'. >> >> Not sure I'm understanding your question... for def functions the >> return type is always object, which is implied (i.e. one doesn't have >> to explicitly write "object"). > >Well, I believe saying a function returns 'object' is equivalent to >say nothing :-). Perhaps this should wait until functions annotations >are supported. > >>> We should handle default aguments; I'm currently using >>> 'arg.default.compile_time_value(None)' (I'm passing 'None' to the call >>> because not sure at this point what to pass). If this fails, I just >>> use ''. >> >> That sounds like a good first pass to me. I think a context object >> goes here, right? > >Yes, but I'm not sure where to get it. Dag, please help me!! > > > >> - Robert >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > >-- >Lisandro Dalc?n >--------------- >Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >PTLC - G?emes 3450, (3000) Santa Fe, Argentina >Tel/Fax: +54-(0)342-451.1594 >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Tue Sep 9 00:48:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 09 Sep 2008 00:48:00 +0200 Subject: [Cython] transform efficiency Message-ID: <3303766138.1266244@smtp.netcom.no> (answering the question below, sorry about top-post (on cellphone)): You do something like this: classname = None def visit_ClassDefNode(self, node): oldname = self.classname self.classname = node.X # name? # or perhaps node.entry.name self.visitchildren(node) self.classname = oldname def visit_FuncDefNode(self, node): .... use self.classname .... Dag Sverre Seljebotn -----Original Message----- From: "Lisandro Dalcin" Date: Monday, Sep 8, 2008 7:02 pm Subject: Re: [Cython] transform efficiency To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Mon, Sep 8, 2008 at 1:46 PM, Dag Sverre Seljebotn > wrote: >>> >> Lisandro simply shouldn't call self.visitchildren in visit_FuncDefNode, >> no new transform type is needed. >> > >OK, I'm not visiting children. However, what should I do if I want to >'push/pop' the class name where a method is defined? > > >> >> For my own uses, gcc still uses *a lot* more running-time than Cython, >> so I don't have an itch to scratch here -- how are things in SAGE? > >That's the situation for me, too. As a side note, I never experienced >any speed-up from using a compiled 'Scanners.so' in my projects. > > > >-- >Lisandro Dalc?n >--------------- >Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >PTLC - G?emes 3450, (3000) Santa Fe, Argentina >Tel/Fax: +54-(0)342-451.1594 >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Tue Sep 9 05:56:05 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 9 Sep 2008 00:56:05 -0300 Subject: [Cython] about subclassing strings and tuples Message-ID: Stefan, I've just saw your comments on Python-Dev list. You always stress the point that if is not possible (or at least difficult) to inherit (at the C-level) from the Py2.X builtin 'str' type. Just to be completely sure I understand the whole issues... This problem also applies to ANY object using 'PyObject_VAR_HEAD' macro (like PyTupleObject or PyFrameObject), right? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Sep 9 07:12:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Sep 2008 07:12:57 +0200 Subject: [Cython] autodoc function/method signatures In-Reply-To: References: <53781.213.61.181.86.1220603247.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <153A35AF-A3F6-46C2-9319-6EB80EFDF073@math.washington.edu> <48C58877.6010209@behnel.de> Message-ID: <48C605D9.2070007@behnel.de> Hi, Lisandro Dalcin wrote: > On Mon, Sep 8, 2008 at 5:17 PM, Stefan Behnel wrote: >> I don't think there's much gained by additionally providing type information, >> for example. > > But what about IPython? Type information is valuable when typing, for example > > In [1]: from petsc4py import PETSc > In [2]: PETSc.Mat.mult? > Type: method_descriptor > Base Class: > String Form: > Namespace: Interactive > Docstring: > mult(self, Vec x, Vec y) Ok, I agree that that's helpful. Just go for it. >>> Other stuff I'm not sure how to render is __get__/__set__/__del__ on >>> properties. Should I generate docstrings for them? >> There's only one visible docstring for them, which is the docstring of the >> property itself (i.e. of the descriptor). So augmenting the docstring doesn't >> make much sense. > > OK, so getting this (at least inside IPython) is just fine, right? > > In [3]: PETSc.Vec.size > Out[3]: > > In [4]: PETSc.Mat.size > Out[4]: Yes, that's just fine. Being a property or not is just an implementation detail. In extension classes, attributes tend to be properties much more often than in Python code, but users don't need to see that. Stefan From stefan_ml at behnel.de Tue Sep 9 08:05:59 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 09 Sep 2008 08:05:59 +0200 Subject: [Cython] about subclassing strings and tuples In-Reply-To: References: Message-ID: <48C61247.3030507@behnel.de> Hi, Lisandro Dalcin wrote: > Stefan, I've just saw your comments on Python-Dev list. Actually the py3k list. > You always > stress the point that if is not possible (or at least difficult) to > inherit (at the C-level) from the Py2.X builtin 'str' type. Just to be > completely sure I understand the whole issues... This problem also > applies to ANY object using 'PyObject_VAR_HEAD' macro (like > PyTupleObject or PyFrameObject), right? Yes. The problem is the single-malloc initialisation that includes a variable sized buffer tail right after the fixed size object struct. Strings use it for their content and tuples for their items. If you inherit from such an object in Cython (or Pyrex) in a cdef class, the normal machinery will assume that additional fields and the vtab lie right behind the original struct, so they end up inside the buffer - a sure crasher. The current result of that discussion is that any PyVarObject, i.e. str/tuples (and maybe unicode in Py3, that's not decided yet) needs special casing when cdef subclassing it in Cython. The idea is to calculate the address of their additional struct fields instead of assuming a fixed size combined struct. I added the latest comments below. We might still run into unexpected problems here (type casting and whatnot), so this still needs some more thought. Also, I will definitely not have the time to implement it. Stefan Martin von L?wis wrote: --------------------------- As people have pointed out: add new fields *after* the variable-sized members. To access it, you need to compute the length of the base object, and then cast the pointer to an extension struct. That extends to further subtypes, too. Access is slightly slower, i.e. it's not a compile-time constant, but base_address + base_address[ob_len]*elem_size - more_fields_size This still compiles efficiently, e.g. on x86, gcc compiles a struct field access to movl 20(%eax), %eax and an access with a var-sized offset into movl 8(%eax), %edx; fetch length into edx movl -20(%eax,%edx,2), %eax; access 20-byte sized struct, assuming elements of size 2 --------------------------- He then corrected the term into base_address + base_address[ob_len]*elem_size + more_fields_size in a later post. Similarly, Antoine Pitrou answered on an example I gave: --------------------- Stefan Behnel writes: > > > > cdef class MyListSubType(PyListObject): > > cdef int some_additional_int_field > > cdef my_struct* some_struct > > > > def __init__(self): > > self.some_struct = get_the_struct_pointer(...) > > self.some_additional_int_field = 1 In your example, you could wrap the additional fields (additional_int_field and some_struct) in a dedicated struct, and define a macro which gives a pointer to this struct when given the address of the object. Once you have the pointer to the struct, accessing additional fields is as simple as in the non-PyVarObject case. Something like (pseudocode): #define MyStrSubType_FIELDS_ADDR(op) \ ((struct MyStrSubType_subfields*) &((void*)op + \ PyString_Type->tp_basicsize + \ op->size * PyString_Type->tp_itemsize)) It's not as trivially cheap as a straight field access, but much less expensive than a dictionary lookup. (perhaps this needs to be a bit more complicated if you want a specific alignment for your fields) --------------------- From stefan_ml at behnel.de Wed Sep 10 16:35:25 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Sep 2008 16:35:25 +0200 (CEST) Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> Message-ID: <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Ondrej Certik wrote: > On Mon, Sep 8, 2008 at 5:12 PM, Lisandro Dalcin wrote: >> In Python 2.3 and 2.4 we have this: >> >> PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, char *); >> >> In Python 2.5 and above, we have this: >> >> PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, const char *); >> >> >> So pehaps we need the a define of inline function >> __Pyx_PyObject_GetAttr() that do the special-case according to the >> Python version. > > Either way is fine. Clearly the current state, that it prints the > warnings is a bug, so it should be fixed (either way). This specific case is easy to handle as we can replace PyObject_GetAttrString() by the equivalent PyObject_GetAttr() call and create the Python string ourselves (which would normally happen inside of PyObject_GetAttrString). I'll fix it. Stefan From ondrej at certik.cz Wed Sep 10 16:51:48 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 10 Sep 2008 16:51:48 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <85b5c3130809100751i2ae74a5ewf301ff09be09e964@mail.gmail.com> On Wed, Sep 10, 2008 at 4:35 PM, Stefan Behnel wrote: > Ondrej Certik wrote: >> On Mon, Sep 8, 2008 at 5:12 PM, Lisandro Dalcin wrote: >>> In Python 2.3 and 2.4 we have this: >>> >>> PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, char *); >>> >>> In Python 2.5 and above, we have this: >>> >>> PyAPI_FUNC(PyObject *) PyObject_GetAttrString(PyObject *, const char *); >>> >>> >>> So pehaps we need the a define of inline function >>> __Pyx_PyObject_GetAttr() that do the special-case according to the >>> Python version. >> >> Either way is fine. Clearly the current state, that it prints the >> warnings is a bug, so it should be fixed (either way). > > This specific case is easy to handle as we can replace > PyObject_GetAttrString() by the equivalent PyObject_GetAttr() call and > create the Python string ourselves (which would normally happen inside of > PyObject_GetAttrString). > > I'll fix it. Thanks a lot! Ondrej From dalcinl at gmail.com Wed Sep 10 17:10:41 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 10 Sep 2008 12:10:41 -0300 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Wed, Sep 10, 2008 at 11:35 AM, Stefan Behnel wrote: > This specific case is easy to handle as we can replace > PyObject_GetAttrString() by the equivalent PyObject_GetAttr() call and > create the Python string ourselves (which would normally happen inside of > PyObject_GetAttrString). As a reminder, please use PyString_InternFromString() (or the py3k equivalent). > > I'll fix it. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Wed Sep 10 17:53:50 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Sep 2008 17:53:50 +0200 (CEST) Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C2E131.4030600@behnel.de> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Wed, Sep 10, 2008 at 11:35 AM, Stefan Behnel > wrote: >> This specific case is easy to handle as we can replace >> PyObject_GetAttrString() by the equivalent PyObject_GetAttr() call and >> create the Python string ourselves (which would normally happen inside >> of PyObject_GetAttrString). > > As a reminder, please use PyString_InternFromString() (or the py3k > equivalent). I considered doing that and rejected it as I do not see the gain in re-re-building the Python string instead of creating it once. Why do you think this is necessary? Stefan From dalcinl at gmail.com Wed Sep 10 19:20:23 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 10 Sep 2008 14:20:23 -0300 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Wed, Sep 10, 2008 at 12:53 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> On Wed, Sep 10, 2008 at 11:35 AM, Stefan Behnel >> wrote: >>> This specific case is easy to handle as we can replace >>> PyObject_GetAttrString() by the equivalent PyObject_GetAttr() call and >>> create the Python string ourselves (which would normally happen inside >>> of PyObject_GetAttrString). >> >> As a reminder, please use PyString_InternFromString() (or the py3k >> equivalent). > > I considered doing that and rejected it as I do not see the gain in > re-re-building the Python string instead of creating it once. Why do you > think this is necessary? Well, this is the way that PyObject_GetAttrString() is implemented. If the call ends-up doing many dicts lookups (think about the chain instance->class->[intermediate_class]->base_class), then using a interned string is much faster (as string comparison ends-up being a pointer comparison). In short, I believe this is a good optimization if Cython wants its own version of PyObject_GetAttrString() implemented in terms of PyObject_GetAttr(). > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Sep 10 20:38:40 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 10 Sep 2008 20:38:40 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48C81430.5000807@student.matnat.uio.no> Lisandro Dalcin wrote: > On Wed, Sep 10, 2008 at 12:53 PM, Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> On Wed, Sep 10, 2008 at 11:35 AM, Stefan Behnel >>> wrote: >>>> This specific case is easy to handle as we can replace >>>> PyObject_GetAttrString() by the equivalent PyObject_GetAttr() call and >>>> create the Python string ourselves (which would normally happen inside >>>> of PyObject_GetAttrString). >>> As a reminder, please use PyString_InternFromString() (or the py3k >>> equivalent). >> I considered doing that and rejected it as I do not see the gain in >> re-re-building the Python string instead of creating it once. Why do you >> think this is necessary? > > Well, this is the way that PyObject_GetAttrString() is implemented. If > the call ends-up doing many dicts lookups (think about the chain > instance->class->[intermediate_class]->base_class), then using a > interned string is much faster (as string comparison ends-up being a > pointer comparison). In short, I believe this is a good optimization > if Cython wants its own version of PyObject_GetAttrString() > implemented in terms of PyObject_GetAttr(). I think Stefan is referring to a different approach: The string will be created at module startup time and held on to. So your point is valid but there's an even better approach :-) (assuming here that the module startup interns the string, which seems very very likely but I didn't check it). -- Dag Sverre From stefan_ml at behnel.de Wed Sep 10 21:04:57 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 10 Sep 2008 21:04:57 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <48C81430.5000807@student.matnat.uio.no> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <85b5c3130809070553q1d9d455dla4a99a3fa7e0e695@mail.gmail.com> <48C3D6F5.9020005@behnel.de> <85b5c3130809070658k45d71415qf5869bf158ab34ff@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48C81430.5000807@student.matnat.uio.no> Message-ID: <48C81A59.5050507@behnel.de> Hi, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Wed, Sep 10, 2008 at 12:53 PM, Stefan Behnel wrote: >>> Lisandro Dalcin wrote: >>>> On Wed, Sep 10, 2008 at 11:35 AM, Stefan Behnel >>>> wrote: >>>>> This specific case is easy to handle as we can replace >>>>> PyObject_GetAttrString() by the equivalent PyObject_GetAttr() call and >>>>> create the Python string ourselves (which would normally happen inside >>>>> of PyObject_GetAttrString). >>>> As a reminder, please use PyString_InternFromString() (or the py3k >>>> equivalent). >>> I considered doing that and rejected it as I do not see the gain in >>> re-re-building the Python string instead of creating it once. Why do you >>> think this is necessary? >> Well, this is the way that PyObject_GetAttrString() is implemented. If >> the call ends-up doing many dicts lookups (think about the chain >> instance->class->[intermediate_class]->base_class), then using a >> interned string is much faster (as string comparison ends-up being a >> pointer comparison). No, not in all cases. When the pointer comparison fails, Python still has to compare the strings. > In short, I believe this is a good optimization >> if Cython wants its own version of PyObject_GetAttrString() >> implemented in terms of PyObject_GetAttr(). We are totally not discussing our own GetAttrString() at this point (which is only used in a couple of less important places anyway). > I think Stefan is referring to a different approach: The string will be > created at module startup time and held on to. So your point is valid > but there's an even better approach :-) (assuming here that the module > startup interns the string, which seems very very likely but I didn't > check it). While true in general, this isn't currently done in this situation. The string is passed as a char*. We were discussing the specific case of importing a type from a module here, and the whole import machinery is so complex that it won't pay off to intern the class name at this point. It's a single lookup we are doing here, right after the entire module has been searched, imported and initialised. Interning the string will create it, look it up in the dictionary of interned strings, and then either insert or decref it, before it is finally looked up in the module dict and decrefed. The current implementation will create the string, look it up in the module dict and decref it. *That* is the faster way of doing it. Interning the string at module init time would only pay off if it's used elsewhere in a performance critical part of the code later on, which is not so likely for a class name. Stefan From dalcinl at gmail.com Wed Sep 10 21:42:34 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 10 Sep 2008 16:42:34 -0300 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <48C81A59.5050507@behnel.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48C81430.5000807@student.matnat.uio.no> <48C81A59.5050507@behnel.de> Message-ID: On Wed, Sep 10, 2008 at 4:04 PM, Stefan Behnel wrote: > Hi, > We are totally not discussing our own GetAttrString() at this point (which is > only used in a couple of less important places anyway). But if GetAttrString() is still used in C code, then it has to handle the different signatures for 2.3/24 and 2.5/2.6/3.0. So then perhaps the less intrusive change is to generate a custom function PyObject* __Pyx_GetAttrString(PyObject*, const char*); > Interning the string at module init time would only pay off if it's used > elsewhere in a performance critical part of the code later on, which is not so > likely for a class name. I actually agree in this point, but perhaps the approach I commented above still applies. I do not see any semantic difference in using PyString_FromString() or PyString_InternFromString(). Both calls return a new reference you own and next have to DECREF. I do not really have any strong complaint about Stefan's way for the particular case of importing a type from a module. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From parlar at gmail.com Wed Sep 10 23:14:06 2008 From: parlar at gmail.com (Jay Parlar) Date: Wed, 10 Sep 2008 17:14:06 -0400 Subject: [Cython] Cython, py2exe and cimport Message-ID: Does anyone have any tips on getting py2exe to know that a Cython module is doing a cimport of another Cython module? In the top level of my directory, I have a file PC_viewer.py, and a Cython module reader.pyx. reader.pyx does the following cimport: from filters.dc cimport DC So there's a 'filters' directory, in there I have a 'dc' module, which has a DC class. Everything works fantastically when just running with straight Python, but the executables created by py2exe don't seem very happy. $ ./PC_viewer.exe Traceback (most recent call last): File "PC_viewer.py", line 14, in File "reader.pyc", line 12, in File "reader.pyc", line 10, in __load File "dc.pxd", line 1, in reader (reader.c:8732) ImportError: No module named dc Any thoughts or tips? The 'dc' module doesn't show up in the library.zip created by py2exe. Jay P. From dalcinl at gmail.com Wed Sep 10 23:32:15 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 10 Sep 2008 18:32:15 -0300 Subject: [Cython] Cython, py2exe and cimport In-Reply-To: References: Message-ID: On Wed, Sep 10, 2008 at 6:14 PM, Jay Parlar wrote: > Does anyone have any tips on getting py2exe to know that a Cython > module is doing a cimport of another Cython module? Well, I cannot imagine a direct way. However, you can hack things. Try to add a '_deps.py' file in your package, and put there all the required imports for Cython-generated extension modules. Note that I've never used py2exe, but regarding the way (I believe) it works, perhaps the suggested hackery will hopefully solve your problems. > > In the top level of my directory, I have a file PC_viewer.py, and a > Cython module reader.pyx. reader.pyx does the following cimport: > > from filters.dc cimport DC > > > So there's a 'filters' directory, in there I have a 'dc' module, which > has a DC class. > > Everything works fantastically when just running with straight Python, > but the executables created by py2exe don't seem very happy. > > $ ./PC_viewer.exe > Traceback (most recent call last): > File "PC_viewer.py", line 14, in > File "reader.pyc", line 12, in > File "reader.pyc", line 10, in __load > File "dc.pxd", line 1, in reader (reader.c:8732) > ImportError: No module named dc > > Any thoughts or tips? The 'dc' module doesn't show up in the > library.zip created by py2exe. > > > > Jay P. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Thu Sep 11 07:13:59 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 10 Sep 2008 22:13:59 -0700 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48C81430.5000807@student.matnat.uio.no> <48C81A59.5050507@behnel.de> Message-ID: <586DD444-EFCC-492D-AFDE-C2A65E428C3A@math.washington.edu> On Sep 10, 2008, at 12:42 PM, Lisandro Dalcin wrote: > On Wed, Sep 10, 2008 at 4:04 PM, Stefan Behnel > wrote: >> Hi, >> We are totally not discussing our own GetAttrString() at this >> point (which is >> only used in a couple of less important places anyway). > > But if GetAttrString() is still used in C code, then it has to handle > the different signatures for 2.3/24 and 2.5/2.6/3.0. So then perhaps > the less intrusive change is to generate a custom function > > PyObject* __Pyx_GetAttrString(PyObject*, const char*); +1 to this idea. >> Interning the string at module init time would only pay off if >> it's used >> elsewhere in a performance critical part of the code later on, >> which is not so >> likely for a class name. > > I actually agree in this point, but perhaps the approach I commented > above still applies. > > I do not see any semantic difference in using PyString_FromString() or > PyString_InternFromString(). Both calls return a new reference you own > and next have to DECREF. I do not really have any strong complaint > about Stefan's way for the particular case of importing a type from a > module. If we're going to be using this elsewhere, we should be using PyString_InternFromString. For most attribute lookups we should be using pre-generated interned strings. - Robert From robertwb at math.washington.edu Thu Sep 11 07:15:27 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 10 Sep 2008 22:15:27 -0700 Subject: [Cython] Cython, py2exe and cimport In-Reply-To: References: Message-ID: <8DA1FA5D-6493-4481-8B76-E26D98EBA5C4@math.washington.edu> On Sep 10, 2008, at 2:32 PM, Lisandro Dalcin wrote: > On Wed, Sep 10, 2008 at 6:14 PM, Jay Parlar wrote: >> Does anyone have any tips on getting py2exe to know that a Cython >> module is doing a cimport of another Cython module? > > Well, I cannot imagine a direct way. However, you can hack things. Try > to add a '_deps.py' file in your package, and put there all the > required imports for Cython-generated extension modules. > > Note that I've never used py2exe, but regarding the way (I believe) it > works, perhaps the suggested hackery will hopefully solve your > problems. I've never used py2exe either, but if you do get it to work could you please put the info up on http://wiki.cython.org? > >> >> In the top level of my directory, I have a file PC_viewer.py, and a >> Cython module reader.pyx. reader.pyx does the following cimport: >> >> from filters.dc cimport DC >> >> >> So there's a 'filters' directory, in there I have a 'dc' module, >> which >> has a DC class. >> >> Everything works fantastically when just running with straight >> Python, >> but the executables created by py2exe don't seem very happy. >> >> $ ./PC_viewer.exe >> Traceback (most recent call last): >> File "PC_viewer.py", line 14, in >> File "reader.pyc", line 12, in >> File "reader.pyc", line 10, in __load >> File "dc.pxd", line 1, in reader (reader.c:8732) >> ImportError: No module named dc >> >> Any thoughts or tips? The 'dc' module doesn't show up in the >> library.zip created by py2exe. >> >> >> >> Jay P. >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Thu Sep 11 09:12:18 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 11 Sep 2008 09:12:18 +0200 Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48C81430.5000807@student.matnat.uio.no> <48C81A59.5050507@behnel.de> Message-ID: <48C8C4D2.4010607@behnel.de> Hi, Lisandro Dalcin wrote: > On Wed, Sep 10, 2008 at 4:04 PM, Stefan Behnel wrote: >> We are totally not discussing our own GetAttrString() at this point (which is >> only used in a couple of less important places anyway). > > But if GetAttrString() is still used in C code, then it has to handle > the different signatures for 2.3/24 and 2.5/2.6/3.0. So then perhaps > the less intrusive change is to generate a custom function > > PyObject* __Pyx_GetAttrString(PyObject*, const char*); Fine with me. >> Interning the string at module init time would only pay off if it's used >> elsewhere in a performance critical part of the code later on, which is not so >> likely for a class name. > > I actually agree in this point, but perhaps the approach I commented > above still applies. > > I do not see any semantic difference in using PyString_FromString() or > PyString_InternFromString(). Both calls return a new reference you own > and next have to DECREF. I'm not talking about a semantic difference, the interning itself is just more overhead, so it's not worth doing unless you know it will be payed for by the usage later on. Stefan From stefan_ml at behnel.de Thu Sep 11 13:09:35 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 11 Sep 2008 13:09:35 +0200 (CEST) Subject: [Cython] [PATCH] Make __Pyx_ArgTypeTest() use "const char*" to suppress warning In-Reply-To: <48C8C4D2.4010607@behnel.de> References: <85b5c3130809051241v126c740eya8c112ef66b021b@mail.gmail.com> <48C3E246.5060201@behnel.de> <85b5c3130809080817r67ccf27qd079cc4adaf1ff3a@mail.gmail.com> <64486.213.61.181.86.1221057325.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <52657.213.61.181.86.1221062030.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48C81430.5000807@student.matnat.uio.no> <48C81A59.5050507@behnel.de> <48C8C4D2.4010607@behnel.de> Message-ID: <51083.213.61.181.86.1221131375.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Stefan Behnel wrote: > Lisandro Dalcin wrote: >> I do not see any semantic difference in using PyString_FromString() or >> PyString_InternFromString(). Both calls return a new reference you own >> and next have to DECREF. > > I'm not talking about a semantic difference, the interning itself is just > more > overhead, so it's not worth doing unless you know it will be payed for by > the usage later on. To stop this discussion from taking a bikeshed planning route, I looked through the places where we currently use GetAttrString(). With the possible (!) exception of the C-API import (which can benefit from some general optimisation, but likely isn't important enough to bother), none of them is in any way performance critical, so adding the additional burden of a second dict lookup for interning is in no way rectified by whatever speedup we get *after* having interned the name that is looked up here. Stefan From dalcinl at gmail.com Thu Sep 11 18:21:32 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 11 Sep 2008 13:21:32 -0300 Subject: [Cython] cdef classes and staticmethod's Message-ID: While working in auto-generation of function/method signatures (almost ready, plus a custom script monkeypatching epydoc), I've noticed that the following snippet fails (Cython complains because 'foo' has no arguments): cdef class A: @staticmethod def foo(): return 1 Do you believe this can be fixed in an easy way? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From parlar at gmail.com Thu Sep 11 22:20:44 2008 From: parlar at gmail.com (Jay Parlar) Date: Thu, 11 Sep 2008 16:20:44 -0400 Subject: [Cython] Cython, py2exe and cimport In-Reply-To: <8DA1FA5D-6493-4481-8B76-E26D98EBA5C4@math.washington.edu> References: <8DA1FA5D-6493-4481-8B76-E26D98EBA5C4@math.washington.edu> Message-ID: On Thu, Sep 11, 2008 at 1:15 AM, Robert Bradshaw wrote: > I've never used py2exe either, but if you do get it to work could you > please put the info up on http://wiki.cython.org? Ok, the answer wasn't anything Cython specific, just some distutils particulars for py2exe. The setup() function takes a few keyword args, including 'options'. You can pass multiple things to this. So I did: setup(console = ['PC_viewer.py'], options={'py2exe': {'includes': ['module1', 'module2'], 'packages': ['package1', 'package2'] } } ) And this included the modules and packages that the automatic detection by py2exe hadn't discovered. I think that py2exe (at a high level) just scans Python code for 'import' statements, and makes sure to bundle whatever it finds. It can't scan compiled extensions, so it just needs the user to explicitly list them. I haven't made an entry on the wiki yet, but I'll try to add a short example by the end of next week. I've added it as an action OmniFocus, so I guarantee it'll get done :) Lisandro's _deps.py solution did work, but this way is a bit more explicit and correct. Thanks, Jay P. From dagss at student.matnat.uio.no Sat Sep 13 13:57:10 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 13 Sep 2008 13:57:10 +0200 Subject: [Cython] On result_code and work load status Message-ID: <48CBAA96.7010809@student.matnat.uio.no> I currently don't have much time for Cython (studies takes up most of it), and what few hours I have for Cython I always keep getting into the result_code problem; that result_code on a node should be calculated during code generation rather than during analysis. This has been changed in Pyrex, but the changes going into Cython after that is substantial enough that I get problems (as in: I need to spend much time in understanding the changes that went into Cython etc.) http://trac.cython.org/cython_trac/ticket/67 In short, this is the kind of task that I feel someone who knew the code and the changes that went into Cython better would spend half a day to a day on, while myself I could spend two days and do a worse job. OTOH, I am the one craving it. What I can say is that until this gets done, I'm blocked on my projects. While I'm happy to stay blocked until I fix it myself (it is my itch after all) I at least wanted to air the issue and see if anyone else would think this is an easier task (because they know the area better) and is willing to take it up. So my questions are: 1) Do you agree with me in the high priority I assign to this task? 2) How are we (/am I) going to get it done? Failing anything else, I think at least I'd be able to put in a few days for this in January. That's a while though. Details: With this done, one could have: - (Incrementally cleaner code when it comes to temp allocation) - Closures/inner functions with approx. a week of of work. - A "LetNode", which would make transformations much stronger: Transforming the loops to a common construct, transforming various assignments to more basic constructs etc. would be very easy. (Not a priority though.) - One would jump over an issue I currently have with how CloneNode interfers with the order of phases so that I cannot get buffers for complex numbers to work... So, I believe my motivation is practical rather than theoretical. Pyrex already having done this seems to support my argument as well. (In fact there's a lot of stuff gone into Pyrex which seems to be quite an improvement -- some of this reinvented by me over the summer in fact -- and Cython is lagging behind when it comes to code structure in some sense. Well, that's how it is, and I think remerging all Pyrex would be way too much work?) Dag Sverre From robertwb at math.washington.edu Sat Sep 13 20:41:17 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 13 Sep 2008 11:41:17 -0700 Subject: [Cython] On result_code and work load status In-Reply-To: <48CBAA96.7010809@student.matnat.uio.no> References: <48CBAA96.7010809@student.matnat.uio.no> Message-ID: On Sep 13, 2008, at 4:57 AM, Dag Sverre Seljebotn wrote: > I currently don't have much time for Cython (studies takes up most of > it), and what few hours I have for Cython I always keep getting > into the > result_code problem; that result_code on a node should be calculated > during code generation rather than during analysis. This has been > changed in Pyrex, but the changes going into Cython after that is > substantial enough that I get problems (as in: I need to spend much > time > in understanding the changes that went into Cython etc.) > > http://trac.cython.org/cython_trac/ticket/67 > > In short, this is the kind of task that I feel someone who knew the > code > and the changes that went into Cython better would spend half a day > to a > day on, while myself I could spend two days and do a worse job. > OTOH, I > am the one craving it. > > What I can say is that until this gets done, I'm blocked on my > projects. > While I'm happy to stay blocked until I fix it myself (it is my itch > after all) I at least wanted to air the issue and see if anyone else > would think this is an easier task (because they know the area better) > and is willing to take it up. > > So my questions are: > 1) Do you agree with me in the high priority I assign to this task? Yes. > 2) How are we (/am I) going to get it done? Depends. I just finished up my job here and may have some time this next week. If so I will try and do this. > > Failing anything else, I think at least I'd be able to put in a few > days > for this in January. That's a while though. > > Details: > With this done, one could have: > - (Incrementally cleaner code when it comes to temp allocation) > - Closures/inner functions with approx. a week of of work. > - A "LetNode", which would make transformations much stronger: > Transforming the loops to a common construct, transforming various > assignments to more basic constructs etc. would be very easy. (Not a > priority though.) > - One would jump over an issue I currently have with how CloneNode > interfers with the order of phases so that I cannot get buffers for > complex numbers to work... > > So, I believe my motivation is practical rather than theoretical. > Pyrex > already having done this seems to support my argument as well. (In > fact > there's a lot of stuff gone into Pyrex which seems to be quite an > improvement -- some of this reinvented by me over the summer in > fact -- > and Cython is lagging behind when it comes to code structure in some > sense. Well, that's how it is, and I think remerging all Pyrex > would be > way too much work?) Our codebases have diverged too far for a straight merge...and moving structural issues over could be hard (sometimes not worth the investment compared to other stuff we could do). It would be nice to have a centralized page with any stuff you feel we're lagging behind on. - Robert From marcin.cieslik at gmail.com Sun Sep 14 02:38:10 2008 From: marcin.cieslik at gmail.com (=?UTF-8?Q?Marcin_Cie=C5=9Blik?=) Date: Sat, 13 Sep 2008 20:38:10 -0400 Subject: [Cython] NumPy output works on 32bit not on 64bit Message-ID: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> Hello, Does anyone know why this code fails on a 64bit linux machine (it is working on 32 just fine): The compiler complains: bit.c:792: warning: passing argument 3 of '*(PyArray_API + 744u)' from incompatible pointer type and I get a segmentation fault If it is not an issue with Cython could you please direct me somewhere, if have almost no C/C++ skill. Thank you, Marcin ##### cimport numpy as np ctypedef np.double_t DTYPE_t from numpy cimport NPY_DOUBLE, NPY_UINT from stdlib cimport malloc, realloc, free cdef extern from "arrayobject.h": cdef object PyArray_SimpleNewFromData(int nd, int *dims,\ int typenum, void *data) cdef void import_array() import_array() print sizeof(DTYPE_t) print sizeof(void*) print sizeof(DTYPE_t*) cdef int i, k = 100 cdef DTYPE_t *dst = malloc(k * sizeof(DTYPE_t)) for 0 <= i < k: dst[i] = i cdef np.ndarray dist = PyArray_SimpleNewFromData(1, &k, NPY_DOUBLE, dst) print dist #### From dagss at student.matnat.uio.no Sun Sep 14 12:15:54 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 14 Sep 2008 12:15:54 +0200 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> Message-ID: <48CCE45A.1040703@student.matnat.uio.no> Marcin Cie?lik wrote: > Hello, > > Does anyone know why this code fails on a 64bit linux machine (it is > working on 32 just fine): > The compiler complains: > > bit.c:792: warning: passing argument 3 of '*(PyArray_API + 744u)' from > incompatible pointer type > > and I get a segmentation fault > > If it is not an issue with Cython could you please direct me > somewhere, if have almost no C/C++ skill. See comments below. > ##### > cimport numpy as np > ctypedef np.double_t DTYPE_t This isn't related to the problem, but when you are using the NPY_DOUBLE etc. constants directly you should instead do ctypedef np.npy_double DTYPE_t This is because there isn't a 1:1 correspondance between the "shorthand" types and the NPY_TYPE constants (see bottom of Cython/Includes/numpy.pxd for details. BTW, patches for numpy.pxd to add the Py_Array_SimpleNew... calls are welcome if submitted). > from numpy cimport NPY_DOUBLE, NPY_UINT > from stdlib cimport malloc, realloc, free > > cdef extern from "arrayobject.h": > cdef object PyArray_SimpleNewFromData(int nd, int *dims,\ > int typenum, void *data) dims should be declared with the type npy_intp instead (and this is also what your compiler complained about). > cdef void import_array() > import_array() > > print sizeof(DTYPE_t) > print sizeof(void*) > print sizeof(DTYPE_t*) > > cdef int i, k = 100 Here is the problem. If you make k an npy_intp instead then it will be a 64-bit int. So: cdef np.npy_intp k = 100 This wouldn't matter in many circumstances but as you are taking the address to k the call to SimpleNewFromData it is crucial. (So NumPy thought your array had a random billion elements coming from whatever bits were after the 32-bit "k" variable on the stack...) -- Dag Sverre From dagss at student.matnat.uio.no Sun Sep 14 12:17:20 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 14 Sep 2008 12:17:20 +0200 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <48CCE45A.1040703@student.matnat.uio.no> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> Message-ID: <48CCE4B0.8060901@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Marcin Cie?lik wrote: >> Hello, >> >> Does anyone know why this code fails on a 64bit linux machine (it is >> working on 32 just fine): >> The compiler complains: >> >> bit.c:792: warning: passing argument 3 of '*(PyArray_API + 744u)' from >> incompatible pointer type >> >> and I get a segmentation fault >> >> If it is not an issue with Cython could you please direct me >> somewhere, if have almost no C/C++ skill. > > See comments below. > >> ##### >> cimport numpy as np >> ctypedef np.double_t DTYPE_t > > This isn't related to the problem, but when you are using the NPY_DOUBLE > etc. constants directly you should instead do > > ctypedef np.npy_double DTYPE_t > > This is because there isn't a 1:1 correspondance between the "shorthand" > types and the NPY_TYPE constants (see bottom of > Cython/Includes/numpy.pxd for details. BTW, patches for numpy.pxd to add > the Py_Array_SimpleNew... calls are welcome if submitted). > >> from numpy cimport NPY_DOUBLE, NPY_UINT >> from stdlib cimport malloc, realloc, free >> >> cdef extern from "arrayobject.h": >> cdef object PyArray_SimpleNewFromData(int nd, int *dims,\ >> int typenum, void *data) > > dims should be declared with the type npy_intp instead (and this is also > what your compiler complained about). Sorry, I meant npy_intp*. -- Dag Sverre From marcin.cieslik at gmail.com Sun Sep 14 16:32:13 2008 From: marcin.cieslik at gmail.com (=?UTF-8?Q?Marcin_Cie=C5=9Blik?=) Date: Sun, 14 Sep 2008 10:32:13 -0400 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <48CCE4B0.8060901@student.matnat.uio.no> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> Message-ID: <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> Thanks, I followed your advices and got my first useful cython program cross-platform. It is a kd-tree implementation which takes a numpy array as input and allows for k-nearest-neighbor and points-within-radius queries. A crude benchmark shows that for 7000 3D points (~number of heavy atoms in a protein) it is almost as fast as ANN via scikits-ANN and more than 3x faster than the current implementation of Bio.KDTree from biopython (SWIGed C++), which both have problems on 64bit machines. pyx file: http://pastebin.com/maade362 pxd file: http://pastebin.com/m51effa8c For an array of 7000 3D points (0.0; 1.0): 1) find k=20 from (0.5, 0.5, 0.5) # scikits-ann: best of 3: 8.19 ms per loop # cython kd-tree: best of 3: 9.3 ms per loop 2) find all within radius 0.04 (~300 points): # Bio.KDTree: best of 3: 30.9 ms per loop # cython kd-tree: best of 3: 9.35 ms per loop The numpy over-head from random matrix computation is ~1ms Yours, Marcin >> This isn't related to the problem, but when you are using the NPY_DOUBLE >> etc. constants directly you should instead do >> >> ctypedef np.npy_double DTYPE_t >> >> This is because there isn't a 1:1 correspondance between the "shorthand" >> types and the NPY_TYPE constants (see bottom of >> Cython/Includes/numpy.pxd for details. BTW, patches for numpy.pxd to add >> the Py_Array_SimpleNew... calls are welcome if submitted). >> >>> from numpy cimport NPY_DOUBLE, NPY_UINT >>> from stdlib cimport malloc, realloc, free >>> >>> cdef extern from "arrayobject.h": >>> cdef object PyArray_SimpleNewFromData(int nd, int *dims,\ >>> int typenum, void *data) >> >> dims should be declared with the type npy_intp instead (and this is also >> what your compiler complained about). From dagss at student.matnat.uio.no Wed Sep 17 12:30:11 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 17 Sep 2008 12:30:11 +0200 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> Message-ID: <48D0DC33.7000200@student.matnat.uio.no> Marcin Cie?lik wrote: > Thanks, > > I followed your advices and got my first useful cython program > cross-platform. It is a kd-tree implementation which takes a numpy > array as input and allows for k-nearest-neighbor and > points-within-radius queries. A crude benchmark shows that for 7000 3D > points (~number of heavy atoms in a protein) it is almost as fast as > ANN via scikits-ANN and more than 3x faster than the current > implementation of Bio.KDTree from biopython (SWIGed C++), which both > have problems on 64bit machines. > Nice. Will you contribute it to SciPy/any other library? Your code inspired me to make some additions to the buffer code, up in the devel branch now. Your calls to PyArray_SimpleNewFromData should no longer be necesarry, instead you can (if you so wish, of course...) do cdef np.ndarray[unsigned int, mode="c"] ridx = np.empty((k,), dtype='I') ... for 0 <= i < k: ridx[i] = self.kdpnts[idx[i]].index return (ridx, ...) without any penalty. I.e. mode="c" causes no stride multiplication to happen on the last dimension (the only on in 1D). (The only additional overhead should be calling np.empty vs. PyArray_SimpleNewFromData + malloc). If you do this I'm very interested in the benchmarks and whether there's a performance regression, and also what happens to the timings if mode is set to "strided" instead. (With "strided" you have the advantage that you can take an optional "out" parameter like numpy often does...) Dag Sverre From dagss at student.matnat.uio.no Wed Sep 17 12:57:56 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 17 Sep 2008 12:57:56 +0200 Subject: [Cython] Builtin declarations + should BufferException be backported? Message-ID: <48D0E2B4.5010001@student.matnat.uio.no> According to the buffer PEP, __getbuffer__ should raise a BufferError under certain circumstances. This is not present in older versions of Python. Backport/emulate or not? Any good ideas for how it should happen? (I'm blank when it comes to both builtins and creating exception types.) One option could be to simply make BufferError be a ValueError in Python 2.5-; still, the main question I guess is whether we import BufferError into the Cython builtin scope. Slightly related, look at this code in Symtab.py: def declare_builtin(self, name, pos): if not hasattr(__builtin__, name): If I'm not entirely wrong, this means that BufferError (and other new builtin symbols) will be available only if you run Cython under Python 2.6+. I find the fact that results of a compilation depends on the Python version used for running Cython somewhat disturbing. Should perhaps a literal list of accepted Cython builtins be present in Symtab.py instead? (Or couple it to Builtins.py somehow?) Dag Sverre From dalcinl at gmail.com Wed Sep 17 20:39:42 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 17 Sep 2008 15:39:42 -0300 Subject: [Cython] embeding functions signatures almost done Message-ID: I've almost finished the transform in charge of embeding functions signatures in docstrings. I want to ask your about some issues I need to handle to finish it. * Position in the transform pipeline: I'm curently inserting the auto-signature transform after the 'ResolveOptions' one in the transform pipeline. Dag, tell me if this is OK. * Enabling option?: Should I used a pragma-like option (like 'boundscheck') or a global option (like '--embed-positions')?. I do not really see the benefit of selectively enabling the embeding of signatures, so I believe we should use a global option. * Always embed?: If a function do not have a docstring (ie., node.doc is None), should I embed the signature anyway? Perhaps this could be controlled with the '--embed-signatures' options providing a level, like let say the '--cleanup' option? * Remove identation?: What do you thing about previously call 'textwrap.dedent()' on the original function docstring if it is present? Do you believe there ar use cases for preserving the common level of identation in docstrings? If docstrings are not 'dedented' (Do such word exists?), then we are going to be carefull about the other option '--embed-positions', that do not take into account docstring identation. * Signature format: I'm currently generating a dotted name 'ClassName.methodname' for the signature. Not completelly sure If I should add the 'ClassName', or a full namespace name 'ModuleName.ClassName.NestedClassName', or just the undotted 'methodname'. About argument list, 'self' is there, and types are there; no 'return_type' is yet generated, not sure how to build a meanigful value. In short: appart from the ClassName being added, the signature looks almost the same that the function definition in Cython source code (though I'm not currently generating the 'not None' stuff. it looks really ugly!!) Finally, builtin help() and IPython seems to work just fine with the currently generated signatures. Epydoc needs a custom script with a few lines monkeypatching the regex used for parsing builting functions signatures. I've not tried Sphinx yet, not even sure if it handle builtin functions like Epydoc has. All my current implementation lives in two scripts: one monkeypatching Cython, and other for Epydoc. If any of you want to give a try before pushing this in cython-devel, just let me know, I'll send the scripts. Regards, -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Sep 17 21:54:11 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 17 Sep 2008 21:54:11 +0200 (CEST) Subject: [Cython] embeding functions signatures almost done In-Reply-To: References: Message-ID: <62571.193.157.228.10.1221681251.squirrel@webmail.uio.no> Lisandro wrote: > I've almost finished the transform in charge of embeding functions > signatures in docstrings. I want to ask your about some issues I need > to handle to finish it. > > * Position in the transform pipeline: I'm curently inserting the > auto-signature transform after the 'ResolveOptions' one in the > transform pipeline. Dag, tell me if this is OK. For this, anywhere where you have the needed information available is ok. I guess you read the type info from nodes then, and not the "parsed" info that is available later? (that could make sense and is not necesarrily a bad sign) > * Remove identation?: What do you thing about previously call > 'textwrap.dedent()' on the original function docstring if it is > present? Do you believe there ar use cases for preserving the common > level of identation in docstrings? If docstrings are not 'dedented' > (Do such word exists?), then we are going to be carefull about the > other option '--embed-positions', that do not take into account > docstring identation. -1. An aim is as high degree of Python compatability as possible, and dedenting automatically would mean less Python compatability (without any really good reason). Taking the inverse approach seems better: Scan for common indentation and insert that prior to the signature you are inserting. Remember that some issues/features (like any issues with indendation) can just be left for later though; "release early, release often!". I don't have an opinion about the rest. > All my current implementation lives in two scripts: one monkeypatching > Cython, and other for Epydoc. If any of you want to give a try before > pushing this in cython-devel, just let me know, I'll send the scripts. Perhaps attach patches to the ticket and ping the list when the issues above is sorted out? If you don't like working on a hg clone, have you looked at the hg "queue" functionality for creating, pushing and popping patches? Then you don't need a script. (Just a tip.) Open question: Do you plan to submit a patch to epydoc (would they even consider Cython support?), or is it something that should be shipped with Cython somehow? Dag Sverre From dalcinl at gmail.com Wed Sep 17 22:32:38 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 17 Sep 2008 17:32:38 -0300 Subject: [Cython] embeding functions signatures almost done In-Reply-To: <62571.193.157.228.10.1221681251.squirrel@webmail.uio.no> References: <62571.193.157.228.10.1221681251.squirrel@webmail.uio.no> Message-ID: On Wed, Sep 17, 2008 at 4:54 PM, Dag Sverre Seljebotn wrote: > Lisandro wrote: >> I've almost finished the transform in charge of embeding functions >> signatures in docstrings. I want to ask your about some issues I need >> to handle to finish it. >> >> * Position in the transform pipeline: I'm curently inserting the >> auto-signature transform after the 'ResolveOptions' one in the >> transform pipeline. Dag, tell me if this is OK. > > For this, anywhere where you have the needed information available is ok. > I guess you read the type info from nodes then, and not the "parsed" info > that is available later? (that could make sense and is not necesarrily a > bad sign) > I did not understand you here, but anyway it seems to work at the place I'm inserting it. Or perhaps I could get more info (like return type in 'cpdef' funcions) if I insert the transform near the end? >> * Remove identation?: What do you thing about previously call >> 'textwrap.dedent()' on the original function docstring if it is >> present? Do you believe there ar use cases for preserving the common >> level of identation in docstrings? If docstrings are not 'dedented' >> (Do such word exists?), then we are going to be carefull about the >> other option '--embed-positions', that do not take into account >> docstring identation. > > -1. An aim is as high degree of Python compatability as possible, and > dedenting automatically would mean less Python compatability (without any > really good reason). > > Taking the inverse approach seems better: Scan for common indentation and > insert that prior to the signature you are inserting. > > Remember that some issues/features (like any issues with indendation) can > just be left for later though; "release early, release often!". > Well, then this will be for the future, I'm not sure what to do in every case (docstrings could start with 0 or more blanks and next a newline, or docstrings could start with some text provining a short summary, like in numpy, and all that variants!): > Perhaps attach patches to the ticket and ping the list when the issues > above is sorted out? OK, > If you don't like working on a hg clone, have you looked at the hg "queue" > functionality for creating, pushing and popping patches? Then you don't > need a script. (Just a tip.) Yes, I know about hg queues, bet never got learn really understand them. I'll just clone and post patches. > Open question: Do you plan to submit a patch to epydoc (would they even > consider Cython support?), or is it something that should be shipped with > Cython somehow? Perhaps in the near future we can ask the Epydoc developer so suport Cython. But we should ship a tool with Cython for the moment. Epydoc is prepared for building signatures for core Python builtin functions and methods, and the 'format' of those signatures are quite different to what I'm currently generating. If we want to use Epydoc right now (and in the future, with current 3.0.X version) we will need my monkeypatching script. It is a real short script, take a look: http://petsc.cs.iit.edu/petsc4py/petsc4py-dev/file/dba78e8da92a/misc/epydoc-cython.py -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Thu Sep 18 09:53:38 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 18 Sep 2008 09:53:38 +0200 Subject: [Cython] embeding functions signatures almost done In-Reply-To: References: <62571.193.157.228.10.1221681251.squirrel@webmail.uio.no> Message-ID: <48D20902.8040506@student.matnat.uio.no> Lisandro Dalcin wrote: > On Wed, Sep 17, 2008 at 4:54 PM, Dag Sverre Seljebotn > wrote: > >> Lisandro wrote: >> >>> I've almost finished the transform in charge of embeding functions >>> signatures in docstrings. I want to ask your about some issues I need >>> to handle to finish it. >>> >>> * Position in the transform pipeline: I'm curently inserting the >>> auto-signature transform after the 'ResolveOptions' one in the >>> transform pipeline. Dag, tell me if this is OK. >>> >> For this, anywhere where you have the needed information available is ok. >> I guess you read the type info from nodes then, and not the "parsed" info >> that is available later? (that could make sense and is not necesarrily a >> bad sign) >> >> > > I did not understand you here, but anyway it seems to work at the > place I'm inserting it. Or perhaps I could get more info (like return > type in 'cpdef' funcions) if I insert the transform near the end? > Sorry, my fault. Basically, when I wrote my description I meant to imply (and should have clarified better) that the AnalyseDeclarationsTransform interprets the nodes containing the type declarations into a purer Python representation (see PyrexTypes.py). However for what you are doing, walking the CSimpleTypeNodes might be just as good. (Inserting the transform after AnalyseDeclarationsTransform makes certain things The end of the story is: If things work, then the position you picked is fine! And as for the return type of cpdef functions, it seems like the information you need is in the "base_type" attribute of CFuncDefNode. One thing I thought about for compiler directives vs. command line options: I am now +1 for a directive! Reason: Whether or not you want this auto-generated line in your docstrings is tightly connected with the source code you write, so it makes sense that it can be controlled using #cython: autofunctiondoc=True in the beginning of the file. (If you want we can disable decorator and with statement usage of this directive, though it doesn't really hurt to leave it in even if nobody would use it?) I.e. if you write your signatures in the docstrings yourself you would want to turn it off in the same source file, and so on. > Perhaps in the near future we can ask the Epydoc developer so suport > Cython. But we should ship a tool with Cython for the moment. > I guess you can just stick it in the Tools subdirectory. Dag Sverre From dagss at student.matnat.uio.no Thu Sep 18 09:56:50 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 18 Sep 2008 09:56:50 +0200 Subject: [Cython] embeding functions signatures almost done In-Reply-To: <48D20902.8040506@student.matnat.uio.no> References: <62571.193.157.228.10.1221681251.squirrel@webmail.uio.no> <48D20902.8040506@student.matnat.uio.no> Message-ID: <48D209C2.6060600@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Wed, Sep 17, 2008 at 4:54 PM, Dag Sverre Seljebotn >> wrote: >> >>> Lisandro wrote: >>> >>>> I've almost finished the transform in charge of embeding functions >>>> signatures in docstrings. I want to ask your about some issues I need >>>> to handle to finish it. >>>> >>>> * Position in the transform pipeline: I'm curently inserting the >>>> auto-signature transform after the 'ResolveOptions' one in the >>>> transform pipeline. Dag, tell me if this is OK. >>>> >>> For this, anywhere where you have the needed information available >>> is ok. >>> I guess you read the type info from nodes then, and not the "parsed" >>> info >>> that is available later? (that could make sense and is not >>> necesarrily a >>> bad sign) >>> >>> >> >> I did not understand you here, but anyway it seems to work at the >> place I'm inserting it. Or perhaps I could get more info (like return >> type in 'cpdef' funcions) if I insert the transform near the end? >> > Sorry, my fault. Basically, when I wrote my description I meant to > imply (and should have clarified better) that the > AnalyseDeclarationsTransform interprets the nodes containing the type > declarations into a purer Python representation (see PyrexTypes.py). > However for what you are doing, walking the CSimpleTypeNodes might be > just as good. > > (Inserting the transform after AnalyseDeclarationsTransform makes > certain things (I need a countdown timer on my send button...) I meant to say: Inserting the transform after AnalyseDeclarationsTransform makes certain things more difficult -- because for cpdef functions, a "python version node" is created and the docstring copied so you would need to change it in multiple locations and so on. So actually, what you have done will be more convenient and make this more isolated and less likely to break if the cpdef implementation is changed. Looking forward to see this :-) Dag Sverre From marcin.cieslik at gmail.com Thu Sep 18 16:47:02 2008 From: marcin.cieslik at gmail.com (=?UTF-8?Q?Marcin_Cie=C5=9Blik?=) Date: Thu, 18 Sep 2008 10:47:02 -0400 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <48D0DC33.7000200@student.matnat.uio.no> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> <48D0DC33.7000200@student.matnat.uio.no> Message-ID: <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> Hello, > Nice. Will you contribute it to SciPy/any other library? I'd love to. It will find it's way into my zenpdb: http://code.google.com/p/zenpdb/ which is the reason i developed it. I do not know if projects like scipy/biopython accept cython code. Biopython has a very fresh (not yet released) implementation of kd-trees in C, but without knn look-up. further i've been told that the maker of h-cluste/scipy-cluster is involved in some big effort to create a partitioning / clustering + nearest neighbor look-up / classification C-library with bindings to everything. Is there any scientific cython code besides sage? > Your code inspired me to make some additions to the buffer code, up in > the devel branch now. Your calls to PyArray_SimpleNewFromData should no > longer be necesarry, instead you can (if you so wish, of course...) do > > cdef np.ndarray[unsigned int, mode="c"] ridx = np.empty((k,), dtype='I') > ... > for 0 <= i < k: > ridx[i] = self.kdpnts[idx[i]].index > return (ridx, ...) It looks fantastic, I think the direction of avoiding the PyArray_* calls is the right one. > without any penalty. I.e. mode="c" causes no stride multiplication to > happen on the last dimension (the only on in 1D). (The only additional > overhead should be calling np.empty vs. PyArray_SimpleNewFromData + malloc). This should not be a problem since most of the time the algorithm spends in qsort anyway. I don't know why but recursion overhead seems quite high for cython generated code. The resursive version of qsort was about 2x slower. The numpy qsort code (in C) is actually ~60% faster than mine (http://svn.scipy.org/svn/numpy/trunk/numpy/core/src/_sortmodule.c.src), but using it would require to make qsort a cpdef function (and slice arrays / concatenate ) which i assumed would be slower. I think that if cython had a do ... while loop (i don't think it should), which is at the heart of qsort my code would be closer to numpy. What i could improve performance wise is to make tree-creation non-recursive, but in my first application I will create a static tree and query it multiple times so it's not on my priority list. > If you do this I'm very interested in the benchmarks and whether there's > a performance regression, and also what happens to the timings if mode > is set to "strided" instead. (With "strided" you have the advantage that > you can take an optional "out" parameter like numpy often does...) Here are timings for my non-strided version. For an array of 7000 3D points (0.0; 1.0): 1) find k=20 from (0.5, 0.5, 0.5) # scikits-ann: best of 3: 8.19 ms per loop # cython kd-tree: best of 3: 9.3 ms per loop 2) find all within squared radius 0.04 (~300 points): # Bio.KDTree: best of 3: 30.9 ms per loop # cython kd-tree: best of 3: 9.35 ms per loop he numpy over-head from random matrix computation is ~1ms strdided using the devel-cython api, will come :) Marcin From dagss at student.matnat.uio.no Thu Sep 18 17:44:27 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 18 Sep 2008 17:44:27 +0200 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> <48D0DC33.7000200@student.matnat.uio.no> <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> Message-ID: <48D2775B.5000306@student.matnat.uio.no> Marcin Cie?lik wrote: > Hello, > >> Nice. Will you contribute it to SciPy/any other library? > > I'd love to. It will find it's way into my zenpdb: > http://code.google.com/p/zenpdb/ which is the reason i developed it. I > do not know if projects like scipy/biopython accept cython code. > Biopython has a very fresh (not yet released) implementation of > kd-trees in C, but without knn look-up. further i've been told that > the maker of h-cluste/scipy-cluster is involved in some big effort to > create a partitioning / clustering + nearest neighbor look-up / > classification C-library with bindings to everything. Is there any > scientific cython code besides sage? SciPy had a longish Cython thread some time ago (June-ish) though I never checked back to see the results. >> cdef np.ndarray[unsigned int, mode="c"] ridx = np.empty((k,), dtype='I') >> ... >> for 0 <= i < k: >> ridx[i] = self.kdpnts[idx[i]].index >> return (ridx, ...) > > It looks fantastic, I think the direction of avoiding the PyArray_* > calls is the right one. Note that the only new thing is mode="c"; you can do this with strided in the latest release. (np.ndarray[...] can be assigned any numpy array objects, also those you create yourself). BTW you need to import numpy as a Python module as well: import numpy as np (for calling np.empty the cimport is not enough) > This should not be a problem since most of the time the algorithm > spends in qsort anyway. I don't know why but recursion overhead seems > quite high for cython generated code. The resursive version of qsort > was about 2x slower. The numpy qsort code (in C) is actually ~60% > faster than mine > (http://svn.scipy.org/svn/numpy/trunk/numpy/core/src/_sortmodule.c.src), > but using it would require to make qsort a cpdef function (and slice > arrays / concatenate ) which i assumed would be slower. I think that cpdef is not slower in itself (it uses a cdef calling from other Cython code), but when passing references out NumPy it would be horrible, yes. This is a long-standing concern of mine: There's no fast and convenient standard algorithmic library to use with Cython code, without going into Python land. Good wrappers for C++ and STL (and better C++ template support) would get us somewhere. Then you could just call C++ qsort. Recursion overhead *should* be almost the same as C in Cython (and if not we need to find a mechanism, or even compiler directive, to fix it). You could try to have a look at the generated C (use the -a switch to Cython!) and see if there's any slow operations perhaps? Does NumPy use recursion? And are you sure you compile your code with optimization turned on? (If gcc doesn't inline those "swap" for instance you'll have a problem). And BTW, you can write A[i], A[r] = A[r], A[i] and it will turn into efficient C for swapping elements :-) -- Dag Sverre From dalcinl at gmail.com Thu Sep 18 17:59:35 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Sep 2008 12:59:35 -0300 Subject: [Cython] short option clash with -X Message-ID: Currently, short option '-X' is associated to '--link' and '--directive' (see Cython/Compiler/CmdLine.py). I believe the -X option was for '--directive'. Can any of you push a fix in cython and cython-devel repos? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Thu Sep 18 19:27:41 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 18 Sep 2008 19:27:41 +0200 Subject: [Cython] short option clash with -X In-Reply-To: References: Message-ID: <48D28F8D.30302@student.matnat.uio.no> Lisandro Dalcin wrote: > Currently, short option '-X' is associated to '--link' and > '--directive' (see Cython/Compiler/CmdLine.py). I believe the -X > option was for '--directive'. Can any of you push a fix in cython and > cython-devel repos? > Ouch. I suppose that is my fault... The thing is, -X was there first, and the current release uses it for --link. So I dare not do this myself, I'll leave any change for the project leads. It is listed as experimental and not documented so I'm +1 for the change though (as using -X was agreed upon on the list anyway). (A note in the release notes would be good though.) (You can just modify your own copy if you want to for now, also any testcases you write in tests/run should use #cython: headers if you are using directives so there it doesn't matter.) -- Dag Sverre From dalcinl at gmail.com Thu Sep 18 23:28:54 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Sep 2008 18:28:54 -0300 Subject: [Cython] PATCH: embed signatures Message-ID: Function/method signature embedding finally implemented. Currently, this seems to work just fine for mpi4py and petsc4py. I would love Stefan give a try in lxml, and Robert in sage. Fernando and Brian CC'd (sorry if you get this twice!), as they could provide further comments on this, especially regarding the way signatures are formatted. After some rounds, perhaps IPython could be made even smarter at the time of displaying info on the console. Finally, some notes about the attached patch. * All this needs to be properly documented (I hardly find spare time for documenting my own projects!) * A testcase was added, perhaps it needs to be extended. * This do not play well with '--embed-positions' global option * I did not handle any indentation issues. * Pehaps the determination of the value for default arguments could be made smarter. * The epydoc monkeypatching in Tools/cython-epydoc.py only tested with epydoc-3.0.1. Enjoy! -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: embedsignature.diff Type: text/x-patch Size: 9655 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080918/2a06c192/attachment.bin From dalcinl at gmail.com Thu Sep 18 23:38:31 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Sep 2008 18:38:31 -0300 Subject: [Cython] PATCH: embed signatures In-Reply-To: References: Message-ID: Sorry, in the patch attached in last mail, the 'Tools/cython-epydoc.py' script is a bit broken. The attached script is perhaps more appropriate. On Thu, Sep 18, 2008 at 6:28 PM, Lisandro Dalcin wrote: > Function/method signature embedding finally implemented. Currently, > this seems to work just fine for mpi4py and petsc4py. I would love > Stefan give a try in lxml, and Robert in sage. Fernando and Brian CC'd > (sorry if you get this twice!), as they could provide further comments > on this, especially regarding the way signatures are formatted. After > some rounds, perhaps IPython could be made even smarter at the time of > displaying info on the console. > > Finally, some notes about the attached patch. > > * All this needs to be properly documented (I hardly find spare time > for documenting my own projects!) > * A testcase was added, perhaps it needs to be extended. > * This do not play well with '--embed-positions' global option > * I did not handle any indentation issues. > * Pehaps the determination of the value for default arguments could be > made smarter. > * The epydoc monkeypatching in Tools/cython-epydoc.py only tested with > epydoc-3.0.1. > > Enjoy! > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: cython-epydoc.py Type: text/x-python Size: 1435 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080918/5f199586/attachment.py From strawman at astraw.com Fri Sep 19 01:11:26 2008 From: strawman at astraw.com (Andrew Straw) Date: Thu, 18 Sep 2008 16:11:26 -0700 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> <48D0DC33.7000200@student.matnat.uio.no> <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> Message-ID: <48D2E01E.2010707@astraw.com> Marcin Cie?lik wrote: > Hello, > >> Nice. Will you contribute it to SciPy/any other library? > > I'd love to. It will find it's way into my zenpdb: > http://code.google.com/p/zenpdb/ which is the reason i developed it. I > do not know if projects like scipy/biopython accept cython code. > Biopython has a very fresh (not yet released) implementation of > kd-trees in C, but without knn look-up. further i've been told that > the maker of h-cluste/scipy-cluster is involved in some big effort to > create a partitioning / clustering + nearest neighbor look-up / > classification C-library with bindings to everything. Is there any > scientific cython code besides sage? numpy's mtrand.c is compiled from mtrand.pyx. (Using Pyrex at my last check.) -Andrew From dalcinl at gmail.com Fri Sep 19 03:00:22 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 18 Sep 2008 22:00:22 -0300 Subject: [Cython] Cython-based wrapper to ANN Message-ID: Marcin, some time ago I've implented a Cython-based wrapper to ANN. Only kSearch is available, just because I stopped there writing code. Are you interested in give it a try? I can send you a tarball if you wish. I'm really curious about a performance comparison between your All-Cython implementations of a kdtree, and the C++ one of ANN ;-). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Fri Sep 19 16:34:31 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 19 Sep 2008 16:34:31 +0200 Subject: [Cython] PATCH: embed signatures In-Reply-To: References: Message-ID: <48D3B877.1030200@student.matnat.uio.no> Lisandro Dalcin wrote: > Function/method signature embedding finally implemented. Currently, > this seems to work just fine for mpi4py and petsc4py. I would love > Stefan give a try in lxml, and Robert in sage. Fernando and Brian CC'd > (sorry if you get this twice!), as they could provide further comments > on this, especially regarding the way signatures are formatted. After > some rounds, perhaps IPython could be made even smarter at the time of > displaying info on the console. > Excellent. I've commited it now. I didn't close the ticket though as def f(unsigned short i): pass ...will get "int" as the description. Furthermore def f(object[int, ndim=3] i): pass won't render into a docstring either. In order to remedy this cleanly some internal Cython refactorings should probably be made, so I propose we just leave it like this for now. My thoughts on that are up on http://trac.cython.org/cython_trac/ticket/2 (If you want to have a shot at that as well I can outline it in more detail, but if you just leave it I might pick it up myself at some point too..) > Finally, some notes about the attached patch. > > * All this needs to be properly documented (I hardly find spare time > for documenting my own projects!) > * A testcase was added, perhaps it needs to be extended. > I added some stuff to the testcase, have a look. Dag Sverre From dalcinl at gmail.com Fri Sep 19 17:24:15 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 19 Sep 2008 12:24:15 -0300 Subject: [Cython] PATCH: embed signatures In-Reply-To: <48D3B877.1030200@student.matnat.uio.no> References: <48D3B877.1030200@student.matnat.uio.no> Message-ID: On Fri, Sep 19, 2008 at 11:34 AM, Dag Sverre Seljebotn wrote: > I didn't close the ticket though as > > def f(unsigned short i): pass > > ...will get "int" as the description. Ups! My fault, I'm working right now in a fix for this.... > Furthermore > > def f(object[int, ndim=3] i): pass > > won't render into a docstring either. In order to remedy this cleanly > some internal Cython refactorings should probably be made, so I propose > we just leave it like this for now. My thoughts on that are up on > > http://trac.cython.org/cython_trac/ticket/2 > > (If you want to have a shot at that as well I can outline it in more > detail, but if you just leave it I might pick it up myself at some point > too..) Appart from the specifics of buffer syntax, I can continue working on this, just ask for fixes! >> > I added some stuff to the testcase, have a look. > OK. I believe I'll add more stuff. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Fri Sep 19 17:33:25 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 19 Sep 2008 17:33:25 +0200 Subject: [Cython] PATCH: embed signatures In-Reply-To: References: <48D3B877.1030200@student.matnat.uio.no> Message-ID: <48D3C645.6040305@student.matnat.uio.no> Lisandro Dalcin wrote: > On Fri, Sep 19, 2008 at 11:34 AM, Dag Sverre Seljebotn > wrote: > > >> I didn't close the ticket though as >> >> def f(unsigned short i): pass >> >> ...will get "int" as the description. >> > > Ups! My fault, I'm working right now in a fix for this.... > Please have a look at PyrexTypes.py first though, and make sure that you don't duplicate code -- I think there's already code for making Cython representations of types. (However you need to move your transform to after AnalyseDeclarations and inspect funcnode.local_scope.arg_entries to access the type objects) Dag Sverre From dagss at student.matnat.uio.no Fri Sep 19 18:10:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 19 Sep 2008 18:10:00 +0200 Subject: [Cython] PATCH: embed signatures Message-ID: <3304692645.446586@smtp.netcom.no> Have some time for more details while waiting for my bus...: - When moving the transform to after AnalyseDeclarations, you must modify funcnode.py_func.doc as well as funcnode.doc for "cpdef" functions. - Rather than inspecting the syntax tree nodes for the arguments, funcnode.local_scope.arg_entries can be iterated. It contains a quite clean view of the arguments (should lean to fewer special cases in your code). Got to go. Dag Sverre Seljebotn -----Original Message----- From: Dag Sverre Seljebotn Date: Friday, Sep 19, 2008 5:33 pm Subject: Re: [Cython] PATCH: embed signatures To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Lisandro Dalcin wrote: > On Fri, Sep 19, 2008 at 11:34 AM, Dag Sverre Seljebotn > wrote: > >> >> I didn't close the ticket though as >> >> def f(unsigned short i): pass >> >> ...will get "int" as the description. >> > >> Ups! My fault, I'm working right now in a fix for this.... > >Please have a look at PyrexTypes.py first though, and make sure that you >don't duplicate code -- I think there's already code for making Cython >representations of types. (However you need to move your transform to >after AnalyseDeclarations and inspect funcnode.local_scope.arg_entries >to access the type objects) > >Dag Sverre >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Fri Sep 19 19:02:54 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 19 Sep 2008 14:02:54 -0300 Subject: [Cython] PATCH: embed signatures In-Reply-To: <3304692645.446586@smtp.netcom.no> References: <3304692645.446586@smtp.netcom.no> Message-ID: OK, Dag. For the moment, apply the attached patch. If fix the rendering of basic C type modifiers in argument lists and return type of cpdef functions. I'll take a look next week on the new way you are suggesting. BTW, the Epydoc monkeypatching in Tools/cython-epydoc.py is really naive. I'm not good at writing regular expresions. That tool needs a better regex for argument lists, at least if we want Epydoc still be able to parse the format in core Python builtin functions/methods. On Fri, Sep 19, 2008 at 1:10 PM, Dag Sverre Seljebotn wrote: > Have some time for more details while waiting for my bus...: > > - When moving the transform to after AnalyseDeclarations, you must modify funcnode.py_func.doc as well as funcnode.doc for "cpdef" functions. > > - Rather than inspecting the syntax tree nodes for the arguments, funcnode.local_scope.arg_entries can be iterated. It contains a quite clean view of the arguments (should lean to fewer special cases in your code). > > Got to go. > > Dag Sverre Seljebotn > -----Original Message----- > From: Dag Sverre Seljebotn > Date: Friday, Sep 19, 2008 5:33 pm > Subject: Re: [Cython] PATCH: embed signatures > To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > > Lisandro Dalcin wrote: >> On Fri, Sep 19, 2008 at 11:34 AM, Dag Sverre Seljebotn >> wrote: >> >>> >>> I didn't close the ticket though as >>> >>> def f(unsigned short i): pass >>> >>> ...will get "int" as the description. >>> >> >>> Ups! My fault, I'm working right now in a fix for this.... >> >>Please have a look at PyrexTypes.py first though, and make sure that you >>don't duplicate code -- I think there's already code for making Cython >>representations of types. (However you need to move your transform to >>after AnalyseDeclarations and inspect funcnode.local_scope.arg_entries >>to access the type objects) >> >>Dag Sverre >>_______________________________________________ >>Cython-dev mailing list >>Cython-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/cython-dev >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: embedsignature2.diff Type: text/x-patch Size: 5651 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080919/d2e0e27f/attachment.bin From dagss at student.matnat.uio.no Sat Sep 20 13:45:58 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 20 Sep 2008 13:45:58 +0200 Subject: [Cython] PATCH: embed signatures In-Reply-To: References: Message-ID: <48D4E276.9030600@student.matnat.uio.no> Lisandro Dalcin wrote: > * This do not play well with '--embed-positions' global option For the record (not saying I expect you to), I believe the way to fix this is to pull out the embed_position stuff (from the top of Nodes.py) and stick it into the same transform you just created -- that should be rather trivial (and opens up the way for making things more consistent, i.e. make "embedpositions" a compiler directive as well and just have --embed-positions be an alias for it). The transform could be named DocstringTransform or something like that to keep the number of transforms down. -- Dag Sverre From robertwb at math.washington.edu Sat Sep 20 19:17:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 20 Sep 2008 10:17:07 -0700 Subject: [Cython] PATCH: embed signatures In-Reply-To: <48D4E276.9030600@student.matnat.uio.no> References: <48D4E276.9030600@student.matnat.uio.no> Message-ID: <3BF18D32-1ABD-480B-8212-7DDF8A333786@math.washington.edu> On Sep 20, 2008, at 4:45 AM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: > >> * This do not play well with '--embed-positions' global option > > For the record (not saying I expect you to), I believe the way to fix > this is to pull out the embed_position stuff (from the top of > Nodes.py) > and stick it into the same transform you just created -- that > should be > rather trivial (and opens up the way for making things more > consistent, > i.e. make "embedpositions" a compiler directive as well and just have > --embed-positions be an alias for it). That's exactly what I was going to say. Sage. uses this option to make it possible to view the source in .pyx files via introspection (as in ipython). > The transform could be named DocstringTransform or something like that > to keep the number of transforms down. +1 - Robert From robertwb at math.washington.edu Sat Sep 20 19:40:24 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 20 Sep 2008 10:40:24 -0700 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <48D2775B.5000306@student.matnat.uio.no> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> <48D0DC33.7000200@student.matnat.uio.no> <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> <48D2775B.5000306@student.matnat.uio.no> Message-ID: <90A37F84-5353-4B1B-920A-3CBD3C1E0726@math.washington.edu> On Sep 18, 2008, at 8:44 AM, Dag Sverre Seljebotn wrote: > Marcin Cie?lik wrote: >> Hello, >> >>> Nice. Will you contribute it to SciPy/any other library? >> >> I'd love to. It will find it's way into my zenpdb: >> http://code.google.com/p/zenpdb/ which is the reason i developed >> it. I >> do not know if projects like scipy/biopython accept cython code. >> Biopython has a very fresh (not yet released) implementation of >> kd-trees in C, but without knn look-up. further i've been told that >> the maker of h-cluste/scipy-cluster is involved in some big effort to >> create a partitioning / clustering + nearest neighbor look-up / >> classification C-library with bindings to everything. Is there any >> scientific cython code besides sage? > > SciPy had a longish Cython thread some time ago (June-ish) though I > never checked back to see the results. I gave a 2 hour talk at SciPy 2008 and the audience was very enthusiastic about it. I'm not sure how much Cython code is used in the SciPy codebase itself, but lots of people are using it on their own projects. >>> cdef np.ndarray[unsigned int, mode="c"] ridx = np.empty((k,), >>> dtype='I') >>> ... >>> for 0 <= i < k: >>> ridx[i] = self.kdpnts[idx[i]].index >>> return (ridx, ...) >> >> It looks fantastic, I think the direction of avoiding the PyArray_* >> calls is the right one. > > Note that the only new thing is mode="c"; you can do this with strided > in the latest release. (np.ndarray[...] can be assigned any numpy > array > objects, also those you create yourself). > > BTW you need to import numpy as a Python module as well: > > import numpy as np > > (for calling np.empty the cimport is not enough) > >> This should not be a problem since most of the time the algorithm >> spends in qsort anyway. I don't know why but recursion overhead seems >> quite high for cython generated code. The resursive version of qsort >> was about 2x slower. The numpy qsort code (in C) is actually ~60% >> faster than mine >> (http://svn.scipy.org/svn/numpy/trunk/numpy/core/src/ >> _sortmodule.c.src), >> but using it would require to make qsort a cpdef function (and slice >> arrays / concatenate ) which i assumed would be slower. I think that > > cpdef is not slower in itself (it uses a cdef calling from other > Cython > code), but when passing references out NumPy it would be horrible, > yes. > > This is a long-standing concern of mine: There's no fast and > convenient > standard algorithmic library to use with Cython code, without going > into > Python land. Good wrappers for C++ and STL (and better C++ template > support) would get us somewhere. Then you could just call C++ qsort. This would be very nice. > Recursion overhead *should* be almost the same as C in Cython (and if > not we need to find a mechanism, or even compiler directive, to fix > it). > You could try to have a look at the generated C (use the -a switch to > Cython!) and see if there's any slow operations perhaps? Yes, if you have some code that is significantly slower than the equivalent code in C, please send it to us. As Dag said, there shouldn't be additional overhead. Have you tried using the -a option to investigate? - Robert From prabhu at aero.iitb.ac.in Sat Sep 20 20:39:15 2008 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Sun, 21 Sep 2008 00:09:15 +0530 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <90A37F84-5353-4B1B-920A-3CBD3C1E0726@math.washington.edu> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> <48D0DC33.7000200@student.matnat.uio.no> <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> <48D2775B.5000306@student.matnat.uio.no> <90A37F84-5353-4B1B-920A-3CBD3C1E0726@math.washington.edu> Message-ID: <48D54353.5000400@aero.iitb.ac.in> Robert Bradshaw wrote: > On Sep 18, 2008, at 8:44 AM, Dag Sverre Seljebotn wrote: > >> SciPy had a longish Cython thread some time ago (June-ish) though I >> never checked back to see the results. > > I gave a 2 hour talk at SciPy 2008 and the audience was very > enthusiastic about it. I'm not sure how much Cython code is used in That was a very nice tutorial! Thanks! > the SciPy codebase itself, but lots of people are using it on their > own projects. I was part of that thread in April and *just* finished a blog post of a simple comparison between Python, Cython, C++ and D for a use case I was interested in during the discussion. I'm happy to say that Cython won the competition! Thanks for the great work on Cython! Here is the blog in case it is of interest: http://prabhuramachandran.blogspot.com/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig.html cheers, prabhu From ondrej at certik.cz Sun Sep 21 01:02:11 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 21 Sep 2008 01:02:11 +0200 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <48D54353.5000400@aero.iitb.ac.in> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> <48D0DC33.7000200@student.matnat.uio.no> <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> <48D2775B.5000306@student.matnat.uio.no> <90A37F84-5353-4B1B-920A-3CBD3C1E0726@math.washington.edu> <48D54353.5000400@aero.iitb.ac.in> Message-ID: <85b5c3130809201602i846453ejef62861b0b98e28@mail.gmail.com> On Sat, Sep 20, 2008 at 8:39 PM, Prabhu Ramachandran wrote: > Robert Bradshaw wrote: >> On Sep 18, 2008, at 8:44 AM, Dag Sverre Seljebotn wrote: >> >>> SciPy had a longish Cython thread some time ago (June-ish) though I >>> never checked back to see the results. >> >> I gave a 2 hour talk at SciPy 2008 and the audience was very >> enthusiastic about it. I'm not sure how much Cython code is used in > > That was a very nice tutorial! Thanks! > >> the SciPy codebase itself, but lots of people are using it on their >> own projects. > > I was part of that thread in April and *just* finished a blog post of a > simple comparison between Python, Cython, C++ and D for a use case I was > interested in during the discussion. I'm happy to say that Cython won > the competition! Thanks for the great work on Cython! I am glad you changed your mind since April and do not oppose Cython anymore. :)) > Here is the blog in case it is of interest: > > http://prabhuramachandran.blogspot.com/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig.html Nice post. Ondrej From prabhu at aero.iitb.ac.in Sun Sep 21 10:55:00 2008 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Sun, 21 Sep 2008 14:25:00 +0530 Subject: [Cython] NumPy output works on 32bit not on 64bit In-Reply-To: <85b5c3130809201602i846453ejef62861b0b98e28@mail.gmail.com> References: <3f9ed1a70809131738h65747009s6297da3282d25537@mail.gmail.com> <48CCE45A.1040703@student.matnat.uio.no> <48CCE4B0.8060901@student.matnat.uio.no> <3f9ed1a70809140732x4d54dad2g215f3cce52f90c98@mail.gmail.com> <48D0DC33.7000200@student.matnat.uio.no> <3f9ed1a70809180747o6929391gc064f3fc64f49f10@mail.gmail.com> <48D2775B.5000306@student.matnat.uio.no> <90A37F84-5353-4B1B-920A-3CBD3C1E0726@math.washington.edu> <48D54353.5000400@aero.iitb.ac.in> <85b5c3130809201602i846453ejef62861b0b98e28@mail.gmail.com> Message-ID: <48D60BE4.2090304@aero.iitb.ac.in> Ondrej Certik wrote: > I am glad you changed your mind since April and do not oppose Cython > anymore. :)) Well, I never opposed the use of Cython. I actually use Pyrex in TVTK. I was only concerned about the particular use case I mentioned there and thought wrongly that it would not be suitable. At the time no one proved me wrong with a real example. It took a while for me to get around to doing it myself. Better late than never. :) >> Here is the blog in case it is of interest: >> >> http://prabhuramachandran.blogspot.com/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig.html > > Nice post. Thanks! regards, prabhu From aaron.devore at gmail.com Mon Sep 22 01:39:29 2008 From: aaron.devore at gmail.com (Aaron DeVore) Date: Sun, 21 Sep 2008 16:39:29 -0700 Subject: [Cython] Speed of IsInstance vs TypeCheck Message-ID: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> I'm working on a Cython-based library where I need to check the type/class of objects very, very often. Depending on the data that is fed in and the calls the developer makes I may have to do hundreds of thousands of checks. Because of that any significant difference between PyObject_IsInstance and PyObject_TypeCheck is important. Via grep I found that typecheck seems to be disabled in Cython. I couldn't find any documentation on why that is. That brings up three questions: 1) Is there a significant speed difference? "Significant difference" for me is anything above 20% 2) If there is a speed difference how do I use PyObject_TypeCheck? It was never quite clear to me from the documentation that I could find. 3) Why was typecheck removed/disabled? -Aaron DeVore -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080921/67f7d2db/attachment.htm From lists at cheimes.de Mon Sep 22 11:46:50 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 22 Sep 2008 11:46:50 +0200 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> Message-ID: Aaron DeVore wrote: > I'm working on a Cython-based library where I need to check the type/class > of objects very, very often. Depending on the data that is fed in and the > calls the developer makes I may have to do hundreds of thousands of checks. > Because of that any significant difference between PyObject_IsInstance and > PyObject_TypeCheck is important. [...] Do you want to check for builtin types like PyString, PyUnicode etc. or subclasses? Python has super fast checks for builtin types, for example PyString_Check(obj) or PyString_CheckExact(obj). PyObject_TypeCheck is a macro while PyObject_IsInstance is a recursive function that does much more work than PyObject_TypeCheck. Therefore PyObject_TypeCheck should be a bit faster than PyObject_IsInstance. By the way the latter is invoked by isinstance(), too. Christian From dalcinl at gmail.com Mon Sep 22 15:36:40 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 22 Sep 2008 10:36:40 -0300 Subject: [Cython] PATCH: embed signatures In-Reply-To: <3BF18D32-1ABD-480B-8212-7DDF8A333786@math.washington.edu> References: <48D4E276.9030600@student.matnat.uio.no> <3BF18D32-1ABD-480B-8212-7DDF8A333786@math.washington.edu> Message-ID: On Sat, Sep 20, 2008 at 2:17 PM, Robert Bradshaw wrote: > > That's exactly what I was going to say. Sage. uses this option to > make it possible to view the source in .pyx files via introspection > (as in ipython). > Robert, are you using some special hackery for instrospecting pyx files inside IPython. I never could make it work!. Of course, I want this to work after you 'install' a package. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Mon Sep 22 19:10:46 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 22 Sep 2008 10:10:46 -0700 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> Message-ID: On Sep 22, 2008, at 2:46 AM, Christian Heimes wrote: > Aaron DeVore wrote: >> I'm working on a Cython-based library where I need to check the >> type/class >> of objects very, very often. Depending on the data that is fed in >> and the >> calls the developer makes I may have to do hundreds of thousands >> of checks. >> Because of that any significant difference between >> PyObject_IsInstance and >> PyObject_TypeCheck is important. > > [...] > > Do you want to check for builtin types like PyString, PyUnicode > etc. or > subclasses? Python has super fast checks for builtin types, for > example > PyString_Check(obj) or PyString_CheckExact(obj). > > PyObject_TypeCheck is a macro while PyObject_IsInstance is a recursive > function that does much more work than PyObject_TypeCheck. Therefore > PyObject_TypeCheck should be a bit faster than PyObject_IsInstance. By > the way the latter is invoked by isinstance(), too. Note also that PyObject_TypeCheck checks for types, and normal python classes aren't types. - Robert From dagss at student.matnat.uio.no Mon Sep 22 21:58:01 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 22 Sep 2008 21:58:01 +0200 Subject: [Cython] PATCH: embed signatures In-Reply-To: References: <3304692645.446586@smtp.netcom.no> Message-ID: <48D7F8C9.6070003@student.matnat.uio.no> Lisandro Dalcin wrote: > OK, Dag. For the moment, apply the attached patch. If fix the > rendering of basic C type modifiers in argument lists and return type > of cpdef functions. I'll take a look next week on the new way you are > suggesting. BTW, when you get time it would be good if you could add a paragraph in the table here: http://wiki.cython.org/enhancements/compilerdirectives (It's not a great place to put documentation but I suppose that whenever somebody (perhaps me) gets around to put it in the real docs (whatever those are) it will be pulled from there.) -- Dag Sverre From dagss at student.matnat.uio.no Mon Sep 22 22:38:39 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 22 Sep 2008 22:38:39 +0200 Subject: [Cython] Simple nonecheck directive in Message-ID: <48D8024F.4010408@student.matnat.uio.no> In cython-devel there's now (experimental) support for a "nonecheck" directive. It emulates the Python AttributeError on typed cdef class variables set to None: ===== """ >>> getattr_(None) Traceback (most recent call last): ... AttributeError: 'NoneType' object has no attribute 'a' """ cimport cython cdef class MyClass: cdef int a ... @cython.nonecheck(True) def getattr_(MyClass var): print var.a ===== It will create inefficient code, inserting a check on any and all attribute accesses. Haven't tested performance, any benchmarks welcome. -- Dag Sverre From dagss at student.matnat.uio.no Mon Sep 22 22:39:02 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 22 Sep 2008 22:39:02 +0200 Subject: [Cython] A small change in compiler directive implementation Message-ID: <48D80266.6060701@student.matnat.uio.no> Just a note that I've changed some internals with compiler directives. Before each node would get an "options" attribute in a transform that ran pretty early. This was getting increasingly cumbersome to deal with (as transforms needed to make sure options was copied over to new nodes etc.). So now I've created a CompilerDirectivesNode that sit in the tree instead. Each recursive process then stores the directives these sets in different places: - env.directives - code.globalstate.directives - For CythonTransform descendants, self.current_directives is automatically updated as long as you leave the default CompilerDirectivesNode visit alone. If anyone protests I'll take it upon me to back it out. -- Dag Sverre From collette at physics.ucla.edu Tue Sep 23 00:13:19 2008 From: collette at physics.ucla.edu (Andrew Collette) Date: Mon, 22 Sep 2008 15:13:19 -0700 Subject: [Cython] Issues porting to Cython Message-ID: <1222121599.14956.21.camel@tachyon-laptop> Hi, I am in the midst of porting a project to Cython, and have run into a few issues. I had been using Pyrex 0.9.8.4 and am trying the latest public release of Cython (0.9.8.1.1). I'm interfacing to the HDF5 C library, which has a huge API. Since I heard Cython (unlike Pyrex) supports "cimport *", I wanted consolidate the whole thing into "hdf5.pxd" and simply cimport everything into each module I'm building. However, this doesn't seem to import typedefs: # file "ext.pxd": cdef extern from "something.h": ctypedef int INT # file "implementation.pyx": from ext cimport * def foo(): cdef INT q = 1 print q This fails with a syntax error. If I replace the cimport line with "from ext cimport INT" it works fine. Is this a bug or an intentional limitation (or am I using it wrong)? The second issue is that Cython.Compiler.Main.compile_multiple doesn't work if you use either the "recursive" or "timestamps" options; I get a NameError referring to what seems to be a missing global variable "context" inside Cython itself: /usr/lib/python2.5/site-packages/Cython/Compiler/Main.py in compile_multiple(sources, options) 639 # Compiling multiple sources in one context doesn't quite 640 # work properly yet. --> 641 if not timestamps or context.c_file_out_of_date(source): 642 if verbose: 643 sys.stderr.write("Compiling %s\n" % source) : global name 'context' is not defined I'd like to eventually use this if possible because it lets me run Cython from a setup script independent of the actual C build process. Will this be fixed in a later version of Cython? Finally I noticed that Cython doesn't have the new Pyrex "typecheck" builtin. I've switched back to "isinstance" for now. Is there/will there be a Cython equivalent? Other than that everything works! Thanks in particular for the decorator support, which is the main reason I'm switching. Andrew Collette From greg.ewing at canterbury.ac.nz Tue Sep 23 01:12:32 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Sep 2008 11:12:32 +1200 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> Message-ID: <48D82660.1040105@canterbury.ac.nz> Robert Bradshaw wrote: > Note also that PyObject_TypeCheck checks for types, and normal python > classes aren't types. Old-style classes aren't types, but nowadays I'd say that new-style classes are regarded as equally "normal", perhaps more so. Anyway, this makes no difference for testing against builtin or extension types, which are always new-style. -- Greg From aaron.devore at gmail.com Tue Sep 23 01:26:11 2008 From: aaron.devore at gmail.com (Aaron DeVore) Date: Mon, 22 Sep 2008 16:26:11 -0700 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> Message-ID: <2ead2fb0809221626k7fa5a7d4x18c9ac4b45e7243a@mail.gmail.com> On Mon, Sep 22, 2008 at 1:46 AM, Christian Heimes wrote: > Do you want to check for builtin types like PyString, PyUnicode etc. or > subclasses? Python has super fast checks for builtin types, for example > PyString_Check(obj) or PyString_CheckExact(obj). > > PyObject_TypeCheck is a macro while PyObject_IsInstance is a recursive > function that does much more work than PyObject_TypeCheck. Therefore > PyObject_TypeCheck should be a bit faster than PyObject_IsInstance. By > the way the latter is invoked by isinstance(), too. > > Christian In most cases I'm checking for whatever is generated by cdef class Foo: ... I'm not sure if that would be a type or a subclass. There are a few places where I am do checks for dict, str, unicode, and list. I have absolutely no idea how that would affect things, though. -Aaron -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080922/c63c1698/attachment.htm From greg.ewing at canterbury.ac.nz Tue Sep 23 01:22:01 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Sep 2008 11:22:01 +1200 Subject: [Cython] Issues porting to Cython In-Reply-To: <1222121599.14956.21.camel@tachyon-laptop> References: <1222121599.14956.21.camel@tachyon-laptop> Message-ID: <48D82899.9050902@canterbury.ac.nz> Andrew Collette wrote: > Since I heard Cython (unlike Pyrex) > supports "cimport *", I wanted consolidate the whole thing into > "hdf5.pxd" and simply cimport everything into each module I'm building. > However, this doesn't seem to import typedefs: The reason Pyrex doesn't support 'cimport *' is because it needs to know at parse time which names are potentially type names. If Cython hasn't done anything about that problem, its support for cimport * may not work properly. > 639 # Compiling multiple sources in one context doesn't > quite > 640 # work properly yet. The latest Pyrex supports multiple compilation properly now, but this may not have made it into Cython yet. -- Greg From robertwb at math.washington.edu Tue Sep 23 01:35:11 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 22 Sep 2008 16:35:11 -0700 Subject: [Cython] Issues porting to Cython In-Reply-To: <1222121599.14956.21.camel@tachyon-laptop> References: <1222121599.14956.21.camel@tachyon-laptop> Message-ID: <9D7C2A64-0D58-4B73-A36D-558A6E05C3B1@math.washington.edu> On Sep 22, 2008, at 3:13 PM, Andrew Collette wrote: > Hi, > > I am in the midst of porting a project to Cython, and have run into a > few issues. I had been using Pyrex 0.9.8.4 and am trying the latest > public release of Cython (0.9.8.1.1). I'm interfacing to the HDF5 C > library, which has a huge API. Since I heard Cython (unlike Pyrex) > supports "cimport *", I wanted consolidate the whole thing into > "hdf5.pxd" and simply cimport everything into each module I'm > building. > However, this doesn't seem to import typedefs: > > # file "ext.pxd": > cdef extern from "something.h": > ctypedef int INT > > # file "implementation.pyx": > from ext cimport * > def foo(): > cdef INT q = 1 > print q > > This fails with a syntax error. If I replace the cimport line with > "from ext cimport INT" it works fine. Is this a bug or an intentional > limitation (or am I using it wrong)? No, this looks like a bug. http://trac.cython.org/cython_trac/ticket/ 73 For now, import your ctypedef types explicitly (functions and classes should cimport fine). > The second issue is that Cython.Compiler.Main.compile_multiple doesn't > work if you use either the "recursive" or "timestamps" options; I > get a > NameError referring to what seems to be a missing global variable > "context" inside Cython itself: > > /usr/lib/python2.5/site-packages/Cython/Compiler/Main.py in > compile_multiple(sources, options) > 639 # Compiling multiple sources in one context > doesn't > quite > 640 # work properly yet. > --> 641 if not timestamps or > context.c_file_out_of_date(source): > 642 if verbose: > 643 sys.stderr.write("Compiling %s\n" % > source) > > : global name 'context' is not defined > > I'd like to eventually use this if possible because it lets me run > Cython from a setup script independent of the actual C build process. > Will this be fixed in a later version of Cython? Yes, I certainly hope so. I don't think many people have used this yet, so I'm sure it's not robust. > Finally I noticed that Cython doesn't have the new Pyrex "typecheck" > builtin. I've switched back to "isinstance" for now. Is there/will > there be a Cython equivalent? Probably not. One of the goals of Cython is to reduce the need for special syntax, trying to stay as close to Python as possible. The only problem with isinstance is that the type may lie via its __class__ attribute, but if the second argument is an extension class then it should to a type test. http://article.gmane.org/gmane.comp.python.cython.devel/1434 > Other than that everything works! Thanks in particular for the > decorator support, which is the main reason I'm switching. Great. Let us know if you run into anything else. - Robert From greg.ewing at canterbury.ac.nz Tue Sep 23 01:30:25 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Sep 2008 11:30:25 +1200 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: <2ead2fb0809221626k7fa5a7d4x18c9ac4b45e7243a@mail.gmail.com> References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> <2ead2fb0809221626k7fa5a7d4x18c9ac4b45e7243a@mail.gmail.com> Message-ID: <48D82A91.9010903@canterbury.ac.nz> Aaron DeVore wrote: > In most cases I'm checking for whatever is generated by cdef class Foo: > ... I'm not sure if that would be a type or a subclass. It's a type, aka new-style class, so both PyObject_TypeCheck and PyObject_IsInstance will work equally well, and they will both accept either that exact type or a subclass of it. PyObject_TypeCheck is probably a bit faster, since it doesn't have to worry about dealing with old-style classes and instances. -- Greg From greg.ewing at canterbury.ac.nz Tue Sep 23 01:35:02 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 23 Sep 2008 11:35:02 +1200 Subject: [Cython] Issues porting to Cython In-Reply-To: <9D7C2A64-0D58-4B73-A36D-558A6E05C3B1@math.washington.edu> References: <1222121599.14956.21.camel@tachyon-laptop> <9D7C2A64-0D58-4B73-A36D-558A6E05C3B1@math.washington.edu> Message-ID: <48D82BA6.6010101@canterbury.ac.nz> Robert Bradshaw wrote: > The only problem with isinstance is that the type may lie via its > __class__ attribute, but if the second argument is an extension class > then it should to a type test. However, if there's a noticeable speed difference between PyObject_IsInstance and PyObject_TypeCheck, it may still be desirable to have both available. -- Greg From aaron.devore at gmail.com Tue Sep 23 02:53:06 2008 From: aaron.devore at gmail.com (Aaron DeVore) Date: Mon, 22 Sep 2008 17:53:06 -0700 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: <48D82A91.9010903@canterbury.ac.nz> References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> <2ead2fb0809221626k7fa5a7d4x18c9ac4b45e7243a@mail.gmail.com> <48D82A91.9010903@canterbury.ac.nz> Message-ID: <2ead2fb0809221753n3c61d1a1w45d544d6616dd036@mail.gmail.com> So how do I go about getting PyObject_TypeCheck working? -Aaron On Mon, Sep 22, 2008 at 4:30 PM, Greg Ewing wrote: > Aaron DeVore wrote: > > > In most cases I'm checking for whatever is generated by cdef class Foo: > > ... I'm not sure if that would be a type or a subclass. > > It's a type, aka new-style class, so both PyObject_TypeCheck > and PyObject_IsInstance will work equally well, and they > will both accept either that exact type or a subclass of > it. > > PyObject_TypeCheck is probably a bit faster, since it doesn't > have to worry about dealing with old-style classes and > instances. > > -- > Greg > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080922/433c5080/attachment.htm From dalcinl at gmail.com Tue Sep 23 03:26:09 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 22 Sep 2008 22:26:09 -0300 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: <2ead2fb0809221753n3c61d1a1w45d544d6616dd036@mail.gmail.com> References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> <2ead2fb0809221626k7fa5a7d4x18c9ac4b45e7243a@mail.gmail.com> <48D82A91.9010903@canterbury.ac.nz> <2ead2fb0809221753n3c61d1a1w45d544d6616dd036@mail.gmail.com> Message-ID: I'm on Greg's side. As there are real performance issues, 'typecheck' should be supported in Cython. My use cases are mainly (though actually a small buch of calls) for checking against cdef classes. Aaron, you could do the following: Add the snippet below in a 'typecheck.pxi' file, then include it with 'include "typecheck.pxi"' at the beginning of every .pyx file you want to have 'typecheck? available. If Cython ever get built-in support for typecheck, then you should just remove the pxi file and the places where it is included. I just wrote this for you, but also just included it in my petsc4py project (I had typecheck, but actually implemented calling PyObject_IsInstance). cdef extern from "Python.h": ctypedef struct PyTypeObject int PyObject_TypeCheck(object, PyTypeObject*) cdef inline bint typecheck(object ob, object tp): return PyObject_TypeCheck(ob, tp) Let me know if this hackery worked for you. Regards, On Mon, Sep 22, 2008 at 9:53 PM, Aaron DeVore wrote: > So how do I go about getting PyObject_TypeCheck working? > > -Aaron > > On Mon, Sep 22, 2008 at 4:30 PM, Greg Ewing > wrote: >> >> Aaron DeVore wrote: >> >> > In most cases I'm checking for whatever is generated by cdef class Foo: >> > ... I'm not sure if that would be a type or a subclass. >> >> It's a type, aka new-style class, so both PyObject_TypeCheck >> and PyObject_IsInstance will work equally well, and they >> will both accept either that exact type or a subclass of >> it. >> >> PyObject_TypeCheck is probably a bit faster, since it doesn't >> have to worry about dealing with old-style classes and >> instances. >> >> -- >> Greg >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From collette at physics.ucla.edu Tue Sep 23 05:54:04 2008 From: collette at physics.ucla.edu (Andrew Collette) Date: Mon, 22 Sep 2008 20:54:04 -0700 Subject: [Cython] Issues porting to Cython In-Reply-To: <9D7C2A64-0D58-4B73-A36D-558A6E05C3B1@math.washington.edu> References: <1222121599.14956.21.camel@tachyon-laptop> <9D7C2A64-0D58-4B73-A36D-558A6E05C3B1@math.washington.edu> Message-ID: <1222142045.9872.19.camel@tachyon-laptop> > No, this looks like a bug. http://trac.cython.org/cython_trac/ticket/ > 73 For now, import your ctypedef types explicitly (functions and > classes should cimport fine). Thanks! Since hdf5.pxd is 1154 lines and is purely external definitions, I'm getting around this by just including it in my module .pxd files. This seems to work for now. Andrew From robertwb at math.washington.edu Tue Sep 23 08:49:44 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 22 Sep 2008 23:49:44 -0700 Subject: [Cython] Speed of IsInstance vs TypeCheck In-Reply-To: References: <2ead2fb0809211639y18fe7778r771b13f0fc44fe68@mail.gmail.com> <2ead2fb0809221626k7fa5a7d4x18c9ac4b45e7243a@mail.gmail.com> <48D82A91.9010903@canterbury.ac.nz> <2ead2fb0809221753n3c61d1a1w45d544d6616dd036@mail.gmail.com> Message-ID: On Sep 22, 2008, at 6:26 PM, Lisandro Dalcin wrote: > I'm on Greg's side. As there are real performance issues, 'typecheck' > should be supported in Cython. My use cases are mainly (though > actually a small buch of calls) for checking against cdef classes. How about http://trac.cython.org/cython_trac/ticket/74 ? > Aaron, you could do the following: Add the snippet below in a > 'typecheck.pxi' file, then include it with 'include "typecheck.pxi"' > at the beginning of every .pyx file you want to have 'typecheck? > available. If Cython ever get built-in support for typecheck, then you > should just remove the pxi file and the places where it is included. I > just wrote this for you, but also just included it in my petsc4py > project (I had typecheck, but actually implemented calling > PyObject_IsInstance). > > > cdef extern from "Python.h": > ctypedef struct PyTypeObject > int PyObject_TypeCheck(object, PyTypeObject*) The above is actually already defined, you can simply write "from python_object import PyTypeObject, PyObject_TypeCheck". > cdef inline bint typecheck(object ob, object tp): > return PyObject_TypeCheck(ob, tp) > > Let me know if this hackery worked for you. > > Regards, > > On Mon, Sep 22, 2008 at 9:53 PM, Aaron DeVore > wrote: >> So how do I go about getting PyObject_TypeCheck working? >> >> -Aaron >> >> On Mon, Sep 22, 2008 at 4:30 PM, Greg Ewing >> >> wrote: >>> >>> Aaron DeVore wrote: >>> >>>> In most cases I'm checking for whatever is generated by cdef >>>> class Foo: >>>> ... I'm not sure if that would be a type or a subclass. >>> >>> It's a type, aka new-style class, so both PyObject_TypeCheck >>> and PyObject_IsInstance will work equally well, and they >>> will both accept either that exact type or a subclass of >>> it. >>> >>> PyObject_TypeCheck is probably a bit faster, since it doesn't >>> have to worry about dealing with old-style classes and >>> instances. >>> >>> -- >>> Greg >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >> >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Tue Sep 23 09:41:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Sep 2008 00:41:42 -0700 Subject: [Cython] Builtin declarations + should BufferException be backported? In-Reply-To: <48D0E2B4.5010001@student.matnat.uio.no> References: <48D0E2B4.5010001@student.matnat.uio.no> Message-ID: <67A49D67-A79C-45EB-8F03-DA05D048ED9F@math.washington.edu> On Sep 17, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote: > According to the buffer PEP, __getbuffer__ should raise a BufferError > under certain circumstances. This is not present in older versions of > Python. Backport/emulate or not? Any good ideas for how it should > happen? (I'm blank when it comes to both builtins and creating > exception > types.) One option could be to simply make BufferError be a ValueError > in Python 2.5-; still, the main question I guess is whether we import > BufferError into the Cython builtin scope. Yes, I think we should. Like you said, the easy way is to #define it to be ValueError for Python 2.5-. The other way would be to fill it in during initialization with a manually created class (see ExprNodes.ClassNode). Care must be taken to have a globally unique BufferError, so perhaps it should be injected into (and an attempt to looked up from) the __builtin__ module. > Slightly related, look at this code in Symtab.py: > def declare_builtin(self, name, pos): > if not hasattr(__builtin__, name): > > If I'm not entirely wrong, this means that BufferError (and other new > builtin symbols) will be available only if you run Cython under Python > 2.6+. I find the fact that results of a compilation depends on the > Python version used for running Cython somewhat disturbing. Should > perhaps a literal list of accepted Cython builtins be present in > Symtab.py instead? (Or couple it to Builtins.py somehow?) I think maintaining such a list would be less elegant than looking them up at runtime, especially as it is fairly stable and monotonically increasing. If one wants to use the code with, say, Python 2.3, then one simply has to avoid using newer symbols in ones code (just as with Python), and it is easy to check (just compile with Python 2.3). This allows people to write code with newer builtins if one wants without having to maintain a list of when every symbol was introduced. - Robert From dagss at student.matnat.uio.no Tue Sep 23 10:08:32 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 23 Sep 2008 10:08:32 +0200 Subject: [Cython] Builtin declarations + should BufferException be backported? In-Reply-To: <67A49D67-A79C-45EB-8F03-DA05D048ED9F@math.washington.edu> References: <48D0E2B4.5010001@student.matnat.uio.no> <67A49D67-A79C-45EB-8F03-DA05D048ED9F@math.washington.edu> Message-ID: <48D8A400.7060306@student.matnat.uio.no> Robert Bradshaw wrote: > On Sep 17, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote: > > >> Slightly related, look at this code in Symtab.py: >> def declare_builtin(self, name, pos): >> if not hasattr(__builtin__, name): >> >> If I'm not entirely wrong, this means that BufferError (and other new >> builtin symbols) will be available only if you run Cython under Python >> 2.6+. I find the fact that results of a compilation depends on the >> Python version used for running Cython somewhat disturbing. Should >> perhaps a literal list of accepted Cython builtins be present in >> Symtab.py instead? (Or couple it to Builtins.py somehow?) >> > > I think maintaining such a list would be less elegant than looking > them up at runtime, especially as it is fairly stable and > monotonically increasing. If one wants to use the code with, say, > Python 2.3, then one simply has to avoid using newer symbols in ones > code (just as with Python), and it is easy to check (just compile > with Python 2.3). This allows people to write code with newer > builtins if one wants without having to maintain a list of when every > symbol was introduced. > I might have misunderstood some earlier discussions then. So, in effect, one has to run cython.py using an interpreter that has at least the same Python version as the builtins you are using. Well, I can certainly live with that. But I think it kind of contradicts what Stefan has tried to achieve with 3.0 compatability etc. (i.e one then has to use Python 2.6 if one wants to compile Cython code for Python 3.0 using BufferError...) Can't this just be deferred until run-time anyway? I.e. have the list in question be the set of all strings :-) I don't think there's a strong reason we have to have a compile-time error here? Then sys.version_info etc. could be used to create code that runs on multiple targets (and compile, using Cython, under multiple targets!) just like in Python. Dag Sverre From robertwb at math.washington.edu Tue Sep 23 10:29:47 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Sep 2008 01:29:47 -0700 Subject: [Cython] Builtin declarations + should BufferException be backported? In-Reply-To: <48D8A400.7060306@student.matnat.uio.no> References: <48D0E2B4.5010001@student.matnat.uio.no> <67A49D67-A79C-45EB-8F03-DA05D048ED9F@math.washington.edu> <48D8A400.7060306@student.matnat.uio.no> Message-ID: <09586463-9800-4AD7-898B-9C07913E740E@math.washington.edu> On Sep 23, 2008, at 1:08 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Sep 17, 2008, at 3:57 AM, Dag Sverre Seljebotn wrote: >> >> >>> Slightly related, look at this code in Symtab.py: >>> def declare_builtin(self, name, pos): >>> if not hasattr(__builtin__, name): >>> >>> If I'm not entirely wrong, this means that BufferError (and other >>> new >>> builtin symbols) will be available only if you run Cython under >>> Python >>> 2.6+. I find the fact that results of a compilation depends on the >>> Python version used for running Cython somewhat disturbing. Should >>> perhaps a literal list of accepted Cython builtins be present in >>> Symtab.py instead? (Or couple it to Builtins.py somehow?) >>> >> >> I think maintaining such a list would be less elegant than looking >> them up at runtime, especially as it is fairly stable and >> monotonically increasing. If one wants to use the code with, say, >> Python 2.3, then one simply has to avoid using newer symbols in ones >> code (just as with Python), and it is easy to check (just compile >> with Python 2.3). This allows people to write code with newer >> builtins if one wants without having to maintain a list of when every >> symbol was introduced. >> > I might have misunderstood some earlier discussions then. So, in > effect, > one has to run cython.py using an interpreter that has at least the > same > Python version as the builtins you are using. More precisely, it is possible to get a runtime error rather than a compile time error if there is a mismatch. For example, if one uses the reversed builtin from 2.4, then it will complain if you compile with 2.3, and if you compile with 2.4+ it won't run under 2.3. In this sense it is just as compatible as .py files. > Well, I can certainly live with that. Me too, especially as the set of builtins is rather stable. (At least it was for 2.3-2.6, perhaps someone could comment on how disruptive 3.0 is). > But I think it kind of contradicts > what Stefan has tried to achieve with 3.0 compatability etc. (i.e one > then has to use Python 2.6 if one wants to compile Cython code for > Python 3.0 using BufferError...) Buffers are a special case, because we decided to backport them (mainly at the request of the NumPy folks, who support back to 2.3). > Can't this just be deferred until run-time anyway? I.e. have the > list in > question be the set of all strings :-) I don't think there's a strong > reason we have to have a compile-time error here? Then > sys.version_info > etc. could be used to create code that runs on multiple targets (and > compile, using Cython, under multiple targets!) just like in Python. Yes, we could. This is exactly what Pyrex does--anything that it doesn't recognize is treated as a builtin. But catching this kind of stuff early is very nice, especially as the edit-compile-run cycle is necessarily longer with Cython code than with Python. I find it catches all sorts of typos and other errors way more often then it gets in the way (and other people on this list have said the same--my favorite is someone who complained that Cython wouldn't let him raise ValueErr anymore...). It also makes import * feasible. We could have an option to turn it off, but I don't know that it'd do much good (it'd only be useful if you want to write code for a platform you don't have access to, and even then you could put "from __builtin__ import foo" at the top of the file to manually declare it). - Robert From dalcinl at gmail.com Tue Sep 23 17:51:59 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 23 Sep 2008 12:51:59 -0300 Subject: [Cython] about refactoring signature embeding transform Message-ID: Dag, I was looking at refactoring the signature embeding transform as you suggested. But using the 'arg_entries' in 'local_scope' do not seems to provide all the stuff needed to avoid the special cases I had to handle in my former implementation. Unless I'm missing something, I believe the current approach more or less works fine and I'm not sure that the refactoring will pay too much. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Sep 23 18:20:51 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 23 Sep 2008 13:20:51 -0300 Subject: [Cython] isinstance optimization breaks cdef methods Message-ID: Latests Robert's commit optimizing 'isinstance' call breaked the call to cdef methods. The one line patch below seems to fix the problem, but anyway this need to be reviewed more carefully. Perhaps an unimportant gotcha: the generated code for, let say 'insinstance(None, list)' has two superfluous casts: /* "/tmp/qq.pyx":1 * isinstance(None, list) # <<<<<<<<<<<<<< */ __pyx_1 = PyObject_TypeCheck(Py_None, ((PyTypeObject *)((PyObject*)&PyList_Type))); diff -r 30b7a6c2c7c1 Cython/Compiler/Optimize.py --- a/Cython/Compiler/Optimize.py Mon Sep 22 23:45:08 2008 -0700 +++ b/Cython/Compiler/Optimize.py Tue Sep 23 13:09:36 2008 -0300 @@ -163,7 +163,7 @@ class FinalOptimizePhase(Visitor.CythonT def visit_SimpleCallNode(self, node): self.visitchildren(node) if node.function.type.is_cfunction: - if node.function.name == 'isinstance': + if getattr(node.function, 'name', None) == 'isinstance': type_arg = node.args[1] if type_arg.type.is_builtin_type and type_arg.type.name == 'type': object_module = self.context.find_module('python_object') -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Tue Sep 23 19:47:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Sep 2008 10:47:20 -0700 Subject: [Cython] isinstance optimization breaks cdef methods In-Reply-To: References: Message-ID: <1C400DCB-A22D-4ED2-9A4E-0A4F0ADF5D3A@math.washington.edu> On Sep 23, 2008, at 9:20 AM, Lisandro Dalcin wrote: > Latests Robert's commit optimizing 'isinstance' call breaked the call > to cdef methods. The one line patch below seems to fix the problem, > but anyway this need to be reviewed more carefully. Perhaps an > unimportant gotcha: the generated code for, let say > 'insinstance(None, list)' has two superfluous casts: > > /* "/tmp/qq.pyx":1 > * isinstance(None, list) # <<<<<<<<<<<<<< > */ > __pyx_1 = PyObject_TypeCheck(Py_None, ((PyTypeObject > *)((PyObject*)&PyList_Type))); Yep, I was looking at this last night. There's not a clean way to "undo" the first cast, so I've just left it in as it doesn't hurt. > diff -r 30b7a6c2c7c1 Cython/Compiler/Optimize.py > --- a/Cython/Compiler/Optimize.py Mon Sep 22 23:45:08 2008 -0700 > +++ b/Cython/Compiler/Optimize.py Tue Sep 23 13:09:36 2008 -0300 > @@ -163,7 +163,7 @@ class FinalOptimizePhase(Visitor.CythonT > def visit_SimpleCallNode(self, node): > self.visitchildren(node) > if node.function.type.is_cfunction: > - if node.function.name == 'isinstance': > + if getattr(node.function, 'name', None) == 'isinstance': > type_arg = node.args[1] > if type_arg.type.is_builtin_type and > type_arg.type.name == 'type': > object_module = self.context.find_module > ('python_object') Ah, I see what the issue was. I've committed a fix. All tests pass now. - Robert From dfdeshom at gmail.com Tue Sep 23 20:01:53 2008 From: dfdeshom at gmail.com (didier deshommes) Date: Tue, 23 Sep 2008 14:01:53 -0400 Subject: [Cython] cython license: PSF or Apache? Message-ID: <4db014670809231101m463bf9cenafbdb0c1f59632c6@mail.gmail.com> Hi there, The main page of cython (http://cython.org/#download) mentions that it is released under the Apache license (http://opensource-definition.org/licenses/apache2.0.html) while LICENSE.txt (http://hg.cython.org/cython/file/f4b39b81687b/LICENSE.txt) says cython is released under the PSF license. Which one is it? didier From aaron.devore at gmail.com Tue Sep 23 20:41:45 2008 From: aaron.devore at gmail.com (Aaron DeVore) Date: Tue, 23 Sep 2008 10:41:45 -0800 Subject: [Cython] Backward compatibility for optimized isinstance Message-ID: <2ead2fb0809231141s2aa5a9b5u535f36b6eaa6fef4@mail.gmail.com> I'm really happy with the automatic isinstance -> PyObject_TypeCheck change, but another question comes up. How do I make my code backward compatible? Just having isinstance be slower on older versions (pre 0.9.8.2) would work, but there would still be a performance hit. My first thought is to use Lisandro's hack ( http://codespeak.net/pipermail/cython-dev/2008-September/002358.html) in the immediate future until Cython 0.9.8.2+ is more widely adopted. Does that sound like the best option? -Aaron DeVore -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20080923/f52f9182/attachment.htm From robertwb at math.washington.edu Tue Sep 23 20:44:54 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Sep 2008 11:44:54 -0700 Subject: [Cython] Backward compatibility for optimized isinstance In-Reply-To: <2ead2fb0809231141s2aa5a9b5u535f36b6eaa6fef4@mail.gmail.com> References: <2ead2fb0809231141s2aa5a9b5u535f36b6eaa6fef4@mail.gmail.com> Message-ID: <537263C6-4F80-49A2-9F1B-FF487D8327A6@math.washington.edu> On Sep 23, 2008, at 11:41 AM, Aaron DeVore wrote: > I'm really happy with the automatic isinstance -> > PyObject_TypeCheck change, but another question comes up. How do I > make my code backward compatible? Just having isinstance be slower > on older versions (pre 0.9.8.2) would work, but there would still > be a performance hit. > > My first thought is to use Lisandro's hack (http://codespeak.net/ > pipermail/cython-dev/2008-September/002358.html) in the immediate > future until Cython 0.9.8.2+ is more widely adopted. Does that > sound like the best option? Yes, I think it is. - Robert From robertwb at math.washington.edu Tue Sep 23 20:55:22 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Sep 2008 11:55:22 -0700 Subject: [Cython] cython license: PSF or Apache? In-Reply-To: <4db014670809231101m463bf9cenafbdb0c1f59632c6@mail.gmail.com> References: <4db014670809231101m463bf9cenafbdb0c1f59632c6@mail.gmail.com> Message-ID: <43A61DF6-14D8-48EC-BB70-392EDDAA891F@math.washington.edu> On Sep 23, 2008, at 11:01 AM, didier deshommes wrote: > Hi there, > The main page of cython (http://cython.org/#download) mentions that it > is released under the Apache license > (http://opensource-definition.org/licenses/apache2.0.html) while > LICENSE.txt (http://hg.cython.org/cython/file/f4b39b81687b/ > LICENSE.txt) > says cython is released under the PSF license. Which one is it? > > didier Sorry for the confusion. It is the Apache license, as we found out the PSF license can't be applied to anything but Python itself. If Cython becomes part of the standard Python distribution, then we can change to the PSF. I've changed LICENSE.txt t reflect this. - Robert From dalcinl at gmail.com Tue Sep 23 23:48:52 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 23 Sep 2008 18:48:52 -0300 Subject: [Cython] Backward compatibility for optimized isinstance In-Reply-To: <2ead2fb0809231141s2aa5a9b5u535f36b6eaa6fef4@mail.gmail.com> References: <2ead2fb0809231141s2aa5a9b5u535f36b6eaa6fef4@mail.gmail.com> Message-ID: Anyway, I really believe it is a good idea to ship the generated sources with your release tarballs. Then the users of your releases do not even need to have Cython installed to build your distribution. Cython is gaining more and more features and fixes every week. I'm really happy with this fast enhacement process, then I'm always running on head of cython-devel repo. And this also helps to Cython core developers, as they some extra code outside the Cython test suite being continuouly built and possible regresions being catched. Of course, perhaps the above comments do not fit well with your workflow; in such case, just forget about it. On Tue, Sep 23, 2008 at 3:41 PM, Aaron DeVore wrote: > I'm really happy with the automatic isinstance -> PyObject_TypeCheck change, > but another question comes up. How do I make my code backward compatible? > Just having isinstance be slower on older versions (pre 0.9.8.2) would work, > but there would still be a performance hit. > > My first thought is to use Lisandro's hack > (http://codespeak.net/pipermail/cython-dev/2008-September/002358.html) in > the immediate future until Cython 0.9.8.2+ is more widely adopted. Does that > sound like the best option? > > -Aaron DeVore > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed Sep 24 00:28:40 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 23 Sep 2008 19:28:40 -0300 Subject: [Cython] more comments on isinstance optimization Message-ID: Could we extend the 'builtin_types_table' (if this is actually the place) in 'Cython/Compiler/Builtin.py' in order to have more built-int type to work with Robert's "isinstance" optimization? PyInt_Type is not there, not PyFloat_Type; I'm also using PySlice_Type (it is commented out) in petsc4py (in order to support __setitem__ in vectors and matrices). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From greg.ewing at canterbury.ac.nz Wed Sep 24 02:22:04 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 24 Sep 2008 12:22:04 +1200 Subject: [Cython] isinstance optimization breaks cdef methods In-Reply-To: References: Message-ID: <48D9882C.5040305@canterbury.ac.nz> Lisandro Dalcin wrote: > the generated code for, let say > 'insinstance(None, list)' has two superfluous casts: > > __pyx_1 = PyObject_TypeCheck(Py_None, ((PyTypeObject > *)((PyObject*)&PyList_Type))); That's because PyList_Type is known to have type 'type' (at least Pyrex knows this, I'm assuming Cyton does too). If you declare typecheck() as taking a type for the second argument, rather than object, you shouldn't get those casts. -- Greg From greg.ewing at canterbury.ac.nz Wed Sep 24 02:46:14 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 24 Sep 2008 12:46:14 +1200 Subject: [Cython] more comments on isinstance optimization In-Reply-To: References: Message-ID: <48D98DD6.5000806@canterbury.ac.nz> Lisandro Dalcin wrote: > Could we extend the 'builtin_types_table' ... PyInt_Type is > not there, not PyFloat_Type; In Pyrex, these are in the builtin_constant_table (with type 'type'). The builtin_type_table is only used for types having methods that are treated specially. -- Greg From robertwb at math.washington.edu Wed Sep 24 03:41:53 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Sep 2008 18:41:53 -0700 Subject: [Cython] more comments on isinstance optimization In-Reply-To: <48D98DD6.5000806@canterbury.ac.nz> References: <48D98DD6.5000806@canterbury.ac.nz> Message-ID: On Sep 23, 2008, at 5:46 PM, Greg Ewing wrote: > Lisandro Dalcin wrote: > >> Could we extend the 'builtin_types_table' ... PyInt_Type is >> not there, not PyFloat_Type; > > In Pyrex, these are in the builtin_constant_table (with > type 'type'). The builtin_type_table is only used for > types having methods that are treated specially. I don't see any reason why we wouldn't, as long as they don't interfere with the c types of the same name (though I don't see why they would). - Robert From robertwb at math.washington.edu Wed Sep 24 03:43:26 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 23 Sep 2008 18:43:26 -0700 Subject: [Cython] isinstance optimization breaks cdef methods In-Reply-To: <48D9882C.5040305@canterbury.ac.nz> References: <48D9882C.5040305@canterbury.ac.nz> Message-ID: <3A934234-B798-44F2-A1C8-CBF0416D432E@math.washington.edu> On Sep 23, 2008, at 5:22 PM, Greg Ewing wrote: > Lisandro Dalcin wrote: >> the generated code for, let say >> 'insinstance(None, list)' has two superfluous casts: >> >> __pyx_1 = PyObject_TypeCheck(Py_None, ((PyTypeObject >> *)((PyObject*)&PyList_Type))); > > That's because PyList_Type is known to have type 'type' > (at least Pyrex knows this, I'm assuming Cyton does too). > If you declare typecheck() as taking a type for the > second argument, rather than object, you shouldn't get > those casts. The problem is that isinstance is declared to take an object as the second parameter, and the optimization happens later on. I'm sure we could get around it if we wanted to put in the effort. - Robert From dalcinl at gmail.com Wed Sep 24 17:00:07 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 24 Sep 2008 12:00:07 -0300 Subject: [Cython] more comments on isinstance optimization In-Reply-To: References: <48D98DD6.5000806@canterbury.ac.nz> Message-ID: On Tue, Sep 23, 2008 at 10:41 PM, Robert Bradshaw wrote: > On Sep 23, 2008, at 5:46 PM, Greg Ewing wrote: > >> Lisandro Dalcin wrote: >> >>> Could we extend the 'builtin_types_table' ... PyInt_Type is >>> not there, not PyFloat_Type; >> >> In Pyrex, these are in the builtin_constant_table (with >> type 'type'). The builtin_type_table is only used for >> types having methods that are treated specially. > > I don't see any reason why we wouldn't, as long as they don't > interfere with the c types of the same name (though I don't see why > they would). > Well, it seems that Robert's isinstance optimization only see the types in 'builtin_types_table', perhaps this could be improved. Anyway, I'll try to populate 'builtin_types_table' and see what happens. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Wed Sep 24 18:35:31 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 24 Sep 2008 09:35:31 -0700 Subject: [Cython] more comments on isinstance optimization In-Reply-To: References: <48D98DD6.5000806@canterbury.ac.nz> Message-ID: On Sep 24, 2008, at 8:00 AM, Lisandro Dalcin wrote: > On Tue, Sep 23, 2008 at 10:41 PM, Robert Bradshaw > wrote: >> On Sep 23, 2008, at 5:46 PM, Greg Ewing wrote: >> >>> Lisandro Dalcin wrote: >>> >>>> Could we extend the 'builtin_types_table' ... PyInt_Type is >>>> not there, not PyFloat_Type; >>> >>> In Pyrex, these are in the builtin_constant_table (with >>> type 'type'). The builtin_type_table is only used for >>> types having methods that are treated specially. >> >> I don't see any reason why we wouldn't, as long as they don't >> interfere with the c types of the same name (though I don't see why >> they would). >> > > Well, it seems that Robert's isinstance optimization only see the > types in 'builtin_types_table', perhaps this could be improved. > Anyway, I'll try to populate 'builtin_types_table' and see what > happens. Specifically, it operates on anything of type type (which includes extension classes). - Robert From dagss at student.matnat.uio.no Thu Sep 25 00:09:37 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 25 Sep 2008 00:09:37 +0200 Subject: [Cython] Review: TempsBlockNode Message-ID: <48DABAA1.6080506@student.matnat.uio.no> Thanks to Robert's recent work with result_code (thanks!), I'm able to clean up some long-standing transform issues with temps. I've been wrong about such approaches before so I thought I'd check by if there are any opinions before I push to -devel. The solution I coded up is that one can create a TempsBlockNode which allocates temporaries during code generation, and use them by using a TempRefNode. (This is then used in TreeFragment, one can see the effect on slightly cleaner generated "with" statement C code.) Example: tempblock = TempsBlockNode(pos, [PyrexTypes.c_int_type, PyrexTypes.py_object_type], body=None) tempblock.body = SingleAssignmentNode(pos, lhs=tempblock.new_ref_node(1, pos), rhs=node) i.e. tempblock.new_ref_node is used to construct a TempRefNode which magically refers to a temporary which was set up by the TempsBlockNode (this is done by sharing a simple handle). The argument to new_ref_node is the index of the temporary (which is created by the type at the same index in the list given to the constructor). TempRefNode can be used in roughly the same situations as a NameNode referring to a local typed variable (i.e as an expression and as an assignment target). See: http://hg.cython.org/cython-dagss/rev/28b0311d6bff -- Dag Sverre From greg.ewing at canterbury.ac.nz Thu Sep 25 04:25:34 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 25 Sep 2008 14:25:34 +1200 Subject: [Cython] isinstance optimization breaks cdef methods In-Reply-To: <3A934234-B798-44F2-A1C8-CBF0416D432E@math.washington.edu> References: <48D9882C.5040305@canterbury.ac.nz> <3A934234-B798-44F2-A1C8-CBF0416D432E@math.washington.edu> Message-ID: <48DAF69E.9090801@canterbury.ac.nz> Robert Bradshaw wrote: > The problem is that isinstance is declared to take an object as the > second parameter, and the optimization happens later on. I'm not sure why you need compiler hackery to do this in the first place. Why not just declare isinstance in the builtin functions table as the equivalent of int isinstance "PyObject_TypeCheck" (object, type) A runtime type test will get generated if the second argument isn't known to be a type, but that's probably a good idea anyway, since PyObject_TypeCheck is expecting a type and will likely crash if you give it something else. -- Greg From robertwb at math.washington.edu Thu Sep 25 04:51:39 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 24 Sep 2008 19:51:39 -0700 Subject: [Cython] isinstance optimization breaks cdef methods In-Reply-To: <48DAF69E.9090801@canterbury.ac.nz> References: <48D9882C.5040305@canterbury.ac.nz> <3A934234-B798-44F2-A1C8-CBF0416D432E@math.washington.edu> <48DAF69E.9090801@canterbury.ac.nz> Message-ID: <65819D4C-7E90-4CA6-9B77-5CD8F51C5962@math.washington.edu> On Sep 24, 2008, at 7:25 PM, Greg Ewing wrote: > Robert Bradshaw wrote: > >> The problem is that isinstance is declared to take an object as the >> second parameter, and the optimization happens later on. > > I'm not sure why you need compiler hackery to do this > in the first place. Why not just declare isinstance > in the builtin functions table as the equivalent of > > int isinstance "PyObject_TypeCheck" (object, type) > > A runtime type test will get generated if the second > argument isn't known to be a type, but that's probably > a good idea anyway, since PyObject_TypeCheck is > expecting a type and will likely crash if you give > it something else. Because isinstance is much more versatile than this, it can take tuples, and non-types (though that will raise an error instead of crashing). I've optimized the case that the second argument of isinstance is of type type only. When we implement polymorphism, we can do it the "easy" way. - Robert From greg.ewing at canterbury.ac.nz Thu Sep 25 04:53:48 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 25 Sep 2008 14:53:48 +1200 Subject: [Cython] more comments on isinstance optimization In-Reply-To: References: <48D98DD6.5000806@canterbury.ac.nz> Message-ID: <48DAFD3C.10705@canterbury.ac.nz> Robert Bradshaw wrote: > Specifically, it operates on anything of type type (which includes > extension classes). In that case it should already work on int and float, as they're known to be types. -- Greg From dagss at student.matnat.uio.no Thu Sep 25 12:33:59 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 25 Sep 2008 12:33:59 +0200 Subject: [Cython] Ok to refactor InPlaceAssignmentNode? Message-ID: <48DB6917.9060404@student.matnat.uio.no> I keep running into issues with making struct buffer access work :-) As the temps part of result_code is not moved to generation yet (and that is a change that perhaps cannot happen very quickly), I get problems with CloneNode, even if my node is NOT a temp (as it makes it a temp). I've tried to wrap my head around InPlaceAssignmentNode for over an hour now without making any headway, so I'm prone to instead splitting it into CInPlaceAssignmentNode and PyInPlaceAssignmentNode in a transform. This should get rid of the need for CloneNodes in the C version and sidestep the issue. What do you think? However I am slightly worried: If the result of a buffer lookup cannot be put in a CloneNode, will other things/obscure usecases suddenly start breaking? I.e. *must* CloneNode be fixed before one can make use of result_code being moved to code generation? Robert: This applies to your recent changes too. "isinstance(x, y) += 2" is not legal, but are there other situations in which an isinstance could be put in a CloneNode? If so we have a bug already. Dag Sverre From dalcinl at gmail.com Thu Sep 25 15:07:12 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 25 Sep 2008 10:07:12 -0300 Subject: [Cython] more comments on isinstance optimization In-Reply-To: <48DAFD3C.10705@canterbury.ac.nz> References: <48D98DD6.5000806@canterbury.ac.nz> <48DAFD3C.10705@canterbury.ac.nz> Message-ID: On Wed, Sep 24, 2008 at 11:53 PM, Greg Ewing wrote: > Robert Bradshaw wrote: > >> Specifically, it operates on anything of type type (which includes >> extension classes). > > In that case it should already work on int and float, > as they're known to be types. > Well, by some reason Robert's patch do not work on int, float, slices. It end-ups optimizing cdef classes and some of built-in types. If 'int', 'float', 'slice', etc. are added in 'builtin_types_table' at Cython/Compiler/Builtin.py then they will be optimized. I'll send a patch soon. I just have to run the testsuite to be sure nothing breaks. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Thu Sep 25 18:18:01 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 25 Sep 2008 13:18:01 -0300 Subject: [Cython] isinstance and str (bytes/str/unicode again!) Message-ID: If we do 'isinstance(obj, str)', then what should be the behavior in Py2 and Py3? In fact, this question is actually related to the map in builtin_types_table at Cython/Compiler/Builtin.py. Do it make sense to do the map like this bytes -> PyBytes_Type str -> PyString_Type unicode -> PyUnicode_Type But this would require to #define 'PyString_Type' to 'PyUnicode_Type' for the Py3 case, instead of the current approach of being it #defined a 'PyBytes_Type'. My mind still resist the idea of not being able to tell Cython 'this is the built-in 'str' type you use in Python code, whatever it is in Py2 or Py3'. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Thu Sep 25 19:24:53 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 10:24:53 -0700 Subject: [Cython] more comments on isinstance optimization In-Reply-To: References: <48D98DD6.5000806@canterbury.ac.nz> <48DAFD3C.10705@canterbury.ac.nz> Message-ID: <7F083273-93B8-4A0D-85AA-C7AB5D9C3D2C@math.washington.edu> On Sep 25, 2008, at 6:07 AM, Lisandro Dalcin wrote: > On Wed, Sep 24, 2008 at 11:53 PM, Greg Ewing > wrote: >> Robert Bradshaw wrote: >> >>> Specifically, it operates on anything of type type (which includes >>> extension classes). >> >> In that case it should already work on int and float, >> as they're known to be types. >> > > Well, by some reason Robert's patch do not work on int, float, slices. > It end-ups optimizing cdef classes and some of built-in types. If > 'int', 'float', 'slice', etc. are added in 'builtin_types_table' at > Cython/Compiler/Builtin.py then they will be optimized. > > I'll send a patch soon. I just have to run the testsuite to be sure > nothing breaks. Thats' because they were not declared as types, just as ordinary objects. - Robert From robertwb at math.washington.edu Thu Sep 25 19:26:34 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 10:26:34 -0700 Subject: [Cython] isinstance and str (bytes/str/unicode again!) In-Reply-To: References: Message-ID: <1388E403-AFFB-407A-88CC-9CCE8092EB0C@math.washington.edu> On Sep 25, 2008, at 9:18 AM, Lisandro Dalcin wrote: > If we do 'isinstance(obj, str)', then what should be the behavior in > Py2 and Py3? In fact, this question is actually related to the map in > builtin_types_table at Cython/Compiler/Builtin.py. Do it make sense to > do the map like this > > bytes -> PyBytes_Type > str -> PyString_Type > unicode -> PyUnicode_Type > > But this would require to #define 'PyString_Type' to 'PyUnicode_Type' > for the Py3 case, instead of the current approach of being it #defined > a 'PyBytes_Type'. I think this is the right idea. > My mind still resist the idea of not being able to tell Cython 'this > is the built-in 'str' type you use in Python code, whatever it is in > Py2 or Py3'. Me too. - Robert From robertwb at math.washington.edu Thu Sep 25 20:55:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 11:55:07 -0700 Subject: [Cython] Ok to refactor InPlaceAssignmentNode? In-Reply-To: <48DB6917.9060404@student.matnat.uio.no> References: <48DB6917.9060404@student.matnat.uio.no> Message-ID: <29F35167-1C27-44E5-8A43-B7D714AF70AB@math.washington.edu> On Sep 25, 2008, at 3:33 AM, Dag Sverre Seljebotn wrote: > I keep running into issues with making struct buffer access > work :-) As > the temps part of result_code is not moved to generation yet (and that > is a change that perhaps cannot happen very quickly), I'm hopeful that there is away to move the temps out in an incremental fashion (using a different prefix to avoid conflict). > I get problems > with CloneNode, even if my node is NOT a temp (as it makes it a temp). > > I've tried to wrap my head around InPlaceAssignmentNode for over an > hour > now without making any headway, so I'm prone to instead splitting it > into CInPlaceAssignmentNode and PyInPlaceAssignmentNode in a > transform. > This should get rid of the need for CloneNodes in the C version and > sidestep the issue. What do you think? I don't see how this would help--the CloneNodes are created before analysis of whether or not it is a C operation is performed. It might be easier to simply unwrap the CloneNodes when you detect that it can be done safely and would cause problems. The whole inplace assignment node could probably be re-written using the better temps. > However I am slightly worried: If the result of a buffer lookup cannot > be put in a CloneNode, will other things/obscure usecases suddenly > start > breaking? I.e. *must* CloneNode be fixed before one can make use of > result_code being moved to code generation? > > Robert: > > This applies to your recent changes too. "isinstance(x, y) += 2" is > not > legal, but are there other situations in which an isinstance could be > put in a CloneNode? If so we have a bug already. Perhaps. I think all the CloneNode stuff could go away with better handling of temps (as in your previous email). - Robert From robertwb at math.washington.edu Thu Sep 25 21:23:07 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 12:23:07 -0700 Subject: [Cython] Review: TempsBlockNode In-Reply-To: <48DABAA1.6080506@student.matnat.uio.no> References: <48DABAA1.6080506@student.matnat.uio.no> Message-ID: On Sep 24, 2008, at 3:09 PM, Dag Sverre Seljebotn wrote: > Thanks to Robert's recent work with result_code (thanks!), I'm able to > clean up some long-standing transform issues with temps. > > I've been wrong about such approaches before so I thought I'd check by > if there are any opinions before I push to -devel. > > The solution I coded up is that one can create a TempsBlockNode which > allocates temporaries during code generation, and use them by using a > TempRefNode. (This is then used in TreeFragment, one can see the > effect > on slightly cleaner generated "with" statement C code.) > > Example: > > tempblock = TempsBlockNode(pos, [PyrexTypes.c_int_type, > PyrexTypes.py_object_type], body=None) > > tempblock.body = SingleAssignmentNode(pos, > lhs=tempblock.new_ref_node(1, pos), rhs=node) > > i.e. tempblock.new_ref_node is used to construct a TempRefNode which > magically refers to a temporary which was set up by the TempsBlockNode > (this is done by sharing a simple handle). The argument to > new_ref_node > is the index of the temporary (which is created by the type at the > same > index in the list given to the constructor). > > TempRefNode can be used in roughly the same situations as a NameNode > referring to a local typed variable (i.e as an expression and as an > assignment target). > > See: > > http://hg.cython.org/cython-dagss/rev/28b0311d6bff I think this is a nice approach that could clean up a lot of stuff. (In particular, I think we could get rid of almost every CloneNode and the (admittedly somewhat hackish) PersistentNode.) The one thing I would change is having to refer to these handles by index into a list passed at creation time. I would rather have an interface something like tempblock = TempsBlockNode(pos, body=None) foo = tempblock.new_temps(PyrexTypes.c_int_type) a, b = tempblock.new_temps(PyrexTypes.py_object_type, PyrexTypes.py_object_type) tempblock.body = SingleAssignmentNode(pos, lhs=foo.ref(), rhs=node) Actually, it does feel a bit weird setting the body after it is created. What about foo = TempRefNode(PyrexTypes.c_int_type) body = SingleAssignmentNode(pos, lhs=foo.ref(), rhs=node) return TempsBlockNode(pos, body=body, temps=[foo]) - Robert P.S. When I mentioned incremental temp migration, FunctionState.allocate_temp is exactly what I had in mind. Nice. From dagss at student.matnat.uio.no Thu Sep 25 21:54:26 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 25 Sep 2008 21:54:26 +0200 (CEST) Subject: [Cython] Review: TempsBlockNode In-Reply-To: References: <48DABAA1.6080506@student.matnat.uio.no> Message-ID: <55418.193.157.228.10.1222372466.squirrel@webmail.uio.no> Robert wrote: > On Sep 24, 2008, at 3:09 PM, Dag Sverre Seljebotn wrote: >> See: >> >> http://hg.cython.org/cython-dagss/rev/28b0311d6bff > > I think this is a nice approach that could clean up a lot of stuff. > (In particular, I think we could get rid of almost every CloneNode > and the (admittedly somewhat hackish) PersistentNode.) The one thing > I would change is having to refer to these handles by index into a > list passed at creation time. I would rather have an interface > something like > > tempblock = TempsBlockNode(pos, body=None) > foo = tempblock.new_temps(PyrexTypes.c_int_type) > a, b = tempblock.new_temps(PyrexTypes.py_object_type, > PyrexTypes.py_object_type) > tempblock.body = SingleAssignmentNode(pos, lhs=foo.ref(), rhs=node) > > Actually, it does feel a bit weird setting the body after it is > created. What about > > foo = TempRefNode(PyrexTypes.c_int_type) > body = SingleAssignmentNode(pos, lhs=foo.ref(), rhs=node) > return TempsBlockNode(pos, body=body, temps=[foo]) I didn't like it either, but it was the best I came up with there and then. The problem I have with your suggestions is this: I took care to make a) the temporary and b) a node in the tree referring to a temporary, seperate things. Even if node.pos is made optional/ignored (which could vary among TempRefNodes referring to the same temp), I think it makes sense to keep this distinction -- for instance, control flow analysis might want to tag possible variable values to TempRefNode instances which would only hold about the temp at that location. It just doesn't feel right to have a node sit in more than one place in the tree... So, what about just being explicit and do: foo = TempHandle(type) a = TempRefNode(pos, foo) b = TempRefNode(pos, foo) ... TempsBlockNode(pos, temps=[foo]) ? > P.S. When I mentioned incremental temp migration, > FunctionState.allocate_temp is exactly what I had in mind. Nice. Yes, that was don in in summer. Once everything is moved over in a year or so we can go back to only one temp namespace. Dag Sverre From robertwb at math.washington.edu Thu Sep 25 22:05:42 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 13:05:42 -0700 Subject: [Cython] Review: TempsBlockNode In-Reply-To: <55418.193.157.228.10.1222372466.squirrel@webmail.uio.no> References: <48DABAA1.6080506@student.matnat.uio.no> <55418.193.157.228.10.1222372466.squirrel@webmail.uio.no> Message-ID: <8496418A-6D8E-4BB7-AFE8-30E75EC02C56@math.washington.edu> On Sep 25, 2008, at 12:54 PM, Dag Sverre Seljebotn wrote: > Robert wrote: >> On Sep 24, 2008, at 3:09 PM, Dag Sverre Seljebotn wrote: >>> See: >>> >>> http://hg.cython.org/cython-dagss/rev/28b0311d6bff >> >> I think this is a nice approach that could clean up a lot of stuff. >> (In particular, I think we could get rid of almost every CloneNode >> and the (admittedly somewhat hackish) PersistentNode.) The one thing >> I would change is having to refer to these handles by index into a >> list passed at creation time. I would rather have an interface >> something like >> >> tempblock = TempsBlockNode(pos, body=None) >> foo = tempblock.new_temps(PyrexTypes.c_int_type) >> a, b = tempblock.new_temps(PyrexTypes.py_object_type, >> PyrexTypes.py_object_type) >> tempblock.body = SingleAssignmentNode(pos, lhs=foo.ref(), >> rhs=node) >> >> Actually, it does feel a bit weird setting the body after it is >> created. What about >> >> foo = TempRefNode(PyrexTypes.c_int_type) >> body = SingleAssignmentNode(pos, lhs=foo.ref(), rhs=node) >> return TempsBlockNode(pos, body=body, temps=[foo]) > > I didn't like it either, but it was the best I came up with there > and then. > > The problem I have with your suggestions is this: I took care to > make a) > the temporary and b) a node in the tree referring to a temporary, > seperate > things. Sorry, I had a really bad typo in my example. What I meant was "foo = TempHandle(PyrexTypes.c_int_type)" > Even if node.pos is made optional/ignored (which could vary among > TempRefNodes referring to the same temp), I think it makes sense to > keep > this distinction -- for instance, control flow analysis might want > to tag > possible variable values to TempRefNode instances which would only > hold > about the temp at that location. It just doesn't feel right to have > a node > sit in more than one place in the tree... Yes. > So, what about just being explicit and do: > > foo = TempHandle(type) > a = TempRefNode(pos, foo) > b = TempRefNode(pos, foo) > ... TempsBlockNode(pos, temps=[foo]) > > ? Yes. That's what I meant to say :). I was thinking foo.ref() as shorthand for TempRefNode(pos, foo). I guess we should pas pos in as well. >> P.S. When I mentioned incremental temp migration, >> FunctionState.allocate_temp is exactly what I had in mind. Nice. > > Yes, that was don in in summer. Once everything is moved over in a > year or > so we can go back to only one temp namespace. I'd forgotten about that (or didn't realize it at the time--I read a lot of code but didn't go through it all with a fine-toothed comb :). - Robert From dalcinl at gmail.com Thu Sep 25 22:23:24 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 25 Sep 2008 17:23:24 -0300 Subject: [Cython] more comments on isinstance optimization In-Reply-To: <7F083273-93B8-4A0D-85AA-C7AB5D9C3D2C@math.washington.edu> References: <48D98DD6.5000806@canterbury.ac.nz> <48DAFD3C.10705@canterbury.ac.nz> <7F083273-93B8-4A0D-85AA-C7AB5D9C3D2C@math.washington.edu> Message-ID: On Thu, Sep 25, 2008 at 2:24 PM, Robert Bradshaw wrote: > On Sep 25, 2008, at 6:07 AM, Lisandro Dalcin wrote: > >> On Wed, Sep 24, 2008 at 11:53 PM, Greg Ewing >> wrote: >>> Robert Bradshaw wrote: >>> >>>> Specifically, it operates on anything of type type (which includes >>>> extension classes). >>> >>> In that case it should already work on int and float, >>> as they're known to be types. >>> >> >> Well, by some reason Robert's patch do not work on int, float, slices. >> It end-ups optimizing cdef classes and some of built-in types. If >> 'int', 'float', 'slice', etc. are added in 'builtin_types_table' at >> Cython/Compiler/Builtin.py then they will be optimized. >> >> I'll send a patch soon. I just have to run the testsuite to be sure >> nothing breaks. > > Thats' because they were not declared as types, just as ordinary > objects. > Yes, I see the difference now. So, what should I do? There is a bit of code duplication here. I'll try to populate all the tables as appropriate, and let's put in the TODO a unification if that make sense and it is possible. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Thu Sep 25 22:24:15 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 25 Sep 2008 17:24:15 -0300 Subject: [Cython] isinstance and str (bytes/str/unicode again!) In-Reply-To: <1388E403-AFFB-407A-88CC-9CCE8092EB0C@math.washington.edu> References: <1388E403-AFFB-407A-88CC-9CCE8092EB0C@math.washington.edu> Message-ID: OK, let's wait for Stefan/Dag/Greg comments about this. On Thu, Sep 25, 2008 at 2:26 PM, Robert Bradshaw wrote: > On Sep 25, 2008, at 9:18 AM, Lisandro Dalcin wrote: > >> If we do 'isinstance(obj, str)', then what should be the behavior in >> Py2 and Py3? In fact, this question is actually related to the map in >> builtin_types_table at Cython/Compiler/Builtin.py. Do it make sense to >> do the map like this >> >> bytes -> PyBytes_Type >> str -> PyString_Type >> unicode -> PyUnicode_Type >> >> But this would require to #define 'PyString_Type' to 'PyUnicode_Type' >> for the Py3 case, instead of the current approach of being it #defined >> a 'PyBytes_Type'. > > I think this is the right idea. > >> My mind still resist the idea of not being able to tell Cython 'this >> is the built-in 'str' type you use in Python code, whatever it is in >> Py2 or Py3'. > > Me too. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Thu Sep 25 22:29:02 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 13:29:02 -0700 Subject: [Cython] more comments on isinstance optimization In-Reply-To: References: <48D98DD6.5000806@canterbury.ac.nz> <48DAFD3C.10705@canterbury.ac.nz> <7F083273-93B8-4A0D-85AA-C7AB5D9C3D2C@math.washington.edu> Message-ID: On Sep 25, 2008, at 1:23 PM, Lisandro Dalcin wrote: > On Thu, Sep 25, 2008 at 2:24 PM, Robert Bradshaw > wrote: >> On Sep 25, 2008, at 6:07 AM, Lisandro Dalcin wrote: >> >>> On Wed, Sep 24, 2008 at 11:53 PM, Greg Ewing >>> wrote: >>>> Robert Bradshaw wrote: >>>> >>>>> Specifically, it operates on anything of type type (which includes >>>>> extension classes). >>>> >>>> In that case it should already work on int and float, >>>> as they're known to be types. >>>> >>> >>> Well, by some reason Robert's patch do not work on int, float, >>> slices. >>> It end-ups optimizing cdef classes and some of built-in types. If >>> 'int', 'float', 'slice', etc. are added in 'builtin_types_table' at >>> Cython/Compiler/Builtin.py then they will be optimized. >>> >>> I'll send a patch soon. I just have to run the testsuite to be sure >>> nothing breaks. >> >> Thats' because they were not declared as types, just as ordinary >> objects. >> > > Yes, I see the difference now. So, what should I do? There is a bit of > code duplication here. I'll try to populate all the tables as > appropriate, and let's put in the TODO a unification if that make > sense and it is possible. Yes, builtin_entries should probably be moved form symtab to Builtins.pyx for consistancy. One could also declare them to have the more specialized type. - Robert From dalcinl at gmail.com Fri Sep 26 00:29:32 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 25 Sep 2008 19:29:32 -0300 Subject: [Cython] issues with the GIL in Python 2.3 Message-ID: I've just discovered that Python2.3 has some problems (in my case, it segfaults) if when the GIL is release but PyEval_InitThreads() has not ben called. Adding this code at the very begining of my pyx file seems to solve the problem. cdef extern from "Python.h": enum: PY_VERSION_HEX enum: WITH_THREAD void PyEval_InitThreads() if PY_VERSION_HEX < 0x02040000: if WITH_THREAD: PyEval_InitThreads() However, I believe Cython/Pyrex could generate the fix somewere in the module init function (at the begining? at the end?) if the GIL is ever aquired/released in Cython code (I mean, using 'with gil/nogil' ?). What do you think? As reference, you can see some comments in 'Python-2.3.6/Modules/_bsddb.c'. For your convenience, pasted below: /* PyEval_InitThreads is called here due to a quirk in python 1.5 * - 2.2.1 (at least) according to Russell Williamson : * The global interepreter lock is not initialized until the first * thread is created using thread.start_new_thread() or fork() is * called. that would cause the ALLOW_THREADS here to segfault due * to a null pointer reference if no threads or child processes * have been created. This works around that and is a no-op if * threads have already been initialized. * (see pybsddb-users mailing list post on 2002-08-07) */ #ifdef WITH_THREAD PyEval_InitThreads(); #endif I really need advice here, because my Python+threads knowledge is not really good. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Fri Sep 26 01:11:29 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 25 Sep 2008 20:11:29 -0300 Subject: [Cython] issues with the GIL in Python 2.3 In-Reply-To: References: Message-ID: BTW, the fix I'm using in my pyx file is a broken, PyEval_InitThreads() is available only if WITH_THREAD is #define'd. On Thu, Sep 25, 2008 at 7:29 PM, Lisandro Dalcin wrote: > I've just discovered that Python2.3 has some problems (in my case, it > segfaults) if when the GIL is release but PyEval_InitThreads() has not > ben called. > > Adding this code at the very begining of my pyx file seems to solve > the problem. > > cdef extern from "Python.h": > enum: PY_VERSION_HEX > enum: WITH_THREAD > void PyEval_InitThreads() > > if PY_VERSION_HEX < 0x02040000: > if WITH_THREAD: > PyEval_InitThreads() > > However, I believe Cython/Pyrex could generate the fix somewere in the > module init function (at the begining? at the end?) if the GIL is ever > aquired/released in Cython code (I mean, using 'with gil/nogil' ?). > > What do you think? As reference, you can see some comments in > 'Python-2.3.6/Modules/_bsddb.c'. For your convenience, pasted below: > > /* PyEval_InitThreads is called here due to a quirk in python 1.5 > * - 2.2.1 (at least) according to Russell Williamson : > * The global interepreter lock is not initialized until the first > * thread is created using thread.start_new_thread() or fork() is > * called. that would cause the ALLOW_THREADS here to segfault due > * to a null pointer reference if no threads or child processes > * have been created. This works around that and is a no-op if > * threads have already been initialized. > * (see pybsddb-users mailing list post on 2002-08-07) > */ > #ifdef WITH_THREAD > PyEval_InitThreads(); > #endif > > > I really need advice here, because my Python+threads knowledge is not > really good. > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From greg.ewing at canterbury.ac.nz Fri Sep 26 02:11:34 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Sep 2008 12:11:34 +1200 Subject: [Cython] Ok to refactor InPlaceAssignmentNode? In-Reply-To: <48DB6917.9060404@student.matnat.uio.no> References: <48DB6917.9060404@student.matnat.uio.no> Message-ID: <48DC28B6.4030505@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > However I am slightly worried: If the result of a buffer lookup cannot > be put in a CloneNode, will other things/obscure usecases suddenly start > breaking? I would be worried if it couldn't. I can't remember offhand all the ways that a CloneNode is used, but one that comes to mind is the handling of cascaded comparisons, i.e. things like x < y < z. The middle expression gets split in two using a CloneNode so that its result can be used more than once. So you can expect any expression to potentially become wrapped in a CloneNode given the right context. -- Greg From greg.ewing at canterbury.ac.nz Fri Sep 26 02:21:07 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 26 Sep 2008 12:21:07 +1200 Subject: [Cython] more comments on isinstance optimization In-Reply-To: References: <48D98DD6.5000806@canterbury.ac.nz> <48DAFD3C.10705@canterbury.ac.nz> Message-ID: <48DC2AF3.9010602@canterbury.ac.nz> Lisandro Dalcin wrote: > Well, by some reason Robert's patch do not work on int, float, slices. It must be doing something in an odd way, then. Is this being done after type analysis? If so, you should just be able to look at whether the type of the second argument is PyrexTypes.py_type_type or a subclass of it. -- Greg From robertwb at math.washington.edu Fri Sep 26 03:01:12 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 18:01:12 -0700 Subject: [Cython] more comments on isinstance optimization In-Reply-To: <48DC2AF3.9010602@canterbury.ac.nz> References: <48D98DD6.5000806@canterbury.ac.nz> <48DAFD3C.10705@canterbury.ac.nz> <48DC2AF3.9010602@canterbury.ac.nz> Message-ID: <203B1E95-6ADE-4A08-92E9-6A93BE0515DC@math.washington.edu> On Sep 25, 2008, at 5:21 PM, Greg Ewing wrote: > Lisandro Dalcin wrote: > >> Well, by some reason Robert's patch do not work on int, float, >> slices. > > It must be doing something in an odd way, then. Is this > being done after type analysis? If so, you should just be > able to look at whether the type of the second argument > is PyrexTypes.py_type_type or a subclass of it. There's no mystery here, int, float, slice, etc. were declared as being normal objects (as this predates the builtin types, and I didn't see any advantage at the moment to typing them). - Robert From kevinar18 at hotmail.com Fri Sep 26 04:33:31 2008 From: kevinar18 at hotmail.com (Kevin Ar18) Date: Thu, 25 Sep 2008 22:33:31 -0400 Subject: [Cython] cython license: PSF or Apache? Message-ID: > Sorry for the confusion. It is the Apache license, as we found out > the PSF license can't be applied to anything but Python itself. If > Cython becomes part of the standard Python distribution, then we can > change to the PSF. > > I've changed LICENSE.txt t reflect this. > > - Robert There is also the option of a "generic PSF license" ... if that might be preferred. :) Per http://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq If you feel the PSF License is the one to use with your code, you must change the following parts of the license: * Replace all occurrences of "Python Software Foundation" and "PSF" with your name or organization. * Replace "Python" with the name of your project. _________________________________________________________________ See how Windows Mobile brings your life together?at home, work, or on the go. http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/ From kevinar18 at hotmail.com Fri Sep 26 04:46:54 2008 From: kevinar18 at hotmail.com (Kevin Ar18) Date: Thu, 25 Sep 2008 22:46:54 -0400 Subject: [Cython] =?windows-1256?q?Microsoft_Visual_C++_2008_Express_Editi?= =?windows-1256?b?b27+?= Message-ID: Are there any step by step instructions for installing Cython with Microsoft Visual C++ 2008 Express Edition? I just get install errors. After manually editing msvccompiler.py, I got farther: def initialize(self): self.__paths = [] if os.environ.has_key("DISTUTILS_USE_SDK") and os.environ.has_key("MSSdk") and self.find_exe("cl.exe"): # Assume that the SDK set up everything alright; don't try to be # smarter self.cc = "cl.exe" self.linker = "link.exe" self.lib = "lib.exe" self.rc = "rc.exe" self.mc = "mc.exe" else: to: def initialize(self): self.__paths = [] if True: # Assume that the SDK set up everything alright; don't try to be # smarter self.cc = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\cl.exe" self.linker = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\link.exe" self.lib = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\lib.exe" self.rc = "C:\\Program Files\\Microsoft SDKs\\Windows\\v6.0A\\bin\\RC.exe" self.mc = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\mc.exe" else: Note: I can't actually find mc.exe in MSVC 2008. Is this bad? ...and here's the latest error: C:\...\Cython-0.9.8.1.1>python setup.py install Compiling module Cython.Plex.Scanners ... running install running build running build_py running build_ext building 'Cython.Plex.Scanners' extension C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG "-IC:\Program Files\Python 2.5.2\include" "-IC:\Program Fi les\Python 2.5.2\PC" /TcC:\...\Cython-0.9.8.1.1\Cython\Plex\Scanners.c /Fobuild\temp.win32-2.5\Release\...\Cython-0.9.8.1.1\Cython\Plex\Scanners.obj Scanners.c c:\program files\python 2.5.2\include\pyconfig.h(61) : fatal error C1083: Cannot open include file: 'io.h': No such file or directory error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\cl.ex e"' failed with exit status 2 Been working at it for hours now... at least I've gotten a little farther. _________________________________________________________________ Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 From michael.abshoff at googlemail.com Fri Sep 26 05:07:18 2008 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Thu, 25 Sep 2008 20:07:18 -0700 Subject: [Cython] =?windows-1256?q?Microsoft_Visual_C++_2008_Express_Editi?= =?windows-1256?b?b27+?= In-Reply-To: References: Message-ID: <48DC51E6.1080704@gmail.com> Kevin Ar18 wrote: > Are there any step by step instructions for installing Cython with Microsoft Visual C++ 2008 Express Edition? > I just get install errors. IIRC Python 2.5.x's distutils do not support MSVC 2008 of any kind, just 2003 and 2005. Python 2.6 (now in rc status) does support it. > After manually editing msvccompiler.py, I got farther: > def initialize(self): > self.__paths = [] > if os.environ.has_key("DISTUTILS_USE_SDK") and os.environ.has_key("MSSdk") and self.find_exe("cl.exe"): > # Assume that the SDK set up everything alright; don't try to be > # smarter > self.cc = "cl.exe" > self.linker = "link.exe" > self.lib = "lib.exe" > self.rc = "rc.exe" > self.mc = "mc.exe" > else: > to: > def initialize(self): > self.__paths = [] > if True: > # Assume that the SDK set up everything alright; don't try to be > # smarter > self.cc = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\cl.exe" > self.linker = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\link.exe" > self.lib = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\lib.exe" > self.rc = "C:\\Program Files\\Microsoft SDKs\\Windows\\v6.0A\\bin\\RC.exe" > self.mc = "C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\bin\\mc.exe" > else: > > > Note: I can't actually find mc.exe in MSVC 2008. Is this bad? Maybe :) According to http://msdn.microsoft.com/en-us/library/aa385638(VS.85).aspx mc.exe is the message compiler, which might or might be not required to build DLLs on Windows. It might be that the Express edition of 2008 does not include it, but I do not have a MSVC 2008 professional handy at this moment, but I can likely check later. > ...and here's the latest error: > C:\...\Cython-0.9.8.1.1>python setup.py install > Compiling module Cython.Plex.Scanners ... > running install > running build > running build_py > running build_ext > building 'Cython.Plex.Scanners' extension > C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\cl.exe /c /nologo /Ox > /MD /W3 /GS- /DNDEBUG "-IC:\Program Files\Python 2.5.2\include" "-IC:\Program Fi > les\Python 2.5.2\PC" /TcC:\...\Cython-0.9.8.1.1\Cython\Plex\Scanners.c > /Fobuild\temp.win32-2.5\Release\...\Cython-0.9.8.1.1\Cython\Plex\Scanners.obj > Scanners.c > c:\program files\python 2.5.2\include\pyconfig.h(61) : fatal error C1083: Cannot > open include file: 'io.h': No such file or directory > error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\cl.ex > e"' failed with exit status 2 > > > Been working at it for hours now... at least I've gotten a little farther. This is probably due to 2008 not being supported by Python 2.5. Some people out there have patches for MSVC 2008 and Python 2.5.2, but you have to Google for them since I did not bookmark those when I last came across them. I have not looked at pyconfig.h, so I might be completely wrong here. Did you build Python yourself or did you use an installer? If you used a binary build with MSVC 2005 I would not be surprised that even after building the extension it might still blow up on loading due to ABI breakage. Cheers, Michael > _________________________________________________________________ > Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie. > http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From kevinar18 at hotmail.com Fri Sep 26 05:48:33 2008 From: kevinar18 at hotmail.com (Kevin Ar18) Date: Thu, 25 Sep 2008 23:48:33 -0400 Subject: [Cython] =?windows-1256?q?_RE=3A__Microsoft_Visual_C++_2008_Expre?= =?windows-1256?q?ss_Edition=FE?= In-Reply-To: References: Message-ID: ---------------------------------------- > From: kevinar18 at hotmail.com > To: cython-dev at codespeak.net > Date: Thu, 25 Sep 2008 22:46:54 -0400 > Subject: [Cython] Microsoft Visual C++ 2008 Express Edition? > > > Been working at it for hours now... at least I've gotten a little farther. Made some progress by editing msvccompiler.py some more: macros, objects, extra_postargs, pp_opts, build = \ self._setup_compile(output_dir, macros, include_dirs, sources, depends, extra_postargs) pp_opts += ["-IC:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\include"] pp_opts += ["-IC:\\Program Files\\Microsoft SDKs\\Windows\\v6.0A\\Include"] lib_opts += ["/LIBPATH:C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\lib"] lib_opts += ["/LIBPATH:C:\\Program Files\\Microsoft SDKs\\Windows\\v6.0A\\Lib"] lib_opts += ["/LIBPATH:C:\\Program Files\\Microsoft SDKs\\Windows\\v6.0A\\Lib\\x64"] ld_args = (ldflags + lib_opts + export_opts + objects + ['/OUT:' + output_filename]) current error: C:\Users\Kevin\Downloads\dev\Cython-0.9.8.1.1\Cython-0.9.8.1.1>python setup.py i nstall Compiling module Cython.Plex.Scanners ... running install running build running build_py running build_ext building 'Cython.Plex.Scanners' extension ... Scanners.c ... C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\link.exe ... Creating library build\temp.win32-2.5\Release\...\dev\Cytho n-0.9.8.1.1\Cython-0.9.8.1.1\Cython\Plex\Scanners.lib and object build\temp.win3 2-2.5\Release\Users\Kevin\Downloads\dev\Cython-0.9.8.1.1\Cython-0.9.8.1.1\Cython \Plex\Scanners.exp Scanners.obj : error LNK2019: unresolved external symbol __imp__PyObject_GetItem referenced in function ___Pyx_GetItemInt Scanners.obj : error LNK2019: unresolved external symbol __imp__PyInt_FromLong r eferenced in function ___Pyx_GetItemInt ... Scanners.obj : error LNK2019: unresolved external symbol __imp__Py_InitModule4 r eferenced in function _initScanners build\lib.win32-2.5\Cython\Plex\Scanners.pyd : fatal error LNK1120: 72 unresolved externals error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\link. exe"' failed with exit status 1120 _________________________________________________________________ Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 From kevinar18 at hotmail.com Fri Sep 26 05:58:38 2008 From: kevinar18 at hotmail.com (Kevin Ar18) Date: Thu, 25 Sep 2008 23:58:38 -0400 Subject: [Cython] =?windows-1256?q?_RE=3A__Microsoft_Visual_C++_2008_Expre?= =?windows-1256?q?ss_Edition=FE?= In-Reply-To: <48DC51E6.1080704@gmail.com> References: <48DC51E6.1080704@gmail.com> Message-ID: ---------------------------------------- > Date: Thu, 25 Sep 2008 20:07:18 -0700 > To: cython-dev at codespeak.net > From: michael.abshoff at googlemail.com > Subject: Re: [Cython] Microsoft Visual C++ 2008 Express Edition? > > Kevin Ar18 wrote: >> Are there any step by step instructions for installing Cython with Microsoft Visual C++ 2008 Express Edition? >> I just get install errors. > > IIRC Python 2.5.x's distutils do not support MSVC 2008 of any kind, just > 2003 and 2005. Python 2.6 (now in rc status) does support it. Is 2.6 stable enough? The app needs accurate computation. >> ...and here's the latest error: >> C:\...\Cython-0.9.8.1.1>python setup.py install >> Compiling module Cython.Plex.Scanners ... >> running install >> running build >> running build_py >> running build_ext >> building 'Cython.Plex.Scanners' extension >> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\cl.exe /c /nologo /Ox >> /MD /W3 /GS- /DNDEBUG "-IC:\Program Files\Python 2.5.2\include" "-IC:\Program Fi >> les\Python 2.5.2\PC" /TcC:\...\Cython-0.9.8.1.1\Cython\Plex\Scanners.c >> /Fobuild\temp.win32-2.5\Release\...\Cython-0.9.8.1.1\Cython\Plex\Scanners.obj >> Scanners.c >> c:\program files\python 2.5.2\include\pyconfig.h(61) : fatal error C1083: Cannot >> open include file: 'io.h': No such file or directory >> error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\cl.ex >> e"' failed with exit status 2 >> >> >> Been working at it for hours now... at least I've gotten a little farther. > > This is probably due to 2008 not being supported by Python 2.5. Some > people out there have patches for MSVC 2008 and Python 2.5.2, but you > have to Google for them since I did not bookmark those when I last came > across them. I have not looked at pyconfig.h, so I might be completely > wrong here. > > Did you build Python yourself or did you use an installer? If you used a > binary build with MSVC 2005 I would not be surprised that even after > building the extension it might still blow up on loading due to ABI > breakage. It's pre-built, but I also tried one I built with msvc 2008, but I don't know if I built it right.... Should building my own Python fix Cython? hmm... let me try again... > > Cheers, > > Michael > Thanks BTW, I noticed there are no registry entries for MSVC like msvccompiler.py was looking for... odd. _________________________________________________________________ See how Windows connects the people, information, and fun that are part of your life. http://clk.atdmt.com/MRT/go/msnnkwxp1020093175mrt/direct/01/ From kevinar18 at hotmail.com Fri Sep 26 06:29:01 2008 From: kevinar18 at hotmail.com (Kevin Ar18) Date: Fri, 26 Sep 2008 00:29:01 -0400 Subject: [Cython] =?windows-1256?q?_RE=3A___RE=3A__Microsoft_Visual_C++_20?= =?windows-1256?q?08_Express_Edition=FE?= In-Reply-To: References: <48DC51E6.1080704@gmail.com> Message-ID: >> Did you build Python yourself or did you use an installer? If you used a >> binary build with MSVC 2005 I would not be surprised that even after >> building the extension it might still blow up on loading due to ABI >> breakage. > > It's pre-built, but I also tried one I built with msvc 2008, but I don't know if I built it right.... Should building my own Python fix Cython? > > hmm... let me try again... using custom msvccompiler.py Python-2.5.2\PCbuild Release Win32 ... * It installs * Demos It works? I get c files * runtests.py gives: running doctests in wundram1 ... ====================================================================== ERROR: compiling and running specialfloat ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 276, in run self.runCompileTest() File "runtests.py", line 173, in runCompileTest self.directory, self.expect_errors, self.annotate) File "runtests.py", line 262, in compile self.assertEquals(None, unexpected_error) AssertionError: None != '50:21: Error in compile-time expression: ValueError: in valid literal for float(): nan' ---------------------------------------------------------------------- Ran 629 tests in 86.580s FAILED (errors=1) Following tests excluded because of missing dependencies on your system: run.numpy_test _________________________________________________________________ See how Windows Mobile brings your life together?at home, work, or on the go. http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/ From lists at cheimes.de Fri Sep 26 08:14:39 2008 From: lists at cheimes.de (Christian Heimes) Date: Fri, 26 Sep 2008 08:14:39 +0200 Subject: [Cython] =?windows-1256?q?Microsoft_Visual_C++_2008_Express_Editi?= =?windows-1256?b?b27+?= In-Reply-To: References: Message-ID: Kevin Ar18 wrote: > Are there any step by step instructions for installing Cython with Microsoft Visual C++ 2008 Express Edition? > I just get install errors. You can't use VS 2005 or 2008 with Python 2.5 or older because these Python versions are compiled with VS 7.1. Each major version of the VC compiler comes with its own version of the MSVC runtime dll. The DLLs aren't compatible in the sense that you can't pass a FILE* from one DLL to another. For example http://effbot.org/pyfaq/pyrun-simplefile-crashes-on-windows-but-not-on-unix-why.htm explains one major issue. (This issue should probably be a FAQ entry on cython.org.) Either you have to get VC 7.1 (hard and ugly) or you can use MinGW32 to compile your code under Windows. I'm using it all the time - mostly with success ;) Christian From robertwb at math.washington.edu Fri Sep 26 08:34:24 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 25 Sep 2008 23:34:24 -0700 Subject: [Cython] cython license: PSF or Apache? In-Reply-To: References: Message-ID: On Sep 25, 2008, at 7:33 PM, Kevin Ar18 wrote: > >> Sorry for the confusion. It is the Apache license, as we found out >> the PSF license can't be applied to anything but Python itself. If >> Cython becomes part of the standard Python distribution, then we can >> change to the PSF. >> >> I've changed LICENSE.txt t reflect this. >> >> - Robert > > There is also the option of a "generic PSF license" ... if that > might be preferred. :) > > Per http://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq > If you feel the PSF License is the one to use with your code, you > must change the following parts of the license: > > * Replace all occurrences of "Python Software Foundation" and > "PSF" with your name or organization. > * Replace "Python" with the name of your project. Yep. Actually, the reason we chose the license we did is on that same page: What if I want to contribute my code to the PSF? The PSF does not want contributions of any code other than that which will end up in Python or its standard libraries, or in Jython. If your code is going to end up in Python or the standard library, the PSF will require you to: * License your code under an acceptable open source license. These currently include only the Academic Free License and the Apache License 2.0, although this list may be expanded in the future. (No, the PSF License is not acceptable; see below) * Fill out and submit a contributor agreement. Though we haven't made any formal movements to have Cython as part of the Python standard library, this is certainly something we would like to consider in the future and we want to keep things open in that direction. Fortunately the Apache License is a fairly widely used and well known license, and matches well with our goals. - Robert From dagss at student.matnat.uio.no Fri Sep 26 12:51:43 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 26 Sep 2008 12:51:43 +0200 Subject: [Cython] [Numpy-discussion] Cython: Passing numpy array into C via buffer? In-Reply-To: <48DCBA66.1080008@cam.ac.uk> References: <48DCBA66.1080008@cam.ac.uk> Message-ID: <48DCBEBF.3040701@student.matnat.uio.no> Richard Shaw wrote: > Hello, > > Forgive me if this is a stupid question, I've been looking around all > the Cython documentation and I can't find out if this is possible. > > What I would like to do is generally is wrap a C function that takes a > double array, and be able to pass in a numpy array, I was wondering if > it's possible to do this using the buffer interface? I understand I > could do it using this method (http://wiki.cython.org/WrappingNumpy), > but the buffer interface seems much neater. Questions like this should be posted to the Cython list (otherwise us Cython developers might easily ignore it, we do not follow everything that goes on in numpy-discussion). > Specifically, I have a C function that I would like to wrap with signature: > > #### cfile.h > double do_something(double * array, int len); > > I was hoping with the buffer interface, I could do something like this: > > #### pyfile.pyx > import numpy > cimport numpy > > cdef extern from "cfile.h" > double do_something(double * array, int len) > > cdef do_something_python(numpy.ndarray[numpy.float64, ndim = 1] numarr): numpy.float64_t, currently at least. > cdef double x > cdef double * data > cdef int l > > data = ... # Not sure what to do here > l = numarr.shape[0] Yes, I understand this. I definitely plan to make this easier in a coming Cython release, i.e. something like cimport cython ... data = cython.buffer.bufptr(array) I just haven't got there yet. For now, there's an ugly hack you can do (untested though): your_header.h: #define GET_BUFPTR_HACK(x) __pyx_bstruct_#x.buf yourfile.pyx: cdef extern from "your_header.h": void* GET_BUFPTR_HACK(x) ... data = GET_BUFPTR_HACK(buf) (The reason this is so horribly ugly is the "#x" in the macro, meaning this depends very much of the particular mangling scheme used. This will likely break in a future version of Cython, but then cython.buffer.butptr should be available.) -- Dag Sverre From dagss at student.matnat.uio.no Fri Sep 26 12:57:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 26 Sep 2008 12:57:44 +0200 Subject: [Cython] [Numpy-discussion] Cython: Passing numpy array into C via buffer? In-Reply-To: <48DCBA66.1080008@cam.ac.uk> References: <48DCBA66.1080008@cam.ac.uk> Message-ID: <48DCC028.1010507@student.matnat.uio.no> Richard Shaw wrote: > Hello, > > Forgive me if this is a stupid question, I've been looking around all > the Cython documentation and I can't find out if this is possible. > > What I would like to do is generally is wrap a C function that takes a > double array, and be able to pass in a numpy array, I was wondering if > it's possible to do this using the buffer interface? I understand I > could do it using this method (http://wiki.cython.org/WrappingNumpy), > but the buffer interface seems much neater. > > Specifically, I have a C function that I would like to wrap with signature: > > #### cfile.h > double do_something(double * array, int len); > > I was hoping with the buffer interface, I could do something like this: > > #### pyfile.pyx > import numpy > cimport numpy > > cdef extern from "cfile.h" > double do_something(double * array, int len) > > cdef do_something_python(numpy.ndarray[numpy.float64, ndim = 1] numarr): > cdef double x > cdef double * data > cdef int l > > data = ... # Not sure what to do here > l = numarr.shape[0] > > x = do_something(data, l) I'm sorry, my previous answer was a really, really bad one. Here's a better one: data = numarr.data :-) With the addition of the _t on float64 this should just work. Please insert an extra check to make sure that your data is contiguous though! Otherwise you need to make a copy to pass the data to an external lib (one way is by using numpy.array with a contiguouos argument to construct a copy, and then copy the data back again afterwards). -- Dag Sverre From dagss at student.matnat.uio.no Fri Sep 26 13:21:44 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 26 Sep 2008 13:21:44 +0200 Subject: [Cython] Coordination note: temp alloc Message-ID: <48DCC5C8.6040306@student.matnat.uio.no> I started some work on refactoring temp allocation but didn't finish it and I am going away for the weekend. Please see http://trac.cython.org/cython_trac/ticket/77 for the current state before doing anything related. -- Dag Sverre From ondrej at certik.cz Fri Sep 26 14:42:36 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 26 Sep 2008 14:42:36 +0200 Subject: [Cython] cython license: PSF or Apache? In-Reply-To: References: Message-ID: <85b5c3130809260542s7fc1ff10q4407234231e5b051@mail.gmail.com> On Fri, Sep 26, 2008 at 8:34 AM, Robert Bradshaw wrote: > On Sep 25, 2008, at 7:33 PM, Kevin Ar18 wrote: > >> >>> Sorry for the confusion. It is the Apache license, as we found out >>> the PSF license can't be applied to anything but Python itself. If >>> Cython becomes part of the standard Python distribution, then we can >>> change to the PSF. >>> >>> I've changed LICENSE.txt t reflect this. >>> >>> - Robert >> >> There is also the option of a "generic PSF license" ... if that >> might be preferred. :) >> >> Per http://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq >> If you feel the PSF License is the one to use with your code, you >> must change the following parts of the license: >> >> * Replace all occurrences of "Python Software Foundation" and >> "PSF" with your name or organization. >> * Replace "Python" with the name of your project. > > Yep. Actually, the reason we chose the license we did is on that same > page: > > What if I want to contribute my code to the PSF? > > The PSF does not want contributions of any code other than that which > will end up in Python or its standard libraries, or in Jython. If > your code is going to end up in Python or the standard library, the > PSF will require you to: > > * License your code under an acceptable open source license. > These currently include only the Academic Free License and the Apache > License 2.0, although this list may be expanded in the future. (No, > the PSF License is not acceptable; see below) > * Fill out and submit a contributor agreement. > > Though we haven't made any formal movements to have Cython as part of > the Python standard library, this is certainly something we would > like to consider in the future and we want to keep things open in > that direction. Fortunately the Apache License is a fairly widely > used and well known license, and matches well with our goals. pyprocessing, that was recently accepted, is BSD. And the license in Lib/multiprocessing/__init__.py is BSD. So I don't understand much this apache license requirement, but it seems to me that plain BSD is enough. Ondrej From ondrej at certik.cz Fri Sep 26 15:59:56 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 26 Sep 2008 15:59:56 +0200 Subject: [Cython] Support .pxd files for plain Python source files? In-Reply-To: <85b5c3130808270902h60ea5636q754f54fb3657eb91@mail.gmail.com> References: <33046.213.61.181.86.1219822755.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <44879.213.61.181.86.1219829632.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48B5324F.9010209@student.matnat.uio.no> <9A30F4EE-E8E6-4842-8055-23051CCBF03D@math.washington.edu> <85b5c3130808270902h60ea5636q754f54fb3657eb91@mail.gmail.com> Message-ID: <85b5c3130809260659n496056a7oa886522fd3a088a1@mail.gmail.com> On Wed, Aug 27, 2008 at 6:02 PM, Ondrej Certik wrote: > On Wed, Aug 27, 2008 at 5:51 PM, Robert Bradshaw > wrote: >> >> On Aug 27, 2008, at 3:54 AM, Dag Sverre Seljebotn wrote: >> >>> Stefan Behnel wrote: >>>> >>>> Stefan Behnel wrote: >>>>> >>>>> we've already discussed a couple of times how to provide type >>>>> information >>>>> for pure Python source code. It was generally agreed that it's difficult >>>>> to do so inside of the function body and easier to do for function >>>>> signatures if we use Py3 annotations. I don't remember any good ideas >>>>> how >>>>> to provide type information for classes, i.e. how to make them extension >>>>> types. >>> >>> Someone must have mentioned this?: >>> >>> @cython.cdef >>> class A: >>> ... >>> >>> @cython.cdef >>> def func(): ... >>> >>> Just mentioning it as a possibility. >> >> This is the way I was imagining doing it, especially as .pxd files move >> towards being auto-generated. The proposal of being able to simply declare >> them in the pxd file is an interesting one, and though it feels a bit sloppy >> to me perhaps it would be good to support as well (curious for more input). > > There also needs to by some syntax to define variables etc, but > anything that allows me to use the full power of Cython yet stay in > pure Python is fine with me. I am scarce on time, but I'd like to see this happen soon. How would you suggest me to modify Cython to support that? I can try to write some preparser, that takes pure python code (for example with decorators specifying types) and turn it into a regular .pyx file that Cython understands. But I am not sure if such a work would be helpful to you, I guess the ultimate solution would be to modify the Cython itself, but I am not sure how, without thorough studying of it, which I am not sure how much time (days?) will take. Those of you who know Cython well, if you have some ideas where I should start to help get this done, it'd be awesome. I can spend couple evenings on that. Ondrej From dagss at student.matnat.uio.no Fri Sep 26 16:43:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 26 Sep 2008 16:43:00 +0200 Subject: [Cython] Support .pxd files for plain Python source files? Message-ID: <3305292230.603409@smtp.netcom.no> If you have a couple of evenings I'm happy to help with pointers as it would be a good investment of time. I can write more later when I'm not on my phone, but: I'd suggest starting with implementing @cython.earlybind def f(x, y): ... and have it turn to cpdef f(x, y): ... (note on the name below) Just getting this working, with arguments and result typed to "object", should get your feet wet. Then I suppose the next step is implementing the Py3 PEP supporting decorators on arguments and return type. (I think one should aim for Py2.6+ here, though I suppose arguments to the decorator could work too: @cython.earlybind(x=cython.uint, localvar=cython.float)...opinions? Brief scetch on the first step anyway (please ask about any hazy points and I'll give better help when I'm at a computer): 1) The file you want to work in is ParseTreeTransforms.py. Your goal is to intercept decorated DefNode-s and transform them into CFuncDefNodes. Have a look and see how transforms are written... 2) The ResolveDirectives transform (or something like that) already has code in it to intercept @cython.X-decorators on functions for compiler directives, I think you want to extend that transform, in time changing the name appropriately to give it a wider scope. 3) the general workflow is this: Create a small testcase with both "from" code and "to" code as given above. Then add this function to ResolveDirectives (not entirely sure about the node class name, see Nodes.py): def visit_CFuncDefNode(self, node): print node.dump() return node this gives you an idea about the node structure you must convert to. 5) To transform, edit the function visit_DefNode to also take into account "earlybind" (hardcode it) before checking the decorator against directives, and if so, simply set node = self.convert_def_to_cpdef(node), which you write. "node.dump()" here gives you the from-tree. 6) if you fancy unit tests in addition to integration tests, have a look at TestWithTransform.py (or similar) for how to do it. You should literally be able to input the decorated def example above as a string and assert that the result after the transform is the expected cpdef version... though you may need to extend CodeWriter.py to serialize cpdef to strings. Send an email whenever something is not obvious -- it would be really cool to see this in Cython. Note on names: It is easily changeable so it is not a big point...but here goes: I think personally the cdef and cpdef are very cython-specific and communicate badly with python users. Also, considering cpdef exists, I think the need for cdef should be small? If wanted it could be called "@cython.nativeonly" perhaps. Anyway, "earlybind" also gets in the important message that you cannot reassign the symbol it binds to at runtime, which is a difference with how you expect a "def" to work. Dag Sverre Seljebotn -----Original Message----- From: "Ondrej Certik" Date: Friday, Sep 26, 2008 4:00 pm Subject: Re: [Cython] Support .pxd files for plain Python source files? To: "Robert Bradshaw" CC: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Wed, Aug 27, 2008 at 6:02 PM, Ondrej Certik wrote: > On Wed, Aug 27, 2008 at 5:51 PM, Robert Bradshaw > wrote: >> >> On Aug 27, 2008, at 3:54 AM, Dag Sverre Seljebotn wrote: >> >>> Stefan Behnel wrote: >>>> >>>> Stefan Behnel wrote: >>>>> >>>>> we've already discussed a couple of times how to provide type >>>>> information >>>>> for pure Python source code. It was generally agreed that it's difficult >>>>> to do so inside of the function body and easier to do for function >>>>> signatures if we use Py3 annotations. I don't remember any good ideas >>>>> how >>>>> to provide type information for classes, i.e. how to make them extension >>>>> types. >>> >>> Someone must have mentioned this?: >>> >>> @cython.cdef >>> class A: >>> ... >>> >>> @cython.cdef >>> def func(): ... >>> >>> Just mentioning it as a possibility. >> >> This is the way I was imagining doing it, especially as .pxd files move >> towards being auto-generated. The proposal of being able to simply declare >> them in the pxd file is an interesting one, and though it feels a bit sloppy >> to me perhaps it would be good to support as well (curious for more input). > >> There also needs to by some syntax to define variables etc, but > anything that allows me to use the full power of Cython yet stay in > pure Python is fine with me. > >I am scarce on time, but I'd like to see this happen soon. How would you suggest me to modify Cython to support that? I can try to write >some preparser, that takes pure python code (for example with >decorators specifying types) and turn it into a regular .pyx file that Cython understands. But I am not sure if such a work would be helpful to you, I guess the ultimate solution would be to modify the Cython >itself, but I am not sure how, without thorough studying of it, which I am not sure how much time (days?) will take. > >Those of you who know Cython well, if you have some ideas where I >should start to help get this done, it'd be awesome. I can spend >couple evenings on that. > >Ondrej >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From ondrej at certik.cz Fri Sep 26 17:12:58 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 26 Sep 2008 17:12:58 +0200 Subject: [Cython] Support .pxd files for plain Python source files? In-Reply-To: <3305292230.603409@smtp.netcom.no> References: <3305292230.603409@smtp.netcom.no> Message-ID: <85b5c3130809260812u4b9a24bfn84871c1e734a2fe2@mail.gmail.com> On Fri, Sep 26, 2008 at 4:43 PM, Dag Sverre Seljebotn wrote: > If you have a couple of evenings I'm happy to help with pointers as it would be a good investment of time. > > I can write more later when I'm not on my phone, but: I'd suggest starting with implementing > > @cython.earlybind > def f(x, y): ... > > and have it turn to > > cpdef f(x, y): ... > > (note on the name below) > > Just getting this working, with arguments and result typed to "object", should get your feet wet. Then I suppose the next step is implementing the Py3 PEP supporting decorators on arguments and return type. (I think one should aim for Py2.6+ here, though I suppose arguments to the decorator could work too: @cython.earlybind(x=cython.uint, localvar=cython.float)...opinions? I need it to be working with python2.4+, so I'll choose something that works in python2.4. Debian just switched to python2.5 recently, so it's a long, long way before python2.6 could become a de facto standard. But generally I agree one should design this so that things are especially easy in python2.6+. > > Brief scetch on the first step anyway (please ask about any hazy points and I'll give better help when I'm at a computer): > > 1) The file you want to work in is ParseTreeTransforms.py. Your goal is to intercept decorated DefNode-s and transform them into CFuncDefNodes. Have a look and see how transforms are written... > > 2) The ResolveDirectives transform (or something like that) already has code in it to intercept @cython.X-decorators on functions for compiler directives, I think you want to extend that transform, in time changing the name appropriately to give it a wider scope. > > 3) the general workflow is this: Create a small testcase with both "from" code and "to" code as given above. Then add this function to ResolveDirectives (not entirely sure about the node class name, see Nodes.py): > > def visit_CFuncDefNode(self, node): > print node.dump() > return node > > this gives you an idea about the node structure you must convert to. > > 5) To transform, edit the function visit_DefNode to also take into account "earlybind" (hardcode it) before checking the decorator against directives, and if so, simply set node = self.convert_def_to_cpdef(node), which you write. "node.dump()" here gives you the from-tree. > > 6) if you fancy unit tests in addition to integration tests, have a look at TestWithTransform.py (or similar) for how to do it. You should literally be able to input the decorated def example above as a string and assert that the result after the transform is the expected cpdef version... though you may need to extend CodeWriter.py to serialize cpdef to strings. > > Send an email whenever something is not obvious -- it would be really cool to see this in Cython. > > Note on names: It is easily changeable so it is not a big point...but here goes: I think personally the cdef and cpdef are very cython-specific and communicate badly with python users. Also, considering cpdef exists, I think the need for cdef should be small? If wanted it could be called "@cython.nativeonly" perhaps. Anyway, "earlybind" also gets in the important message that you cannot reassign the symbol it binds to at runtime, which is a difference with how you expect a "def" to work. Thanks a lot for the help! I'll give it a shot and report back. Ondrej From dagss at student.matnat.uio.no Fri Sep 26 18:55:45 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 26 Sep 2008 18:55:45 +0200 (CEST) Subject: [Cython] Python-compatible syntax In-Reply-To: <85b5c3130809260812u4b9a24bfn84871c1e734a2fe2@mail.gmail.com> References: <3305292230.603409@smtp.netcom.no> <85b5c3130809260812u4b9a24bfn84871c1e734a2fe2@mail.gmail.com> Message-ID: <50542.85.165.152.85.1222448145.squirrel@webmail.uio.no> Ondrej wrote: > On Fri, Sep 26, 2008 at 4:43 PM, Dag Sverre Seljebotn > wrote: >> I can write more later when I'm not on my phone, but: I'd suggest >> starting with implementing >> >> @cython.earlybind >> def f(x, y): ... >> >> and have it turn to >> >> cpdef f(x, y): ... >> >> (note on the name below) >> >> Just getting this working, with arguments and result typed to "object", >> should get your feet wet. Then I suppose the next step is implementing >> the Py3 PEP supporting decorators on arguments and return type. (I think >> one should aim for Py2.6+ here, though I suppose arguments to the >> decorator could work too: @cython.earlybind(x=cython.uint, >> localvar=cython.float)...opinions? > > I need it to be working with python2.4+, so I'll choose something that > works in python2.4. Debian just switched to python2.5 recently, so > it's a long, long way before python2.6 could become a de facto > standard. But generally I agree one should design this so that things > are especially easy in python2.6+. (Writing this just so that you can look it up when you are ready for types, ideally when declaring cpdef without any non-object types works already). We've had long discussions about options for this earlier and it is something people care strongly about, so please check with the mailing list before fixing too hard on a given syntax. Until something else is proposed then I'm +1 on my following proposal (but at least Stefan or Robert should +1 it before it has any kind of official status): @cython.earlybind(cython.uint, x=cython.uint, v=cython.float) def f(x): v = x return x The first positional argument is the return type, while any keyword arguments are for either arguments or local variables. Support for the Py2.6+ syntax can be added in addition later. We had a discussion already about the type names, I think we agreed on something like (though others should comment too): - cython.uint, cython.int, cython.float, cython.longdouble, etc. - cython.p_uint for unsigned*. Up to three pointers provided like this: cython.ppp_uint is unsigned***. (Hmm, in fact, as this is likely parsed as a string and not declared anywhere, one can make the number of p-s arbitrary). - For people preferring verbosity: cython.ptr(cython.uint), with nesting of ptr as many times as one like. Finally, cython.ptr(cython.uint, levels=4) for unsigned****... - Pointers to functions provided in a similar fashion, i.e. cython.func_ptr(cython.int, cython.float) etc. Notes you can look at again when you get as far (I enjoy thinking about how to do stuff, coding it is the boring part :-) of course, if you do things in your own way instead there's no harm to it though): 1) This is already parseable by Parsing.py, so you should be able to implement this purely by staying in the transform I wrote about and output type nodes in the appropriate places. 2) It would work by having the transform simply inserting "cdef unsigned int v" statements (i.e. CVarDeclNode) at the beginning of the function body, or changing info in the argument list of the function (anything in the argument list goes there, the rest is inserted at beginning of function without checking whether it is used). 3) As cpdef functions cannot take pointer arguments, one should generate cdef functions instead when there are pointer arguments. IMO this should be automagic, i.e. cpdef is selected whenever the arguments are coercable from objects and cdef otherwise. (An alternative I would like better, though, is to have cpdef functions taking pointer arguments be allowed in the language, but lead to a "def" version containing a stub raising a run-time error! This way things could be more consistent with good error reporting. Long term, ctypes pointers could be coercable to Cython pointers too.) >> 2) The ResolveDirectives transform (or something like that) already has >> code in it to intercept @cython.X-decorators on functions for compiler >> directives, I think you want to extend that transform, in time changing >> the name appropriately to give it a wider scope. Having thought about this, the transform should be renamed and redocumented to "Handle the parts of the cython scope that are not normal run-time functions, i.e. any 'magic' functions and decorators in the cython cimportable module." Something like CythonScopeTransform perhaps. Dag Sverre From lists at cheimes.de Fri Sep 26 19:36:37 2008 From: lists at cheimes.de (Christian Heimes) Date: Fri, 26 Sep 2008 19:36:37 +0200 Subject: [Cython] isinstance and str (bytes/str/unicode again!) In-Reply-To: References: Message-ID: Lisandro Dalcin wrote: > If we do 'isinstance(obj, str)', then what should be the behavior in > Py2 and Py3? In fact, this question is actually related to the map in > builtin_types_table at Cython/Compiler/Builtin.py. Do it make sense to > do the map like this > > bytes -> PyBytes_Type > str -> PyString_Type > unicode -> PyUnicode_Type It doesn't make sense for some applications of str. Python 3.0 uses PyUnicode for identifiers (attribute names, function and class names etc.) while Python 2.x uses PyString. You can use PyUnicode instances for attributes in Python 2.x through obj.__dict__[u"key"] = key but you cannot use PyBytes in 3.0. This mapping makes more sense for me: bytes:: whatever Python uses to store binary data Python 2.x: PyString Python 3.x: PyBytes unicode:: text data PyUnicode for all Python versions str:: whatever Python uses for attributes and names Python 2.x: PyString Python 3.x: PyUnicode The suggestion follows my forward compatibility code for Python 2.6. In Python 2.6 b"" and bytes are simple aliases for PyString. "from __future__ import unicode_literals" turns "" into PyUnicode objects. Christian From dagss at student.matnat.uio.no Fri Sep 26 19:43:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 26 Sep 2008 19:43:00 +0200 Subject: [Cython] Support .pxd files for plain Python source files? Message-ID: <3305302990.684309@smtp.netcom.no> BTW as you may have gathered I may have misunderstood your original request; if you would rather like to implement Stefan's original (and, I believe, complementary) proposal then the procedure is a bit different. But ask for details if you want to take that route... Dag Sverre Seljebotn -----Original Message----- From: "Ondrej Certik" Date: Friday, Sep 26, 2008 5:13 pm Subject: Re: [Cython] Support .pxd files for plain Python source files? To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Fri, Sep 26, 2008 at 4:43 PM, Dag Sverre Seljebotn > wrote: > If you have a couple of evenings I'm happy to help with pointers as it would be a good investment of time. > >> I can write more later when I'm not on my phone, but: I'd suggest starting with implementing > >> @cython.earlybind > def f(x, y): ... > >> and have it turn to > >> cpdef f(x, y): ... > >> (note on the name below) > >> Just getting this working, with arguments and result typed to "object", should get your feet wet. Then I suppose the next step is implementing the Py3 PEP supporting decorators on arguments and return type. (I think one should aim for Py2.6+ here, though I suppose arguments to the decorator could work too: @cython.earlybind(x=cython.uint, localvar=cython.float)...opinions? > >I need it to be working with python2.4+, so I'll choose something that works in python2.4. Debian just switched to python2.5 recently, so >it's a long, long way before python2.6 could become a de facto >standard. But generally I agree one should design this so that things are especially easy in python2.6+. > >> > Brief scetch on the first step anyway (please ask about any hazy points and I'll give better help when I'm at a computer): > >> 1) The file you want to work in is ParseTreeTransforms.py. Your goal is to intercept decorated DefNode-s and transform them into CFuncDefNodes. Have a look and see how transforms are written... > >> 2) The ResolveDirectives transform (or something like that) already has code in it to intercept @cython.X-decorators on functions for compiler directives, I think you want to extend that transform, in time changing the name appropriately to give it a wider scope. > >> 3) the general workflow is this: Create a small testcase with both "from" code and "to" code as given above. Then add this function to ResolveDirectives (not entirely sure about the node class name, see Nodes.py): > >> def visit_CFuncDefNode(self, node): > print node.dump() > return node > >> this gives you an idea about the node structure you must convert to. > >> 5) To transform, edit the function visit_DefNode to also take into account "earlybind" (hardcode it) before checking the decorator against directives, and if so, simply set node = self.convert_def_to_cpdef(node), which you write. "node.dump()" here gives you the from-tree. > >> 6) if you fancy unit tests in addition to integration tests, have a look at TestWithTransform.py (or similar) for how to do it. You should literally be able to input the decorated def example above as a string and assert that the result after the transform is the expected cpdef version... though you may need to extend CodeWriter.py to serialize cpdef to strings. > >> Send an email whenever something is not obvious -- it would be really cool to see this in Cython. > >> Note on names: It is easily changeable so it is not a big point...but here goes: I think personally the cdef and cpdef are very cython-specific and communicate badly with python users. Also, considering cpdef exists, I think the need for cdef should be small? If wanted it could be called "@cython.nativeonly" perhaps. Anyway, "earlybind" also gets in the important message that you cannot reassign the symbol it binds to at runtime, which is a difference with how you expect a "def" to work. > >Thanks a lot for the help! I'll give it a shot and report back. > >Ondrej >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Fri Sep 26 20:02:41 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 26 Sep 2008 15:02:41 -0300 Subject: [Cython] isinstance and str (bytes/str/unicode again!) In-Reply-To: References: Message-ID: On Fri, Sep 26, 2008 at 2:36 PM, Christian Heimes wrote: > Lisandro Dalcin wrote: >> If we do 'isinstance(obj, str)', then what should be the behavior in >> Py2 and Py3? In fact, this question is actually related to the map in >> builtin_types_table at Cython/Compiler/Builtin.py. Do it make sense to >> do the map like this >> >> bytes -> PyBytes_Type >> str -> PyString_Type >> unicode -> PyUnicode_Type > > It doesn't make sense for some applications of str. Python 3.0 uses > PyUnicode for identifiers (attribute names, function and class names > etc.) while Python 2.x uses PyString. You can use PyUnicode instances > for attributes in Python 2.x through obj.__dict__[u"key"] = key but you > cannot use PyBytes in 3.0. > > This mapping makes more sense for me: > > bytes:: > whatever Python uses to store binary data > Python 2.x: PyString > Python 3.x: PyBytes > > unicode:: > text data > PyUnicode for all Python versions > > str:: > > Python 2.x: PyString > Python 3.x: PyUnicode Perhaps I was not clear enough, but the mapping in my mind is just the mapping you proposed. In the Python side, 'str' sould be the the type Python uses for attributes and names. In the C side, that would be 'PyString_Type'. Of course, Cython has to #define PyString_Type to PyUnicode_Type for Py3 (currently, PyString_Type is #define'd as PyBytes_Type for Py3). > The suggestion follows my forward compatibility code for Python 2.6. In > Python 2.6 b"" and bytes are simple aliases for PyString. "from > __future__ import unicode_literals" turns "" into PyUnicode objects. The only gotcha I have with all this is that I do not have an easy way to write a string literal that matchs what Python uses for identifiers, in such a way that it can run in both Py2 and Py3. Let's take an example: Suppose I want to create a new type calling 'type(name, bases, dict)', then I can write in Cython code: # Note: Cython code inside a pyx file type( b'A', (), {}) # -> fails in Py3 type( u'A', (), {}) # -> fails in Py2 type( 'A', (), {}) # -> fails in Py2 or Py3 depending on __future__ import So I really believe Cython needs an extension, let say type( s'A', (), {}) # Success in Py2 and Py3! Then the 's' prefix build a string with the whatever type the Python runtime uses for identifiers. What to you think, Christian? Do my later comments make sense for you? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Sep 26 20:31:47 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 26 Sep 2008 11:31:47 -0700 Subject: [Cython] cython license: PSF or Apache? In-Reply-To: <85b5c3130809260542s7fc1ff10q4407234231e5b051@mail.gmail.com> References: <85b5c3130809260542s7fc1ff10q4407234231e5b051@mail.gmail.com> Message-ID: <4AAADC27-DB6E-4501-A3DE-BE690D0ED211@math.washington.edu> On Sep 26, 2008, at 5:42 AM, Ondrej Certik wrote: > On Fri, Sep 26, 2008 at 8:34 AM, Robert Bradshaw > wrote: >> On Sep 25, 2008, at 7:33 PM, Kevin Ar18 wrote: >> >>> >>>> Sorry for the confusion. It is the Apache license, as we found out >>>> the PSF license can't be applied to anything but Python itself. If >>>> Cython becomes part of the standard Python distribution, then we >>>> can >>>> change to the PSF. >>>> >>>> I've changed LICENSE.txt t reflect this. >>>> >>>> - Robert >>> >>> There is also the option of a "generic PSF license" ... if that >>> might be preferred. :) >>> >>> Per http://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq >>> If you feel the PSF License is the one to use with your code, you >>> must change the following parts of the license: >>> >>> * Replace all occurrences of "Python Software Foundation" and >>> "PSF" with your name or organization. >>> * Replace "Python" with the name of your project. >> >> Yep. Actually, the reason we chose the license we did is on that same >> page: >> >> What if I want to contribute my code to the PSF? >> >> The PSF does not want contributions of any code other than that which >> will end up in Python or its standard libraries, or in Jython. If >> your code is going to end up in Python or the standard library, the >> PSF will require you to: >> >> * License your code under an acceptable open source license. >> These currently include only the Academic Free License and the Apache >> License 2.0, although this list may be expanded in the future. (No, >> the PSF License is not acceptable; see below) >> * Fill out and submit a contributor agreement. >> >> Though we haven't made any formal movements to have Cython as part of >> the Python standard library, this is certainly something we would >> like to consider in the future and we want to keep things open in >> that direction. Fortunately the Apache License is a fairly widely >> used and well known license, and matches well with our goals. > > pyprocessing, that was recently accepted, is BSD. And the license in > Lib/multiprocessing/__init__.py is BSD. So I don't understand much > this apache license requirement, but it seems to me that plain BSD is > enough. Yeah, me either, except that this was brought to our attention by some folks at python.org when we declared our code to be under PSF. The Apache is similar enough in spirit to BSD that I don't think it's worth worrying about unless someone has some strong concerns. - Robert From dagss at student.matnat.uio.no Sat Sep 27 00:30:00 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 27 Sep 2008 00:30:00 +0200 Subject: [Cython] isinstance and str (bytes/str/unicode again!) Message-ID: <3305320231.877679@smtp.netcom.no> Note that def s(x): return x if py3 else x.encode("...") fills the same purpose without changing Cython... (On a related note, we're getting closer to the long and intense discussion about such matters from earlier this year. If this leads to discussing the matter in a more fundamental way then that discussion should at least be properly summarized first.) Dag Sverre Seljebotn -----Original Message----- From: "Lisandro Dalcin" Date: Friday, Sep 26, 2008 8:02 pm Subject: Re: [Cython] isinstance and str (bytes/str/unicode again!) To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Fri, Sep 26, 2008 at 2:36 PM, Christian Heimes wrote: >> Lisandro Dalcin wrote: >>> If we do 'isinstance(obj, str)', then what should be the behavior in >>> Py2 and Py3? In fact, this question is actually related to the map in >>> builtin_types_table at Cython/Compiler/Builtin.py. Do it make sense to >>> do the map like this >>> >>> bytes -> PyBytes_Type >>> str -> PyString_Type >>> unicode -> PyUnicode_Type >> >> It doesn't make sense for some applications of str. Python 3.0 uses >> PyUnicode for identifiers (attribute names, function and class names >> etc.) while Python 2.x uses PyString. You can use PyUnicode instances >> for attributes in Python 2.x through obj.__dict__[u"key"] = key but you >> cannot use PyBytes in 3.0. >> >> This mapping makes more sense for me: >> >> bytes:: >> whatever Python uses to store binary data >> Python 2.x: PyString >> Python 3.x: PyBytes >> >> unicode:: >> text data >> PyUnicode for all Python versions >> >> str:: >> >> Python 2.x: PyString >> Python 3.x: PyUnicode > >Perhaps I was not clear enough, but the mapping in my mind is just the >mapping you proposed. > >In the Python side, 'str' sould be the the type Python uses for >attributes and names. In the C side, that would be 'PyString_Type'. >Of course, Cython has to #define PyString_Type to PyUnicode_Type for >Py3 (currently, PyString_Type is #define'd as PyBytes_Type for Py3). > > >> The suggestion follows my forward compatibility code for Python 2.6. In >> Python 2.6 b"" and bytes are simple aliases for PyString. "from >> __future__ import unicode_literals" turns "" into PyUnicode objects. > >The only gotcha I have with all this is that I do not have an easy way >to write a string literal that matchs what Python uses for >identifiers, in such a way that it can run in both Py2 and Py3. Let's >take an example: Suppose I want to create a new type calling >'type(name, bases, dict)', then I can write in Cython code: > ># Note: Cython code inside a pyx file >type( b'A', (), {}) # -> fails in Py3 > >type( u'A', (), {}) # -> fails in Py2 > >type( 'A', (), {}) # -> fails in Py2 or Py3 depending on __future__ import > >So I really believe Cython needs an extension, let say > >type( s'A', (), {}) # Success in Py2 and Py3! > >Then the 's' prefix build a string with the whatever type the Python >runtime uses for identifiers. > >What to you think, Christian? Do my later comments make sense for you? > > > > > > >-- >Lisandro Dalc?n >--------------- >Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >PTLC - G?emes 3450, (3000) Santa Fe, Argentina >Tel/Fax: +54-(0)342-451.1594 >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From kevinar18 at hotmail.com Sat Sep 27 03:12:23 2008 From: kevinar18 at hotmail.com (Kevin Ar18) Date: Fri, 26 Sep 2008 21:12:23 -0400 Subject: [Cython] The basics? Message-ID: Well it finally works. Now, I gotta learn it... but one problem: I cannot find a basics tutorial/intro -- the kind that says here's how you take your .py app and make cython compile and finally run it. There's the Demos, but that doesn't explain what anything is. There's the wiki, but it skips over the "how to take a .py file and make cython compile and finally run it" and instead talks about how to start editing and optomizing .py/.pyx files after they are setup. I can read code to distutils.core, distutils.extension, Cython.Distutils and their python docs to see if I can figure it out, but that's not nearly as effective as a real tutorial. Am I overlooking something? _________________________________________________________________ Stay up to date on your PC, the Web, and your mobile phone with Windows Live. http://clk.atdmt.com/MRT/go/msnnkwxp1020093185mrt/direct/01/ From dfdeshom at gmail.com Sat Sep 27 03:19:46 2008 From: dfdeshom at gmail.com (didier deshommes) Date: Fri, 26 Sep 2008 21:19:46 -0400 Subject: [Cython] The basics? In-Reply-To: References: Message-ID: <4db014670809261819o3dd6a7f1t3af2dafa98393994@mail.gmail.com> On Fri, Sep 26, 2008 at 9:12 PM, Kevin Ar18 wrote: > > Well it finally works. > > Now, I gotta learn it... but one problem: I cannot find a basics tutorial/intro -- the Have you looked at http://ldots.org/pyrex-guide/ ? didier From robertwb at math.washington.edu Sat Sep 27 03:41:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 26 Sep 2008 18:41:14 -0700 Subject: [Cython] The basics? In-Reply-To: <4db014670809261819o3dd6a7f1t3af2dafa98393994@mail.gmail.com> References: <4db014670809261819o3dd6a7f1t3af2dafa98393994@mail.gmail.com> Message-ID: <07C18849-135C-4FB9-A557-4A33E82A9ADA@math.washington.edu> On Sep 26, 2008, at 6:19 PM, didier deshommes wrote: > On Fri, Sep 26, 2008 at 9:12 PM, Kevin Ar18 > wrote: >> >> Well it finally works. >> >> Now, I gotta learn it... but one problem: I cannot find a basics >> tutorial/intro -- the > > Have you looked at http://ldots.org/pyrex-guide/ ? That is a good resource. I would recommend, looking at the slides for some of the slides for talks I've given too (the most recent SciPy 2008). Like most programming languages, the easiest way to learn it is from a huge variety of actual code. - Robert From simon at arrowtheory.com Sat Sep 27 05:18:06 2008 From: simon at arrowtheory.com (Simon Burton) Date: Fri, 26 Sep 2008 23:18:06 -0400 Subject: [Cython] The basics? In-Reply-To: References: Message-ID: <20080926231806.d2362a57.simon@arrowtheory.com> Go to the pyrex website. There are some differences, but the majority of the documentation is still the pyrex docs. Simon. On Fri, 26 Sep 2008 21:12:23 -0400 Kevin Ar18 wrote: > > Well it finally works. > > Now, I gotta learn it... but one problem: I cannot find a basics tutorial/intro -- the kind that says here's how you take your .py app and make cython compile and finally run it. > > There's the Demos, but that doesn't explain what anything is. > There's the wiki, but it skips over the "how to take a .py file and make cython compile and finally run it" and instead talks about how to start editing and optomizing .py/.pyx files after they are setup. > I can read code to distutils.core, distutils.extension, Cython.Distutils and their python docs to see if I can figure it out, but that's not nearly as effective as a real tutorial. > > Am I overlooking something? > _________________________________________________________________ > Stay up to date on your PC, the Web, and your mobile phone with Windows Live. > http://clk.atdmt.com/MRT/go/msnnkwxp1020093185mrt/direct/01/ > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From lists at cheimes.de Sat Sep 27 15:13:51 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 27 Sep 2008 15:13:51 +0200 Subject: [Cython] Call for help: official C-API 2 to 3 porting guide Message-ID: Hello fellow developers! Benjamin is working on a C-API 2 to 3 porting guide for the official Python docs. I know that you - the Cython developers - have spent lots of time on making Cython compatible with 2.3+ as well as 3.0. Could you please assist Benjamin with his task and give him some hints? We are especially interested in a list of gotchas and tricks how to write cross version compatible C code. Thanks in advance! Christian From lists at cheimes.de Sat Sep 27 15:39:33 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 27 Sep 2008 15:39:33 +0200 Subject: [Cython] isinstance and str (bytes/str/unicode again!) In-Reply-To: References: Message-ID: Lisandro Dalcin wrote: > Perhaps I was not clear enough, but the mapping in my mind is just the > mapping you proposed. Oh well, ok. :) > In the Python side, 'str' sould be the the type Python uses for > attributes and names. In the C side, that would be 'PyString_Type'. > Of course, Cython has to #define PyString_Type to PyUnicode_Type for > Py3 (currently, PyString_Type is #define'd as PyBytes_Type for Py3). The proposal could cause confusion. Python 3.0's PyBytes type is a slightly modified and renamed version of Python 2.0's PyString type. A few months ago the PyBytes_* functions were still called PyString_* until I renamed them. Does Cython provide aliases for PyBytes -> PyString for all versions of Python 2.x? It would be helpful for Python 3.0 compatible code. [...] > What to you think, Christian? Do my later comments make sense for you? I see your problem ... I've to think about the implications more carefully. Christian From ggellner at uoguelph.ca Sat Sep 27 16:49:13 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 27 Sep 2008 10:49:13 -0400 Subject: [Cython] Documentation Message-ID: <20080927144913.GA20932@encolpuis> Well seeing the need I am writing up a tutorial for simple compilation and usage for my Sphinx docs. This leads me to update some of the other pages to reflect recent changes, and I would like some feedback to ensure I understand everything! * CEP 106 (Package pxds): should I update the wiki (and add to the docs) that we have a bunch of basic .pxd's now? Or are we waiting for something? * CEP 301 (from blah import *): says DONE, is this true in the released version, it seems to work for me using the PyPI version, but I never use this feature, so I don't know if I am missing something. * CEP 302 (with statement): again status is DONE, again seems to work for me, but the wiki still says this is a limitation, is there a go ahead to remove this from the limitation wiki? Finally is the current wiki state describing the differences between Cython and Pyrex true? It seems Pyrex has done a lot of updating, and that it has some features know that Cython doesn't (I am thinking of the h5py code, something about type checking ... I haven't really been following). If anyone has knowledge of this could they give the wiki page a once over, I would then make the changes to the Sphinx docs. Thanks, Gabriel From ggellner at uoguelph.ca Sat Sep 27 16:59:11 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 27 Sep 2008 10:59:11 -0400 Subject: [Cython] Documentation In-Reply-To: <20080927144913.GA20932@encolpuis> References: <20080927144913.GA20932@encolpuis> Message-ID: <20080927145911.GB20932@encolpuis> Also is the following points still all True? * The :func:``globals`` and :func:``locals`` functions cannot be used. * Class and function definitions cannot be placed inside control structures. * There is no support for Unicode. # I believe this isn't true . . . definitive answer? * Special methods of extension types cannot have functioning docstrings. * The use of string literals as comments is not recommended at present, because Cython doesn't optimize them away, and won't even accept them in places where executable statements are not allowed. Semantic differences between Python and Cython ---------------------------------------------- Behaviour of class scopes ^^^^^^^^^^^^^^^^^^^^^^^^^ In Python, referring to a method of a class inside the class definition, i.e. while the class is being defined, yields a plain function object, but in Cython it yields an unbound method [#]_. A consequence of this is that the usual idiom for using the ``classmethod`` and ``staticmethod`` functions, e.g.:: class Spam: def method(cls): ... method = classmethod(method) will not work in Cython. This can be worked around by defining the function outside the class, and then assigning the result of ``classmethod`` or ``staticmethod`` inside the class, i.e.:: def Spam_method(cls): ... class Spam: method = classmethod(Spam_method) From ggellner at uoguelph.ca Sat Sep 27 17:03:25 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 27 Sep 2008 11:03:25 -0400 Subject: [Cython] Documentation In-Reply-To: <20080927144913.GA20932@encolpuis> References: <20080927144913.GA20932@encolpuis> Message-ID: <20080927150325.GA21172@encolpuis> At http://wiki.cython.org/Unsupported optimized support for builtin types is listed, shouldn't this be in a CEP instead? It is not really an unsupported python feature, just not a perfectly implemented feature . . . Also should I add a quick line that function closures are not supported, so that all the other pages that list this as needed are more complete? Gabriel From dagss at student.matnat.uio.no Sat Sep 27 17:27:29 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 27 Sep 2008 17:27:29 +0200 (CEST) Subject: [Cython] Documentation In-Reply-To: <20080927144913.GA20932@encolpuis> References: <20080927144913.GA20932@encolpuis> Message-ID: <52388.85.165.152.85.1222529249.squirrel@webmail.uio.no> Gabriel Gellner wrote: > Well seeing the need I am writing up a tutorial for simple compilation and > usage for my Sphinx docs. This leads me to update some of the other pages > to > reflect recent changes, and I would like some feedback to ensure I > understand > everything! Note that I wrote some paragraphs on simple compilation etc. in my NumPy tutorial (wiki.cython.org/tutorials/numpy). Question: Can I as a Cython developer modify these docs? For instance, take the compilerdirectives docs currently at http://wiki.cython.org/enhancements/compilerdirectives where should I have added this? Similar for documenting the behaviour at http://wiki.cython.org/enhancements/buffer. Ideally I'd like either a) a "docs" subdirectory in the Cython repository, or b) a wiki-like interface to the official docs. (Is there a Sphinx plugin to MoinMoin perhaps?) a) has the advantage that it is simple and means that documentation is versioned together with the sources which has many advantages (easy to check whether the "with" statement was supported in version X). A single commit that adds a feature would then change the docs as well. (At least last time I checked, the docs weren't in the repo...) > * CEP 106 (Package pxds): should I update the wiki (and add to the docs) > that we have a bunch > of basic .pxd's now? Or are we waiting for something? Sounds good. > * CEP 302 (with statement): again status is DONE, again seems to work for > me, > but the wiki still says this is a limitation, is there a go ahead to > remove this from the limitation wiki? This is OK now, go. BTW, great that you are doing this, documentation has a way of falling behind... Dag Sverre From fperez.net at gmail.com Sat Sep 27 20:49:53 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 27 Sep 2008 11:49:53 -0700 Subject: [Cython] Documentation In-Reply-To: <20080927144913.GA20932@encolpuis> References: <20080927144913.GA20932@encolpuis> Message-ID: Howdy, On Sat, Sep 27, 2008 at 7:49 AM, Gabriel Gellner wrote: > Well seeing the need I am writing up a tutorial for simple compilation and > usage for my Sphinx docs. This leads me to update some of the other pages to > reflect recent changes, and I would like some feedback to ensure I understand > everything! BTW, sorry if I missed an earlier announcement... Are your docs/tutorials available online yet? Do you plan on making them so? Cheers, f From ggellner at uoguelph.ca Sat Sep 27 20:52:14 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 27 Sep 2008 14:52:14 -0400 Subject: [Cython] Check out the most recent docs . . . Message-ID: <20080927185214.GA24136@encolpuis> Well I have done some major updating to the Sphinx docs located at: www.mudskipper.ca/cython-doc/ (and under cython mercurial at: hg.cython.org/cython-docs) Highlights: * The very start of a tutorial (much more work will be done over the next week) * Added Dag's Numpy Tutorial * Basic Pygments Cython highlighter (still some bugs, but I will iron them out), so Cython is highlighted! * Ipython highlighter added from the matplotlib guys (check out the numpy tutorial) * Begun giving some real information on how to write setup.py files in 'Source files and Compilation'. * Removed some crufty information from sections like limitations. Short run plans: * Finish the Tutorial, relying heavily on Robert's scipy08 talk. * Finish the setup.py examples * Find all old Pyrex'y information and remove it. * Document that range(n) expansion is the default. As always tell me what you think. Gabriel From robertwb at math.washington.edu Sat Sep 27 21:16:21 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 27 Sep 2008 12:16:21 -0700 Subject: [Cython] The basics? In-Reply-To: <07C18849-135C-4FB9-A557-4A33E82A9ADA@math.washington.edu> References: <4db014670809261819o3dd6a7f1t3af2dafa98393994@mail.gmail.com> <07C18849-135C-4FB9-A557-4A33E82A9ADA@math.washington.edu> Message-ID: On Sep 26, 2008, at 6:41 PM, Robert Bradshaw wrote: > On Sep 26, 2008, at 6:19 PM, didier deshommes wrote: > >> On Fri, Sep 26, 2008 at 9:12 PM, Kevin Ar18 >> wrote: >>> >>> Well it finally works. >>> >>> Now, I gotta learn it... but one problem: I cannot find a basics >>> tutorial/intro -- the >> >> Have you looked at http://ldots.org/pyrex-guide/ ? > > That is a good resource. I would recommend, looking at the slides for > some of the slides for talks I've given too (the most recent SciPy > 2008). Like most programming languages, the easiest way to learn it > is from a huge variety of actual code. And here's the URL for the docs that I was thinking of, but couldn't remember where they were: http://www.mudskipper.ca/cython-doc/ (We really need to feature these more prominently.) - Robert From robertwb at math.washington.edu Sat Sep 27 21:28:51 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 27 Sep 2008 12:28:51 -0700 Subject: [Cython] Check out the most recent docs . . . In-Reply-To: <20080927185214.GA24136@encolpuis> References: <20080927185214.GA24136@encolpuis> Message-ID: <9528A195-FA6B-46D7-B103-27018B5D933C@math.washington.edu> On Sep 27, 2008, at 11:52 AM, Gabriel Gellner wrote: > Well I have done some major updating to the Sphinx docs located at: > > www.mudskipper.ca/cython-doc/ > > (and under cython mercurial at: > hg.cython.org/cython-docs) Thank you again so much! Improving the docs is something that has been on my todolist for a long time, but never quite made it to the top. > Highlights: > > * The very start of a tutorial (much more work will be done over > the next > week) > * Added Dag's Numpy Tutorial > * Basic Pygments Cython highlighter (still some bugs, but I will > iron them > out), so Cython is highlighted! > * Ipython highlighter added from the matplotlib guys (check out the > numpy > tutorial) > * Begun giving some real information on how to write setup.py files in > 'Source files and Compilation'. > * Removed some crufty information from sections like limitations. It's looking good. The tutorial part is especially useful for beginners. One comment about highlighted, it might be nice to have a highlighted style where the new keywords (cdef, cimport, ...) stand out. People often find this useful. > Short run plans: > > * Finish the Tutorial, relying heavily on Robert's scipy08 talk. > * Finish the setup.py examples > * Find all old Pyrex'y information and remove it. > * Document that range(n) expansion is the default. > > As always tell me what you think. While it's good to clean up the inconsistent (and sometimes confusing) use of the name Pyrex vs. Cython, I feel strongly that we need to credit Pyrex and all that Greg Ewing did (and maybe a bit about our relationship with that project), preferably right there on the first page in the overview. - Robert From robertwb at math.washington.edu Sat Sep 27 21:29:20 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 27 Sep 2008 12:29:20 -0700 Subject: [Cython] Documentation In-Reply-To: References: <20080927144913.GA20932@encolpuis> Message-ID: <8C0FFC5D-C27B-44D2-B5B6-1AD2EE669945@math.washington.edu> On Sep 27, 2008, at 11:49 AM, Fernando Perez wrote: > Howdy, > > On Sat, Sep 27, 2008 at 7:49 AM, Gabriel Gellner > wrote: >> Well seeing the need I am writing up a tutorial for simple >> compilation and >> usage for my Sphinx docs. This leads me to update some of the >> other pages to >> reflect recent changes, and I would like some feedback to ensure I >> understand >> everything! > > BTW, sorry if I missed an earlier announcement... Are your > docs/tutorials available online yet? Do you plan on making them so? http://www.mudskipper.ca/cython-doc/ From fperez.net at gmail.com Sat Sep 27 21:32:22 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 27 Sep 2008 12:32:22 -0700 Subject: [Cython] The basics? In-Reply-To: References: <4db014670809261819o3dd6a7f1t3af2dafa98393994@mail.gmail.com> <07C18849-135C-4FB9-A557-4A33E82A9ADA@math.washington.edu> Message-ID: On Sat, Sep 27, 2008 at 12:16 PM, Robert Bradshaw wrote: > And here's the URL for the docs that I was thinking of, but couldn't > remember where they were: http://www.mudskipper.ca/cython-doc/ (We > really need to feature these more prominently.) I just helped a little :) I advertised them on the numpy list and on our UC Berkeley py4science internal list. Great job guys, and thanks! Cheers, f From robertwb at math.washington.edu Sat Sep 27 22:00:09 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 27 Sep 2008 13:00:09 -0700 Subject: [Cython] Documentation In-Reply-To: <20080927144913.GA20932@encolpuis> References: <20080927144913.GA20932@encolpuis> Message-ID: On Sep 27, 2008, at 7:49 AM, Gabriel Gellner wrote: > Well seeing the need I am writing up a tutorial for simple > compilation and > usage for my Sphinx docs. This leads me to update some of the other > pages to > reflect recent changes, and I would like some feedback to ensure I > understand > everything! > > * CEP 106 (Package pxds): should I update the wiki (and add to the > docs) that we have a bunch > of basic .pxd's now? Or are we waiting for something? Yep, this is done. > * CEP 301 (from blah import *): says DONE, is this true in the > released > version, it seems to work for me using the PyPI version, but I > never use > this feature, so I don't know if I am missing something. This is nearly done, short of this bug http://trac.cython.org/ cython_trac/ticket/73 > * CEP 302 (with statement): again status is DONE, again seems to > work for me, > but the wiki still says this is a limitation, is there a go > ahead to > remove this from the limitation wiki? Yep, done. > Finally is the current wiki state describing the differences > between Cython > and Pyrex true? It seems Pyrex has done a lot of updating, and that > it has > some features know that Cython doesn't (I am thinking of the h5py > code, > something about type checking ... I haven't really been following). > If anyone > has knowledge of this could they give the wiki page a once over, I > would then > make the changes to the Sphinx docs. I did a quick pass on that page to get it (more) up to date. I am sure there's stuff in Cython not in Pyrex not on that page, but I've removed the things that Pyrex caught up on. On Sep 27, 2008, at 7:59 AM, Gabriel Gellner wrote: > Also is the following points still all True? > > * The :func:``globals`` and :func:``locals`` functions cannot be used. Yes. > * Class and function definitions cannot be placed inside control > structures. Unfortunately... > * There is no support for Unicode. # I believe this isn't > true . . . definitive answer? I believe we support it fully now, but Stefan would be able to give the definitive answer. > * Special methods of extension types cannot have functioning > docstrings. They can now. > * The use of string literals as comments is not recommended at > present, > because Cython doesn't optimize them away, and won't even accept > them in > places where executable statements are not allowed. They can now. On Sep 27, 2008, at 8:03 AM, Gabriel Gellner wrote: > At > http://wiki.cython.org/Unsupported > > optimized support for builtin types is listed, shouldn't this be in > a CEP > instead? It is not really an unsupported python feature, just not a > perfectly > implemented feature . . . True, but they're supported now. > Also should I add a quick line that function closures are not > supported, so > that all the other pages that list this as needed are more complete? I updated http://wiki.cython.org/Unsupported with these points (and those above). Thanks again for putting this together. - Robert From robertwb at math.washington.edu Sat Sep 27 22:08:49 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 27 Sep 2008 13:08:49 -0700 Subject: [Cython] Documentation In-Reply-To: <52388.85.165.152.85.1222529249.squirrel@webmail.uio.no> References: <20080927144913.GA20932@encolpuis> <52388.85.165.152.85.1222529249.squirrel@webmail.uio.no> Message-ID: <50E0FD1C-0E09-488C-A133-CE52D01D23E6@math.washington.edu> On Sep 27, 2008, at 8:27 AM, Dag Sverre Seljebotn wrote: > Gabriel Gellner wrote: >> Well seeing the need I am writing up a tutorial for simple >> compilation and >> usage for my Sphinx docs. This leads me to update some of the >> other pages >> to >> reflect recent changes, and I would like some feedback to ensure I >> understand >> everything! > > Note that I wrote some paragraphs on simple compilation etc. in my > NumPy > tutorial (wiki.cython.org/tutorials/numpy). > > Question: Can I as a Cython developer modify these docs? The repository is up at http://hg.cython.org/cython-docs/ , so you can make changes, submit patches/bundles, etc. If Gabriel is willing, I could give you push access. > For instance, take the compilerdirectives docs currently at > > http://wiki.cython.org/enhancements/compilerdirectives > > where should I have added this? Similar for documenting the > behaviour at > http://wiki.cython.org/enhancements/buffer. > > Ideally I'd like either a) a "docs" subdirectory in the Cython > repository, That does make sense. We do have a Doc subdirectory already, but it's not near as up to date (or as nice looking). > or b) a wiki-like interface to the official docs. (Is there a Sphinx > plugin to MoinMoin perhaps?) The "all documentation is on the wiki" approach hasn't (yet) yielded the most helpful docs so far, which is why I'm so enthusiastic about this project. Both certainly have their place, but having a (more) stable set of refined docs is very valuable. The process of migrating information from the wiki into the docs themselves (as in this thread) is a useful one to get something polished and accurate. > a) has the advantage that it is simple and means that documentation is > versioned together with the sources which has many advantages (easy to > check whether the "with" statement was supported in version X). A > single > commit that adds a feature would then change the docs as well. [...] > BTW, great that you are doing this, documentation has a way of falling > behind... Unless someone keeps it up :). Most of the basic stuff isn't changing though, and that's often the hardest stuff for new users. - Robert From ggellner at uoguelph.ca Sat Sep 27 22:40:36 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 27 Sep 2008 16:40:36 -0400 Subject: [Cython] Documentation In-Reply-To: <50E0FD1C-0E09-488C-A133-CE52D01D23E6@math.washington.edu> References: <20080927144913.GA20932@encolpuis> <52388.85.165.152.85.1222529249.squirrel@webmail.uio.no> <50E0FD1C-0E09-488C-A133-CE52D01D23E6@math.washington.edu> Message-ID: <20080927204036.GA25437@encolpuis> On Sat, Sep 27, 2008 at 01:08:49PM -0700, Robert Bradshaw wrote: > On Sep 27, 2008, at 8:27 AM, Dag Sverre Seljebotn wrote: > > > Gabriel Gellner wrote: > >> Well seeing the need I am writing up a tutorial for simple > >> compilation and > >> usage for my Sphinx docs. This leads me to update some of the > >> other pages > >> to > >> reflect recent changes, and I would like some feedback to ensure I > >> understand > >> everything! > > > > Note that I wrote some paragraphs on simple compilation etc. in my > > NumPy > > tutorial (wiki.cython.org/tutorials/numpy). > > > > Question: Can I as a Cython developer modify these docs? > > The repository is up at http://hg.cython.org/cython-docs/ , so you > can make changes, submit patches/bundles, etc. If Gabriel is willing, > I could give you push access. > I am very willing! I am happy to take the lead on this, but I think the more the merrier, as far as I am concerned all dev's should have push access. Gabriel From kevinar18 at hotmail.com Sun Sep 28 02:04:15 2008 From: kevinar18 at hotmail.com (Kevin Ar18) Date: Sat, 27 Sep 2008 20:04:15 -0400 Subject: [Cython] The basics? In-Reply-To: References: <4db014670809261819o3dd6a7f1t3af2dafa98393994@mail.gmail.com> <07C18849-135C-4FB9-A557-4A33E82A9ADA@math.washington.edu> Message-ID: > On Sep 26, 2008, at 6:41 PM, Robert Bradshaw wrote: > >> On Sep 26, 2008, at 6:19 PM, didier deshommes wrote: >> >>> On Fri, Sep 26, 2008 at 9:12 PM, Kevin Ar18 >>> wrote: >>>> >>>> Well it finally works. >>>> >>>> Now, I gotta learn it... but one problem: I cannot find a basics >>>> tutorial/intro -- the >>> >>> Have you looked at http://ldots.org/pyrex-guide/ ? >> >> That is a good resource. I would recommend, looking at the slides for >> some of the slides for talks I've given too (the most recent SciPy >> 2008). Like most programming languages, the easiest way to learn it >> is from a huge variety of actual code. > > And here's the URL for the docs that I was thinking of, but couldn't > remember where they were: http://www.mudskipper.ca/cython-doc/ (We > really need to feature these more prominently.) > > - Robert Looks good. That's probably what I need. Thanks _________________________________________________________________ Want to do more with Windows Live? Learn ?10 hidden secrets? from Jamie. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008 From ggellner at uoguelph.ca Sun Sep 28 17:41:00 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sun, 28 Sep 2008 11:41:00 -0400 Subject: [Cython] ieee 754 best practices Message-ID: <20080928154100.GA30881@encolpuis> I need to deal with some floating point operations and I want to have inf, -inf, nan, etc with functions like is_inf(x) is_nan(x) etc Are there any best practices for doing this in Cython, I have some old C code that I could just convert into a pxd file, but I am not sure if there is a idomatic/standard way. Pointers to code that already deals with this (in Sage maybe) would be greatly appreciated. Gabriel From lists at cheimes.de Sun Sep 28 20:45:12 2008 From: lists at cheimes.de (Christian Heimes) Date: Sun, 28 Sep 2008 20:45:12 +0200 Subject: [Cython] ieee 754 best practices In-Reply-To: <20080928154100.GA30881@encolpuis> References: <20080928154100.GA30881@encolpuis> Message-ID: Gabriel Gellner wrote: > Are there any best practices for doing this in Cython, I have some old C code > that I could just convert into a pxd file, but I am not sure if there is a > idomatic/standard way. > > Pointers to code that already deals with this (in Sage maybe) would be greatly > appreciated. I don't think that Cython has special code for IEEE 754 code. But you can grab Python 2.6 sources and look how Mark and I have implemented the IEEE 754 fixes for floats and complex. The interesting parts are in Include/pymath.h, pyconfig.h, PC/pyconfig.h, Objects/floatobject.c, Modules/mathmodule.c and Modules/cmathmodule.c The changes are heavily based on the draft of the C99 standard, WG14/N1124 Annex F and G. Christian From amirnntp at gmail.com Mon Sep 29 08:40:47 2008 From: amirnntp at gmail.com (Amir) Date: Mon, 29 Sep 2008 06:40:47 +0000 (UTC) Subject: [Cython] embedding Cython and wiki link Message-ID: I would like to create a simple embedded cython executable like the example in the Python docs (5. Embedding Python in Another Application). I started with the Cython wiki link pointing to David McNab's pyrex embedding tutorial but am not sure how to proceed. Was wondering if anyone had done this with a recent Cython version. The python pxd's don't seem to have Py_Initialize, Py_Finalize, PySys_SetArgv used in David's tutorial. I'm guessing there is a good reason for this. And I am wondering if there is a way to use Cython.distutils to build the executable. Any pointers to docs or an example application would be appreciated. Thanks, Amir. From ondrej at certik.cz Mon Sep 29 15:30:31 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 29 Sep 2008 15:30:31 +0200 Subject: [Cython] Documentation In-Reply-To: <20080927144913.GA20932@encolpuis> References: <20080927144913.GA20932@encolpuis> Message-ID: <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> On Sat, Sep 27, 2008 at 4:49 PM, Gabriel Gellner wrote: > Well seeing the need I am writing up a tutorial for simple compilation and > usage for my Sphinx docs. This leads me to update some of the other pages to > reflect recent changes, and I would like some feedback to ensure I understand > everything! > > * CEP 106 (Package pxds): should I update the wiki (and add to the docs) that we have a bunch > of basic .pxd's now? Or are we waiting for something? > > * CEP 301 (from blah import *): says DONE, is this true in the released > version, it seems to work for me using the PyPI version, but I never use > this feature, so I don't know if I am missing something. > > * CEP 302 (with statement): again status is DONE, again seems to work for me, > but the wiki still says this is a limitation, is there a go ahead to > remove this from the limitation wiki? > > Finally is the current wiki state describing the differences between Cython > and Pyrex true? It seems Pyrex has done a lot of updating, and that it has > some features know that Cython doesn't (I am thinking of the h5py code, > something about type checking ... I haven't really been following). If anyone > has knowledge of this could they give the wiki page a once over, I would then > make the changes to the Sphinx docs. Could this please be hosted at docs.cython.org? I use yours sphinx docs all the time and it is just excellent! You've then really, really great job. Thanks, Ondrej From ivanov.maxim at gmail.com Mon Sep 29 15:44:56 2008 From: ivanov.maxim at gmail.com (Max Ivanov) Date: Mon, 29 Sep 2008 17:44:56 +0400 Subject: [Cython] How to return char* Message-ID: I contsuct string inside function, how could I return it? ---------- 20 cdef char* close(self): 21 return ''.join(self._data) --------- If I copy it to temporary "p" as docs says, it wouldn't help I suppose, because p will not be referenced by anyone. The only option is to return python object? From dagss at student.matnat.uio.no Mon Sep 29 16:08:46 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 29 Sep 2008 16:08:46 +0200 Subject: [Cython] How to return char* In-Reply-To: References: Message-ID: <48E0E16E.8020003@student.matnat.uio.no> Max Ivanov wrote: > I contsuct string inside function, how could I return it? > ---------- > 20 cdef char* close(self): > 21 return ''.join(self._data) > --------- > > If I copy it to temporary "p" as docs says, it wouldn't help I > suppose, because p will not be referenced by anyone. The only option > is to return python object? This is really a matter of working with C, and not so much a Cython question. A char* is just a pointer to some memory, and the question of managing that memory is an important question but will vary from occasion to occasion. As what you are trying to do doesn't make so much sense one must have more context to give a better answer -- i.e. what are you trying to achieve overall, not only this snippet. I.e: 1) You want to use char* because it is feels "faster" than Python objects: No, you're better off with Python objects, char* is fundamentally something different. 2) The string is always a constant, but generated run-time: Assign it to a module global variable. 3) Something else: Perhaps allocate the memory manually and pass the target char* /in/ to the close func, and then write to it, rather than returning something... and so on. If you provide more details on why you want to do this perhaps we can give a better answer. Dag Sverre From dalcinl at gmail.com Mon Sep 29 16:27:51 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Sep 2008 11:27:51 -0300 Subject: [Cython] isinstance and str (bytes/str/unicode again!) In-Reply-To: References: Message-ID: On Sat, Sep 27, 2008 at 10:39 AM, Christian Heimes wrote: >> In the Python side, 'str' sould be the the type Python uses for >> attributes and names. In the C side, that would be 'PyString_Type'. >> Of course, Cython has to #define PyString_Type to PyUnicode_Type for >> Py3 (currently, PyString_Type is #define'd as PyBytes_Type for Py3). > > The proposal could cause confusion. Python 3.0's PyBytes type is a > slightly modified and renamed version of Python 2.0's PyString type. A > few months ago the PyBytes_* functions were still called PyString_* > until I renamed them. Yes, but now the Python C-API has finally stabilized, right?. So this should not make confusion in the near future when users have the first final release of Python 3.0. > Does Cython provide aliases for PyBytes -> PyString for all versions of > Python 2.x? It would be helpful for Python 3.0 compatible code. Yes, more or less. It also supports the b"" prefix for all Python versions, and also the u"" prefix (unfortunatelly unavailable in Py3). The 'unicode' builting is associated with PyUnicode_Type, I'm working on a (few lines) pach for associating the 'bytes' builtin as appropriate. Was has to be discussed a bit more is the association of the 'str' builtin; currently, it is an alias por 'bytes' in Py3, and I'm not confortable with that. >> What to you think, Christian? Do my later comments make sense for you? > > I see your problem ... I've to think about the implications more carefully. > OK. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ggellner at uoguelph.ca Mon Sep 29 16:28:13 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Mon, 29 Sep 2008 10:28:13 -0400 Subject: [Cython] Documentation In-Reply-To: <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> Message-ID: <20080929142813.GA8804@encolpuis> > Could this please be hosted at docs.cython.org? > > I use yours sphinx docs all the time and it is just excellent! You've > then really, really great job. > I am not sure what is involved in this, but I would be happy to have them hosted on the cython site, I use my site just because it was easy, and I wasn't sure what the response would be. Do the server admins think this is feasible? If we want I could have the built html pages put under the hg.cython versioning, and we could just do a url rename to point to this? Anyway, I am open to any ideas. Gabriel From ivanov.maxim at gmail.com Mon Sep 29 16:29:03 2008 From: ivanov.maxim at gmail.com (Max Ivanov) Date: Mon, 29 Sep 2008 18:29:03 +0400 Subject: [Cython] How to return char* In-Reply-To: <48E0E16E.8020003@student.matnat.uio.no> References: <48E0E16E.8020003@student.matnat.uio.no> Message-ID: > 1) You want to use char* because it is feels "faster" than Python > objects: That was the reason >No, you're better off with Python objects, char* is > fundamentally something different. And this is the answer. Thanks a lot =) From ggellner at uoguelph.ca Mon Sep 29 16:33:47 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Mon, 29 Sep 2008 10:33:47 -0400 Subject: [Cython] ieee 754 best practices In-Reply-To: References: <20080928154100.GA30881@encolpuis> Message-ID: <20080929143347.GA8847@encolpuis> On Sun, Sep 28, 2008 at 08:45:12PM +0200, Christian Heimes wrote: > Gabriel Gellner wrote: > > Are there any best practices for doing this in Cython, I have some old C code > > that I could just convert into a pxd file, but I am not sure if there is a > > idomatic/standard way. > > > > Pointers to code that already deals with this (in Sage maybe) would be greatly > > appreciated. > > I don't think that Cython has special code for IEEE 754 code. But you > can grab Python 2.6 sources and look how Mark and I have implemented the > IEEE 754 fixes for floats and complex. The interesting parts are in > Include/pymath.h, pyconfig.h, PC/pyconfig.h, Objects/floatobject.c, > Modules/mathmodule.c and Modules/cmathmodule.c > > The changes are heavily based on the draft of the C99 standard, > WG14/N1124 Annex F and G. > Thanks for the pointer. I will look at this this weekend. Thanks for doing this for python! Gabriel From ondrej at certik.cz Mon Sep 29 16:35:59 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 29 Sep 2008 16:35:59 +0200 Subject: [Cython] Documentation In-Reply-To: <20080929142813.GA8804@encolpuis> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> Message-ID: <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> On Mon, Sep 29, 2008 at 4:28 PM, Gabriel Gellner wrote: >> Could this please be hosted at docs.cython.org? >> >> I use your sphinx docs all the time and it is just excellent! You've >> done really, really great job. >> > I am not sure what is involved in this, but I would be happy to have them > hosted on the cython site, I use my site just because it was easy, and I > wasn't sure what the response would be. > > Do the server admins think this is feasible? If we want I could have the built > html pages put under the hg.cython versioning, and we could just do a url > rename to point to this? > > Anyway, I am open to any ideas. Technically I think just Robert or William needs to setup dns redirection and add a new rule to apache. Socially, let's see if other people also like your docs as the default docs for Cython. Ondrej From dalcinl at gmail.com Mon Sep 29 16:41:31 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Sep 2008 11:41:31 -0300 Subject: [Cython] isinstance and str (bytes/str/unicode again!) In-Reply-To: <3305320231.877679@smtp.netcom.no> References: <3305320231.877679@smtp.netcom.no> Message-ID: On Fri, Sep 26, 2008 at 7:30 PM, Dag Sverre Seljebotn wrote: > Note that > > def s(x): return x if py3 else x.encode("...") > > fills the same purpose without changing Cython... OK, my request for a s"" prefix is not a strong one (though IMHO it is very convenient). What really bothers me at this point is the mapping of the 'str' builtin to PyBytes_Type. > Dag Sverre Seljebotn > -----Original Message----- > From: "Lisandro Dalcin" > Date: Friday, Sep 26, 2008 8:02 pm > Subject: Re: [Cython] isinstance and str (bytes/str/unicode again!) > To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > > On Fri, Sep 26, 2008 at 2:36 PM, Christian Heimes wrote: >>> Lisandro Dalcin wrote: >>>> If we do 'isinstance(obj, str)', then what should be the behavior in >>>> Py2 and Py3? In fact, this question is actually related to the map in >>>> builtin_types_table at Cython/Compiler/Builtin.py. Do it make sense to >>>> do the map like this >>>> >>>> bytes -> PyBytes_Type >>>> str -> PyString_Type >>>> unicode -> PyUnicode_Type >>> >>> It doesn't make sense for some applications of str. Python 3.0 uses >>> PyUnicode for identifiers (attribute names, function and class names >>> etc.) while Python 2.x uses PyString. You can use PyUnicode instances >>> for attributes in Python 2.x through obj.__dict__[u"key"] = key but you >>> cannot use PyBytes in 3.0. >>> >>> This mapping makes more sense for me: >>> >>> bytes:: >>> whatever Python uses to store binary data >>> Python 2.x: PyString >>> Python 3.x: PyBytes >>> >>> unicode:: >>> text data >>> PyUnicode for all Python versions >>> >>> str:: >>> >>> Python 2.x: PyString >>> Python 3.x: PyUnicode >> >>Perhaps I was not clear enough, but the mapping in my mind is just the >>mapping you proposed. >> >>In the Python side, 'str' sould be the the type Python uses for >>attributes and names. In the C side, that would be 'PyString_Type'. >>Of course, Cython has to #define PyString_Type to PyUnicode_Type for >>Py3 (currently, PyString_Type is #define'd as PyBytes_Type for Py3). >> >> >>> The suggestion follows my forward compatibility code for Python 2.6. In >>> Python 2.6 b"" and bytes are simple aliases for PyString. "from >>> __future__ import unicode_literals" turns "" into PyUnicode objects. >> >>The only gotcha I have with all this is that I do not have an easy way >>to write a string literal that matchs what Python uses for >>identifiers, in such a way that it can run in both Py2 and Py3. Let's >>take an example: Suppose I want to create a new type calling >>'type(name, bases, dict)', then I can write in Cython code: >> >># Note: Cython code inside a pyx file >>type( b'A', (), {}) # -> fails in Py3 >> >>type( u'A', (), {}) # -> fails in Py2 >> >>type( 'A', (), {}) # -> fails in Py2 or Py3 depending on __future__ import >> >>So I really believe Cython needs an extension, let say >> >>type( s'A', (), {}) # Success in Py2 and Py3! >> >>Then the 's' prefix build a string with the whatever type the Python >>runtime uses for identifiers. >> >>What to you think, Christian? Do my later comments make sense for you? >> >> >> >> >> >> >>-- >>Lisandro Dalc?n >>--------------- >>Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >>Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >>Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >>PTLC - G?emes 3450, (3000) Santa Fe, Argentina >>Tel/Fax: +54-(0)342-451.1594 >>_______________________________________________ >>Cython-dev mailing list >>Cython-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/cython-dev >> > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Sep 29 17:00:03 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Sep 2008 12:00:03 -0300 Subject: [Cython] embedding Cython and wiki link In-Reply-To: References: Message-ID: On Mon, Sep 29, 2008 at 3:40 AM, Amir wrote: > > I would like to create a simple embedded cython executable like the example in > the Python docs (5. Embedding Python in Another Application). I started with the > Cython wiki link pointing to David McNab's pyrex embedding tutorial but am not > sure how to proceed. Was wondering if anyone had done this with a recent Cython > version. Cython is primarily targeted to build extension modules, and not executables embeding Python. However, Cython could provide support for this, though perhaps that is not strongly needed, as things could be done more or less easy writting some C code making appropriate Python C-API calls (Py_Initialize, Py_Finalize, PyRun_SimpleString, etc.). > The python pxd's don't seem to have Py_Initialize, Py_Finalize, PySys_SetArgv > used in David's tutorial. I'm guessing there is a good reason for this. And I am > wondering if there is a way to use Cython.distutils to build the executable. > I have some custom support for building python executable in my mpi4py project. Look at http://code.google.com/p/mpi4py/source/browse/trunk/conf/mpidistutils.py . You will find some distutils command 'build_exe', 'install_exe', and 'clean_exe', and a Executable class (is just inherit from Extension, but do not extend it in any way). Of course, you will need to remove the MPI-configuration related stuff. Additionally, this was never tried on Windows, and likely will not work in that platform. > Any pointers to docs or an example application would be appreciated. > Feel free to contact me by private mail if you have trouble. Regards, > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ivanov.maxim at gmail.com Mon Sep 29 17:08:19 2008 From: ivanov.maxim at gmail.com (Max Ivanov) Date: Mon, 29 Sep 2008 19:08:19 +0400 Subject: [Cython] List of builtin types? Message-ID: Is there any list of builtin types which Cython supports in more optimized way than just plain "object"? From dalcinl at gmail.com Mon Sep 29 18:23:43 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 29 Sep 2008 13:23:43 -0300 Subject: [Cython] PATCH: population of builtin types for 'isinstance' optimization Message-ID: Rober, here you have a patch populating builtin types in Bultin.py and Symtab.py. The full testsuite pass with Py2.3.6 and Py2.5.2 and Py2.6 (recent trunk checkout). Additionally, you have attached a new test (should go to 'tests/run' dir) for isinstance optimization with builtin types. Some notes below * I deliberately added 'set' to the builtin types. If set is ever used in Cython sources, then that source will not compile in Python 2.3.x. Perhaps the 'set' and 'frozenset' builtins could be somewhat emulated in Py2.3.x with 'sets.Set' and 'set.InmutableSet' from the Python(2.3) stdlib. * The 'str' builtin is mapped to PyString_Type. In Py3, Cython #define's PyString_Type to PyBytes_Type, so this map is safe. Of course, I still believe that Cython should #define PyString_Type to PyUnicode_Type for Py3. But that discussion is for the future. Regards, and let me know if you ever push this. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: builtin_types.diff Type: text/x-patch Size: 3230 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080929/e109a148/attachment-0001.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: testisinstance.pyx Type: application/octet-stream Size: 1177 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20080929/e109a148/attachment-0001.obj From ivanov.maxim at gmail.com Mon Sep 29 19:16:49 2008 From: ivanov.maxim at gmail.com (Max Ivanov) Date: Mon, 29 Sep 2008 21:16:49 +0400 Subject: [Cython] [BUG] PyFrozenset_CheckExact instead of PyFrozenSet_CheckExact Message-ID: If you try to use frozenset inside your code, in .c file there will be "PyFrozenset_CheckExact" but in python2.5/setobject.h "PyFrozenSet_CheckExact" defined (notice "set" and "Set"). How could I define new macro in pyx file? I would like to workaround this bug by defining macro "PyFrozenset_CheckExact" which points to another macro: "PyFrozenSet_CheckExact" From dagss at student.matnat.uio.no Mon Sep 29 20:23:47 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 29 Sep 2008 20:23:47 +0200 Subject: [Cython] Asking for review: More result_code refactor etc. Message-ID: <48E11D33.9050004@student.matnat.uio.no> Buffer access towards complex types (treating them as a struct of two floats) are now up in -dagss. In order to get there I had to do a couple of dangerous things regarding result_code though. These patches are really small but someone with more knowledge may have more or less confidence in them than I: http://hg.cython.org/cython-dagss/rev/b2bcc32021fd http://hg.cython.org/cython-dagss/rev/a8a9e4c7d5bd Note that this should fix the CloneNode issue I talked about. (And all tests pass btw; yet this is the kind of thing that would cause horrendous failures in obscure cases which are likely not tested for...) Also in -dagss I've introduces NewTempExprNode, which is a transitionary class where the temp allocation is moved to code generation. Ideally all ExprNodes should descend from this one instead (and then it should be folded back into ExprNode to complete the migration). Some classes fail hard when they are changed to the new scheme. It works for BinOpNode which now uses it. This should allow partial migration. (One must make sure a class can work with the new temp allocation scheme before constructing them in a transform after analysis). -- Dag Sverre From robertwb at math.washington.edu Mon Sep 29 22:48:16 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Sep 2008 13:48:16 -0700 Subject: [Cython] [BUG] PyFrozenset_CheckExact instead of PyFrozenSet_CheckExact In-Reply-To: References: Message-ID: On Sep 29, 2008, at 10:16 AM, Max Ivanov wrote: > If you try to use frozenset inside your code, in .c file there will be > "PyFrozenset_CheckExact" but in python2.5/setobject.h > "PyFrozenSet_CheckExact" defined (notice "set" and "Set"). > > How could I define new macro in pyx file? I would like to workaround > this bug by defining macro "PyFrozenset_CheckExact" which points to > another macro: "PyFrozenSet_CheckExact" Hmm... perhaps we should #define these so that PyFrozenSet_CheckExact is always available. The best short term thing to do, if you want to compile against both 2.5 and < 2.5, I would make a .h file with the appropriate macro, then in your file you can put cdef extern from "file.h": pass From robertwb at math.washington.edu Mon Sep 29 22:52:26 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Sep 2008 13:52:26 -0700 Subject: [Cython] List of builtin types? In-Reply-To: References: Message-ID: No, and this is constantly evolving, though currently there isn't too much. See the bottom of http://hg.cython.org/cython-devel/file/3152381ccf89/Cython/Compiler/ Builtin.py for what we have now. It should be noted that Cython tests for the most obvious types at runtime, e.g. on conversion to a c int/double it uses the macro form if one has a python int/float, and to index into lists/tuples it checks types dynamically to see if it can use the macros as well. - Robert On Sep 29, 2008, at 8:08 AM, Max Ivanov wrote: > Is there any list of builtin types which Cython supports in more > optimized way than just plain "object"? > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Mon Sep 29 22:55:48 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Sep 2008 13:55:48 -0700 Subject: [Cython] embedding Cython and wiki link In-Reply-To: References: Message-ID: On Sep 28, 2008, at 11:40 PM, Amir wrote: > > I would like to create a simple embedded cython executable like the > example in > the Python docs (5. Embedding Python in Another Application). I > started with the > Cython wiki link pointing to David McNab's pyrex embedding tutorial > but am not > sure how to proceed. Was wondering if anyone had done this with a > recent Cython > version. > > The python pxd's don't seem to have Py_Initialize, Py_Finalize, > PySys_SetArgv > used in David's tutorial. I'm guessing there is a good reason for > this. Actually, there's not as far as I know. We should add them (could you want to submit a pxd file?) > And I am wondering if there is a way to use Cython.distutils to > build the executable. > > Any pointers to docs or an example application would be appreciated. I've never done it myself, but Lisandro's suggestions look good. - Robert From robertwb at math.washington.edu Mon Sep 29 23:53:41 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Sep 2008 14:53:41 -0700 Subject: [Cython] docs.cython.org In-Reply-To: <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> Message-ID: <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> On Sep 29, 2008, at 7:35 AM, Ondrej Certik wrote: > On Mon, Sep 29, 2008 at 4:28 PM, Gabriel Gellner > wrote: >>> Could this please be hosted at docs.cython.org? >>> >>> I use your sphinx docs all the time and it is just excellent! You've >>> done really, really great job. >>> >> I am not sure what is involved in this, but I would be happy to >> have them >> hosted on the cython site, I use my site just because it was easy, >> and I >> wasn't sure what the response would be. >> >> Do the server admins think this is feasible? If we want I could >> have the built >> html pages put under the hg.cython versioning, and we could just >> do a url >> rename to point to this? >> >> Anyway, I am open to any ideas. > > Technically I think just Robert or William needs to setup dns > redirection and add a new rule to apache. I think this could be done, or would it be better to host the docs on cython.org itself? > Socially, let's see if other people also like your docs as the default > docs for Cython. I do! Given that there aren't any "default" docs now, unless there's objection (that would be a real suprise) let's use them now. - Robert From ggellner at uoguelph.ca Mon Sep 29 23:59:31 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Mon, 29 Sep 2008 17:59:31 -0400 Subject: [Cython] docs.cython.org In-Reply-To: <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> Message-ID: <20080929215931.GA11292@encolpuis> > I think this could be done, or would it be better to host the docs on > cython.org itself? > I have no opinion, they both seem equally fine to me. > > Socially, let's see if other people also like your docs as the default > > docs for Cython. > > I do! Given that there aren't any "default" docs now, unless there's > objection (that would be a real suprise) let's use them now. > Great! Someone just tell me where to upload the generated docs, and I will do so. I can also have a pre built pdf linked to in the html docs. Gabriel From amirnntp at gmail.com Tue Sep 30 01:01:53 2008 From: amirnntp at gmail.com (Amir) Date: Mon, 29 Sep 2008 23:01:53 +0000 (UTC) Subject: [Cython] embedding Cython and wiki link References: Message-ID: Robert Bradshaw writes: > > On Sep 28, 2008, at 11:40 PM, Amir wrote: > > > > > I would like to create a simple embedded cython executable like the > > example in > > the Python docs (5. Embedding Python in Another Application). I > > started with the > > Cython wiki link pointing to David McNab's pyrex embedding tutorial > > but am not > > sure how to proceed. Was wondering if anyone had done this with a > > recent Cython > > version. > > > > The python pxd's don't seem to have Py_Initialize, Py_Finalize, > > PySys_SetArgv > > used in David's tutorial. I'm guessing there is a good reason for > > this. > > Actually, there's not as far as I know. We should add them (could you > want to submit a pxd file?) I don't know enough yet about Python internals to submit a pxd. I asked the question out of curiosity. I was reading the Python docs to understand the annotated code generated by Cython and noticed the section on embedding python, which didn't seem to have a counterpart in Cython. Thanks, Amir From greg.ewing at canterbury.ac.nz Tue Sep 30 01:37:15 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 30 Sep 2008 11:37:15 +1200 Subject: [Cython] How to return char* In-Reply-To: References: <48E0E16E.8020003@student.matnat.uio.no> Message-ID: <48E166AB.20408@canterbury.ac.nz> Max Ivanov wrote: >>1) You want to use char* because it is feels "faster" than Python >>objects: > > That was the reason > >>No, you're better off with Python objects, char* is >>fundamentally something different. > > And this is the answer. Yep. You have to allocate memory for the string one way or another, so you might as well let Python manage the details for you. Note that you can get direct access to the chars inside a Python string any time you want at no extra cost. -- Greg From robertwb at math.washington.edu Tue Sep 30 02:41:50 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 29 Sep 2008 17:41:50 -0700 Subject: [Cython] Call for help: official C-API 2 to 3 porting guide In-Reply-To: References: Message-ID: <5186651C-6A0C-4D72-95DF-CCEF28915B20@math.washington.edu> On Sep 27, 2008, at 6:13 AM, Christian Heimes wrote: > Hello fellow developers! > > Benjamin is working on a C-API 2 to 3 porting guide for the official > Python docs. I know that you - the Cython developers - have spent lots > of time on making Cython compatible with 2.3+ as well as 3.0. Could > you > please assist Benjamin with his task and give him some hints? We are > especially interested in a list of gotchas and tricks how to write > cross > version compatible C code. > > Thanks in advance! Stefan and Lisandro did most of the work on this front, but much of what we did was use a lot of #defines to point things to the right places, and write (inline) wrapper functions that wrapped the API where things differed. While it works very well for our target (machine generated C) I don't know how applicable it would be to his domain. - Robert From fperez.net at gmail.com Tue Sep 30 08:48:21 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 29 Sep 2008 23:48:21 -0700 Subject: [Cython] docs.cython.org In-Reply-To: <20080929215931.GA11292@encolpuis> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> <20080929215931.GA11292@encolpuis> Message-ID: On Mon, Sep 29, 2008 at 2:59 PM, Gabriel Gellner wrote: > Great! > Someone just tell me where to upload the generated docs, and I will do so. > I can also have a pre built pdf linked to in the html docs. This is great, many thanks to all involved! Just one minor suggestion/request: could you do something like what we have for ipython: http://ipython.scipy.org/moin/Documentation where we keep persistent URLs of the docs for each release (well, only 0.9.1 so far, which was the first on the new system, but we'll keep them all): http://ipython.scipy.org/doc/rel-0.9.1/html (and also a pdf link) and one for a nightly updated copy from our bzr trunk: http://ipython.scipy.org/doc/manual/html This gives everyone easy access to the freshest docs, as well as a reliable reference to how things were for any particular release, which can be useful later on. Cheers, f From dagss at student.matnat.uio.no Tue Sep 30 09:34:05 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 30 Sep 2008 09:34:05 +0200 Subject: [Cython] docs.cython.org In-Reply-To: <20080929215931.GA11292@encolpuis> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> <20080929215931.GA11292@encolpuis> Message-ID: <48E1D66D.7080408@student.matnat.uio.no> Gabriel Gellner wrote: >> I think this could be done, or would it be better to host the docs on >> cython.org itself? >> > I have no opinion, they both seem equally fine to me. > >>> Socially, let's see if other people also like your docs as the default >>> docs for Cython. >> I do! Given that there aren't any "default" docs now, unless there's >> objection (that would be a real suprise) let's use them now. >> > Great! > Someone just tell me where to upload the generated docs, and I will do so. > I can also have a pre built pdf linked to in the html docs. An opinion: Ideally (to follow Fernando's request, which I agree with) a) docs is moved to a subdirectory of the normal cython repo b) A script to build them is shipped with Cython (setup.py could do this I suppose) b) On each release, Robert runs a script on the server which generates those html and docs in a subdirectory for the release, and updates the current link to the docs. -- Dag Sverre From ggellner at uoguelph.ca Tue Sep 30 17:13:29 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Tue, 30 Sep 2008 11:13:29 -0400 Subject: [Cython] docs.cython.org In-Reply-To: <48E1D66D.7080408@student.matnat.uio.no> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> <20080929215931.GA11292@encolpuis> <48E1D66D.7080408@student.matnat.uio.no> Message-ID: <20080930151329.GA15724@encolpuis> > > An opinion: Ideally (to follow Fernando's request, which I agree with) > > a) docs is moved to a subdirectory of the normal cython repo > b) A script to build them is shipped with Cython (setup.py could do this > I suppose) > c) On each release, Robert runs a script on the server which generates > those html and docs in a subdirectory for the release, and updates the > current link to the docs. > b) already exists, there is a top level makefile that will generate anything people want. Just get the repo (make sure you have Sphinx installed, which is in PyPi) and run 'make html', or 'make latex', tell me if this doesn't work for anybody. c) currently this approach would be far to slow, the docs need *a lot* of work to catch up to current Cython, as well as editing to make them feel less like a collection of disparate write ups (which in fact they are . . .). I am committed to this project, and will set aside a couple of hours a week to make this happen for the foreseeable future. As far as having links to historical versions, I think this is an important idea, but a bit premature for the current state of the docs, which are really of alpha quality (though better than anything else currently, in my opinion). To remind people what I have done: I combed over all the links that were given in the old Cython webpage for documentation, which was composed mostly of Greg Ewing's Pyrex manual and a bunch of Wiki cookbook like entries, FAQ, etc. I found this annoying so I just compiled everything together and Switched all mention of Pyrex to Cython. I have written a paragraph here and a paragraph there, but currently that is basically what we have. (So once again I think it is important to thank Greg for all his amazing work! I am going to add a section into the docs mentioning this history so that more credit is given than just being one of the authors.) Before we are committed to keeping historical versions online I would like at least to have reviewed each section, and made sure the prose and voice is consistent, and have much more of a tutorial to help beginning Cythoneers get up to speed. I think this will realistically take at least couple of months (given nobody else gets involved). In summary I think the current method of having a separate repo for the docs and a link to the generated docs being made from the main Cython page is the best, until we have a real release candidate ready. Once this happens we can think of including the docs into the main Cython repo, and having historical pointers. Until then I think the docs will be changing to rapidly and would make this a bit of a logistic hassle. So these are my opinions, but I, as always, am happy to go with what we all decide to do. Gabriel From dagss at student.matnat.uio.no Tue Sep 30 17:22:30 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 30 Sep 2008 17:22:30 +0200 Subject: [Cython] docs.cython.org In-Reply-To: <20080930151329.GA15724@encolpuis> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> <20080929215931.GA11292@encolpuis> <48E1D66D.7080408@student.matnat.uio.no> <20080930151329.GA15724@encolpuis> Message-ID: <48E24436.9040606@student.matnat.uio.no> Gabriel Gellner wrote: > > In summary I think the current method of having a separate repo for the docs and > a link to the generated docs being made from the main Cython page is the best, > until we have a real release candidate ready. Once this happens we can think > of including the docs into the main Cython repo, and having historical > pointers. Until then I think the docs will be changing to rapidly and would > make this a bit of a logistic hassle. > Very good points. +1. Dag Sverre From tgrav at mac.com Tue Sep 30 17:24:23 2008 From: tgrav at mac.com (Tommy Grav) Date: Tue, 30 Sep 2008 11:24:23 -0400 Subject: [Cython] docs.cython.org In-Reply-To: <20080930151329.GA15724@encolpuis> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> <20080929215931.GA11292@encolpuis> <48E1D66D.7080408@student.matnat.uio.no> <20080930151329.GA15724@encolpuis> Message-ID: On Sep 30, 2008, at 11:13 AM, Gabriel Gellner wrote: > > To remind people what I have done: I combed over all the links that > were given > in the old Cython webpage for documentation, which was composed > mostly of Greg > Ewing's Pyrex manual and a bunch of Wiki cookbook like entries, FAQ, > etc. I > found this annoying so I just compiled everything together and > Switched all > mention of Pyrex to Cython. I have written a paragraph here and a > paragraph > there, but currently that is basically what we have. (So once again > I think it > is important to thank Greg for all his amazing work! I am going to > add a > section into the docs mentioning this history so that more credit is > given than just being one of the authors.) I am so glad that someone finally did the pyrex -> cython replacement :) I have been following the mailing list for months and have on and off been trying to apply cython to my code and the pyrex/cython mix have always been a very, very confusing part for me. Thanks to everyone for this great project. Cheers Tommy From uschmitt at mineway.de Tue Sep 30 19:13:07 2008 From: uschmitt at mineway.de (Uwe Schmitt) Date: Tue, 30 Sep 2008 19:13:07 +0200 Subject: [Cython] quick guide moved ??? Message-ID: Hi, does anybody know what happened to "michaels quick guide to pyrex" at http://iopen.net/pyrex-guide/ ??? Greetings, Uwe From robertwb at math.washington.edu Tue Sep 30 19:35:15 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 30 Sep 2008 10:35:15 -0700 Subject: [Cython] docs.cython.org In-Reply-To: <48E1D66D.7080408@student.matnat.uio.no> References: <20080927144913.GA20932@encolpuis> <85b5c3130809290630w4c1055dma2ce4a4fbb7d5831@mail.gmail.com> <20080929142813.GA8804@encolpuis> <85b5c3130809290735r62bb6434oe725c8eb2df24f2b@mail.gmail.com> <476444D5-5A11-444B-879D-867BFE0A34BE@math.washington.edu> <20080929215931.GA11292@encolpuis> <48E1D66D.7080408@student.matnat.uio.no> Message-ID: On Sep 30, 2008, at 12:34 AM, Dag Sverre Seljebotn wrote: > Gabriel Gellner wrote: >>> I think this could be done, or would it be better to host the >>> docs on >>> cython.org itself? >>> >> I have no opinion, they both seem equally fine to me. >> >>>> Socially, let's see if other people also like your docs as the >>>> default >>>> docs for Cython. >>> I do! Given that there aren't any "default" docs now, unless there's >>> objection (that would be a real suprise) let's use them now. >>> >> Great! >> Someone just tell me where to upload the generated docs, and I >> will do so. >> I can also have a pre built pdf linked to in the html docs. > > An opinion: Ideally (to follow Fernando's request, which I agree with) > > a) docs is moved to a subdirectory of the normal cython repo See below. > b) A script to build them is shipped with Cython (setup.py could do > this > I suppose) If Sphinx is detected, and it's fast. I certainly don't want it to be a requirement. > c) On each release, Robert runs a script on the server which generates > those html and docs in a subdirectory for the release, and updates the > current link to the docs. I agree with Gabriel that at this point we are simply trying to make docs for Cython, the difference between versions being miniscule compared to the incompleteness that has been there. I am also excited for rapid progress on this front and if it is decoupled with the codebase to make this easier, it is certainly worth it for the moment. When things stabilize, we'll move over to the new model. - Robert From robertwb at math.washington.edu Tue Sep 30 22:16:14 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 30 Sep 2008 13:16:14 -0700 Subject: [Cython] embeding functions signatures almost done In-Reply-To: <48D209C2.6060600@student.matnat.uio.no> References: <62571.193.157.228.10.1221681251.squirrel@webmail.uio.no> <48D20902.8040506@student.matnat.uio.no> <48D209C2.6060600@student.matnat.uio.no> Message-ID: On Sep 18, 2008, at 12:56 AM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Lisandro Dalcin wrote: >>> On Wed, Sep 17, 2008 at 4:54 PM, Dag Sverre Seljebotn >>> wrote: >>> >>>> Lisandro wrote: >>>> >>>>> I've almost finished the transform in charge of embeding functions >>>>> signatures in docstrings. I want to ask your about some issues >>>>> I need >>>>> to handle to finish it. >>>>> >>>>> * Position in the transform pipeline: I'm curently inserting the >>>>> auto-signature transform after the 'ResolveOptions' one in the >>>>> transform pipeline. Dag, tell me if this is OK. >>>>> >>>> For this, anywhere where you have the needed information available >>>> is ok. >>>> I guess you read the type info from nodes then, and not the >>>> "parsed" >>>> info >>>> that is available later? (that could make sense and is not >>>> necesarrily a >>>> bad sign) >>>> >>>> >>> >>> I did not understand you here, but anyway it seems to work at the >>> place I'm inserting it. Or perhaps I could get more info (like >>> return >>> type in 'cpdef' funcions) if I insert the transform near the end? >>> >> Sorry, my fault. Basically, when I wrote my description I meant to >> imply (and should have clarified better) that the >> AnalyseDeclarationsTransform interprets the nodes containing the type >> declarations into a purer Python representation (see PyrexTypes.py). >> However for what you are doing, walking the CSimpleTypeNodes might be >> just as good. >> >> (Inserting the transform after AnalyseDeclarationsTransform makes >> certain things > (I need a countdown timer on my send button...) > > I meant to say: Inserting the transform after > AnalyseDeclarationsTransform makes certain things more difficult -- > because for cpdef functions, a "python version node" is created and > the > docstring copied so you would need to change it in multiple locations > and so on. So actually, what you have done will be more convenient and > make this more isolated and less likely to break if the cpdef > implementation is changed. I went ahead and moved it to after AnalyseDeclarationsTransform because it wasn't picking up some of the types correctly after some other changes I made (specifically, the parser no longer keeps track of a list of all types seen, but this means it may mistake a argument name for a type name in a function declaration, which is corrected later). Setting the signature is a little less elegant, but it does make some of the other code quite a bit cleaner. - Robert From robertwb at math.washington.edu Tue Sep 30 22:18:53 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 30 Sep 2008 13:18:53 -0700 Subject: [Cython] PATCH: population of builtin types for 'isinstance' optimization In-Reply-To: References: Message-ID: <8DCFF676-7C7B-41FA-9129-1BD0EF9D4981@math.washington.edu> On Sep 29, 2008, at 9:23 AM, Lisandro Dalcin wrote: > Rober, here you have a patch populating builtin types in Bultin.py and > Symtab.py. The full testsuite pass with Py2.3.6 and Py2.5.2 and Py2.6 > (recent trunk checkout). Additionally, you have attached a new test > (should go to 'tests/run' dir) for isinstance optimization with > builtin types. Some notes below > > * I deliberately added 'set' to the builtin types. If set is ever used > in Cython sources, then that source will not compile in Python 2.3.x. > Perhaps the 'set' and 'frozenset' builtins could be somewhat emulated > in Py2.3.x with 'sets.Set' and 'set.InmutableSet' from the Python(2.3) > stdlib. I think this is fine. Currently, one can use builtins from higher versions of python, it just pushes the minimum requirement up (just as with .py files). > * The 'str' builtin is mapped to PyString_Type. In Py3, Cython > #define's PyString_Type to PyBytes_Type, so this map is safe. Of > course, I still believe that Cython should #define PyString_Type to > PyUnicode_Type for Py3. But that discussion is for the future. > > Regards, and let me know if you ever push this. Done, and passes all tests. Thanks. - Robert From robertwb at math.washington.edu Tue Sep 30 22:53:30 2008 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 30 Sep 2008 13:53:30 -0700 Subject: [Cython] quick guide moved ??? In-Reply-To: References: Message-ID: No idea. I guess http://mudskipper.ca/cython-doc/ is coming just in time. - Robert On Sep 30, 2008, at 10:13 AM, Uwe Schmitt wrote: > > Hi, > > does anybody know what happened > to "michaels quick guide to pyrex" > at http://iopen.net/pyrex-guide/ ??? > > Greetings, Uwe > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev